Artifacts and store¶
Everything the system learns is a sealed, digest-pinned artifact with provenance (RFC-0004); runs and artifacts have store ids that configs reference instead of paths (RFC-0005 part 2). Scorecards and active versions are the formats bench and serve exchange. User guides: learning, the store, benchmarking, deployment.
Artifacts and the split guard¶
artifact ¶
Derived artifacts (RFC-0004): everything the system learns is a versioned, digest-pinned directory with provenance.
The trace store is the single source of truth. Action memories, critics, value models, fine-tuned weights, datasets,
judges are all derived from traces (or from external data sources) by a builder, and each one carries an
artifact.json manifest: what it is, which role it plays, a content digest, and which runs / splits it was built
from. Three roles, three trust paths:
- behaviour changes what the robot does (memory, critic, noise policy, weights): only the RSI gate activates it;
- instrument measures or labels (success judge, progress model, world model): trusted only after calibration against ground truth, never the gate's verdict;
- proposal only changes what gets proposed (lessons, skill index); curated by outcome votes.
One split guard for every builder: development splits (search, search_val) and calibrated external sources may be read; gate and held-out data never. The gate re-checks the provenance of every artifact a candidate references, so a leak is caught even if a builder forgot the guard.
FORBIDDEN_SPLIT_PREFIXES
module-attribute
¶
ArtifactKind
module-attribute
¶
ArtifactKind = Literal['dataset', 'action_memory', 'value_model', 'noise_policy', 'policy_weights', 'policy_params', 'success_judge', 'progress_model', 'world_model', 'lesson_book', 'skill_library', 'other']
ArtifactRole
module-attribute
¶
ArtifactError ¶
Bases: ValueError
Missing / corrupted manifest, digest mismatch, or forbidden data in an artifact's provenance.
SplitLeak ¶
Bases: ArtifactError
A builder or a candidate would use gate / held-out data.
SourceRef ¶
Bases: Contract
One input of a build: a run directory (kind 'run') or an external data source (kind 'external').
Provenance ¶
ArtifactManifest ¶
check_splits ¶
Raise if any requested split is gate / held-out data (call this before reading anything).
synthetic_run ¶
A run produced inside a learned model (world-model pre-screening), not by a real or simulated environment.
find_runs ¶
Run directories (those with an evallog.json) given run directories, store ids or roots to search.
development_runs ¶
development_runs(roots: Iterable[str | Path], splits: Iterable[str] = DEV_SPLITS, *, strict: bool = False) -> list[tuple[Path, SourceRef]]
The run directories under roots whose split is in splits, with their provenance records.
strict=True raises on a run of any other split (a CLI given an explicit list of runs); strict=False skips them
(a builder scanning a campaign directory, which also contains gate and held-out runs). Ingested external data
(split external:splits.
write_manifest ¶
write_manifest(root: str | Path, *, kind: ArtifactKind, role: ArtifactRole, provenance: Provenance, name: str = '', metrics: dict[str, Any] | None = None, mutable: bool = False, calibration: dict[str, Any] | None = None) -> ArtifactManifest
Seal a built directory: compute its digest and write artifact.json (written last, after every other file).
manifest_root ¶
The artifact directory containing path (the path itself or one of its first max_up parents).
verify ¶
Load and check an artifact: manifest present, digest intact (unless mutable), no gate / held-out sources.
referenced_artifacts ¶
Every artifact a config (patch) points to, verified: {artifact dir: manifest}. Store ids are resolved (and their digest must match the content); strings that are not existing paths (repo ids, factory refs) are ignored; an existing path inside an artifact must verify or this raises.
Store¶
store ¶
Run and artifact store (hylo.store/v0.1): global ids, references instead of paths, deletion protection, sync.
A store is a directory tree on one machine (this machine's profile says where: hylo store init PATH, else $HYLO_STORE,
else ./runs). Runs (directories with an evallog.json) and sealed artifacts (an artifact.json) anywhere under it get
an id:
run:<run_id> run ids are unique (they end in a random suffix)
artifact:<name>@<digest> name from the manifest, digest = content digest (a prefix of >= 8 hex works)
artifact:<name> allowed when exactly one artifact has that name; the run manifest pins it
Configs reference artifacts by id (critic: {path: artifact:l10-v1@3d8d6096f7111acc}); the registry resolves ids to
this machine's paths when it builds a component, after the experiment digest is taken, so the same config means the
same content on every machine. Learner job inputs accept run and artifact ids too.
The index (hylo store rm refuses to delete anything still referred to unless forced, and a forced
or plain removal leaves a tombstone, so a dangling reference says what happened instead of "not found".
hylo store push|pull HOST:PATH [ID ...] copies runs and artifacts between machines with rsync (runs from the
robot computer to the workstation, artifacts the other way). Sealed artifacts are never overwritten: the same path
holding a different digest on the other side is a conflict and nothing is copied.
Store ¶
id_for ¶
The id of the run or artifact at path, if it is one inside this store.
ref_for ¶
An id for path when it is a sealed artifact (or a run) in this store, else the path itself: what a
generator writes into a candidate patch.
remove ¶
Delete a run or artifact. Refused while anything refers to it (an artifact built from it, a campaign's archive or ledger) unless forced; either way a tombstone records what was removed and why.
resolve ¶
obj with every store id replaced by this machine's absolute path.
pin ¶
The ids obj references, each with its full id and current path (for a run's manifest).
closure ¶
The entries and, recursively, what they were built from.
Index ¶
Location ¶
StoreError ¶
Bases: ValueError
Unknown or ambiguous id, a removed entry, a refused deletion or a sync conflict.
resolve_refs ¶
Replace store ids in a component spec with paths (a no-op without ids, so no store is opened).
adopt ¶
adopt(path: str | Path, *, kind: str, role: str, name: str = '', sources: Sequence[str] = (), store: Store | None = None) -> dict[str, Any]
Seal a directory that predates manifests. Its sources are what the caller states (run directories or run ids); none means unknown provenance, which is recorded, and which the RSI gate refuses for behaviour artifacts.
sync ¶
sync(local: Store, other: str | Location, *, direction: Literal['push', 'pull'], ids: Sequence[str] | None = None, with_sources: bool = False, dry_run: bool = False) -> dict[str, Any]
Copy runs and artifacts between this machine's store and another (push: here -> there, pull: there -> here).
plan_sync ¶
plan_sync(src: dict[str, Entry], dst: dict[str, Entry], ids: Sequence[str] | None, with_sources: bool) -> dict[str, list[Any]]
What a sync copies: selected entries not on the other side (runs and mutable artifacts also when present, to bring them up to date), and conflicts (the same path holding something else).
Scorecards¶
scorecard ¶
Scorecards (hylo.scorecard/v0.1): the result of running one policy under one benchmark protocol.
Written by hylo-bench (hylo bench run), read by leaderboards, the catalog, the playground and the project site, so
the format lives here. A scorecard is self-contained: the protocol and policy it measured, the headline metrics
with a 95% interval on success, per-task success, and the per-episode outcomes (episodes, keyed
"task#seed") that let two scorecards of the same protocol be compared pairwise.
{"schema_version": "hylo.scorecard/v0.1", "protocol": "libero-10", "protocol_version": "1",
"protocol_digest": "...", "policy": "smolvla", "agent": {"type": ..., "params": ...},
"metrics": {"success_mean": 0.54, "steps_mean": 312.0, ...}, "success_ci95": [0.40, 0.67],
"per_task": {"libero_10/0": {"success": 0.6, "n": 5}, ...}, "episodes": {"libero_10/0#30000": 1.0, ...},
"run_dir": "runs/...", "config_digest": "...", "created_at": ...}
TaskScore ¶
Scorecard ¶
wilson_interval ¶
95% Wilson score interval for a success rate (well behaved at 0 and 1, unlike the normal approximation).
Active versions¶
deployment ¶
The active version of an improvement campaign, as a file (hylo.active/v0.1).
The campaign (hylo-evolve) writes <campaign>/active.json whenever its active version changes: at start, on a
promotion, on an approval, on a rollback. Anything that deploys the campaign's current behaviour (the policy server
in hylo-serve) reads only this file, so the deploying side needs neither the improvement code nor the candidate
registry, and the file can be copied to another machine.
{"schema_version": "hylo.active/v0.1", "campaign": "libero10-steering", "version": "cand-1a2b3c4d5e",
"agent": {"type": "lerobot.steered", "params": {...}}, "patch": {"agent.params.kp": 2.0},
"lineage": ["cand-1a2b3c4d5e"], "updated_at": 1793400000.0}
version is None while the base configuration is active. agent is the fully resolved agent spec (base config
with the lineage's patches applied), so a reader builds it with the registry and nothing else.