hylo-bench

Hylo benchmarking: fixed evaluation protocols, scorecards, leaderboards and pairwise comparison (hylo bench)

Repository hylo-bench, packages/hylo-bench. Licence: Apache-2.0. Depends on: hylo-core, hylo-eval, pyyaml>=6.

Registered entry points

Group	Name	Object
commands	bench	hylo.bench.cli:main

Module	Summary
hylo.bench	Benchmarking: run a policy under a fixed protocol and get a scorecard (core.scorecard) for a leaderboard
hylo.bench.cli	hylo bench: benchmark protocols, scorecards and leaderboards
hylo.bench.leaderboard	Leaderboards: scorecards of one protocol (version and digest), ranked by success
hylo.bench.protocol	Benchmark protocols (hylo.protocol/v0.1): everything about an evaluation except the policy
hylo.bench.run	Run a policy under a protocol and write its scorecard; compare two scorecards pairwise

hylo.bench

Benchmarking: run a policy under a fixed protocol and get a scorecard (core.scorecard) for a leaderboard.

The evaluation engine is hylo-eval; this package fixes what is measured (protocols), records it (scorecards) and ranks and compares results (leaderboards, paired comparison).

hylo.bench.cli

hylo bench: benchmark protocols, scorecards and leaderboards.

hylo bench list                                         protocols that ship with hylo-bench
hylo bench show PROTOCOL                                its definition and digest
hylo bench run PROTOCOL --policy SPEC --name NAME [--out DIR] [--set env.remote.python=...] [--workers N]
                                                      run a policy: runs in OUT/runs, OUT/scorecards/PROTOCOL/NAME.json
hylo bench leaderboard [DIR ...] [--json]               rank the scorecards found under DIR (default: scorecards)
hylo bench compare CANDIDATE.json BASELINE.json         paired one-sided McNemar on shared episodes

SPEC is an agent spec as JSON, an experiment config YAML (its agent is used) or a plugin name.

main

main(argv: Sequence[str]) -> int

benchmark protocols, scorecards and leaderboards

hylo.bench.leaderboard

Leaderboards: scorecards of one protocol (version and digest), ranked by success.

collect

collect(root: str | Path) -> list[Scorecard]

boards

boards(cards: list[Scorecard]) -> dict[tuple[str, str, str], list[Scorecard]]

(protocol, version, digest) -> scorecards, best first. Different digests are different boards.

markdown

markdown(cards: list[Scorecard]) -> str

hylo.bench.protocol

Benchmark protocols (hylo.protocol/v0.1): everything about an evaluation except the policy.

A protocol fixes the benchmark and split (which tasks and init states), the environment, the safety limits, the trace options, the timing and the seed. The policy is the only free input, so two scorecards of the same protocol digest are comparable, episode by episode. timing decides whether the world waits for the policy (sync) or keeps moving while it thinks (realtime: what a workcell runs). Where the simulator runs (env.remote) is not measured.

Protocols ship with this package (protocols/*.yaml, by name) or are given as a YAML path. Changing anything measured is a new version; workers and description are not measured and do not change the digest.

PROTOCOL_SCHEMA module-attribute

PROTOCOL_SCHEMA = 'hylo.protocol/v0.1'

Protocol

Bases: Contract

schema_version class-attribute instance-attribute

schema_version: Literal['hylo.protocol/v0.1'] = PROTOCOL_SCHEMA

name instance-attribute

name: str

version instance-attribute

version: str

description class-attribute instance-attribute

description: str = ''

benchmark instance-attribute

benchmark: BenchmarkRef

env instance-attribute

env: EnvRef

safety class-attribute instance-attribute

safety: dict[str, JsonAny] = Field(default_factory=dict)

trace class-attribute instance-attribute

trace: TraceOptions = Field(default_factory=TraceOptions)

timing class-attribute instance-attribute

timing: Timing = Field(default_factory=Timing)

seed class-attribute instance-attribute

seed: int = 0

workers class-attribute instance-attribute

workers: int = Field(default=1, ge=1)

requires class-attribute instance-attribute

requires: tuple[str, ...] = ()

digest

digest() -> str

experiment

experiment(agent: ComponentRef | dict[str, Any], policy: str, *, output_dir: str = 'runs') -> ExperimentConfig

builtin_protocols

builtin_protocols() -> dict[str, Path]

load_protocol

load_protocol(name_or_path: str) -> Protocol

hylo.bench.run

Run a policy under a protocol and write its scorecard; compare two scorecards pairwise.

policy_spec

policy_spec(text: str) -> dict[str, Any]

An agent spec from JSON ({"type", "params"}), an experiment config YAML (its agent), or a plugin name.

scorecard

scorecard(protocol: Protocol, policy: str, cfg: ExperimentConfig, result: EvalResult, *, notes: str | None = None) -> Scorecard

run_protocol

run_protocol(protocol: Protocol, agent: dict[str, Any], policy: str, *, out: str | Path = '.', overrides: dict[str, Any] | None = None, notes: str | None = None, progress: Callable[[dict], None] | None = None) -> tuple[Scorecard, Path]

Run agent under protocol; runs go to OUT/runs, the scorecard to OUT/scorecards//.json.

overrides may only change how the run executes (env.remote.*, workers, output_dir): anything else would change what is measured, and the scorecard would no longer belong to this protocol.

compare

compare(candidate: Scorecard, baseline: Scorecard) -> PairedComparison

One-sided exact McNemar on the episodes both scorecards share (H1: candidate succeeds more often).