hylo-eval¶
Hylo evaluation engine: benchmark contract and splits, scorers, paired statistics and gate tests, evaluation runner, experiments
Repository hylo-harness, packages/hylo-eval. Licence: Apache-2.0. Depends on: hylo-core.
Registered entry points
| Group | Name | Object |
|---|---|---|
| benchmarks | testing/reach-v0 |
hylo.eval.builtin:testing_benchmark |
| gate_tests | mcnemar |
hylo.eval.stats:McNemar |
| gate_tests | paired_permutation |
hylo.eval.stats:PairedPermutation |
| gate_tests | sign |
hylo.eval.stats:SignTest |
| scorers | safety_interventions |
hylo.eval.scoring:SafetyInterventionScorer |
| scorers | steps |
hylo.eval.scoring:StepsScorer |
| scorers | success |
hylo.eval.scoring:SuccessScorer |
| Module | Summary |
|---|---|
hylo.eval |
Hylo benchmark layer |
hylo.eval.benchmark |
Moved to hylo.core.benchmark (the benchmarks contract lives in hylo-core); kept as an alias |
hylo.eval.builtin |
The test benchmark "testing/reach-v0" over the test double environment (hylo.testing): for unit tests |
hylo.eval.conformance |
Conformance for gate-test plugins (hylo.eval.stats.GateTest) |
hylo.eval.experiment |
Build components from an ExperimentConfig through the plugin registry and run it |
hylo.eval.runner |
EvaluationRunner: runs a benchmark split through the kernel, then scores from the trace |
hylo.eval.scoring |
Scorers read the trace only; they never touch the Agent or the Environment |
hylo.eval.stats |
Gate tests (plugin group "gate_tests"): the paired statistic the improvement protocol uses to decide |
hylo.eval¶
Hylo benchmark layer.
META
module-attribute
¶
META = {'contracts': {'benchmarks': 'hylo.benchmark/v0.1', 'scorers': 'hylo.scorer/v0.1', 'gate_tests': 'hylo.gate_test/v0.1'}, 'maturity': 'implemented', 'plugins': {'gate_tests/paired_permutation': {'maturity': 'unverified', 'notes': 'not yet run in a campaign'}, 'gate_tests/sign': {'maturity': 'unverified', 'notes': 'not yet run in a campaign'}}}
hylo.eval.benchmark¶
Moved to hylo.core.benchmark (the benchmarks contract lives in hylo-core); kept as an alias.
standard_splits ¶
Disjoint seed ranges: train 0.., validation 10000.., heldout 20000..
rsi_splits ¶
rsi_splits(n_states: int = 50, bounds: tuple[int, int, int] = (20, 30, 40), replicates: int = 1, gate_looks: int = 1) -> dict[str, tuple[int, ...]]
RFC-0002 split of init-state indices: search (diagnosis), search_val (selection), gate (confirmation), heldout (report only). Replicate r reuses init state (seed % n_states) with new policy noise (seed + r * n_states). gate_looks > 1 adds gate_r1 .. gate_r{looks-1}: further noise replicates of the gate init states, evaluated only when an earlier look is inconclusive (group-sequential gate).
hylo.eval.builtin¶
The test benchmark "testing/reach-v0" over the test double environment (hylo.testing): for unit tests only. Real benchmarks live in hylo-bench.
testing_benchmark ¶
testing_benchmark(tags: list[str] | None = None, train: int = 10, validation: int = 20, heldout: int = 20, protocol: str = 'standard', gate_looks: int = 1) -> Benchmark
hylo.eval.conformance¶
hylo.eval.experiment¶
Build components from an ExperimentConfig through the plugin registry and run it.
Only core and eval are imported here; envs, agents and memory backends arrive as plugins.
run_experiment ¶
run_experiment(cfg: ExperimentConfig, *, split: str | None = None, run_root: str | Path | None = None, label: str | None = None, agent: Agent | None = None, progress: Callable[[dict], None] | None = None) -> EvalResult
hylo.eval.runner¶
EvaluationRunner: runs a benchmark split through the kernel, then scores from the trace.
Sequential in v0.1. Sharding / resume follow the vla-eval SQLite work-queue pattern later.
EvalResult
dataclass
¶
score_run ¶
score_run(run_dir: Path, log_samples: list[dict[str, Any]], tasks: Mapping[str, Any], scorers: Sequence[Scorer]) -> list[dict[str, Any]]
evaluate ¶
evaluate(benchmark: Benchmark, split: str, *, env_factory: Callable[[], Environment], agent: Agent | None = None, agent_factory: Callable[[], Agent] | None = None, run_root: str | Path, safety: Mapping[str, Any] | None = None, scorers: Sequence[Scorer] = DEFAULT_SCORERS, manifest: Mapping[str, Any] | None = None, run_id: str | None = None, record_privileged: bool = True, workers: int = 1, progress: Callable[[dict[str, Any]], None] | None = None, image_every: int = 1, timing: Timing | None = None) -> EvalResult
Run every TaskInstance of split. With workers > 1 each worker gets its own environment and
agent (from the factories) and a disjoint shard of episodes; episode results do not depend on
which worker ran them.
hylo.eval.scoring¶
Scorers read the trace only; they never touch the Agent or the Environment. A Score of None is an abstention (infra failure, agent error) and is excluded from metrics.
DEFAULT_SCORERS
module-attribute
¶
SuccessScorer ¶
SafetyInterventionScorer ¶
StepsScorer ¶
one_sided_exact_mcnemar ¶
P[X >= candidate_wins], X ~ Binomial(discordant, 0.5). Same test as Zetta gating.py.
paired_compare ¶
paired_compare(candidate: Mapping[str, float | None], parent: Mapping[str, float | None]) -> PairedComparison
Pairs on TaskInstance.key (task id + seed). Pairs where either side abstained are dropped.
hylo.eval.stats¶
Gate tests (plugin group "gate_tests"): the paired statistic the improvement protocol uses to decide.
A gate test compares candidate and parent on the same (task, seed) keys and returns a PairedComparison:
candidate_wins / parent_wins (pairs where one side is better, used by early rejection and futility), and a
one-sided p_value for H1 "the candidate is better". The protocol around it (cascade, group-sequential looks,
alpha spending, Holm, regression and safety vetoes, held-out report) is unchanged whichever test is chosen.
protocol: {gate_test: {type: sign, params: {metric: steps, direction: lower}}}
Built in: - mcnemar (default): exact one-sided McNemar on a binary metric (success); - sign: exact one-sided sign test on any metric, with a direction and an optional minimum difference; - paired_permutation: sign-flip permutation test on the mean paired difference of a continuous metric.
GateTest ¶
McNemar ¶
SignTest ¶
PairedPermutation ¶
binomial_check ¶
Calibration helper for new tests: rejection rate at alpha 0.05 under H0 (identical agents) and under a clear effect. A valid one-sided test stays near or below 0.05 under H0.