hylo-policies¶

Hylo reference policies: one small, weights-free implementation per policy family (retrieval, regression, generative, tokenized, model-based), trained from your own runs

Repository hylo-harness, packages/hylo-policies. Licence: Apache-2.0. Depends on: hylo-core, hylo-agents, numpy>=1.24.

Registered entry points

Group	Name	Object
learners	`policy.bc`	`hylo.policies.reference.learners:BCLearner`
learners	`policy.flow`	`hylo.policies.reference.learners:FlowLearner`
learners	`policy.knn`	`hylo.policies.reference.learners:KNNLearner`
learners	`policy.tokens`	`hylo.policies.reference.learners:TokenLearner`
policies	`policy.bc`	`hylo.policies.reference.bc:BCPolicy`
policies	`policy.flow`	`hylo.policies.reference.flow:FlowPolicy`
policies	`policy.knn`	`hylo.policies.reference.knn:KNNPolicy`
policies	`policy.mpc`	`hylo.policies.reference.mpc:MPCPolicy`
policies	`policy.tokens`	`hylo.policies.reference.tokens:TokenPolicy`

Module	Summary
`hylo.policies.reference`	Reference policies: one small implementation per policy family, numpy only, no downloaded weights
`hylo.policies.reference.base`	Demonstrations from runs, and the artifact format and runtime shared by the reference policies
`hylo.policies.reference.bc`	policy.bc: regression behaviour cloning with action chunking: an MLP from the observation to the next T
`hylo.policies.reference.flow`	policy.flow: a flow-matching action head (the objective of pi0's and SmolVLA's action experts)
`hylo.policies.reference.knn`	policy.knn: retrieval (VINN, Pari et al. 2021): the k demonstration observations nearest to the current one,
`hylo.policies.reference.learners`	Learners for the reference policies (RFC-0004): development runs -> demonstrations -> sealed policy artifact
`hylo.policies.reference.mpc`	policy.mpc: model-predictive control with a learned world model (the planning half of TD-MPC / PETS, without a
`hylo.policies.reference.nn`	A small multilayer perceptron with manual gradients and Adam (numpy). Enough for low-dimensional state
`hylo.policies.reference.tokens`	policy.tokens: tokenized actions. Each action dimension at each chunk step is one of `bins` uniform bins over

`hylo.policies.reference`¶

Reference policies: one small implementation per policy family, numpy only, no downloaded weights.

They are trained from your own runs (development splits only, sealed as artifacts with provenance) and run through the same agents, safety shield, benchmarks and gate as any other policy. They are meant for teaching, testing the stack end to end in seconds, and as baselines on low-dimensional state; they are not the published architectures at scale. For those, use the LeRobot policies (ACT, Diffusion, VQ-BeT, pi0, SmolVLA, ...) or openpi servers.

Plugin	Family	Idea (reference)	Stochastic
policy.knn	retrieval / non-parametric	nearest neighbours in observation space, average their chunks (VINN)	no
policy.bc	regression behaviour cloning	MLP from observation to an action chunk, MSE (BC with chunking)	no
policy.flow	generative, flow matching	velocity field from noise to the chunk, Euler integration (pi0, SmolVLA action experts)	yes
policy.tokens	tokenized actions	each action dimension binned, categorical heads, sampled (RT-1, OpenVLA-style discretization)	yes
policy.mpc	model-based planning	cross-entropy method over action sequences in a learned world model (MPC, TD-MPC without the value)	yes

Run any of them with the "policy" agent, ACT-style with "temporal_ensemble", or steer the stochastic ones with "steered" (seeded noise, sample-and-select). Learners of the same names build the artifacts: hylo learn job.json with {"learner": "policy.flow", "inputs": ["runs/demos"], "out": "artifacts/flow"}.

META `module-attribute` ¶

META = {'contracts': {'policies': 'hylo.policy/v0.1', 'learners': 'hylo.learner/v0.1'}, 'maturity': 'implemented'}

`hylo.policies.reference.base`¶

Demonstrations from runs, and the artifact format and runtime shared by the reference policies.

An artifact directory holds policy.json (method, observation keys and sizes, chunk length, the demonstrator's action semantics, hyper-parameters) and policy.npz (normalisers and parameters), sealed by the learner (core.artifact). Nothing is pickled.

SCHEMA `module-attribute` ¶

SCHEMA = 'hylo.reference_policy/v0.1'

Demos `dataclass` ¶

Demos(obs: ndarray, chunks: ndarray, keys: list[str], dims: list[int], semantics: dict[str, Any] | None, episodes: int)

obs `instance-attribute` ¶

obs: ndarray

chunks `instance-attribute` ¶

chunks: ndarray

keys `instance-attribute` ¶

keys: list[str]

dims `instance-attribute` ¶

dims: list[int]

semantics `instance-attribute` ¶

semantics: dict[str, Any] | None

episodes `instance-attribute` ¶

episodes: int

Normalizer ¶

Normalizer(mean: ndarray, std: ndarray)

fit `classmethod` ¶

fit(x: ndarray) -> 'Normalizer'

inverse ¶

inverse(z: ndarray) -> np.ndarray

ReferencePolicy ¶

ReferencePolicy(path: str, execution_steps: int | None = None)

Loads an artifact; subclasses implement _sample(x, rngs, noise_scale) -> (N, T, A) in action units.

method `class-attribute` `instance-attribute` ¶

method = 'reference'

meta `instance-attribute` ¶

meta = json.loads((root / 'policy.json').read_text())

a `instance-attribute` ¶

a = {k: z[k] for k in z.files}

obs_norm `instance-attribute` ¶

obs_norm = Normalizer(self.a['obs_mean'], self.a['obs_std'])

act_norm `instance-attribute` ¶

act_norm = Normalizer(self.a['act_mean'], self.a['act_std'])

name `instance-attribute` ¶

name = f'{self.method}@{digest}'

exec_steps `instance-attribute` ¶

exec_steps = int(execution_steps or self.meta['hyper'].get('execution_steps') or max(1, self.chunk // 2))

spec ¶

spec() -> AgentSpec

reset ¶

reset(ctx: EpisodeContext) -> None

default_execution_steps ¶

default_execution_steps() -> int

sample ¶

sample(observation: Observation, seeds: list[int] | None, noise_scale: float = 1.0) -> np.ndarray

load_demos ¶

load_demos(sources: list[tuple[Path, Any]], *, chunk: int, keys: list[str] | None = None, statuses: tuple[str, ...] = ('success',), max_episodes: int | None = None) -> Demos

Every (observation, next chunk actions) pair of the episodes whose status is in statuses. keys defaults to every small (<= 64 values) state entry of the first observation.

save ¶

save(out: str | Path, method: str, demos: Demos, arrays: dict[str, ndarray], hyper: dict[str, Any]) -> None

`hylo.policies.reference.bc`¶

policy.bc: regression behaviour cloning with action chunking: an MLP from the observation to the next T actions, trained with mean squared error (the parametric baseline of VINN's "BC-Rep"; chunked like ACT).

BCPolicy ¶

BCPolicy(path: str, execution_steps: int | None = None)

Bases: ReferencePolicy

method `class-attribute` `instance-attribute` ¶

method = 'bc'

net `instance-attribute` ¶

net = MLP.from_params({k[4:]: v for k, v in self.a.items() if k.startswith('net.')})

`hylo.policies.reference.flow`¶

policy.flow: a flow-matching action head (the objective of pi0's and SmolVLA's action experts).

Training: for a demonstration chunk x1 (normalised) and noise x0 ~ N(0, I), x_tau = (1 - tau) x0 + tau x1, and an MLP v(x_tau, tau, obs) regresses the velocity x1 - x0. Sampling: start from seeded noise (times noise_scale) and integrate dx/dtau = v from tau = 0 to 1 with Euler steps. Different seeds give different, plausible chunks, so the policy can be steered (seeded noise, sample-and-select) like the large flow-matching VLAs.

FlowPolicy ¶

FlowPolicy(path: str, execution_steps: int | None = None, steps: int | None = None)

Bases: ReferencePolicy

method `class-attribute` `instance-attribute` ¶

method = 'flow'

net `instance-attribute` ¶

net = MLP.from_params({k[4:]: v for k, v in self.a.items() if k.startswith('net.')})

steps `instance-attribute` ¶

steps = int(steps or self.meta['hyper'].get('integration_steps', 10))

tau_features ¶

tau_features(tau: ndarray) -> np.ndarray

`hylo.policies.reference.knn`¶

policy.knn: retrieval (VINN, Pari et al. 2021): the k demonstration observations nearest to the current one, their action chunks averaged with softmax(-distance / temperature) weights. No training beyond storing the data.

KNNPolicy ¶

KNNPolicy(path: str, execution_steps: int | None = None, k: int | None = None, temperature: float | None = None)

Bases: ReferencePolicy

method `class-attribute` `instance-attribute` ¶

method = 'knn'

`hylo.policies.reference.learners`¶

Learners for the reference policies (RFC-0004): development runs -> demonstrations -> sealed policy artifact.

{"learner": "policy.bc", "inputs": ["runs/demos"], "out": "artifacts/bc",
 "params": {"chunk": 8, "epochs": 150, "hidden": 128, "execution_steps": 4}}

Common params: chunk (actions per prediction, default 8), state_keys (default: every small state entry), statuses (episodes to imitate, default ["success"]), max_episodes, execution_steps, seed. Only development splits are read (core.artifact.development_runs), and the artifact's role is "behaviour": only the gate activates it.

KNNLearner ¶

Bases: _Learner

Stores the normalised observations and their chunks (in action units).

BCLearner ¶

Bases: _Learner

FlowLearner ¶

Bases: _Learner

TokenLearner ¶

Bases: _Learner

`hylo.policies.reference.mpc`¶

policy.mpc: model-predictive control with a learned world model (the planning half of TD-MPC / PETS, without a learned value function).

Each decision runs the cross-entropy method over horizon-step action sequences in the world model (an artifact of the "world_model.state" learner): sample sequences from a Gaussian (seeded), roll them out with the ensemble, keep the best elites, refit, repeat. The objective is either the model's learned success probability ("success"), or a task cost given as data, {"reach": ["tip_pos", "target_pos"]}: the distance between two state entries. uncertainty_penalty > 0 adds the ensemble's disagreement to the cost, keeping plans where the model has data (off by default: it slowed reaching on MuJoCo planar reach).

Train the world model on data that explores: expert runs alone cover too few actions, and the planner then exploits the model's errors. On MuJoCo planar reach (40 held-out episodes): expert-only data 0.88 success, expert + random-action runs 0.95 in 44 steps, with these defaults; 64 samples x 3 iterations, re-planning every 2 steps: 0.42 / 0.75.

policy: {type: policy.mpc, params: {world_model: artifacts/wm, objective: {reach: [tip_pos, target_pos]}}}

MPCPolicy ¶

MPCPolicy(world_model: str, objective: str | dict[str, Any] = 'success', horizon: int = 10, population: int = 128, elites: int = 8, iterations: int = 5, execution_steps: int = 1, uncertainty_penalty: float = 0.0, action_low: float = -1.0, action_high: float = 1.0)

model `instance-attribute` ¶

model = StateWorldModel.load(world_model)

action_dim `instance-attribute` ¶

action_dim = int((action.get('shape') or [None])[0] or 0) or None

low `instance-attribute` ¶

low = np.asarray(action.get('low') or [action_low] * (self.action_dim or 1), dtype=float)

high `instance-attribute` ¶

high = np.asarray(action.get('high') or [action_high] * (self.action_dim or 1), dtype=float)

semantics `instance-attribute` ¶

semantics = action.get('semantics')

idx `instance-attribute` ¶

idx = idx

spec ¶

spec() -> AgentSpec

reset ¶

reset(ctx: EpisodeContext) -> None

default_execution_steps ¶

default_execution_steps() -> int

sample ¶

sample(observation: Observation, seeds: list[int] | None, noise_scale: float = 1.0) -> np.ndarray

`hylo.policies.reference.nn`¶

A small multilayer perceptron with manual gradients and Adam (numpy). Enough for low-dimensional state.

MLP ¶

MLP(sizes: list[int], rng: Generator | None = None)

tanh hidden layers, linear output. Parameters are a flat dict of arrays (saved with np.savez).

n `instance-attribute` ¶

n = len(sizes) - 1

p `instance-attribute` ¶

p = {}

from_params `classmethod` ¶

from_params(params: dict[str, ndarray]) -> 'MLP'

forward ¶

forward(x: ndarray) -> tuple[np.ndarray, list[np.ndarray]]

backward ¶

backward(acts: list[ndarray], dout: ndarray) -> dict[str, np.ndarray]

Adam ¶

Adam(params: dict[str, ndarray], lr: float = 0.001)

m `instance-attribute` ¶

m = {k: np.zeros_like(v) for k, v in params.items()}

v `instance-attribute` ¶

v = {k: np.zeros_like(v) for k, v in params.items()}

step ¶

step(params: dict[str, ndarray], grads: dict[str, ndarray]) -> None

train ¶

train(net: MLP, n: int, loss_grad, *, epochs: int, batch: int = 256, lr: float = 0.001, seed: int = 0) -> float

Minibatch Adam. loss_grad(net, idx, rng) returns (loss, grads) for the samples idx. Returns the last epoch's mean loss.

`hylo.policies.reference.tokens`¶

policy.tokens: tokenized actions. Each action dimension at each chunk step is one of bins uniform bins over the demonstrated range (RT-1 and OpenVLA discretize to 256), predicted by a categorical head and sampled.

The heads are independent given the observation (RT-1 style), not decoded autoregressively as in OpenVLA or pi0-FAST. noise_scale is the sampling temperature; greedy=True takes the most likely bin.

TokenPolicy ¶

TokenPolicy(path: str, execution_steps: int | None = None, greedy: bool = False)

Bases: ReferencePolicy

method `class-attribute` `instance-attribute` ¶

method = 'tokens'

net `instance-attribute` ¶

net = MLP.from_params({k[4:]: v for k, v in self.a.items() if k.startswith('net.')})

centers ¶

centers() -> np.ndarray

softmax ¶

softmax(z: ndarray, axis: int = -1) -> np.ndarray

hylo-policies¶

hylo.policies.reference¶

META module-attribute ¶

hylo.policies.reference.base¶

SCHEMA module-attribute ¶

Demos dataclass ¶

obs instance-attribute ¶

chunks instance-attribute ¶

keys instance-attribute ¶

dims instance-attribute ¶

semantics instance-attribute ¶

episodes instance-attribute ¶

Normalizer ¶

fit classmethod ¶

inverse ¶

ReferencePolicy ¶

method class-attribute instance-attribute ¶

meta instance-attribute ¶

a instance-attribute ¶

obs_norm instance-attribute ¶

act_norm instance-attribute ¶

name instance-attribute ¶

exec_steps instance-attribute ¶

spec ¶

reset ¶

default_execution_steps ¶

sample ¶

load_demos ¶

save ¶

hylo.policies.reference.bc¶

BCPolicy ¶

method class-attribute instance-attribute ¶

net instance-attribute ¶

hylo.policies.reference.flow¶

FlowPolicy ¶

method class-attribute instance-attribute ¶

net instance-attribute ¶

steps instance-attribute ¶

tau_features ¶

hylo.policies.reference.knn¶

KNNPolicy ¶

method class-attribute instance-attribute ¶

hylo.policies.reference.learners¶

KNNLearner ¶

BCLearner ¶

FlowLearner ¶

TokenLearner ¶

hylo.policies.reference.mpc¶

MPCPolicy ¶

model instance-attribute ¶

action_dim instance-attribute ¶

low instance-attribute ¶

high instance-attribute ¶

semantics instance-attribute ¶

idx instance-attribute ¶

spec ¶

reset ¶

default_execution_steps ¶

sample ¶

hylo.policies.reference.nn¶

MLP ¶

n instance-attribute ¶

p instance-attribute ¶

from_params classmethod ¶

forward ¶

backward ¶

Adam ¶

m instance-attribute ¶

v instance-attribute ¶

step ¶

train ¶

hylo.policies.reference.tokens¶

TokenPolicy ¶

method class-attribute instance-attribute ¶

net instance-attribute ¶

centers ¶

softmax ¶

`hylo.policies.reference`¶

META `module-attribute` ¶

`hylo.policies.reference.base`¶

SCHEMA `module-attribute` ¶

Demos `dataclass` ¶

obs `instance-attribute` ¶

chunks `instance-attribute` ¶

keys `instance-attribute` ¶

dims `instance-attribute` ¶

semantics `instance-attribute` ¶

episodes `instance-attribute` ¶

fit `classmethod` ¶

method `class-attribute` `instance-attribute` ¶

meta `instance-attribute` ¶

a `instance-attribute` ¶

obs_norm `instance-attribute` ¶

act_norm `instance-attribute` ¶

name `instance-attribute` ¶

exec_steps `instance-attribute` ¶

`hylo.policies.reference.bc`¶

method `class-attribute` `instance-attribute` ¶

net `instance-attribute` ¶

`hylo.policies.reference.flow`¶

method `class-attribute` `instance-attribute` ¶

net `instance-attribute` ¶

steps `instance-attribute` ¶

`hylo.policies.reference.knn`¶

method `class-attribute` `instance-attribute` ¶

`hylo.policies.reference.learners`¶

`hylo.policies.reference.mpc`¶

model `instance-attribute` ¶

action_dim `instance-attribute` ¶

low `instance-attribute` ¶

high `instance-attribute` ¶

semantics `instance-attribute` ¶

idx `instance-attribute` ¶

`hylo.policies.reference.nn`¶

n `instance-attribute` ¶

p `instance-attribute` ¶

from_params `classmethod` ¶

m `instance-attribute` ¶

v `instance-attribute` ¶

`hylo.policies.reference.tokens`¶

method `class-attribute` `instance-attribute` ¶

net `instance-attribute` ¶