hylo-policy-lerobot¶

Hylo adapter that runs LeRobot policies (SmolVLA, pi0, ...) as Agents, usually behind an AgentServer

Repository hylo-harness, adapters/hylo-policy-lerobot. Licence: Apache-2.0. Depends on: hylo-core, hylo-agents, hylo-memory.

Registered entry points

Group	Name	Object
agents	`lerobot`	`hylo.policies.lerobot.agent:LeRobotAgent`
agents	`lerobot.critic_steered`	`hylo.policies.lerobot.critic_agent:make_critic_steered`
agents	`lerobot.memory_steered`	`hylo.policies.lerobot.experience_agent:make_experience_agent`
agents	`lerobot.noise_steered`	`hylo.policies.lerobot.experience_agent:make_noise_agent`
agents	`lerobot.steered`	`hylo.policies.lerobot.agent:make_steered`
datasources	`lerobot.dataset`	`hylo.policies.lerobot.ingest:LeRobotDatasetSource`
learners	`lerobot.action_memory`	`hylo.policies.lerobot.memory_build:ActionMemoryLearner`
learners	`lerobot.expert_iteration`	`hylo.policies.lerobot.selftrain:ExpertIterationLearner`
learners	`lerobot.noise_prior`	`hylo.policies.lerobot.memory_build:NoisePriorLearner`
policies	`lerobot`	`hylo.policies.lerobot.agent:LeRobotAgent`

Module	Summary
`hylo.policies.lerobot`	LeRobot policies as Hylo Agents
`hylo.policies.lerobot.agent`	A LeRobot policy (SmolVLA, pi0, pi0.5, ...) as a Hylo Agent
`hylo.policies.lerobot.critic_agent`	LeRobot policy + SteeredAgent + learned chunk critic (hylo.agents.critic)
`hylo.policies.lerobot.experience_agent`	LeRobot flow policies with experience memory (RFC-0003 M1)
`hylo.policies.lerobot.export`	Export Hylo traces to a LeRobotDataset (v3) for training (imitation, RECAP-style advantage
`hylo.policies.lerobot.ingest`	LeRobotDataset (v3) as an external DataSource (RFC-0004 W1): demonstrations or other robots' logs -> ingested runs
`hylo.policies.lerobot.memory_build`	Build a frozen ActionBank from Hylo traces, with keys from the policy's own visual tokens (RFC-0003 M1)
`hylo.policies.lerobot.selftrain`	Expert iteration for a LeRobot VLA (RFC-0002 L4): fine-tune the policy on its own successful episodes

`hylo.policies.lerobot`¶

LeRobot policies as Hylo Agents.

WEIGHTS `module-attribute` ¶

WEIGHTS = 'per checkpoint (its model card); PaliGemma-based pi0 / pi0.5: Gemma Terms of Use'

META `module-attribute` ¶

META = {'contracts': {'agents': 'hylo.agent/v0.1', 'policies': 'hylo.policy/v0.1', 'datasources': 'hylo.datasource/v0.1', 'learners': 'hylo.learner/v0.1'}, 'maturity': 'implemented', 'plugins': {'agents/lerobot': {'weights': WEIGHTS}, 'policies/lerobot': {'weights': WEIGHTS}, 'agents/lerobot.steered': {'weights': WEIGHTS}, 'agents/lerobot.critic_steered': {'weights': WEIGHTS}, 'agents/lerobot.memory_steered': {'weights': WEIGHTS}, 'agents/lerobot.noise_steered': {'weights': WEIGHTS}, 'datasources/lerobot.dataset': {'maturity': 'unverified', 'notes': 'dataset downloads not yet approved here; each dataset has its own licence'}, 'learners/lerobot.expert_iteration': {'maturity': 'unverified', 'notes': 'smoke run only', 'weights': WEIGHTS}}}

`hylo.policies.lerobot.agent`¶

A LeRobot policy (SmolVLA, pi0, pi0.5, ...) as a Hylo Agent.

Meant to run behind an AgentServer in a venv that has lerobot + torch; the main process talks to it through RemoteAgent. Observation keys are mapped explicitly (no positional guessing): camera names to the policy's image features, and an ordered list of state keys concatenated into observation.state. The policy returns a whole action chunk; the kernel's ChunkExecutor runs n_action_steps of it.

LIBERO_CAMERAS `module-attribute` ¶

LIBERO_CAMERAS = {'agentview': 'observation.images.camera1', 'wrist': 'observation.images.camera2'}

LIBERO_STATE `module-attribute` ¶

LIBERO_STATE = ('eef_pos', 'eef_axis_angle', 'gripper_qpos')

LeRobotAgent ¶

LeRobotAgent(repo_id: str = 'lerobot/smolvla_libero', device: str = 'cpu', camera_map: dict[str, str] | None = None, state_keys: list[str] | tuple[str, ...] = LIBERO_STATE, empty_cameras: int | None = 1, n_action_steps: int | None = None, num_threads: int | None = None, weights_license: str | None = None, action_semantics: dict[str, Any] | None = None)

Bases: AgentBase

policy `instance-attribute` ¶

policy = get_policy_class(cfg.type).from_pretrained(repo_id, config=cfg).to(device).eval()

camera_map `instance-attribute` ¶

camera_map = dict(camera_map or LIBERO_CAMERAS)

state_keys `instance-attribute` ¶

state_keys = tuple(state_keys)

weights_license `instance-attribute` ¶

weights_license = weights_license or f'per checkpoint: see the model card of {repo_id}'

semantics `instance-attribute` ¶

semantics = ActionSemantics(**(action_semantics or {'control_mode': 'eef_delta_pose', 'frame': 'base', 'rotation_repr': 'axis_angle', 'gripper': 'binary_close_positive'}))

load_s `instance-attribute` ¶

load_s = time.perf_counter() - t0

spec ¶

spec() -> AgentSpec

reset ¶

reset(ctx: EpisodeContext) -> None

default_execution_steps ¶

default_execution_steps() -> int

sample ¶

sample(observation: Observation, seeds: list[int] | None, noise_scale: float = 1.0) -> np.ndarray

(N, T, A) unnormalized chunks; one batched forward pass, noise drawn from the given seeds.

act ¶

act(observation: Observation) -> ActionChunk

make_steered ¶

make_steered(steering: dict[str, Any] | None = None, **lerobot_kwargs: Any)

A LeRobot policy wrapped in SteeredAgent (seeded noise, sample-and-select, runtime recovery rules).

`hylo.policies.lerobot.critic_agent`¶

LeRobot policy + SteeredAgent + learned chunk critic (hylo.agents.critic).

make_critic_steered ¶

make_critic_steered(critic: dict[str, Any], steering: dict[str, Any] | None = None, **lerobot_kwargs: Any)

Samples steering.n_samples chunks per decision and executes the one the critic values most.

`hylo.policies.lerobot.experience_agent`¶

LeRobot flow policies with experience memory (RFC-0003 M1).

PriorLeRobotSampler extends LeRobotAgent with the PriorSampler protocol: - the retrieval key is the policy's own visual tokens (SigLIP embeddings of each real camera, 2x2 grid-pooled, concatenated, L2-normalised), captured from the forward pass the policy runs anyway; - prior_fn(key) is called between the prefix computation and the denoising loop; if it returns a prior, the Euler integration starts at x_t0 = (1 - t0) * normalise(prior) + t0 * noise and runs t0 -> 0 with the policy's unchanged velocity field. With no prior (or t0 = 1) the result is bit-identical to LeRobotAgent.sample. Implemented by wrapping embed_image and euler_integrate for the duration of one call; LeRobot is not modified.

agent: {type: remote, params: {ref: hylo.policies.lerobot.experience_agent:make_experience_agent,
        params: {repo_id: lerobot/smolvla_libero, memory: {type: action_bank, params: {path: ...}}, ...}}}

PriorLeRobotSampler ¶

PriorLeRobotSampler(*args: Any, key_cameras: int = 2, key_grid: int = 2, **kwargs: Any)

Bases: LeRobotAgent

retrieval_key ¶

retrieval_key(observation: Observation) -> np.ndarray

sample ¶

sample(observation: Observation, seeds: list[int] | None, noise_scale: float = 1.0, prior_fn: PriorFn | None = None) -> np.ndarray

seed_noise ¶

seed_noise(seed: int, shape: tuple[int, int], noise_scale: float = 1.0) -> np.ndarray

The exact initial noise LeRobotAgent.sample draws for a seed (torch's generator, so this needs torch).

make_noise_agent ¶

make_noise_agent(memory: dict[str, Any], noise_steer: dict[str, Any] | None = None, steering: dict[str, Any] | None = None, **lerobot_kwargs: Any)

LeRobot policy + SteeredAgent + noise prior retrieved from successful episodes (hylo.agents.noise_steer).

make_experience_agent ¶

make_experience_agent(memory: dict[str, Any], memory_steer: dict[str, Any] | None = None, steering: dict[str, Any] | None = None, **lerobot_kwargs: Any)

LeRobot policy + SteeredAgent + action-memory prior (hylo.agents.memory_steer).

`hylo.policies.lerobot.export`¶

Export Hylo traces to a LeRobotDataset (v3) for training (imitation, RECAP-style advantage conditioning, RL warm starts). Runs in a venv with lerobot (envs/libero).

Each frame pairs the observation seen before an action with the action actually executed (after the SafetyShield), plus next.reward / next.done / next.success. Video features need traces recorded with trace.image_every = 1; episodes without per-step images are skipped and reported. A provenance file maps every exported episode back to its run, episode id, status and agent, and the dataset is sealed as a dataset artifact (RFC-0004). Gate and held-out runs are refused whatever --splits says.

python -m hylo.policies.lerobot.export --runs runs/<run_dir> ... --out data/<name>         [--only-success] [--state-keys eef_pos eef_axis_angle gripper_qpos] [--cameras agentview wrist]

DEFAULT_STATE `module-attribute` ¶

DEFAULT_STATE = ('eef_pos', 'eef_axis_angle', 'gripper_qpos')

DEFAULT_CAMERAS `module-attribute` ¶

DEFAULT_CAMERAS = {'agentview': 'observation.images.image', 'wrist': 'observation.images.image2'}

POLICY_CAMERAS `module-attribute` ¶

POLICY_CAMERAS = {'agentview': 'observation.images.camera1', 'wrist': 'observation.images.camera2'}

episode_frames ¶

episode_frames(tr: EpisodeTrace, state_keys: tuple[str, ...], cameras: dict[str, str]) -> Iterator[dict[str, Any]]

Yield (observation before action, executed action, outcome) frames; raises if images are missing.

export ¶

export(run_dirs: list[str], out: str, *, repo_id: str = 'local/hylo', fps: int = 20, state_keys: tuple[str, ...] = DEFAULT_STATE, cameras: dict[str, str] | None = None, only_success: bool = False, robot_type: str = 'panda', splits: tuple[str, ...] | None = None) -> dict[str, Any]

main ¶

main() -> None

`hylo.policies.lerobot.ingest`¶

LeRobotDataset (v3) as an external DataSource (RFC-0004 W1): demonstrations or other robots' logs -> ingested runs.

State and action vectors come from the parquet table; camera frames are decoded only for kept frames (every image_every, plus the last). The dataset's own next.success column is the outcome label when present; otherwise episodes stay unlabeled unless the ingest spec says the source is a curated demonstration set (assume_success). Runs in a venv with lerobot (envs/libero).

{"name": "libero10-demos", "licence": "MIT", "image_every": 10, "out": "runs/external",
 "source": {"type": "lerobot.dataset", "params": {"root": "data/libero_10", "repo_id": "HuggingFaceVLA/libero",
            "cameras": {"observation.images.image": "agentview", "observation.images.wrist_image": "wrist"}}}}
envs/libero/.venv/bin/python -m hylo.learn.data ingest spec.json

DEFAULT_STATE `module-attribute` ¶

DEFAULT_STATE = ('eef_pos', 'eef_axis_angle', 'gripper_qpos')

LIBERO_STATE `module-attribute` ¶

LIBERO_STATE = {'eef_pos': 3, 'eef_axis_angle': 3, 'gripper_qpos': 2}

LeRobotDatasetSource ¶

LeRobotDatasetSource(root: str, repo_id: str = 'local/dataset', cameras: dict[str, str] | None = None, state_split: dict[str, int] | None = None, episodes: list[int] | None = None, max_episodes: int | None = None, image_every: int = 1, task_prefix: str = '', licence: str = 'unknown')

cameras `instance-attribute` ¶

cameras = cameras or {v: k for k, v in POLICY_CAMERAS.items()}

state_split `instance-attribute` ¶

state_split = state_split or dict(LIBERO_STATE)

origin `instance-attribute` ¶

origin = f'lerobot:{repo_id}@{root}'

name `instance-attribute` ¶

name = f'lerobot.dataset[{repo_id}]'

episodes ¶

episodes() -> Iterator[ExternalEpisode]

`hylo.policies.lerobot.memory_build`¶

Build a frozen ActionBank from Hylo traces, with keys from the policy's own visual tokens (RFC-0003 M1).

python -m hylo.policies.lerobot.memory_build --runs runs/<search runs> ... --out runs/memory/<name>         [--repo-id lerobot/smolvla_libero] [--workers 8] [--task-prefix libero/libero_10/]

Only chunks of simulator-verified successful episodes from development splits are stored (search / search_val; gate and held-out never). Replays of identical episodes (common random numbers across runs) are stored once. Writes bank.npz / bank.json plus build_report.json with nearest-neighbour similarity statistics, which is what memory_steer.sim_floor should be calibrated against, and seals the directory as an action_memory artifact (RFC-0004). ActionMemoryLearner is the same builder behind the learner interface ("lerobot.action_memory").

ActionMemoryLearner ¶

Learner interface (core.learning): job.params are build keyword arguments.

name `class-attribute` `instance-attribute` ¶

name = 'hylo.policies.lerobot.memory_build@0.1'

fit ¶

fit(job)

NoisePriorLearner ¶

Bases: ActionMemoryLearner

Learner "lerobot.noise_prior": the same builder storing each successful chunk's initial flow noise.

name `class-attribute` `instance-attribute` ¶

name = 'hylo.policies.lerobot.memory_build[noise]@0.1'

fit ¶

fit(job)

build ¶

build(runs: list[str], out: str, *, repo_id: str = 'lerobot/smolvla_libero', workers: int = 8, threads: int = 4, splits: tuple[str, ...] = ('search', 'search_val'), task_prefix: str | None = None, capacity: int = 50000, store: str = 'actions') -> dict[str, Any]

main ¶

main() -> None

`hylo.policies.lerobot.selftrain`¶

Expert iteration for a LeRobot VLA (RFC-0002 L4): fine-tune the policy on its own successful episodes.

python -m hylo.policies.lerobot.selftrain --runs runs/<search-run> ... --out runs/selftrain/<name>         --pretrained lerobot/smolvla_libero --steps 300 --batch-size 8

export: successful episodes of development-split runs (search / search_val only; gate and held-out data must stay unseen) -> LeRobotDataset v3, cameras named as the policy expects, with provenance;
pin normalisation: the dataset's state/action statistics are replaced by the pretrained policy's own, so fine-tuning changes weights, not the input/output scaling (lerobot-train otherwise re-normalises with the new dataset's stats);
fine-tune with lerobot-train (action expert only by default: cheap on CPU, little room to overfit);
write selftrain.json: dataset + weights digests, training arguments, and the config patch that turns the result into an improvement candidate (it is then gated like any other candidate); the weights directory is sealed as a policy_weights artifact (RFC-0004) whose provenance lists the runs and the dataset artifact. Runs in the policy venv (envs/libero/.venv: lerobot + torch).

PINNED `module-attribute` ¶

PINNED = ('observation.state', 'action')

ExpertIterationLearner ¶

Learner interface (core.learning): job.params are selftrain keyword arguments; seals the weights dir.

name `class-attribute` `instance-attribute` ¶

name = 'hylo.policies.lerobot.selftrain@0.1'

fit ¶

fit(job) -> ArtifactManifest

pretrained_dir ¶

pretrained_dir(ref: str) -> Path

pin_stats ¶

pin_stats(dataset_dir: str | Path, pretrained: str) -> dict[str, Any]

Overwrite meta/stats.json entries for state and action with the pretrained normaliser's statistics.

finetune ¶

finetune(dataset_dir: str | Path, pretrained: str, out_dir: str | Path, *, steps: int, batch_size: int, lr: float, num_workers: int = 2, expert_only: bool = True, seed: int = 0, extra: list[str] | None = None) -> Path

selftrain ¶

selftrain(run_dirs: list[str], out: str, *, pretrained: str = 'lerobot/smolvla_libero', steps: int = 300, batch_size: int = 8, lr: float = 1e-05, splits: tuple[str, ...] = ('search', 'search_val'), expert_only: bool = True, num_workers: int = 2, seed: int = 0, patch_path: str = 'agent.params.params.repo_id') -> dict[str, Any]

main ¶

main() -> None

hylo-policy-lerobot¶

hylo.policies.lerobot¶

WEIGHTS module-attribute ¶

META module-attribute ¶

hylo.policies.lerobot.agent¶

LIBERO_CAMERAS module-attribute ¶

LIBERO_STATE module-attribute ¶

LeRobotAgent ¶

policy instance-attribute ¶

camera_map instance-attribute ¶

state_keys instance-attribute ¶

weights_license instance-attribute ¶

semantics instance-attribute ¶

load_s instance-attribute ¶

spec ¶

reset ¶

default_execution_steps ¶

sample ¶

act ¶

make_steered ¶

hylo.policies.lerobot.critic_agent¶

make_critic_steered ¶

hylo.policies.lerobot.experience_agent¶

PriorLeRobotSampler ¶

retrieval_key ¶

sample ¶

seed_noise ¶

make_noise_agent ¶

make_experience_agent ¶

hylo.policies.lerobot.export¶

DEFAULT_STATE module-attribute ¶

DEFAULT_CAMERAS module-attribute ¶

POLICY_CAMERAS module-attribute ¶

episode_frames ¶

export ¶

main ¶

hylo.policies.lerobot.ingest¶

DEFAULT_STATE module-attribute ¶

LIBERO_STATE module-attribute ¶

LeRobotDatasetSource ¶

cameras instance-attribute ¶

state_split instance-attribute ¶

origin instance-attribute ¶

name instance-attribute ¶

episodes ¶

hylo.policies.lerobot.memory_build¶

ActionMemoryLearner ¶

name class-attribute instance-attribute ¶

fit ¶

NoisePriorLearner ¶

name class-attribute instance-attribute ¶

fit ¶

build ¶

main ¶

hylo.policies.lerobot.selftrain¶

PINNED module-attribute ¶

ExpertIterationLearner ¶

name class-attribute instance-attribute ¶

fit ¶

pretrained_dir ¶

pin_stats ¶

finetune ¶

selftrain ¶

main ¶

`hylo.policies.lerobot`¶

WEIGHTS `module-attribute` ¶

META `module-attribute` ¶

`hylo.policies.lerobot.agent`¶

LIBERO_CAMERAS `module-attribute` ¶

LIBERO_STATE `module-attribute` ¶

policy `instance-attribute` ¶

camera_map `instance-attribute` ¶

state_keys `instance-attribute` ¶

weights_license `instance-attribute` ¶

semantics `instance-attribute` ¶

load_s `instance-attribute` ¶

`hylo.policies.lerobot.critic_agent`¶

`hylo.policies.lerobot.experience_agent`¶

`hylo.policies.lerobot.export`¶

DEFAULT_STATE `module-attribute` ¶

DEFAULT_CAMERAS `module-attribute` ¶

POLICY_CAMERAS `module-attribute` ¶

`hylo.policies.lerobot.ingest`¶

DEFAULT_STATE `module-attribute` ¶

LIBERO_STATE `module-attribute` ¶

cameras `instance-attribute` ¶

state_split `instance-attribute` ¶

origin `instance-attribute` ¶

name `instance-attribute` ¶

`hylo.policies.lerobot.memory_build`¶

name `class-attribute` `instance-attribute` ¶

name `class-attribute` `instance-attribute` ¶

`hylo.policies.lerobot.selftrain`¶

PINNED `module-attribute` ¶

name `class-attribute` `instance-attribute` ¶