hylo-agents¶
Hylo intelligence layer: agent composition, reasoners, baseline agents
Repository hylo-harness, packages/hylo-agents. Licence: Apache-2.0. Depends on: hylo-core.
Registered entry points
| Group | Name | Object |
|---|---|---|
| agents | baseline.nearest_object |
hylo.agents.baseline:NearestObjectAgent |
| agents | baseline.planar_ik |
hylo.agents.baseline:PlanarIKAgent |
| agents | baseline.random |
hylo.agents.baseline:RandomAgent |
| agents | composable.object_reach |
hylo.agents.composable:make_object_reach_agent |
| agents | memory.steered |
hylo.agents.memory_steer:MemorySteeredAgent |
| agents | noise.steered |
hylo.agents.noise_steer:NoiseSteeredAgent |
| agents | planner.subgoal |
hylo.agents.planner:SubgoalAgent |
| agents | policy |
hylo.agents.policy_agents:PolicyAgent |
| agents | shadow |
hylo.agents.shadow:ShadowAgent |
| agents | skills |
hylo.agents.skills:SkillAgent |
| agents | steered |
hylo.agents.steering:SteeredAgent |
| agents | temporal_ensemble |
hylo.agents.policy_agents:TemporalEnsembleAgent |
| agents | testing.landmark |
hylo.agents.composable:make_landmark_agent |
| agents | testing.p_controller |
hylo.agents.baseline:PControllerAgent |
| choosers | chooser.rule_color |
hylo.agents.object_reasoner:RuleColorChooser |
| envs | world_model |
hylo.agents.world_model:WorldModelEnvironment |
| instruments | judge.trace |
hylo.agents.judges:TraceSuccessJudge |
| planners | planner.identity |
hylo.agents.planner:IdentityPlanner |
| reasoners | testing.rule_landmark |
hylo.agents.reasoner:LandmarkReasoner |
| Module | Summary |
|---|---|
hylo.agents |
Hylo intelligence layer: agents, reasoners, policies |
hylo.agents.baseline |
Baseline agents. They implement the Agent protocol directly (no reasoner, no memory) |
hylo.agents.composable |
ComposableAgent: Reasoner (slow, event-triggered) + goal-conditioned Policy (every step) + Memory |
hylo.agents.critic |
Chunk value critic (RFC-0002 L3): a small learned Q(s, chunk) that re-ranks the N chunks a frozen VLA samples |
hylo.agents.judges |
Success judges trained on our own traces (RFC-0004 A3; the instrument RFC-0003 M5 needs) |
hylo.agents.memory_steer |
Action-memory steering for a frozen flow VLA (RFC-0003 M1; mechanism after Retrieve-then-Steer, arXiv 2605.10094) |
hylo.agents.noise_steer |
Noise-space steering (RFC-0004 W3): an offline, retrieval-based counterpart of DSRL for a frozen flow VLA |
hylo.agents.object_reasoner |
Language-conditioned object selection with spatial memory |
hylo.agents.planner |
Subgoal planning around any Agent (RFC-0002 L2) |
hylo.agents.policy_agents |
Agents that run a policy (core.policy.ChunkPolicy) given by configuration |
hylo.agents.reasoner |
A rule-based reference reasoner (the Reasoner contract is hylo.core.reasoning) |
hylo.agents.shadow |
Shadow mode (RFC-0004 W4; roadmap v0.3): evaluate a candidate on real hardware without letting it act |
hylo.agents.skills |
SkillAgent (RFC-0003 M3): apply verified skills from a SkillLibrary to the agent it wraps, per episode |
hylo.agents.steering |
Steering a frozen stochastic policy (flow-matching / diffusion VLAs) without touching its weights |
hylo.agents.world_model |
World model (RFC-0004 W2): an action-conditioned model of the environment, learned from our own traces |
hylo.agents¶
Hylo intelligence layer: agents, reasoners, policies.
META
module-attribute
¶
META = {'contracts': {'envs': 'hylo.env/v0.1', 'agents': 'hylo.agent/v0.1', 'reasoners': 'hylo.reasoner/v0.1', 'choosers': 'hylo.chooser/v0.1', 'planners': 'hylo.planner/v0.1', 'instruments': 'hylo.instrument/v0.1'}, 'maturity': 'implemented', 'plugins': {'agents/planner.subgoal': {'maturity': 'unverified'}, 'agents/shadow': {'maturity': 'unverified'}, 'agents/skills': {'maturity': 'unverified'}, 'agents/memory.steered': {'maturity': 'unverified', 'notes': 'the LeRobot variant (lerobot.memory_steered) has run'}, 'agents/noise.steered': {'maturity': 'unverified', 'notes': 'the LeRobot variant (lerobot.noise_steered) has run'}, 'planners/planner.identity': {'maturity': 'unverified'}, 'envs/world_model': {'maturity': 'unverified', 'notes': 'world-model pre-screening not yet used in a campaign'}}}
hylo.agents.baseline¶
Baseline agents. They implement the Agent protocol directly (no reasoner, no memory).
PControllerAgent ¶
RandomAgent ¶
PlanarIKAgent ¶
hylo.agents.composable¶
ComposableAgent: Reasoner (slow, event-triggered) + goal-conditioned Policy (every step) + Memory.
This is the reference composition, not a kernel concept. An end-to-end VLA implements Agent
directly; a VLA can also be the policy here, with an LLM/VLM as the reasoner. Components
declare which observation keys they read, so the same agent works on any environment.
DEFAULT_MEMORY
module-attribute
¶
GoalPolicy ¶
Bases: Protocol
PGoalPolicy ¶
Planar velocity P-controller toward a task-space subgoal (testing.point_reach).
semantics
class-attribute
instance-attribute
¶
PlanarIKGoalPolicy ¶
Analytic 2-link IK toward a task-space subgoal, joint-delta output (mujoco planar arms).
semantics
class-attribute
instance-attribute
¶
ComposableAgent ¶
make_landmark_agent ¶
make_landmark_agent(kp: float = 3.0, memory: dict | None = DEFAULT_MEMORY, replan_every: int = 10) -> ComposableAgent
Factory for configs. The memory backend is resolved through the plugin registry, so this package never imports a memory implementation (memory=None disables long-term memory).
make_object_reach_agent ¶
make_object_reach_agent(kp: float = 0.3, chooser: dict | None = None, memory: dict | None = DEFAULT_MEMORY, replan_every: int = 10) -> ComposableAgent
Language-conditioned object reaching: an ObjectReasoner picks the instructed object with a
pluggable chooser (rule-based or LLM, resolved from the reasoners plugin group), remembers
objects in spatial memory while they are visible, and an IK policy drives the arm to it.
hylo.agents.critic¶
Chunk value critic (RFC-0002 L3): a small learned Q(s, chunk) that re-ranks the N chunks a frozen VLA samples.
V-GPS-style test-time steering, scaled to CPU and to the data we already have:
- data: our own traces. Each agent.chunk event plus the observation before it is one (state, chunk) sample;
the target is the discounted success-to-go gamma**(steps until success), 0 if the episode failed
(Monte Carlo, no bootstrapping: the behaviour policy is the policy being steered).
- features: proprio state, episode progress, task one-hot, a summary of the chunk's executed part
(net translation / rotation, gripper level and flips), optionally a coarse camera thumbnail.
- model: an ensemble of small numpy MLPs; selection maximises mean - beta * std (pessimism against the
critic's own extrapolation, the role Cal-QL's conservatism plays in V-GPS).
Weights are .npz + JSON metadata (no pickle), so the critic loads in any venv.
agent: {type: remote, params: {ref: hylo.policies.lerobot.critic_agent:make_critic_steered, params: {
repo_id: ..., steering: {n_samples: 4, execution_steps: 25}, critic: {path: critics/l10-v1, beta: 0.5}}}}
ChunkCritic ¶
ChunkCritic(features: FeatureConfig, mean: ndarray, std: ndarray, members: list[dict[str, ndarray]], meta: dict[str, Any] | None = None)
score_chunks ¶
score_chunks(obs: Observation, chunks: ndarray, *, step: int, task_id: str, execution_steps: int, beta: float = 0.0) -> tuple[np.ndarray, np.ndarray, np.ndarray]
fit
classmethod
¶
fit(x: ndarray, y: ndarray, features: FeatureConfig, *, weights: ndarray | None = None, members: int = 5, hidden: int = 64, epochs: int = 60, lr: float = 0.001, l2: float = 0.0001, batch: int = 256, seed: int = 0, meta: dict[str, Any] | None = None) -> 'ChunkCritic'
CriticSteeredAgent ¶
CriticSteeredAgent(sampler: Any, steering: Any = None, critic: CriticConfig | dict[str, Any] | None = None)
Bases: SteeredAgent
SteeredAgent whose sample-and-select step picks the chunk the critic scores highest.
chunk_features ¶
Summary of the part of a (T, 7) chunk that will actually run: net 6-D motion over the prefix and the executed part, mean gripper command over each, and the number of gripper sign flips.
state_features ¶
state_features(state: dict[str, Any], step: int, task_id: str, images: dict[str, Any] | None, cfg: FeatureConfig) -> np.ndarray
hylo.agents.judges¶
Success judges trained on our own traces (RFC-0004 A3; the instrument RFC-0003 M5 needs).
A judge decides "is the task done?" from what a real robot has: camera frames, proprio state and the task. It never sees privileged simulator state, the episode length or the termination reason (in LIBERO an episode ends the moment it succeeds, so length alone would leak the answer).
Model: the critic's numpy MLP ensemble over proprio + task one-hot + grayscale thumbnails of each camera. Trained by the "judge.trace" learner in hylo-learn (hylo.learn.judges).
instrument: {type: judge.trace, params: {path: "artifact:l10-trace-v2@54a1efadf00d34d8"}}
TraceSuccessJudge ¶
SuccessJudge instrument over a sealed judge.trace artifact.
judge_features ¶
judge_features(state: dict[str, Any], images: dict[str, Any], task_id: str, cfg: JudgeConfig) -> np.ndarray
imaged_frames ¶
(observation index, payload) of every observation that carries all cameras.
hylo.agents.memory_steer¶
Action-memory steering for a frozen flow VLA (RFC-0003 M1; mechanism after Retrieve-then-Steer, arXiv 2605.10094).
At every decision the sampler computes a retrieval key inside its own forward pass and calls back here: 1. retrieve the top-K verified segments by cosine similarity (same task by default) above a similarity floor; 2. drop candidates whose chunk is inconsistent with the others (median pairwise DTW over the executed prefix); 3. aggregate the rest into an elite prior with softmax-over-similarity weights (weighted mean for motion, weighted vote for the gripper, nearest neighbour when the vote is split); 4. map retrieval confidence (similarity, agreement) to a flow start time t0: confident -> small t0 -> strong guidance; nothing usable -> t0 = 1, i.e. the unchanged policy. The sampler then integrates the policy's own velocity field from x_t0 = (1 - t0) prior + t0 noise, so the current observation still decides the action: memory biases the sample, it never replays it.
MemorySteerConfig ¶
MemorySteeredAgent ¶
MemorySteeredAgent(sampler: Any, steering: Any = None, memory: ActionMemory | dict[str, Any] | None = None, memory_steer: MemorySteerConfig | dict[str, Any] | None = None)
Bases: SteeredAgent
SteeredAgent whose samples start from an action-memory prior (needs a PriorSampler).
dtw ¶
Multivariate dynamic time warping with Euclidean step cost, normalised by path length bound (len(a)+len(b)).
elite_prior ¶
elite_prior(cands: list[tuple[ActionSegment, float]], cfg: MemorySteerConfig, horizon: int, compare: int) -> PriorDecision
Filter, aggregate and score retrieved candidates (already above the similarity floor).
verified_decision_points ¶
verified_decision_points(trace: Any, *, splits: tuple[str, ...] = ('search', 'search_val')) -> Iterator[dict[str, Any]]
Policy chunks of a successful development-split episode whose observation has camera images.
Yields {"observation": payload, "actions": (T, A), "step", "progress", "task_id", "episode_id", "split"}. Only simulator-verified successes qualify (unverified memory is worse than none).
hylo.agents.noise_steer¶
Noise-space steering (RFC-0004 W3): an offline, retrieval-based counterpart of DSRL for a frozen flow VLA.
DSRL learns a policy over the flow policy's initial noise and leaves the VLA untouched. Offline, from our own
successes: every successful chunk was produced from a known seed, so its exact initial noise is recoverable. The
bank (memory_build --store noise) keys those noises by the policy's own visual tokens. At decision time the agent
retrieves the nearest same-task noises, averages them (similarity-weighted), rescales the mean to unit variance and
hands it to the sampler, which starts the full flow from sqrt(w) * prior noise + sqrt(1 - w) * fresh noise. With
w = 0 this is exactly the policy; nothing is retrieved below sim_floor.
Unlike an action prior, the chunk is still produced by the policy's velocity field for the current observation: the noise only biases which mode the flow lands in.
agent: {type: remote, params: {ref: hylo.policies.lerobot.experience_agent:make_noise_agent, params: {
repo_id: lerobot/smolvla_libero, memory: {type: action_bank, params: {path: "artifact:l10-noise-search-v1@2687768f", frozen: true}},
noise_steer: {weight: 0.5, k: 5, sim_floor: 0.95}, steering: {n_samples: 1, seed_noise: true}}}}
NoiseSteeredAgent ¶
NoiseSteeredAgent(sampler: Any, steering: Any = None, memory: ActionMemory | dict[str, Any] | None = None, noise_steer: NoiseSteerConfig | dict[str, Any] | None = None)
Bases: SteeredAgent
hylo.agents.object_reasoner¶
Language-conditioned object selection with spatial memory.
ObjectReasoner turns an instruction + perceived (or remembered) objects into a subgoal. Which object
the instruction refers to is decided by a pluggable Chooser: a rule-based colour matcher here, an
LLM/VLM in the hylo-reasoner-llm adapter. Objects are written to spatial memory while visible and
recalled from it once they are occluded, so the same code path serves both cases.
COLOR_NAMES
module-attribute
¶
COLOR_NAMES: dict[str, tuple[float, float, float]] = {'red': (0.85, 0.1, 0.1), 'green': (0.1, 0.7, 0.2), 'blue': (0.15, 0.35, 0.9), 'yellow': (0.95, 0.85, 0.1), 'purple': (0.55, 0.2, 0.75), 'orange': (0.95, 0.5, 0.1)}
RuleColorChooser ¶
Matches an explicit colour word in the instruction to the nearest object colour. Falls back to the nearest object when no colour word is present.
hylo.agents.planner¶
Subgoal planning around any Agent (RFC-0002 L2).
SubgoalAgent asks a Planner to decompose the task instruction into short sequential sub-instructions, feeds the current one to the wrapped agent (typically a VLA) in place of the original instruction, and advances when the planner judges it done or its step budget runs out. Plans and switches are trace events, so failures can be attributed to planning vs execution. The planner's prompt and check cadence are config, i.e. evolvable.
IdentityPlanner ¶
Baseline: one subgoal, the original instruction (SubgoalAgent then behaves like the wrapped agent).
SubgoalAgent ¶
SubgoalAgent(inner: Agent | dict[str, Any], planner: Planner | dict[str, Any] | None = None, check_every: int = 1, final_instruction: str = 'original')
hylo.agents.policy_agents¶
Agents that run a policy (core.policy.ChunkPolicy) given by configuration.
- "policy": draw one chunk per decision (seeded noise, so episodes replay and paired comparisons share noise)
and execute its first
execution_stepsactions open loop. The plain way to run any policy plugin. -
"temporal_ensemble": ACT's inference (Zhao et al. 2023): query the policy every step and execute, for the current step, the exponentially weighted average of every chunk predicted so far that covers it, w_i = exp(-m * i) with i = 0 the oldest prediction. Smoother motion at the price of one query per step.
agent: {type: policy, params: {policy: {type: policy.bc, params: {path: artifacts/bc}}, execution_steps: 8}} agent: {type: temporal_ensemble, params: {policy: {type: policy.bc, params: {path: artifacts/bc}}, m: 0.1}}
PolicyAgent ¶
PolicyAgent(policy: ChunkPolicy | dict[str, Any], execution_steps: int | None = None, seed_noise: bool = True, noise_scale: float = 1.0)
TemporalEnsembleAgent ¶
hylo.agents.reasoner¶
A rule-based reference reasoner (the Reasoner contract is hylo.core.reasoning).
A Reasoner is the slow, event-triggered part of an Agent. It turns observations + memory into Decisions (here: a subgoal position). An LLM/VLM reasoner, a LangGraph graph or a DSPy program implements the same protocol; the kernel never sees the difference.
SWEEP
module-attribute
¶
SWEEP = [(-0.8, 0.8), (0.8, 0.8), (0.8, 0.4), (-0.8, 0.4), (-0.8, 0.0), (0.8, 0.0), (0.8, -0.4), (-0.8, -0.4), (-0.8, -0.8), (0.8, -0.8)]
LandmarkReasoner ¶
Go to the goal if visible; else to a remembered goal; else explore. Remembers what it saw.
Uses working memory within an episode and spatial memory across episodes. A remembered location that turns out to be empty is invalidated (forgotten), not trusted forever.
hylo.agents.shadow¶
Shadow mode (RFC-0004 W4; roadmap v0.3): evaluate a candidate on real hardware without letting it act.
ShadowAgent executes the active agent and hands the same observation to a shadow (the candidate), whose chunk is
never executed. Each decision emits an x.shadow event: how far the shadow's executed prefix would have been from
what the robot actually did (mean / max absolute difference over arm dims, gripper agreement), and the shadow's
latency. The shadow also receives the real step results, so it follows the episode the active agent produces.
With background=True the shadow runs in a worker thread and its result is emitted at the next decision, so the
control loop never waits for it (on a robot the shadow must not add latency). The shadow's own events are prefixed
x.shadow. so nothing it emits is mistaken for the active agent's.
agent: {type: shadow, params: {active: {type: ...}, shadow: {type: ...}, background: true}}
ShadowAgent ¶
divergence ¶
divergence(active: ndarray, shadow: ndarray, steps: int, gripper_dim: int | None = -1) -> dict[str, float]
shadow_report ¶
Aggregate a shadow run: how often and how far the candidate would have acted differently, per task.
hylo.agents.skills¶
SkillAgent (RFC-0003 M3): apply verified skills from a SkillLibrary to the agent it wraps, per episode.
At reset the library is matched against the task (instruction pattern, task tags, env). The best verified skill
of each kind is applied for this episode only and undone at its end:
- steering: a partial SteeringConfig merged into the nearest SteeredAgent in the wrapped chain;
- recovery: trigger rules appended to that SteeredAgent's rules (the steering cap of 4 rules still holds);
- plan: subgoal templates, filled with the trigger's slot bindings, given to the nearest SubgoalAgent.
Weights skills are not swapped per episode (a policy reload); they are applied as config by the improvement loop.
Each application is a skill.applied trace event, so outcomes can be attributed to the skill.
agent: {type: skills, params: {skills: {type: skill_library, params: {path: ..., frozen: true}},
inner: {type: planner.subgoal, params: {inner: {type: lerobot.steered, params: {...}}}}}}
TemplatePlanner ¶
TemplatePlanner(templates: list[str], binds: dict[str, str], fallback: Any, skill_id: str, max_steps: int = 260)
One-episode planner: subgoals from a plan skill; completion checks delegate to the original planner.
hylo.agents.steering¶
Steering a frozen stochastic policy (flow-matching / diffusion VLAs) without touching its weights.
SteeredAgent wraps any ChunkSampler (core.policy.ChunkPolicy: a "policies" plugin, given as a spec or an object) that can draw action chunks for given noise seeds: - seeded noise: chunk noise is derived from (task key, chunk index, sample index, resample round), so a candidate and its parent face the same randomness on the same init state (common random numbers for paired tests) and every episode is replayable; - sample-and-select: draw N chunks and execute their medoid ("medoid"), or keep the gripper-majority ones first ("gripper_vote_medoid", RoboMonkey-style vote); no extra work when N = 1; - runtime triggers: declarative rules over cheap features (still steps, grasp attempts, sample disagreement) that fire recovery actions (resample with more noise, or back off and reopen the gripper). Everything tunable is data (SteeringConfig), so the improvement loop can propose and gate it as config patches.
Feature
module-attribute
¶
SteeringConfig ¶
SteeredAgent ¶
SteeredAgent(sampler: ChunkSampler | dict[str, Any], steering: SteeringConfig | dict[str, Any] | None = None)
select_chunk ¶
Index of the chunk to execute and the sample disagreement (mean distance to the mean prefix).
hylo.agents.world_model¶
World model (RFC-0004 W2): an action-conditioned model of the environment, learned from our own traces.
StateWorldModel predicts the next proprioceptive / object state from (state, action) with an ensemble of small MLPs (their disagreement is the model's uncertainty), and the probability that the task is complete in a state. WorldModelEnvironment wraps it behind the Environment protocol, so the unchanged kernel can run an agent inside the model: episodes start from initial states recorded in development runs and end when the model says the task is complete or the step budget runs out.
It is an instrument. Its use in the improvement loop is pre-screening (protocol.prescreen): candidates the model predicts to be clearly worse than the parent are dropped before any real rollout; the gate never sees model results. This version models low-dimensional state, so it can host agents that act on state (MuJoCo / test baselines, planners); a VLA acting on camera images needs a video world model (GPU), which plugs into the same protocol later.
env: {type: world_model, params: {path: "artifact:world_model@<digest>"}} # trained by the "world_model.state"
# learner (hylo.learn.world_model)
StateWorldModel ¶
StateWorldModel(keys: list[str], dims: list[int], mean: dict[str, ndarray], std: dict[str, ndarray], dyn: list[dict[str, ndarray]], done: list[dict[str, ndarray]], meta: dict[str, Any])
predict_batch ¶
predict() for (B, S) states and (B, A) actions: next states (B, S) and ensemble disagreement (B,).
WorldModelEnvironment ¶
Environment plugin "world_model": the kernel runs an agent inside a learned StateWorldModel.