hylo-reasoner-llm

Hylo LLM/VLM runtime components: client, planner, object choosers, VLM success judge (Ollama native or OpenAI-compatible endpoints)

Repository hylo-harness, adapters/hylo-reasoner-llm. Licence: Apache-2.0. Depends on: hylo-core, pillow>=10.

Registered entry points

Group	Name	Object
choosers	chooser.llm	hylo.reasoners.llm.chooser:LLMChooser
choosers	chooser.vlm	hylo.reasoners.llm.vlm_chooser:VLMChooser
instruments	judge.vlm	hylo.reasoners.llm.judge:VLMSuccessJudge
planners	planner.llm	hylo.reasoners.llm.planner:LLMSubgoalPlanner

Module	Summary
hylo.reasoners.llm	LLM/VLM-backed reasoner components
hylo.reasoners.llm.chooser	LLM-backed object chooser: grounds a free-form instruction to one of the perceived objects
hylo.reasoners.llm.client	Minimal JSON-mode chat client for Ollama (native API) and OpenAI-compatible endpoints
hylo.reasoners.llm.judge	Zero-shot VLM success judge (RFC-0004 A3): a vision-language model looks at the final camera frames and the task
hylo.reasoners.llm.planner	LLM/VLM subgoal planner: a text LLM decomposes the instruction; a VLM checks completion from the camera
hylo.reasoners.llm.vlm_chooser	VLM object chooser with set-of-mark prompting

hylo.reasoners.llm

LLM/VLM-backed reasoner components.

META module-attribute

META = {'contracts': {'choosers': 'hylo.chooser/v0.1', 'planners': 'hylo.planner/v0.1', 'instruments': 'hylo.instrument/v0.1'}, 'maturity': 'implemented', 'plugins': {'choosers/chooser.llm': {'weights': "the served model's own licence (e.g. a Qwen or GLM model run by Ollama)"}, 'choosers/chooser.vlm': {'weights': "the served model's own licence (e.g. a Qwen or GLM model run by Ollama)"}, 'planners/planner.llm': {'maturity': 'unverified', 'weights': "the served model's own licence (e.g. a Qwen or GLM model run by Ollama)"}, 'instruments/judge.vlm': {'maturity': 'unverified', 'notes': 'not yet calibrated against ground truth', 'weights': "the served model's own licence (e.g. a Qwen or GLM model run by Ollama)"}}}

hylo.reasoners.llm.chooser

LLM-backed object chooser: grounds a free-form instruction to one of the perceived objects.

Drop-in replacement for chooser.rule_color (same Chooser protocol), selected purely by config: agent: {type: composable.object_reach, params: {chooser: {type: chooser.llm, params: {model: ...}}}}

SYSTEM module-attribute

SYSTEM = 'You ground natural-language instructions for a robot arm working on a table. You receive the instruction and a list of objects with their RGB colour (0-1 floats and hex) and position in metres. Decide which single object the instruction refers to. Reply with JSON only: {"object_id": "<one of the ids>", "rationale": "<one short sentence>"}.'

LLMChooser

LLMChooser(model: str = 'qwen3.5:35b-a3b', base_url: str = 'http://127.0.0.1:11434', api: str = 'ollama', cache_dir: str | None = None, **client_kwargs: Any)

client instance-attribute

client = LLMClient(model=model, base_url=base_url, api=api, cache_dir=cache_dir, **client_kwargs)

name instance-attribute

name = f'chooser.llm[{model}]@0.1'

choose

choose(instruction: str, objects: list[SceneObject], robot_xy: tuple[float, float], obs: Observation | None = None) -> Choice

hylo.reasoners.llm.client

Minimal JSON-mode chat client for Ollama (native API) and OpenAI-compatible endpoints.

Standard library only. Responses are cached on disk by prompt hash so that evaluations are reproducible and re-runs do not hit the model; every result reports whether it was cached.

LLMResult

Bases: Contract

content instance-attribute

content: dict[str, Any]

raw instance-attribute

raw: str

model instance-attribute

model: str

latency_s instance-attribute

latency_s: float

cached instance-attribute

cached: bool

prompt_hash instance-attribute

prompt_hash: str

tokens_in class-attribute instance-attribute

tokens_in: int | None = None

tokens_out class-attribute instance-attribute

tokens_out: int | None = None

LLMError

Bases: RuntimeError

LLMClient

LLMClient(model: str = 'qwen3.5:35b-a3b', base_url: str = 'http://127.0.0.1:11434', api: Literal['ollama', 'openai'] = 'ollama', api_key_env: str | None = None, temperature: float = 0.0, max_tokens: int = 300, timeout_s: float = 120.0, cache_dir: str | None = None, think: bool = False)

cache instance-attribute

cache = Path(cache_dir) if cache_dir else None

chat_json

chat_json(messages: list[dict[str, Any]], images: list[bytes] | None = None) -> LLMResult

hylo.reasoners.llm.judge

Zero-shot VLM success judge (RFC-0004 A3): a vision-language model looks at the final camera frames and the task instruction and says whether the task is complete. No training; trusted only after calibrate_judge against simulator outcomes. Sees exactly what a real robot would record (no privileged state, no episode length).

instrument: {type: judge.vlm, params: {model: qwen3.5:35b-a3b, cameras: [agentview, wrist]}}

PROMPT module-attribute

PROMPT = 'You see the final state of a robot manipulation attempt: {views}.\nTask instruction: "{instruction}"\nDecide whether the task is fully completed in this final state. Every object named in the instruction must be in the\nstated place or state (e.g. inside the basket, on the plate, drawer closed, stove turned on). An object still held by\nthe gripper or merely near its target does not count.\nReply with JSON only: {{"completed": true or false, "confidence": <probability from 0 to 1 that the task is completed>,\n"reason": "<at most 20 words>"}}'

VIEW_NAMES module-attribute

VIEW_NAMES = {'agentview': 'a third-person camera view', 'wrist': "the robot's wrist camera view"}

VLMSuccessJudge

VLMSuccessJudge(model: str = 'qwen3.5:35b-a3b', base_url: str = 'http://127.0.0.1:11434', api: str = 'ollama', cameras: tuple[str, ...] = ('agentview', 'wrist'), flip: bool = False, cache_dir: str | None = None, timeout_s: float = 600.0, max_tokens: int = 200, **client_kwargs: Any)

client instance-attribute

client = LLMClient(model=model, base_url=base_url, api=api, cache_dir=cache_dir, timeout_s=timeout_s, max_tokens=max_tokens, **client_kwargs)

name instance-attribute

name = f'judge.vlm[{model}]@0.1'

judge

judge(episode: EpisodeTrace) -> Judgement

hylo.reasoners.llm.planner

LLM/VLM subgoal planner: a text LLM decomposes the instruction; a VLM checks completion from the camera.

agent: {type: planner.subgoal, params: {inner: {...}, planner: {type: planner.llm, params: {...}}}}

PLAN_PROMPT module-attribute

PLAN_PROMPT = 'You plan for a robot arm controlled by a vision-language-action policy that was trained on short\ntabletop instructions such as "pick up the alphabet soup and place it in the basket", "open the top drawer of the\ncabinet", "put the bowl on the plate", "turn on the stove". Decompose the instruction into at most {max_subgoals}\nsequential sub-instructions in that style, one object or one action each. If it is already a single step, return it\nunchanged as the only sub-instruction. Keep object names exactly as written in the instruction.\nInstruction: "{instruction}"\nReply with JSON only: {{"subgoals": ["...", "..."], "rationale": "<one sentence>"}}'

DONE_PROMPT module-attribute

DONE_PROMPT = 'The image shows a robot workspace from a fixed camera. Has this sub-task been completed?\nSub-task: "{subgoal}"\nAnswer conservatively: only "true" if the end state is clearly visible.\nReply with JSON only: {{"done": true|false, "rationale": "<one short sentence>"}}'

LLMSubgoalPlanner

LLMSubgoalPlanner(model: str = 'qwen3.5:35b-a3b', base_url: str = 'http://127.0.0.1:11434', api: str = 'ollama', camera: str = 'agentview', max_subgoals: int = 3, subgoal_max_steps: int = 260, plan_prompt: str = PLAN_PROMPT, done_prompt: str = DONE_PROMPT, cache_dir: str | None = None, **client_kwargs: Any)

client instance-attribute

client = LLMClient(model=model, base_url=base_url, api=api, cache_dir=cache_dir, **client_kwargs)

name instance-attribute

name = f'planner.llm[{model}]@0.1'

plan

plan(instruction: str, observation: Observation) -> Plan

done

done(subgoal: Subgoal, observation: Observation) -> tuple[bool, str, dict[str, Any]]

hylo.reasoners.llm.vlm_chooser

VLM object chooser with set-of-mark prompting.

The model receives only the camera image, with each candidate object marked by its number; it never sees colour values. Object positions are projected into the image with the pinhole parameters the environment publishes in Observation.extra["camera_params"][camera].

FONT module-attribute

FONT = '/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf'

PROMPT module-attribute

PROMPT = 'The image is a top-down view of a table with a robot arm. Candidate objects are marked with white number tags {marks}. Instruction: "{instruction}". Which numbered object does the instruction refer to? Reply with JSON only: {{"marker": <number>, "rationale": "<one short sentence about what you see>"}}.'

VLMChooser

VLMChooser(model: str = 'qwen3.5:35b-a3b', base_url: str = 'http://127.0.0.1:11434', api: str = 'ollama', camera: str = 'top', cache_dir: str | None = None, **client_kwargs: Any)

client instance-attribute

client = LLMClient(model=model, base_url=base_url, api=api, cache_dir=cache_dir, **client_kwargs)

camera instance-attribute

camera = camera

name instance-attribute

name = f'chooser.vlm[{model}, set-of-mark]@0.1'

choose

choose(instruction: str, objects: list[SceneObject], robot_xy: tuple[float, float], obs: Observation | None = None) -> Choice

project

project(points_xyz: ndarray, cam: dict[str, Any]) -> np.ndarray

World points -> pixel (u, v) for a pinhole camera looking along its local -z (MuJoCo convention).

mark

mark(image: ndarray, uv: ndarray, labels: list[str]) -> Image.Image