perceiving-objects-oneshot

Single VLM call over a set-of-marks overlay. Pipeline:

observe → perceive → filter_obb

perceive runs:

1. `grounding-dino.detect with a broad object.` text prompt
2. One `vlm.query` showing the image with letter-labeled boxes:

"Which letter is the <target>? Reply with one letter or 'none'."

1. On `none: emit found: False` so the subgraph exits

`not_found. On a letter: sam3.segment_box on the chosen box, geometry.mask_to_world_points` for the cloud.

When to use

• Clean-all-items / multi-item loops where the cycle needs a clean

"no match" signal to terminate via `target.not_found → done`.

• Tasks where the target description is generic ("any item on the

floor", "the next remaining grocery item") rather than a specific scene-spec id.

• Uncluttered scenes with distinct, reasonably sized targets where the

set-of-marks letter pick is reliable.

When NOT to use

• Small / cluttered targets (< 40 px wide) — prefer `perceiving-objects`

whose pairwise crop tournament is far more reliable in that regime.

Recommended subgraph state flow

3 states: `observe → perceive → filter_obb (mirrors perceiving-objects`).

State details:

About `object_name` below: it is a literal Python string — the natural noun phrase for the object you are perceiving, drawn from this subgraph's description (e.g. "alphabet soup", "basket", "any grocery item on the floor"). It is a constant per subgraph instance, NOT a binding. DO NOT write Ref("in.object_name") or any other Ref(...); the coordinator does not declare object_name as a subgraph input. Write the string directly, e.g. "object_name": "any grocery item on the floor". The same rule applies to object_description if you set it.

1. `observe` — type: tool, tool: "robot.get_observation",

inputs: {}. Connector tool; flat name only.

1. `perceive` — type: script, file scripts/<sg>/perceive_simple.py

from this bundle. Inputs: cameras=Ref("observe.cameras"), object_name="<noun phrase from the subgraph description>", plus any optional literals (object_description, dino_prompt). Returns {found, cloud, mask, score}. When the VLM picks "none" or DINO emits no detections, `found is False and the downstream filter_obb step then raises (empty cloud) — caught by the subgraph's on_error: "not_found"` exit.

1. `filter_obb` — type: tool,

tool: "geometry.filter_and_compute_obb", inputs={"points": Ref("perceive.cloud")}. Returns {"obb": <OrientedBoundingBox>}.

Wiring the exit (HARD)

Linear perceive → filter_obb → found → END. The filter_obb tool raises on empty clouds (the not-found path), and the subgraph's on_error: "not_found" catches that. Do NOT add conditional edges on perceive — the linear path plus set_on_error is sufficient.

sg.add_node("filter_obb", type="tool",
            tool="geometry.filter_and_compute_obb",
            inputs={"points": Ref("perceive.cloud")})
sg.add_exit("found")
sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)
sg.set_on_error("not_found")

Bind the subgraph outputs (ALL THREE — required, no exceptions):

sg.set_outputs(
    target_obb=Ref("filter_obb.obb"),
    target_mask=Ref("perceive.mask"),
    target_cloud=Ref("perceive.cloud"),
)

Note that geometry.filter_and_compute_obb returns {"obb": ...}, so the OBB binding walks into the obb field (Ref("filter_obb.obb"), NOT a bare Ref("filter_obb")). See references/geometry_calling_conventions.md.