Support pick and place next to

[ryanlindeborg]

Summary

Adds experimental support for "pick and place next to" tasks (e.g. "place the cup next to the apple") via a new near predicate. Gated behind a config flag and off by default, so the stable pipeline is unaffected.

What's included

• New near(movable, reference) predicate — "place A laterally next to B," where B is a positional reference, not a surface to stack on. Distinct from on(A, surface).
• Experimental flag — experimental.pick_place_next_to in tiptop.yml (default false). When enabled, perception loads a separate prompt (detect_and_translate_pick_place_next_to.txt) that teaches Gemini the near predicate and the on-vs-near distinction. When disabled, the original prompt is used unchanged, so existing behavior is untouched.
• Goal handling — create_tamp_environment grounds near atoms into Near(movable, reference) goals, which cuTAMP solves with its PlaceNear operator.

Enabling

Set experimental.pick_place_next_to: true in tiptop.yml.

Known limitations

• near reference must be a movable, not a surface. A compound goal like "put block X in the bowl and block Y next to the bowl" fails: the bowl is a surface (an On-target), but cuTAMP's PlaceNear requires the near reference to be a movable, so the goal is rejected with a ValueError before planning. Tracked in tiptop-robot/cuTAMP#26; the fix lives entirely in cuTAMP. Only affects the experimental flag when enabled.

[williamshen-nz]

Thanks for the changes! Left some comments.

[Copilot]

Pull request overview

This PR extends TipTop’s task translation + TAMP goal construction to support “pick/place next to” instructions by introducing a near(movable, reference) goal predicate and adding logic to ensure near goals still imply a concrete placement surface.

Changes:

• Adds near(movable, reference) to the perception prompt guidelines and examples to steer the VLM away from misusing on(A, B) for “next to” language.
• Updates TAMP goal construction to parse near atoms and (when needed) auto-inject an on(movable, table) placement target.
• Refactors hand-state goal handling to add HandEmpty up-front when no holding goal is present.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tiptop/tiptop_run.py	Adds Near to goal-state construction and injects default On(..., table) for near goals.
tiptop/perception/prompts/detect_and_translate.txt	Updates prompt spec + examples to emit near(...) for “next to/beside” instructions and avoid stacking semantics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[williamshen-nz]

Great work!

Let's also bump cuTAMP version here and make sure to test on real robot before approval and merge

[williamshen-nz]

One last comment!

[Copilot]

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

tiptop/tiptop_run.py:721

• cfg.experimental.pick_place_next_to will crash if a user runs with a config file that hasn’t been updated to include the new experimental section. To keep backward compatibility (and match existing optional-config patterns), read this flag via .get(..., False).

        max_planning_time=max_planning_time,
        opt_steps=opt_steps_per_skeleton,
        robot_type=cfg.robot.type,
        time_dilation_factor=cfg.robot.time_dilation_factor,
        near_placement=cfg.experimental.pick_place_next_to,
        collision_activation_distance=0.0,
        enable_visualizer=cutamp_visualize,

tiptop/tiptop_offline.py:86

• cfg.experimental.pick_place_next_to will raise if the loaded config file predates the new experimental section. Consider defaulting to False when the key is absent so older configs continue to work.

    config = build_tamp_config(
        num_particles=num_particles,
        max_planning_time=max_planning_time,
        opt_steps=opt_steps_per_skeleton,
        robot_type=cfg.robot.type,
        time_dilation_factor=cfg.robot.time_dilation_factor,
        near_placement=cfg.experimental.pick_place_next_to,
        collision_activation_distance=0.0,
        enable_visualizer=cutamp_visualize,
    )

tiptop/tiptop_websocket_server.py:83

• self._cfg.experimental.pick_place_next_to will raise if the server is started with a config file that doesn’t define the new experimental section. Defaulting the flag to False when missing keeps older configs compatible.

        self._config = build_tamp_config(
            num_particles=num_particles,
            max_planning_time=max_planning_time,
            opt_steps=500,
            robot_type=self._cfg.robot.type,
            time_dilation_factor=self._cfg.robot.time_dilation_factor,
            near_placement=self._cfg.experimental.pick_place_next_to,
        )

tiptop/perception/gemini.py:26

• Direct attribute access on tiptop_cfg().experimental.pick_place_next_to will raise ConfigAttributeError if a user loads an older config file that doesn’t have the new experimental section. Other parts of the codebase treat optional config keys defensively via .get(...) (e.g. camera config), so it’d be safer to default this flag to False when missing.

def _detect_and_translate_prompt_name() -> str:
    if tiptop_cfg().experimental.pick_place_next_to:
        return "detect_and_translate_pick_place_next_to"
    return "detect_and_translate"

[williamshen-nz]

Peeled the incidental (non-feature) changes out of this PR into their own focused PRs, so this one can stay purely the pick-and-place-next-to feature:

• Pin warp-lang below 1.13 #32 — pin warp-lang below 1.13 (pixi.toml)
• Decode git output and read prompts as UTF-8 #33 — decode git output + read prompt templates as UTF-8 (recording.py, gemini.py's load_prompt)
• Fill missing config keys from packaged defaults on offline replay #34 — fill missing config keys from packaged defaults on offline replay (config/__init__.py, tiptop_offline.py) + tests/test_config.py and the test-unit pixi task

Note on #34: the tests were rewritten to exercise the generic fill/merge behavior using keys that already exist on main (perception.m2t2.apply_bounds, perception.depth_trunc_m, cameras), so they don't depend on the experimental block this PR adds.

Plan: once #32–#34 merge, rebase this branch onto main. The extracted diffs should drop out, leaving only the feature: the near predicate + goal handling, the experimental.pick_place_next_to flag and its threading through planning.py / tiptop_run.py / tiptop_websocket_server.py, the Gemini prompt-selection logic, the new detect_and_translate_pick_place_next_to.txt prompt, the tiptop.yml experimental block, and the utils.py cuTAMP 0.0.6 bump.