Add RecGen for object shape completion

[williamshen-nz]

Summary

Replaces the convex-hull mesh built per object in segment_pointcloud_by_masks with a RecGen reconstruction, behind perception.recgen.enabled in tiptop.yml (off by default). Per object: POST the camera-frame RGB-D + mask + intrinsics to the RecGen server, transform the returned camera-frame mesh into world frame via world_from_cam, and feed the result through the existing cuRobo conversion. Grasp linking continues to use the masked depth point cloud — behaviour is unchanged for that path.

perception.recgen.target_faces enables server-side quadric edge-collapse decimation (uses fast-simplification, lives in recgen/scripts/server.py) to keep wire size down — raw RecGen meshes are ~0.5–1.5M faces / ~30 MB each; decimated to ~1000 faces / ~30 KB. The recgen server change is already on main there.

Per-object requests fan out concurrently (bounded by perception.recgen.concurrency, ~= the gateway's GPU count) rather than serially. RecGen is awaited up-front in run_perception; process_scene_geometry stays synchronous and runs in a worker thread so the event loop isn't blocked, deriving each object's point cloud from its mask. Objects whose masked depth has too few valid points are skipped, and the mesh/pcd/grasp dicts are kept in sync.

RecGen is recommended off in production: it adds ~10–20s per object vs convex hulls.

Also includes a perception-server health check (FoundationStereo, M2T2, and RecGen when enabled) before the slow cuRobo IK/MotionGen warmup, so a down server fails fast.

Test plan

• pixi run test-integration — 5/5 passed on the latest full run (scene 2 is borderline; see known issues)
• pixi run tiptop-h5 --h5-path tests/assets/tiptop_scene1_obs.h5 --task-instruction "Put the Rubik's cube in the bowl." — perception + plan complete, recgen meshes look right in rerun
• pixi run tiptop-h5 --h5-path tests/assets/tiptop_scene5_obs.h5 --task-instruction "Put 3 blocks in the bowl." — 4 movables + bowl reconstructed, 25-step plan found
• pixi run viz-tiptop-run --save-dir <run> — perception + plan viz both render correctly
• Convex-hull fallback (enabled: false) — full integ suite passes, confirming no regression to the existing path
• Live robot demo path — verified on the FR3, including trajectory execution with the RecGen path active

Known issues / follow-ups

• Scene 2 (Put the can in the mug.) is borderline. RecGen sometimes reconstructs the (partially occluded) can over-extended in X — observed 9.4 × 4.7 × 7.6 cm vs the expected ~6.5 × 6.5 × 12 — which makes it wider than the 8 cm mug and fails cuTAMP's StablePlacement constraint (0/256 satisfying). It passed on the latest run, but the reconstruction quality on this occluded object is variable. Recgen-side quality issue, not an integration bug; the convex-hull path is stable here because it's dominated by observed depth.
• RecGen occasionally returns multi-shell reconstructions (observed on green_block in scene 5: two near-identical overlapping cuboid meshes). Quadric edge collapse can't merge geometrically distinct surfaces, so decimation can't bring those below ~50k faces. Workarounds (largest-component filtering / vertex welding) not implemented here — flagged for recgen-side investigation.
• target_faces=10000 is the default (raised from 1000 in 9d42739). 1000 visibly under-tessellates meshes in Rerun. The bump is effectively free: face count only affects visualisation and the static world mesh, since cuTAMP samples movable-object collision spheres via area-weighted surface sampling (sample_greedy_surface_spheres, 50 spheres @ 5mm), which is insensitive to tessellation. Per-object response grows 0.02 → 0.20 MB against a 7.4 MB per-object request, and RecGen wall time is unchanged (5-object runs: 20.1s avg at 1k, 19.7s at 10k).

---

2026-07-31 — synced with main. Merged main in (no rebase/force-push, history preserved). The two incidental fixes this PR originally bundled have since landed separately and were dropped from the branch during the merge (they now match main):

• viz_tiptop_run.py apostrophe-label regex fix → shipped in Allow non-word characters in viz_tiptop_run object label regex #35
• UTF-8 git-diff decode fix → shipped in Decode git output and read prompts as UTF-8 #33

The branch now reduces to the RecGen shape-completion feature plus the pre-warmup server health check. enabled: false is already the default, so no pre-merge config flip is needed.

[Copilot]

Pull request overview

This PR adds an optional RecGen-based per-object mesh reconstruction path to replace the existing convex-hull mesh generation in segment_pointcloud_by_masks, gated by perception.recgen.enabled in tiptop.yml. It introduces a new async RecGen client + shape-completion wrapper, integrates it into run_perception/scene processing, and includes a small regex fix in the run-visualization script for object labels containing apostrophes.

Changes:

• Add tiptop.perception.recgen (HTTP msgpack client + health check) and tiptop.perception.shape_completion (per-mask RecGen reconstruction + masked depth PCD creation).
• Update tiptop_run.py to optionally call RecGen during scene geometry processing and include RecGen in server health checks when enabled.
• Fix label parsing in viz_tiptop_run.py to handle object names with apostrophes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tiptop/tiptop_run.py	Integrates RecGen shape completion behind config and updates server health checks + perception pipeline call sites.
tiptop/scripts/viz_tiptop_run.py	Adjusts regex for parsing grasped object names from action labels.
tiptop/perception/shape_completion.py	New RecGen-based per-object reconstruction + masked depth point cloud generation.
tiptop/perception/recgen.py	New async RecGen HTTP client (msgpack payloads) + health check helper.
tiptop/config/tiptop.yml	Adds perception.recgen config block (url/enabled/target_faces).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

[williamshen-nz]

Added comments

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[williamshen-nz]

Getting closer and closer!

[williamshen-nz]

I think recgen doesn't like top down viewpoints. Maybe try front facing more on the robot

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[williamshen-nz]

Self-review after the main sync. Overall this is in good shape to land as an experimental, off-by-default feature.

The thing that mattered most — the default convex-hull path is unchanged. The only code shared between the two paths is the extracted masked_object_points, and I checked it line-for-line against the old inlined logic in segment_pointcloud_by_masks: same erosion, same < 10 valid-point threshold, same un-eroded fallback, same return. The recgen_meshes is None branch calls segment_pointcloud_by_masks exactly as before, and removing the never-used object_pcds param is safe. The new if not object_pcds: raise guard fires on both paths but only replaces a would-be cryptic np.vstack([]) crash with a clear error — strict improvement. So no regression risk to the production path.

A few things I verified while reviewing, for the record: depth handed to RecGen (depth_results["depth_map"]) is the same meters-scale map depth_to_xyz consumes, so units are consistent; the asyncio.run(_check()) health check is only reached from the synchronous main(), so there's no nested-event-loop hazard; and nothing else in the tree uses msgpack, so the global patch is currently isolated.

The inline notes below are all non-blocking engineering discussion, not correctness blockers. One more minor consistency nit not worth an inline: reconstruct_objects_with_recgen normalizes masks with a guarded masks[:, 0], while the RecGen branch in process_scene_geometry uses an unguarded masks.squeeze(1) — harmless since both assume (n, 1, h, w), but worth unifying on one idiom eventually.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

[predicators]

Self-review pass before merge. Three corrections to the record above, since the thread currently describes behaviour the branch no longer has.

Mask erosion — the earlier note is superseded. My comment on 41ca6ff said the RecGen path "now mirrors segment_pointcloud_by_masks: erode, fall back to un-eroded mask." That is no longer true and is no longer the intent. 2b6c4da deliberately dropped erosion for RecGen: erosion degrades its reconstructions, whereas the point cloud path still needs it to suppress depth edge noise. So the two now intentionally derive from different masks. The asymmetry reads as a bug on inspection, so it is documented at the source in a47ea7f.

Worth noting the old failure signal is now masked: the ~56k → ~159k face blowup can't appear anymore because target_faces caps every mesh, so face count no longer indicates mask noise.

The pre-POST skip from b5db43b is gone, and that was not deliberate. That commit added a < 10 valid depth points check to skip the RecGen POST for degenerate objects. 2b6c4da moved point cloud derivation out to process_scene_geometry and dropped xyz_world/rgb_world from the signature — the guard was computed off xyz_world in the same block, so it went with its inputs.

Leaving it out. The object is still logged when dropped (masked_object_points warns and returns None), so nothing disappears quietly; the only cost is one wasted inference on a mask with <10 valid depth points, which is rare enough not to justify re-plumbing xyz_world back in.

Health check scope. The description's "fails fast before the slow cuRobo warmup" holds for the live path only. check_server_health is called from get_demo_container, but tiptop_offline.py and tiptop_websocket_server.py call run_perception directly, so an unreachable RecGen server surfaces at request time instead — still a RuntimeError naming the object, just later. Left as is rather than changing existing behaviour on those paths.

Also in this pass

• a47ea7f — the previously silent skip when a label has no RecGen mesh now logs a warning.
• 9d42739 — target_faces default raised 1000 → 10000; rationale in the updated description.

Verified unchanged for the convex-hull path: the masked_object_points extraction is behaviour-preserving (the old code rebound mask_2d to the eroded mask, but it is not read after that block); process_scene_geometry's signature change has a single caller; and the KeyError raised earlier in review is genuinely fixed — filtered_grasps is keyed off object_pcds and object_trimeshes is filtered to those keys, so object_trimeshes ⊆ object_pcds holds.

Known follow-ups, none blocking: RecGen re-uploads identical rgb+depth per object (~6.4 MB of each 7.4 MB request, ~38 MB redundant for a 6-object scene) — needs a server-side change to upload the scene once and reference it per mask. Multi-shell reconstructions remain a recgen-side issue as described above. The RecGen path has no CI coverage; acceptable while enabled: false is the default.