Guidance on representative local evaluation setup for qualification phase

[jashshah999]

Question

We're running local evaluation via docker compose -f docker/docker-compose.yaml up with the default sample_config.yaml. We'd like to understand how to make local testing more representative of the official portal evaluation.

Specifically:

1. Does the official portal evaluation use randomized task board poses for each submission? The docs mention randomization, but it's unclear whether each submission sees a fresh random config or a fixed (but secret) one.

2. Are the randomization ranges for the official eval the same as those documented in task_board_limits in sample_config.yaml? (NIC translation: [-0.0215, 0.0234]m, SC translation: [-0.06, 0.055]m, etc.)

3. Is the task board yaw fully randomized (0-360°) or constrained to a range where insertion is kinematically feasible?

4. Is there a recommended way to test locally with varied configurations? For example, should we rebuild the eval image with modified configs, or is there a launch parameter approach that works with the pre-built ghcr.io/intrinsic-dev/aic/aic_eval image?

We've noticed that policies perform very differently on the fixed sample_config.yaml vs slightly varied board poses, and want to ensure our local testing is representative before using limited daily submissions.

Thanks for any guidance!

[jashshah999]

Adding some concrete data that illustrates why this is hard to debug with 1 submission/day:

• Policy A: scores ~24 locally on sample_config.yaml → 112.8 on official eval
• Policy B: scores ~56 locally on sample_config.yaml → 39 on official eval

This inversion makes it nearly impossible to iterate effectively. A few additional questions beyond the original post:

5. Are there differences in cable physics/friction parameters between the local sim and official eval? We've noticed cable behavior can vary significantly and wonder if this is a source of discrepancy.

6. Is there a way to generate a set of test configs that better approximate the official eval distribution? Even 5-10 representative configs would help teams calibrate before using their daily submission.

With only 5 days remaining and 1 submission/day, any clarity on what makes official eval different from local would be greatly appreciated.

[kingdeng05]

Same question here. From the task description it only defines the sub component limit (sc rail, etc), is there any limit on the task board itself?