Summary
Running allenai/MolmoAct-7B-D-0812 with the documented HF quick-start recipe — and, as a cross-check, with your own vLLM path (SteerSimplerEnv/molmoact_model_test.py style, vllm==0.8.5) — the model's visual reasoning is coherent and input-dependent (it emits a sensible depth map and an image-space end-effector trajectory that changes with the scene), but the decoded 7-D low-level action collapses to a near-zero, effectively input-independent vector. I wanted to flag this in case a step in the action decoding is missing on my end, or there's a known checkpoint issue.
Evidence
1. The model's own example decodes near-zero. Using the exact quick-start (its prompt, example_1.png + example_2.png, "close the box"):
parse_action(generated_text, unnorm_key="molmoact")
# -> [-8.6e-06, -6.4e-05, -1.4e-04, -4.0e-04, 1.3e-04, 8.0e-05, 0.996]
Every position/rotation dim is ~an order of magnitude below the action std from norm_stats["molmoact"] (std ≈ [0.0052, 0.0075, 0.0064, 0.0132, 0.0109, 0.0149]), and far from the sample action shown in the model card ([0.073, 0.082, ...], which itself exceeds the norm-stats max of ~0.06).
2. The action is nearly input-independent. Two clearly different scenes (different object layouts, same instruction) decode to the byte-identical position delta [-1e-05, -6e-05, -0.00029, ...], even though the predicted image-space trajectories differ (so the model is conditioning on the image for the trajectory, just not for the decoded action).
3. It reproduces in vLLM (your stack), not just HF. Running the vLLM path (olmo.vllm.molmoact.MolmoActForActionReasoning, vllm==0.8.5, SamplingParams(temperature=0)) on the same scene yields byte-identical action tokens to the HF .generate() path — so it does not appear to be an HF-serving artifact.
4. Invariant to dtype and length. Same result in bfloat16 and float32, and at max_new_tokens ∈ {256, 512, 900} (generation completes; not truncation).
Environment
transformers==4.52.3, vllm==0.8.5, huggingface_hub==0.33.4- 1× A100 80GB, CUDA 12.4,
torch 2.4
- checkpoint:
allenai/MolmoAct-7B-D-0812, unnorm_key="molmoact"
Question
Is there an action-decoding step (or a different unnorm_key / generation config) required to recover usable low-level actions from this checkpoint via the documented recipe? The depth + trajectory outputs look great; only the final 7-D action appears degenerate. Happy to share a minimal repro script.
Thanks for releasing MolmoAct — the visual-reasoning trace is genuinely useful.
Summary
Running
allenai/MolmoAct-7B-D-0812with the documented HF quick-start recipe — and, as a cross-check, with your own vLLM path (SteerSimplerEnv/molmoact_model_test.pystyle,vllm==0.8.5) — the model's visual reasoning is coherent and input-dependent (it emits a sensible depth map and an image-space end-effector trajectory that changes with the scene), but the decoded 7-D low-level action collapses to a near-zero, effectively input-independent vector. I wanted to flag this in case a step in the action decoding is missing on my end, or there's a known checkpoint issue.Evidence
1. The model's own example decodes near-zero. Using the exact quick-start (its prompt,
example_1.png+example_2.png, "close the box"):Every position/rotation dim is ~an order of magnitude below the action
stdfromnorm_stats["molmoact"](std ≈ [0.0052, 0.0075, 0.0064, 0.0132, 0.0109, 0.0149]), and far from the sample action shown in the model card ([0.073, 0.082, ...], which itself exceeds the norm-statsmaxof~0.06).2. The action is nearly input-independent. Two clearly different scenes (different object layouts, same instruction) decode to the byte-identical position delta
[-1e-05, -6e-05, -0.00029, ...], even though the predicted image-space trajectories differ (so the model is conditioning on the image for the trajectory, just not for the decoded action).3. It reproduces in vLLM (your stack), not just HF. Running the vLLM path (
olmo.vllm.molmoact.MolmoActForActionReasoning,vllm==0.8.5,SamplingParams(temperature=0)) on the same scene yields byte-identical action tokens to the HF.generate()path — so it does not appear to be an HF-serving artifact.4. Invariant to dtype and length. Same result in
bfloat16andfloat32, and atmax_new_tokens ∈ {256, 512, 900}(generation completes; not truncation).Environment
transformers==4.52.3,vllm==0.8.5,huggingface_hub==0.33.4torch 2.4allenai/MolmoAct-7B-D-0812,unnorm_key="molmoact"Question
Is there an action-decoding step (or a different
unnorm_key/ generation config) required to recover usable low-level actions from this checkpoint via the documented recipe? The depth + trajectory outputs look great; only the final 7-D action appears degenerate. Happy to share a minimal repro script.Thanks for releasing MolmoAct — the visual-reasoning trace is genuinely useful.