[reward] feat: expose valid_response_length and max_resp_len in extra_info

[Secbone]

What does this PR do?

Expose valid_response_length and max_resp_len in the reward_extra_info dict of DAPORewardManager. These values were already computed internally but were not written to the batch output, making them inaccessible to downstream consumers for logging and analysis.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: gh pr list --search "extra_info valid_response_length" — no duplicates found.
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

pytest tests/workers/reward_manager/test_dapo_on_cpu.py::test_call_return_dict_includes_valid_response_and_max_resp_len -v

Unit test verifies that valid_response_length and max_resp_len appear in reward_extra_info when return_dict=True.

API and Usage Example

# Values are now available in the output batch
reward_extra_info["valid_response_length"]  # list of per-sample response lengths
reward_extra_info["max_resp_len"]           # list of per-sample max response lengths

Design & Code Changes

• verl/workers/reward_manager/dapo.py: 2 lines changed — write valid_response_length and max_resp_len to reward_extra_info (which is explicitly returned to the caller) instead of the non-persistent extra_info dict.
• tests/workers/reward_manager/test_dapo_on_cpu.py: Add unit test verifying the new keys appear in reward_extra_info.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: Unit test added to tests/workers/reward_manager/test_dapo_on_cpu.py.
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[gemini-code-assist]

Code Review

This pull request adds valid_response_length and max_resp_len to extra_info in the DAPO reward manager. However, modifying extra_info in-place is problematic because it may not write back to the batch and will be lost in distributed environments due to Ray actor isolation. It is recommended to append these metrics to reward_extra_info instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Modifying extra_info in-place here has two major issues that prevent downstream consumers from accessing these values:

1. No Write-Back: extra_info is retrieved via data_item.non_tensor_batch.get("extra_info", {}). If "extra_info" is not present in non_tensor_batch, a new dictionary is created. Modifying this dictionary in-place does not write it back to data_item.non_tensor_batch.
2. Ray Actor Isolation: In a distributed training setup, DAPORewardManager typically runs as a Ray actor. Since arguments and return values are serialized/deserialized (passed by value), any in-place modifications to data or data_item inside the actor are lost and not propagated back to the caller.

To correctly expose these metrics to downstream consumers (e.g., for logging), they should be appended to reward_extra_info, which is returned to the caller when return_dict=True.

Suggested change

	extra_info["valid_response_length"] = valid_response_length
	extra_info["max_resp_len"] = self.max_resp_len
	reward_extra_info["valid_response_length"].append(valid_response_length)
	reward_extra_info["max_resp_len"].append(self.max_resp_len)