[algo, doc] feat: add UP-GRPO unbounded positive asymmetric policy loss

[a-F1]

What does this PR do?

Adds UP-GRPO (Unbounded Positive Asymmetric Optimization, arXiv:2607.06987) as a new, plug-and-play policy-loss mode up.

Motivation. Standard GRPO/PPO combines a symmetric clip (single ε) with importance sampling against π_old. For rare, low-confidence but correct tokens (large positive advantage, small probability), the symmetric upper clip and the π_old anchor suppress exactly the gradient signal needed for exploration, while lifting the clip risks IS-induced gradient explosion. UP resolves this exploration-stability dilemma by treating the two advantage signs asymmetrically.

The up loss (UP-GRPO, Eq. 15):

• Positive advantages (Â > 0): an unbounded, self-anchored REINFORCE objective. Implemented via the stop-gradient trick — the self-anchored ratio r̃ = πθ / sg(πθ) equals 1 in value but carries gradient ∇log πθ, removing the π_old anchor and the upper clip to maximize exploration without IS blow-up.
• Negative advantages (Â ≤ 0): the standard GRPO symmetric clip + dual-clip safeguard, unchanged for stability.

KL handling. The KL term is not added inside the policy loss; it is handled separately by verl via use_kl_loss / kl_loss_coef, consistent with GRPO.

Checklist Before Starting

• Search for similar PRs: https://github.com/volcengine/verl/pulls?q=is%3Apr+UP-GRPO
• Format the PR title as [{modules}] {type}: {description}

Test

Not covered by CI (new algorithm). Validated locally:

• pre-commit on changed files — ruff, ruff-format, check-example-naming, check-docs-time-info, check-license, check-docstrings, check-naming-conventions all pass.
• python -c "from verl.trainer.ppo.core_algos import get_policy_loss_fn; print(get_policy_loss_fn('up'))" resolves the registered loss.
• End-to-end reference run: bash examples/grpo_trainer/run_up_grpo_qwen3_8b_fsdp.sh (Qwen3-8B, GSM8K/MATH, Table A3 hyperparameters). No full training run is included in CI (out of scope).

API and Usage Example

Enable UP-GRPO by selecting the policy-loss mode; everything else follows the standard GRPO recipe:

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    actor_rollout_ref.actor.policy_loss.loss_mode=up \
    actor_rollout_ref.actor.clip_ratio=0.2 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.loss_agg_mode=token-mean \
    ...

The upper clip no longer affects positive advantages under loss_mode=up.

Design & Code Changes

• verl/trainer/ppo/core_algos.py: new @register_policy_loss("up") compute_policy_loss_up, mirroring vanilla's agg_loss(..., **config.global_batch_info) call, dual-clip handling, and metric keys.
• verl/workers/config/actor.py: add up to the PolicyLossConfig.loss_mode docstring.
• verl/trainer/config/actor/actor.yaml: note up (UP-GRPO) in the loss_mode comment.
• docs/algo/up.md: new algorithm page (motivation, Eq. 15, configuration, reference); registered in the docs/index.rst Algorithms toctree after GRPO.
• examples/grpo_trainer/run_up_grpo_qwen3_8b_fsdp.sh: runnable example based on the Qwen3-8B GRPO script with loss_mode=up and Table A3 hyperparameters.
• README.md: cite UP-GRPO in the algorithm lists.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks (ruff / ruff-format / sanity hooks pass on changed files).
• Add / Update the documentation (docs/algo/up.md + toctree).
• Add unit or end-to-end test(s) to the CI workflow. Not feasible here: this is a new RL algorithm whose behavior is validated by training experiments rather than CI; a CI smoke test is intentionally out of scope for this PR.
• Once your PR is ready for CI, send a message in the ci-request channel.
• N/A — this PR does not modify the recipe submodule.

Made with Cursor

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gemini-code-assist]

Code Review

This pull request introduces support for UP-GRPO (Unbounded Positive Asymmetric Optimization), a plug-and-play policy loss modification of GRPO designed to address the exploration-stability dilemma. The changes include the core implementation of the UP-GRPO policy loss in verl/trainer/ppo/core_algos.py, configuration updates to support the new up loss mode, a comprehensive documentation guide, and an example FSDP training script for Qwen3-8B. There are no review comments to address, and I have no additional feedback to provide as the implementation is clean and well-documented.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.