Add UP (Unbounded Positive Asymmetric Optimization) to trl.experimental

[yashb98]

What does this PR do?

Adds trl/experimental/up/, implementing Unbounded Positive Asymmetric Optimization from the UP paper as a GRPOTrainer subclass that overrides _compute_loss. It follows the trl/experimental/gmpo/ layout.

Fixes huggingface#6407

I originally wrote this as a core loss_type="up" value, then noticed recent new methods have been landing in trl/experimental/ instead, so I moved it. That branch still exists and the two are numerically identical, so if you would rather have this as a core loss_type I can switch it back in a few minutes. Happy either way.

Method

UP splits on the sign of the advantage. For positive advantages it replaces the importance ratio with a self-anchored one:

anchored_log_ratio = per_token_logps - per_token_logps.detach()
up_per_token_loss = -anchored_log_ratio.exp() * advantages

Its forward value is exactly 1 and its gradient is the plain REINFORCE gradient, so positive updates are never clipped and do not depend on π_old. Non-positive advantages keep the usual clipped surrogate as a safeguard. Aggregation follows DAPO's global active-token normalization, matching the paper's UP-DAPO instantiation.

Notes for review

• One core line, in base_trainer.py: "UPTrainer" added to _TELEMETRY_TRAINERS, since the comment there says a new trainer needs an explicit entry. Nothing else outside trl/experimental/up/.
• use_liger_kernel=True raises NotImplementedError. GRPOTrainer.compute_loss routes to the fused Liger loss before _compute_loss is reached, so without the guard the trainer would silently optimize the GRPO objective instead of UP. Worth flagging that GMPOTrainer looks like it has the same hole; I have not opened anything about it since I may be misreading the dispatch.
• epsilon_high has no effect with the default delta=None, and more generally whenever delta >= 1 + epsilon_high. Positive advantages skip clipping, and for non-positive ones the upper bound is dominated. Only a delta below 1 + epsilon_high makes it bind. Table A1 of the paper lists no ε_high for UP-DAPO accordingly. There is a test for this.
• importance_sampling_level="sequence" changes only the non-positive branch. The positive branch is unaffected, because the global active-token normalization exactly cancels the sequence-level length normalization. Also tested.
• UPConfig adds no fields. It exists as the autodoc surface for the semantics above and for symmetry with the other experimental configs.
• Only the low-side clip metrics are logged. The high-side ones would be identically zero, since is_high_clipped requires a positive advantage and UP routes those to the unclipped branch.

Tests

UP's positive branch has a forward value of exactly 1, so asserting on the loss alone cannot tell a correct implementation from one that detached the wrong tensor. The gradient is what distinguishes them, so the tests check gradients, and then the loss value separately:

• positive advantages: per-parameter gradients match the REINFORCE reference, the loss equals the closed form, and both stay bitwise identical when old_per_token_logps changes
• non-positive advantages: loss and gradients bitwise identical to GRPOTrainer's dapo
• delta caps the non-positive branch, and gates whether epsilon_high binds at all
• importance_sampling_level leaves the positive branch unchanged
• beta > 0 adds and logs the KL term
• the Liger guard raises, and end-to-end training runs

Verification

Rebased on de63151. tests/experimental/test_up_trainer.py: 8 passed. ruff, ruff format and the doc-builder hook pass on the changed files.

I also checked this module against the core loss_type="up" branch it replaces, on identical seeded inputs: loss, gradient norm and gradient sum are bit-identical across six cases (token and sequence levels, with all-positive, all-negative and mixed advantages). The move did not change the objective.

Results, including a negative one

Small-scale runs on Qwen3-0.6B and Qwen3-4B with GSM8K. These do not show a reward win and I am not claiming one.

What reproduced: the clip is one-sided, and entropy separates upward for UP, matching Fig. 3(a).

What did not: with beta=0, UP's entropy crosses 2.0 nats for good around step 236 and reward collapses without recovering. It is capability loss rather than truncation, since completions mostly finish and still score near zero. At 4B the same pattern starts, entropy passes 2.0 by step 173 and mean reward falls from 0.87 over the first 100 steps to 0.30 over the last 100 while dapo holds 0.71, but that arm is only 500 steps and partially recovers late. So it is a weaker version of the same failure rather than a repeat, and at least not a 0.6B-only artifact.

Adding a KL anchor (beta=0.04) removes the collapse. Entropy stays below 2.0 for 2000 steps and reward holds around 0.35. That fits the paper: its UP-GRPO keeps the KL term (Eq. 15), and its UP-DAPO results are at 14B.

The docs carry a one-line version of that next to the config example, pointing at the paper's own UP-GRPO variant rather than at my numbers. I have no 14B run and am not extrapolating to one.

curves: 2000 steps at 0.6B, the KL-anchor overlay, and the 4B run

Prior art

verl#7022 adds the same method to verl. Opened 2026-07-13, still open and unmerged, and the author appears to be the paper's first author. The positive branch is mathematically identical. The negative branches differ only in defaults: verl's dual clip (clip_ratio_c) is on at 3.0, TRL's equivalent (delta) is off unless you set it.

Jackie2049/trl#6 and Jackie2049/verl#9 are fork-internal, not upstream.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case. I opened Add UP (Unbounded Positive) loss as a GRPO loss_type huggingface/trl#6407 for this; no maintainer has replied there yet.
• Did you make sure to update the documentation with your changes? New docs/source/up.md, a paper_index.md subsection, and a _toctree.yml entry.
• Did you write any new necessary tests? See above.

AI writing disclosure

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone once CI is green. @qgallouedec or @kashif may be interested, given the GRPO surface.