[trainer] feat: early stopping via early_stop_patience config

[Secbone]

What does this PR do?

Add optional early_stop_patience config field for RayPPOTrainer. When set > 0 and critic/rewards/mean does not improve for that many consecutive steps, the training loop terminates early. This helps avoid wasting compute when the model has converged.

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: gh pr list --search "early stop early_stop" — no duplicates found.
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

Set early_stop_patience=3 in the trainer config and verify that training stops after 3 steps without reward improvement. The feature is disabled by default (early_stop_patience=0) and has no side effects when not configured.

API and Usage Example

trainer:
  early_stop_patience: 5  # stop training if reward doesn't improve for 5 consecutive steps

Design & Code Changes

• verl/trainer/ppo/ray_trainer.py: Insert ~20 lines of early stopping logic before progress_bar.update(1) in the training loop.
  • Reads self.config.trainer.get("early_stop_patience", 0) — no config schema change needed.
  • Tracks best reward via self._best_reward and a patience counter.
  • On patience exhaustion, sets is_last_step = True and self.total_training_steps = self.global_steps to trigger normal loop termination, checkpoint saving, and cleanup.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: End-to-end test requires multi-GPU training setup not available in CI. The change is small (~20 lines) and guarded by an explicit config flag (default off).
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)
• If your PR is related to the recipe submodule, please also update the reference to the submodule commit via git submodule update --remote or cd recipe && git pull origin main.

[gemini-code-assist]

Code Review

This pull request introduces early stopping logic to the PPO Ray trainer based on the mean critic rewards and a configured patience threshold. However, there is a logic bug in the initialization of the best reward tracking: using sequential if statements instead of an elif causes the patience counter to prematurely increment on the very first step, which can trigger early stopping immediately if patience is set to 1. Using an elif statement resolves this issue.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

There is a logic bug in the initialization of _best_reward. When _best_reward is not yet set, the first if block initializes it and sets _patience_counter = 0. However, because the subsequent check is a separate if statement rather than an elif, it immediately compares reward > self._best_reward (which is False since they are equal) and executes the else block, incrementing _patience_counter to 1 on the very first step. If early_stop_patience is set to 1, this will trigger early stopping immediately on the first step. Changing the second if to elif resolves this issue.

Suggested change

	if not hasattr(self, "_best_reward"):
	self._best_reward = reward
	self._patience_counter = 0
	if reward > self._best_reward:
	self._best_reward = reward
	self._patience_counter = 0
	else:
	self._patience_counter += 1
	if not hasattr(self, "_best_reward"):
	self._best_reward = reward
	self._patience_counter = 0
	elif reward > self._best_reward:
	self._best_reward = reward
	self._patience_counter = 0
	else:
	self._patience_counter += 1

[wuxibin89]

Legacy RayPPOTrainer is deprecated, please use V1 trainer instead.