acceptodds
Under review as a conference paper at ICLR 2027

ESPO: Early-Stopping Proximal Policy Optimization

Abstract

When a reasoning trajectory becomes unrecoverable, standard reinforcement learning algorithms still generate until an end-of-sequence (EOS) token is generated or the rollout horizon is reached. This can waste computation and add estimation noise from post-failure tokens. We propose (arly-topping Proximal olicy ptimization), which identifies likely trajectory failures on-the-fly and terminates rollouts early. At each generation step, ESPO computes a surrogate regret using only the logits already computed during sampling, and terminates when the normalized, smoothed deviation exceeds a critic-dependent threshold. Truncated trajectories are treated as absorbing failure states with a terminal penalty at the detected stopping step, propagated through the retained prefix by GAE, without an additional reward model or human annotation. Across three backbones from two model series, on both mathematical reasoning and code generation, ESPO reaches on AIME 2024, on AMC 2023 and on MATH-500 with DeepSeek-R1-Distill-Qwen-7B — each the best figure among the five methods we compare, as is its average over six benchmarks — while decoding 21.7% fewer rollout tokens than PPO and cutting per-step wall-clock by 23.2%. Our code is available in the supplementary material.

Primary area:
reinforcement learning
Keywords:
reinforcement learning for LLMs, proximal policy optimization, efficient rl training, llm reasoning, early stopping

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X45QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper addresses wasted computation in reinforcement learning for LLM reasoning, where rollouts continue even when trajectories are unlikely to recover from errors.

  2. The authors propose ESPO, which uses logit-based surrogate regret and critic values to terminate likely failing rollouts early. It assigns a terminal penalty and propagates it through GAE.

Critical strengths
  1. ESPO demonstrates consistent performance improvements while reducing rollout tokens and training time across different model scales and reasoning tasks.

  2. The method requires no additional reward models or annotations. The paper also provides ablation studies and diagnostic analyses to evaluate its effectiveness.

Critical weaknesses
  1. ESPO combines post-action surrogate regret and pre-action critic values to determine whether a rollout should be terminated. However, it remains unclear how reliably these signals reflect the probability of successful continuation after the sampled token. Although Appendix A.4 introduces a monotonicity assumption relating these signals to future success, the paper does not directly examine whether this assumption holds in practice. Comparing the stopping signals with estimated post-action success probabilities would help justify the proposed criterion.

  2. Although the paper reports correct-yet-truncated rates and compares several stopping strategies, these results do not fully establish whether ESPO can distinguish low-recoverability prefixes from temporary reasoning errors that may be corrected later. Moreover, premature termination assigns a negative terminal reward to potentially recoverable prefixes, altering both critic targets and prefix advantages. Although the paper includes a terminal-penalty ablation and a theoretical analysis of these changes, their effects on recoverable prefixes that are prematurely terminated remain insufficiently examined. An analysis of stopping positions using multiple continuations from stopped prefixes would help clarify this limitation.

Questions for authors

Could the authors examine how the proposed stopping signals relate to actual prefix-level recoverability? For example, success probabilities could be estimated from multiple continuations at selected stopping points and compared with the regret and critic signals. This analysis would help assess the reliability of the proposed stopping criterion.

Reviewer_fkmROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ESPO, an early-stopping mechanism for PPO-based RL of language models. ESPO monitors a normalized token-level log-probability gap and a critic value threshold to terminate potentially unmeaningful rollouts, assigning a terminal penalty for current step and propagating its through GAE to prefix. Experiments across mathematical reasoning and code generation show consistent improvements with reduced rollout tokens and training time.

Critical strengths
  1. The method reuses actor logits and critic values without requiring a separately trained process reward model or stopping network, and could be lightly integrated into PPO algorithm.
  2. The proposed ESPO shows effectiveness with improved performance on mathematical and coding benchmarks, while reducing rolout tokens and practical training-time.
Critical weaknesses
  1. Prior works such as ARRoL and Step-GRPO have already explored early termination during RL training. Although ESPO introduces a different stopping criterion and penalized prefix updates, the lack of direct comparison makes its incremental contribution difficult to assess.
  2. The stopping threshold uses the pre-action critic value with a fixed lower bound (Eq. 11). Since the critic may predict negative values after learning from penalized trajectories, the gate can effectively reduce to a regret-only threshold or floor determined threshold. The paper does not analyze how frequently this occurs or how much the critic actually influences stopping decisions.
  3. ESPO introduces several additional hyperparameters, including the target stop rate, EMA coefficients, and controller update rate. Appendix H shows noticeable variation in accuracy and rollout length across these settings, raising questions about the sensitivity to stopping hyperparameters across different models and tasks.
  4. The running normalization statistics are updated using only generated tokens (Eqs. 8–9), while early stopping systematically removes suffix tokens. This may introduce intervention between the stopping decisions and the estimated regret distribution, but the paper does not investigate its effect on training stability.
Questions for authors
  1. Can ESPO be extended to critic-free algorithms such as GRPO, GSPO? If so, how would the value-gated stopping criterion be adapted?
  2. Could the authors provide additional evidence that the stopping signal predicts low continuation success probability, rather than merely low token probability?
Reviewer_F85pOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ESPO, a modification to PPO rollout collection that terminates potentially unsuccessful reasoning trajectories early. It computes a token-level log-probability gap between the policy mode and the sampled token, normalizes and smooths this signal, and applies a critic-dependent stopping threshold. Truncated trajectories receive a terminal failure penalty, which changes retained-prefix advantages through GAE. Experiments cover mathematical reasoning on three backbones and code generation on one backbone, with comparisons against PPO, GRPO, GSPO, and DAPO. On the 7B math setting, ESPO increases average accuracy from PPO's 62.56 to 64.89 while reducing rollout tokens by 21.7% and per-update wall-clock time by 23.2%. The paper also presents component ablations, counterfactual continuation diagnostics, and conditional theoretical analysis.

Critical strengths
  1. The paper addresses a practical source of cost in long-rollout RL training. The method is integrated clearly with PPO, including the trigger token, terminal reward, zero bootstrap, and retained-prefix GAE. This makes the learning effect of truncation explicit.

  2. The efficiency evaluation goes beyond token counts. Results cover three backbones and two domains, and Tables 3 and 9 report total update time including critic computation, optimization, and warmup. Disabling stopping during evaluation also separates training-time savings from the accuracy of complete model responses.

  3. The paper includes useful ablations and diagnostics. Rate-matched random stopping performs worse at a similar average rollout length, supporting selective stopping over arbitrary shortening. Counterfactual continuations help distinguish omitted suffixes from changes in natural response length. The theoretical discussion also carefully separates oracle masking from practical ESPO and states the limits of its guarantees.

Critical weaknesses
  1. The analysis could further examine whether stopped reasoning prefixes can still lead to correct answers. The authors already provide useful checks by allowing generation to continue after a stop is triggered. However, one incorrect completion does not establish that a prefix has little chance of success. The reported stopping and false-stop rates imply that approximately 10.8%–12.9% of stopped rollouts produce correct answers when allowed to finish, although these rollouts receive a failure reward of −1 during training. Sampling several complete responses from a subset of stopped prefixes, and comparing their success rates with those of unstopped prefixes, would help clarify whether the rule identifies reasoning with low chances of success and whether it penalizes useful exploration.

  2. The comparisons do not fully establish the advantage of the specific stopping rule. The paper includes single-signal and rate-matched random baselines, as well as comparisons of accuracy and token cost. However, the selection and tuning ranges of the value-only and regret-only thresholds are not specified, and only limited operating points are reported. The paper also discusses confidence-based stopping but does not compare an alternative confidence rule within the same PPO training pipeline. Matching the terminal penalty and generation budget while varying the stopping signal would better isolate the value of the proposed regret–value design. This would be a controlled training-time comparison, rather than a direct comparison with the original inference-time results of DeepConf.

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.