- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
The paper addresses wasted computation in reinforcement learning for LLM reasoning, where rollouts continue even when trajectories are unlikely to recover from errors.
-
The authors propose ESPO, which uses logit-based surrogate regret and critic values to terminate likely failing rollouts early. It assigns a terminal penalty and propagates it through GAE.
Critical strengths
-
ESPO demonstrates consistent performance improvements while reducing rollout tokens and training time across different model scales and reasoning tasks.
-
The method requires no additional reward models or annotations. The paper also provides ablation studies and diagnostic analyses to evaluate its effectiveness.
Critical weaknesses
-
ESPO combines post-action surrogate regret and pre-action critic values to determine whether a rollout should be terminated. However, it remains unclear how reliably these signals reflect the probability of successful continuation after the sampled token. Although Appendix A.4 introduces a monotonicity assumption relating these signals to future success, the paper does not directly examine whether this assumption holds in practice. Comparing the stopping signals with estimated post-action success probabilities would help justify the proposed criterion.
-
Although the paper reports correct-yet-truncated rates and compares several stopping strategies, these results do not fully establish whether ESPO can distinguish low-recoverability prefixes from temporary reasoning errors that may be corrected later. Moreover, premature termination assigns a negative terminal reward to potentially recoverable prefixes, altering both critic targets and prefix advantages. Although the paper includes a terminal-penalty ablation and a theoretical analysis of these changes, their effects on recoverable prefixes that are prematurely terminated remain insufficiently examined. An analysis of stopping positions using multiple continuations from stopped prefixes would help clarify this limitation.
Questions for authors
Could the authors examine how the proposed stopping signals relate to actual prefix-level recoverability? For example, success probabilities could be estimated from multiple continuations at selected stopping points and compared with the regret and critic signals. This analysis would help assess the reliability of the proposed stopping criterion.