- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Single-Rollout Pass@k Policy Optimization (SR-PPO), which combines Monte Carlo advantages with prompt-level Pass@k reweighting. An online critic provides both prefix-dependent baselines and estimates of prompt success probabilities, enabling training with one rollout per prompt. Experiments using Qwen3-1.7B on DeepScaleR report improved stability over the tested PPO configurations, with the on-policy variant achieving performance broadly comparable to GRPO. Additional analyses investigate critic quality, gradient noise, and interference across prompt difficulty groups.
Critical strengths
-
A practical approach to single-rollout training. The method uses an online critic without dedicated large-scale critic pretraining, providing a relatively simple way to incorporate Pass@k reweighting when multiple contemporaneous rollouts per prompt are unavailable.
-
Encouraging empirical stability. Experiments across three training seeds show sustained learning for the proposed configuration, and the on-policy variant is broadly competitive with GRPO across several reasoning benchmarks. These results support the practical feasibility of the approach within the evaluated setting.
Critical weaknesses
-
Insufficient validation of the practical estimator. The theoretical gradient assumes true success probabilities, whereas the implementation uses critic-derived weights whose calibration and induced gradient bias are not evaluated. Sentence-level correctness AUC does not directly validate these probability estimates.
-
Unclear attribution of the stability gains. The main PPO comparison also changes the critic design, while the noise analysis combines an SR-PPO checkpoint with GRPO-generated trajectories. Without clearer controls and policy-mismatch handling, these results do not sufficiently establish the proposed stabilization mechanism.
-
Limited evidence of efficiency and robustness. Computational budgets are matched approximately without reporting actual runtime or GPU-hours. Experiments further cover only one model and training dataset with limited hyperparameter exploration, leaving both the practical cost advantage and robustness across configurations insufficiently established.
Questions for authors
-
Does the Pass@1 MC ablation in Figure 3 retain exactly the same critic architecture, initialization, sigmoid output, BCE-plus-MSE loss, and update schedule as SR-PPO? Does the stability difference persist after controlling for update magnitude and learning rate?
-
How accurately does estimate the current policy’s prompt-level success probability? How do the estimated weights and gradient directions compare with those obtained from independent multi-rollout success estimates?
-
In §5.5, which policy defines the score gradients, and how is the mismatch between the SR-PPO checkpoint and GRPO-generated trajectories handled? Do the principal noise and interference findings persist with matched-policy trajectories and checkpoints preceding training collapse?
Additional Experiments and Clarifications Part II
We agree that Figure 2 and Table 2 do not show consistent improvements over GRPO. Our main contribution is to stabilize a PPO-style algorithm with one rollout per prompt while achieving performance comparable to GRPO.
We also study why a learned critic may not provide further performance gains over GRPO. Using sentence-level AUC, we assess whether the critic can distinguish correct from incorrect intermediate reasoning. Our hypothesis is that a critic that cannot reliably locate errors may offer limited benefits for credit assignment. This is a possible explanation, not a proven limit on PPO’s performance.
Our contribution is therefore twofold: a method for stable single-rollout training, and an analysis suggesting that limited critic accuracy may constrain further performance gains. We believe this analysis is valuable to the community because it highlights challenges in improving critic-based credit assignment, even without consistent gains over GRPO.
We thank the reviewer for highlighting the need to evaluate the critic’s probability estimates directly. Our analysis suggests that the critic captures relative prompt-level success probabilities somewhat better than those at intermediate states. This may help explain why SR-PPO achieves performance comparable to GRPO without consistently outperforming it.
To examine this hypothesis, we directly evaluate the critic’s probability predictions against empirical success rates. These evaluations are distinct from our sentence-level correctness AUC, which measures agreement with external correctness judgments.
For each of 128 prompts, we generated one response using a shared actor and selected five breakpoints near 10%, 30%, 50%, 70%, and 90% of its length. We sampled eight continuations per breakpoint from the target policy, yielding 640 prefix states and 5,120 continuations. At the prompt level, we estimated success probabilities using 20 sampled responses per prompt.
Alongside MSE and BCE, we report ordering scores that evaluate the critic’s ability to rank intermediate breakpoints within a response and to rank prompts by success probability. The higher prompt-level ordering scores suggest that the critic captures relative difficulty across prompts more reliably than changes in success probability within a response. However, this advantage does not extend consistently to absolute probability estimation. The corresponding RMSE is the square root of the reported MSE, which is about 33% error in predicting the probability, implying poor calibration for the critic. BCE at the prompt-level can be higher because prompt-level outcomes are more uncertain, with a longer horizon to the end of a response than breakpoints.
We also compute the per-breakpoint losses for pass@4. SR-PPO critic achieves lower MSE and BCE than the PPO critic for both Pass@1 and Pass@4 prediction.