acceptodds
Under review as a conference paper at ICLR 2027

Learning with a Single Rollout via Pass@k Policy Optimization

Abstract

Group Relative Policy Optimization (GRPO) typically requires multiple rollouts per prompt to estimate relative credit, incurring substantial sampling cost and exacerbating training asynchrony. This limitation is particularly severe for scientific reasoning agents, where each trajectory may involve long-horizon reasoning, multiple turns, and parallel sub-runs. We study single-rollout reinforcement learning, where only one rollout is collected per prompt, and find that naive training can stagnate or become severely unstable. We introduce Single-Rollout Pass@ Policy Optimization (SR-PPO), which combines Monte Carlo token-level advantage estimation with Pass@ gradient reweighting. An online critic without large-scale pretraining is used both as a prefix-dependent baseline for credit assignment and to estimate the Pass@ gradient weight. Using only one rollout per prompt, SR-PPO achieves stable learning and performance comparable to GRPO across reasoning and logic benchmarks, even under off-policy drift that degrades the performance of standard Pass@1 PPO. A key factor in this stability is Pass@ reweighting, which emphasizes harder prompts while downweighting high-success prompts, for which imperfect credit estimates tend to produce noisier policy updates. Through analyses of critic quality, gradient noise, and update behavior, we investigate how Pass@ reweighting stabilizes single-rollout learning and why residual critic errors may limit SR-PPO’s ability to outperform GRPO.

Primary area:
reinforcement learning
Keywords:
Large language model, Reinforcement learning, Math reasoning, Single-rollout, Pass@k policy optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8cfBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Single-Rollout Pass@k Policy Optimization (SR-PPO), which combines Monte Carlo advantages with prompt-level Pass@k reweighting. An online critic provides both prefix-dependent baselines and estimates of prompt success probabilities, enabling training with one rollout per prompt. Experiments using Qwen3-1.7B on DeepScaleR report improved stability over the tested PPO configurations, with the on-policy variant achieving performance broadly comparable to GRPO. Additional analyses investigate critic quality, gradient noise, and interference across prompt difficulty groups.

Critical strengths
  • A practical approach to single-rollout training. The method uses an online critic without dedicated large-scale critic pretraining, providing a relatively simple way to incorporate Pass@k reweighting when multiple contemporaneous rollouts per prompt are unavailable.

  • Encouraging empirical stability. Experiments across three training seeds show sustained learning for the proposed configuration, and the on-policy variant is broadly competitive with GRPO across several reasoning benchmarks. These results support the practical feasibility of the approach within the evaluated setting.

Critical weaknesses
  • Insufficient validation of the practical estimator. The theoretical gradient assumes true success probabilities, whereas the implementation uses critic-derived weights whose calibration and induced gradient bias are not evaluated. Sentence-level correctness AUC does not directly validate these probability estimates.

  • Unclear attribution of the stability gains. The main PPO comparison also changes the critic design, while the noise analysis combines an SR-PPO checkpoint with GRPO-generated trajectories. Without clearer controls and policy-mismatch handling, these results do not sufficiently establish the proposed stabilization mechanism.

  • Limited evidence of efficiency and robustness. Computational budgets are matched approximately without reporting actual runtime or GPU-hours. Experiments further cover only one model and training dataset with limited hyperparameter exploration, leaving both the practical cost advantage and robustness across configurations insufficiently established.

Questions for authors
  • Does the Pass@1 MC ablation in Figure 3 retain exactly the same critic architecture, initialization, sigmoid output, BCE-plus-MSE loss, and update schedule as SR-PPO? Does the stability difference persist after controlling for update magnitude and learning rate?

  • How accurately does q^(s0)\hat q(s_0) estimate the current policy’s prompt-level success probability? How do the estimated weights and gradient directions compare with those obtained from independent multi-rollout success estimates?

  • In §5.5, which policy defines the score gradients, and how is the mismatch between the SR-PPO checkpoint and GRPO-generated trajectories handled? Do the principal noise and interference findings persist with matched-policy trajectories and checkpoints preceding training collapse?

Reviewer_6fwWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tries to reduce rollouts needed in GRPO, and considers to do RL with a single rollout. Two techniques are involved: 1) a critic network to learn the success rate given any prefix in the response; 2) pass@k reweighting to stabilize training. Experiments are done on Math and logical tasks. The paper also analyzes gradient noise.

Critical strengths

The paper tries to address a practical challenge in RL post-training for reasoning: rollout is becoming expensive and is a huge overhead in RL. The paper directly proposes a method that uses a single rollout.

Critical weaknesses
  1. The motivation of the method is not sound. The paper involves pass@k reweighting, and this is estimated from pass@1 according to Equation (2). However, Equation (2) only gives a biased estimation of pass@k. Additionally, pass@1 is learned from a single rollout, making the estimate of pass@1 itself highly noisy.
  2. The empirical performance is not significant and a critical experiment need to be added. Figure 2 does not show a clear trend that SR-PPO is better than GRPO. Given the variance reported in Table 2, the performance gain on logical tasks is insignificant as well. The paper misses a critic experiment, showing that pass@1 as well pass@k can be estimated well with the critic network 2, which answers weakness 1 from an empirical perspective.

Update: I admit that the paper's main contribution is increasing stability under single rollout. Therefore, I would question more why a biased policy gradient stabilizes training in the long run from a theoretical perspective.

Questions for authors

I would like the authors to address weakness 1 and 2, either theoretically or empirically.

Reviewer_GBRiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SR-PPO, aiming to achieve stable reinforcement learning training under the condition of sampling only one answer per question. The method utilizes the final success probability of the online critic's prediction prefix. On the one hand, it constructs a Monte Carlo advantage by subtracting the value prediction from the final reward; on the other hand, it reduces the relative contribution of easy questions through Pass@k weights. Experiments on Qwen3-1.7B and DeepScaleR mathematical training settings show that the method can learn stably and achieve performance close to GRPO on mathematical and some transfer benchmarks. The authors further analyze its stability and the relationship between gradient interference between questions of different difficulty levels through component ablation and gradient diagnostics.

Critical strengths

The method is concise, with a clear mathematical connection between the objective and reweighting. Sections 4.1–4.3 transform the Pass@k objective into success-rate-based question weights and incorporate Monte Carlo advantage for training. The gradient derivation under true success rate holds, clearly explaining why this objective reduces the relative contribution of easy questions; furthermore, optimizing k in the objective does not require sampling k responses during training. Moreover, under Qwen3-1.7B/DeepScaleR settings, SR-PPO can be trained continuously up to 600 steps and approaches GRPO on multiple metrics. Figure 3 shows that replacing MC advantage or removing Pass@k reweighting significantly impairs training; Figure 4 further provides directional evidence of gradient interference between questions. These results provide substantial support for the explanation of component effectiveness and stability.

Critical weaknesses
  1. The incremental benefit of SR-PPO over existing PPO stabilization approaches is not sufficiently established. VC-PPO and VAPO address instability through critic pretraining and improved value/advantage estimation, while ORZ already employs Monte Carlo advantages of the form R−V(st)R-V(s_t). Although Figure 3’s Pass@1 MC ablation and the gradient-noise analysis provide evidence for Pass@k reweighting, the main PPO–SR-PPO comparison simultaneously changes the reweighting mechanism and the critic parameterization/training objective: an unconstrained value head with MSE versus a sigmoid output with BCE+MSE (§4.3). Moreover, Appendix B reports shared, untuned hyperparameters without explicitly documenting the critic and advantage-normalization settings for each ablation. Comparisons with competitive stabilized PPO configurations, adapted to the single-rollout setting or matched for total sampling and training cost, would help establish whether the proposed reweighting provides additional benefits beyond existing stabilization techniques.

  2. All training experiments use Qwen3-1.7B on DeepScaleR, leaving the stability gains unvalidated across model families, parameter scales, and initialization conditions. Evaluation on multiple reasoning benchmarks demonstrates task transfer, but does not establish the robustness of the training method across models. This limitation is particularly relevant because the proposed explanation depends on the relationship between prompt difficulty, critic estimation quality, and gradient noise—properties that may change substantially with model capability and scale. Although the authors explicitly acknowledge this restricted scope, the current evidence does not establish whether Pass@k reweighting consistently improves stability beyond the tested configuration or under what conditions its benefits diminish.

Questions for authors
  1. Do the Pass@1 MC and Pass@k MC variants in Figure 3 share identical critic initialization, parameterization, training loss, update schedules, and advantage normalization, differing only in kk? Could you quantify the contribution of critic design by comparing Pass@1 MC with an unconstrained critic trained using MSE, a sigmoid critic with MSE, and a sigmoid critic with BCE+MSE? Given the shared, untuned hyperparameters in Appendix B, does the benefit of Pass@k reweighting persist under comparable, limited tuning budgets for learning rates and advantage normalization?

  2. How does SR-PPO compare with ORZ-style MC PPO and PPO incorporating VC-PPO/VAPO-style value-stabilization techniques? Could you provide comparisons adapted to the single-rollout setting or matched for total sampling and training cost, including any additional critic-pretraining cost, to establish the incremental benefit of Pass@k reweighting?

  3. It is recommended to conduct experiments with another model family or a larger-scale model to verify the applicability of the method.

If you answer my question, I will consider raising my rating.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Additional Experiments and Clarifications Part II

  1. Figure 2 does not show a clear trend that SR-PPO is better than GRPO. Given the variance reported in Table 2, the performance gain on logical tasks is insignificant as well.

We agree that Figure 2 and Table 2 do not show consistent improvements over GRPO. Our main contribution is to stabilize a PPO-style algorithm with one rollout per prompt while achieving performance comparable to GRPO.

We also study why a learned critic may not provide further performance gains over GRPO. Using sentence-level AUC, we assess whether the critic can distinguish correct from incorrect intermediate reasoning. Our hypothesis is that a critic that cannot reliably locate errors may offer limited benefits for credit assignment. This is a possible explanation, not a proven limit on PPO’s performance.

Our contribution is therefore twofold: a method for stable single-rollout training, and an analysis suggesting that limited critic accuracy may constrain further performance gains. We believe this analysis is valuable to the community because it highlights challenges in improving critic-based credit assignment, even without consistent gains over GRPO.

  1. The paper misses a critic experiment, showing that pass@1 as well pass@k can be estimated well with the critic network 2, which answers weakness 1 from an empirical perspective.

We thank the reviewer for highlighting the need to evaluate the critic’s probability estimates directly. Our analysis suggests that the critic captures relative prompt-level success probabilities somewhat better than those at intermediate states. This may help explain why SR-PPO achieves performance comparable to GRPO without consistently outperforming it.

To examine this hypothesis, we directly evaluate the critic’s probability predictions against empirical success rates. These evaluations are distinct from our sentence-level correctness AUC, which measures agreement with external correctness judgments.

For each of 128 prompts, we generated one response using a shared actor and selected five breakpoints near 10%, 30%, 50%, 70%, and 90% of its length. We sampled eight continuations per breakpoint from the target policy, yielding 640 prefix states and 5,120 continuations. At the prompt level, we estimated success probabilities using 20 sampled responses per prompt.

Evaluation levelMean ordering score ↑Success rate weighted ordering score ↑Pass@1 MSE ↓Pass@1 BCE ↑
Intermediate breakpoints0.65150.54270.11230.4365
Prompt s0s_00.73800.69860.11000.5393

Alongside MSE and BCE, we report ordering scores that evaluate the critic’s ability to rank intermediate breakpoints within a response and to rank prompts by success probability. The higher prompt-level ordering scores suggest that the critic captures relative difficulty across prompts more reliably than changes in success probability within a response. However, this advantage does not extend consistently to absolute probability estimation. The corresponding RMSE is the square root of the reported MSE, which is about 33% error in predicting the probability, implying poor calibration for the critic. BCE at the prompt-level can be higher because prompt-level outcomes are more uncertain, with a longer horizon to the end of a response than breakpoints.

We also compute the per-breakpoint losses for pass@4. SR-PPO critic achieves lower MSE and BCE than the PPO critic for both Pass@1 and Pass@4 prediction.

MetricSR-PPOPPO
Pass@4 MSE ↓0.14190.1920
Pass@4 cross-entropy ↓0.69450.9528
MSE against empirical Pass@1 ↓0.11230.1228
Cross-entropy against empirical Pass@1 ↓0.43650.4991

Thank the authors for the response. I understand that the paper's main contribution is increasing stability under single rollout, rather than improving over GRPO. However, stability is highly related to the bias of the gradient. I would raise my concern why a biased policy gradient stabilizes training in the long run from a theoretical perspective.

Therefore, I decide to raise my score to 2 (weak reject), but the paper is still insufficient to reach an "accept" recommendation.

AuthorsOpenReview ↗

Why Pass@k Reweighting Stabilizes with An Imperfect Critic

We sincerely appreciate your thoughtful engagement and the time you have taken to carefully consider our work. Thank you for reconsidering your assessment and recognizing the paper’s contribution to single-rollout training stability. We greatly appreciate it.

We understand that the remaining concern is whether the observed stabilization can be justified when the practical gradient estimator uses an imperfect critic.

Please note that any value estimator has approximation error, whether for GRPO or PPO. Our task is to stabilize learning under such value approximation error.

As the reviewer requested, please see Xu, Wang, and Liang (2020), “Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithms” (Theorem 3), and Agarwal, Kakade, Lee, and Mahajan (2021), “On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift” (Theorem 29), which show that policy gradient methods can still learn a near-optimal policy under such approximation biases.

Our analysis further provides an empirical explanation for the observed stabilization.

The critic evaluation shows that prompt-level ranking is more reliable than ranking intermediate states, as reported in the first table addressing point 3, and that the critic performs near-randomly when judging the local correctness of each sentence, as shown in Table 6. This leads to easy prompts producing noisier updates, since 85% of steps are correct for an easy prompt versus 49% for a hard prompt on average.

Pass@k reweighting mainly uses the prompt-level value to increase the share of hard data in the gradient from 30.5% to 65%, since hard data produces less noisy updates than easy data.

Our contribution is therefore a simple method that improves the stability of single-rollout training under an imperfect, evolving critic, together with a thorough analysis of the critic’s limitations and the mechanism by which Pass@k stabilizes learning. We believe these empirical and analytical findings, supported by a rigorous and thorough experimental analysis, are useful to the community. We would be grateful if you could revisit your assessment in light of our contribution and our thorough experiments. We greatly appreciate your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.