acceptodds
Withdrawn from ICLR 2027

Success-aware Policy Learning in RLVR

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) remains sample-inefficient on hard prompts, where most rollouts fail and the learner receives only a binary correctness signal. In the standard sample–verify–update loop, a rare successful rollout is used primarily as a target for parameter updates, leaving its reasoning content unused for acquiring subsequent training experience. We introduce Success-Aware Policy Learning (SAPL), which gives each verified success a second role of temporary in-context guidance for further sampling before the policy is updated. SAPL first discovers verifier-correct on-policy rollouts and selects the success with the highest likelihood under the current policy. The same model, with unchanged parameters, then conditions on this success to generate additional candidates. The verifier filters these candidates, and a weighted objective assimilates the verified responses into the prompt-only policy. SAPL requires no reference solution, external teacher, or verifier feedback beyond correctness, and uses no auxiliary context at inference. We show that SAPL achieves improved accuracy across Qwen3-4B, 8B, and 14B on eleven competition mathematics benchmarks by – percentage points over the strongest evaluated RLVR baseline and by – points over the strongest evaluated ground-truth-conditioned baseline. These results show that rare successes are valuable not only as update targets, but also as guides for acquiring new verified experience.

Primary area:
reinforcement learning
Keywords:
LLM Reasoning, RLVR, Sample Efficiency, Sparse Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BDR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Success-Aware Policy Learning (SAPL) to improve the sample efficiency of reinforcement learning with verifiable rewards (RLVR) on challenging reasoning tasks. SAPL follows a discover–amplify–assimilate framework: it first identifies successful on-policy trajectories, selects the highest-likelihood success to guide additional sampling, and then assimilates the verified guided responses into the original policy using surprisal-gated cross-entropy loss. The method requires neither external teacher models nor ground-truth solutions during training. Experiments on Qwen3-4B, 8B, and 14B across eleven mathematical reasoning benchmarks demonstrate consistent improvements over existing RLVR and answer-conditioned baselines.

Critical strengths
  • SAPL provides a practical approach to improving sample efficiency in sparse-reward RLVR by using successful trajectories not only for policy updates but also to guide subsequent data generation. This directly addresses a fundamental bottleneck in training on difficult reasoning tasks.

  • SAPL consistently outperforms multiple RLVR baselines across three Qwen3 model sizes and eleven mathematical reasoning benchmarks. The improvements in sample efficiency and gradient alignment provide encouraging evidence of the proposed framework's effectiveness.

Critical weaknesses
  • SAPL selects the highest-likelihood successful trajectory to guide subsequent sampling, which may bias exploration toward reasoning patterns already favored by the current policy. Although the authors provide a qualitative example demonstrating different solution strategies, a systematic analysis of exploration diversity is missing. This raises concerns about whether SAPL could reinforce existing reasoning modes and hinder exploration on tasks requiring novel solutions.

  • SAPL relies on the base model's ability to discover initial successful trajectories and generate useful alternatives conditioned on them. Although the experiments cover three Qwen3 model sizes, the training prompts are selected to have similarly low initial success rates. It remains unclear whether the method generalizes to substantially weaker models that struggle to discover initial successes or stronger models that may benefit less from success-conditioned sampling. This limits the evidence for the method's applicability across different model capabilities.

Questions for authors
  • How does success-conditioned sampling affect the diversity of generated reasoning trajectories compared with standard on-policy sampling? Have the authors considered whether selecting the highest-likelihood success could restrict exploration or lead to mode collapse? A comparison with random or diversity-aware success selection would help clarify this concern.
  • How does SAPL perform across different initial pass@1 regimes or on models with substantially different reasoning capabilities? In particular, does the method remain effective when the base model rarely discovers successful trajectories, or when the base model is already strong enough that success-conditioned sampling provides limited additional benefit?
  • How much of SAPL's performance improvement comes from success-conditioned sampling, highest-likelihood success selection, and surprisal-gated assimilation, respectively? A component-wise ablation, particularly against random success selection and standard cross-entropy assimilation, would help establish which components are essential to the reported gains.
1 comment

We thank the reviewer for recognizing that SAPL directly addresses a fundamental bottleneck in sparse-reward RLVR by using successful trajectories to guide data generation, and for noting the consistent improvements across three model sizes and eleven benchmarks, together with the evidence on sample efficiency and gradient alignment. We address the questions below.

W1 / Q1: Diversity and highest-likelihood selection

We thank you for the suggestion, we will consider it in the revised version.

W2 / Q2: Different pass@1 regimes

The low pass@1 regime, where the base model rarely solves the task, is exactly the setting SAPL targets. For each model, we select training prompts that it rarely solves, with an average pass@1 of 3% (Appendix A.2), and the gains hold across all three model sizes (Table 1). At the other end of the spectrum, where successes are frequent, standard RLVR already receives sufficient learning signal, which is not the setting we consider.

Q3: Component ablation

We thank you for the constructive feedback, we will consider it in the revised version of our work.

Reviewer_Xsr7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduced Success-Aware Policy Learning, which during the GRPO rollout process, rollout two rounds overall, and select the correct samples as the context for the second round rollouts. For this the model use the trajectory with the correct answer to generate the logits. Experimental results indicate that being suprvised on this term help to improve the performance in math reasoning.

Critical strengths
  • The paper is well-written and easy to follow.
  • The proposed are conducted experiments with various scales of the model.
Critical weaknesses

(1) The proposed method use the binary verifier to select correct samples, but have not considered that the quality of the Chain-of-Thought process, as well as the trajectory quality. What if the reasoning process is wrong but the final answer is right, how model benefit from training with this kind of data?

(2) The training process seems cannot be robustness: as for each term the rollout varies, the selected context sample varies and the generated logit varies. Please explain why under these variation how to guarantee the training process to be reproduced.

(3) The proposed method is lack of novelty, all this kind of design can be find in other current papers. Please discuss and compare with these methods both in method design also in experiments.

[1] Reinforcement Learning via Self-Distillation

[2] Multi-Rollout On-Policy Distillation via Peer Successes and Failures

[3] Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

(4) The setting in Table 1, Table 2 is quite strange, why the number of output token is set to 1024? Normally it should be 20k to aviod the output being cut. Please explain this reasons of this setting and add the experiments with longer token output.

Questions for authors

Refer to the Weakness part;

1 comment

We thank the reviewer for finding the paper well written and easy to follow, and for noting the experiments across multiple model scales. We clarify the novelty of SAPL and address the remaining concerns below.

W1: Correct answer, flawed reasoning

Spurious solutions, where the final answer is correct but the reasoning is flawed, are a known limitation of outcome-verified RL. This applies equally to every baseline we compare against, since all of them use the same binary verifier. SAPL does not target this issue: its focus is on how to obtain and learn from verified successes on hard prompts, and like any RLVR method it inherits the verifier's limitations.

W3: Novelty relative to [1], [2], [3]

We take this opportunity to clarify our novelty. The works mentioned by the reviewer are related, but they do not undermine the contribution of SAPL. The key difference is where a success is used: existing approaches use it to sharpen the training signal, while SAPL uses it to change the sampling.

In [1]–[3], a teacher is conditioned on privileged context (successful peer rollouts as feedback in [1], successful and failed peer rollouts in [2], a successful peer rollout in [3]) and provides token-level supervision on the student's existing on-policy rollouts. These methods differ in how existing samples are scored.

SAPL instead uses the model's own verified success to improve sampling. Conditioned on it, the model generates new responses, which are verified independently and used to train the prompt-only policy. On hard prompts, where successes are rare, this increases the number of verified successes available for learning, rather than reweighting credit on existing ones.

We cite [1] and [2] in the introduction and discuss them in Appendix A.1.

W4: Why 1,024 output tokens

The 1,024-token budget is a design choice of our environment: we study sample efficiency on hard prompts under a fixed, compute-limited response budget. We use the same budget to construct the hard training set and for all training and evaluation, identically for every method (Appendix A.3; Table 1 and Table 2 captions). Truncation affects all methods equally and only makes the problem harder, which is exactly the setting SAPL targets. Experiments with longer output budgets are important future work.

Reviewer_kbBROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Success-aware policy learning (SAPL) uses rare successes during RLVR as conditioning to produce more successes for a given prompt, then trains on these successes without the conditioning (using a loss that downweights highly unlikely tokens to facilitate stable training). SAPL is evaluated on several math datasets at three model sizes, up to 14B, and consistently provides more performance per rollout and better final performance than baselines.

Critical strengths

The paper is well written and motivated – the in-context-learning analogy is helpful, and the data acquisition value of rollouts is an insightful perspective.

Showing that conditioning on higher likelihood responses leads to a higher fraction of successful rollouts is helpful, and this is a key part of the methodological novelty.

Critical weaknesses

My main concern is the experiment setup/evaluation soundness. Qwen3 4B and 8B (non thinking) obtain 84.8% and 87.4%, respectively, according to the Qwen3 technical report (2025). With thinking turned on, they score 97%. The pass@8 in the submission for these base models is only 71.2% and 81%, respectively. The baseline methods barely improve on these performances, while the proposed approach SAPL fails to exceed the base numbers reported in the technical report.

  • I'm wondering if 1K tokens is insufficient for these experiments, and whether it's inducing a high response truncation rate.

Relatedly, it appears that the baselines are evaluated after many fewer optimizer steps. I understand the paper is emphasizing rollout efficiency, but that isn't the only axis of interest.

Questions for authors

Can you please try to figure out why your baselines are below the technical report numbers? They should match. I think your sequence length of 1024 may be too short and is one possible source of issues.

The above question is by far the most critical. If it can be addressed successfully, please consider the following questions.

  • Why is thinking disabled? This is fairly critical for reasoning LLM performance.
  • What happens as training continues, e.g. to 150,000 rollouts instead of just 15,000? Do the baselines converge to higher peak performance than SAPL?
  • How does plain SFT on the successful trajectories perform?
2 comments

We thank the reviewer for finding the paper well written and motivated, for finding the in-context-learning analogy helpful, and for recognizing that conditioning on higher-likelihood successes yields a higher fraction of successful rollouts. We address the concerns below.

W1 / Q1: Base numbers and the 1,024-token budget

The 1,024-token completion budget is a deliberate design choice of our environment. We use it to construct the hard training set and for all training and evaluation, and we keep it identical for every method (Appendix A.3; Table 1 and Table 2 captions). The technical report evaluates with much longer generations, so its numbers are not directly comparable to ours.

We agree that the baselines can suffer from truncation under this budget. Truncation, however, affects all methods equally and only makes the problem harder. This is precisely the setting SAPL is designed for, and it achieves its gains under the same constraint.

Q2: Why thinking is disabled

Thinking traces on these benchmarks often run to tens of thousands of tokens, which is outside the compute regime we study. Our question is how efficiently a method turns a limited number of sampled responses into learning signal on hard prompts. RLVR in thinking mode is a natural and important extension, and we list it as future work.

Q3: Longer training (150k rollouts)

Our main focus is rollout efficiency, which is the x-axis in all our training figures. SAPL addresses the slow learning of RLVR on hard tasks, so we stopped training once our method converged. We did not train at the 150k-rollout scale and therefore cannot say whether the baselines eventually catch up.

Q4: Plain SFT on successful trajectories

Table 1 already contains this baseline. With a binary reward and no baseline, the REINFORCE gradient is

g^K(x)=1K∑i∈SK∇θlog⁡πθ(yi∣x)\hat{g}_K(\mathbf{x}) = \frac{1}{K}\sum_{i\in\mathcal{S}_K}\nabla_\theta\log\pi_\theta(\mathbf{y}_i\mid\mathbf{x})

Failed responses contribute nothing, so each update is a cross-entropy step on the model's own verified successes (Section 2). REINFORCE stays close to the base model, while SAPL improves substantially (average pass@8):

ModelBaseREINFORCE (SFT on successes)SAPL
4B16.516.925.7
8B21.423.028.2
14B21.521.829.0

Thank you for the response, but I will maintain my rating because my main concern is not adequately addressed. I believe it's a problem that the submission's evaluation setup prevents attainment of the performance of the base model from the Qwen technical report, and I would go further than this submission's authors, who "agree that the baselines can suffer from truncation under this budget."

In particular, I believe the following explanation of why the above truncation is okay is flawed.

Truncation, however, affects all methods equally

My understanding is that SAPL is exactly the kind of method that would be least hurt by truncation. For example, say 95% of successful rollouts are >1024 tokens, the point at which truncation happens. Baselines like GRPO might find <=1024 token successes never or very rarely. However, SAPL is designed to mine many <=1024 token successes, training on a rich set of successful trajectories because it can condition on the very rare <=1024 token successes to generate more short successes.

In other words, I believe the experiments in this paper are basically showing that SAPL can teach Qwen to output successes with fewer tokens, at the cost of reducing performance relative to the base model (evaluated at a >1024 but still reasonable token budget).

If the authors can present evidence supporting their assertion that truncation hurts all methods equally, I'll happily reconsider my position. For example, run experiments at a larger token budget to provide such evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.