- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Success-Aware Policy Learning (SAPL) to improve the sample efficiency of reinforcement learning with verifiable rewards (RLVR) on challenging reasoning tasks. SAPL follows a discover–amplify–assimilate framework: it first identifies successful on-policy trajectories, selects the highest-likelihood success to guide additional sampling, and then assimilates the verified guided responses into the original policy using surprisal-gated cross-entropy loss. The method requires neither external teacher models nor ground-truth solutions during training. Experiments on Qwen3-4B, 8B, and 14B across eleven mathematical reasoning benchmarks demonstrate consistent improvements over existing RLVR and answer-conditioned baselines.
Critical strengths
-
SAPL provides a practical approach to improving sample efficiency in sparse-reward RLVR by using successful trajectories not only for policy updates but also to guide subsequent data generation. This directly addresses a fundamental bottleneck in training on difficult reasoning tasks.
-
SAPL consistently outperforms multiple RLVR baselines across three Qwen3 model sizes and eleven mathematical reasoning benchmarks. The improvements in sample efficiency and gradient alignment provide encouraging evidence of the proposed framework's effectiveness.
Critical weaknesses
-
SAPL selects the highest-likelihood successful trajectory to guide subsequent sampling, which may bias exploration toward reasoning patterns already favored by the current policy. Although the authors provide a qualitative example demonstrating different solution strategies, a systematic analysis of exploration diversity is missing. This raises concerns about whether SAPL could reinforce existing reasoning modes and hinder exploration on tasks requiring novel solutions.
-
SAPL relies on the base model's ability to discover initial successful trajectories and generate useful alternatives conditioned on them. Although the experiments cover three Qwen3 model sizes, the training prompts are selected to have similarly low initial success rates. It remains unclear whether the method generalizes to substantially weaker models that struggle to discover initial successes or stronger models that may benefit less from success-conditioned sampling. This limits the evidence for the method's applicability across different model capabilities.
Questions for authors
- How does success-conditioned sampling affect the diversity of generated reasoning trajectories compared with standard on-policy sampling? Have the authors considered whether selecting the highest-likelihood success could restrict exploration or lead to mode collapse? A comparison with random or diversity-aware success selection would help clarify this concern.
- How does SAPL perform across different initial pass@1 regimes or on models with substantially different reasoning capabilities? In particular, does the method remain effective when the base model rarely discovers successful trajectories, or when the base model is already strong enough that success-conditioned sampling provides limited additional benefit?
- How much of SAPL's performance improvement comes from success-conditioned sampling, highest-likelihood success selection, and surprisal-gated assimilation, respectively? A component-wise ablation, particularly against random success selection and standard cross-entropy assimilation, would help establish which components are essential to the reported gains.
We thank the reviewer for recognizing that SAPL directly addresses a fundamental bottleneck in sparse-reward RLVR by using successful trajectories to guide data generation, and for noting the consistent improvements across three model sizes and eleven benchmarks, together with the evidence on sample efficiency and gradient alignment. We address the questions below.
W1 / Q1: Diversity and highest-likelihood selection
We thank you for the suggestion, we will consider it in the revised version.
W2 / Q2: Different pass@1 regimes
The low pass@1 regime, where the base model rarely solves the task, is exactly the setting SAPL targets. For each model, we select training prompts that it rarely solves, with an average pass@1 of 3% (Appendix A.2), and the gains hold across all three model sizes (Table 1). At the other end of the spectrum, where successes are frequent, standard RLVR already receives sufficient learning signal, which is not the setting we consider.
Q3: Component ablation
We thank you for the constructive feedback, we will consider it in the revised version of our work.