- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies an often overlooked issue in RLVR: although training methods such as GRPO can improve overall success rates, these improvements do not necessarily translate into comparable gains under larger sampling budgets, such as obtaining at least one correct answer from 64 samples. The authors examine this issue through the allocation of probability across correct reasoning modes, noting that higher overall success rates do not necessarily imply better coverage of correct solutions with low initial probabilities. Under an idealized policy update framework, they show that the relative probabilities of correct modes remain fixed at their initial ratios, a phenomenon they call prior locking. To address this, they propose adjusting update weights based on mode probabilities and prove that, under appropriate conditions, the resulting idealized dynamics converge to a uniform distribution over correct modes. In practice, since latent reasoning modes and their probabilities cannot be directly observed, PC-RLVR constructs bounded weights from response generation probabilities and historical statistics to reweight GRPO advantages. The paper further analyzes the errors introduced by this approximation and the estimation of historical statistics, and evaluates the method on Qwen3-1.7B and Qwen3-4B across five mathematical reasoning benchmarks. The results show higher mean Avg@64 and Pass@64, with more pronounced likelihood gains over GRPO for correct responses with lower initial probabilities.
Critical strengths
The paper offers a valuable analytical perspective by clearly distinguishing improvements in overall success rates from the allocation of probability among correct reasoning modes. Starting from idealized policy updates, Section 3 establishes the prior-locking phenomenon. Specifically, Theorem 3.1 shows that, under standard KL regularization, the relative probabilities of correct reasoning modes remain fixed at their values under the initial base policy. The paper further analyzes how entropy regularization and prior correction reshape this allocation, with detailed mathematical proofs provided in Appendix A. Section 4.2 examines the errors introduced when moving from mode-level correction to a response-level approximation, distinguishing structural approximation residuals from errors that can be controlled through optimization and historical-statistics estimation. The derivations are detailed, and the assumptions and conclusions are clearly stated, providing a systematic theoretical framework for understanding the relationship between success rates and reasoning coverage in RLVR.
The experimental results support the practical value of the method. At both Qwen3 model scales, PC-RLVR achieves higher mean Avg@64 and Pass@64 across the five mathematical benchmarks than the baselines evaluated. The ablation studies in Section 5.4 and Appendix D.5 are also convincing: the authors retain the same asymmetric clipping settings and remove only the prior correction. In the controlled 4B setting, removing the correction reduces mean Avg@64 from 50.9% to 46.5% and mean Pass@64 from 81.4% to 80.1%, supporting the contribution of the correction weights themselves. The authors also compare the bounded mapping with a linear alternative and examine the effects of the main hyperparameters, providing empirical support for the algorithm’s design choices.
Critical weaknesses
It remains unclear whether PC-RLVR improves coverage across different correct reasoning strategies in practice. The theoretical results in Section 3 rely on mode probabilities, whereas the practical method uses the generation probabilities of individual responses. A response may have low probability simply because its wording is uncommon, even when the model frequently uses the underlying solution strategy. Assigning higher weights to low-probability responses therefore does not necessarily benefit underrepresented correct strategies. Section 4.2 accounts for this discrepancy through a structural residual but does not establish its magnitude during actual training. Moreover, the practical correction depends on GRPO’s group-relative advantages. When all responses in a group are correct, that group provides no further signal for rebalancing, even if the mode allocation remains highly uneven. The authors acknowledge these limitations in Appendix E, but their practical impact remains unclear, and the current experiments do not adequately address this issue.
Appendix C.2 states that checkpoints are selected based on the highest Pass@1 on AIME24/25, yet final results are also reported on these benchmarks. Since these benchmarks are used for model selection, the resulting scores may be optimistic. The main performance results do not report variability across training seeds, making it difficult to assess whether the gains are stable, particularly where the reported improvements are small. The bootstrap intervals provided for the likelihood diagnostics do not substitute for an uncertainty analysis of the main performance results. I recommend using an independent validation set for checkpoint selection and conducting repeated training and evaluation of PC-RLVR and the matched control without prior correction to establish whether the gains are reproducible.
Questions for authors
-
Can response probabilities reliably reflect reasoning-mode probabilities? How can we distinguish between a rarely used correct solution strategy and a low-probability expression of a common strategy? Could the authors run the practical algorithm on tasks with clearly distinguishable solution strategies and directly examine changes in the probabilities of different correct modes?
-
Do the gains under larger sampling budgets persist when checkpoints are selected using an independent validation set? Could the authors select checkpoints using data that do not overlap with the final test sets, then compare PC-RLVR against the control without prior correction but with matching clipping settings to determine whether the Pass@64 gains persist?
-
Are the main performance gains reproducible across repeated training runs? Please report the means and variability of PC-RLVR and the matched control across different training seeds, along with the uncertainty introduced by evaluation sampling. The goal is to assess the stability of the final Avg@64 and Pass@64 results; the bootstrap intervals reported for the likelihood diagnostics do not address this question.