acceptodds
Under review as a conference paper at ICLR 2027

Prior Locking in RLVR: Prior-Corrected Optimization for Broader Reasoning Coverage

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving reasoning in large language models (LLMs). However, increasing aggregate success probability does not necessarily imply broad coverage over distinct correct reasoning modes. We show that, under the common binary-reward setting, even idealized KL-regularized policy improvement can exhibit prior locking, whereby the allocation among correct reasoning modes remains tied to initialization and limits coverage. To overcome this limitation, we propose Prior-Corrected RLVR (PC-RLVR), which dynamically rebalances inherited probability allocation; its oracle dynamics converge to the coverage-optimal uniform allocation over correct modes. To bridge the gap between the mode-level oracle and practical LLM training, we develop a historical-reference-based response-level prior correction with bounded weights, yielding a more stable practical realization under limited rollout budgets. We further derive a finite-time stationarity bound separating structural approximation residuals from controllable optimization and historical-reference estimation errors, with the controllable terms decaying as . Experiments on mathematical reasoning benchmarks show improved Avg@64 and Pass@64, together with stronger log-likelihood gains for initially lower-prior correct responses, providing complementary evidence consistent with broader coverage of successful responses.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, large language models, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BgY9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an often overlooked issue in RLVR: although training methods such as GRPO can improve overall success rates, these improvements do not necessarily translate into comparable gains under larger sampling budgets, such as obtaining at least one correct answer from 64 samples. The authors examine this issue through the allocation of probability across correct reasoning modes, noting that higher overall success rates do not necessarily imply better coverage of correct solutions with low initial probabilities. Under an idealized policy update framework, they show that the relative probabilities of correct modes remain fixed at their initial ratios, a phenomenon they call prior locking. To address this, they propose adjusting update weights based on mode probabilities and prove that, under appropriate conditions, the resulting idealized dynamics converge to a uniform distribution over correct modes. In practice, since latent reasoning modes and their probabilities cannot be directly observed, PC-RLVR constructs bounded weights from response generation probabilities and historical statistics to reweight GRPO advantages. The paper further analyzes the errors introduced by this approximation and the estimation of historical statistics, and evaluates the method on Qwen3-1.7B and Qwen3-4B across five mathematical reasoning benchmarks. The results show higher mean Avg@64 and Pass@64, with more pronounced likelihood gains over GRPO for correct responses with lower initial probabilities.

Critical strengths

The paper offers a valuable analytical perspective by clearly distinguishing improvements in overall success rates from the allocation of probability among correct reasoning modes. Starting from idealized policy updates, Section 3 establishes the prior-locking phenomenon. Specifically, Theorem 3.1 shows that, under standard KL regularization, the relative probabilities of correct reasoning modes remain fixed at their values under the initial base policy. The paper further analyzes how entropy regularization and prior correction reshape this allocation, with detailed mathematical proofs provided in Appendix A. Section 4.2 examines the errors introduced when moving from mode-level correction to a response-level approximation, distinguishing structural approximation residuals from errors that can be controlled through optimization and historical-statistics estimation. The derivations are detailed, and the assumptions and conclusions are clearly stated, providing a systematic theoretical framework for understanding the relationship between success rates and reasoning coverage in RLVR.

The experimental results support the practical value of the method. At both Qwen3 model scales, PC-RLVR achieves higher mean Avg@64 and Pass@64 across the five mathematical benchmarks than the baselines evaluated. The ablation studies in Section 5.4 and Appendix D.5 are also convincing: the authors retain the same asymmetric clipping settings and remove only the prior correction. In the controlled 4B setting, removing the correction reduces mean Avg@64 from 50.9% to 46.5% and mean Pass@64 from 81.4% to 80.1%, supporting the contribution of the correction weights themselves. The authors also compare the bounded mapping with a linear alternative and examine the effects of the main hyperparameters, providing empirical support for the algorithm’s design choices.

Critical weaknesses

It remains unclear whether PC-RLVR improves coverage across different correct reasoning strategies in practice. The theoretical results in Section 3 rely on mode probabilities, whereas the practical method uses the generation probabilities of individual responses. A response may have low probability simply because its wording is uncommon, even when the model frequently uses the underlying solution strategy. Assigning higher weights to low-probability responses therefore does not necessarily benefit underrepresented correct strategies. Section 4.2 accounts for this discrepancy through a structural residual but does not establish its magnitude during actual training. Moreover, the practical correction depends on GRPO’s group-relative advantages. When all responses in a group are correct, that group provides no further signal for rebalancing, even if the mode allocation remains highly uneven. The authors acknowledge these limitations in Appendix E, but their practical impact remains unclear, and the current experiments do not adequately address this issue.

Appendix C.2 states that checkpoints are selected based on the highest Pass@1 on AIME24/25, yet final results are also reported on these benchmarks. Since these benchmarks are used for model selection, the resulting scores may be optimistic. The main performance results do not report variability across training seeds, making it difficult to assess whether the gains are stable, particularly where the reported improvements are small. The bootstrap intervals provided for the likelihood diagnostics do not substitute for an uncertainty analysis of the main performance results. I recommend using an independent validation set for checkpoint selection and conducting repeated training and evaluation of PC-RLVR and the matched control without prior correction to establish whether the gains are reproducible.

Questions for authors
  1. Can response probabilities reliably reflect reasoning-mode probabilities? How can we distinguish between a rarely used correct solution strategy and a low-probability expression of a common strategy? Could the authors run the practical algorithm on tasks with clearly distinguishable solution strategies and directly examine changes in the probabilities of different correct modes?

  2. Do the gains under larger sampling budgets persist when checkpoints are selected using an independent validation set? Could the authors select checkpoints using data that do not overlap with the final test sets, then compare PC-RLVR against the control without prior correction but with matching clipping settings to determine whether the Pass@64 gains persist?

  3. Are the main performance gains reproducible across repeated training runs? Please report the means and variability of PC-RLVR and the matched control across different training seeds, along with the uncertainty introduced by evaluation sampling. The goal is to assess the stability of the final Avg@64 and Pass@64 results; the bootstrap intervals reported for the likelihood diagnostics do not address this question.

Reviewer_SYGxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies "prior locking" in binary-reward RLVR, where KL-regularized policy updates increase total probability on correct reasoning modes but preserve their initial relative probabilities, limiting coverage. It proposes PC-RLVR, which reweights GRPO advantages using bounded correction weights derived from response surprisal and exponentially smoothed historical statistics, and proves that its oracle dynamics converge to a uniform allocation over correct modes while controllable errors decay at (O(T^{-2/5})). Experiments with Qwen3 models on several math benchmarks show improved avg@64 and pass@64, with larger likelihood gains for initially low-probability correct responses.

Critical strengths
  • The paper provides a clear theoretical characterization of prior locking, showing that standard KL-regularized policy improvement can increase the total probability of correct solutions while leaving their relative allocation unchanged. This distinction between overall correctness and coverage of different correct modes is useful.
  • The proposed method is simple to implement on top of GRPO. It only reweights the advantage using response-level statistics, does not require explicit reasoning-mode labels, and does not increase the rollout budget.
  • The experiments cover two model sizes and several reasoning benchmarks. The consistent improvements in pass@64, together with the larger likelihood gains for initially low-probability correct responses, provide evidence that the method may improve the coverage of successful responses.
Critical weaknesses
  • Why not use quantitative metrics to measure diversity among correct solutions, such as n-gram diversity or cosine similarity between solutions? Since the paper’s main goal is to encouraging a more uniform distribution across correct solutions, reporting only avg@k and pass@k is insufficient.
  • The paper does not provide an intuitive case study showing that the proposed method actually produces a broader range of correct solutions. The authors could follow the case study used in [1].
  • Reweighting GRPO using rollout trajectory probability or perplexity (PPL) to more strongly reinforce low-probability correct samples has already been explored in several works, including STARE [2], and GRPO-likelihood [3] besides considered ProGRPO. The paper needs more discussion or experiments to demonstrate clear advantages over these closely related methods.
  • The necessity of the historical reference is not validated. The paper claims that the historical EMA is more stable than statistics estimated from the current rollouts, but does not compare it against using current-batch statistics directly without historical smoothing.

[1] He, Haoran, et al. "Random policy valuation is enough for llm reasoning with verifiable rewards." International Conference on Learning Representations. Vol. 2026. 2026. [2] Luo, Haipeng, et al. "STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability." arXiv preprint arXiv:2606.19236 (2026). [3] He, Andre Wang, Daniel Fried, and Sean Welleck. "Rewarding the unlikely: Lifting grpo beyond distribution sharpening." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Questions for authors

No.

Reviewer_zujaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper shows prior locking in RLVR, where KL-regularized policy improvement can preserve the initial relative probability allocation among correct reasoning modes. Thus, the paper proposes PC-RLVR to encourage broader coverage of correct reasoning modes. The paper provides theoretical analysis and a practical historical-reference-based correction, with results on mathematical benchmarks.

Critical strengths
  1. Interesting topics about improving RLVR performance from correct reasoning modes.

  2. It describes an important distinction between improving overall correctness and maintaining coverage across correct reasoning modes and introduces the concept of prior locking.

Critical weaknesses
  1. All correct modes receive the same binary reward, so Pass@k depends on the total probability mass assigned to the correct set, rather than how this mass is distributed among correct modes. Therefore, uniform allocation does not directly improve Pass@k under this setting. Please clarify under what conditions uniform allocation can improve Pass@k or generalization, and explain why the Pass@64 improvements should be attributed to reallocation among correct modes rather than an increase in the total probability mass of the correct set?

  2. Different reasoning modes may have heterogeneous rewards. It is unclear whether the coverage-optimal uniform allocation over correct modes is valid at the level of thinking and reasoning.

  3. Limited base model tested.

  4. No model-generated output examples for comparison and review.

Questions for authors
  1. Last year's famous NeurIPS paper, "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" shows that RL actually does not help much, especially in the pass@k setting when k increases. Here in the authors' manuscript, it is set to an unusually high value (64), which is quite different from the common k=8 setting.

  2. The title is PC-RLVR, but actually the main work is only the binary rewards (so not even consider a process reward) and only compare with the GPRO (and its variants). It is definitely overclaimed.

  3. I remember last year's COLM paper Dr. GRPO which made some progress in the KL divergence implementation and reward clipping, which also helps in terms of math reasoning ability. However, Dr. GRPO is not compared or discussed here.

  4. There are lots of concerns with the Qwen family of models when people use them as the base benchmark for math reasoning tests. See this arXiv paper https://arxiv.org/abs/2506.10947); it is suggested to use other hands as well as baselines for comparison and discussion。

  5. Please provide model output generation examples at lesast for AIME 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.