acceptodds
Under review as a conference paper at ICLR 2027

When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewards

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a central paradigm for turning large language models (LLMs) into reliable problem solvers, especially in logic‑heavy domains. Despite its empirical success, it remains unclear whether RLVR elicits novel capabilities or merely sharpens the distribution over existing knowledge. We study this by formalizing over-sharpening, a phenomenon where the policy collapses onto limited modes, suppressing valid alternatives. At a high level, we discover finite-batch updates intrinsically bias learning toward sampled modes, triggering a collapse that propagates globally via semantic coupling. To mitigate this, we propose inverse-success advantage calibration to prioritize difficult queries and distribution-level calibration to diversify sampling via a memory network. Empirical evaluations validate that our strategies can effectively improve generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Reinforcement Learning with Verifiable Rewards, Over-sharpening

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CUSwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates over-sharpening in RLVR, where policy optimization suppresses valid alternative reasoning modes. The authors provide a theoretical analysis identifying two mechanisms: finite-batch sampling bias and semantic coupling, which propagates distribution collapse across related queries. Based on this analysis, they introduce Inverse-success Advantage Calibration (IAC), which reweights positive advantages according to empirical query difficulty, and Distribution-Level Calibration (DLC), which encourages exploration through an auxiliary memory network.

I find IAC simple and practically useful, with reasonably consistent empirical improvements. However, since a substantial part of the paper's contribution lies in its theoretical analysis, I have serious concerns about its validity. In particular, the finite-sample analysis relies on an empirical objective that is not an estimator of Eq. (1), and the theoretical objectives used across the paper are inconsistent. Furthermore, the empirical evidence does not sufficiently establish the proposed mechanism of preserving alternative correct reasoning modes. Overall, despite the promising empirical results, I find that the paper's central theoretical claims are not established by the presented analysis. My assessment is clear reject.

Critical strengths

The main strength of this paper is the simplicity and practicality of IAC. The method only requires rescaling positive advantages based on empirical success rates, making it straightforward to integrate into existing RLVR pipelines without additional computational overhead.

The empirical results are also reasonably consistent across three Qwen3 model sizes and six mathematical reasoning benchmarks. Additional experiments on different model families, code generation, and a larger MoE model further suggest that IAC can be a useful practical intervention, independently of the proposed theoretical explanation.

Critical weaknesses

1. The finite-sample analysis does not establish the paper's central theoretical claim.

Theorem 3.2 is presented as the finite-batch counterpart of Theorem 3.1. However, Appendix B defines the empirical objective as J^(π)=∑iπ(oi∣q)NiGA(q,oi)−βDKL(π∥πref).\hat J(\pi)=\sum_i\pi(o_i\mid q)\frac{N_i}{G}A(q,o_i)-\beta D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}}).

I do not see how this objective follows from Eq. (1). In particular, the additional factor Ni/GN_i/G does not provide a valid finite-sample approximation of the original objective. As G→∞G\to\infty, Ni/GN_i/G converges to the sampling probability πθ(oi∣q)\pi_\theta(o_i\mid q), meaning that Eq. (3) does not recover Eq. (2) even with infinitely many samples.

This is a fundamental issue because the paper attributes the difference between Theorems 3.1 and 3.2 to finite-sampling bias, whereas the discrepancy already exists in the infinite-sample limit.

Furthermore, Appendix C uses a different empirical objective involving log⁡πθ(oi)\log\pi_\theta(o_i) rather than π(oi)\pi(o_i), while reusing the batch-optimal policy derived in Theorem 3.2. The derivation of Theorem C.1 therefore appears inconsistent with Appendix B.

Since Theorem 3.3 and the motivation for IAC build directly on Theorem 3.2, I find the paper's central theoretical explanation insufficiently supported.

2. Insufficient empirical evidence that IAC preserves distinct correct reasoning strategies.

Although IAC achieves reasonably consistent improvements in AVG@8 and PASS@8, these metrics do not directly establish that the method preserves diverse, correct reasoning strategies.

The paper provides controlled experiments on distribution collapse and reports entropy evolution, but the main evaluations on realistic mathematical reasoning benchmarks do not directly measure the preservation of distinct correct reasoning modes. In particular, higher entropy or pass@k does not necessarily imply greater diversity of valid reasoning strategies.

It therefore remains unclear whether the observed improvements arise from mitigating over-sharpening as proposed, or simply from emphasizing positive trajectories on harder queries. A direct analysis of semantic strategy diversity throughout training would substantially strengthen the paper's central mechanistic claim.

Questions for authors

See weakness

Reviewer_SaNNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the causes of over-sharpening and distribution collapse in RLVR, questioning whether they are inevitable. The core approach formalizes the causes as finite-sample bias and semantic coupling, proposes IAC which re-weights positive advantages according to query difficulty to suppress domination by simple questions, and introduces DLC which uses a memory network to calibrate the sampling distribution to enhance exploration. The main contribution is revealing the mechanism of over-sharpening and providing a lightweight, plug-in calibration method. Experiments on six mathematical reasoning benchmarks and multiple Qwen3 models show improvements of approximately 2%–3% in AVG@8 and PASS@8, are compatible with DAPO, and demonstrate cross-model and cross-domain generalizability.

Critical strengths
  1. The paper refines RLVR's 'entropy collapse' into 'over-sharpening,' pointing out that finite sampling bias uniformly suppresses unobserved correct patterns, and semantic coupling propagates local collapse to related queries. The important point is that this is closer to the algorithm design level than simply 'entropy decrease' and can explain why gains are more pronounced on difficult, unsaturated benchmarks and PASS@8, while also providing a verifiable framework for subsequent improvements. Even if the derivation does not fully prove the method's optimality, this mechanistic narrative still has independent value.

  2. IAC only reweights positive advantages according to query difficulty, requiring no additional forward computations, and can suppress updates dominated by easy questions; this has been additionally validated on Qwen3-4B/8B/14B main experiments, 1.5B/8B cross-model setups, code tasks, and 104B Ling-Flash, and is compatible with DAPO, providing additive gains. The important point is that although a 2%–3% average increase is not groundbreaking, the method is low-cost and easy to integrate, and its benefits are more visible on difficult benchmarks and PASS@8. As a paper emphasizing 'mechanistic contribution and practical calibration,' this is sufficient to support acceptance, provided the theoretical inevitability is downplayed and DLC evidence is supplemented.

Critical weaknesses
  1. The theorem establishes batch-optimal attractors, which are difficult to attain via single-step updates or in practical training (i.e., reaching π∗\pi^* or π^\hat{\pi}). The optimality of IAC stems from a heuristic agent rather than a derived necessity. The crux of the matter is that the paper's core contribution lies in its "mechanism explanation →\rightarrow method design" narrative. However, given the insufficient theoretical necessity and the lack of direct empirical verification, the theoretical claims appear overstated.

  2. The average performance gain is merely 2%–3%, with marginal improvements on several benchmarks. Although DLC is heavily emphasized in the introduction and methodology, it lacks independent, comprehensive experiments, ablation studies, and cost-benefit analyses, with key results relegated to the appendix. Consequently, it is difficult to isolate the specific contribution of DLC, which undermines its generality and its positioning as the primary contribution, thereby affecting the confidence in its acceptance.

Questions for authors
  1. Noticing in the proof of Theorem 3.2, you point out that J(π)J(\pi) is πoi∗(Ni/G)∗A\pi_{o_i}*(N_i/G)*A, but the actual situation is that either Ni/GN_i/G is an estimate of pioi, and only one of the two can be chosen. This fundamentally leads to a derivation error, resulting in a significant discrepancy from the conclusion claimed by the author.

  2. in the proof of Theorem C.1, line 790, you caculation of ∂J∂wi\frac{\partial J}{\partial w_i} is wrong, missing a π(o)\pi(o), which leads to a wrong conclusion.

  3. Does the paper claim that the IAC is "necessarily derived" from the theorem or has optimality? If so, please provide the lower bound from (Z'(q)) to ((G−∣S∣)α)((G-|S|) ^\alpha) derivation or approximate guarantee; If not, please clarify that IAC is only a heuristic proxy. This directly affects the theoretical basis of the core method.

  4. It is difficult for single-step SGD to reach π∗\pi^* or π^\hat{\pi}. Please clarify whether Theorem C.1's interpolation perspective is sufficient to support the "over-sharpening mechanism," and provide direct tests: the distance between the update direction and π^\hat{\pi}, and whether sharpening disappears at different learning rates/batches. Otherwise, there is a substantial gap between theoretical explanation and real-world training.

  5. If DLC is the main method, additional independent DLC, ablation, μ sensitivity, and rollout overhead are required; If only expansion, the main text introduction should be condensed and the combined evidence clearly listed in the appendix. Currently, "heavy introduction, little independent evidence" affects contribution evaluation.

Reviewer_rub9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates over-sharpening in reinforcement learning with verifiable rewards, defined as a decrease in the probability of at least one correct output mode. It attributes this phenomenon to finite-batch sampling and its propagation across related queries through shared parameters. The paper proposes inverse-success advantage calibration (IAC), which increases the weight of positive trajectories from difficult queries, and distribution-level calibration (DLC), which uses a memory model to diversify sampling. Experiments across mathematical reasoning and additional model/task settings report improvements. IAC is practically attractive, but several inconsistencies in the theoretical derivations currently prevent the analysis from establishing the claimed mechanism.

Critical strengths
  1. The paper addresses an important distinction between accuracy and solution coverage. Improvements at small sampling budgets need not imply preservation of alternative correct strategies. Studying the distribution over correct modes is therefore a useful complement to aggregate accuracy and token entropy.

  2. IAC is simple and inexpensive. It modifies advantage weights without requiring additional policy forward passes, making it straightforward to incorporate into existing RLVR pipelines. The paper also distinguishes this low-cost component from the more expensive DLC extension.

  3. The empirical results provide evidence that the heuristic merits further investigation. Relative to the strongest listed baselines, Table 1 reports average PASS@8 gains of 1.40, 1.36, and 1.78 percentage points for the 4B, 8B, and 14B models. The DAPO combinations, code-generation experiments, and LLaMA PASS@128 results extend the evidence beyond a single backbone or small sampling budget.

Critical weaknesses
  1. The finite-batch objective is inconsistent with the population objective and the subsequent optimization argument. Appendix B defines the empirical reward term as ∑iπi(Ni/G)Ai\sum_i \pi_i (N_i/G) A_i. For samples drawn from a fixed rollout policy pp, its expectation is ∑iπipiAi\sum_i \pi_i p_i A_i, rather than the reward term in Equation (1). The additional sampling-probability factor persists as G→∞G \to \infty. Thus, the optimizer derived for this objective does not establish the claimed finite-sample relationship to Equation (1). Appendix C then changes the objective to ∑i(Ni/G)Ailog⁡πi−βDKL\sum_i (N_i/G) A_i \log \pi_i - \beta D_{\mathrm{KL}}, while retaining the optimizer from Appendix B. Its derivative with respect to log⁡πi\log \pi_i also omits the probability factor arising from the KL term. These issues undermine the presented justification of Theorem C.1 for the stated parameter-gradient update (pp. 14–15).

  2. The semantic-coupling proof uses inequalities that do not follow from the stated assumptions. Appendix G applies Loewner ordering to bound 1⊤K−1y\mathbf{1}^{\top}K^{-1}y and individual components of K−1yK^{-1}y, although matrix ordering does not generally imply these bilinear or componentwise inequalities. Subsequent termwise bounds also require sign control: y≥0y \ge 0 does not imply K−1y≥0K^{-1}y \ge 0. For example, K=[2112]K=\begin{bmatrix}2&1\\1&2\end{bmatrix} and y=(1,0.1)⊤y=(1,0.1)^\top yield a negative second component of K−1yK^{-1}y. Moreover, a larger logit increase for a seen mode establishes a relative-odds change, not necessarily an absolute probability decrease for an unseen mode. Additional conditions or a revised argument are needed to establish the claimed propagation result (pp. 6, 18–19).

  3. IAC does not reduce the proposed partition function at a fixed batch. For a mixed-success batch and α≥0\alpha \ge 0, the multiplier (G−∣S+∣)α(G-|S^+|)^\alpha is at least one. IAC therefore increases or preserves positive advantages while leaving negative advantages unchanged. Holding the batch, reference policy, and regularization fixed, substitution into Equation (3) makes Z′Z' nondecreasing. This is difficult to reconcile with the proposed explanation that IAC mitigates suppression by limiting partition-function growth. Relative reweighting across queries or longer-term training dynamics could still explain the empirical gains, but these would require a different justification.

  4. The empirical attribution and uncertainty require clarification. IAC changes difficulty weighting, the positive-to-negative update balance, and the overall update scale simultaneously. The experiments do not isolate these effects from simple positive-advantage scaling or matched-gradient-norm controls. In addition, Section 5.1 reports selecting the best performance over checkpoints evaluated every 50 steps, without sufficiently specifying independent validation-based selection or across-seed uncertainty. This matters particularly for small differences, such as Qwen3-8B AVG@8=65.63\mathrm{AVG@8}=65.63 versus 65.6065.60 for the strongest listed baseline.

Questions for authors
  1. Which objective corresponds to the actual implementation? Can the authors provide a consistent derivation linking Equation (1), the finite-batch objective, and the parameter update, including its G→∞G \to \infty limit and the corrections needed in Appendix C?

  2. What additional assumptions justify the bilinear/componentwise bounds and sign conditions used in Appendix G? How does the argument establish an unseen mode’s absolute probability decrease, rather than only a change in relative odds?

  3. How should the fixed-batch nondecrease of Z′Z' under IAC be reconciled with the stated explanation? Does the implementation contain an additional normalization, or is the intended mechanism instead cross-query reweighting and longer-term dynamics?

  4. Please clarify checkpoint selection, training seeds, and uncertainty for the main comparisons. In one existing training configuration, does IAC retain its advantage over constant positive-advantage scaling when the overall update scale is matched?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.