- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates over-sharpening in RLVR, where policy optimization suppresses valid alternative reasoning modes. The authors provide a theoretical analysis identifying two mechanisms: finite-batch sampling bias and semantic coupling, which propagates distribution collapse across related queries. Based on this analysis, they introduce Inverse-success Advantage Calibration (IAC), which reweights positive advantages according to empirical query difficulty, and Distribution-Level Calibration (DLC), which encourages exploration through an auxiliary memory network.
I find IAC simple and practically useful, with reasonably consistent empirical improvements. However, since a substantial part of the paper's contribution lies in its theoretical analysis, I have serious concerns about its validity. In particular, the finite-sample analysis relies on an empirical objective that is not an estimator of Eq. (1), and the theoretical objectives used across the paper are inconsistent. Furthermore, the empirical evidence does not sufficiently establish the proposed mechanism of preserving alternative correct reasoning modes. Overall, despite the promising empirical results, I find that the paper's central theoretical claims are not established by the presented analysis. My assessment is clear reject.
Critical strengths
The main strength of this paper is the simplicity and practicality of IAC. The method only requires rescaling positive advantages based on empirical success rates, making it straightforward to integrate into existing RLVR pipelines without additional computational overhead.
The empirical results are also reasonably consistent across three Qwen3 model sizes and six mathematical reasoning benchmarks. Additional experiments on different model families, code generation, and a larger MoE model further suggest that IAC can be a useful practical intervention, independently of the proposed theoretical explanation.
Critical weaknesses
1. The finite-sample analysis does not establish the paper's central theoretical claim.
Theorem 3.2 is presented as the finite-batch counterpart of Theorem 3.1. However, Appendix B defines the empirical objective as
I do not see how this objective follows from Eq. (1). In particular, the additional factor does not provide a valid finite-sample approximation of the original objective. As , converges to the sampling probability , meaning that Eq. (3) does not recover Eq. (2) even with infinitely many samples.
This is a fundamental issue because the paper attributes the difference between Theorems 3.1 and 3.2 to finite-sampling bias, whereas the discrepancy already exists in the infinite-sample limit.
Furthermore, Appendix C uses a different empirical objective involving rather than , while reusing the batch-optimal policy derived in Theorem 3.2. The derivation of Theorem C.1 therefore appears inconsistent with Appendix B.
Since Theorem 3.3 and the motivation for IAC build directly on Theorem 3.2, I find the paper's central theoretical explanation insufficiently supported.
2. Insufficient empirical evidence that IAC preserves distinct correct reasoning strategies.
Although IAC achieves reasonably consistent improvements in AVG@8 and PASS@8, these metrics do not directly establish that the method preserves diverse, correct reasoning strategies.
The paper provides controlled experiments on distribution collapse and reports entropy evolution, but the main evaluations on realistic mathematical reasoning benchmarks do not directly measure the preservation of distinct correct reasoning modes. In particular, higher entropy or pass@k does not necessarily imply greater diversity of valid reasoning strategies.
It therefore remains unclear whether the observed improvements arise from mitigating over-sharpening as proposed, or simply from emphasizing positive trajectories on harder queries. A direct analysis of semantic strategy diversity throughout training would substantially strengthen the paper's central mechanistic claim.
Questions for authors
See weakness