- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the measurement reliability of the small gains reported by RL post-training (especially RLVR) on reasoning tasks. It proposes no new RL algorithm, but examines how evaluation protocols affect the measured RL-Base differences. Main findings: the same checkpoints yield stable GSM8K gains of about +2.2–2.7 points across three seeds under zero-shot greedy evaluation (CIs excluding zero), but these gains nearly vanish under 4-shot evaluation; prompt wording changes alone produce accuracy shifts of 6.4–7.3 points, exceeding the RL gains themselves; under stochastic decoding, RL-Base differences shrink to +0.2–0.5 points with all paired bootstrap CIs including zero; and RL changes output-length distributions without monotonically corresponding to accuracy gains. The authors introduce the Detectability Ratio as a descriptive diagnostic and provide cross-model validation on Qwen2.5-7B-Instruct.
Critical strengths
-
The paper directly demonstrates the separation between training reproducibility and evaluation robustness using the same set of checkpoints. This is the most solid part of the paper. Figure 1 shows that all three seeds yield positive gains with CIs excluding zero under zero-shot evaluation, while the same checkpoints yield gains indistinguishable from zero under 4-shot evaluation. This comparison is clean and reproducible, and it is a compelling counterexample to the common practice of treating "multiple seeds + small gain + CI excluding zero" as evidence of robustness.
-
The restrained interpretation of DR and the explicit boundary statements. The authors repeatedly emphasize that DR is not a hypothesis test, that is not a p-value, that is not an intrinsic property of a benchmark, and that generation length is not a proxy for reasoning quality. This caution is valuable given the current tendency in RLVR papers to over-interpret small gains, and it prevents the paper itself from falling into the same trap.
Critical weaknesses
-
The paper stops at "the problem exists" without giving an actionable criterion. It shows small RL gains are sensitive to evaluation protocol, but does not answer when a small gain can be trusted. DR is proposed as a diagnostic yet has no threshold or empirical correspondence, making it unusable for judgment. The evaluation-size scaling shows uncertainty decreases with but gives no guidance on how large must be. The Discussion 4.3 checklist is principled but lacks executable standards.
-
Limited novelty. Prompt sensitivity, decoding effects, the importance of bootstrap CIs, and small-subset instability are all well established in the NLP evaluation literature. The paper applies these tools to RLVR but proposes no new method, theory, or generalizable conclusion. Cross-model validation covers only Qwen2.5-7B, and its pattern is not fully consistent with Llama.
-
The presentation is poor. The layout is cluttered, the figures are not visually appealing, and the paper is hard to read.
Questions for authors
See Critical Weaknesses
We thank the reviewer for recognizing the importance of separating evaluation robustness from apparent training reproducibility. We agree that the paper needs a more actionable evaluation standard and a clearer explanation of its contribution.
1. An actionable criterion for small RL gains. We propose a practical, protocol-conditional reporting standard. For each RL–Base comparison, researchers should report:
(Δ, paired 95% CI, N, prompt protocol, decoding parameters, sampling budget, scorer version, checkpoint provenance).
We distinguish two levels of evidence:
A positive result under one protocol should not be interpreted as evidence of general robustness. We will make this distinction operational in the revised Discussion and provide worked examples.
DR is a descriptive effect-to-uncertainty ratio, closely related to conventional standardized statistics. We do not present DR>1 as a novel significance test or universal decision threshold.
2. Evidence supporting the criterion. Our paired GSM8K analysis shows that Static-KL−Base is +2.199 pp (95% CI [+0.758, +3.639]) under zero-shot greedy evaluation but -0.076 pp ([-1.137, +0.986]) under four-shot evaluation. The difference between protocol-specific effects is +2.274 pp ([+0.455, +4.094]).
Thus, a statistically resolved improvement under one prompting protocol does not necessarily remain resolved under another. We emphasize that the difference may involve answer-format compliance or extraction behavior, not necessarily a change in mathematical reasoning.
On MATH500, the Static-KL effect reverses under stochastic Mean@16 evaluation: -1.2625 pp (95% CI [-2.3500, -0.1750]), contrasting with its positive zero-shot greedy result. We will move this finding into the main text.
3. Stochastic uncertainty and scoring sensitivity. We identified an important reporting distinction: the original stochastic Table 7 corresponds to first-generation Pass@1 rather than Mean@16.
Under the original string-matching scorer, GRPO/No-KL/Static-KL improvements are +0.531/+0.227/+0.303 pp for first-generation Pass@1 and +0.578/+0.602/+0.787 pp for Mean@16. The first-generation confidence intervals include zero; the exact Mean@16 intervals remain under verification.
Numerical normalization also changes first-generation effect estimates to +0.08/-0.23/-0.38 pp, respectively, with all paired 95% CIs including zero. We will distinguish estimator precision from scorer sensitivity and avoid generalizing conclusions across different estimands.
4. Novelty and positioning. We agree that prompt sensitivity, bootstrap confidence intervals, and evaluation-size effects are established topics. Our contribution is not a new RL algorithm or statistical test. Instead, we provide a focused empirical audit of small RLVR gains, examining how prompting, decoding, scoring, and uncertainty estimation affect conclusions about fixed checkpoints.
We will strengthen comparisons with prior evaluation literature and narrow our novelty claims accordingly.
5. Training reproducibility and presentation. An additional audit found that archived runs labeled 1337 and 3407 both record
seed=42anddata_seed=None. We therefore withdraw the claim that these results establish reproducibility across three independently controlled training seeds.We will revise Figure 1 and the associated discussion to describe three archived runs, improve figure readability, remove redundant DR presentation, and distinguish verified observations from unresolved training provenance.
Revised contribution. The paper's central message is not that all small RL gains are unreliable, but that claims of reliable improvement require explicit statistical evidence under specified evaluation protocols and careful examination of relevant alternatives. We will make this criterion and its limitations clearer in the revision.