acceptodds
Under review as a conference paper at ICLR 2027

When Evaluation Changes the Conclusion: Measuring the Reliability of Small RL Gains in Reasoning LLMs

Abstract

Reinforcement learning (RL) post-training has become a widely adopted approach for improving the reasoning capabilities of large language models (LLMs). However, many reported improvements are relatively small, raising an important question: whether current evaluation protocols provide sufficient resolution to reliably measure such incremental gains. We conduct a systematic empirical study of RL gain measurement reliability under different evaluation protocols. We evaluate Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct with RL-trained variants including Static-KL GRPO, GRPO, and No-KL using greedy decoding, stochastic sampling, multi-seed evaluation, paired bootstrap confidence intervals, and evaluation-size scaling analysis. Our results show that reproducibility across training seeds does not necessarily imply robustness across evaluation protocols. Under zero-shot greedy GSM8K evaluation, Static-KL achieves consistent improvements across three training seeds, whereas the same checkpoints yield near-zero gains under 4-shot evaluation. Under stochastic sampling with 16 generations, the measured RL–Base differences are only 0.2–0.5 percentage points, with confidence intervals including zero. Evaluation-size scaling further shows that larger evaluations provide more stable estimates of small effects.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, RL Post-Training, Reasoning Models, Evaluation Methodology, Benchmark Reliability, Uncertainty Quantification, Group Relative Policy Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gTvbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the measurement reliability of the small gains reported by RL post-training (especially RLVR) on reasoning tasks. It proposes no new RL algorithm, but examines how evaluation protocols affect the measured RL-Base differences. Main findings: the same checkpoints yield stable GSM8K gains of about +2.2–2.7 points across three seeds under zero-shot greedy evaluation (CIs excluding zero), but these gains nearly vanish under 4-shot evaluation; prompt wording changes alone produce accuracy shifts of 6.4–7.3 points, exceeding the RL gains themselves; under stochastic decoding, RL-Base differences shrink to +0.2–0.5 points with all paired bootstrap CIs including zero; and RL changes output-length distributions without monotonically corresponding to accuracy gains. The authors introduce the Detectability Ratio as a descriptive diagnostic and provide cross-model validation on Qwen2.5-7B-Instruct.

Critical strengths
  1. The paper directly demonstrates the separation between training reproducibility and evaluation robustness using the same set of checkpoints. This is the most solid part of the paper. Figure 1 shows that all three seeds yield positive gains with CIs excluding zero under zero-shot evaluation, while the same checkpoints yield gains indistinguishable from zero under 4-shot evaluation. This comparison is clean and reproducible, and it is a compelling counterexample to the common practice of treating "multiple seeds + small gain + CI excluding zero" as evidence of robustness.

  2. The restrained interpretation of DR and the explicit boundary statements. The authors repeatedly emphasize that DR is not a hypothesis test, that PbootP_{\text{boot}} is not a p-value, that R95R_{95} is not an intrinsic property of a benchmark, and that generation length is not a proxy for reasoning quality. This caution is valuable given the current tendency in RLVR papers to over-interpret small gains, and it prevents the paper itself from falling into the same trap.

Critical weaknesses
  1. The paper stops at "the problem exists" without giving an actionable criterion. It shows small RL gains are sensitive to evaluation protocol, but does not answer when a small gain can be trusted. DR is proposed as a diagnostic yet has no threshold or empirical correspondence, making it unusable for judgment. The evaluation-size scaling shows uncertainty decreases with NN but gives no guidance on how large NN must be. The Discussion 4.3 checklist is principled but lacks executable standards.

  2. Limited novelty. Prompt sensitivity, decoding effects, the importance of bootstrap CIs, and small-subset instability are all well established in the NLP evaluation literature. The paper applies these tools to RLVR but proposes no new method, theory, or generalizable conclusion. Cross-model validation covers only Qwen2.5-7B, and its pattern is not fully consistent with Llama.

  3. The presentation is poor. The layout is cluttered, the figures are not visually appealing, and the paper is hard to read.

Questions for authors

See Critical Weaknesses

1 comment

We thank the reviewer for recognizing the importance of separating evaluation robustness from apparent training reproducibility. We agree that the paper needs a more actionable evaluation standard and a clearer explanation of its contribution.

1. An actionable criterion for small RL gains. We propose a practical, protocol-conditional reporting standard. For each RL–Base comparison, researchers should report:

(Δ, paired 95% CI, N, prompt protocol, decoding parameters, sampling budget, scorer version, checkpoint provenance).

We distinguish two levels of evidence:

  • Protocol-specific statistical resolution: the paired confidence interval for the RL–Base difference excludes zero under a prespecified evaluation configuration.
  • Cross-protocol robustness: the direction and uncertainty of the effect remain consistent across relevant, explicitly reported alternative evaluation configurations.

A positive result under one protocol should not be interpreted as evidence of general robustness. We will make this distinction operational in the revised Discussion and provide worked examples.

DR is a descriptive effect-to-uncertainty ratio, closely related to conventional standardized statistics. We do not present DR>1 as a novel significance test or universal decision threshold.

2. Evidence supporting the criterion. Our paired GSM8K analysis shows that Static-KL−Base is +2.199 pp (95% CI [+0.758, +3.639]) under zero-shot greedy evaluation but -0.076 pp ([-1.137, +0.986]) under four-shot evaluation. The difference between protocol-specific effects is +2.274 pp ([+0.455, +4.094]).

Thus, a statistically resolved improvement under one prompting protocol does not necessarily remain resolved under another. We emphasize that the difference may involve answer-format compliance or extraction behavior, not necessarily a change in mathematical reasoning.

On MATH500, the Static-KL effect reverses under stochastic Mean@16 evaluation: -1.2625 pp (95% CI [-2.3500, -0.1750]), contrasting with its positive zero-shot greedy result. We will move this finding into the main text.

3. Stochastic uncertainty and scoring sensitivity. We identified an important reporting distinction: the original stochastic Table 7 corresponds to first-generation Pass@1 rather than Mean@16.

Under the original string-matching scorer, GRPO/No-KL/Static-KL improvements are +0.531/+0.227/+0.303 pp for first-generation Pass@1 and +0.578/+0.602/+0.787 pp for Mean@16. The first-generation confidence intervals include zero; the exact Mean@16 intervals remain under verification.

Numerical normalization also changes first-generation effect estimates to +0.08/-0.23/-0.38 pp, respectively, with all paired 95% CIs including zero. We will distinguish estimator precision from scorer sensitivity and avoid generalizing conclusions across different estimands.

4. Novelty and positioning. We agree that prompt sensitivity, bootstrap confidence intervals, and evaluation-size effects are established topics. Our contribution is not a new RL algorithm or statistical test. Instead, we provide a focused empirical audit of small RLVR gains, examining how prompting, decoding, scoring, and uncertainty estimation affect conclusions about fixed checkpoints.

We will strengthen comparisons with prior evaluation literature and narrow our novelty claims accordingly.

5. Training reproducibility and presentation. An additional audit found that archived runs labeled 1337 and 3407 both record seed=42 and data_seed=None. We therefore withdraw the claim that these results establish reproducibility across three independently controlled training seeds.

We will revise Figure 1 and the associated discussion to describe three archived runs, improve figure readability, remove redundant DR presentation, and distinguish verified observations from unresolved training provenance.

Revised contribution. The paper's central message is not that all small RL gains are unreliable, but that claims of reliable improvement require explicit statistical evidence under specified evaluation protocols and careful examination of relevant alternatives. We will make this criterion and its limitations clearer in the revision.

Reviewer_ZY4FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studied the evaluation of small perfromance gains from RLVR post-training. It systematically examined the influence of evaluation protocols, including zero-shot versus few-shot prompting, greedy versus stochastic decoding, and evaluation-set size, for measuring RL gains. Experiments on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct provided some interesting observations and information, which can help researchers design better benchmarks and evaluation ways for measuring small gains from RLVR.

Critical strengths
  1. This paper studied a practical and common issue about measuring the gain from RLVR post-training. It's meaningful for improving evaluation ways, since the results of many new RLVR algorithms are not easy to reproduced.
  2. The experiments considered various evaluation protocols, including zero-shot versus few-shot prompting, greedy versus stochastic decoding, and evaluation-set size, for measuring RL gains.
Critical weaknesses
  1. Limited experimental scope and insufficient coverage of evaluation configurations. The experimental analysis primarily focuses on two model families with relatively similar parameter scales (7B–8B). Most evaluation analyses are concentrated on GSM8K. The stochastic evaluation only uses a temperature of 0.9 and 16 generations per question. A more comprehensive evaluation across model scales, sampling temperatures, generation budgets, and prompting configurations would substantially strengthen the empirical evidence.
  2. This paper presents some experimental phenomena, while didn't give some deeper analysis for how to design better evaluation protocols.
Questions for authors
  1. How do you set the KL coefficient for GRPO, rather than Static-KL GRPO and No-KL?
  2. How about the influence of the responce token length for evaluation?
1 comment

We thank the reviewer for recognizing the practical importance of reliable evaluation of small RLVR gains. We address the experimental scope, evaluation guidance, KL configuration, and response-length concerns below.

1. Experimental scope and limitations. We agree that our study is limited to two 7B–8B model families and does not establish universal behavior across model scales. The stochastic GSM8K evaluation uses T=0.9 and 16 generations per question. We will explicitly restrict our conclusions to the evaluated models, datasets, and decoding configurations rather than generalize to other temperatures or sampling budgets.

Our additional audit also highlights cross-benchmark sensitivity. On MATH500, Static-KL−Base changes from a positive effect under zero-shot greedy decoding to -1.2625 pp under stochastic Mean@16 evaluation (95% paired CI [-2.3500, -0.1750]). This provides evidence that evaluation-dependent conclusions are not restricted to GSM8K, although the result does not isolate decoding as the sole cause.

2. Actionable evaluation guidance. We agree that observing protocol sensitivity alone is insufficient. We will strengthen the discussion with an operational reporting procedure: (i) specify prompting, decoding, generation budget, scorer, and evaluated checkpoint; (ii) report absolute accuracies and paired effect estimates with 95% confidence intervals; (iii) repeat comparisons under at least one relevant alternative evaluation protocol; and (iv) distinguish protocol-specific statistical resolution from cross-protocol robustness.

A positive effect whose paired CI excludes zero supports a protocol-specific finding, not a universal improvement. DR will remain a descriptive effect-to-uncertainty ratio rather than a new significance test.

3. KL coefficient and training configuration. Our audited Static-KL implementation uses a fixed KL coefficient β=0.02, learning rate 2×10⁻⁶, two epochs, four generations per prompt, and a binary correctness reward based on answer extraction. No-KL uses β=0.

For the ordinary GRPO checkpoint used in stochastic evaluation, the mapping to its original training configuration remains unresolved. We therefore cannot reliably report its KL coefficient and will not infer one from the model label. We will clarify this limitation and avoid interpreting the current comparison as an isolated causal test of KL regularization.

4. Response length. We audited mean GSM8K response lengths under greedy evaluation:

ModelZero-shotFour-shot
Base176.14102.92
Static-KL (42)170.69102.93
Static-KL (1337)171.09101.08
Static-KL (3407)172.02102.57

Static-KL produces modestly shorter responses under zero-shot evaluation, while lengths are similar under four-shot evaluation. These are descriptive associations; they do not establish that response length causes accuracy changes or serves as a proxy for reasoning quality.

5. Reproducibility clarification. Our training audit found that archived runs labeled 1337 and 3407 both record seed=42 and data_seed=None. We therefore withdraw the characterization of these runs as verified independent-seed replications.

We will revise the manuscript to make these limitations explicit and emphasize reproducible, protocol-specific evaluation evidence rather than unsupported generalization.

Reviewer_LMhrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks if smaller RL posttraining gains in 7-8B reasoning models can be measured robustly. They use Llama-3.1-8B-Instruct (and one Qwen2.5-7B-Instruct variant) with LoRA based GRPO variants. They found:

  • Changing the prompt changes the accuracy by more than the RL gain
  • A zero shot gain (greedy decoding) in GSM8K of around 2-2.7 pp that replicates across three seeds vanishes under 4 shot evaluation
  • RL changes output length with any corresponding changes in accuracy
Critical strengths
  • The main question the paper is asking is important. There’s a lot of RL post training papers that report gains of a few points without any proper uncertainty estimates or good evaluation practices.
  • The distinction between reproducibility in training seed and evaluation protocol robustness is stated in a clear and useful way
  • The comparisons in the paper are paired at the question level along with the CIs being reported throughout the paper
  • Their claims are hedged carefully and the limitations are acknowledged.
Critical weaknesses
  1. The novelty is quite limited

    • The prompt format, few shot setup, decoding and sample size change measured LLM performance and reasoning progress conclusion is already well established. The paper itself cites Hochlehnert et al. (2025), Madaan et al. (2024), Miller (2024), Wu et al. (2025) and Chilaka et al. (2026) and they appear to cover very similar things.
    • Their DR metric is essentially |Δ| divided by ~1.96*SE, i.e. a rescaled z statistic. For an approximately symmetric CI, DR > 1 is equivalent to the CI excluding zero, so Table 1 basically just restates Figure 2.
  2. The main finding that zero shot gains disappear under 4 shot evaluation is not explained and could be due to a format confound.

    A natural explanation is that RL mainly taught the model the answer format or termination behaviour which is expected by the extractor, and the 4 shot examples (which end in #### N) give the base model the same benefit. The paper doesn't have any analysis to test this: i.e. no extraction failure rates, format compliance statistics, etc. The training prompt format and the reward function are also not reported. Table 6 tries to support this but is not discussed.

  3. There are several results that require verification:

    • Very wide stochastic GSM8K CIs: with 16 samples averaged per question, the paired per question difference should have lower variance than the binary greedy comparison. However, the stochastic half widths are wider than the greedy ones (~2.1 pp vs ~1.1-1.5 pp). The magnitude seems closer to what an unpaired bootstrap over question level accuracies would give. The MATH500 stochastic CIs look consistent with pairing, which makes the GSM8K numbers stand out. Please verify how pairing is preserved in this analysis, since the main stochastic claim "all CIs include zero" depends on it.
    • Small arithmetic mismatches in Table 6: the reported prompt effects don't match the accuracies in the same table. 75.21 - 68.01 = 7.20 (table text: 7.25), 6.90 vs 6.94, and 6.37 vs 6.42. Please clarify whether the effect column is a bootstrap mean.
  4. An important result is in the appendix and contradicts the main text:

    • Table 8 (stochastic MATH500) shows Static-KL significantly worse than base (-1.3 pp). Under zero shot greedy decoding, the same model is significantly better (~+3.4 pp, Fig. 2). A sign reversal with both CIs excluding zero is the most direct demonstration of the paper's title claim. However, the main text only mentions MATH500 as a case where "larger improvements may remain distinguishable", which refers to GRPO. The abstract's 0.2-0.5 pp stochastic result describes GSM8K only.
  5. Missing details:

    • The "GRPO" checkpoint used throughout the stochastic analysis has no training config.
    • Absolute greedy accuracies for base and RL under the main zero shot and 4 shot protocols are not reported.
    • Answer extraction is not described, despite the reproducibility statement saying that it is.
    • The reproducibility statement refers to a "methodology section" that does not exist.
    • Appendix C.4 mentions a bootstrap ranking analysis whose results are not present.
Questions for authors
  1. What reward function and training prompt format were used? Does the bare question zero shot setting match the training format?
  2. What fraction of Base vs. RL failures under 0 shot greedy decoding are extraction / format failures instead of wrong answers?
  3. What is the training configuration of the “GRPO” model?
2 comments

We thank the reviewer for the constructive feedback. We audited the archived predictions, scoring implementations, and training records, identifying several issues that require substantive corrections.

1. Stochastic evaluation and uncertainty. We verified 1,319 aligned GSM8K questions with 16 generations per model (84,416 generations total). Table 7 reports first-generation Pass@1, not Mean@16. Both estimate expected single-generation accuracy under the same sampling distribution, but Mean@16 uses more observations.

Under the original string-matching scorer, GRPO/No-KL/Static-KL improvements over Base are +0.531/+0.227/+0.303 pp for first-generation Pass@1, versus +0.578/+0.602/+0.787 pp for Mean@16. All first-generation paired 95% CIs include zero. An earlier Mean@16 audit suggested intervals excluding zero, but their exact endpoints remain unverified; we therefore treat that significance finding as provisional. We will correct the manuscript's overly broad stochastic uncertainty statement.

Using 10,000 question-level paired bootstrap resamples, the recomputed GRPO and No-KL first-generation CIs match Table 7. Static-KL differs: reported [-1.74, +2.50] versus recomputed [-1.90, +2.65] pp. We will reconcile this discrepancy.

2. Scoring sensitivity. Exact string matching produced 2,965 false negatives across 84,416 generations relative to numerical normalization. Normalized first-generation accuracies are 74.07% (Base), 74.15% (GRPO), 73.84% (No-KL), and 73.69% (Static-KL).

The normalized paired differences and 95% CIs are GRPO +0.08 [-1.97, +2.20], No-KL -0.23 [-2.50, +1.97], and Static-KL -0.38 [-2.58, +1.90] pp. All include zero. Normalized Maj@16 comparisons likewise have CIs including zero. These estimators address different quantities; normalized Mean@16 remains to be verified before making scorer-robustness claims. We also identified 312 nonstandard extracted predictions requiring separate adjudication.

3. Prompt-protocol sensitivity. Under greedy GSM8K evaluation, Static-KL−Base is +2.199 pp (95% CI [+0.758, +3.639]) in zero-shot and -0.076 pp ([-1.137, +0.986]) in four-shot. The difference between protocol-specific effects is +2.274 pp ([+0.455, +4.094]). This supports protocol sensitivity, but does not establish whether answer formatting, extraction, or reasoning causes it. We will document the exact training prompt and its relationship to bare-question zero-shot evaluation.

4. MATH500 reversal. Under stochastic Mean@16 evaluation with the audited v5 scorer, Static-KL−Base is -1.2625 pp (95% CI [-2.3500, -0.1750]), contrasting with its positive zero-shot greedy effect. We will emphasize this reversal in the main text while avoiding a single-factor causal interpretation.

5. Training reproducibility. The audited Static-KL configuration uses fixed KL β=0.02, learning rate 2×10⁻⁶, two epochs, four generations per prompt, and a binary correctness reward dependent on answer extraction. No-KL uses β=0.

Crucially, archived runs labeled 1337 and 3407 both record seed=42 and data_seed=None. We therefore withdraw the claim of three verified independent-seed replications and will revise Figure 1 and the manuscript. Ordinary GRPO checkpoint provenance remains unresolved; we do not claim a verified causal KL comparison.

6. Reporting corrections. Table 6's 75.21%−68.01% discrepancy (7.20 versus reported 7.25 pp) cannot be explained by pairing alone and requires correction. DR is a descriptive effect-to-uncertainty ratio, not a novel significance test; we will clarify this relationship and remove redundant presentation. We will also document extraction rules, reward and prompt templates, response-length statistics, checkpoint provenance, and correct unsupported appendix references.

7. Novelty. We agree that prompt sensitivity and uncertainty estimation are established topics. Our contribution is not a new statistical test, but an empirical audit of small RL post-training gains across prompting, decoding, scoring, and uncertainty estimation. Unlike analyses that examine these factors individually, we assess how they jointly affect conclusions about fixed RL checkpoints. We will sharpen the comparison with prior work and avoid overstating methodological novelty.

Revised conclusion. Our contribution is not that all small RL gains are insignificant or that improvements reproduce across independent training seeds. Rather, the verified evidence demonstrates that measured RL effects and their uncertainty depend on specified evaluation conditions. We will narrow our claims accordingly and clearly separate verified results from unresolved limitations.

I thank the authors for being thorough and honest with their rebuttal and audit, and adding the comparison of protocol specific effects. The corrections however affect the paper's main claims rather than just the presentation.

Specifically, I believe there are issues that go beyond just the points I flagged in my review. Many of the main supporting points of the paper's claim (especially the three seed replication, stochastic results with single generation) remain unverified. It is also not clear from the rebuttal if the greedy results were scored again with the normalized scorer.

The newer but narrower conclusion that RL 'improvements' depend on evaluation conditions, is already very close to has been shown in prior works. I maintain my score, I believe the paper needs another review cycle.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional Statistical and Reproducibility Audits (E1/E3)

We thank the reviewers for their constructive feedback. We conducted additional statistical and reproducibility audits to clarify the reliability of small RL improvements.

1. Statistical Significance: Paired Bootstrap (E1)

We reanalyzed 1,319 GSM8K questions, with 16 sampled responses per question (temperature = 0.9, top-p = 0.9), using 10,000 paired question-level bootstrap resamples. These results use the original scorer; normalized-scoring results were reported in our prior response and will be included in the revision.

All differences are relative to the Base model, in percentage points (pp). Confidence intervals are paired 95% percentile bootstrap intervals.

MetricGRPO − BaseNo-KL − BaseStatic-KL − Base
First-sample Pass@1+0.53+0.23+0.30
95% CI[−1.59, +2.65][−2.05, +2.43][−1.90, +2.65]
Mean@16+0.58+0.60+0.79
95% CI[+0.08, +1.08][+0.06, +1.14][+0.28, +1.29]
Pass@8−0.53−0.30−0.45
95% CI[−1.52, +0.38][−1.29, +0.68][−1.44, +0.53]
Pass@160.00−0.08+0.61
95% CI[−0.83, +0.83][−0.76, +0.61][−0.08, +1.29]

All first-sample Pass@1, Pass@8, and Pass@16 confidence intervals include zero, whereas all Mean@16 intervals lie above zero. These results highlight how estimator precision and metric choice affect the statistical resolution of small RL comparisons, underscoring the need for explicit uncertainty reporting and clear metric specification.

2. Inference Reproducibility (E3)

We evaluated a separately reproduced Static-KL checkpoint on 1,319 GSM8K questions using 4-shot prompting, greedy decoding, BF16, and vLLM 0.19.0.

Execution configurationRun 1Run 2Accuracy difference
Eager execution80.59%81.50%0.91 pp
Eager + synchronous80.89%80.97%0.08 pp
Full-list synchronous80.97%80.97%0.00 pp
Sequential single-prompt81.27%81.27%0.00 pp

Repeated runs were identical within each of the two tested controlled configurations, whereas full-list and sequential execution differed by 0.30 pp despite identical nominal decoding settings. Greedy decoding alone therefore does not ensure execution-level reproducibility, making execution settings an important factor when interpreting sub-percentage-point differences.

3. Training Configuration and Seed Provenance

We audited the Static-KL training scripts and historical records for seeds 42, 1337, and 3407. The audited configurations specify β = 0.02, learning rate 2 × 10⁻⁶, and two training epochs, with the three seed-specific scripts referencing seeds 42, 1337, and 3407, respectively. We also recovered a historical checkpoint dated September 3, 2026, containing 256 optimizer parameter states at step 500.

We separately organized the training configurations, checkpoint records, and evaluation results for the three historical seed-labeled runs, and distinguished them from the subsequently reproduced and independently verified seed-3407 run.

4. Clarifications and Scope

We frame the paper around how metric definitions, paired uncertainty estimates, and inference execution settings affect measured RL comparisons.

Mean@16 averages correctness across 16 generations per question, whereas empirical Pass@k measures whether at least one of the first k generations is correct. Paired bootstrap quantifies question-level uncertainty conditional on the fixed generations, with unadjusted confidence intervals, and is distinct from seed-level or execution-level variation.

In the revision, we will document these estimator definitions and execution settings explicitly and align all claims with the audited evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.