acceptodds
Under review as a conference paper at ICLR 2027

Phantom Gains: Auditing Self-Improvement Against a Measured Null

Abstract

Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank- LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of . The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain (). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess. Code and evaluation artifacts are included as supplementary material.

Primary area:
datasets and benchmarks
Keywords:
LLMs, Self-Improvement, Evaluation Auditing, Measured Null, Catastrophic Forgetting, Benchmark Design

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_nYA5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides statistical audit of transition-level analysis in LLM self-improvement methods. Standard evaluation methods of LLMs focus on mean accuracy which can't distinguish if model acquired new capability, or doing distribution sharpening, and can't expose underlying regressions and forgetting. The submission has mainly 3 contributions:

  1. Identified and reproduced the 7 measurement failure modes for transition audits.
  2. Proposed a principled threshold-free measurement framework for capability expansion
  3. Conducted rigorous audit using Qwen3-8B across MATH-500, AIME, and 1163 difficulty band of MATH problems, discovered that self-training mainly sharpens reachable problems while inducing capability corruption.
Critical strengths
  1. The paper's key strength is it decoupled of real capability gain from the measurement noise via empirical controls, it demonstrates that standard transition heuristics produce phantom learning and degradation purely from the batching non-determinism. by measuring identical pipelines against a frozen backbone and proposing an pooled exact test under FDR control, it establishes a solid methodological foundation for auditing self-improvement.

  2. I appreciate that the author carefully equalize problem streams, retained volume and grading protocals between self-training regimes and external distillation. This isolates supervision sources from confounding variables, providing solid evidence that self-training predominantly sharpens existing sampling reach while inducing measureable capability corruption.

Critical weaknesses

The empirical audit only runs 3 rounds of rank-32 LoRA with roughly 270 optimizer steps on a single Qwen3-8B backbone, the experiment isn't representative of modern post training and RL pipelines. Given the experiment coverage, it's unclear whether the lack of capability expansion is an result of self-training or just the model is under-trained.

The dataset setup seems narrow, AIME only has 10 "never reached" problems across 1408 draws, relying on such a small set of data size makes the claim underpowered. It would be great if there's more evidence on this.

Questions for authors
  1. Could you provide the specification of the table 15 model, are the problems fixed? which seeds enter? how are zero counts handled?

  2. To separate capability loss from sharpening on the band, could you compare the difficulty band transitions against a pure sharpening control method (e.g., evaluating the frozen base model at a lower temperature)? If temperature sharpening alone reproduces a large share of the 88–106 corrupted problems (lines 439–448), the "destruction" claim might need to be reframed

  3. Was Qwen3's thinking mode enabled during training and evaluation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nYA5 (Part 1)

Thank you for the thoughtful and generous review, we are sincerely appreciative of it. For three questions that each led to a new experiment or a tighter analysis.


1. A sharpening control, and the "destruction" claim (Q2)

We ran the control you proposed, with the decision rule written down before the full run (Table 5, p. 10; Appendix C.6, p. 36).

Choice of temperature. A pilot on 175 band problems (k = 32) gave accuracies of 0.531, 0.530 and 0.528 at T = 0.6, 0.4 and 0.2. These are statistically tied (the differences are 0.1–0.2 standard errors), and our rule for a tie selects the lowest temperature, T = 0.2, which is the strongest sharpening.

Result. Sampling the frozen model at T = 0.2 on all 1,163 band problems (k = 128) regresses 74–78 problems and gains 28–41, measured against each of the five T = 0.8 baselines. Lowering the temperature therefore moves many problems on its own. But they are mostly different problems: only 14% of STaR's regressed problems and 16% of majority-vote SFT's are also regressed at T = 0.2, where chance gives about 11%. Our pre-specified threshold for "not reproduced by sharpening" was 25%.

What the regressions are. Your question led us to two further measurements, and the paper's wording now follows them.

  • Reach is preserved. Of STaR's 106 regressed problems, 104 still have at least 5 correct samples in 128 at the final checkpoint (median 43.5). The paper now calls these events pass@1 regressions.
  • The training signal is not the cause. A placebo arm, fine-tuned on a random sample of the model's own output with no correctness filter, regresses 19, 24 and 18 of 175 problems per seed (STaR: 19, 17, 10; majority-vote SFT: 11, 29, 24). They are largely the same problems (odds ratio 6.2 given the baseline solve rate), and the drops are as large (median 0.29, against 0.25 and 0.32). What the correctness filter adds is net gain: +5.6 to +6.9 points for STaR, against +1.2 to +2.4 for the placebo.

So the regressions are real and are not decoding-level sharpening, but they belong to fine-tuning on self-generated text rather than to self-training's signal.


2. Specification of the Table 15 model (Q1)

As submitted. The 60 AIME problems were fixed. Only seed 0 of each arm entered. Counts were pooled within two strata (the 22 low-base problems, with at most 5 successes in 1,408 pooled base draws, and the other 38), with no per-problem effect. A problem with zero successes simply added its draws to its stratum's total.

Why we replaced it. Pooling treats the 128 draws of each problem as independent evidence across problems whose base rates differ by orders of magnitude. The fit is overdispersed by a factor of 297, so its p < 10⁻⁸ is anti-conservative.

Now (Table 21, p. 39; Appendix D.1, p. 37), with the problem as the unit and all seeds:

SpecificationResultOne-sided p
Sign test on the seeds detecting each problem, vs. majority-vote SFT12 problems to 10.0017
Same, vs. STaR (three seeds)11 to 10.0032
Fixed effect per problem, quasi-likelihood errors, vs. majority-vote SFT+1.95 log-odds (s.e. 1.12)0.041
Same, vs. STaR+1.33 (s.e. 0.71)0.030
Submitted model, cluster bootstrap over problems+1.91, 95% CI [+0.48, +4.22]0.008

In the fixed-effects model, the four problems with no success in any evaluation carry no information and drop out. The direction is the same in every specification. The strength is not, so we report all of them and note that the fixed-effects p-values would not reach 0.05 two-sided. The abstract cites the sign test. One sentence is updated to match: a uniform-lift simulation is rejected for the self-training arms too (by one or two problems, against five to eight for the teacher), so the paper does not say that a uniform lift accounts for them.

AuthorsOpenReview ↗

Response to Reviewer nYA5 (Part 2)

3. Only 10 never-reached problems (W2)

Ten problems are too few to decide this: on those 10 AIME problems, distillation's 5 detected problems against self-training's 2 is not significant (p = 0.35). MATH-Beyond supplies a larger pool: under our protocol the base model fails 128 of its problems in all 512 draws (Appendix B.2, p. 28). All nine final ladder checkpoints were evaluated on them (Table 4, p. 9; Table 31, p. 45).

ArmDetections per seedDistinct problems
Distillation11, 8, 814
Majority-vote SFT0, 1, 01
STaR0, 0, 00

That is 14 problems to 1 (sign test, one-sided p = 5×10⁻⁴). There are two caveats. The pool is defined under our protocol, not MATH-Beyond's, under which the base model reaches more of these problems. And the distilled student truncates 61–71% of its completions here; with every truncated completion counted as incorrect it still detects 7, 5 and 8 (12 problems to 1, p = 0.002).


4. Under-training, a single backbone, and thinking mode (W1, Q3)

Is the lack of expansion due to under-training? Within this regime, the evidence says no.

  • The distillation control trains under the same LoRA rank, stream and volume. After 16 optimizer steps of its first round it already detects 8 low-base AIME problems (7 with truncated completions counted as incorrect); its final counts are 8–11.
  • All 18 self-training checkpoints, across three rounds and six runs, stay at 0–2.
  • The difference remains at a matched overall gain (Table 29, p. 42).

So the optimizer budget is not what keeps self-training from reaching new problems here. We do not claim more than that. The abstract and Limitations (§6, p. 10) scope the result to three rounds of rank-32 LoRA in non-thinking mode, and say that this does not establish that self-training cannot expand.

A single backbone, and modern RL pipelines. The training arms stay on one backbone, and the paper scopes them that way: the newer trainable backbones available to us truncate 44–62% of completions under our protocol (Table 17, p. 34). The measurement results do not depend on the training regime, and we extended those. The frozen nulls replicate on a second model family (Table 18, p. 34), and the expansion null is non-zero at every budget for all 21 models in MATH-Beyond's released curves, RL-trained models included (Table 28, p. 42).

Thinking mode. It was disabled in training and evaluation alike (enable_thinking=False), with token caps of 2,048 on MATH-500 and the band, and 8,192 on AIME and MATH-Beyond. The abstract, §3 (p. 5) and Appendix A.1 (p. 22) now say so.


Thank you once again for your generous feedback. We would be glad to answer further questions.

Reviewer_D4KKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits transition-level evaluation of LLM self-improvement, i.e. tracking which individual problems a model gains or loses. The setup is three rounds of rank-32 LoRA self-training on Qwen3-8B. The audited arms are STaR, majority-vote SFT in the style of TTRL, and a majority-vote policy-gradient arm. Distillation from gpt-oss-120b serves as a positive control. A frozen model is pushed through the identical pipeline as a control. The paper lists seven measurement failures.

Critical strengths

1.The key observation is that every arm's checkpoint 0 is an independent evaluation of the untrained model, so a multi-arm study already owns many frozen replicates. 2.Released per-problem counts and code make the claims checkable.

Critical weaknesses
  1. Several failure modes are already known in general form: F1 (batch nondeterminism, He & Thinking Machines 2025), F3 (truncation under length drift), F4/F6 (power), and F5 (seed variance, Dodge et al. 2020).

  2. It differs from the self-training arms in teacher capability, style, and 36–46% truncation. Truncation is argued to make the teacher's counts conservative, but it also changes which problems can register at all.

  3. It is not stated whether the frozen control is served through the same LoRA-adapter sampling path (e.g. a zero-initialized adapter) as the trained checkpoints.

Questions for authors

Is the STaR rung of the matched ladder a single seed? If so, please state this clearly and either add seeds or remove the per-seed combined p-value against STaR.

Is the frozen control sampled through the same adapter-serving path as the trained checkpoints? If not, how might that affect F1 and the floors?

2 comments
AuthorsOpenReview ↗

Response to Reviewer D4KK (Part 1)

Thank you for the careful and generous review, we are sincerely appreciative of it. Each of your points led to an added check, and the paper's main comparison holds under all of them.


1. Is the STaR rung a single seed? (Q1)

In the original submission it was. STaR now has three seeds (Table 3, p. 8), and problem-level tests replace the per-seed combined p-values. "Low-base" below means the 22 AIME problems with at most 5 successes in 1,408 pooled base draws.

Arm (three seeds)Accuracy change (points)Low-base problems detected
STaR+1.8, +3.3, +3.92, 1, 2
Majority-vote SFT+3.3, −1.5, +0.72, 1, 0
Distillation+13.0, +14.7, +15.810, 11, 8

As you noted, per-seed tests on the same 22 problems are not independent. The tests now take the problem as the unit (Table 21, p. 39; Appendix D.1, p. 37).

  • Sign test on the number of seeds that detect each problem. Distillation exceeds majority-vote SFT on 12 problems and falls short on 1 (one-sided p = 0.0017), and exceeds STaR 11 to 1 (p = 0.0032).
  • A model with a fixed effect per problem and quasi-likelihood errors. Distillation's extra lift on low-base problems is +1.95 log-odds (s.e. 1.12, one-sided p = 0.041) against majority-vote SFT and +1.33 (s.e. 0.71, p = 0.030) against STaR. This is the most conservative specification. Doubled, its p-values do not reach 0.05, and Appendix D.1 says so.

On 128 never-reached MATH-Beyond problems the sign test gives 14 problems to 1 (p = 5×10⁻⁴; Table 4, p. 9).


2. Is the frozen control sampled through the adapter path? (W3, Q2)

Mostly yes, and the path makes no measurable difference (§3, p. 4; Appendix C.3, p. 33).

Every arm's checkpoint 0 samples the untrained model through a zero-initialized rank-32 adapter saved before any optimizer step. That is the path the trained checkpoints use, and it covers 9 of the 11 AIME baselines and 3 of the 5 band baselines. The two dedicated no-op evaluations per benchmark sampled the bare weights.

Adapter-path evaluations onlyAll evaluations
AIME one-success expansion null0.1750.176
AIME two-success null0.0650.058
Band floor, learned / corrupted (two-checkpoint medians)3.5 / 34 / 3

The adapter-only AIME column uses eleven adapter-path evaluations: the nine in the pooled baseline and the checkpoint 0 of two further runs.

  • Path effect, from a binomial model with problem fixed effects: +0.040 ± 0.039 log-odds on AIME (p = 0.30) and −0.002 ± 0.005 on the band (p = 0.75).
  • F1. The serialization test of §4.1 ran on the adapter path. There, 16 of 200 greedy verdicts flip (8.0%), against 15 of 200 (7.5%) on the bare path.
  • Trained checkpoints, re-evaluated through the same path, show no increase against their own first evaluation and one decrease (Table 16, p. 33).
AuthorsOpenReview ↗

Response to Reviewer D4KK (Part 2)

3. Distillation differs in capability, style and truncation (W2)

All three differ on the teacher rung, and the revision measures each.

What differs on the teacher rung. The teacher is a positive control. On its rung the author of the training text changes, not only the filter, and the paper now says "source of supervision" (§5, p. 7; Table 3).

What the style is. A distillation target is the teacher's complete output, its reasoning as well as its final solution. The distilled student reproduces that format: long written-out reasoning, then a final solution. It often keeps writing after it has boxed an answer, until the token cap (Appendix A.1, p. 22).

Truncation, and which problems can register. Appendix A.2 (p. 23) now states the grading rule exactly: a truncated completion is graded on the last answer it boxed, where the earlier text said it was always graded incorrect. For the base model the difference is negligible (1 of 3,541 truncated completions in the AIME baselines is graded correct). For the distilled student it is not: at its final checkpoints, 2–9% of truncated completions are graded correct. Truncation also falls hardest where the comparison is made, on 66–75% of the student's completions on the 22 low-base problems against 30–41% on the rest (Appendix D.1, p. 38).

So that nothing rests on truncated completions, we re-scored every arm, and the baseline, with every truncated completion counted as incorrect (Table 20, p. 39; Table 31, p. 45).

DetectionsAs gradedTruncated counted as incorrect
AIME low-base, distillation10, 11, 89, 8, 6
AIME low-base, majority-vote SFT / STaR / placebo2, 1, 0 / 2, 1, 2 / 1, 0unchanged
AIME sign tests (one-sided p)12:1 (0.002), 11:1 (0.003)10:1 (0.006), 9:1 (0.011)
MATH-Beyond, distillation11, 8, 87, 5, 8
MATH-Beyond, majority-vote SFT / STaR0, 1, 0 / 0, 0, 0unchanged
MATH-Beyond sign test (one-sided p)14:1 (5×10⁻⁴)12:1 (0.002)

The policy-gradient arm's single low-base detection does not survive the stricter rule, so the paper does not describe its count as a floor (Appendix A.3, p. 25).

Gain. To test whether the teacher's advantage is only its larger overall gain, we stopped distillation inside its first round. At step 32 its gain is +6.2 points, close to STaR's best checkpoint (+6.0), and it detects 8 low-base problems against STaR's 2. This comparison is sensitive to how truncated completions are scored: 70% of that checkpoint's correct samples are truncated completions that had already boxed an answer. Under the stricter rule the matched checkpoint is step 16 (+3.0 points, 7 detections), and all 18 self-training checkpoints stay at 0–2 under either rule (Table 29, p. 42). The pre-specified window for this comparison was a gain of +1.5 to +4 points. As graded, no checkpoint fell inside it, and Appendix D.3 reports that deviation.

The comparison therefore holds under the strictest reading of truncation. The teacher's advantage comes with its style and the two are not separated here; §5 and §6 state that as a scope condition.


4. Several failure modes are known in general form (W1)

Table 1 (p. 6) now marks F1 and F3–F6 as known and credits He & Thinking Machines Lab (2025), Hochlehnert et al. (2025), Card et al. (2020) and Dodge et al. (2020). F3–F6 take one paragraph (§4.3, p. 7). The main text can then concentrate on what we believe is new: the expansion null that no sampling budget removes (§4.2, p. 6), and the placebo arm, which shows that a frozen null cannot attribute regressions to a method's signal (Table 5, p. 10).


Thank you once again for your generous feedback. We would be glad to answer further questions.

Reviewer_PFhLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that existing methods do not robustly capture problem-level capability acquisition and forgetting in LLMs. The authors identify seven measurement failures that cause the evaluation method to report false gains or losses. To address these failures, they propose a statistically controlled evaluation framework and validate it experimentally using Qwen3-8B across self-training and external distillation settings. The results of the paper demonstrate that capability acquisition and forgetting require more complex methods of measurement rather than simple before-and-after comparisons.

Critical strengths
  1. Tackles an important evaluation problem: The paper highlights how noise in the evaluation method can lead to completely different conclusions about model capabilities and how easy it is to make such evaluation errors.

  2. Statistical Validation: The paper is careful in tackling evaluation noise and uses appropriate statistical methods to reduce the risk of false capability gains or losses.

Critical weaknesses
  1. Limited novelty: The proposed approach is a combination of established methods, such as repeated evaluations, Fisher's exact test, and FDR corrections. The sensitivity of LLM evaluation results to repeated runs and sampling variability is already well documented, as the paper itself establishes. It is therefore unclear to me exactly what substantial methodological or conceptual contribution this work is making beyond applying established methods.

  2. Limited generalizability: While the authors acknowledge this limitation, relying only on 22 problems for the expansion analysis and a single base model (Qwen3-8B) makes it difficult to justify the generalizability of this work.

  3. Clarity: Some parts of the paper are unnecessarily verbose and difficult to follow. Explanations could be simplified without the loss of technical details, which would improve the paper's readability.

Questions for authors
  1. Does the result hold for other base models beyond Qwen3-8B and a larger data sample?

  2. Baseline: Does the evaluation noise measured on the untrained model represent the trained model, which might have changed significantly after training?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PFhL (Part 1)

Thank you for the insightful review, we are sincerely appreciative of it. You asked for a clearer statement of the contribution, broader evidence and a more readable paper. We have tried to keep this response short for the same reason.


1. What is new beyond established tools (W1)

Fisher's test, FDR control and repeated evaluation are standard tools, and run-to-run variance in aggregate accuracy is well documented; §2 (p. 3) now cites that literature (Madaan et al., 2025; Hochlehnert et al., 2025; Bouthillier et al., 2021). The tools are not the contribution. We claim three findings about per-problem statistics, which behave differently from aggregate accuracy.

  1. More samples do not remove the expansion statistic's null. The expansion rate is the share of problems the base model fails in k draws that the trained model then solves at least once. For mean accuracy, sampling noise shrinks as k grows. Here it does not: re-sample a frozen model and a roughly constant share of its unsolved problems gets solved, because that set was selected on a noisy zero. A short derivation gives the limit 1−2−a1 - 2^{-a} (Eq. 4, p. 6; Appendix D.2, p. 39). Measured, the rate is 0.18–0.21 from k = 128 to 768 on AIME, and 0.099 then 0.100 on a second model family (Table 2, p. 7). It is non-zero at every doubling of k up to 1,024 for all 21 models in MATH-Beyond's released curves (Table 28, p. 42).
  2. One frozen comparison cannot certify a repaired threshold. Requiring two successes gives a null of 0 on one frozen pair and 0.058 over all 110 pairs; nine replicates are needed to fix it within ±0.02 (Table 24, p. 40). The test we recommend instead compares each problem with a baseline pooled from every arm's checkpoint 0. It needs no new experiments, and its held-out null is zero.
  3. The null has to match the claim. Against a frozen model, self-training regresses 88–106 problems (floor: 8). A placebo arm with no correctness information regresses largely the same problems by as much (odds ratio 6.2; median drop 0.29, against 0.25 and 0.32). A lower sampling temperature regresses mostly different ones (14–16% overlap; chance 11%). A frozen null alone would have attributed these regressions to the training signal (Table 5, p. 10).

2. Does noise measured on the untrained model represent the trained model? (Q2)

We tested this directly by re-evaluating four stored trained checkpoints at k = 128 (Appendix C.3, Table 16, p. 33).

  • Test–retest. Testing each trained AIME checkpoint against its own re-evaluation gives no increase in 180 tests and one decrease (a STaR problem: 56, then 29 correct of 128). On the band the test–retest floor is 0 learned and 0 corrupted; the frozen range goes up to 4.
  • Replication. 35 of distillation's 36 AIME detections and all 19 of STaR's band regressions recur. Majority-vote SFT's and STaR's AIME detections recur at 15/19 and 8/11. Apart from the one problem above, those that do not recur were within three samples of the original count. Detections on the 22 low-base problems number 10 in both evaluations of the distillation checkpoint, and fall from 2 to 1 for each self-training checkpoint.

Two design choices make this the expected outcome. The exact test uses the trained checkpoint's own counts, so it assumes nothing about the trained model's variance. And the ledger's band is sized at the worst-case standard error of a binomial proportion, which no solve rate can exceed. Untrained and trained checkpoints are also sampled through the same adapter path, with no measurable path effect (Appendix C.3, p. 33).

AuthorsOpenReview ↗

Response to Reviewer PFhL (Part 2)

3. One base model and 22 problems (W2, Q1)

A larger sample. AIME has only 10 problems that the base model never solves. On MATH-Beyond, under our protocol, the base model fails 128 problems in all 512 draws. We evaluated all nine final ladder checkpoints on them (Table 4, p. 9; Table 31, p. 45).

ArmDetections per seedDistinct problems
Distillation11, 8, 814
Majority-vote SFT0, 1, 01
STaR0, 0, 00

That is 14 problems to 1 (sign test, one-sided p = 5×10⁻⁴). The paper states two caveats. The pool is defined under our protocol (non-thinking mode, an 8,192-token cap); under MATH-Beyond's own protocol the base model reaches more of these problems. And the distilled student reaches the token cap on 61–71% of its completions; with every truncated completion counted as incorrect it still detects 7, 5 and 8 (12 problems to 1, p = 0.002).

On AIME's 22 low-base problems (at most 5 successes in 1,408 pooled base draws), STaR now has three seeds and the tests take the problem as the unit: 12 problems to 1 against majority-vote SFT and 11 to 1 against STaR (one-sided p = 0.002 and 0.003; Table 21, p. 39).

Other base models. The training arms stay on one backbone, and Limitations states that scope. The newer trainable backbones available to us truncate 44–62% of completions under our protocol (Table 17, p. 34), so arms trained on them would mostly measure truncation drift (F3). We therefore extended the measurements that need no training.

  • A second family, frozen. Four evaluations of Nemotron-3.5-Lightning on AIME (Table 18, p. 34). Greedy verdicts flip on 6.9% of problems between evaluations (Qwen3-8B on AIME: 6.7%). The one-success expansion null is 0.099, and 0.100 at k = 256; the two-success null is 0.027. The pooled exact test returns no detection on any of the four held-out evaluations.
  • 21 released models. The expansion null is non-zero at every budget in MATH-Beyond's curves (Table 28, p. 42).

So the measurement findings hold on more than one model. The findings about trained arms rest on Qwen3-8B, and we scope them that way.


4. Readability (W3)

We restructured the paper as well as editing it.

  • The abstract and introduction are rewritten around two findings, followed by a four-item list of contributions (p. 2).
  • A "Definitions at a glance" box collects every term before the experiments (p. 4).
  • The five known failure modes are credited in one table (Table 1, p. 6), and four of them take one paragraph (§4.3, p. 7).
  • §5 is organised by question, with the two new result tables in the main text (Tables 4 and 5).
  • §6 states the recommendation as a short reporting standard (p. 10).

If particular passages are still hard to follow, we would be glad to revise them.


We would be glad to answer further questions. If the revision resolves your concerns, we would be grateful if you would reconsider your assessment.

I appreciate the effort put in by the authors to address my questions. I still find the novelty somewhat limited, and the generalizability of the training arms is still shown only for Qwen. I will retain my score.

Thank you for taking the time to review our rebuttal and revised manuscript. We appreciate your feedback and respect your assessment.

Reviewer_qC6VOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that per-problem metrics for self-improvement (which problems a model gains, loses, or newly solves) are noisy and can lead to wrong conclusions without a measured baseline. The authors fine-tune Qwen3-8B with LoRA for three rounds using STaR, majority-vote SFT and majority-vote policy gradient, with distillation from gpt-oss-120b as a positive control. An untrained copy of the model goes through the same evaluation to measure noise.

They describe seven measurement mistakes (F1–F7). The central one is that the standard expansion metric gives an untrained model a score of 0.28, and raising the threshold does not fully fix it. They propose a per-problem Fisher test against a baseline pooled from all untrained evaluations, with FDR control. Under this test, distillation improves rarely-solved AIME problems while self-training does not, and self-training loses many problems it could already solve.

Critical strengths
  1. The problem is real. Per-problem analysis is becoming common in the self-improvement and RLVR debate, and the result that the expansion metric gives a frozen model a nonzero score is a useful warning.

  2. The pooled baseline is a good practical idea. Every arm’s checkpoint 0 is a free evaluation of the untrained model, and pooling them gives a shared baseline at no extra cost.

Critical weaknesses
  1. The “seven failure modes": Most of the seven are standard good practices: multiple seeds, power analysis, logging truncation, not trusting a 10-prompt probe, and knowing greedy decoding is not deterministic. The claim that each one “inverts a reported finding” is mostly shown against the authors’ own pilots or hypothetical analyses, not against published papers.

  2. The null only covers evaluation noise, not training noise. The frozen model shows what happens when nothing changes. It does not show what happens when a model is fine-tuned on something that should add no capability.

  3. Conclusions rest on single numbers from single runs, not on distributions over training runs. The paper shows that training randomness is large (CLR ranges from 0.55 to 1.53 across seeds for the same method), but it still compares single numbers (CLR, ER, detection counts) against a null. The Fisher test handles sampling noise within one run, not variation across runs. Nice to have a discussion on this. Look at a few relevant papers from other fields below:

a. Dong, J., Roth, A., & Su, W. J. (2022). Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1), 3–37.

b. Panda, S., Tarsadiya, D., Sourav, S., A.P., P., & Karimireddy, S. P. (2025). f-INE: A hypothesis testing framework for estimating influence under training randomness. ICLR, 2026, arXiv preprint arXiv:2510.10510.

Questions for authors

Address the weakness. Also the following

  1. How do the corruption counts and detections compare against a placebo-training arm?
  2. If outcomes are compared as distributions across training seeds (trained vs. null runs), do the main conclusions hold?
  3. Which published papers made each of the seven mistakes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer qC6V (Part 1)

Thank you for your thoughtful and constructive review, we are sincerely appreciative of it. Your three questions led to the most informative additions in the revision. We’ll start by providing short answers first, then the detail.

  • Q1, placebo arm. A placebo arm with no correctness information regresses as many problems as self-training does, and largely the same ones. The regressions therefore come from fine-tuning on the model's own text, not from the training signal, and the paper now says so.
  • Q2, runs as the unit. Compared run by run, every trained and placebo run lies outside the frozen range; trained and placebo runs are not separable from each other; distillation runs are separated from all others. With three seeds the smallest attainable p is 0.05, and the paper says so.
  • Q3, published papers. We found published protocols that share the design feature behind five of the failures (F1, F2, F3, F5 and F8), and for one of them we could measure the missing null on the authors' released data. For F4, F6 and F7 we found no instance we could verify; the paper says so and explains why.

1. A null for training noise (W2, Q1)

This was the right control to ask for, and we ran it (§3 "Three nulls", p. 4; Table 5, p. 10; Appendix C.6, p. 35).

Design. The placebo is STaR's pipeline with the same stream, rounds, LoRA rank, learning rate and training volume. Only the filter changes: instead of a correct sample, it keeps one random sample of the model's own output per problem (temperature 1, top-p 1). Truncated samples are dropped, a rule that never reads the answer; apart from that, the expected first gradient is zero. Three seeds ran on the 175-problem band subsample and two on AIME.

Results.

Problems regressed per seed (of 175)Accuracy change (points)
Frozen model, re-evaluatedat most 40
Placebo19, 24, 18+2.4, +1.2, +1.7
STaR19, 17, 10+6.7, +5.6, +6.9
Majority-vote SFT11, 29, 24+6.1, −2.8, −1.0
  • Same problems. Over three seeds the placebo and STaR share 23 regressed problems (of 39 and 33), where chance predicts 7.4. The odds ratio is 6.2 after stratifying by baseline solve rate.
  • Same size. The median drop in solve rate is 0.29 for the placebo, 0.25 for STaR and 0.32 for majority-vote SFT.
  • What the correctness filter adds is net gain: +5.6 to +6.9 points for STaR, against +1.2 to +2.4 for the placebo.
  • On AIME the placebo detects 1 and 0 low-base problems (those with at most 5 successes in 1,408 pooled base draws), like the self-training arms (0–2) and unlike distillation (8–11) (Table 3, p. 8).

Read against a frozen model alone, the regressions looked like harm done by self-training. Read against the placebo, they are what fine-tuning on self-generated text does with or without a signal. This is now failure mode F8 (Table 1, p. 6) and a main result of §5 (p. 9).


2. Single runs compared with a null (W3, Q2)

We now also treat each training run as one draw, following the hypothesis-testing view of Dong et al. (2022) and Panda et al. (2026), which we discuss in §2 (p. 3), §5 "Runs as the unit" (p. 9) and Appendix C.6 (p. 36).

Per runTrained runsPlacebo runsFrozen model
Band problems regressed (of 175)9–29 (seven runs)18–24 (three runs)at most 4 over 3,000 design-matched orderings
Low-base AIME problems detecteddistillation 10, 11, 8; majority-vote SFT 2, 1, 0; STaR 2, 1, 21, 00 on all 11 held-out replicates

On 128 MATH-Beyond problems that the base model never solves, distillation detects 11, 8 and 8 per seed, majority-vote SFT 0, 1 and 0, and STaR 0, 0 and 0 (Table 4, p. 9).

The main conclusions therefore hold run by run. Trained and placebo runs are separable from evaluation noise and not from each other, and distillation is separated from every other arm.

The resolution is limited, and the paper states the limit. With three seeds per arm, the smallest attainable one-sided permutation p between two arms is 1/20 = 0.05, and between an arm and the 11 frozen replicates it is 1/364. The problem-level sign tests have more resolution (12 problems to 1 and 11 to 1 on AIME, one-sided p = 0.002 and 0.003; 14 to 1 on MATH-Beyond, p = 5×10⁻⁴), but they condition on the runs we have. We report both kinds of test and say which is which. We also added STaR seeds 1 and 2, so each of the three filtered SFT arms has three seeds.

AuthorsOpenReview ↗

Response to Reviewer qC6V (Part 2)

3. Standard practice, and which papers (W1, Q3)

The revision separates what is known from what is new, and credits the former.

What is known. Table 1 (p. 6) has a new status column. F1 and F3–F6 are marked as known in general form and credited (He & Thinking Machines Lab, 2025; Hochlehnert et al., 2025; Card et al., 2020; Dodge et al., 2020), and F3–F6 are reduced to one paragraph (§4.3, p. 7). The table's "would have concluded" column describes our own runs, and the TL;DR, abstract and text no longer say "inverts a published conclusion".

What we claim as new. First, the expansion statistic's null does not shrink as the sampling budget grows. Of the problems a frozen model fails in k draws, a roughly constant share is solved in the next k. A short derivation gives the limit 1−2−a1 - 2^{-a} (Eq. 4, p. 6). Measured on AIME, the share is 0.18–0.21 from k = 128 to 768 while the unsolved set shrinks from 23 to 13 problems; on a second model family it is 0.099, then 0.100 (Table 2, p. 7). Second, a frozen null cannot attribute a per-problem change to a method's training signal; that takes the placebo above.

Published instances. We read nine protocols in this literature (Appendix E.1, Table 35, p. 49). Seven share a design feature behind F1 (Wu et al., 2025), F2 (Yue et al., 2025; Mayilvahanan et al., 2026; Wu et al., 2026), F3 and F5 (Zuo et al., 2025) or F8 (Lin et al., 2026; Yu et al., 2026a), and the table lists each with the controls it does include. A shared feature is not a demonstrated error. For F3, an independent audit has measured the mechanism on public checkpoints: at an 8k-token limit, a base model truncates 30.6% of its completions where two RLVR models trained from it truncate 12.2% and 2.0% (Yang, 2026). For F4, F6 and F7 we found no instance that can be verified from the papers alone. F4 and F6 are documented in general form in this literature (Hochlehnert et al., 2025; Fraser et al., 2025), and F7 presupposes a measured frozen null, which none of the nine protocols reports.

One case we could measure. MATH-Beyond releases pass@k curves for 21 models. The share of problems unsolved at k that are first solved by 2k is the frozen one-success rate at budget k, and it is non-zero for every model at every doubling up to k = 1,024 (Table 28, p. 42). For R1-Distill-Qwen-1.5B it is 7–12% per doubling at large k, comparable to the 5.2%, 7.8% and 9.6% expansion rates reported for three RL models trained from it. Skywork-OR1-7B's reported 21.2% is about twice its base model's rate (8–13%). A measured null therefore does not dismiss reported expansion; it shows which results clear it. We present this as a calibration that MATH-Beyond's data release makes possible.


We would be glad to answer further questions. If the revision resolves your concerns, we would be grateful if you would reconsider your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revision

Dear AC and Reviewers,

Thank you for the careful and constructive reviews. They made this a better paper. Below is a short summary of our response.

1. New experiments for every request To answer each concern with data rather than argument, the revision adds 46 runs (48 → 94 in total):

  • Placebo arm (qC6V): fine-tuning on the model's own unfiltered samples, as a null for training noise.
  • Decoding control (nYA5): the frozen model sampled at a lower temperature on all 1,163 band problems.
  • Larger test set (nYA5, PFhL): 128 MATH-Beyond problems the base model never solves.
  • More seeds, stricter tests (D4KK, nYA5): STaR now has three seeds, and problem-level tests replace the pooled regression.
  • Generality and robustness (PFhL, D4KK): frozen nulls on a second model family and in the released curves of 21 models, re-evaluated checkpoints, and adapter-path and truncation checks.

2. What the new evidence shows The main comparison holds under every added control. One conclusion changed: the placebo arm showed that the regressions we had attributed to self-training also occur with no correctness signal, and the paper now says so.

3. What changed in the paper The research question, setup and original measurements are unchanged. We narrowed two claims, credited the failure modes that were already known, and reorganized the text for readability.

4. Reproducibility We have updated both the revised PDF and the supplementary code. The code and per-problem records cover every new experiment, so each reported number can be recomputed.

We are grateful for your time and would be glad to answer any further questions.

Sincerely,

The Authors 1867

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.