- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides statistical audit of transition-level analysis in LLM self-improvement methods. Standard evaluation methods of LLMs focus on mean accuracy which can't distinguish if model acquired new capability, or doing distribution sharpening, and can't expose underlying regressions and forgetting. The submission has mainly 3 contributions:
- Identified and reproduced the 7 measurement failure modes for transition audits.
- Proposed a principled threshold-free measurement framework for capability expansion
- Conducted rigorous audit using Qwen3-8B across MATH-500, AIME, and 1163 difficulty band of MATH problems, discovered that self-training mainly sharpens reachable problems while inducing capability corruption.
Critical strengths
-
The paper's key strength is it decoupled of real capability gain from the measurement noise via empirical controls, it demonstrates that standard transition heuristics produce phantom learning and degradation purely from the batching non-determinism. by measuring identical pipelines against a frozen backbone and proposing an pooled exact test under FDR control, it establishes a solid methodological foundation for auditing self-improvement.
-
I appreciate that the author carefully equalize problem streams, retained volume and grading protocals between self-training regimes and external distillation. This isolates supervision sources from confounding variables, providing solid evidence that self-training predominantly sharpens existing sampling reach while inducing measureable capability corruption.
Critical weaknesses
The empirical audit only runs 3 rounds of rank-32 LoRA with roughly 270 optimizer steps on a single Qwen3-8B backbone, the experiment isn't representative of modern post training and RL pipelines. Given the experiment coverage, it's unclear whether the lack of capability expansion is an result of self-training or just the model is under-trained.
The dataset setup seems narrow, AIME only has 10 "never reached" problems across 1408 draws, relying on such a small set of data size makes the claim underpowered. It would be great if there's more evidence on this.
Questions for authors
-
Could you provide the specification of the table 15 model, are the problems fixed? which seeds enter? how are zero counts handled?
-
To separate capability loss from sharpening on the band, could you compare the difficulty band transitions against a pure sharpening control method (e.g., evaluating the frozen base model at a lower temperature)? If temperature sharpening alone reproduces a large share of the 88–106 corrupted problems (lines 439–448), the "destruction" claim might need to be reframed
-
Was Qwen3's thinking mode enabled during training and evaluation?
2 comments
Response to Reviewer nYA5 (Part 2)
3. Only 10 never-reached problems (W2)
Ten problems are too few to decide this: on those 10 AIME problems, distillation's 5 detected problems against self-training's 2 is not significant (p = 0.35). MATH-Beyond supplies a larger pool: under our protocol the base model fails 128 of its problems in all 512 draws (Appendix B.2, p. 28). All nine final ladder checkpoints were evaluated on them (Table 4, p. 9; Table 31, p. 45).
| Arm | Detections per seed | Distinct problems |
|---|---|---|
| Distillation | 11, 8, 8 | 14 |
| Majority-vote SFT | 0, 1, 0 | 1 |
| STaR | 0, 0, 0 | 0 |
That is 14 problems to 1 (sign test, one-sided p = 5×10⁻⁴). There are two caveats. The pool is defined under our protocol, not MATH-Beyond's, under which the base model reaches more of these problems. And the distilled student truncates 61–71% of its completions here; with every truncated completion counted as incorrect it still detects 7, 5 and 8 (12 problems to 1, p = 0.002).
4. Under-training, a single backbone, and thinking mode (W1, Q3)
Is the lack of expansion due to under-training? Within this regime, the evidence says no.
- The distillation control trains under the same LoRA rank, stream and volume. After 16 optimizer steps of its first round it already detects 8 low-base AIME problems (7 with truncated completions counted as incorrect); its final counts are 8–11.
- All 18 self-training checkpoints, across three rounds and six runs, stay at 0–2.
- The difference remains at a matched overall gain (Table 29, p. 42).
So the optimizer budget is not what keeps self-training from reaching new problems here. We do not claim more than that. The abstract and Limitations (§6, p. 10) scope the result to three rounds of rank-32 LoRA in non-thinking mode, and say that this does not establish that self-training cannot expand.
A single backbone, and modern RL pipelines. The training arms stay on one backbone, and the paper scopes them that way: the newer trainable backbones available to us truncate 44–62% of completions under our protocol (Table 17, p. 34). The measurement results do not depend on the training regime, and we extended those. The frozen nulls replicate on a second model family (Table 18, p. 34), and the expansion null is non-zero at every budget for all 21 models in MATH-Beyond's released curves, RL-trained models included (Table 28, p. 42).
Thinking mode. It was disabled in training and evaluation alike (enable_thinking=False), with token caps of 2,048 on MATH-500 and the band, and 8,192 on AIME and MATH-Beyond. The abstract, §3 (p. 5) and Appendix A.1 (p. 22) now say so.
Thank you once again for your generous feedback. We would be glad to answer further questions.
Response to Reviewer nYA5 (Part 1)
Thank you for the thoughtful and generous review, we are sincerely appreciative of it. For three questions that each led to a new experiment or a tighter analysis.
1. A sharpening control, and the "destruction" claim (Q2)
We ran the control you proposed, with the decision rule written down before the full run (Table 5, p. 10; Appendix C.6, p. 36).
Choice of temperature. A pilot on 175 band problems (k = 32) gave accuracies of 0.531, 0.530 and 0.528 at T = 0.6, 0.4 and 0.2. These are statistically tied (the differences are 0.1–0.2 standard errors), and our rule for a tie selects the lowest temperature, T = 0.2, which is the strongest sharpening.
Result. Sampling the frozen model at T = 0.2 on all 1,163 band problems (k = 128) regresses 74–78 problems and gains 28–41, measured against each of the five T = 0.8 baselines. Lowering the temperature therefore moves many problems on its own. But they are mostly different problems: only 14% of STaR's regressed problems and 16% of majority-vote SFT's are also regressed at T = 0.2, where chance gives about 11%. Our pre-specified threshold for "not reproduced by sharpening" was 25%.
What the regressions are. Your question led us to two further measurements, and the paper's wording now follows them.
So the regressions are real and are not decoding-level sharpening, but they belong to fine-tuning on self-generated text rather than to self-training's signal.
2. Specification of the Table 15 model (Q1)
As submitted. The 60 AIME problems were fixed. Only seed 0 of each arm entered. Counts were pooled within two strata (the 22 low-base problems, with at most 5 successes in 1,408 pooled base draws, and the other 38), with no per-problem effect. A problem with zero successes simply added its draws to its stratum's total.
Why we replaced it. Pooling treats the 128 draws of each problem as independent evidence across problems whose base rates differ by orders of magnitude. The fit is overdispersed by a factor of 297, so its p < 10⁻⁸ is anti-conservative.
Now (Table 21, p. 39; Appendix D.1, p. 37), with the problem as the unit and all seeds:
In the fixed-effects model, the four problems with no success in any evaluation carry no information and drop out. The direction is the same in every specification. The strength is not, so we report all of them and note that the fixed-effects p-values would not reach 0.05 two-sided. The abstract cites the sign test. One sentence is updated to match: a uniform-lift simulation is rejected for the self-training arms too (by one or two problems, against five to eight for the teacher), so the paper does not say that a uniform lift accounts for them.