- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies looped self-distillation, where a code model repeatedly trains on all of its own unfiltered samples and then generates the next round's data. On MBPP it shows that pass@1 rises while the diversity of correct programs (normalized-AST richness) shrinks. SPECTRUM re-estimates, each round, the gradient second moment of the loss on 50 fixed reference solutions with respect to the K/V outputs of two layers, and folds the resulting full-rank shrinkage operator into those weights only while generating training data. With Qwen2.5-Coder-1.5B over five rounds, it retains 89.9% of the initial 64-sample correct-AST richness versus 66.4% for vanilla self-distillation, gaining 2.2 pp pass@64 but losing 1.1 pp pass@1.
Critical strengths
- Separating success from diversity among correct outputs (pass@k versus correct-AST richness and correct-count-matched richness, with rarefaction estimators) is a clean measurement. The five-round tracking shows that rising pass@1 can coexist with a one-third drop in correct-AST richness.
- The intervention is cheap and absent at inference. A closed-form operator is folded into K/V weights only during data generation, the final student is a plain model, and its richness advantage survives matching the number of correct samples (D4 +0.30, D16 +1.82 classes, paired intervals excluding zero).
Critical weaknesses
-
At τ=1 all gains lie in [0.5, 1]. Unless the normalized spectrum is unusually flat, the operator is roughly 0.5·I plus a low-rank term, i.e., a near-uniform halving of K/V in two layers. No isotropic or random-basis control, τ/layer sweep, or spectrum is reported, so the gain is not shown to come from loss-sensitive geometry. The half-rank projection control is also indistinguishable from Vanilla SD (C16 3.066 vs. 3.090).
-
SPECTRUM trails Vanilla SD by 1.1 pp pass@1, and SSD retains more C16 and marginal D4. The paper has no temperature/top-p sweeps, no weight interpolation or KL toward the initial model, and no baseline that also uses the 50 reference solutions (Vanilla SD and SSD use none). It is therefore unclear that SPECTRUM beats simple knobs on the correctness–diversity frontier.
-
The evidence is narrow: one 1.5B model, MBPP-only training, five rounds, and one training run per design, with intervals covering task resampling only. The four-model matrix in Appendix E reports only initial-model pass@16. The one-round pilot is copied from an archived report after a post-hoc change to code extraction.
-
The transfer and breadth claims are overstated. Transfer rests on D4 over 111 and 42 eligible tasks (APPS lower bound 0.007), while pass@1 is lower on both benchmarks and C16 is lower on APPS. The AST fingerprint keeps literals and function names, and the control-flow gain over projection is not significant, so much of the retained breadth may be surface variation.
Questions for authors
Please refer to Weaknesses.
3 comments
3.The requested simple controls are important, and we have now separated changes to the synthesis policy from changes to the evaluation decoder. The following are trained five-round endpoints under the common native evaluation decoder:
| Training-data generation policy | Pass@1 (%) | Pass@64 (%) | Correct AST C64 |
|---|---|---|---|
| Plain, default | 41.47 | 71.2 | 8.158 |
| Plain, temperature 1.0 | 39.08 | 73.0 | 9.932 |
| Plain, top-p 0.8 | 42.87 | 70.4 | 6.734 |
| SSD generation recipe (T=1.5, p=0.8, k=20) | 35.92 | 72.0 | 12.048 |
| Fixed SPECTRUM | 40.24 | 72.4 | 10.870 |
SSD achieves higher C64 than Fixed but lower Pass@1. Increased synthesis temperature is also an effective competing trade-off. These observations will be reported as such, without a claim that Fixed dominates the full accuracy–diversity frontier.
Independently, evaluating the same frozen Fixed checkpoint at temperatures 0.6 / 0.8 / 1.0 yields Pass@1 of 42.07% / 40.24% / 37.57% and C64 of 9.288 / 10.870 / 12.240. These are evaluation-time changes, not additional trained models.
We also evaluated full-weight interpolation with the initial model, :
| Interpolated student | Pass@1 (%) | Pass@64 (%) | Correct AST C64 |
|---|---|---|---|
| Plain, a=0.50 | 39.27 | 73.4 | 10.778 |
| Plain, a=0.75 | 40.45 | 71.0 | 9.384 |
| Fixed, a=0.50 | 38.58 | 73.6 | 12.426 |
| Fixed, a=0.75 | 39.28 | 72.6 | 11.388 |
Interpolation is a meaningful alternative, and the same interpolation can further increase Fixed's richness. This is an inference-time checkpoint-interpolation control; a separate training-time KL-to-initial regularization baseline has not been executed. We will include these distinctions in the revised comparison.
4.We expanded the single-run evidence with three independently trained, paired 1.5B Plain/Fixed trajectories (five rounds, MBPP500, 512-token synthesis cap, 64 evaluation samples per task):
| Seed | Plain C64 | Fixed C64 | Difference | Plain Pass@64 (%) | Fixed Pass@64 (%) |
|---|---|---|---|---|---|
| 43 | 8.158 | 10.870 | +2.712 | 71.2 | 72.4 |
| 44 | 8.484 | 11.498 | +3.014 | 71.6 | 72.8 |
| 45 | 8.314 | 11.156 | +2.842 | 70.2 | 71.2 |
| Mean ± sample SD | 8.319 ± 0.163 | 11.175 ± 0.314 | +34.3% relative | 71.00 | 72.13 |
At a matched budget of four correct samples on tasks eligible for both arms, paired AST-D4 differences are +0.262 / +0.289 / +0.300 on 312 / 316 / 315 common tasks. These are distinct training seeds, not bootstrap resamplings of one checkpoint.
We also trained additional backbones for five rounds:
| Backbone and protocol | Plain Pass@64 (%) | Fixed Pass@64 (%) | Plain C64 | Fixed C64 |
|---|---|---|---|---|
| Qwen2.5-Coder-3B, 512-token cap | 73.2 | 72.0 | 13.320 | 18.406 |
| DeepSeek-Coder-6.7B, 512-token cap | 72.6 | 72.4 | 8.026 | 9.446 |
| Qwen2.5-Coder-7B, separate 4096-token protocol | 82.4 | 84.8 | 9.174 | 14.570 |
| Llama-3.1-8B, separate 4096-token protocol | 62.2 | 66.0 | 4.852 | 7.066 |
The C64 increase extends across these trained settings; Pass@64 does not increase for every backbone. The 4096-token runs are a separate protocol, not replacement cells for the original 512-token experiment. Each additional backbone currently has one matched training lineage.
A separate, post-submission learned-gain extension completed ten rounds on Qwen2.5-Coder-7B (MBPP500, 4096-token synthesis cap). At R10, Plain and Learned obtain Pass@1 56.40% vs 56.30%, Pass@64 82.0% vs 82.4%, and correct-program unique fraction 0.621 vs 0.667; mean evaluation output lengths are 166 vs 167 tokens. At R8, Fixed40 and Learned obtain Pass@1 50.37% vs 57.89%, Pass@64 83.8% vs 83.2%, and mean lengths 247 vs 159 tokens. These observations motivate learning intervention gains rather than fixing their strength throughout recursion. Fixed40 was measured through R8 and SSD through R9, not R10. Learned is explicitly an extension, distinct from the submitted Fixed method.
5.The reviewer is correct that Fixed SPECTRUM uses 50 reference answers for calibration, whereas Plain and SSD do not. The generation-stage intervention is removed before standard LoRA SFT; all generated completions are retained without execution-based filtering. To test whether simple access to the same answers explains the result, we added supervised reference replay:
| Five-round 1.5B method | Pass@1 (%) | Pass@64 (%) | Correct AST C64 |
|---|---|---|---|
| Plain | 41.47 | 71.2 | 8.158 |
| Plain + 50-reference supervised replay | 41.79 | 70.6 | 7.936 |
| Fixed, 50 gradient-calibration references | 40.24 | 72.4 | 10.870 |
| Fixed, 40 calibration references | 40.28 | 72.6 | 11.134 |
Simple reference replay does not reproduce the Fixed method's correct-structure richness in this control. It does not make the information usage or calibration compute identical: supervised replay and gradient-based modulation use the reference answers differently.
The original main comparison holds the 291 synthesis prompts, 16 raw samples per prompt, five training rounds, LoRA settings, and evaluation protocol constant. It does not establish identical realized supervised-token exposure or equal total FLOPs. We additionally ran a protocol with a common configured minimum of 7.2 million shifted supervised target-token presentations per round, allowing bounded whole-record overshoot:
| Five-round 1.5B method, 7.2M target-token minimum | Pass@1 (%) | Pass@64 (%) | Correct AST C64 |
|---|---|---|---|
| Plain | 20.95 | 45.4 | 5.666 |
| Fixed | 9.04 | 29.4 | 3.072 |
| Plain + replay | 20.66 | 49.4 | 6.132 |
This high-exposure setting is unfavorable to Fixed. The configured equal target is not evidence of identical realized exposure, wall time, or FLOPs; complete per-round consumption and runtime accounting are not included in the present summary. We will report the main comparison as matched in prompt count, sampling count, training algorithm, and evaluation, not as an equal-compute result. The negative token-exposure outcome will be included as protocol sensitivity in the revision.
6.We added a stricter correctness check using the MBPP+ v0.2.0 Plus tests on existing generated samples. The common evaluated population is 224 tasks shared by the original MBPP500 outputs and the MBPP+ fixture, with 64 pre-existing samples per task. This is a re-verification of an intersecting test set, not a new training dataset or a full 378-task MBPP+ evaluation. C64 below uses normalized-AST implementation fingerprints rather than exact-source hashes.
| Model / student | MBPP+ Pass@1 (%) | MBPP+ Pass@64 (%) | Correct AST C64 |
|---|---|---|---|
| 1.5B Plain R5 | 43.22 | 69.64 | 6.571 |
| 1.5B Fixed R5 | 42.56 | 70.54 | 9.188 |
| 1.5B Random R5 | 41.20 | 70.54 | 11.277 |
| 7B Plain R10 | 52.96 | 74.55 | 8.594 |
| 7B Learned R10 | 55.64 | 76.79 | 10.571 |
For 1.5B Plain versus Fixed, common-eligible correct-count-matched AST-D4 increases by +0.246 on 142 tasks, while control-flow D4 changes by −0.002. For 7B Plain versus Learned, the common-eligible AST-D4 increase is +0.019 on 154 tasks, whereas full-population C64 increases. We will distinguish full-population richness from diversity conditional on a fixed number of correct samples.
Frozen-checkpoint transfer results are mixed, including adverse results:
| Benchmark / evaluation | Plain | Fixed |
|---|---|---|
| HumanEval+ (164 tasks), Pass@16 | 86.59% | 86.59% |
| HumanEval+ (164 tasks), C16 | 5.848 | 6.000 |
| Adapted APPS Intro (200 tasks), Pass@16 | 35.5% | 34.0% |
| Adapted APPS Intro (200 tasks), C16 | 1.930 | 1.845 |
| Adapted APPS Intro (200 tasks), Pass@64 (separate rerun) | 48.0% | 43.5% |
AST classes measure implementation structure, not semantic algorithm families. An exploratory, non-blind LLM-only strategy annotation on 30 MBPP+ tasks and 501 correct programs finds 1.200 / 1.233 / 1.233 mean task-local strategies for Plain / Fixed / Random; this is not human validation and does not establish a meaningful increase in algorithm-family breadth. We will remove the stronger semantic interpretation.
We will separate the archived extraction-changing one-round pilot from matched, newly evaluated five-round comparisons, and specify the code-extraction rule, verifier, task set, sample counts, and provenance for each result. Conditional D4 comparisons will always disclose their common eligible-task denominator. The revised manuscript will focus on reproducible correct implementation-structure richness and the measured accuracy–coverage trade-off, explicitly reporting negative transfer and methodological limits.
spectral orientation versus calibrated attenuation
1.We thank the reviewer for the careful and constructive technical analysis, particularly the proposed isotropic, random-basis, and simple-control comparisons. Following these suggestions, we ran a broad set of additional experiments. The results have clarified the mechanism and changed several interpretations we will make in the revised paper. We agree that the original manuscript did not isolate gradient-aligned orientation from attenuation strength.
For τ=1, let μi=λi/λmax. The operator admits the exact decomposition
T=21I+Udiag(2(2−μi)μi)U⊤.It is therefore isotropic attenuation plus a spectrum-dependent correction. The size and effective rank of that correction depend on the measured spectrum; the gain bound [0.5,1] alone does not establish that the correction is negligible.
We added the following five-round controls under the original Qwen2.5-Coder-1.5B / MBPP500 protocol (64 native evaluation samples per task):
Random-basis soft gains retain the gradient-derived gain spectrum and replace the eigenbasis with an independently sampled orthogonal basis. Isotropic soft attenuation uses one scalar gain matched to the soft operator's Frobenius distance from identity. Neither is a gradient-free control. Their higher C64 and slightly lower Pass@1 show that gradient-aligned eigenvectors are not necessary for the richness gain, and that attenuation strength and orientation lead to different accuracy–richness trade-offs. Matched Frobenius displacement does not imply matched output-distribution KL.
We will revise the paper accordingly: the proximal map is a classical mathematical construction, and our supported empirical contribution is generation-time, gradient-calibrated attenuation in recursive self-distillation and its measured effect on correct implementation-structure richness. We will not claim that the specific gradient eigenvectors are required. The hard-projection entry is a reconstructed control, not an independently verified upstream SPD reproduction.
2.We agree that the behavior of the operator must be established empirically rather than inferred from its formula. We ran the following five-round sensitivity experiments under the same 1.5B / MBPP500 / 64-sample evaluation protocol:
Increasing τ from 0.5 to 2 increases C64 from 9.458 to 12.694 while reducing Pass@1 from 41.36% to 38.96%. The middle-only and last-only experiments show that placement affects the trade-off. Reusing the complete round-one operator also retains high C64; this experiment does not establish that re-estimating geometry at every round is necessary.
The implementation records each targeted K/V operator's eigenvalues, mean gain, and departure from its scalar component. We distinguish (i) the exact algebraic decomposition, (ii) operator-level eigenvalue diagnostics, and (iii) performance-level controls. Full per-layer/per-round spectral histograms and numerical anisotropy ratios are not present in the current shareable result package, so we do not report inferred values as measurements. We will include measured spectral plots only with the corresponding verified operator artifacts in the revised manuscript.