- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate teacher supervision across examples in on-policy self-distillation (OPSD). IG-OPSD appends the reference solution after the student response and scores each example by the length-normalized log-likelihood gap between the privileged teacher and the student on the reference. The score is z-normalized against a window made of the current batch plus 256 recent values, and the positive part is used as the example weight. With LoRA (100 steps) on Qwen3-1.7B/8B/14B, the four-benchmark Avg@12 mean (AIME24/25, AMC23, HMMT25) improves from 39.6 to 41.5 at 8B and from 49.5 to 52.8 at 14B, with little change at 1.7B (20.1 to 20.5).
Critical strengths
- A useful, orthogonal design axis with a simple, cheap method. Existing OPSD extensions change the target distribution or reweight tokens; this paper instead asks which examples should receive more teacher influence, which is complementary to that work. The method leaves rollout generation and the token-level objective unchanged and only adds a stop-gradient per-example weight, so it is easy to drop into existing pipelines. The measured cost is low: total 14B training time rises from 14,034 s to 14,637 s, about 4.3% (Table 8).
- A positive main result, reported with uncertainty estimates and careful scoping. At 14B, the four-benchmark Avg@12 gain over OPSD is +3.28 points with a paired problem-level bootstrap 95% CI of [+1.27, +5.43] (Table 4). IG-OPSD also has the highest four-benchmark mean among the compared methods at both 8B and 14B (Table 2). The authors are candid about the limits of their evidence: they state that the intervals do not include training-seed variation and that the difficulty stratification is descriptive only. This makes the claims easier to assess.
- A well-designed key control, supported by ablations and diagnostics. The shuffled-weight control keeps the same set of weights and only breaks their assignment to examples, which separates "which example gets the weight" from "reweighting changes the effective learning rate." IG-OPSD beats this control at both 8B and 14B, with aggregate CIs of [+1.02, +7.22] and [+0.56, +5.56] (Table 4), which supports the claim that correct pairing matters. The weighting ablations (binary gate, current-batch z-score, raw IG; Table 3), the Pass@12 coverage decomposition (Appendix G), and the repetition sanity check (Appendix H) further help interpret the result. The submission also includes logged diagnostics and code with unit tests, which supports reproducibility.
Critical weaknesses
- Robustness across training runs is unknown. The paper does not report the number of training seeds, and the CIs "do not include training-seed variation" (Table 4). The ribbons in Figure 3 show variation across sample indices, not across runs. At 8B, all four per-benchmark CIs include zero (Table 9), and the 8B aggregate interval is not reported. The ~1.3-point lead over PW-OPSD at 14B (Table 2) has no uncertainty estimate.
- The value of IG over plain reference NLL is not tested. The reference is both teacher input ("Here is a reference solution: {y_star}", Appendix B) and scoring target. If the teacher's score on the reference varies little across examples, IG reduces to roughly the student's reference NLL, i.e., difficulty weighting. The decomposition in Section 5.5 does not resolve this, because it is computed offline "on the pooled competition evaluation sets" (Figure 7) rather than on training batches.
- Implementation details are insufficient for reproduction. Algorithm 1 does not show gradient accumulation, so it is unclear whether the normalization group B_g contains 4 examples or the full 32 (Table 12). The token loss is not fully specified: it is unclear whether teacher probabilities are renormalized after top-k truncation, and the 10⁻⁷ clip is described in a single sentence (Table 13). Baseline configurations are incomplete; at 14B, SFT (35.1) and Purified-OPSD (36.1) are close to the untrained model (35.6), which deserves a sanity check. Deduplication between the training data and the evaluation sets is not mentioned.
Minor Weaknesses:
- At 1.7B, the per-benchmark results go in mixed directions (AIME24 −1.7, AMC23 +2.1). The paper offers only a one-sentence post hoc explanation.
- The abstract's "consistently outperforms" should be scoped to the reported single-run results.
Questions for authors
- Report mean ± std over at least 3 training seeds for OPSD and IG-OPSD at 14B, and the 8B four-benchmark aggregate CI.
- Add a student-reference-NLL weighting control under the same gating rule and budget. If IG clearly wins, my score could go up; if NLL explains most of the gain, an accurately positioned and robust simple method is still a valid contribution.
- Clarify B_g relative to gradient accumulation, and give the exact token-loss formula (top-k renormalization, clipping).
- Provide full baseline configurations and the train/eval deduplication check.
- (Optional, does not affect my score.) PACED (Xu et al., arXiv:2603.11178) also reweights problems in (self-)distillation, using the pass rate with w = p(1−p). It may be a useful point of comparison given the hard-problem gains reported in Section 5.6.
3 comments
Thanks for the additional experiments. The NLL control and HMMT25 results are helpful, although the matched three-seed gain is smaller and less consistent than the original result.
One clarification: does the implementation really apply min(p_T * (log p_T − log p_S), 1e−7) to each vocabulary component? This caps positive terms but keeps negative ones. Is this also used in both methods and the NLL control?
I'll keep my current rating pending this clarification.
Clarification on Loss Clipping and Three-Seed Results
Thank you for checking this carefully.
Three-seed results and performance on harder problems
Your observation is correct: the matched three-seed results show a smaller and less consistent improvement in the four-benchmark Avg@12 mean. Nevertheless, on HMMT25, the hardest benchmark in our evaluation, IG-OPSD improves over OPSD in all three seeds (+3.61, +4.17, and +2.22 percentage points), with the largest or joint-largest gain among the four benchmarks in each seed. These results provide some evidence of the method’s potential on harder problems.
Clipping operation and consistency across methods
Regarding clipping, yes to both questions. For each token position (i, t), the implementation computes:
c_v = p_T(v) * (log p_T(v) - log p_S(v))
loss(i, t) = sum over v in S(i, t) of min(c_v, 1e-7)
The upper clipping is applied before summing over vocabulary: positive contributions above 1e-7 are capped, while negative contributions remain unchanged.
OPSD, IG-OPSD, and the student-reference-NLL control all use the same loss routine and clipping threshold, including the matched full-vocabulary three-seed OPSD/IG-OPSD runs. The NLL control replaces the example-weighting score while retaining this loss computation.
Objective properties and clarification in the paper
Your observation also identifies an important distinction: after this asymmetric clipping, the objective is no longer a standard KL divergence and need not be nonnegative. We will make this explicit and replace the imprecise description “per-token divergence clipping” with “component-wise upper clipping before vocabulary summation.” Our empirical conclusions are scoped to example weighting under this shared clipped objective.
Response to Reviewer K6Fh
Thank you for the detailed suggestions. We added the requested training-seed comparison, NLL control, confidence intervals, and overlap check.
1. Three training seeds and confidence intervals
The original 14B comparison used top-256 loss support for OPSD and full vocabulary for IG-OPSD. We reran both methods with full vocabulary for seeds 42, 43, and 44, keeping the other settings identical. These results replace the original comparison.
The table reports the equal-weighted four-benchmark Avg@12 mean. ± denotes the sample standard deviation across training seeds.
On HMMT25, IG-OPSD improves all three seeds by +3.61, +4.17, and +2.22 points, the largest or joint-largest gain among the four benchmarks in each seed.
At 8B, the four-benchmark mean increases from 39.57 to 41.51 (+1.94 points), with a paired problem-level bootstrap 95% CI of [−0.52, +4.65].
For the Table 2 single-run 14B comparison with PW-OPSD, the mean difference is +1.30 points, with 95% CI [−0.76, +3.42]. This compares the reported configurations: top-256 support for PW-OPSD and full vocabulary for IG-OPSD.
Both intervals use 100,000 paired bootstrap replicates, resampling within each benchmark and averaging the four benchmark means equally.
2. Student-reference-NLL control
We replaced the IG score with the student's length-normalized reference NLL, retaining the same gate, normalization, schedule, full-vocabulary objective, 100-step budget, and seed 42.
IG-OPSD outperforms student-NLL weighting on all four benchmarks, with a +3.09-point mean gain. This matched run supports the value of the teacher-conditioned score beyond student-reference difficulty. We will add this control to the main ablation table.
3. Normalization group and token loss
At each 14B microstep, four workers each contribute one example, so B_g contains 4 examples. The normalization reference includes these four scores plus up to 256 recent scores. Eight microsteps form the optimizer batch of 32.
For top-k runs, teacher and student logits are separately normalized with log_softmax over the same teacher-selected support. With beta = 0, the token loss is:
loss(i,t) = sum over v in S(i,t) of min[p_T(v) * (log p_T(v) - log p_S(v)), 1e-7]
Here, p_T and p_S are the normalized probabilities on S(i,t). Clipping applies to each vocabulary-component contribution before summation. The new three-seed runs use the full vocabulary for both methods. We will make these details explicit in Algorithm 1.
4. Baseline configurations and overlap check
The 14B SFT and distillation runs use the same 29,434-example training split, 100 steps, optimizer batch 32, learning rate 5e-6, a linear schedule without warmup, and LoRA rank 64/alpha 128/dropout 0.05.
OPSD and PW-OPSD use beta = 0 and component clip 1e-7. Purified-OPSD uses beta = 0.5 without component clipping.
Distillation rollouts use temperature 1.1, top-p 0.95, top-k 20, and a 1024-token completion limit. Evaluation uses non-thinking mode, temperature 1.0, top-p 0.8, an 8192-token limit, and 12 completions per problem.
For deduplication, we applied Unicode NFKC, lowercasing, removal of LaTeX \left/\right, and whitespace removal. Comparing all training examples against the four evaluation sets found zero normalized exact overlaps.
5. Minor comments
At 1.7B, the four-benchmark mean changes from 20.12 to 20.43, with mixed benchmark-level results. We will describe this pattern separately from the larger-model gains and scope “consistently outperforms” to the reported runs.
We will cite PACED as related work. PACED uses pass-rate weighting p(1−p), whereas our score uses reference likelihoods from a single rollout.