acceptodds
Under review as a conference paper at ICLR 2027

Distill Where It Matters: Selective On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) provides dense supervision without a stronger external teacher by conditioning a frozen teacher initialized from the same model on the reference solution. However, standard OPSD distills every example equally, overlooking that privileged information may benefit the teacher differently across problems. We introduce Information-Gain Weighted OPSD (IG-OPSD), which measures the teacher's advantage over the student through their reference-likelihood gap and assigns greater weight to examples where this advantage is larger. Experiments with Qwen3 models show that IG-OPSD consistently outperforms uniform OPSD across four mathematical reasoning benchmarks at both 8B and 14B scales. On the two most difficult benchmarks under uniform OPSD, the 14B model improves Avg@12 by 4.2 points on AIME25 and 5.6 points on HMMT25. Ablation and diagnostic analyses show that the gains depend on matching weights to the correct examples: IG primarily captures the advantage provided by privileged context and directs learning toward harder problems. These results establish sample selection as a key design axis for on-policy self-distillation: the student should learn more from the teacher precisely when privileged information makes the teacher more informative.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, self-distillation, information gain, privileged information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K6FhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate teacher supervision across examples in on-policy self-distillation (OPSD). IG-OPSD appends the reference solution after the student response and scores each example by the length-normalized log-likelihood gap between the privileged teacher and the student on the reference. The score is z-normalized against a window made of the current batch plus 256 recent values, and the positive part is used as the example weight. With LoRA (100 steps) on Qwen3-1.7B/8B/14B, the four-benchmark Avg@12 mean (AIME24/25, AMC23, HMMT25) improves from 39.6 to 41.5 at 8B and from 49.5 to 52.8 at 14B, with little change at 1.7B (20.1 to 20.5).

Critical strengths
  1. A useful, orthogonal design axis with a simple, cheap method. Existing OPSD extensions change the target distribution or reweight tokens; this paper instead asks which examples should receive more teacher influence, which is complementary to that work. The method leaves rollout generation and the token-level objective unchanged and only adds a stop-gradient per-example weight, so it is easy to drop into existing pipelines. The measured cost is low: total 14B training time rises from 14,034 s to 14,637 s, about 4.3% (Table 8).
  2. A positive main result, reported with uncertainty estimates and careful scoping. At 14B, the four-benchmark Avg@12 gain over OPSD is +3.28 points with a paired problem-level bootstrap 95% CI of [+1.27, +5.43] (Table 4). IG-OPSD also has the highest four-benchmark mean among the compared methods at both 8B and 14B (Table 2). The authors are candid about the limits of their evidence: they state that the intervals do not include training-seed variation and that the difficulty stratification is descriptive only. This makes the claims easier to assess.
  3. A well-designed key control, supported by ablations and diagnostics. The shuffled-weight control keeps the same set of weights and only breaks their assignment to examples, which separates "which example gets the weight" from "reweighting changes the effective learning rate." IG-OPSD beats this control at both 8B and 14B, with aggregate CIs of [+1.02, +7.22] and [+0.56, +5.56] (Table 4), which supports the claim that correct pairing matters. The weighting ablations (binary gate, current-batch z-score, raw IG; Table 3), the Pass@12 coverage decomposition (Appendix G), and the repetition sanity check (Appendix H) further help interpret the result. The submission also includes logged diagnostics and code with unit tests, which supports reproducibility.
Critical weaknesses
  1. Robustness across training runs is unknown. The paper does not report the number of training seeds, and the CIs "do not include training-seed variation" (Table 4). The ribbons in Figure 3 show variation across sample indices, not across runs. At 8B, all four per-benchmark CIs include zero (Table 9), and the 8B aggregate interval is not reported. The ~1.3-point lead over PW-OPSD at 14B (Table 2) has no uncertainty estimate.
  2. The value of IG over plain reference NLL is not tested. The reference is both teacher input ("Here is a reference solution: {y_star}", Appendix B) and scoring target. If the teacher's score on the reference varies little across examples, IG reduces to roughly the student's reference NLL, i.e., difficulty weighting. The decomposition in Section 5.5 does not resolve this, because it is computed offline "on the pooled competition evaluation sets" (Figure 7) rather than on training batches.
  3. Implementation details are insufficient for reproduction. Algorithm 1 does not show gradient accumulation, so it is unclear whether the normalization group B_g contains 4 examples or the full 32 (Table 12). The token loss is not fully specified: it is unclear whether teacher probabilities are renormalized after top-k truncation, and the 10⁻⁷ clip is described in a single sentence (Table 13). Baseline configurations are incomplete; at 14B, SFT (35.1) and Purified-OPSD (36.1) are close to the untrained model (35.6), which deserves a sanity check. Deduplication between the training data and the evaluation sets is not mentioned.

Minor Weaknesses:

  1. At 1.7B, the per-benchmark results go in mixed directions (AIME24 −1.7, AMC23 +2.1). The paper offers only a one-sentence post hoc explanation.
  2. The abstract's "consistently outperforms" should be scoped to the reported single-run results.
Questions for authors
  1. Report mean ± std over at least 3 training seeds for OPSD and IG-OPSD at 14B, and the 8B four-benchmark aggregate CI.
  2. Add a student-reference-NLL weighting control under the same gating rule and budget. If IG clearly wins, my score could go up; if NLL explains most of the gain, an accurately positioned and robust simple method is still a valid contribution.
  3. Clarify B_g relative to gradient accumulation, and give the exact token-loss formula (top-k renormalization, clipping).
  4. Provide full baseline configurations and the train/eval deduplication check.
  5. (Optional, does not affect my score.) PACED (Xu et al., arXiv:2603.11178) also reweights problems in (self-)distillation, using the pass rate with w = p(1−p). It may be a useful point of comparison given the hard-problem gains reported in Section 5.6.
3 comments
AuthorsOpenReview ↗

Response to Reviewer K6Fh

Thank you for the detailed suggestions. We added the requested training-seed comparison, NLL control, confidence intervals, and overlap check.

1. Three training seeds and confidence intervals

The original 14B comparison used top-256 loss support for OPSD and full vocabulary for IG-OPSD. We reran both methods with full vocabulary for seeds 42, 43, and 44, keeping the other settings identical. These results replace the original comparison.

The table reports the equal-weighted four-benchmark Avg@12 mean. ± denotes the sample standard deviation across training seeds.

MethodSeed 42Seed 43Seed 44Mean ± std
OPSD50.3051.1350.1450.52 ± 0.53
IG-OPSD52.8350.5951.5151.64 ± 1.13
IG − OPSD+2.53−0.54+1.37+1.12 ± 1.55

On HMMT25, IG-OPSD improves all three seeds by +3.61, +4.17, and +2.22 points, the largest or joint-largest gain among the four benchmarks in each seed.

At 8B, the four-benchmark mean increases from 39.57 to 41.51 (+1.94 points), with a paired problem-level bootstrap 95% CI of [−0.52, +4.65].

For the Table 2 single-run 14B comparison with PW-OPSD, the mean difference is +1.30 points, with 95% CI [−0.76, +3.42]. This compares the reported configurations: top-256 support for PW-OPSD and full vocabulary for IG-OPSD.

Both intervals use 100,000 paired bootstrap replicates, resampling within each benchmark and averaging the four benchmark means equally.

2. Student-reference-NLL control

We replaced the IG score with the student's length-normalized reference NLL, retaining the same gate, normalization, schedule, full-vocabulary objective, 100-step budget, and seed 42.

BenchmarkIG-OPSDStudent-NLLIG − NLL
AIME2452.5049.17+3.33
AIME2543.8941.11+2.78
AMC2389.3887.29+2.08
HMMT2525.5621.39+4.17
Four-benchmark mean52.8349.74+3.09

IG-OPSD outperforms student-NLL weighting on all four benchmarks, with a +3.09-point mean gain. This matched run supports the value of the teacher-conditioned score beyond student-reference difficulty. We will add this control to the main ablation table.

3. Normalization group and token loss

At each 14B microstep, four workers each contribute one example, so B_g contains 4 examples. The normalization reference includes these four scores plus up to 256 recent scores. Eight microsteps form the optimizer batch of 32.

For top-k runs, teacher and student logits are separately normalized with log_softmax over the same teacher-selected support. With beta = 0, the token loss is:

loss(i,t) = sum over v in S(i,t) of min[p_T(v) * (log p_T(v) - log p_S(v)), 1e-7]

Here, p_T and p_S are the normalized probabilities on S(i,t). Clipping applies to each vocabulary-component contribution before summation. The new three-seed runs use the full vocabulary for both methods. We will make these details explicit in Algorithm 1.

4. Baseline configurations and overlap check

The 14B SFT and distillation runs use the same 29,434-example training split, 100 steps, optimizer batch 32, learning rate 5e-6, a linear schedule without warmup, and LoRA rank 64/alpha 128/dropout 0.05.

MethodObjective / allocationLoss support
OPSD, originalUniform example weightsTeacher top-256
OPSD, new three-seed runsUniform example weightsFull vocabulary
IG-OPSDReference-gap z-score + ReLU; history 256Full vocabulary
PW-OPSDPosition sigmoid: floor 0.25, threshold 0.30, scale 0.10Teacher top-256 launch setting
Purified-OPSD, Table 2PMI target; correction beta 1, saturation scale 10; symmetric JSDFull vocabulary
SFTReference-solution cross-entropyFull vocabulary

OPSD and PW-OPSD use beta = 0 and component clip 1e-7. Purified-OPSD uses beta = 0.5 without component clipping.

Distillation rollouts use temperature 1.1, top-p 0.95, top-k 20, and a 1024-token completion limit. Evaluation uses non-thinking mode, temperature 1.0, top-p 0.8, an 8192-token limit, and 12 completions per problem.

For deduplication, we applied Unicode NFKC, lowercasing, removal of LaTeX \left/\right, and whitespace removal. Comparing all training examples against the four evaluation sets found zero normalized exact overlaps.

5. Minor comments

At 1.7B, the four-benchmark mean changes from 20.12 to 20.43, with mixed benchmark-level results. We will describe this pattern separately from the larger-model gains and scope “consistently outperforms” to the reported runs.

We will cite PACED as related work. PACED uses pass-rate weighting p(1−p), whereas our score uses reference likelihoods from a single rollout.

Thanks for the additional experiments. The NLL control and HMMT25 results are helpful, although the matched three-seed gain is smaller and less consistent than the original result.

One clarification: does the implementation really apply min(p_T * (log p_T − log p_S), 1e−7) to each vocabulary component? This caps positive terms but keeps negative ones. Is this also used in both methods and the NLL control?

I'll keep my current rating pending this clarification.

AuthorsOpenReview ↗

Clarification on Loss Clipping and Three-Seed Results

Thank you for checking this carefully.

Three-seed results and performance on harder problems

Your observation is correct: the matched three-seed results show a smaller and less consistent improvement in the four-benchmark Avg@12 mean. Nevertheless, on HMMT25, the hardest benchmark in our evaluation, IG-OPSD improves over OPSD in all three seeds (+3.61, +4.17, and +2.22 percentage points), with the largest or joint-largest gain among the four benchmarks in each seed. These results provide some evidence of the method’s potential on harder problems.

Clipping operation and consistency across methods

Regarding clipping, yes to both questions. For each token position (i, t), the implementation computes:

c_v = p_T(v) * (log p_T(v) - log p_S(v))
loss(i, t) = sum over v in S(i, t) of min(c_v, 1e-7)

The upper clipping is applied before summing over vocabulary: positive contributions above 1e-7 are capped, while negative contributions remain unchanged.

OPSD, IG-OPSD, and the student-reference-NLL control all use the same loss routine and clipping threshold, including the matched full-vocabulary three-seed OPSD/IG-OPSD runs. The NLL control replaces the example-weighting score while retaining this loss computation.

Objective properties and clarification in the paper

Your observation also identifies an important distinction: after this asymmetric clipping, the objective is no longer a standard KL divergence and need not be nonnegative. We will make this explicit and replace the imprecise description “per-token divergence clipping” with “component-wise upper clipping before vocabulary summation.” Our empirical conclusions are scoped to example weighting under this shared clipped objective.

Reviewer_qGpBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper comes up with a weighting system for on-policy self-distillation. For the weighting, they used a measure called information gain, which measures how likely the privileged information is under the teacher vs the student. More specifically, they first measure the average log probability of reference answer tokens when the teacher model is conditioned on the questions, reference answers, and student answers. Then they measure the average log probability of reference answer tokens when the student model is conditioned on the question and its answer. Then they normalize the information gains and move negative ones to zero to get sample weights.

In their experiments, they trained Qwen3 1.7B, Qwen3 8B, Qwen3 14B, and in all of them, they showed that their method has a better pass@12 and avg@12 compared to OPSD in 4 math tasks.

Critical strengths
  1. The idea is novel, and sample weighting seems like a promising direction.
  2. The writing is clear and easy to understand.
Critical weaknesses
  1. All main models are Qwen3, use non-thinking mode. Because the task is math, the non-thinking mode seems not to be fit for the use case.
  2. Models are trained with 100 step of LoRA, with a batch size of 32 means only 3.2k training samples out of ~30k training dataset size. With the upward trend in the figure 3, I think the models are not yet fully trained, and the accuracy will go up by just training more.
  3. Trainings are done only once, and with the small gaps between the methods, it is unclear that gains are random or consistent.
Questions for authors
  1. The interpretation of the likelihood gap as “information gain” is unclear because the teacher has already seen the exact reference solution that it is subsequently scored on. After a few reference tokens, the teacher may recognize the continuation as a repetition and copy the remaining text, so the score may largely reflect copying rather than useful reasoning guidance. Could the authors plot the average token log probability against token position within the appended reference, with separate curves for the teacher and student? I want to see if the teacher’s curve approaches zero after the first few tokens, or not. If visual rebutal is not supported, a similar table would work.
4 comments
AuthorsOpenReview ↗

Response to Reviewer qGpB

Thank you for the specific questions about training robustness and the reference-likelihood score.

1.Training budget, inference mode, and robustness

OPSD and IG-OPSD use the same 100 optimizer steps and rollout budget. At 8B, IG-OPSD separates from OPSD after step 50 and finishes higher on all four benchmarks. This comparison measures the benefit of allocation within the shared training budget. Both teacher and student use non-thinking mode, with the same generation-length limits and evaluation settings across methods.

We added three paired training seeds for the 14B OPSD–IG-OPSD comparison, using identical full-vocabulary loss support. Across AIME24, AIME25, AMC23, and HMMT25, the equal-weighted mean Avg@12 is 50.52 ± 0.53 for OPSD and 51.64 ± 1.13 for IG-OPSD. HMMT25 improves in all three seeds, by +3.61, +4.17, and +2.22 points. The ± values are standard deviations across training seeds; the per-seed results will be included in the revision.

2.Does the teacher simply copy the appended reference?

The table below reports the requested teacher/student reference-token log-probabilities by position. We rescored 40 saved Qwen3-14B training rollouts at checkpoint 100, using the same rollout and reference for both models. Here, IG denotes the measured reference log-likelihood gap.

Reference positionTeacher mean log pStudent mean log pT−S gap
0–10%−0.829−1.887+1.058
10–20%−0.143−0.849+0.706
20–30%−0.088−0.671+0.583
30–40%−0.055−0.477+0.422
40–50%−0.041−0.457+0.416
50–60%−0.037−0.388+0.350
60–70%−0.053−0.506+0.453
70–80%−0.070−0.511+0.440
80–90%−0.076−0.529+0.453
90–100%−0.084−0.584+0.500

The teacher becomes highly confident after the beginning. The gap remains positive in every decile and is 0.35–0.50 nats/token in the second half. These measurements characterize reference predictability with and without privileged context.

Our matched student-NLL control separately tests the score’s usefulness: IG-OPSD exceeds NLL weighting by 3.09 points on the four-benchmark mean at 14B, with higher scores on all four benchmarks. The detailed results appear in our response to K6Fh. We will add the position curve, decile table, and NLL control to the revision.


I want to thank authors for their response. The following issues remain for me.

Weaknesses:

  1. I still think non-thinking mode evaluation of models in a math setup isn't the correct way to evaluate them.
  2. Although the setup is shared between baselines, when the accuracies seem to go up near the end of training, I think the model is not fully trained; hence, comparing them isn't as informative as it could be. This is not a fairness issue, as with the shared setups the comparison is fair.
  3. Thanks for the seed experiments; it partially addressed my concern. However, the intervals of 50.5 ± 0.5 and 51.6 ± 1.1 still share the 50.8-51 interval, which means the improvement might not be significant.

Questions: Thanks for the experiment. It addressed my question.

AuthorsOpenReview ↗

Further clarification: what the allocation controls establish

Thank you for raising the reference-copying concern. We agree that high teacher confidence on a previously seen reference does not, by itself, demonstrate reasoning benefit. Our positional analysis characterizes reference predictability; it does not distinguish copying from reasoning guidance.

Our empirical contribution concerns whether this likelihood-based score is useful for allocating supervision. The two controls address complementary aspects of that question:

  • Student-reference-NLL weighting tests whether including the teacher-conditioned term improves on student difficulty alone, under the matched settings reported in our previous response.
  • Shuffled weights preserve the weight values while breaking their assignment to examples, testing whether the score–example correspondence matters.

Under the reported configurations, IG-OPSD outperforms shuffled weights on the three-benchmark Avg@12 aggregate:

ModelAdvantage over shuffled weightsPaired problem-bootstrap 95% CI
Qwen3-8B+3.98 percentage points[+1.02, +7.22]
Qwen3-14B+2.96 percentage points[+0.56, +5.56]

Together, these controls support a concrete empirical finding: the proposed likelihood-gap score can improve supervision allocation in the tested settings, and assigning weights to their corresponding examples matters.

We will clarify this empirical scope in the revision and define the score operationally as a privileged-context reference-likelihood gap. The allocation controls provide evidence of practical usefulness without establishing that the score directly measures reasoning benefit.

AuthorsOpenReview ↗

Follow-up on Thinking Evaluation and Training Robustness

We apologize that, due to limited time and computational resources, we could not complete all the additional experiments we had planned. We have done our best to address your concerns with further experiments, and summarize the results below.

1. Thinking-mode evaluation

We evaluated the existing 8B checkpoints on HMMT25 with thinking enabled:

Avg@12 (%)OPSDIG-OPSDImprovement
HMMT2545.2748.33+3.06 points

This shows that the benefit on HMMT25 also appears under thinking-enabled inference, directly extending our original non-thinking evaluation.

2. Training duration

You are right that equal budgets do not answer whether the models are fully trained. We checked the existing extended-training trajectories and found mixed results across later checkpoints. We will include these trajectories and describe the main finding as improved supervision allocation within the tested training budget, rather than a higher final performance ceiling.

3. Training-seed variability

The reported ± values denote standard deviations across training seeds. With matched settings, the paired four-benchmark mean gain is +1.12 ± 1.55 points. The aggregate benefit varies across runs. On HMMT25, the improvement appears in all three seeds; Δ denotes IG-OPSD minus OPSD in percentage points:

HMMT25 Avg@12 (%)Seed 42Seed 43Seed 44
OPSD21.9420.0022.50
IG-OPSD25.5624.1724.72
Δ+3.61+4.17+2.22

The new thinking result and the repeated HMMT25 gains strengthen the evidence for the method on this challenging benchmark. We will make this finding, alongside the variability across settings, clearer in the revision. We would appreciate your reconsideration in light of these additions.

Reviewer_2aFcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the on-policy self-distillation setup and examines the importance of different training instances (e.g., if the student can already solve an instance with no problem, the intuition is that this instance should have a less weighting than other instances where the student cannot solve the problem by itself and the teacher provides benefits)---namely, whether training should weigh problems where the privileged teacher has a larger measured advantage heavier rather than considering all instances equal. Experimental setup uses Qwen3-1.7B, 8B, and 14B with thinking disabled throughout and evaluates primarily on AIME25 and HMMT25. IG-OPSD sees a few percentage points' worth of gains.

Critical strengths
  • Rigorous experimental setup.
Critical weaknesses
  • (Major) What is the surprising finding in this paper? Clearly, putting more weighting on instances that the student can learn from will improve training fidelity, and the paper mostly observes 1~2 percent performance gains where the performance numbers are already far off from what is expected of the original models (e.g., Qwen3-1.7B with thinking scores ~40% on AIME).
Questions for authors
  • Please address W1. Perhaps I am not understanding a major nuance in the paper, but the findings seem trivial to me and the improvements are not significant enough to even warrant this reweighing helps.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2aFc

1. For W1: Are the gains substantial and reproducible?

We added three matched Qwen3-14B training runs. IG-OPSD improves HMMT25 Avg@12 by 2.22–4.17 percentage points in every seed. Both methods use non-thinking mode for teacher and student, full-vocabulary loss support, and the same training-step and rollout budgets. Avg@12 averages correctness over 12 completions per problem; Δ denotes IG-OPSD minus OPSD.

HMMT25 Avg@12 (%)Seed 42Seed 43Seed 44
OPSD21.9420.0022.50
IG-OPSD25.5624.1724.72
Δ (percentage points)+3.61+4.17+2.22

Existing OPSD extensions provide context for this magnitude: PW-OPSD reports +1.0 and +1.1 Avg@12 points on AIME24 and AIME25 for Qwen3-4B, while PAINT reports a +2.1-point macro Avg@12 gain over OPSD at 8B, in their respective evaluation settings. Thus, a few percentage points over an already-trained OPSD baseline represent a meaningful improvement.

Our Table 2 also reports HMMT25 scores of 23.1 for PW-OPSD and 25.6 for IG-OPSD in the single-run comparison. The reported training-time increase is approximately 4.3% at 14B, with no increase in the rollout budget.

2.What does the paper establish beyond the intuition that some examples deserve more supervision?

The contribution is an online allocation rule with a repeatable benefit on difficult problems. That intuition leaves open how to identify useful examples as the student changes. IG-OPSD uses the teacher–student reference-likelihood gap to weight examples while retaining the rollout procedure and token-level objective.

The added control also tests whether student difficulty alone explains the benefit. In the matched 14B seed-42 comparison, IG-OPSD exceeds student-reference-NLL weighting on all four benchmarks, with a +3.09-point mean gain. This supports the value of the teacher-conditioned score beyond a student-only difficulty signal; the full results appear in our response to K6Fh.

The shuffled-weight variant randomly reassigns example weights, breaking the score–example assignment. Under the reported run configurations, IG-OPSD exceeds this variant on the three-benchmark aggregate by +3.98 points at 8B (paired problem-level bootstrap 95% CI [+1.02, +7.22]) and +2.96 points at 14B ([+0.56, +5.56]). These comparisons further motivate studying supervision allocation at the example level.

3.Comparison with thinking-mode scores

The cited Qwen3-1.7B score is obtained with thinking enabled. Our teacher and student both use non-thinking mode, with the same generation-length limits and evaluation settings across methods. Our results measure the improvement over OPSD within this shared protocol. We will make the inference mode explicit beside the results.


AuthorsOpenReview ↗

Further Clarification of W1: Student Difficulty versus Teacher Supervision Value

Thank you for your comments on W1. We would like to clarify what the additional controls tell us about supervision allocation. The practical question is how to identify which examples benefit more from teacher supervision as the student changes during on-policy training.

Distinguishing difficulty from supervision value

Student-reference NLL measures the student’s difficulty with an example, but does not directly measure the benefit of privileged teacher supervision. Our added NLL control tests whether including teacher-conditioned information makes a difference. In the matched seed-42 comparison, replacing the teacher–student reference-likelihood gap with student NLL, while retaining the same weighting procedure and training settings, lowers performance on all four benchmarks. IG-OPSD achieves a +3.09-percentage-point advantage in mean Avg@12.

The shuffled-weight control addresses a separate question: does it matter which example receives each weight? It preserves the set of weights and breaks their assignment to examples. IG-OPSD outperforms this control at both 8B and 14B, supporting the importance of the score–example pairing.

The specific empirical insight

Together, these controls provide evidence that the teacher–student reference-likelihood gap carries useful information for allocating supervision beyond student difficulty alone. This gives a concrete basis for choosing example weights during training, using reference likelihoods without additional rollouts. The gains on HMMT25 across all three matched training seeds further suggest potential on harder problems.

We will make this finding more explicit in the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New Summary of Additional Experiments and Request for Reconsideration

We thank the reviewers for their time and feedback. We have addressed the main questions through additional experiments and implementation clarifications.

1. Responses to Reviewer Questions

Our responses now include three matched training seeds, a student-reference-NLL weighting control, reference-token likelihood analysis by position, and confidence intervals. We also provide the normalization procedure, exact loss definition, baseline configurations, and a normalized exact-match check that found no train–evaluation overlaps.

2. A Simple Method with Meaningful Gains

IG-OPSD changes only the supervision weight assigned to each example. It requires no additional rollouts and adds approximately 4.3% to training time.

In the matched student-NLL weighting comparison, IG-OPSD improves mean Avg@12 by 3.09 percentage points, with higher scores on all four benchmarks. This comparison shows that including teacher-conditioned information can improve on weighting by student difficulty alone.

14B, seed 42 — Avg@12 (%)Student-NLL weightingIG-OPSDGain (percentage points)
AIME2449.1752.50+3.33
AIME2541.1143.89+2.78
AMC2387.2989.38+2.08
HMMT2521.3925.56+4.17
Four-benchmark mean49.7452.83+3.09

3. Gains on Challenging Problems

HMMT25 has the lowest OPSD accuracy among the four benchmarks. It also benefits most from IG-OPSD: Avg@12 improves by 3.61, 4.17, and 2.22 percentage points across three matched training seeds, the largest or joint-largest gain in each run.

14B, HMMT25 Avg@12 (%)Seed 42Seed 43Seed 44
OPSD21.9420.0022.50
IG-OPSD25.5624.1724.72
Gain (percentage points)+3.61+4.17+2.22

We see this as the paper’s most interesting result. Alongside the difficulty-stratified analysis, it suggests that allocating teacher supervision is especially useful for problems the student finds difficult.

Additional thinking-enabled evaluation of the existing 8B checkpoints:

ModelBenchmarkEvaluation modeOPSD Avg@12 (%)IG-OPSD Avg@12 (%)Gain (percentage points)
8BHMMT25Thinking45.2748.33+3.06

4. Future Work and Applications

The results on difficult problems motivate us to test IG-OPSD on harder datasets and complex enterprise tasks. We want to examine whether the same allocation rule can improve performance in these settings while keeping the additional training cost low.

We respectfully ask the reviewers and Area Chair to reconsider our paper in light of the additional experimental evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.