acceptodds
Under review as a conference paper at ICLR 2027

-: Taming Entropy Overshoot in On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher’s and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student’s predictive distribution without pulling it back. We introduce - to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student- reachable states. Entropy-aware distillation uses the student–teacher entropy gap to determine the direction and strength of each token’s correction. - improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out- of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, - remains simple, requiring no additional forward passes or networks during distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Self-Distillation, On-Policy Distillation, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BaGpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies on-policy self-distillation (OPSD), where a frozen copy of the student conditioned on the reference solution teaches the student via forward KL on student-generated prefixes. It identifies "entropy overshoot" (student token entropy rising above the teacher's and staying there) as a failure mode and attributes it to (i) answer-conditioned teacher guidance that transfers poorly to student prefixes and (ii) forward KL's one-way mode-covering pressure. E2-OPSD replaces the reference solution in the teacher context with a retrieved solved neighboring problem (EGT) and routes each token to forward or reverse KL by the sign of the student-teacher entropy gap, weighted by its normalized magnitude (BER + AGD). On four backbones (Qwen3-0.6B/1.7B thinking, Qwen3-4B-Instruct, Olmo-3-7B-Instruct) trained 360 steps on DeepMath, it reports math mean@16 gains of +0.6 to +4.3 over OPSD and OOD gains over Base on ZebraLogic/HumanEval, plus ablations and mechanism probes (teacher-context KL, gradient-share by entropy bucket, KL-direction dynamics).

Critical strengths
  1. The diagnosis is concrete and the fix is cheap. The entropy-overshoot phenomenon is shown directly (Figures 1, 4, 6), the teacher-context probe (Table 13: 25.9% lower forward KL on student prefixes, teacher own-prefix length 836 vs student 9,140 tokens) gives a plausible mechanism for why answer-conditioned teachers mismatch student states, and the method adds no forward passes or networks. Figure 6 (FKL vs RKL vs fixed mixture vs BER under a fixed teacher) is a clean, controlled demonstration that state-dependent direction routing behaves differently from any fixed divergence.

  2. The ablation is additive and consistent on Qwen3-1.7B (Table 11): EGT +1.4, BER +3.0, AGD +0.8, combined +4.3 over OPSD, with per-benchmark gains on all four math sets and small evaluation-seed variance. On this model the result is convincing.

Critical weaknesses
  1. All baselines are essentially flat relative to Base, which makes the comparison uninformative. Across the four backbones, OPSD, SDPO, RLSD and OPSD+EOPD move average math mean@16 by at most +0.8 and often by -0.1 to +0.5 (Table 1), whereas their original papers report substantial gains. Under the shared protocol (lr 1e-6, 360 steps, frozen teacher), the competitors appear not to be learning at all, so "beating OPSD" reduces to "beating Base." Either the budget/hyperparameters are unsuited to the baselines or the implementations are off; in either case the headline "up to 4.3 over OPSD" is not evidence that E2-OPSD outperforms properly-tuned self-distillation. The OOD tables (Tables 2, 10) compound this by comparing only to Base, with no baseline rows.

  2. Gains are robust on only one of four backbones, from a single training run each. On Qwen3-0.6B (+0.6 avg over OPSD), Olmo-3-7B (+1.9, with HMMT25 below OPSD and SDPO) and to a lesser extent Qwen3-4B (+2.2, mostly from AIME25), the improvements are within one to two evaluation-seed standard deviations and there are no training-seed repeats. The "consistent across families and modes" claim rests largely on Qwen3-1.7B, which is also the only model used for all ablations and probes. Relatedly, the causal link between entropy overshoot and accuracy loss is asserted rather than established: in Table 1 OPSD does not degrade below Base despite overshooting, and the teacher used as the entropy reference has privileged answer access, so a student exceeding its entropy is not obviously pathological.

A further concern on novelty: the cited "direction-adaptive self-distillation" (Zhang et al., 2026) appears to already switch KL direction based on student/teacher uncertainty, and retrieved-exemplar context for training-time guidance (RA-RFT, XRPO) is also prior work. Neither is compared against, so the incremental contribution of BER and EGT over the closest prior methods is unclear.

Questions for authors

Clarification requests:

  1. Why do OPSD, SDPO and RLSD show near-zero change from Base under your protocol, when their original papers report multi-point gains? Did you tune learning rate or step budget per baseline, and do the baselines improve with a larger budget (e.g., 1,000+ steps) or the learning rates used in their papers?
  2. Please add baseline rows to the OOD tables (ZebraLogic, HumanEval). Gains over Base alone do not show that E2-OPSD transfers better than OPSD/SDPO/RLSD.
  3. How similar are retrieved neighbors to the target problem in DeepMath (which contains many near-duplicate templates)? Please report (a) a leakage check, e.g., fraction of neighbors sharing the final answer or differing only in constants, and (b) a control with a randomly chosen (non-neighbor) solved problem, to separate "remove the current answer" from "provide a transferable strategy."
  4. How does E2-OPSD compare to Zhang et al. (2026) direction-adaptive self-distillation, and what is the distinction between BER and that method?
  5. Can you provide training-seed repeats (at least 2-3 runs) for one small configuration to quantify run-to-run variance relative to the reported gains?

Suggestions for future work: Validating that the method still helps when OPSD itself is run to the point where "gains deteriorate as training proceeds" (longer budgets) would directly support the paper's motivation.

3 comments

We thank the reviewer for the careful and constructive review. All new experiments are in Appendix D of the revised PDF, and new text is shown in blue. Tables R1–R3 are reproduced at the end of this response for your convenience.

Q1. Why do OPSD, SDPO, and RLSD show near-zero change from Base, and do they improve with tuned learning rates or larger budgets?

We thank the reviewer for this important question. Within our fixed compute budget, we carefully tuned each baseline individually, including its learning rate and training steps, and allocated sufficient training budget to all methods. We report the best validation-selected configuration for each baseline. Since our data and experimental protocol differ from those in the original papers, the absolute gains are not directly comparable.

Q2. Please add baseline rows to the OOD tables.

We thank the reviewer for the suggestion. We added GRPO and OPSD baselines and introduced LiveCodeBench v6 to strengthen the OOD evaluation (Table R2; Table 17 in the revised PDF). E²-OPSD achieves the highest three-benchmark average (67.76 vs. 66.60 for OPSD and 65.48 for GRPO), with particularly strong gains on ZebraLogic and LiveCodeBench v6. These results provide stronger evidence that E²-OPSD delivers robust transfer beyond its in-domain advantages, especially on challenging multi-step reasoning tasks.

Q3. Can you provide training-seed repeats to quantify run-to-run variance?

We repeated GRPO, OPSD, and E²-OPSD on Qwen3-4B-Instruct-2507 with two additional training seeds (Table R1, Table 16 in the revised PDF). Across three seeds, E²-OPSD reaches 57.59 ± 0.31, compared with 55.92 ± 0.54 for OPSD and 55.81 ± 0.54 for GRPO. It is best on every seed and on all four benchmarks on average, and its margin over each baseline exceeds the training-seed standard deviation of every method.

Q4. The "consistent across families and modes" claim rests largely on Qwen3-1.7B, which is also the only model used for ablations and probes.

We thank the reviewer for raising this important concern. We substantially expanded the evidence beyond Qwen3-1.7B along two dimensions.

  1. Validation beyond the original setting. The new BER controls, long-horizon dynamics, retrieval ablation, and OOD evaluation use Qwen3-4B-Instruct-2507, while the AGD gradient analysis covers all four original configurations.
  2. Robustness against a strong baseline. We added GRPO and repeated GRPO, OPSD, and E²-OPSD across three training seeds. E²-OPSD achieves 57.59 ± 0.31, versus 55.92 ± 0.54 for OPSD and 55.81 ± 0.54 for GRPO, ranking first on every seed and across all four benchmarks on average.

Together, these results show that both the diagnosed failure and E²-OPSD’s gains are consistent across model scales, families, modes, and random seeds, providing substantially stronger support for our central claims.

Q5. The causal link between overshoot and accuracy loss is asserted rather than established: OPSD does not fall below Base in Table 1, and the teacher has privileged access, so exceeding its entropy is not obviously pathological.

A: We thank the reviewer for this thoughtful question.

  1. Consistent link between overshoot and accuracy degradation (Figures 7 & 9, Tables 18–19): Across all five OPSD runs (0.6B–14B), persistent overshoot consistently precedes late-stage validation decline—every model's post-overshoot accuracy falls below its peak checkpoint (which Table 1's best-checkpoint reporting masks), and on Qwen3-4B, widening the gap to +0.034–0.035 nats coincides with a 3.3–3.4-point drop. Fixing the teacher, seed, and budget while switching only to BER prevents overshoot and raises accuracy from 55.65% to 56.95% (outperforming OPSD at all 12 post-crossing checkpoints), confirming a direct causal link.
  2. Why student overshoot is structurally pathological despite teacher privilege: Although the teacher sees the ground-truth answer, it is conditioned on off-policy student-generated prefixes—a mismatch that already inflates teacher uncertainty relative to its own trajectories. Exceeding even this prefix-inflated teacher entropy therefore signals runaway policy diffusion rather than calibrated exploration.

Q6. How does E²-OPSD compare with Zhang et al. (2026), and what distinguishes BER from that method?

We thank the reviewer for this important question. Although both methods use entropy-aware routing, they address fundamentally different problems.

  1. Failure mode. DASD addresses entropy suppression and diminished exploration, whereas E²-OPSD targets entropy overshoot—the student becoming more uncertain than its teacher.
  2. Teacher relationship. DASD makes the teacher attractive at low-entropy tokens but repulsive at high-entropy tokens. E²-OPSD always preserves the teacher as the target: BER selects between two teacher-attractive KL directions, while AGD controls update strength.
  3. Supervision. DASD is verifier-anchored and applies GRPO-style advantages to sampled tokens. E²-OPSD is reward- and verifier-free, distills the teacher’s full top-KK distribution at every position, and improves its context through EGT.

Thus, E²-OPSD provides a more faithful and broadly applicable solution to OPSD’s intrinsic instability, without repelling the teacher or requiring external verification. We will add this distinction to the related-work section.

Suggestion for future work: validate that the method still helps when OPSD is run until its gains deteriorate.

We thank the reviewer for this insightful suggestion. We extended OPSD to 480 steps and branched the overshooting run to BER at step 240 (Figure 9). BER rapidly eliminates the entropy gap and consistently outperforms continued OPSD, while E²-OPSD prevents the degradation altogether, finishing 2.8–3.6 points higher across two seeds (Table R3). This provides strong direct validation of E²-OPSD’s central design: preventing entropy overshoot is substantially more effective than correcting it afterward.


Table R1. Results over three training seeds (Qwen3-4B-Instruct-2507, math mean@16, %). Seed 1 is the run in Table 1; benchmark columns average the three seeds.

MethodAMC23AIME24AIME25HMMT25Seed 1Seed 2 (new)Seed 3 (new)Mean ± Std.
Base85.3961.7446.4629.03———55.65
GRPO84.9961.6547.4129.1955.1956.1856.0555.81 ± 0.54
OPSD85.5361.9046.7629.4955.5855.6556.5455.92 ± 0.54
E²-OPSD86.8662.6650.4630.3957.7857.2357.7757.59 ± 0.31

Table R2. Out-of-domain results (Qwen3-4B-Instruct-2507, training seed 2, mean@16, %). ± is over three evaluation seeds.

MethodZebraLogicHumanEvalLiveCodeBench v6Avg.
Base78.98 ± 0.8484.04 ± 0.0933.27 ± 0.3065.43
GRPO79.27 ± 0.1283.40 ± 0.0833.77 ± 0.1765.48
OPSD79.85 ± 0.5286.12 ± 0.2733.83 ± 0.3066.60
E²-OPSD82.65 ± 0.1185.10 ± 0.2535.55 ± 0.2667.76

Table R3. Training dynamics over 480 steps (Qwen3-4B-Instruct-2507, validation mean@8, %; gap = student − teacher entropy, nats).

Method (seed)Gap > 0 fromBest (step)Step 480ChangeGap at bestGap at 480
OPSD (2)step 12856.7 (240)53.4−3.3+0.019+0.034
OPSD (3)step 12857.2 (184)53.8−3.4+0.013+0.035
BER only (2)never57.4 (184)54.7−2.7−0.015−0.014
E²-OPSD (2)never58.8 (304)56.2−2.6−0.008−0.008
E²-OPSD (3)never58.2 (240)57.3−0.9−0.009−0.008
AuthorsOpenReview ↗

Looking for your response

Dear Reviewer,

I hope this message finds you well. We sincerely appreciate your valuable feedback. We have carefully addressed all your comments point by point and made substantial revisions to the manuscript with significant effort.

As the discussion period is nearing its end with less than three days remaining, we want to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_UhxBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes E^2-OPSD to address entropy overshoot in on-policy self-distillation (OPSD), where the student’s prediction entropy increases beyond the teacher’s during training. The method consists of two major components. The first is exemplar guided teaching (EGT), which replaces the ground truth solution of the current problem with the solution trace of a similar instance retrieved from the training dataset as the teacher’s context. The second is entropy aware distillation (EAD), which uses the entropy difference between the student’s and teacher’s prediction distributions to dynamically select between forward KL and reverse KL and adjust the distillation loss weight for each token. The paper evaluates the proposed method on mathematical reasoning, logical reasoning, and code generation tasks across different model sizes.

Critical strengths

The proposed method is relatively simple and does not require additional forward passes or models during distillation.

Critical weaknesses

The proposed EAD uses the entropy difference between the student and teacher distributions to determine the KL direction and token weights. However, two distributions can have nearly identical entropy while assigning high probabilities to completely different tokens. In such cases, AGD may suppress important distillation signals despite substantial distributional disagreement. Therefore, entropy difference alone may not be a reliable metric for guiding distillation.

EGT uses ground truth solution traces from similar training instances as the teacher's context. However, semantic similarity between problems does not necessarily imply similarity in their underlying reasoning strategies. The effectiveness of EGT may therefore depend heavily on the quality of retrieved examples, but the paper lacks sufficient analysis of how retrieval quality affects performance.

Questions for authors

Can the authors provide additional evidence that entropy overshoot directly contributes to reasoning performance degradation? For example, is there a consistent correlation between the magnitude of entropy overshoot and downstream performance across different training stages and models?

What evidence supports the assumption that switching to reverse KL when student entropy exceeds teacher entropy will reduce the entropy gap? Could the authors analyze whether the selected KL direction consistently produces the intended entropy changes during optimization?

5 comments

We thank the reviewer for the detailed questions. All new experiments are in Appendix D of the revised PDF, and new text is shown in blue. For convenience, Tables 20, 15, 21, and 18 of the revised PDF are reproduced below as Tables R1–R4, respectively.

Q1. Two distributions can have nearly identical entropy while assigning high probability to different tokens, so AGD may suppress important signals.

We thank the reviewer for raising this insightful point. While theoretically possible, our empirical analyses demonstrate that this edge case is negligible in practice and that AGD provides clear, selective gains. Across all four model configurations, tokens strongly damped by AGD (ct≤0.1c_t \le 0.1) despite high teacher–student disagreement (forward KL ≥1\ge 1 nat) account for only 0.51% of the total pre-AGD logit-gradient norm on average (range 0.24–0.73%, Table 20 in Appendix D.4 of the revised PDF and Table R1 below). Consistently, strongly damped positions exhibit a 0.985 mean teacher–student probability-mass overlap (Table 15 in Appendix C.2 of the revised PDF and Table R2 below), confirming that near-zero entropy gaps overwhelmingly reflect genuine distributional agreement rather than hidden conflict.

Q2. Semantic similarity does not imply similar reasoning strategies, and the paper lacks analysis of how retrieval quality affects performance.

We thank the reviewer for this important question. We compare nearest-neighbor retrieval with uniformly sampled exemplars under identical settings (Table 21 in Appendix D.5 of the revised PDF and Table R3 below).

Both exemplar variants substantially outperform OPSD, showing that EGT is robust across retrieval choices rather than relying on a specific similarity heuristic. Similarity-aware retrieval achieves the best overall result, confirming that retrieval quality provides a complementary benefit.

Q3. Is there evidence that entropy overshoot directly contributes to reasoning degradation, for example through a consistent correlation across training stages and models?

A: We thank the reviewer for this important question. Appendix D provides consistent cross-model, temporal, and intervention evidence that overshoot is a cumulative failure mode rather than a single-step threshold.

  • Cross-model consistency. All five OPSD configurations spanning 0.6B–14B exhibit persistent overshoot; Phi-4-reasoning reaches a +0.35-nat gap (Figure 7).
  • Cumulative degradation. Although crossing occurs at different stages across models, every run’s latest post-overshoot accuracy falls below its best checkpoint. On Qwen3-4B across two seeds, widening the gap to +0.034–0.035 coincides with 3.3–3.4-point declines (Figure 8 and Table 18 in the revised PDF, with Table 18 reproduced as Table R4 below).

Together, these results establish overshoot as an operative contributor to OPSD degradation—not an incidental correlation.

Q4. What evidence supports that switching to reverse KL when student entropy exceeds teacher entropy reduces the gap?

A: We thank the reviewer for this question. Direct interventional evidence in Figure 9 (Appendix D.3) confirms that BER consistently produces its intended directional correction: when we branch an already-overshooting Qwen3-4B-Instruct-2507 OPSD run at step 240 and switch to BER, reverse-KL routing rapidly reverses the runaway diffusion—closing the positive entropy gap within ~60 steps and keeping student entropy strictly below the teacher's through step 480, whereas continued OPSD further widens the overshoot.

Q5: Does the selected direction consistently produce the intended change?

We thank the reviewer for this question. Yes—Figure 4 shows that bidirectional routing consistently achieves its intended self-balancing effect over long-horizon training: without any manual schedule, routing naturally shifts from forward-KL exploration early on to increased reverse-KL concentration later (Figure 4c), keeping student entropy tightly regulated just below the teacher's throughout training (Figures 4a–b) and preventing OPSD's runaway overshoot for stable reasoning gains.

Table R1. Gradient share of strongly damped tokens with high disagreement (Table 20 in the revised PDF). We report the share of summed pre-AGD per-token BER logit-gradient norms attributable to tokens with ct≤0.1c_t \le 0.1 and forward KL ≥1\ge 1 nat. The mean weights all four model configurations equally.

MetricQwen3-0.6BQwen3-1.7BQwen3-4BOlmo-3-7BMean
Gradient share (%)0.240.400.670.730.51

Table R2. Teacher–student probability-mass overlap at strongly damped positions (Table 15 in the revised PDF). Percentages are computed within each row.

SubsetPositionsMean overlapOt<0.5O_t < 0.5 (%)
All strongly damped positions12,708,9620.98480.380
Strongly damped positions with H(pt)≥ln⁡2H(p_t) \ge \ln 2766,6820.83433.662

Table R3. Exemplar retrieval in EGT (Table 21 in the revised PDF). Results use training seed 1 and are reported as math mean@16 (%).

Teacher contextAMC23AIME24AIME25HMMT25Avg.
Reference solution (OPSD)84.8162.3645.6229.5155.58
Random exemplars86.6462.6449.1730.0057.11
Nearest-neighbor exemplars (E²-OPSD)87.1062.8550.9730.2157.78

Table R4. Training dynamics over 480 steps on Qwen3-4B-Instruct-2507 (Table 18 in the revised PDF). Validation scores are mean@8 (%). The entropy gap is student minus teacher entropy in nats. Change is the step-480 score minus the best validation score, in percentage points.

Method (seed)Gap > 0 fromBest (step)Step 480ChangeGap at bestGap at 480
OPSD (2)Step 12856.7 (240)53.4−3.3+0.019+0.034
OPSD (3)Step 12857.2 (184)53.8−3.4+0.013+0.035
BER only (2)Never57.4 (184)54.7−2.7−0.015−0.014
E²-OPSD (2)Never58.8 (304)56.2−2.6−0.008−0.008
E²-OPSD (3)Never58.2 (240)57.3−0.9−0.009−0.008
AuthorsOpenReview ↗

Looking for your response

Dear Reviewer,

I hope this message finds you well. We sincerely appreciate your valuable feedback. We have carefully addressed all your comments point by point and made substantial revisions to the manuscript with significant effort.

As the discussion period is nearing its end with less than three days remaining, we want to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_AnLbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces E²-OPSD, which is a on-policy self-distillation framework that proposes an adaptive recipe for on-policy self-distillation: give the teacher a solved related problem, then use the student–teacher entropy gap to choose both the direction and strength of each token’s distillation loss. The goal is to make the teacher’s guidance more useful while preventing the student’s predictions from becoming persistently too diffuse. The paper also identifies entropy overshoot, where the student’s predictions become persistently more uncertain than the teacher’s, and proposes two remedies. First, exemplar-guided teaching gives the frozen self-teacher a retrieved, solved neighboring problem instead of the current problem’s reference solution, aiming to provide transferable reasoning guidance that better matches the student’s intermediate states. Second, entropy-aware distillation uses the token-level student–teacher entropy gap to select forward KL when the student is more concentrated, encouraging coverage of teacher-supported alternatives, and reverse KL when the student is more diffuse, discouraging unsupported probability mass. It also downweights updates where the entropy gap is small. Empirical results in Math reasoning and out of domain evaluations show promising results.

Critical strengths
  1. The paper’s strength is that it connects an observable training failure to a simple intervention and backs that intervention with clear ablations. Its use of the student–teacher entropy gap provides an inexpensive signal for adapting both the direction and strength of token-level distillation as training progresses.

  2. Promising empirical results: the experiments help establish which components matter: on Qwen3-1.7B, changing only the KL routing improves average math mean@16 from 44.64% to 47.61%, adding gap-based weighting raises it to 48.39%, and exemplar guidance brings the full method to 48.89%.

  3. Practically useful: the method fits into the existing OPSD training loop without additional forward passes during distillation

Critical weaknesses
  1. The authors mentioned about the student-teacher entropy gap, but the paper does not sufficiently establish the student–teacher entropy gap as a reliable signal for deciding when and how to distill. I think the authors should elaborate this part in the rebuttal.
Questions for authors

Do you actually see any trend of the student-teacher entropy gaps when you scale the model sizes? Do you foresee entropy as a good metrics to evaluate the distillation quality?

2 comments

We thank the reviewer for the thoughtful questions. All new experiments are in Appendix D of the revised PDF, and new text is shown in blue. For convenience, Tables 20, 15, 21, and 18 of the revised PDF are reproduced below as Tables R1–R4, respectively.

Q1. The paper does not sufficiently establish the student–teacher entropy gap as a reliable signal for deciding when and how to distill.

We tested the two roles of the gap separately (Appendix D.4).

  1. How to distill (BER). With the seed, budget, and teacher fixed on Qwen3-4B-Instruct-2507, adding BER to OPSD boosts math mean@16 from 55.65 to 56.95. Conversely, inverting BER's routing rule (applying reverse KL when the student is more concentrated and forward KL when more diffuse) drops performance to 55.50 (Table 19 in the revised PDF and Table R1 below). Since both variants mix the same KL directions, this 1.45-point margin confirms that the gain stems directly from routing by the sign of the gap.
  2. When to distill (AGD). Across all four model configurations, tokens strongly damped by AGD (ct≤0.1c_t \le 0.1) despite high teacher–student disagreement (forward KL ≥1\ge 1 nat) carry only 0.51% of the pre-AGD BER logit-gradient norm on average, with a range of 0.24–0.73% across models (Table 20 in the revised PDF and Table R2 below), showing that AGD rarely suppresses informative signals. Moreover, replacing AGD with an inverted weighting that emphasizes small-gap tokens degrades peak validation mean@8 by 1.00 point (57.89% vs. 56.89%) and slows convergence (peaking at step 184 vs. step 304), confirming that gap magnitude selectively prioritizes high-leverage updates.

Q2. Do you see any trend in the student–teacher entropy gap as model size grows?

Adding OPSD on Phi-4-reasoning (14B) expands our evaluation to 0.6B–14B parameters across three model families (Figure 7, Appendix D.2). Across all five settings, the same pattern persists: student entropy starts below the teacher's, crosses over during training, and stays higher afterward. Neither the crossover point shrinks monotonically with scale, nor does the overshoot vanish—in fact, the 14B model shows the largest late-stage gap. Thus, entropy overshoot is a universal pathology that scaling alone cannot resolve, highlighting the critical need for our approach.

Q3. Do you foresee entropy as a good metric to evaluate distillation quality? We thank the reviewer for this question. No—higher or lower entropy does not by itself indicate better or worse model quality, so we do not use entropy as an evaluation metric; held-out accuracy remains our measure of quality. Rather than treating high entropy as inherently bad, we specifically target entropy overshoot (Hstudent>HteacherH_{\text{student}} > H_{\text{teacher}})—a failure mode caused by answer-leaking teacher supervision and one-way forward KL continually diffusing the policy. While the crossing step does not rigidly predict when accuracy peaks or drops (e.g., Qwen3-0.6B peaks before it, while Olmo-3-7B-Instruct peaks after it; Figure 7), using the token-level entropy gap as an online control signal prevents this imbalance and yields consistently higher validation accuracy than OPSD across training.

Table R1. BER routing by the sign of the entropy gap (Table 19 in the revised PDF). Qwen3-4B-Instruct-2507, training seed 2, math mean@16 (%). Each run uses its best validation checkpoint. The ± values are computed across three evaluation seeds.

MethodAMC23AIME24AIME25HMMT25Avg.
OPSD85.9261.1145.9029.6555.65 ± 0.29
OPSD + inverted BER85.3260.6246.8129.2455.50 ± 0.43
OPSD + BER86.4263.1248.7529.5156.95 ± 0.33

Table R2. Share of summed pre-AGD per-token BER logit-gradient norms attributable to tokens with ct≤0.1c_t \le 0.1 and forward KL ≥1\ge 1 nat (Table 20 in the revised PDF). The mean weights the four model configurations equally.

MetricQwen3-0.6BQwen3-1.7BQwen3-4BOlmo-3-7BMean
Gradient share (%)0.240.400.670.730.51

Thanks for the authors for the detailed answers to my questions and also appreciate the additional experiments. The responses provide stronger empirical support. I am happy to remain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.