acceptodds
Under review as a conference paper at ICLR 2027

When Better Prediction Makes Worse Generation: Teacher-Domain Alignment in Knowledge Distillation

Abstract

Teacher-domain alignment can reverse whether knowledge distillation improves generation, even when distillation improves held-out prediction. We establish this through a controlled intervention: adapting one GPT-2 teacher on PubMed changes its student's embedding-fidelity effect from harmful to helpful while keeping the corpus, student, tokenizer, optimization, and generation protocol fixed. The adapted-minus-unadapted Fréchet improvement replicates across three training seeds (−0.043 ± 0.006, mean ± SD). Across exactly shared audio, text, and image token spaces, prediction gains coexist with different generation outcomes: distant teachers degrade audio fidelity or increase text repetition, whereas aligned teachers improve fidelity. A web teacher also worsens protein fidelity at every positive weight tested. We introduce Domain-Anchored KD (DA-KD), a supervised training-time gate that retains teacher KL when the observed target token lies in the teacher's top-K. DA-KD improves fidelity over standard KD across GPT-2, Pythia, and SmolLM2, also improves over from-scratch training in the reported text comparisons, and reduces repetition without additional parameters or inference overhead. Aligned and null controls delimit when gating is useful. A two-risk analysis under quadratic surrogates explains how prediction and generation can favor different distillation weights. Experiments fit within a 16 GB GPU budget; code and result summaries are available at https://anonymous.4open.science/r/domain-aligned-distillation-3894/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, autoregressive models, generation fidelity, selective distillation, language modeling, domain shift, cross-modal learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qw61OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a mismatch between teacher-forced prediction and autonomous generation in knowledge distillation. It argues that a teacher can improve held-out likelihood while degrading target-domain generation when the teacher is poorly aligned with the target domain. The authors demonstrate this phenomenon across audio, text, image, and protein settings, including a controlled intervention where adapting the same GPT-2 teacher on PubMed changes the effect of distillation from harmful to beneficial.

The paper proposes Domain-Anchored Knowledge Distillation (DA-KD), which applies the teacher KL loss only when the observed target token is among the teacher’s top-K predictions. Across GPT-2, Pythia, and SmolLM2 experiments, DA-KD improves several generation-oriented metrics relative to dense KD, including primed Fréchet distance, repetition, and selected lexical/domain metrics. The paper also includes aligned, null, and extreme teacher-domain controls, together with a quadratic bias-variance analysis to explain why prediction and generation may prefer different distillation weights.

Critical strengths

The controlled teacher-adaptation experiment is a strong part of the paper. Fixing the corpus, student, tokenizer, optimization, and generation protocol while changing the teacher checkpoint provides more convincing evidence for the role of teacher-domain alignment than comparing unrelated teachers alone.

The paper evaluates multiple modalities and teacher families and includes aligned, null, extreme, and matched-retention controls. The comparisons against random and confidence-based gates are useful for assessing whether the observed-token criterion contributes beyond reducing the amount of KL supervision.

Critical weaknesses

The evidence for degraded generation is metric- and protocol-dependent. The main evaluation uses embedding Fréchet distance, repetition, lexical overlap, and fixed primed continuations, which do not directly establish factual accuracy, long-horizon quality, or downstream utility. In particular, the GPT-2 dense-KD versus scratch Fréchet contrast is unresolved, so the clearest text evidence for degradation comes from repetition rather than from a consistent set of generation metrics.

The scope of DA-KD’s superiority is not fully established. Some audio controls use a single training seed, K=16 is unresolved, the human evaluation is small, and the on-policy comparison covers only a few fixed configurations. The claims should therefore be scoped to the reported models, datasets, prompts, horizons, and decoding protocols.

Questions for authors

Could the authors provide a more systematic sensitivity analysis over K, prompt length, generation horizon, temperature, and decoding strategy for at least one distant teacher and one aligned teacher?

How much of the DA-KD gain comes from the observed-token gate itself versus the induced position-dependent reduction in the effective KL weight? A per-context analysis of retained positions and effective alpha would help clarify this.

Please consider adding the following recent related works to the discussion: “Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning” (arXiv:2511.21075).

The theoretical section uses quadratic surrogate risks and a static mixture target. Can the authors better distinguish which empirical observations are explained by the theorem and which are only qualitative interpretations?

Can the authors provide end-to-end training cost or FLOP comparisons, especially for the on-policy baseline and the gate implementation, rather than only local throughput measurements?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Qw61 (1/5): W1

The global response summarizes the new experiments and revisions; below we address each weakness (W) and question (Q) in your numbering.

Thank you for your constructive comments and your added insight. We went through each detail and revised our manuscript to adhere to your suggestions and your comments. We are open for additional suggestions.

In short.

  • W1 (Protocol & encoder robustness): Across 3 seeds and two decoders, raising KL weight degrades Heuristic F1 and in-domain PubMedBERT Fréchet in 3/3 seeds under both decoders, plus MeSH F1 (−0.009, 3/3), rep-4 (0.017 → 0.046, 3/3), and glued punctuation (32.7% → 60.5%) under default decoding. Teacher adaptation recovers Heuristic F1, MeSH F1, MiniLM and PubMedBERT Fréchet in 3/3 seeds under both decoders.
  • W2 (3-seed Audio & track-level bootstrap): Dense KD lowers perplexity (30.55 → 26.07) and worsens FAD against scratch in 3/3 seeds under both window- and track-level bootstrap. Resolving K=16 and K=4 at 3 seeds shows that as the gate filters more tokens (90.9% kept at K=64 → 75.5% at K=16 → 52.8% at K=4), DA-KD's mean advantage over random masking grows at each step (0.00000 → −0.00041 → −0.00069). At K=4, DA-KD's FAD is at or below the random mask's and below dense KD's in every seed.
  • Q1–Q5 (Sensitivity, gate attribution & FLOPs): Across K in [1, 200] (unadapted and aligned teachers), prompt lengths (12, 32 words), and horizons (75, 300 tokens), KD and DA-KD improve Fréchet over scratch in 3/3 seeds under both decoders at every setting. DA-KD's gate adds 3×10⁻⁵ FLOPs per step, and on-policy GKD's overhead is sequential sampling wall-clock time (4.1× per step), not FLOPs (0.67×).

W1: Metric, encoder, and protocol robustness

"The evidence for degraded generation is metric- and protocol-dependent... the clearest text evidence for degradation comes from repetition..."

Reporting every contrast across three metrics, two encoders (MiniLM and in-domain PubMedBERT), and two decoders (default and tuned nucleus p=0.95) shows that the degradation is not confined to repetition: it appears on Heuristic F1 and on in-domain Fréchet under both decoders:

  • Across both decoders (3/3 seeds): In the controlled KL-weight test (our reply to 6ys9, Q1), higher teacher weight degrades Heuristic F1 (−0.013 default; −0.008 tuned, 3/3) and in-domain PubMedBERT Fréchet (+0.076 ± 0.019 default; +0.100 ± 0.009 tuned, 3/3). Teacher adaptation at high KL weight recovers Heuristic F1 (+0.012; +0.011, 3/3), MeSH biomedical-term F1 (+0.010 ± 0.001; +0.008 ± 0.001, 3/3), MiniLM Fréchet (−0.042; −0.019, 3/3), and PubMedBERT Fréchet (−0.100; −0.115, 3/3).
  • Under default decoding (3/3 seeds): Higher KL weight also degrades rep-4 (0.017 → 0.046 vs. 0.018 real text, 3/3), MeSH F1 (−0.009 ± 0.003, 3/3), and glued punctuation (32.7% → 60.5% vs. 8.3% real text); rep-4 and MeSH F1 degrade in the same direction in 3/3 seeds when only the KL weight is raised inside DA-KD and the random mask.
  • Why rep-4 is read under the default decoder: Real PubMed continuations have rep-4 = 0.018. Default-decoder generations sit at or above that rate (0.016–0.068), whereas nucleus sampling compresses repetition below real text (0.008–0.015), consistent with the sign flip of high-KL KD vs. scratch on rep-4 between decoders (+0.014 default vs. −0.008 nucleus).
AuthorsOpenReview ↗

Response to Reviewer Qw61 (2/5): W2 (audio)

Contrasts: 3-seed mean ± SD, default decoder; tuned decoder. (k/3) = number of seeds whose own 95% bootstrap interval excludes zero.*

W2: Three-seed MAESTRO audio controls (K=64, 16, 4) and track-level bootstrap

"Some audio controls use a single training seed, K=16 is unresolved, the human evaluation is small, and the on-policy comparison covers only a few fixed configurations."

We trained seeds 1 and 2 for scratch, dense KD, and DA-KD along with matched random and anti-confidence masks across K in {64, 16, 4} (1,000 MAESTRO windows, CLAP FAD), and added a 2,000-replicate whole-track bootstrap across the 23 validation tracks:

Audio CLAP FAD Difference ↓ (negative favors first model)Gate RetentionSeeds 0 / 1 / 23-Seed Mean ± SDSeeds Excluding Zero (Window; Track)
Dense KD vs scratch (PPL 30.55 → 26.07)100%+0.00235 / +0.00132 / +0.00133+0.00167 ± 0.000593/3; 3/3
DA-KD K=64 vs dense KD90.9%−0.00082 / +0.00087 / +0.00022+0.00009 ± 0.000852/3 (opp. signs); 2/3 (opp. signs)
DA-KD K=64 vs random mask90.9%−0.00112 / +0.00083 / +0.00029+0.00000 ± 0.001012/3 (opp. signs); 2/3 (opp. signs)
DA-KD K=64 vs anti-confidence90.9%−0.00081 / +0.00022 / +0.00036−0.00008 ± 0.000641/3; 1/3
DA-KD K=16 vs dense KD75.5%−0.00031 / +0.00013 / −0.00090−0.00036 ± 0.000521/3; 1/3
DA-KD K=16 vs random mask75.5%−0.00034 / −0.00020 / −0.00070−0.00041 ± 0.000261/3; 0/3 (lower in every seed)
DA-KD K=16 vs anti-confidence75.5%+0.00000 / −0.00053 / −0.00071−0.00041 ± 0.000372/3; 2/3
DA-KD K=4 vs dense KD52.8%−0.00076 / −0.00026 / −0.00032−0.00045 ± 0.000271/3; 1/3 (lower in every seed)
DA-KD K=4 vs random mask52.8%−0.00132 / −0.00074 / 0.00000−0.00069 ± 0.000662/3; 2/3 (at or below in every seed)
DA-KD K=4 vs anti-confidence52.8%−0.00042 / −0.00074 / +0.00020−0.00032 ± 0.000481/3; 2/3
  1. Prediction–generation divergence on audio is 3/3: Dense KD lowers perplexity (30.55 → 26.07) while worsening FAD against scratch in all three seeds under both window and track-level resampling.
  2. Selection at lower retention (K=16, K=4): At K=64, the gate retains 90.9% of positions and the gates do not separate consistently across seeds. Tightening the gate to K=16 (75.5% kept) and K=4 (52.8% kept) moves the mean FAD difference in DA-KD's favor against both dense KD (+0.00009 → −0.00036 → −0.00045; lower in every seed at K=4) and the matched random mask (0.00000 → −0.00041 → −0.00069; lower in every seed at K=16).
  3. At K=4 (52.8% kept), DA-KD's FAD is at or below the random mask's in every seed (lower in two, both intervals excluding zero; tied in the third) and below dense KD's in every seed (one interval excluding zero).
  4. The human evaluation and the on-policy comparison are retained, with their size and configurations stated as limitations.
AuthorsOpenReview ↗

Response to Reviewer Qw61 (3/5): Q1 (gate width)

Q1: Sensitivity over K, prompt length, horizon, and decoding strategy

"...sensitivity analysis over K, prompt length, generation horizon, temperature, and decoding strategy for at least one distant teacher and one aligned teacher?"

Of the 107 checkpoints evaluated on the 9-setting temperature/nucleus validation grid, 105 selected nucleus p=0.95 (one pretrained DA-KD seed and one aligned-teacher K=50 seed selected a repetition penalty, 1.05 and 1.10). Across every K in [1, 200], prompt length (12, 24, 32 words), and horizon (75, 150, 300 tokens) with both unadapted and 16k-step aligned teachers, distillation improves Fréchet over scratch in 3/3 seeds under both decoders.

K-sweep (3 seeds, unadapted and aligned teachers). Contrasts show DA-KD at each K minus low-weight standard KD (unadapted PPL 66.3; aligned PPL 64.8):

Teacher & Gate KKeptPerplexity ↓MiniLM Fréchet ↓ vs Std KDrep-4 ↓ vs Std KDHeuristic F1 ↑ vs Std KD
Unadapted, K=146%83.4+0.012 ± 0.003 (3/3); +0.009 ± 0.003 (3/3)+0.007 ± 0.004 (3/3); +0.006 ± 0.002 (3/3)−0.000 ± 0.002 (0/3); +0.002 ± 0.001 (1/3)
Unadapted, K=566%74.1+0.003 ± 0.008 (2/3, opp.); +0.002 ± 0.006 (1/3)+0.003 ± 0.003 (2/3); +0.003 ± 0.001 (3/3)+0.000 ± 0.002 (0/3); +0.002 ± 0.002 (1/3)
Unadapted, K=1073%72.4+0.003 ± 0.005 (1/3); +0.004 ± 0.004 (1/3)+0.001 ± 0.002 (2/3); +0.002 ± 0.001 (2/3)+0.002 ± 0.002 (1/3); +0.001 ± 0.003 (1/3)
Unadapted, K=2078%70.5−0.001 ± 0.003 (0/3); −0.001 ± 0.002 (0/3)−0.000 ± 0.001 (0/3); +0.002 ± 0.001 (2/3)+0.001 ± 0.003 (1/3); +0.002 ± 0.001 (0/3)
Unadapted, K=5084%68.8+0.004 ± 0.004 (1/3); +0.001 ± 0.002 (0/3)+0.000 ± 0.001 (0/3); +0.001 ± 0.002 (2/3)−0.000 ± 0.001 (0/3); +0.002 ± 0.001 (1/3)
Unadapted, K=10088%67.8+0.002 ± 0.009 (2/3, opp.); +0.001 ± 0.005 (1/3)+0.000 ± 0.002 (1/3); +0.001 ± 0.001 (1/3)+0.001 ± 0.003 (0/3); +0.003 ± 0.003 (2/3)
Unadapted, K=20091%67.3+0.002 ± 0.005 (0/3); −0.001 ± 0.003 (0/3)−0.000 ± 0.002 (0/3); +0.001 ± 0.001 (0/3)+0.001 ± 0.001 (0/3); +0.002 ± 0.001 (0/3)
Aligned (16k), K=151%85.5+0.016 ± 0.009 (3/3); +0.015 ± 0.004 (3/3)+0.000 ± 0.001 (0/3); +0.003 ± 0.001 (2/3)−0.003 ± 0.004 (1/3); −0.004 ± 0.002 (2/3)
Aligned (16k), K=1078%71.6+0.006 ± 0.005 (2/3); +0.002 ± 0.004 (1/3)−0.001 ± 0.001 (0/3); +0.001 ± 0.002 (1/3)−0.003 ± 0.002 (1/3); −0.002 ± 0.001 (1/3)
Aligned (16k), K=5088%68.1+0.003 ± 0.007 (1/3); +0.004 ± 0.010 (2/3, opp.)−0.001 ± 0.002 (1/3); −0.002 ± 0.005 (1/3)−0.001 ± 0.004 (2/3, opp.); +0.003 ± 0.006 (2/3, opp.)
Aligned (16k), K=20094%66.3+0.000 ± 0.005 (0/3); −0.002 ± 0.002 (0/3)−0.002 ± 0.002 (2/3); +0.000 ± 0.002 (0/3)−0.001 ± 0.003 (1/3); +0.000 ± 0.003 (1/3)

Only the extreme K=1 gate (dropping ~50% of tokens) degrades Fréchet relative to standard KD; for K in [5, 200], generation quality is stable and beats scratch on Fréchet in 3/3 seeds under both decoders (by −0.037 to −0.045 default and −0.026 to −0.034 tuned), while K in {50, 200} with the aligned teacher also improves F1 over scratch in 3/3 seeds under the tuned decoder.

Reviewer_St87OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why KD can improve next token prediction while make generation worse when the teacher model is not aligned well with the target domain. This is shown experimentally and the authors propose DA-KD that retains teacher KL only when the observed target token falls within the teacher's top K predictions while keeping CE supervision.

Critical strengths

The main strenght in my opinion is the controlled experiment where the data, student, and the evaluation protocol are kept fixed and repeating the comparison across different seeds. Also the paper is well-structured and supported by theoretical insights.

Critical weaknesses

Several parts of the paper are not clear to me, so I mention them below as questions:

  • Looking at Theorem 1, what evidence connects its static, per-context quadratic surrogate to the measured sequence-level FD and repetition? Can the proposed bias–variance explanation be empirically validated?
  • Also, how does top K membership distinguish teacher-domain mismatch from difficult or rare tokens?
  • In the settting, all text students are initialized from scratch. Does the same prediction–generation trade-off exist when adapting pretrained students and after checking that both KD and non-KD training have converged?
Questions for authors

Please see the Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer St87 (1/2): W1, W2

The global response summarizes the new experiments and revisions; below we address each weakness (W) in your numbering.

We thank you for your positive assessment and suggestions that we took into consideration in our revision.

Summary.

  • W1 (Theorem & empirical divergence): Changing only the KL weight within GPT-2 (across dense KD, DA-KD, and random masking) lowers perplexity by 10–13 points while degrading Fréchet, rep-4, and F1 in 3/3 seeds (default) and F1 in 3/3 seeds (both decoders; PubMedBERT Fréchet too for dense KD). On MAESTRO audio, KD lowers perplexity and worsens FAD in 3/3 seeds.
  • W2 (Top-K vs. rare/hard tokens): Cross-tabulating positions by corpus frequency and domain excess surprisal shows that as the teacher adapts (0 → 50 → 250 → 16k steps), gate retention rises monotonically on high-excess positions while staying flat or declining on low-excess positions.
  • W3 (Pretrained students & convergence): For pretrained GPT-2 students at low KL weight, KD and DA-KD improve both perplexity and repetition (lowering rep-4 from 0.068 to 0.049–0.050 in 3/3 seeds), and DA-KD achieves the lowest validation perplexity in every seed. Convergence is reported for the scratch and low-weight runs (below).

W1: Connecting the theorem to measured divergence

"...what evidence connects its static, per-context quadratic surrogate to the measured sequence-level FD and repetition? Can the proposed bias-variance explanation be empirically validated?"

We now test what the theorem allows—that prediction and generation can prefer different teacher weights—directly by varying only the KL supervision weight on GPT-2 across three seeds in dense KD, DA-KD, and the random mask (full table in our reply to 6ys9, Q1). Raising the KL weight lowers validation perplexity across all three objectives (66.3 → 53.2 in dense KD; 72.4 → 59.2 in DA-KD; 65.7 → 55.8 in random mask) while simultaneously worsening MiniLM Fréchet, rep-4, and Heuristic F1 in 3/3 seeds under the default decoder, and Heuristic F1 in 3/3 seeds under both decoders (for dense KD, in-domain PubMedBERT Fréchet too). Likewise, on MAESTRO audio (3 seeds), distillation lowers perplexity from 30.55 to 26.07 while worsening CLAP FAD in 3/3 seeds (our reply to Qw61, W2). The revision states the theorem's scope per observation (table in App. B.1): it gives an analytical bias–variance account of how the optimal teacher weights for token likelihood and for rollout distribution matching can differ, not a fitted numerical predictor (we removed the fitted text decomposition in App. B.1 and note that repetition lies outside the static surrogate).

W2: How top-K membership distinguishes domain mismatch from rare/hard tokens

"...how does top K membership distinguish teacher-domain mismatch from difficult or rare tokens?"

To separate domain mismatch from intrinsic token rarity, we analyzed 800 held-out PubMed windows (K=10) by splitting token positions along two axes: corpus frequency (rare vs. frequent) and domain excess surprisal (Surprisal_unadapted − Surprisal_reference, where the reference teacher is adapted for 16,000 PubMed steps):

Token Position CategoryCorpus ShareKept: UnadaptedKept: Adapted 50Kept: Adapted 250Kept: Reference (16k)
Rare, high excess surprisal8%42%44%46%51% (+9%)
Rare, low excess surprisal12%57%57%55%53% (−4%)
Frequent, high excess surprisal42%72%75%80%83% (+11%)
Frequent, low excess surprisal38%84%83%82%81% (−3%)

While intrinsic rarity influences baseline retention (rare tokens are 20% of the corpus and 36% of dropped positions), teacher domain adaptation selectively restores high-excess positions to the kept set (42% → 51% for rare; 72% → 83% for frequent), while retention on low-excess positions remains flat or slightly decreases (57% → 53%; 84% → 81%). Overall, 8%, 15%, and 20% of positions dropped by the unadapted teacher are recovered after 50, 250, and 16,000 adaptation steps, a measured association: the gate tracks excess surprisal beyond token frequency (the reference is itself a model).

AuthorsOpenReview ↗

Response to Reviewer St87 (2/2): W3

Contrasts: 3-seed mean ± SD, default decoder; tuned decoder. (k/3) = number of seeds whose own 95% bootstrap interval excludes zero.

W3: Pretrained students and training convergence

"Does the same prediction-generation trade-off exist when adapting pretrained students and after checking that both KD and non-KD training have converged?"

We trained a WebText-pretrained GPT-2 small student with the GPT-2 medium teacher on PubMed at three seeds (learning rate 1e-4, low KL weight). Validation perplexity and generation contrasts (default; tuned):

Pretrained Student ArmSeed 0 PPL ↓Seed 1 PPL ↓Seed 2 PPL ↓Best Epoch
No KD (fine-tuning only)21.6721.6721.700.5–1.0
Standard KD20.8320.8220.768.5–10.5
DA-KD20.7520.7020.705.0–7.0
Contrast, 3 seeds (default; tuned)MiniLM Fréchet ↓rep-4 ↓Heuristic F1 ↑
Standard KD − No KD−0.010 ± 0.004 (3/3); −0.008 ± 0.007 (2/3)−0.019 ± 0.006 (3/3); −0.017 ± 0.003 (3/3)+0.003 ± 0.002 (1/3); +0.002 ± 0.001 (0/3)
DA-KD − No KD−0.011 ± 0.004 (3/3); −0.007 ± 0.007 (2/3)−0.018 ± 0.011 (3/3); −0.019 ± 0.004 (3/3)+0.003 ± 0.001 (2/3); +0.005 ± 0.005 (1/3)
DA-KD − Standard KD−0.001 ± 0.001 (0/3); +0.001 ± 0.001 (0/3)+0.000 ± 0.005 (0/3); −0.002 ± 0.006 (1/3)+0.000 ± 0.003 (0/3); +0.004 ± 0.005 (1/3)

Fine-tuning the pretrained student without KD reaches its best validation perplexity within the first epoch and repeats well above the real-text rate (default rep-4 = 0.068 vs. 0.018). Both KD and DA-KD cut rep-4 to 0.049–0.050 (3/3 seeds under both decoders) and improve Fréchet (3/3 default), while DA-KD achieves the lowest validation perplexity in every seed. Thus, at low KL weight, pretrained and from-scratch students behave consistently: prediction and generation improve together; in our scratch-trained runs the trade-off appears at the high KL weight (stated as a scope condition in the revision, App. U.9).

Convergence (App. U.9). Scratch runs stop early at epochs 9–10. The low-weight standard KD, adapted-teacher KD, DA-KD and random-mask runs reach their best validation perplexity at epochs 15.5–19.5 of the 20-epoch schedule, gaining 0.2%–1.6% between epoch 15 and their best. The gate-width and aligned-teacher runs reach theirs at epochs 14.5–19.5 and gain at most 1.3% after epoch 15; the all-position and α-matched dense runs reach theirs at epochs 14–17.5 and gain at most 0.5%.

I thank the authors for answering my concerns. So I would keep my positive score.

Reviewer_6ys9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when knowledge distillation (KD) that improves a student's held-out likelihood also improves what the student generates. Teacher and student share a vocabulary, so the student can match the teacher token by token. Experiments cover audio (MusicGen teacher; FMA and MAESTRO), text (GPT-2, Pythia, SmolLM2 and Qwen3 teacher–student pairs on PubMed), images (ImageGPT; Flowers-102 and EuroSAT) and proteins (GPT-2 teacher; UniRef50).

There are three contributions. (1) A controlled experiment adapts the GPT-2 teacher to PubMed while holding everything else fixed: distilling from the original teacher gives worse generations than training from scratch, distilling from the adapted teacher gives better ones, and the adapted-versus-original gap is replicated over three seeds. (2) Domain-Anchored KD (DA-KD) applies the distillation loss only where the true next token is in the teacher's top-KK. It is compared with random and confidence-based gates, tuned decoding, on-policy distillation, an LLM judge and a small human study. (3) A bias–variance analysis explains why likelihood and generation quality can prefer different distillation weights.

Critical strengths
  1. Clean, controlled design. Teacher and student share a vocabulary, so the student can copy the teacher token by token with no projection or re-tokenization, and the teacher-adaptation experiment (Sec. 7) changes only the teacher. The benefit of adapting the teacher holds in all three training seeds (−0.043 ± 0.006 primed-Fréchet).

  2. Thorough and honest reporting. Strong controls (random and confidence-based gates at matched sparsity, a tuned-decoder baseline, on-policy distillation, LLM and human judges) are reported even when they cut against the method, for example a tuned decoder beating DA-KD (Table 17). The authors also withdraw numbers that did not hold up (App. B.1). This makes the claims easy to check: most of my concerns come from the paper's own tables.

Critical weaknesses
  1. The headline flip from harmful to helpful is not established.
  • The harmful half rests on one seed. In Sec. 7, distilling from the original teacher is worse than scratch (+0.017 primed-Fréchet), but in Table 2 the same teacher is no worse than scratch (0.193 vs 0.196), and Appendix A calls this comparison unresolved. The three-seed replication compares adapted vs original teacher only and does not report scratch at seeds 1–2.
  • Everything uses one fixed decoder. Switching the distilled student to nucleus sampling improves its Fréchet by 0.034 (Table 17), twice the claimed harm, and scratch was never re-scored with that decoder, so we cannot tell whether the harm survives.

What does replicate, that an adapted teacher gives a better student, is expected. The title and abstract rest on the flip, so this weighs most in my rating. It can be checked during the discussion period (Q1).

  1. DA-KD's benefit is narrower than claimed, and a better decoder gives most of it.
  • On GPT-2, a random mask dropping the same 27% of positions does as well as DA-KD (0.165 vs 0.166, Table 3). Since a random mask is, on average, the same objective as standard KD (App. M), the GPT-2 gain is not evidence that DA-KD picks the right positions.
  • GPT-2 and SmolLM2 are single-seed, and the GPT-2 numbers are noisy: uniform KD at α=0.36\alpha = 0.36 scores 0.224, worse than both α=0\alpha = 0 (0.196) and α=0.5\alpha = 0.5 (0.193). Only Pythia shows a gain that holds across seeds.
  • Standard KD with a tuned decoder beats default DA-KD on both Fréchet and repetition, at much lower perplexity (52.9 vs 72.6, Table 17). With both on the tuned decoder, DA-KD's Fréchet gain on GPT-2 disappears; small gains in repetition and word overlap remain (Table 18).

The evidence supports a seed-robust gain on Pythia and small gains after decoding, at a real perplexity cost, rather than the abstract's claim of better fidelity across all three families. Three GPT-2 seeds would settle the selection question (Q3); otherwise the claims should be narrowed.

Questions for authors
  1. Does the flip survive more seeds and a tuned decoder? [Requested analysis] Please report scratch alongside both teacher endpoints at seeds 1 and 2, re-score all three students with a decoder tuned per model on your held-out validation set (as in App. Q), and explain why Sec. 7 and Table 2 disagree on the original teacher versus scratch. If the original teacher remains significantly worse than scratch, I will consider the main claim supported; if not, the claim should become "adapting the teacher improves the student".

  2. What does the adaptation fix? [Clarification] Was the teacher adapted on the student's training text or on separate PubMed text? The flip already appears after 50 adaptation steps, and in Figs. 9–10 the PubMed prompts are lowercased while the KD samples start sentences with capitals ("In", "We") and add citation brackets. Is the mismatch mainly text format rather than domain knowledge? A short adaptation on lowercased non-medical text would tell them apart. If format explains the flip, "domain alignment" overstates the finding.

  3. Is DA-KD's GPT-2 gain due to its selection rule? [Requested analysis] Please run three seeds each of standard KD, the random mask and DA-KD on GPT-2, and report repetition and word-overlap scores for the random mask (only Fréchet is given now). If DA-KD beats the random mask across seeds on at least one generation metric, I will accept the cross-family claim; otherwise it should be limited to Pythia (and audio at K=64K = 64).

  4. Two suggestions. [Suggestion — does not affect my rating] (a) Zhou et al. (ICLR 2021), "Rethinking Soft Labels for Knowledge Distillation", also frame distillation as a bias–variance trade-off and down-weight the teacher on examples where it is less confident in the true label than the student. Please cite it and, if possible, compare DA-KD with a token-level version of that weighting. (b) Evidence for each main claim is spread over many appendices and evaluation runs that report different numbers for the same comparison (Pythia DA-KD vs standard KD appears as −0.019, −0.020, −0.025 and −0.030). One table per main claim, from a single evaluation run, would make the paper much easier to follow.

12 comments
AuthorsOpenReview ↗

Response to Reviewer 6ys9 (1/5): W1, Q1 (main contrasts)

Thank you for highlighting our controlled design and honest reporting and for your constructive review that set a concrete test for each question and made your concerns and request very clear to us that we adhere to them : three-seed scratch and tuned-decoder controls, lowercased non-medical format adaptation, three-seed random masks, and token-level Zhou et al. weighting. We ran each of them at three seeds. The results confirm the prediction–generation trade-off in a controlled KL-weight test across all three objectives, show that PubMed adaptation improves perplexity, F1 and in-domain Fréchet beyond format-only adaptation, place DA-KD ahead of the three-seed random mask on Pythia (F1) and Gemma 4 (in-domain Fréchet), and place DA-KD ahead of Zhou et al. weighting on Fréchet in 3/3 seeds. These tests shaped the revised paper.

W1 and Q1: Does the flip survive more seeds and a tuned decoder?

"If the original teacher remains significantly worse than scratch, I will consider the main claim supported; if not, the claim should become 'adapting the teacher improves the student'."

Yes. Under your Q1 criterion, the unadapted teacher at high KL weight remains worse than scratch on Heuristic F1 in all three seeds under both decoders, and adapting the teacher recovers Fréchet and F1 in 3/3 seeds under both decoders:

  • Unadapted teacher vs. scratch (3 seeds, unified session): The unadapted teacher is worse than scratch in 3/3 seeds under both decoders on Heuristic F1 (−0.010 default; −0.009 tuned), in 3/3 seeds under the default decoder on rep-4 (+0.014), and in 3/3 seeds under tuned nucleus (p=0.95) on both MeSH F1 and in-domain PubMedBERT Fréchet (+0.065 ± 0.021). Under nucleus sampling, only MiniLM Fréchet and rep-4 favor the unadapted teacher.
  • Reconciling Sec. 7 and Table 2: The values in the submission (+0.017 in Sec. 7, −0.003 in Table 2, −0.008 in Table 17) came from separate evaluation sessions. Re-scoring the archived Sec. 7 and Table 2 KD checkpoints in a single unified session shows they do not differ consistently (+0.004 ± 0.011 (1/3) default; +0.000 ± 0.003 (0/3) tuned). Every number below comes from this unified session, with all 15 new checkpoints selecting nucleus p=0.95 on the disjoint validation grid.
Contrast, 3 seeds (default; tuned nucleus 0.95)MiniLM Fréchet ↓rep-4 ↓Heuristic F1 ↑
Unadapted teacher − scratch, high KL weight (Sec. 7)†−0.003 ± 0.010 (1/3); −0.020 ± 0.004 (3/3)+0.014 ± 0.010 (3/3); −0.008 ± 0.002 (3/3)−0.010 ± 0.001 (3/3); −0.009 ± 0.002 (3/3)
Adapted − unadapted, high KL weight (Sec. 7)†−0.042 ± 0.005 (3/3); −0.019 ± 0.004 (3/3)+0.004 ± 0.011 (2/3); +0.007 ± 0.004 (2/3)+0.012 ± 0.001 (3/3); +0.011 ± 0.001 (3/3)
High − low KL weight, unadapted teacher†+0.038 ± 0.018 (3/3); +0.013 ± 0.009 (2/3)+0.030 ± 0.007 (3/3); +0.005 ± 0.002 (3/3)−0.013 ± 0.002 (3/3); −0.008 ± 0.002 (3/3)
Unadapted teacher − scratch, low KL weight−0.044 ± 0.018 (3/3); −0.033 ± 0.011 (3/3)−0.014 ± 0.005 (3/3); −0.013 ± 0.002 (3/3)+0.003 ± 0.002 (1/3); +0.000 ± 0.002 (0/3)
Adapted teacher − scratch, low KL weight−0.046 ± 0.013 (3/3); −0.036 ± 0.007 (3/3)−0.010 ± 0.006 (3/3); −0.008 ± 0.002 (3/3)+0.007 ± 0.001 (3/3); +0.004 ± 0.000 (3/3)
Adapted − unadapted, low KL weight−0.002 ± 0.005 (1/3); −0.003 ± 0.004 (2/3)+0.005 ± 0.002 (3/3); +0.005 ± 0.000 (3/3)+0.004 ± 0.001 (3/3); +0.004 ± 0.001 (2/3)

† Scored with the shared nucleus p=0.95 decoder selected by all 15 new checkpoints.

AuthorsOpenReview ↗

Response to Reviewer 6ys9 (2/5): W1, Q1 (KL-weight test, in-domain encoder)

Contrasts: 3-seed mean ± SD, default decoder; tuned decoder. (k/3) = number of seeds whose own 95% bootstrap interval excludes zero.

Controlled test of the title's claim across all three objectives (W1/Q1, continued; 3 seeds). The high- and low-KL-weight runs share the same student, data, α, seeds, and 20-epoch schedule, differing only in KL weight (Q3). To test whether the effect is specific to dense KD, we applied the same weight increase to Dense KD, DA-KD, and the Random Mask:

High − low KL weight, 3 seeds (default; tuned)Perplexity Change ↓MiniLM Fréchet ↓rep-4 ↓Heuristic F1 ↑
Dense KD†66.3 → 53.2+0.038 ± 0.018 (3/3); +0.013 ± 0.009 (2/3)+0.030 ± 0.007 (3/3); +0.005 ± 0.002 (3/3)−0.013 ± 0.002 (3/3); −0.008 ± 0.002 (3/3)
DA-KD72.4 → 59.2+0.035 ± 0.009 (3/3); +0.009 ± 0.007 (2/3)+0.032 ± 0.003 (3/3); +0.008 ± 0.003 (3/3)−0.012 ± 0.003 (3/3); −0.007 ± 0.003 (3/3)
Random mask65.7 → 55.8+0.045 ± 0.027 (3/3); +0.018 ± 0.010 (3/3)+0.027 ± 0.010 (3/3); +0.003 ± 0.003 (2/3)−0.014 ± 0.005 (3/3); −0.009 ± 0.002 (3/3)

In all three objectives, increasing the teacher weight lowers validation perplexity by 10–13 points while degrading Fréchet, rep-4, and Heuristic F1 in 3/3 seeds under the default decoder and Heuristic F1 in 3/3 seeds under the tuned decoder (dropping F1 below scratch in 3/3 seeds under both decoders for all three arms).

In-domain biomedical encoder (PubMedBERT Fréchet, 3 seeds). Re-scoring the same saved continuations with PubMedBERT (PCA-projected to 384 dims fitted on real text; full 768-dim agrees in sign) shows that the unadapted teacher's student is worse than scratch on in-domain Fréchet under tuned decoding, and that teacher adaptation improves in-domain Fréchet at both high and low KL weights:

PubMedBERT Fréchet ↓, 3 seeds (default; nucleus 0.95)Difference
Unadapted teacher − scratch, high KL weight (Sec. 7)†+0.011 ± 0.022 (2/3, opposite signs); +0.065 ± 0.021 (3/3)
Adapted − unadapted, high KL weight (Sec. 7)†−0.100 ± 0.009 (3/3); −0.115 ± 0.007 (3/3)
High − low KL weight, unadapted teacher†+0.076 ± 0.019 (3/3); +0.100 ± 0.009 (3/3)
Standard KD − scratch, low KL weight−0.066 ± 0.029 (3/3); −0.036 ± 0.021 (3/3)
Adapted − unadapted, low KL weight−0.031 ± 0.010 (3/3); −0.047 ± 0.011 (3/3)

† Scored with the shared nucleus p=0.95 decoder selected by all 15 new checkpoints.

AuthorsOpenReview ↗

Response to Reviewer 6ys9 (3/5): Q2 (format vs. domain)

Q2: What does the adaptation fix — format or domain knowledge?

"Is the mismatch mainly text format rather than domain knowledge? A short adaptation on lowercased non-medical text would tell them apart."

We ran this control at three seeds at both KL weights: adapting the teacher on lowercased OpenWebText (50 steps) and lowercased OpenWebText with PubMed-style spacing around punctuation and brackets (50 and 250 steps). Format adaptation closes most of the punctuation gap but leaves validation perplexity unchanged (53.4 vs. 53.4 unadapted, vs. 47.8 for PubMed adaptation) and trails PubMed adaptation in 3/3 seeds under both decoders on Heuristic F1 and PubMedBERT Fréchet:

High KL weight, 3 seeds (default; nucleus 0.95)MiniLM Fréchet ↓Heuristic F1 ↑PubMedBERT Fréchet ↓
Format − unadapted−0.034 ± 0.002 (3/3); −0.012 ± 0.002 (3/3)+0.007 ± 0.001 (3/3); +0.006 ± 0.002 (3/3)−0.042 ± 0.006 (3/3); −0.048 ± 0.006 (3/3)
PubMed − unadapted−0.042 ± 0.005 (3/3); −0.019 ± 0.004 (3/3)+0.012 ± 0.001 (3/3); +0.011 ± 0.001 (3/3)−0.100 ± 0.009 (3/3); −0.115 ± 0.007 (3/3)
Format − PubMed (Domain gain beyond format)+0.008 ± 0.003 (2/3); +0.007 ± 0.004 (2/3)−0.005 ± 0.002 (3/3); −0.005 ± 0.002 (3/3)+0.058 ± 0.014 (3/3); +0.067 ± 0.006 (3/3)
  • High KL weight: Format adaptation closes 76% of the glued-punctuation gap (32.0% vs. 22.7% PubMed-adapted and 61.1% unadapted), accounting for part of the generic MiniLM shift. However, only PubMed domain adaptation lowers student perplexity (47.8 vs. 53.4), lifts F1 above scratch, and lowers PubMedBERT Fréchet below scratch (−0.050, 3/3 vs. +0.017 for format).
  • Low KL weight: Even after 250 format steps—where glued punctuation is lower than PubMed adaptation's (8.0% vs. 10.5%, real text 8.3%) and default rep-4 stays at the real-text rate (0.016–0.018 vs. 0.021–0.025 for PubMed)—format adaptation never produces a consistent F1 gain (at most 2/3), whereas PubMed adaptation improves F1 in 3/3 seeds (50 steps default; 250 steps tuned) and PubMedBERT Fréchet in 3/3 seeds under both decoders (−0.031; −0.047).

Surface statistics & training split. The PubMed corpus is lowercase (stated in the revision), MiniLM lowercases all inputs before encoding, and citation brackets are native to the corpus (~5 per real passage):

Metric (Default decoder, 1000 passages)RealScratchUnadapted (low KL)Adapted (low KL)DA-KDUnadapted (high KL)Adapted (high KL)
Capital letters0.00%0.00%0.27%0.12%0.11%1.24%0.44%
Punctuation glued to previous word8.3%11.1%32.7%14.4%34.7%60.5%22.7%

Teacher adaptation sampled from the full 20.4M-token PubMed training file containing the student's 2M tokens (noted in App. B.1).

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global response (2/2): DA-KD and the revised PDF

Core Findings (continued):

  • 2. Observation-Gated Distillation (DA-KD): Comparing gated and dense objectives at matched per-position KL scale separates two effects in DA-KD, supervision down-weighting and token-level position selection:
    • Position Selection (DA-KD vs. 3-Seed Matched Random Mask): On Pythia, DA-KD beats both standard KD and the 3-seed matched random mask on Heuristic F1 (all three seeds under both decoders; vs. random mask +0.0113 ± 0.0002; +0.0057 ± 0.0030) and MiniLM Fréchet (all three seeds under the default decoder, two under tuned; vs. random mask −0.0334 ± 0.0027; −0.0116 ± 0.0053). On Gemma 4, DA-KD beats the 3-seed matched random mask on in-domain PubMedBERT Fréchet in all three seeds under both decoders (−0.0143 ± 0.0055; −0.0122 ± 0.0035) and raises default-decoder F1 over standard KD in every seed (+0.0031 ± 0.0014, two with intervals excluding zero). On MAESTRO audio, tightening the gate from K=64 (90.9% kept) to K=16 (75.5% kept) and K=4 (52.8% kept) widens DA-KD's mean FAD advantage over random masking at each step (0.00000 → −0.00041 → −0.00069); at K=4, DA-KD's FAD is at or below the random mask's in every seed (−0.00069 ± 0.00066).
    • Supervision Weighting vs. Token Selection (GPT-2): The archived GPT-2 dense-KD baseline used PyTorch's standard sequence-level F.kl_div(..., reduction="batchmean") on (B, S, V) log-probabilities, which sums the KL over the S=512 positions of each sequence (a high effective KL weight relative to per-token CE), whereas the gated objective averages over kept positions. DA-KD's improvement over this sequence-level standard KD reproduces in all three seeds under the default decoder (Fréchet −0.034 ± 0.016; F1 +0.015 ± 0.003). Matching both objectives at per-position scale shows that on GPT-2 MiniLM this gain comes from the lower per-token KL weight, while on in-domain PubMedBERT Fréchet DA-KD is also below the matched random mask in every seed under tuned nucleus decoding (−0.0146 ± 0.0097, two with intervals excluding zero).

Summary of Revisions in the Updated PDF:

  1. Unified 3-seed GPT-2 evaluation (Scratch, Low/High-KL KD, Adapted KD, DA-KD, Random Mask, Zhou et al. weighting) under default and tuned decoders, plus in-domain PubMedBERT Fréchet (6ys9 Q1, Q3, Q4): Sec. 7, Figure 4, App. U.1 (Tables 23–24).
  2. Controlled KL-weight test across dense KD, DA-KD, and random masking (6ys9 Q1): Sec. 7, App. U.1.
  3. Format vs. domain adaptation ablations at high and low KL weights (6ys9 Q2): App. U.2 (Table 25); token-level gate excess-surprisal analysis (St87 W2): App. U.8 (Table 30).
  4. Three-seed Gemma 4 family, 3-seed Pythia random-mask controls, and per-family abstract wording (6ys9 Q3, W2): Tables 3–4, App. U.4–U.5 (Tables 27–28).
  5. Pretrained GPT-2 student runs and convergence reporting (St87 W3): App. U.9 (Table 31).
  6. Three-seed MAESTRO audio controls across K in {64, 16, 4} with track-level bootstrap intervals (Qw61 W2): Table 12, App. U.6.
  7. Three-seed sensitivity sweeps over K in [1, 200] (unadapted and aligned teachers), prompt lengths (12, 32 words), horizons (75, 300 tokens), and Kaplan FLOP vs. wall-clock analysis (Qw61 Q1, Q5): Table 13, App. U.7 (Table 29), App. U.11 (Table 33).
  8. Scope of the theorem stated per observation (St87 W1, Qw61 Q4): App. B.1; GPT-2 gate controls at matched KL scale (6ys9 Q3): App. U.3 (Table 26); token-level Zhou et al. weighting (6ys9 Q4): App. U.10 (Table 32).
  9. Regenerating 3 figures for better readability

Reading the revised PDF. The new evaluations are in App. U (U.1–U.11, Tables 23–33) and are cited in the main text where each claim is made. Sec. 4 defines the metrics and the three-seed reporting rule, and Sec. 5 opens with a map from each claim to its table or figure. Every section, table and equation number cited in the reviews and in our responses is unchanged.

Planned reordering (final version; not applied in this revision). We plan to present the results text-first: the controlled adaptation and KL-weight results (now Sec. 7) first, then the text divergence (Sec. 6), then audio as the cross-modal replication (Sec. 5), then vision (Sec. 8) and DA-KD (Sec. 9). We kept the submitted order in this revision so that the numbering matches the reviews and a comparison with the submitted PDF (latexdiff) shows the actual changes instead of marking whole sections as moved.

We are grateful that our controlled experimental design and transparent reporting were appreciated

AuthorsOpenReview ↗

Global response (1/2): new experiments and central finding

We thank you to all three reviewers for their constructive reviews and concrete experimental suggestions. In response, we ran a unified three-seed evaluation (1,000 continuations per checkpoint; tuned decoders chosen on a 9-setting validation grid, where 105 of 107 checkpoints selected nucleus p=0.95):

  1. Fourth Model Family (Gemma 4, 3 seeds): Distilled Gemma 4 E2B into a 268M student on PubMed across all four arms (Scratch, Standard KD, Random Mask, DA-KD).
  2. In-Domain Biomedical Encoder (PubMedBERT Fréchet, 3 seeds): Re-scored continuations with an in-domain encoder alongside MiniLM Fréchet, rep-4, and Heuristic/MeSH F1.
  3. Matched Per-Position KL Controls (GPT-2 & Pythia, 3 seeds): Separated teacher supervision weight from token-level gate selection across dense KD, DA-KD, matched random masks (now 3 seeds on Pythia, Gemma 4, and GPT-2), and token-level Zhou et al. (2021) weighting.
  4. Format vs. Domain Ablations (GPT-2, 3 seeds): Adapted teachers on lowercased/spaced OpenWebText vs. PubMed at both high and low KL weights.
  5. Expanded Audio (3 seeds, track-level bootstrap) & Sensitivity Sweeps: Full 3-seed MAESTRO audio controls across K in {64, 16, 4} and 3-seed GPT-2 sweeps over K in [1, 200], prompt lengths (12, 24, 32 words), horizons (75, 150, 300 tokens), and pretrained students.

Reporting Protocol: Each contrast reports the 3-seed mean ± SD and the number of seeds whose 95% bootstrap CI excludes zero, under Default and Tuned Nucleus (p=0.95) decoding; paired values read default; tuned. In all three seeds means that every seed's interval excludes zero with the same sign; in every seed means that the sign agrees in each seed. Heuristic F1 is the paper's technical-word F1. Negative Fréchet/rep-4 (↓) and positive F1 (↑) favor the first model. Real PubMed continuations score rep-4 = 0.018; because nucleus sampling compresses repetition below real text (0.008–0.015), rep-4 is read directionally under the default decoder (0.016–0.068) with nucleus reported for completeness.

Core Findings Across All New Controls:

  • 1. Prediction–Generation Divergence & Teacher Alignment (Central Thesis): Confirmed in all three seeds on text and audio (protein: paper):
    • Controlled KL-Weight Test (GPT-2): Raising only the KL weight—whether inside dense KD, DA-KD, or random masking—lowers validation perplexity (66.3 → 53.2 in dense KD; 72.4 → 59.2 in DA-KD; 65.7 → 55.8 in random mask) while consistently degrading generation: Heuristic F1 worsens in all three seeds under both decoders (dense KD −0.013 ± 0.002; −0.008 ± 0.002), default rep-4 worsens in all three seeds (+0.030 ± 0.007; 0.017 → 0.046 vs. 0.018 real text), and, for dense KD, in-domain PubMedBERT Fréchet worsens in all three seeds under both decoders (+0.076 ± 0.019; +0.100 ± 0.009).
    • Domain vs. Format Adaptation (Sec. 7): Adapting the teacher on PubMed recovers Fréchet (−0.042 ± 0.005; −0.019 ± 0.004), Heuristic F1 (+0.012 ± 0.001; +0.011 ± 0.001), MeSH F1 (+0.010 ± 0.001; +0.008 ± 0.001), and PubMedBERT Fréchet (−0.100 ± 0.009; −0.115 ± 0.007) in all three seeds under both decoders. Format-only adaptation (lowercased/spaced OpenWebText) leaves perplexity flat (53.4 vs. 53.4 unadapted) and trails PubMed adaptation on F1 (−0.005 ± 0.002; −0.005 ± 0.002) and PubMedBERT Fréchet (+0.058 ± 0.014; +0.067 ± 0.006) in all three seeds under both decoders.
    • Cross-Family & Cross-Modal: On Gemma 4, standard KD lowers perplexity (53.4 → 40.8) while dropping Heuristic F1 below scratch in every seed under both decoders (−0.0043 ± 0.0023; −0.0017 ± 0.0011). On MAESTRO audio, dense KD lowers perplexity (30.55 → 26.07) while worsening FAD against scratch in all three seeds (+0.00167 ± 0.00059; both window- and track-level bootstrap).
AuthorsOpenReview ↗

Response on novelty: Why perplexity favors stronger supervision and readers prefer DA-KD

Thank you for confirming that the revisions resolve the concerns in your original review. We kindly disagree with the assessment that the paper’s contribution is incremental. The likelihood–generation gap, teacher adaptation, and selective distillation each have precedents. Our paper asks a more specific question: when held-out prediction favors stronger teacher supervision, what happens to the student’s generations, and can the teacher’s own generation quality explain the outcome?

The controlled KL-weight experiment identifies a distinct model-selection failure. In Kim & Rush’s Table 1, selecting by perplexity would discard the better translator. Their smaller word-level student also shows that direction. In our GPT-2 experiment, selecting by perplexity favors the student whose continuations score worse. Increasing only the token-level KL weight lowers validation perplexity from 66.3 to 53.2 while worsening repetition, technical-word F1, and MiniLM Fréchet in all three seeds under the default decoder (Table 24). Your suggested teacher evaluation makes the distinction concrete. Under nucleus decoding, technical-word F1 is 0.041 for the unadapted teacher, 0.038 for scratch, and 0.029 for the high-weight student. The student falls below both on the very metric where its teacher beats scratch.

Theorem 1 gives an exact condition for this trade-off under the stated quadratic surrogates. You observe that it does not derive repetition, the gate’s gain, or the 50-step adaptation threshold. That is its stated scope, and those observations have separate experimental evidence. The theorem asks when two evaluations of the same mixture error favor different teacher weights in the direction relevant to the paper. Let VjV_j and BjB_j denote sampling variance and squared teacher bias under the prediction (PP) or generation-feature (GG) geometry. Increasing the teacher weight improves prediction while worsening generation on a nonempty interval if and only if VPBG>VGBPV_P B_G > V_G B_P. Different evaluation criteria alone do not guarantee that ordering. With zero teacher bias and positive variances, both surrogate risks improve together. The experiments measure the change in model ordering, and Appendix B.1 states that the condition itself is not measured.

The format-only control quantifies what domain adaptation adds. Gordon & Duh establish the benefit of adapting a translation teacher. In our experiment, surface formatting accounts for 41–81% of the measured recovery, depending on the metric. PubMed adaptation still adds 0.005 technical-word F1 over format-only adaptation under both decoders. It also lowers in-domain Fréchet by a further 0.058 under default decoding and 0.067 under nucleus decoding. Both advantages hold in all three seeds (Table 25).

The cited token-selection rules make competing choices that we test directly. Wang et al. select by student difficulty, SelecTKD verifies a student proposal, and ATKD retains target-oriented supervision on tokens the teacher finds difficult. In our measured Pythia batches, every position DA-KD drops falls in ATKD’s hard half. The ATKD-style arm achieves the lowest perplexity (53.9, versus 59.2 for DA-KD), yet DA-KD achieves higher technical-word F1 and lower MiniLM Fréchet in all three seeds under both decoders (Table 28). Thus, even at the standard per-position weight, the choice of evaluation criterion changes which supervision rule a practitioner would select.

Qwen3 at 8M is the boundary Sec. 10 predicted before the run: with an aligned teacher, standard KD improves perplexity and Fréchet, so gating can only forgo signal (0.013 worse at 8M; not significant at 50M; Table 19).

Readers also detected the GPT-2 difference. Although sequence-level standard KD had lower perplexity than DA-KD (53 vs. 72), all four blinded human raters favored DA-KD overall across the 16 paired items. Its pooled preference was 66.7% of non-tied judgments (95% item-bootstrap interval 52.8–78.9%). The counterbalanced LLM judge gave DA-KD a 57.8% preference score across all 250 prompts, counting ties and order-inconsistent verdicts as half (Tables 20–22). This reader comparison includes the KL-weight difference. The Pythia experiment separately compares supervision rules at the per-position weight.

Thank you for your close reading, particularly your suggestion to evaluate the teachers on the same prompts as the students. That comparison sharpened our central finding: even when the teacher outperforms a student trained from scratch on technical-word F1, increasing the weight of token-level supervision can lower the distilled student’s held-out perplexity while pushing its technical-word F1 below both the teacher and the scratch baseline.

As the discussion draws to a close, we hope this response clarifies why we consider the contribution novel. In light of the revised paper and these results, we would appreciate it if you would reconsider your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.