- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a mismatch between teacher-forced prediction and autonomous generation in knowledge distillation. It argues that a teacher can improve held-out likelihood while degrading target-domain generation when the teacher is poorly aligned with the target domain. The authors demonstrate this phenomenon across audio, text, image, and protein settings, including a controlled intervention where adapting the same GPT-2 teacher on PubMed changes the effect of distillation from harmful to beneficial.
The paper proposes Domain-Anchored Knowledge Distillation (DA-KD), which applies the teacher KL loss only when the observed target token is among the teacher’s top-K predictions. Across GPT-2, Pythia, and SmolLM2 experiments, DA-KD improves several generation-oriented metrics relative to dense KD, including primed Fréchet distance, repetition, and selected lexical/domain metrics. The paper also includes aligned, null, and extreme teacher-domain controls, together with a quadratic bias-variance analysis to explain why prediction and generation may prefer different distillation weights.
Critical strengths
The controlled teacher-adaptation experiment is a strong part of the paper. Fixing the corpus, student, tokenizer, optimization, and generation protocol while changing the teacher checkpoint provides more convincing evidence for the role of teacher-domain alignment than comparing unrelated teachers alone.
The paper evaluates multiple modalities and teacher families and includes aligned, null, extreme, and matched-retention controls. The comparisons against random and confidence-based gates are useful for assessing whether the observed-token criterion contributes beyond reducing the amount of KL supervision.
Critical weaknesses
The evidence for degraded generation is metric- and protocol-dependent. The main evaluation uses embedding Fréchet distance, repetition, lexical overlap, and fixed primed continuations, which do not directly establish factual accuracy, long-horizon quality, or downstream utility. In particular, the GPT-2 dense-KD versus scratch Fréchet contrast is unresolved, so the clearest text evidence for degradation comes from repetition rather than from a consistent set of generation metrics.
The scope of DA-KD’s superiority is not fully established. Some audio controls use a single training seed, K=16 is unresolved, the human evaluation is small, and the on-policy comparison covers only a few fixed configurations. The claims should therefore be scoped to the reported models, datasets, prompts, horizons, and decoding protocols.
Questions for authors
Could the authors provide a more systematic sensitivity analysis over K, prompt length, generation horizon, temperature, and decoding strategy for at least one distant teacher and one aligned teacher?
How much of the DA-KD gain comes from the observed-token gate itself versus the induced position-dependent reduction in the effective KL weight? A per-context analysis of retained positions and effective alpha would help clarify this.
Please consider adding the following recent related works to the discussion: “Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning” (arXiv:2511.21075).
The theoretical section uses quadratic surrogate risks and a static mixture target. Can the authors better distinguish which empirical observations are explained by the theorem and which are only qualitative interpretations?
Can the authors provide end-to-end training cost or FLOP comparisons, especially for the on-policy baseline and the gate implementation, rather than only local throughput measurements?
5 comments
Response to Reviewer Qw61 (2/5): W2 (audio)
Contrasts: 3-seed mean ± SD, default decoder; tuned decoder. (k/3) = number of seeds whose own 95% bootstrap interval excludes zero.*
W2: Three-seed MAESTRO audio controls (K=64, 16, 4) and track-level bootstrap
"Some audio controls use a single training seed, K=16 is unresolved, the human evaluation is small, and the on-policy comparison covers only a few fixed configurations."
We trained seeds 1 and 2 for scratch, dense KD, and DA-KD along with matched random and anti-confidence masks across K in {64, 16, 4} (1,000 MAESTRO windows, CLAP FAD), and added a 2,000-replicate whole-track bootstrap across the 23 validation tracks:
| Audio CLAP FAD Difference ↓ (negative favors first model) | Gate Retention | Seeds 0 / 1 / 2 | 3-Seed Mean ± SD | Seeds Excluding Zero (Window; Track) |
|---|---|---|---|---|
| Dense KD vs scratch (PPL 30.55 → 26.07) | 100% | +0.00235 / +0.00132 / +0.00133 | +0.00167 ± 0.00059 | 3/3; 3/3 |
| DA-KD K=64 vs dense KD | 90.9% | −0.00082 / +0.00087 / +0.00022 | +0.00009 ± 0.00085 | 2/3 (opp. signs); 2/3 (opp. signs) |
| DA-KD K=64 vs random mask | 90.9% | −0.00112 / +0.00083 / +0.00029 | +0.00000 ± 0.00101 | 2/3 (opp. signs); 2/3 (opp. signs) |
| DA-KD K=64 vs anti-confidence | 90.9% | −0.00081 / +0.00022 / +0.00036 | −0.00008 ± 0.00064 | 1/3; 1/3 |
| DA-KD K=16 vs dense KD | 75.5% | −0.00031 / +0.00013 / −0.00090 | −0.00036 ± 0.00052 | 1/3; 1/3 |
| DA-KD K=16 vs random mask | 75.5% | −0.00034 / −0.00020 / −0.00070 | −0.00041 ± 0.00026 | 1/3; 0/3 (lower in every seed) |
| DA-KD K=16 vs anti-confidence | 75.5% | +0.00000 / −0.00053 / −0.00071 | −0.00041 ± 0.00037 | 2/3; 2/3 |
| DA-KD K=4 vs dense KD | 52.8% | −0.00076 / −0.00026 / −0.00032 | −0.00045 ± 0.00027 | 1/3; 1/3 (lower in every seed) |
| DA-KD K=4 vs random mask | 52.8% | −0.00132 / −0.00074 / 0.00000 | −0.00069 ± 0.00066 | 2/3; 2/3 (at or below in every seed) |
| DA-KD K=4 vs anti-confidence | 52.8% | −0.00042 / −0.00074 / +0.00020 | −0.00032 ± 0.00048 | 1/3; 2/3 |
- Prediction–generation divergence on audio is 3/3: Dense KD lowers perplexity (30.55 → 26.07) while worsening FAD against scratch in all three seeds under both window and track-level resampling.
- Selection at lower retention (K=16, K=4): At K=64, the gate retains 90.9% of positions and the gates do not separate consistently across seeds. Tightening the gate to K=16 (75.5% kept) and K=4 (52.8% kept) moves the mean FAD difference in DA-KD's favor against both dense KD (+0.00009 → −0.00036 → −0.00045; lower in every seed at K=4) and the matched random mask (0.00000 → −0.00041 → −0.00069; lower in every seed at K=16).
- At K=4 (52.8% kept), DA-KD's FAD is at or below the random mask's in every seed (lower in two, both intervals excluding zero; tied in the third) and below dense KD's in every seed (one interval excluding zero).
- The human evaluation and the on-policy comparison are retained, with their size and configurations stated as limitations.
Response to Reviewer Qw61 (3/5): Q1 (gate width)
Q1: Sensitivity over K, prompt length, horizon, and decoding strategy
"...sensitivity analysis over K, prompt length, generation horizon, temperature, and decoding strategy for at least one distant teacher and one aligned teacher?"
Of the 107 checkpoints evaluated on the 9-setting temperature/nucleus validation grid, 105 selected nucleus p=0.95 (one pretrained DA-KD seed and one aligned-teacher K=50 seed selected a repetition penalty, 1.05 and 1.10). Across every K in [1, 200], prompt length (12, 24, 32 words), and horizon (75, 150, 300 tokens) with both unadapted and 16k-step aligned teachers, distillation improves Fréchet over scratch in 3/3 seeds under both decoders.
K-sweep (3 seeds, unadapted and aligned teachers). Contrasts show DA-KD at each K minus low-weight standard KD (unadapted PPL 66.3; aligned PPL 64.8):
| Teacher & Gate K | Kept | Perplexity ↓ | MiniLM Fréchet ↓ vs Std KD | rep-4 ↓ vs Std KD | Heuristic F1 ↑ vs Std KD |
|---|---|---|---|---|---|
| Unadapted, K=1 | 46% | 83.4 | +0.012 ± 0.003 (3/3); +0.009 ± 0.003 (3/3) | +0.007 ± 0.004 (3/3); +0.006 ± 0.002 (3/3) | −0.000 ± 0.002 (0/3); +0.002 ± 0.001 (1/3) |
| Unadapted, K=5 | 66% | 74.1 | +0.003 ± 0.008 (2/3, opp.); +0.002 ± 0.006 (1/3) | +0.003 ± 0.003 (2/3); +0.003 ± 0.001 (3/3) | +0.000 ± 0.002 (0/3); +0.002 ± 0.002 (1/3) |
| Unadapted, K=10 | 73% | 72.4 | +0.003 ± 0.005 (1/3); +0.004 ± 0.004 (1/3) | +0.001 ± 0.002 (2/3); +0.002 ± 0.001 (2/3) | +0.002 ± 0.002 (1/3); +0.001 ± 0.003 (1/3) |
| Unadapted, K=20 | 78% | 70.5 | −0.001 ± 0.003 (0/3); −0.001 ± 0.002 (0/3) | −0.000 ± 0.001 (0/3); +0.002 ± 0.001 (2/3) | +0.001 ± 0.003 (1/3); +0.002 ± 0.001 (0/3) |
| Unadapted, K=50 | 84% | 68.8 | +0.004 ± 0.004 (1/3); +0.001 ± 0.002 (0/3) | +0.000 ± 0.001 (0/3); +0.001 ± 0.002 (2/3) | −0.000 ± 0.001 (0/3); +0.002 ± 0.001 (1/3) |
| Unadapted, K=100 | 88% | 67.8 | +0.002 ± 0.009 (2/3, opp.); +0.001 ± 0.005 (1/3) | +0.000 ± 0.002 (1/3); +0.001 ± 0.001 (1/3) | +0.001 ± 0.003 (0/3); +0.003 ± 0.003 (2/3) |
| Unadapted, K=200 | 91% | 67.3 | +0.002 ± 0.005 (0/3); −0.001 ± 0.003 (0/3) | −0.000 ± 0.002 (0/3); +0.001 ± 0.001 (0/3) | +0.001 ± 0.001 (0/3); +0.002 ± 0.001 (0/3) |
| Aligned (16k), K=1 | 51% | 85.5 | +0.016 ± 0.009 (3/3); +0.015 ± 0.004 (3/3) | +0.000 ± 0.001 (0/3); +0.003 ± 0.001 (2/3) | −0.003 ± 0.004 (1/3); −0.004 ± 0.002 (2/3) |
| Aligned (16k), K=10 | 78% | 71.6 | +0.006 ± 0.005 (2/3); +0.002 ± 0.004 (1/3) | −0.001 ± 0.001 (0/3); +0.001 ± 0.002 (1/3) | −0.003 ± 0.002 (1/3); −0.002 ± 0.001 (1/3) |
| Aligned (16k), K=50 | 88% | 68.1 | +0.003 ± 0.007 (1/3); +0.004 ± 0.010 (2/3, opp.) | −0.001 ± 0.002 (1/3); −0.002 ± 0.005 (1/3) | −0.001 ± 0.004 (2/3, opp.); +0.003 ± 0.006 (2/3, opp.) |
| Aligned (16k), K=200 | 94% | 66.3 | +0.000 ± 0.005 (0/3); −0.002 ± 0.002 (0/3) | −0.002 ± 0.002 (2/3); +0.000 ± 0.002 (0/3) | −0.001 ± 0.003 (1/3); +0.000 ± 0.003 (1/3) |
Only the extreme K=1 gate (dropping ~50% of tokens) degrades Fréchet relative to standard KD; for K in [5, 200], generation quality is stable and beats scratch on Fréchet in 3/3 seeds under both decoders (by −0.037 to −0.045 default and −0.026 to −0.034 tuned), while K in {50, 200} with the aligned teacher also improves F1 over scratch in 3/3 seeds under the tuned decoder.
Response to Reviewer Qw61 (1/5): W1
The global response summarizes the new experiments and revisions; below we address each weakness (W) and question (Q) in your numbering.
Thank you for your constructive comments and your added insight. We went through each detail and revised our manuscript to adhere to your suggestions and your comments. We are open for additional suggestions.
In short.
W1: Metric, encoder, and protocol robustness
Reporting every contrast across three metrics, two encoders (MiniLM and in-domain PubMedBERT), and two decoders (default and tuned nucleus p=0.95) shows that the degradation is not confined to repetition: it appears on Heuristic F1 and on in-domain Fréchet under both decoders: