- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates uncertainty collapse in sequence-level knowledge distillation (SeqKD), where a single fixed teacher response is used for each input. The authors show that continued distillation can increase student hallucinations and reduce the semantic-entropy AUROC for detecting incorrect answers to near-chance levels. In contrast, using multiple teacher responses can better preserve the teacher's output distribution and maintain the student's uncertainty awareness.
Critical strengths
The paper clearly formulates and analyzes a fundamental objective mismatch in standard SeqKD. The point-mass optimum and non-identifiability propositions provide concise theoretical explanations for why a single teacher response is insufficient to preserve the teacher's output distribution.
Critical weaknesses
- The information advantage of soft probability targets over hard labels is well established in classical knowledge distillation, including MNIST experiments [1]. However, the paper does not sufficiently distinguish its contribution from this established principle or ordinary hard-target overfitting. For example, the decline in Llama-3.1 AUROC from 0.839 at epoch 2 to 0.543 at epoch 10 may simply reflect repeated cross-entropy training on a small, fixed dataset. It remains unclear whether this degradation would persist under practical distillation budgets or with validation-based early stopping.
- The main experiments train on teacher-generated answers to 500 SimpleQA questions and evaluate on the same questions, making it difficult to distinguish memorization from generalizable uncertainty loss. Although held-out evaluations are included, experiments with independent training corpora, multi-source instruction tuning, and reasoning-enabled distillation are lacking. Moreover, the SimpleQA mitigation experiments confound response diversity with sampling temperature (τ = 0.1 for single targets versus τ = 1.0 for multiple targets). The matched-temperature MMLU-Pro control does not fully resolve this confound for SimpleQA.
[1] Distilling the Knowledge in a Neural Network, 2015.
Questions for authors
- The loss of distributional information when replacing probability targets with hard labels is already well established. What new insights does this work offer beyond classical knowledge distillation and ordinary overfitting to fixed targets?
- Training for 10–20 epochs on only 500 fixed questions is a highly repetitive setting, which is not a practical distillation setting. The AUROC decline from 0.839 to 0.543 may primarily reflect overfitting.
- Using evaluation questions as distillation training data limits the ability of the main experiments to assess generalization, as the student is trained on teacher answers to the same questions used for evaluation. The appendix's held-out results do not fully address this limitation. Please use an independent training corpus and a strictly held-out benchmark for the main experiments.
- Multi-source data mixing is common in instruction tuning but is not evaluated here. Repeatedly training on a small, isolated distillation corpus does not establish whether the same uncertainty collapse occurs when teacher responses are mixed with other instruction data.
- The experiments do not cover dedicated reasoning models or distillation with thinking mode enabled.
- The SimpleQA mitigation experiments vary both response diversity and sampling temperature ( = 0.1 versus = 1.0), making it difficult to attribute the improvements specifically to diversity. Please compare single and multiple responses at the same temperature under matched training budgets.
2 comments
Thank you for the additional experiments and clarifications. However several concerns remain insufficiently addressed. In particular, the distinction between uncertainty collapse and ordinary har-target overfitting is still unclear, the generalization remain limited. Additionally, distilling step-by-step solution does not fully address distillation from dedicated reasoning models.
Overall while the rebuttal improves the paper, it does not sufficiently resolve my fundamental conern regarding the meaning and novelty of the study and experimental valiation. I maintain my original rating.
We thank the reviewer for the detailed comments. We clarify the contribution and address the experimental concerns point by point.
Classical KD establishes the information advantage of distributional targets. Our contribution is to measure its reliability consequence in open-ended factual SeqKD: on the questions used for distillation, factual accuracy improves while semantic uncertainty becomes less able to distinguish answers that remain wrong. This semantic uncertainty collapse is the outcome we study, distinct from a conventional train–held-out performance gap. From the base Llama student to epoch 5, accuracy rises from 0.058 to 0.154 while detection AUROC falls from 0.791 to 0.647 (Section 5.3, Figure 3; Appendix D.1, Table 1). Propositions 1–3 connect this measured effect to the single-response SeqKD objective (Section 4).
The accuracy–detectability separation is already present at epoch 5; epochs 15–20 illustrate its later approach to chance. Validation-based early stopping preserves AUROC but gives up much of the later accuracy gain (see W2 for Reviewer py7B). The mixed-source experiment also shows the effect when the teacher targets are combined with other instruction data (see W2 for Reviewer Fmcm).
The same-question evaluation measures the paper’s primary in-corpus semantic uncertainty finding. The submitted held-out SimpleQA, TriviaQA, MMLU-Pro, and MATH evaluations measure its extension to unseen questions (Section 5.4; Appendix F.1–F.2, Tables 5–7; Appendix E.1–E.2, Table 4). We additionally tested mixed-source training (W2 for Reviewer Fmcm) and a same-temperature SimpleQA control (W1 for Reviewer Fmcm). The submitted MATH experiment distills full step-by-step teacher solutions, rather than final answers alone (Appendix E.1).
Beyond the held-out evaluations in the submitted paper, we evaluated a separate 1,000-question SimpleQA set used for neither training nor validation in the new controlled runs. On this strictly held-out set, hard-target and matched-update K = 5 detection AUROCs are 0.700 and 0.713, respectively. The mixed-source experiment adds 600 general, math, and code instruction examples from independent sources to the SimpleQA teacher targets (see W2 for Reviewer Fmcm).
See W2 for Reviewer Fmcm. With 600 additional general, math, and code instructions, the epoch-10 mixed-source student improves accuracy from 0.058 to 0.156 relative to the base student on the original 500 SimpleQA questions, while detection AUROC falls from 0.791 to 0.588.
The submitted MATH experiment distills complete step-by-step teacher solutions from Llama-3.1-70B-Instruct into Llama-3.1-8B-Instruct, rather than final answers alone (Appendix E.1–E.2, Figure 11 and Table 4). Thus, the paper already tests SeqKD with reasoning traces as training targets. We will clarify this setup and distinguish it from the separate setting of dedicated reasoning models with an explicit thinking mode.
See W1 for Reviewer Fmcm. The new SimpleQA control compares K = 1 and K = 5 at the same teacher temperature T = 1, with 580 matched optimizer updates and three training seeds.