acceptodds
Under review as a conference paper at ICLR 2027

Distilling Answers, Discarding Uncertainty: Uncertainty Collapse in Sequence-Level Knowledge Distillation

Abstract

Sequence-level knowledge distillation (SeqKD) commonly trains a student on a single sampled teacher response per input. While this transfers the teacher’s answer, it need not transfer the uncertainty with which that answer was produced. We study the resulting uncertainty collapse and its consequences for hallucination detection. In expectation, the SeqKD objective is minimised by matching the teacher’s response distribution and therefore preserves its uncertainty. But with one fixed sampled target per input, the empirical objective instead favours concentration on that target, regardless of whether it is correct. We show theoretically that a single response cannot identify the teacher’s dispersion over alternative meanings, and that training provides a systematic pressure towards increasingly concentrated student output distributions. Across four teacher–student pairs on SimpleQA, continued distillation causes students to increasingly reproduce hallucinated teacher responses while becoming more certain of them: semantic-entropy AUROC for detecting incorrect answers falls from roughly 0.8 to chance, even when answer accuracy changes little. The effect also appears, more weakly, on held-out and out-of-distribution data, and recurs in mathematical reasoning. We evaluate filtering, abstention replacement, and multi-response distillation as mitigations. Among the interventions we study, retaining multiple teacher responses directly preserves distributional information and substantially maintains hallucination detectability, at a cost that grows with the number of responses. These results show that transferring a teacher’s outputs and transferring its uncertainty are distinct objectives, and that reliable distillation requires preserving more than a single draw from the teacher’s response distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, hallucination, uncertainty quantification, semantic entropy, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qe6FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates uncertainty collapse in sequence-level knowledge distillation (SeqKD), where a single fixed teacher response is used for each input. The authors show that continued distillation can increase student hallucinations and reduce the semantic-entropy AUROC for detecting incorrect answers to near-chance levels. In contrast, using multiple teacher responses can better preserve the teacher's output distribution and maintain the student's uncertainty awareness.

Critical strengths

The paper clearly formulates and analyzes a fundamental objective mismatch in standard SeqKD. The point-mass optimum and non-identifiability propositions provide concise theoretical explanations for why a single teacher response is insufficient to preserve the teacher's output distribution.

Critical weaknesses
  1. The information advantage of soft probability targets over hard labels is well established in classical knowledge distillation, including MNIST experiments [1]. However, the paper does not sufficiently distinguish its contribution from this established principle or ordinary hard-target overfitting. For example, the decline in Llama-3.1 AUROC from 0.839 at epoch 2 to 0.543 at epoch 10 may simply reflect repeated cross-entropy training on a small, fixed dataset. It remains unclear whether this degradation would persist under practical distillation budgets or with validation-based early stopping.
  2. The main experiments train on teacher-generated answers to 500 SimpleQA questions and evaluate on the same questions, making it difficult to distinguish memorization from generalizable uncertainty loss. Although held-out evaluations are included, experiments with independent training corpora, multi-source instruction tuning, and reasoning-enabled distillation are lacking. Moreover, the SimpleQA mitigation experiments confound response diversity with sampling temperature (τ = 0.1 for single targets versus τ = 1.0 for multiple targets). The matched-temperature MMLU-Pro control does not fully resolve this confound for SimpleQA.

[1] Distilling the Knowledge in a Neural Network, 2015.

Questions for authors
  1. The loss of distributional information when replacing probability targets with hard labels is already well established. What new insights does this work offer beyond classical knowledge distillation and ordinary overfitting to fixed targets?
  2. Training for 10–20 epochs on only 500 fixed questions is a highly repetitive setting, which is not a practical distillation setting. The AUROC decline from 0.839 to 0.543 may primarily reflect overfitting.
  3. Using evaluation questions as distillation training data limits the ability of the main experiments to assess generalization, as the student is trained on teacher answers to the same questions used for evaluation. The appendix's held-out results do not fully address this limitation. Please use an independent training corpus and a strictly held-out benchmark for the main experiments.
  4. Multi-source data mixing is common in instruction tuning but is not evaluated here. Repeatedly training on a small, isolated distillation corpus does not establish whether the same uncertainty collapse occurs when teacher responses are mixed with other instruction data.
  5. The experiments do not cover dedicated reasoning models or distillation with thinking mode enabled.
  6. The SimpleQA mitigation experiments vary both response diversity and sampling temperature (τ\tau = 0.1 versus τ\tau = 1.0), making it difficult to attribute the improvements specifically to diversity. Please compare single and multiple responses at the same temperature under matched training budgets.
2 comments

We thank the reviewer for the detailed comments. We clarify the contribution and address the experimental concerns point by point.

W1 & Q1: What is new beyond classical KD and repeated hard-target training?

Classical KD establishes the information advantage of distributional targets. Our contribution is to measure its reliability consequence in open-ended factual SeqKD: on the questions used for distillation, factual accuracy improves while semantic uncertainty becomes less able to distinguish answers that remain wrong. This semantic uncertainty collapse is the outcome we study, distinct from a conventional train–held-out performance gap. From the base Llama student to epoch 5, accuracy rises from 0.058 to 0.154 while detection AUROC falls from 0.791 to 0.647 (Section 5.3, Figure 3; Appendix D.1, Table 1). Propositions 1–3 connect this measured effect to the single-response SeqKD objective (Section 4).

Q2: Does the AUROC decline require an impractically long training budget?

The accuracy–detectability separation is already present at epoch 5; epochs 15–20 illustrate its later approach to chance. Validation-based early stopping preserves AUROC but gives up much of the later accuracy gain (see W2 for Reviewer py7B). The mixed-source experiment also shows the effect when the teacher targets are combined with other instruction data (see W2 for Reviewer Fmcm).

W2: Same-question evaluation, broader training, reasoning, and sampling temperature.

The same-question evaluation measures the paper’s primary in-corpus semantic uncertainty finding. The submitted held-out SimpleQA, TriviaQA, MMLU-Pro, and MATH evaluations measure its extension to unseen questions (Section 5.4; Appendix F.1–F.2, Tables 5–7; Appendix E.1–E.2, Table 4). We additionally tested mixed-source training (W2 for Reviewer Fmcm) and a same-temperature SimpleQA control (W1 for Reviewer Fmcm). The submitted MATH experiment distills full step-by-step teacher solutions, rather than final answers alone (Appendix E.1).

Q3: Independent training data and a strictly held-out benchmark.

Beyond the held-out evaluations in the submitted paper, we evaluated a separate 1,000-question SimpleQA set used for neither training nor validation in the new controlled runs. On this strictly held-out set, hard-target and matched-update K = 5 detection AUROCs are 0.700 and 0.713, respectively. The mixed-source experiment adds 600 general, math, and code instruction examples from independent sources to the SimpleQA teacher targets (see W2 for Reviewer Fmcm).

Q4: Mixing teacher responses with other instruction data.

See W2 for Reviewer Fmcm. With 600 additional general, math, and code instructions, the epoch-10 mixed-source student improves accuracy from 0.058 to 0.156 relative to the base student on the original 500 SimpleQA questions, while detection AUROC falls from 0.791 to 0.588.

Q5: Reasoning-enabled distillation.

The submitted MATH experiment distills complete step-by-step teacher solutions from Llama-3.1-70B-Instruct into Llama-3.1-8B-Instruct, rather than final answers alone (Appendix E.1–E.2, Figure 11 and Table 4). Thus, the paper already tests SeqKD with reasoning traces as training targets. We will clarify this setup and distinguish it from the separate setting of dedicated reasoning models with an explicit thinking mode.

Q6: Response diversity versus sampling temperature.

See W1 for Reviewer Fmcm. The new SimpleQA control compares K = 1 and K = 5 at the same teacher temperature T = 1, with 580 matched optimizer updates and three training seeds.

Thank you for the additional experiments and clarifications. However several concerns remain insufficiently addressed. In particular, the distinction between uncertainty collapse and ordinary har-target overfitting is still unclear, the generalization remain limited. Additionally, distilling step-by-step solution does not fully address distillation from dedicated reasoning models.

Overall while the rebuttal improves the paper, it does not sufficiently resolve my fundamental conern regarding the meaning and novelty of the study and experimental valiation. I maintain my original rating.

Reviewer_FmcmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates uncertainty collapse in single-response sequence-level knowledge distillation (SeqKD) and its impact on hallucination detection. Experiments across four teacher–student pairs show that continued distillation reduces semantic entropy and detection AUROC, particularly on the training corpus. The authors also provide theoretical analysis and evaluate several mitigation strategies.

My main concern is that the proposed explanation is not fully supported. In the main SimpleQA interventions, single-response targets are generated at temperature 0.1, which strongly favors high-probability answers, while multi-response targets use temperature 1.0, allowing greater answer diversity. This makes it unclear whether the improvement comes from using more responses or simply from changing the sampling distribution. Moreover, the theory shows that fitting a fixed target favors concentrated predictions under simplified assumptions, but does not establish that the same effect necessarily extends to unseen questions. The held-out results are also considerably weaker and task-dependent.

Critical strengths
  1. The paper identifies a meaningful failure mode that accuracy alone would miss.
  2. The paper makes a reasonable effort to test the phenomenon beyond a single configuration.
Critical weaknesses
  1. The main multi-response experiments do not fully isolate the effect of preserving distributional information. In the main SimpleQA experiments, the single-response distillation targets are generated at temperature 0.1, whereas the multi-response targets are sampled at temperature 1.0. The semantic entropy used for evaluation is also estimated at temperature 1.0.
  2. The strongest evidence comes from repeated training and evaluation on the same small corpus, while the generalization results are much weaker. The main experiment repeatedly fine-tunes the student for up to 20 epochs on approximately 500 fixed SimpleQA questions and evaluates it on those same questions. Under this setup, increasing confidence in the training targets is also a natural consequence of fitting a small dataset for many epochs. Besides, the theoretical analysis is consistent with this behavior, but it does not fully distinguish the proposed information-loss mechanism from ordinary overfitting under deterministic supervision. I would like to see more evidence that the observed failure mode remains substantial under a more typical distillation setting, rather than mainly reflecting prolonged fitting to a small fixed corpus.
Questions for authors

Can the authors provide a controlled comparison of single-response and multi-response distillation at the same teacher sampling temperature and with matched optimizer updates?

How sensitive is the uncertainty collapse to the training corpus size and the choice of training checkpoint?

Do the multi-response students actually preserve the teacher's per-question semantic uncertainty distribution, beyond maintaining detection AUROC?

1 comment

W1 & Q1: Same teacher temperature and matched optimizer updates.

The original SimpleQA comparison used T = 0.1 for K = 1 and T = 1 for multiple responses. We therefore compared K = 1 and K = 5 at T = 1 on the same questions, with 580 matched updates (20 epochs × ⌈456 targets / batch size 16⌉) and three seeds. Their AUROCs are 0.543 ± 0.002 and 0.752 ± 0.018, respectively. Teacher temperature alone therefore does not explain the multi-response advantage. Evaluation sampling uses T = 1 throughout.

For reference, the teacher’s accuracy/AUROC is 0.171/0.766, and the base student’s is 0.058/0.791. Each entry below gives accuracy/AUROC on the original 500 questions:

Teacher responsesEpoch 2Epoch 5Epoch 10
K = 10.088/0.8600.136/0.6440.141/0.541
K = 20.109/0.7840.135/0.7060.147/0.696
K = 50.145/0.7860.157/0.7600.145/0.785

The K = 2/5 curves reuse our original T = 1 runs; K = 1 is the new T = 1 control. The three-seed comparison above separately matches optimizer updates.

W2: A more typical distillation setting.

Our finding concerns semantic uncertainty on the distillation questions: factual errors remain while their uncertainty becomes less useful for detecting them (see W1 for Reviewer py7B). Inspired by the math, code, and general-domain composition in DeepSeek-R1, Appendix B.3.3, Table 5, we mixed 456 SimpleQA teacher targets with 200 general, 200 math, and 200 code instruction-response pairs. The added responses fit the original 1,024-token limit. Students at epochs 2, 5, and 10 were trained independently from the same base. On the original 500 SimpleQA questions:

ModelAccuracyDSE AUROC
Teacher0.1710.766
Base student0.0580.791
Mixed, epoch 20.1020.862
Mixed, epoch 50.1460.757
Mixed, epoch 100.1560.588

From the base student to epoch 10, accuracy rises from 0.058 to 0.156 while AUROC falls from 0.791 to 0.588, showing the in-corpus effect with mixed instruction data.

Q2: Corpus size and checkpoint selection.

We tested 200, 456, and 928 teacher-answer targets. The 200 targets are a subset of the original 456; the 928 add 472 teacher-answered questions from the disjoint 500-question SimpleQA set in Appendix F.1 (28 teacher abstentions). We trained each budget independently from the same base and evaluated on its distillation questions. Entries give accuracy/AUROC:

TargetsEpoch 2Epoch 5Epoch 10
2000.110/0.7740.165/0.6270.165/0.503
4560.114/0.8240.169/0.6090.171/0.505
9280.116/0.7670.177/0.6030.183/0.501

On the original 456 questions, base-student accuracy/AUROC is 0.057/0.785. AUROC approaches chance at all three sizes. Accuracy here includes abstentions in its denominator; AUROC uses attempted answers. For validation-based checkpoint selection, see W2 for Reviewer py7B.

Q3: Preservation of per-question teacher uncertainty.

We jointly clustered ten fresh teacher and ten student responses per question at T = 1, then computed mean per-question Jensen–Shannon divergence (JSD; lower is closer). On the original 500 questions:

EpochK = 1K = 2K = 5
20.7880.7400.676
50.6590.6020.541
100.6480.5760.504
150.6440.5650.499
200.6430.5670.505

K = 5 is closest to the teacher at every budget. At epoch 10, the paired K = 5 minus K = 1 difference is −0.145 bits (question-bootstrap 95% CI [−0.166, −0.123]); the independent teacher-sample reference is 0.412 bits. The same K = 5 < K = 2 < K = 1 ordering holds on the 456 distillation questions.

Reviewer_py7BOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that sequence-level knowledge distillation with a single teacher response per input transfers the teacher’s answer but not its uncertainty. The authors show theoretically that the empirical objective is minimised by a point mass on the recorded target. Empirically, across four teacher–student pairs on 500 SimpleQA questions, continued distillation makes students reproduce the teacher’s hallucinations with near-zero semantic entropy. Semantic-entropy AUROC drops from around 0.8 to chance while accuracy barely changes. Retaining multiple teacher responses preserves detectability best among the mitigations tested.

Critical strengths

The main strength is showing that hallucination detectability can collapse while accuracy stays flat, with robustness checks across model families, prompts and entropy estimators.

Critical weaknesses

The setting is closer to memorisation than to realistic distillation. Training on 500 questions for up to 20 epochs and evaluating on the same questions almost guarantees the collapse. On held-out data, the closest setting to practice, the effect is weak or absent: Llama’s held-out SimpleQA AUROC only drops from 0.774 to 0.705, and held-out MATH shows no decline at all.

The novelty is also limited. Propositions 1 and 2 are standard facts about fitting a single observation. The mitigations are not compared with simple baselines such as early stopping or label smoothing. Finally, with accuracy around 5% for most pairs and a single seed per setting, the AUROC estimates rest on few correct answers and report no variance.

Questions for authors

N/A

1 comment

We thank the reviewer for the careful assessment. We address the two concerns below.

W1: Repeated training on the evaluation questions may explain the collapse; held-out effects are weaker.

Our primary question concerns the factual questions used for distillation. We expect training to improve performance on these questions; instead, accuracy improves while semantic uncertainty becomes less useful for detecting the errors that remain. The held-out experiments assess how far this in-corpus effect extends: Llama’s held-out SimpleQA AUROC falls from 0.774 to 0.705, while held-out MATH shows no sustained decline (Section 5.4; Appendix F.2, Tables 5–7; Appendix E.2, Table 4).

The teacher makes factual errors, but its DSE detects them (AUROC 0.766). On the original 500 SimpleQA questions, from the base student to epoch 5, accuracy rises from 0.058 to 0.154 while detection AUROC falls from 0.791 to 0.647. At epoch 10, accuracy is 0.158 and AUROC is 0.543 (Section 5.3, Figure 3; Appendix D.1, Table 1). The accuracy increase and AUROC decline are already evident at epoch 5; epochs 15–20 show the subsequent approach to chance-level detection.

To test whether the effect extends beyond the exact training prompts, we independently retrained the students and evaluated meaning-preserving rephrasings that were never used in training. We reevaluated the original questions in these runs, explaining their small numerical differences from the submitted results:

StudentOriginal accuracyOriginal AUROCRephrased accuracyRephrased AUROC
Base0.0670.7830.0620.707
Epoch 20.1110.7900.0990.809
Epoch 50.1560.6380.1530.651
Epoch 100.1580.5430.1550.571
Epoch 150.1560.5450.1490.557
Epoch 200.1560.5460.1530.548

Accuracy is among attempted answers; AUROC excludes abstentions. The similar trajectories show that the finding extends beyond the exact wording of the training prompts.

W2: Limited novelty, missing simple baselines, low accuracy, and single-seed results.

Propositions 1 and 2 formalize the objective mismatch. Our empirical contribution is its consequence for open-ended factual SeqKD: gold-answer accuracy can improve while uncertainty-based detection of the remaining errors deteriorates (Section 4; Section 5.3, Figure 3).

We added both requested baselines in the Llama setting. The 456 teacher-answer targets and effective batch size of 16 give 29 optimizer updates per epoch; the 20-epoch budget gives 580 updates. Over three independent training seeds, the 580-update single-response student obtains accuracy 0.157 ± 0.001 and AUROC 0.544 ± 0.001. Validation-based early stopping selects update 29 and yields accuracy 0.097 ± 0.002 and AUROC 0.810 ± 0.010: it preserves detectability but forgoes much of the later factual-accuracy gain. Answer-token label smoothing at ε = 0.05 yields accuracy 0.157 ± 0.001 and AUROC 0.545 ± 0.001 after 580 updates.

We chose the difficult factual SimpleQA benchmark to provide many errors for analysis and room for accuracy gains. With independent training seeds 42, 43, and 44, the final single-response runs have 78, 78, and 79 correct answers out of 500; their AUROCs are 0.544, 0.544, and 0.545. The main observation is stable across these runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.