- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The objective of this paper is to distill the decision-making capabilities of a large reasoning-capable LLM into a smaller encoder model for repeated classification tasks. The teacher model produces probability distributions over possible labels. A smaller student (ModernBERT-base) learns from these soft labels alongside ground-truth labels. The student also supports dynamic candidate sets and a "None" option. The teacher operates with reasoning disabled. The paper proves that distilling an LLM's probability distributions into a small encoder improves probability quality but not classification accuracy compared with standard supervised training.
Critical strengths
- The paper proves that distilling an LLM's probability distributions into a small encoder improves probability quality but not classification accuracy compared with standard supervised training.
- Ordinary classification assumes that a model always chooses from a fixed list of labels, while this paper studies a more flexible problem: the list of available choices can change at runtime.
Critical weaknesses
I think one of the main weaknesses is that in the simulations, the teacher has no meaningful performance advantage over the student. Another weakness is that the teacher's reasoning capabilities are never used since its thinking is disabled.
Questions for authors
- What is actually being transferred if there is no teacher headroom?
- Why is reasoning disabled if the goal is to do reasoning distillation?
- What do you consider the main novelty of the work beyond the controlled evaluation of existing techniques?
Thank you for the review. The general response (GR) lists the shared changes. One note on the summary: the paper reports measurements on one dataset, not proofs, and the revised text is worded to match.
W1 (no teacher advantage). Agreed. The submitted teacher was less accurate on the test set than both ground-truth students (0.857 against 0.910 and 0.924). On the tuning cohort, its net against a clean GT-CE student was -0.115. To test whether headroom matters, we added the same model with thinking enabled (T1) and a smaller teacher (T2), and we now measure test-set headroom with 95% intervals for every teacher. None had headroom: against the better base student, T1 was -4.7 points [-5.5, -3.8] and T2 -36.0 [-37.2, -34.6]. Every teacher was also below both ModernBERT-large students. Headroom did not order the conditions either: T1 had the smallest deficit but did not improve on KD(T0) (Section 6).
W2 (thinking disabled). Accepted; see Q2.
Q1 (what is transferred without headroom). Calibrated probability structure, not accuracy. The evidence comes from the thinking-off teacher on CLINC150 (Section 5.1, Section 6):
This agrees with reports that weak teachers can help (Yuan et al. 2020). The new conditions (99% intervals) refine this reading:
Q2 (why reasoning was disabled). The submission scored the teacher's direct judgment. With thinking disabled, one forward pass gives the full distribution over the 151 identifier tokens, which is the soft target KD needs. We agree this does not test reasoning, and the title claimed otherwise; it is changed (GR 4).
T1 now enables thinking: one sampled trace of at most 4,096 tokens, followed by the same constrained scoring. All 18,088 records completed and none reached the cap. The students never see the reasoning. T1 was more accurate than T0 (0.877 vs 0.857; headroom -4.7 points [-5.5, -3.8], 95%) but near one-hot (training-split mean entropy 0.004 vs 0.297 nats). KD(T1) was 0.73 points less accurate than KD(T0) [-1.27, -0.19], and no scaled NLL difference was detected. KD(T1c) was worse than KD(T0) after scaling (Section 5.2, Table 1).
Q3 (main novelty). No new loss or training method is claimed.
Section 2 discusses the closest studies, each of which covers only part of this combination. See also our reply to YGfS W1.