- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how reasoning-trained LLMs select a few labels from a very large label space, using ICD-10 coding of MIMIC-IV discharge summaries as a case study. In particular, it studies whether this internal mechanism can be distilled into smaller models. Each emitted code character is treated as a decision, and the authors identify which attention heads drive these choices by switching them off or porting them into another input. On Qwen3-14B, the authors report that a small set of heads, partly different per decision type, is necessary and sufficient for these decisions. They then propose MISTILL, which adds to chain-of-thought distillation a loss matching the student's pooled attention to the teacher's at the decision positions, and evaluate it on two smaller students.
Critical strengths
- The experimental protocol is extensively documented.
Critical weaknesses
Major
- W1. My main concern is the presentation. I have found the paper very difficult to follow. A non-exhaustive list of recurring issues:
- A large amount of ad-hoc terminology is introduced, e.g., P1/P2/P2C/NULL events, "locus", "prose-time set", "emission set", "region readers", "near-miss loop-back", "span buckets", and the readouts M1–M5. When present, definitions are typically a single quick sentence without intuition; some terms are used before being defined, and others are only explained in the ~40 pages appendix.
- Sentences are densely packed with numbers and qualifications, which prevents building intuition progressively. The abstract is an example.
- Several internal identifiers, which also appear in the main text (e.g., "[R-C3]", "[R-D1]"–"[R-D17]", "HD-002").
I understand this could be personal preference, but I think the paper would need substantial rewriting to be accessible to the broad ICLR audience. I also have a few concern regarding the experimental evidence, I will list two below:
-
W2. If I understood correctly, the characterization does not seem to inform the distillation. On SmolLM3-3B, supervising the 40 heads that most oppose the decision yields at least as much gain as supervising the top-40 (0.841 and 0.835 against 0.789 and 0.774, Sec. 4.5). On Qwen3-4B, forty random heads reproduce MISTILL on one of two seeds. Moreover, the student's top heads after training are largely those of the untrained student. If the identified heads are not needed, it is unclear to me what the characterization contributes to the method. Please correct me if I misread these results.
-
W3. The paper claims that MISTILL transfers the teacher's mechanism without a cost in the task. However, it is unclear to me why transferring the mechanism is desirable in itself, as the paper does not motivate it (e.g., in terms of faithfulness, robustness, or interpretability?). Moreover, the "no cost" claim holds only against Vanilla CoT-KD trained at the same reduced precision: with fp32 master weights, MISTILL remains below DistiLLM-2 and OPD on Qwen3-4B (Table 2).
Minor
- W4. Several results have a single seed.
Questions for authors
- Q1 (related to W2). Could the authors clarify how the heads identified in the characterization contribute to MISTILL?
- Q2 (related to W3). Why should a higher recovery in the student's own heads be desirable?
While I remain open to a constructive discussion, and I acknowledge that I may have misunderstood substantial parts of the paper, I believe it requires substantial revision in both presentation and the link between its two parts.