acceptodds
Under review as a conference paper at ICLR 2027

Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces

Abstract

Reasoning-trained language models can perform, zero-shot, multi-label tasks that require selecting a small set of relevant labels from a universe of thousands to hundreds of thousands of candidates. We ask how they do it mechanistically, and whether the mechanism can be distilled. We make the question measurable by treating each decision as a token-level event scored by the model's own decision margin: the token that picks a coarse region of the label space, the tokens that pick a label within it, and the token where the output departs from a close alternative (a near-miss) named earlier in the reasoning. Attribution, exact mean-ablation, knock-in into another example's context, and a null calibration that discounts generic heads then give individual attention heads causal standing. On clinical coding of hospital discharge summaries (MIMIC-IV), with all 5,651 candidate diagnosis codes in context, a small, global, phase-structured set of heads is necessary and sufficient, by ablation and knock-in, on essentially every summary; distinct head families attend to the candidate region and back to the near-miss named earlier; and, for the mentions decided in the reasoning, the region can already be elicited several tokens before the code, from a disjoint mid-layer set that reads the input. We introduce MISTILL: unlike chain-of-thought distillation, which transfers only the teacher's reasoning text, it also supervises the student's pooled attention at exactly these decision events. Read on heads found after training, it nearly doubles the causal recovery of the contrastive decision in a cross-family student and adds a small, seed-stable gain in one that already carries most of it, with no detected task difference when both objectives train bf16 weights and a task cost with fp32 master weights. Code: https://anonymous.4open.science/r/mistill-code-anon-3D07

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, attention heads, knowledge distillation, reasoning models, chain-of-thought distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W7SfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how reasoning-trained LLMs select a few labels from a very large label space, using ICD-10 coding of MIMIC-IV discharge summaries as a case study. In particular, it studies whether this internal mechanism can be distilled into smaller models. Each emitted code character is treated as a decision, and the authors identify which attention heads drive these choices by switching them off or porting them into another input. On Qwen3-14B, the authors report that a small set of heads, partly different per decision type, is necessary and sufficient for these decisions. They then propose MISTILL, which adds to chain-of-thought distillation a loss matching the student's pooled attention to the teacher's at the decision positions, and evaluate it on two smaller students.

Critical strengths
  • The experimental protocol is extensively documented.
Critical weaknesses

Major

  • W1. My main concern is the presentation. I have found the paper very difficult to follow. A non-exhaustive list of recurring issues:
    • A large amount of ad-hoc terminology is introduced, e.g., P1/P2/P2C/NULL events, "locus", "prose-time set", "emission set", "region readers", "near-miss loop-back", "span buckets", and the readouts M1–M5. When present, definitions are typically a single quick sentence without intuition; some terms are used before being defined, and others are only explained in the ~40 pages appendix.
    • Sentences are densely packed with numbers and qualifications, which prevents building intuition progressively. The abstract is an example.
    • Several internal identifiers, which also appear in the main text (e.g., "[R-C3]", "[R-D1]"–"[R-D17]", "HD-002").

I understand this could be personal preference, but I think the paper would need substantial rewriting to be accessible to the broad ICLR audience. I also have a few concern regarding the experimental evidence, I will list two below:

  • W2. If I understood correctly, the characterization does not seem to inform the distillation. On SmolLM3-3B, supervising the 40 heads that most oppose the decision yields at least as much gain as supervising the top-40 (0.841 and 0.835 against 0.789 and 0.774, Sec. 4.5). On Qwen3-4B, forty random heads reproduce MISTILL on one of two seeds. Moreover, the student's top heads after training are largely those of the untrained student. If the identified heads are not needed, it is unclear to me what the characterization contributes to the method. Please correct me if I misread these results.

  • W3. The paper claims that MISTILL transfers the teacher's mechanism without a cost in the task. However, it is unclear to me why transferring the mechanism is desirable in itself, as the paper does not motivate it (e.g., in terms of faithfulness, robustness, or interpretability?). Moreover, the "no cost" claim holds only against Vanilla CoT-KD trained at the same reduced precision: with fp32 master weights, MISTILL remains below DistiLLM-2 and OPD on Qwen3-4B (Table 2).

Minor

  • W4. Several results have a single seed.
Questions for authors
  • Q1 (related to W2). Could the authors clarify how the heads identified in the characterization contribute to MISTILL?
  • Q2 (related to W3). Why should a higher recovery in the student's own heads be desirable?

While I remain open to a constructive discussion, and I acknowledge that I may have misunderstood substantial parts of the paper, I believe it requires substantial revision in both presentation and the link between its two parts.

Reviewer_WvnmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the internal decision-making mechanisms of large language models (LLMs) in large label spaces and proposes a mechanism-aware distillation method called MISTILL. The authors divide the decision-making process in clinical coding into three stages: label-region selection (P1), specific code generation (P2), and discrimination between similar candidates (P2C). Through Attribution Patching, Mean Ablation, and Knock-in, the authors identify Attention Heads that causally influence different decision stages and demonstrate that some label decisions emerge before the final output. Based on these findings, MISTILL extends conventional Chain-of-Thought (CoT) distillation by additionally supervising aggregated Attention distributions at critical decision positions and their interactions with decision margins, aiming to transfer the teacher model's internal decision mechanisms. Experimental results demonstrate that MISTILL significantly improves the causal recovery of contrastive decisions in student models but does not achieve significant improvements in final task accuracy over conventional CoT distillation at comparable precision.

Critical strengths
  1. Rigorous causal analysis of Attention Heads. Rather than relying solely on Attention weight analysis, the authors combine Mean Ablation, Knock-in, and NULL Calibration to validate the causal influence of critical Attention Heads on model decisions.

  2. Insights into internal mechanisms across different reasoning stages. The authors demonstrate that label-region selection, fine-grained code generation, and candidate discrimination rely on different sets of Attention Heads. They also identify cases where decisions emerge before the final output, providing valuable evidence for understanding internal reasoning mechanisms in LLMs.

  3. A flexible cross-model mechanism distillation framework. MISTILL aggregates Attention distributions through Span Buckets, eliminating the need for one-to-one correspondence between teacher and student Attention Heads. The authors further demonstrate improvements in the causal recovery of contrastive decisions across different model architectures.

Critical weaknesses
  1. Unclear novelty relative to existing research. Existing studies already explore the transfer of internal computational mechanisms and the distillation of critical tokens. For example, Circuit Distillation [1] transfers internal computational mechanisms by aligning teacher and student computational circuits, while KPOD [2] improves CoT distillation through key-token weighting. Although this paper proposes a mechanism-aware distillation approach that combines causal Attention Head selection with supervision at critical decision positions, it lacks targeted comparisons with these existing methods. Consequently, the unique contribution and additional benefits of the proposed mechanism supervision remain insufficiently demonstrated.

    [1] Circuit Distillation. Wadhwa et al., 2025. https://arxiv.org/abs/2509.25002

    [2] KPOD: Keypoint-based Progressive Chain-of-Thought Distillation for LLMs. Feng et al., ICML 2024. https://proceedings.mlr.press/v235/feng24e.html

  2. Improvements in mechanism-level metrics do not translate into practical task-level gains. Although MISTILL significantly improves P2C Knock-in Recovery, its final Micro-F1 does not significantly outperform CoT-KD at comparable precision and even exhibits a noticeable degradation under the fp32 setting. This raises concerns about whether improvements in mechanism recovery metrics genuinely reflect enhancements in practical reasoning capabilities.

  3. Limited experimental scope and unclear generalizability. The experiments focus exclusively on a single clinical coding task, where the prompt itself explicitly encourages coarse-to-fine reasoning. Therefore, it remains unclear whether the identified internal mechanisms generalize beyond this particular task structure. Can the authors provide additional experiments on mathematical reasoning or coding tasks to demonstrate the generality of the proposed mechanisms?

Questions for authors

Please refer to the concerns and questions raised in the Weaknesses section.

Reviewer_MNghOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how a reasoning-trained language model makes multi-label decisions over a very large candidate space and whether the identified decision mechanism can be distilled into smaller models. The authors analyze Qwen3-14B on clinical diagnosis coding from MIMIC-IV, using the full inventory of 5,651 ICD-10-CM diagnosis codes. They define token-level decision events for selecting a coarse code region, selecting subsequent code characters, and diverging from a previously mentioned near-miss code. Attribution, exact mean-ablation, cross-context knock-in, and a null calibration are used to identify attention heads that causally affect the model’s own decision margin. The analysis finds phase-specific global head sets, a distinct set of mid-layer heads that can make some chapter decisions elicitable before the emitted code, and broadly similar behavior on a retrieved shortlist. The authors then propose MISTILL, which supplements chain-of-thought distillation with losses on pooled student attention and contribution at teacher-defined decision events. MISTILL substantially improves a causal contrastive-decision readout for SmolLM3-3B and yields a smaller mechanism gain for Qwen3-4B, while task-level effects depend strongly on training precision. The study is carefully scoped to one teacher family and one clinical coding task.

Critical strengths

1. A carefully constructed causal instrument for decisions in a large output space. The paper turns a difficult interpretability question into explicit token-level events and evaluates individual attention heads using the model’s own decision margin. The combination of attribution screening, exact mean-ablation, cross-context knock-in, count-matched random and bottom-head controls, and null calibration is substantially more informative than relying on attention visualization or gradient rankings alone. The global phase-specific sets are evaluated across essentially every characterization note, and the mechanism is replicated on a retrieved-shortlist presentation used for distillation. This provides a concrete and reproducible operational account of which components influence particular decisions, while the paper appropriately distinguishes causal write-side evidence from merely descriptive read-side attention patterns.

2. Transparent mechanism-focused distillation study with informative negative results. MISTILL is evaluated with separate task, fidelity, and mechanism metrics, and the authors include position, head-selection, phase, dose, precision, and training-set-size controls. The results are not presented as uniformly positive: the task benefit disappears under precision-matched bf16 training, the fp32 comparison favors Vanilla CoT-KD, several mechanism readouts do not separate the arms, and the attention read-side analysis is explicitly described as non-causal. The large gain on SmolLM3-3B’s own causal contrastive-decision readout, together with the smaller but reproducible Qwen3-4B effect, suggests that decision-localized supervision can influence an operationally defined causal carrier even when it does not improve the external task score.

Critical weaknesses

1. Insufficient Evidence for Mechanism Transfer Beyond Decision-Localized Supervision. The evidence does not sufficiently establish that MISTILL transfers the teacher's reasoning mechanism rather than primarily benefiting from decision-localized supervision. Its strongest result is the improvement in M4 causal recovery, measured on the student's post-training heads under teacher-forced traces, without demonstrating that the recovered mechanism influences free-running reasoning or task performance. Moreover, the head-selection ablations weaken the necessity of the proposed causal-head supervision: random-head controls approach MISTILL on Qwen3-4B, while the negatively attributed bottom-40 heads outperform MISTILL on SmolLM3-3B. Crucially, an event-weighted Vanilla CoT-KD baseline, which would isolate the benefit of emphasizing decision positions without transferring attention-level information, is missing. Furthermore, P2C events are defined by previously mentioned neighboring codes rather than independently validated semantic alternatives. Consequently, the results support changes in an operational causal readout but do not yet convincingly establish the transfer of a distinct semantic reasoning mechanism.

2. Limited Task-Level Benefits and Insufficient Generalization Evidence. The proposed distillation method does not demonstrate a reliable improvement in downstream task performance, and its evaluation remains too narrow to support broader claims. Under matched bf16 training, MISTILL provides no statistically significant micro-F1 improvement over Vanilla CoT-KD (0.149 vs. 0.143 on Qwen3-4B and 0.051 vs. 0.055 on SmolLM3-3B). With fp32 master weights, MISTILL performs significantly worse than Vanilla CoT-KD on Qwen3-4B (0.165 vs. 0.187), while its loss coefficients were not re-tuned under fp32. Training costs and baseline tuning budgets are also unmatched. Furthermore, the evaluation is restricted to one Qwen3-14B teacher, two relatively small students (3B/4B), and a single clinical coding task, with only 493 valid distillation training traces and 150 test notes. Several baselines and ablations use only one seed. Although the causal characterization is substantial, the limited model and task diversity, together with the absence of consistent task-level gains, leaves the robustness and general applicability of MISTILL insufficiently demonstrated.

Questions for authors
  1. Could the authors evaluate additional teacher families, more diverse student scales (e.g., 7B/8B or larger), and other large-output-space tasks to establish whether the identified causal mechanisms and MISTILL's improvements generalize beyond the current setting?

  2. Could the authors compare MISTILL with event-weighted Vanilla CoT-KD at the same decision positions? Given the competitive random-head and bottom-40 results, how can the gains be attributed to mechanism transfer rather than decision-localized supervision alone?

  3. Do the M4 improvements persist on students' freely generated reasoning traces and correlate with actual coding performance? Could re-tuning MISTILL under fp32 master-weight training resolve its significant performance gap with Vanilla CoT-KD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.