acceptodds
Under review as a conference paper at ICLR 2027

What Does Distillation Transfer? Causal Tests with Ground-Truth Teachers

Abstract

Distilled students can reproduce a teacher's answers without using the same computation. We test what transfers through logits, hidden states, and state traces using modular-addition and automaton teachers with measurable internal signatures. In modular addition, a model's wrong-answer probabilities carry the Fourier frequencies it computes with, and changing only those probabilities selects which frequencies a freshly initialized student uses. Exchanging these probabilities between teachers transfers the source teacher's causal frequency signature. Permuting them while preserving entropy and the correct-answer probability selects a different signature, predicted before training from the edited target's frequency content. Removing the predicted frequencies sharply reduces accuracy; retaining them preserves accuracy in the trained network. A student that already computes through one frequency set can later match a new target and still depend on the old set. Raising the loss weight on those same wrong-answer probabilities restores transfer from near-deterministic teachers. Program-generated automaton traces teach state tracking but contain no teacher identity at training lengths. Using 1.5B teachers and 0.5B students, the probabilities on tokens other than the reference token again determine which teacher's output signature the student acquires. Swapping the output projection moves most of the measured gap. Distillation can therefore select a causal internal signature, but matching the target does not guarantee that the student computes through it.

Primary area:
interpretability and explainable AI
Keywords:
knowledge distillation, mechanistic interpretability, causal interventions, activation patching, modular addition, grokking, reasoning traces, soft targets

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yVDJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose to distill models trained on several tasks, including modular addition, into student models and analyze the features of the students. In the case of modular addition, the teachers are confirmed to rely on input embeddings whose fourier frequency sets (derived from all number embeddings) are required to perform the models computation. They then ask whether the students use the same frequencies. Three types of distillation are done, direct output distribution, hidden state matching, and using direct prediction outputs (for DFA trained teachers) from the teacher as training data for the students.

The main empirical contribution of the work is showing that the mass distribution on incorrect answers impacts what frequency features the student number embeddings use. This is shown by taking two teachers, A and B, and combining their distributions such that the rescaled wrong answer distribution from B combined with the right answer probability for A leads to a student model whose frequencies resemble those of B. And they show that deterministically shuffling the probabilities of the non-correct answer can predict what fourier features a student picks up in the number embeddings. Subtracting those frequencies leads to a performance drop.

Critical strengths

The core experiments seem reasonably clear and comprehensive as described in the summary, confirming that for an otherwise performing model that the frequencies associated with the tail distribution (bar the mass on the correct answer) have a causal relationship with the choice of embedding frequencies. Connecting the distillation itself, i.e. the output distribution of the parent model with the finding that the internal frequencies of the trained student model match those of the parent is sound and interesting.

Critical weaknesses

The paper could be better written in many places. In particular the abstract immediately stands out as particularly opaque to someone not already familiar with the work. There is also a bit of an overstatement (“A trained network solves this task by rotation”) in terms of asserting that modular addition always happens with the same procedure (a paper is even cited where this is not the case, Zhong et al.)

Questions for authors

None

Reviewer_H5EgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether distillation transfers a teacher’s internal mechanism rather than merely its outputs. Using modular addition and finite-state automata with verifiable ground-truth mechanisms, the authors compare logit, hidden-state, and trace-based distillation. The strongest result is that the incorrect-class probability tail can select the Fourier-frequency subspace used by a fresh student: swapping or remapping the tail moves the student to the predicted frequency set, and ablating that set destroys accuracy. The paper also separates output matching, decodability, causal use, teacher identity, and program transfer. A Qwen2.5-0.5B/GSM8K experiment provides only preliminary evidence for output-signature transfer at pretrained scale.

Critical strengths
  1. The paper addresses a fundamental question with unusually strong causal controls: ground-truth teachers, independent-teacher nulls, structural fingerprints, and interventions.
  2. The tail-remapping experiment is compelling: it predicts the acquired frequency set, and removing that set causes a 0.94–0.99 accuracy loss (Sec. 4, Fig. 1).
  3. It clearly demonstrates that behavioral agreement and linear decodability do not imply inheritance or causal use of the teacher’s mechanism.
Critical weaknesses
  1. The strongest claims come from synthetic, highly instrumented tasks. The pretrained experiment identifies output signatures, not a transferred reasoning circuit, and has low GSM8K accuracy.
  2. “Mechanism” is partly operationalized as a causal Fourier representation signature, which may not characterize the complete algorithm or circuit.
  3. Some negative results reflect limited channel information or loss weighting: a channel that does not contain teacher identity cannot be expected to recover it.
  4. Statistical reporting is heterogeneous, often using 3–5 seeds and ranges rather than consistent effect sizes and confidence intervals.
Questions for authors
  1. Which experiments establish transfer of a complete algorithm, versus only a causal representation signature?
  2. Does tail remapping work with architecturally different students or an independent mechanism metric not derived from the Fourier fingerprint?
  3. Can the authors provide unified seed-level effect sizes and confidence intervals for the main preregistered comparisons?
  4. Can the pretrained bridge be repeated at materially higher task accuracy, or be framed more explicitly as preliminary output-signature evidence?
Reviewer_ivGNOpenReview ↗
Rating
1
Confidence
4(+2)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the alignment of output distributions in knowledge distillation is accompanied by the transfer of internal features or computational processes from the teacher, and compares three types of supervisory signals: logits, hidden states, and state traces. The authors construct teachers with analyzable internal features for the modular addition and DFA state tracking tasks, and combine Fourier analysis, representation probes, activation patching, and feature ablation to examine the representations learned by the student and their actual role in prediction.

Critical strengths
  1. In the modular addition experiment, after the author altered the probability distribution of incorrect answers, the student model began to rely on different Fourier frequencies; further interventions confirmed that these frequencies do indeed influence predictions. Experiments involving continued training further demonstrated that adapting to a new tail can occur simultaneously with the preservation of the original frequency dependencies. These results help distinguish between output mimicry and changes in internal features.
Critical weaknesses
  1. The current manuscript requires a comprehensive restructuring centered on the core issue. The transfer objects measured in each experiment differ, and their interrelationships and levels of argumentation are not sufficiently clear; the correspondence between the main conclusions and the evidence is difficult to trace, which impedes the assessment of the paper’s scientific contribution. This goes beyond what can be resolved through minor textual revisions or clarifications in a rebuttal; therefore, I do not support accepting the current version.
  2. The DFA’s causal test simultaneously patches both the source state and the symbol, making it impossible to distinguish whether the student executed the transition function or directly received the next state of the encoded source. Consequently, the causal evidence supporting the claim that “traces taught the procedure” remains insufficient.
Questions for authors
  1. Have you conducted a control experiment in which you manipulated “state” and “symbol” separately? For the rearranged state–symbol pairs, do the model outputs match the transition table?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.