acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Causal Distillation: Selecting Sufficient Causal Supervision

Abstract

Distilling a large language model (LLM) requires choosing which aspects of its computation a smaller student should preserve. Responses to interventions on intermediate variables and computational routes make this choice testable. We propose Hierarchical Causal Distillation (HCD), based on the principle that a distilled model should preserve a compact causal structure sufficient for the required teacher responses. HCD asks which supervision objectives enable a student to retain these responses under intervention. HCD organizes supervision around behavior, high-level reasoning programs, their circuit realizations, and internal states. It trains students with different combinations of objectives and evaluates them against common task-specific behavioral and interventional requirements. Cross-model correspondences relate semantic variables and circuit roles, allowing teachers and students with different architectures to preserve the same required responses. We formalize the choice of causal granularity using Unified Causal Minimum Description Length (UCMDL), which selects the shortest adequate joint description of a teacher mechanism and cross-model alignment within a fixed candidate family. All candidates face the same intervention requirements. A finite-family confidence bound defines adequacy on independent selection data before description-length comparison. We report results on two controlled transformers with known programs and pretrained Qwen3.5 students on seven generated arithmetic tasks. In the controlled study, HCD selects program-and-circuit supervision (BPC) on both tasks. The evaluated BPC students achieve lower ordinary and intervention response errors than behavioral KD. The independently trained frequency-sorting student also achieves low state-intervention error without a separate state objective in student training. In the pretrained comparison, BPC improves base-answer accuracy by 30.0% relative to behavioral KD and achieves the lowest pooled mean teacher–student total variation on ordinary, program-intervention, and circuit-intervention queries. These results support program and circuit supervision for preserving the evaluated teacher responses during compression.

Primary area:
causal reasoning
Keywords:
hierarchical causal distillation, knowledge distillation, causal representation learning, mechanistic interpretability, causal abstraction, neural network compression, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_tJg5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HCD organizes behavioral, program, circuit, and state supervision and selects an adequate teacher-student correspondence by description length. Controlled tasks and generated arithmetic show promising intervention agreement, but the central selection claim and the added value of causal supervision need stronger empirical support.

Critical strengths

S1. The research question is well motivated: ordinary output agreement need not preserve responses to changes in intermediate computations.

S2. The framework separates training objectives from common evaluation requirements and explicitly restricts the correspondence family, improving conceptual clarity.

Critical weaknesses

W1. The key selection contribution is not empirically demonstrated in sufficient detail. Candidate code lengths, selection-error bounds, and adequacy decisions are missing, so it is unclear whether UCMDL meaningfully selects among several adequate alternatives.

W2. The incremental benefit over existing causal distillation is unresolved. Comparisons with behavioral KD alone do not establish what hierarchical selection adds beyond interchange-intervention or circuit distillation.

W3. The pretrained experiments are difficult to reproduce and assess statistically. Exact intervention sites and selected circuits, dataset sizes, optimization details, per-task results, and run-to-run uncertainty are not specified adequately in Appendix D.

W4. The evidence supports only a narrow form of sufficiency. Full selection is tested on two known-program tasks, and state-response evaluation includes fitting alignments to teacher state targets. Broader transfer claims require separating learned student behavior from post-training correspondence fitting.

Questions for authors

See Weaknesses

Reviewer_SGt6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies which parts of a teacher’s computation a student should preserve during knowledge distillation. It defines four types of supervision: behavior (B), program (P), circuit (C), and state (S). HCD trains students with different combinations of these objectives, evaluates them under shared behavioral and intervention requirements, and uses UCMDL to select a compact adequate realization.

The experiments cover two Tracr-controlled transformer tasks and seven generated arithmetic task families with Qwen3.5 students. HCD selects BPC in the controlled setting. In the pretrained comparison, BPC obtains higher accuracy and lower teacher–student total variation than B and BP.

Critical strengths
  1. The paper asks a clear and important question: ordinary behavioral KD may match final outputs without preserving how the student responds when an intermediate variable or computational route is intervened on. Separating behavior, program, circuit, and state supervision gives a useful vocabulary for this question.

  2. In the evaluated tasks, BPC students obtain lower ordinary and intervention response errors than behavioral KD. In the generated arithmetic experiments, BPC also improves accuracy and teacher–student TV over B and BP, suggesting that the additional supervision can be useful in the studied setting.

Critical weaknesses
  1. The central contribution is selecting sufficient causal supervision, but the main results primarily show that BPC outperforms behavioral KD. Although the paper defines B, BP, BPC, BPS, and BPCS as candidate configurations, it does not present a complete comparison of their adequacy, UCMDL values, and independent test errors. It is therefore difficult to tell whether the gain comes from the HCD selection procedure or from using the BPC supervision itself.

  2. The pretrained experiment does not run the full HCD selection procedure and compares only B, BP, and BPC. The authors acknowledge that the objectives use different training query sets and leave matched query budgets and executable counterfactual evaluation for future work. The paper also lacks direct comparisons with relevant causal-distillation methods such as Causal Distillation for Language Models (Wu et al., 2022, which introduces DIITO) and Circuit distillation (Wadhwa et al., 2025). Thus, the current evidence more clearly supports the usefulness of BPC on the evaluated tasks than the independent value of HCD as a selection framework.

Questions for authors
  1. Could you report the adequacy status, UCMDL value, response errors, and independent test results for every controlled-study candidate, including B, BP, BPC, BPS, and BPCS? This would clarify why HCD selects BPC.

  2. Under matched training-query numbers and supervision budgets, how do B, BP, BPC, BPS, BPCS, and relevant causal-distillation baselines compare? This would help separate the contribution of the HCD selection rule from the benefit of receiving additional intervention supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.