acceptodds
Under review as a conference paper at ICLR 2027

Computation Alignment for Chain-of-Thought Distillation

Abstract

Chain-of-thought (CoT) distillation transfers reasoning capabilities from large teacher models to smaller student models by training on teacher-generated rationales. However, existing methods primarily align observable outputs or intermediate representations, leaving a fundamental question unaddressed: whether the student's computation actually follows the reasoning process prescribed by the teacher's CoT. We identify this overlooked problem as computation alignment and formalize it as the overlap between computational circuits underlying intermediate reasoning steps and final-answer generation, recovered via multi-step circuit tracing. Directly optimizing circuit overlap during distillation, however, is computationally prohibitive. To address this, we propose Computation-Aligned Distillation (CAD), which exploits the sensitivity discrepancy between reasoning and answer circuits to construct a lightweight proxy. Across diverse settings, CAD consistently improves distillation performance over corresponding baselines without it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Chain-of-Thought Distillation, Computation Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D7VsOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Computation Aligned Distillation (CAD), which aims to align the internal computation of CoT reasoning steps and final-answer generation. The authors define computation alignment through circuit overlap and introduce a perturbation-sensitivity proxy for training. Experiments across different distillation objectives, teacher/student models, and reasoning datasets show improvements over the corresponding baselines.

Critical strengths
  1. The paper defines computation alignment through the overlap between reasoning and answer circuits, and proposes a perturbation-based proxy that can be optimized without expensive circuit tracing during training. The reported correlation between the proxy and actual circuit overlap provides initial support for this approach.

  2. CAD improves several existing distillation methods. For example, DFT improves from 56.0% to 63.0% on SVAMP, while Classic CoT improves from 52.78% to 87.20% on MultiArith. The comparison with typo augmentation also shows that the gains cannot be explained by simple input augmentation alone.

Critical weaknesses
  1. The paper defines computation alignment as circuit overlap in Eq. 9, but CAD actually optimizes the ratio of hidden-state perturbation sensitivities in Eqs. 10–12. Similar sensitivity magnitudes do not necessarily imply overlapping computational circuits. Although Figure 3 reports a strong correlation on Llama-3.1-8B-Instruct, the paper does not demonstrate that CAD training increases actual circuit overlap in the student models. It also does not rule out more general representation regularization effects. Therefore, the experiments show performance improvements but do not sufficiently establish that these improvements come from the proposed computation alignment mechanism. This is a major concern given the paper's central claim.

  2. In Table 1, Classic CoT and GEM achieve only 18.94% and 19.55% on GSM8K, both below the undistilled model's 24.73%. CAD substantially improves these weak baselines, but the gains over the stronger DFT baseline are much smaller. The paper uses common training hyperparameters across different objectives and reports only one training seed. Without better-tuned baselines and multi-seed results, it is difficult to determine how reliable the smaller improvements are.

Questions for authors
  1. Can the authors measure the actual circuit overlap defined in Eq. 9 before and after CAD training on Qwen2.5-0.5B and Gemma-2-2B? Does increased circuit overlap correlate with improved reasoning accuracy? This is essential for validating the central claim.

  2. Figure 3 evaluates the relationship between sensitivity discrepancy and circuit overlap on Llama-3.1-8B-Instruct. Does this correlation hold for the student models used in distillation, both before and after training? Since Eq. 11 only compares response magnitudes, how can the authors distinguish shared computational pathways from different circuits with similar sensitivities?

  3. How does CAD compare with general hidden-state consistency regularization or sensitivity matching using randomly selected intermediate tokens? Figure 6 rules out simple typo augmentation, but does not establish that the improvement specifically comes from reasoning–answer circuit alignment.

  4. Why do Classic CoT and GEM perform worse than the undistilled model on GSM8K? Were the different distillation objectives independently tuned? Can the authors report results across multiple training seeds, particularly for the smaller improvements over DFT and other strong baselines?

  5. CAD requires additional perturbed forward passes and hyperparameter search. What is the end-to-end computational overhead, and how does CAD compare under a matched compute budget? Also, the GSM8K DFT+CAD score differs between Tables 1 and 2–3 (42.47 vs. 42.27), and the improvement reported in Table 1 does not match its displayed scores.

4 comments

We sincerely thank the reviewer for the constructive feedback, which has helped improve the quality and clarity of our manuscript. We will upload the revised manuscript shortly.

To major W1, Q1 and Q2:

We trace each student before and after training on 100 GSM8K test examples (seed 42; Nmax⁡=2048N_{\max}=2048, τnode=95%\tau_{\mathrm{node}}=95\%, τedge=98%\tau_{\mathrm{edge}}=98\%) and examine two relationships.

  • Sensitivity discrepancy & circuit overlap (Table A). After CAD training, ∣ρ−1∣\lvert\rho-1\rvert decreases and A(s)A(s) increases in every student–teacher pair. Lower sensitivity discrepancy thus comes with genuinely higher circuit overlap in the actual students.
  • Circuit overlap & accuracy (Table B, C, D and E). In every configuration, CAD attains the highest A(s)A(s) and raises accuracy over the baseline. Typo augmentation shares CAD's perturbed inputs but lacks the alignment loss, and it yields both lower A(s)A(s) and lower accuracy. Accuracy gains therefore accompany increased circuit overlap.

Table A. Sensitivity discrepancy and circuit overlap before → after CAD training.

StudentTeacher∣ρ−1∣\lvert\rho-1\rvert ↓A(s)A(s) (%) ↑
Qwen2.5-0.5BLlama-3.3-70B0.2976 → 0.15448.35 → 10.87
Qwen2.5-0.5BGPT-4.1-mini0.3030 → 0.10157.70 → 10.63
Qwen2.5-0.5BGrok-3-mini0.2801 → 0.15738.79 → 9.63
Gemma-2-2BLlama-3.3-70B0.3538 → 0.011246.06 → 59.60

Table B. Circuit-level validation under the Basic Distillation Setting

Distillation MethodA(s)A(s) (%)Accuracy (%)
No Distillation8.3524.73
Base7.2718.94
Typo Aug.7.0325.8
+CAD10.8738.50

Table C. Circuit-level validation with GPT-4.1-mini as the teacher

Distillation MethodA(s)A(s) (%)Accuracy (%)
No Distillation7.7024.73
Base5.6627.58
+CAD10.6342.73

Table D. Circuit-level validation with Grok-3-mini as the teacher

Distillation MethodA(s)A(s) (%)Accuracy(%)
No Distillation8.7924.73
Base8.1211.97
+CAD9.6322.12

Table E. Circuit-level validation with Gemma 2-2B as Student Model

Distillation MethodA(s)A(s) (%)Accuracy (%)
No Distillation46.060.91
Base47.3736.06
+CAD59.6047.27

To Q3:

We report ablations of answer-only consistency regularization and random CoT-token matching, further supporting computation alignment as the source of CAD's performance gains:

Table J. Ablation study of the sensitivity-discrepancy signal on Basic Distillation Setting

Distillation MethodAccuracy (%)
Base18.94
Answer-only consistency (Δy∗\Delta_{y^*})29.85
Random CoT tokens21.97
+CAD38.50

To Q5:

  • Training overhead (Table K). Hyperparameter search is optional; the necessary overhead of CAD comes from additional forward passes to compute sensitivity discrepancies, quantified per epoch below.
  • Compute-controlled comparison (Table L). We fix CAD's (λ,m,α)=(0.5,0.4,6)(\lambda,m,\alpha)=(0.5,0.4,6) to eliminate search overhead and train the baseline on the same perturbed samples to match forward-pass computation. With LR = 2e-5, 3 epochs and three seeds (42, 456, 6), CAD retains a substantial advantage across three seeds on Basic Distillation Setting.

Table K. Necessary extra costs of CAD under Basic Distillation Setting

Extra costTimePeak GPU MemoryAPI Cost
Critical-token extraction (one-time)4.5 hN/A$0.65
Training overhead (per epoch)+0.0572 GPU-h (+64.22%)+4.28 GB (+22.31%)N/A

Table L. Accuracy (%) under controlled computational settings.

BaseCADΔ\Delta
20.10 ± 0.5333.48 ± 5.18+13.38 ± 5.63

To major W2 and Q4:

We evaluate two tuning protocols over three seeds (42, 456, 6), reporting mean ± std.

  • Tuned training hyperparameters & fixed method hyperparameters (Table F). Baseline and CAD independently tune learning rates {2e-5, 3e-5} and epochs {3, 4, 5}, with CAD's (λ,m,α)(\lambda,m,\alpha) fixed as shown below. CAD improves all three distillation methods.
  • Fixed training hyperparameters & tuned method hyperparameters (Table G, H and I).
    • Consistent gains (Table G). Under identical training settings, CAD improves most configurations. Classic CoT falling below the undistilled student is a known effect of long-CoT supervision on small models (L363-366).
    • Robustness to Selection (Table H and I). Over 97% (70/72) of DFT + CAD configurations on GSM8K and over 91% (66/72) on BoolExp exceed the baseline’s validation accuracy.

Table F. Accuracy (%) under matched tuning budgets in the Basic Distillation Setting

Distillation Methodλ\lambdammα\alphaBase+ CADΔ\Delta
Classic CoT0.50.4628.23 ± 3.9141.46 ± 3.59+13.23 ± 6.31
DFT0.10.1438.23 ± 1.9840.30 ± 2.23+2.07 ± 0.87
GEM0.10.2625.15 ± 3.2138.89 ± 0.53+13.74 ± 2.73

Table G. Accuracy (%) under tuned method hyperparameters in the Basic Distillation Setting (mean ± std).

Distillation MethodGSM8KSVAMPMultiArithMAWPSASDiv
Classic CoT20.05±1.1440.33±6.7542.04±9.4546.20±3.6144.82±4.99
Classic +CAD40.31±1.8155.50±4.0988.51±4.3276.38±5.7262.95±5.14
Δ+20.26±1.63+15.17±10.77+46.47±11.96+30.18±9.33+18.13±7.77
DFT38.23±1.1657.33±3.2183.89±2.5576.82±1.1364.19±0.96
DFT +CAD40.61±1.9357.17±5.2087.41±1.2878.86±1.5464.90±0.57
Δ+2.37±2.72−0.17±7.52+3.52±3.61+2.03±1.71+0.71±0.59
GEM18.59±1.5337.67±2.8442.04±7.2552.20±2.4646.22±1.15
GEM +CAD37.37±0.9857.00±2.1886.30±1.1674.47±2.7663.59±0.92
Δ+18.79±0.76+19.33±3.33+44.26±7.40+22.27±4.81+17.37±1.08

Table H. Validation accuracy (%) across the full DFT+CAD hyperparameter grid on BoolExp (baseline: 62.2%)

λ\lambdammα=0.25\alpha=0.25α=0.5\alpha=0.5α=2\alpha=2α=4\alpha=4
0.0010.180.0079.4090.0071.40
0.0010.270.6086.8045.6068.80
0.0010.480.4068.0081.2090.60
0.0030.184.2067.4042.8094.80
0.0030.280.8092.0093.4091.40
0.0030.483.0086.4087.0043.60
0.0100.176.6052.4088.8094.40
0.0100.277.6088.2088.6088.40
0.0100.474.8079.8091.4093.00
0.0300.183.4063.8085.8091.80
0.0300.286.4090.0093.2091.20
0.0300.490.4092.0085.4075.40
0.1000.190.8088.0094.8093.60
0.1000.288.6061.6093.6079.80
0.1000.479.2087.4092.6088.20
0.3000.175.8082.0091.4092.60
0.3000.288.0092.6091.8093.40
0.3000.466.6093.6015.4089.80

Table I. Validation accuracy (%) across the full DFT+CAD hyperparameter grid on GSM8K (baseline: 39.45%)

λ\lambdammα=0.25\alpha=0.25α=0.5\alpha=0.5α=2\alpha=2α=4\alpha=4
0.0010.140.2140.6740.6740.97
0.0010.241.8841.4340.5241.12
0.0010.441.8841.7340.5241.58
0.0030.140.5240.6742.0342.49
0.0030.240.6742.1941.8840.21
0.0030.441.4340.3641.4341.27
0.0100.142.6442.1941.1242.03
0.0100.240.2142.3442.1940.82
0.0100.441.7340.0642.0340.52
0.0300.142.0340.8241.7342.19
0.0300.241.8840.3641.8840.52
0.0300.439.9140.9739.7639.76
0.1000.141.2741.1241.2744.31
0.1000.240.6740.6742.1942.64
0.1000.441.5841.7340.6738.54
0.3000.140.0641.4342.6440.21
0.3000.240.8242.3438.8541.27
0.3000.442.0340.5240.6740.67

Thank you for the detailed response and additional experiments, which have addressed several of my concerns. One remaining question is whether the authors could report circuit overlap for the answer-only consistency and random CoT token baselines in Table J, to further clarify whether CAD’s improvements specifically arise from increased computation alignment rather than other general effects.

Thank you for the follow-up. The circuit overlap A(s)A(s) is 7.06% for Answer-only consistency and 7.70% for Random CoT tokens, both lower than CAD. We hope our responses and additional experiments have addressed your concerns. Please let us know if any issues remain that we could clarify.

Reviewer_DLV3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Computation-Aligned Distillation (CAD) for chain-of-thought distillation. It defines computation alignment as the overlap between circuits associated with intermediate reasoning and final-answer generation. Since tracing these circuits during training is expensive, CAD instead regularizes the ratio of their hidden-state perturbation magnitudes under typographical changes to the input. Experiments on arithmetic and Boolean/formula reasoning use two student models, three teachers, and three base training objectives.

Critical strengths
  • Reproducing a teacher’s reasoning text does not necessarily mean that the student uses the described reasoning to produce its answer. The paper addresses this distinction with a regularizer that can be added to existing distillation objectives without circuit tracing inside the training loop.

  • CAD improves many combinations of teacher, student, and training objective, including transfer to related evaluation datasets. For example, standard CoT distillation on GSM8K improves from 18.94% to 38.50%. The comparison with typo augmentation is also useful: it shows that simply adding perturbed training examples does not reproduce the full benefit.

Critical weaknesses
  • CAD penalizes deviations of ρ=Δr∗/Δy∗\rho=\Delta_{r^*}/\Delta_{y^*} from one, where each Δ\Delta is the norm of a hidden-state change. Equal magnitudes need not imply similar directions, shared circuits, or use of the reasoning procedure described by the CoT. Likewise, shared circuit nodes alone do not establish that the same operations are performed. Figure 3 provides encouraging correlation evidence, r=−0.90r=-0.90, but this is measured on Llama-3.1-8B-Instruct rather than the Qwen2.5-0.5B and Gemma-2-2B students. The paper does not directly demonstrate that CAD increases circuit overlap or causal dependence on intermediate reasoning in the trained students. The typo-augmentation control establishes an additional benefit, but does not rule out a more general sensitivity-regularization explanation.

  • The experimental controls leave uncertainty about robustness and practical value. Appendix reports a single training seed and substantial CAD-specific hyperparameter searches, but no multi-seed uncertainty or measured training overhead. Matching epochs does not match computation when CAD evaluates three perturbations per example. Some large gains also start from degraded baselines: standard CoT and GEM fall below the undistilled Qwen model on GSM8K. Details of the validation split, generated Boolean dataset sizes, typo-validity checks, and critical-span-to-token mapping would also improve reproducibility.

Questions for authors
  1. Can you show whether CAD increases circuit overlap or causal use of intermediate reasoning in the trained students?
  2. How does CAD compare with a simpler perturbation-consistency regularizer or a control using non-critical token positions? These comparisons would help establish whether selecting reasoning-critical positions is essential.
  3. Can the main results be repeated across seeds with comparable baseline tuning? How is the ratio stabilized when Δy∗\Delta_{y^*} is close to zero?
3 comments

We sincerely thank the reviewer for the constructive feedback, which has helped improve the quality and clarity of our manuscript. We will upload the revised manuscript shortly.

To major W1, Q1 and Q2:

  • Circuit overlap (Table A, B, C and D). CAD increases circuit overlap in trained student models across all settings (100 GSM8K test examples, seed 42; Nmax⁡=2048N_{\max}=2048, τnode=95%\tau_{\mathrm{node}}=95\%, τedge=98%\tau_{\mathrm{edge}}=98\%).
  • Ablations (Table E). We evaluate perturbation-consistency regularization and random non-critical CoT tokens as two ablation settings.

Table A. Circuit-level validation under the Basic Distillation Setting

Distillation MethodA(s)A(s) ↑Accuracy (%)
No Distillation8.3524.73
Base7.2718.94
Typo Aug.7.0325.80
+CAD10.8738.50

Table B. Circuit-level validation with GPT-4.1-mini as the teacher

Distillation MethodA(s)A(s) ↑Accuracy (%)
No Distillation7.7024.73
Base5.6627.58
+CAD10.6342.73

Table C. Circuit-level validation with Grok-3-mini as the teacher

Distillation MethodA(s)A(s) ↑Accuracy (%)
No Distillation8.7924.73
Base8.1211.97
+CAD9.6322.12

Table D. Circuit-level validation with Gemma 2-2B as Student Model

Distillation MethodA(s)A(s) (%)Accuracy (%)
No Distillation46.060.91
Base47.3736.06
+CAD59.6047.27

Table E. Ablation study of the sensitivity-discrepancy signal on Basic Distillation Setting

Distillation MethodAccuracy (%)
Base18.94
+CAD38.50
Perturbation consistency (Δy∗\Delta_{y^*} only)29.85
Random CoT tokens21.97

To major W2 and Q3:

  • Fairness and reliability. We evaluate two tuning protocols over three seeds (42, 456, 6), reporting mean ± std.
    • Tuned training & fixed method hyperparameters (Table F). Baseline and CAD independently tune learning rates {2e-5, 3e-5} and epochs {3, 4, 5}, with CAD's (λ,m,α)(\lambda,m,\alpha) fixed. CAD improves all three distillation methods.
    • Fixed training & tuned method hyperparameters(Table G and H).
      • Consistent gains (Table G). Under identical training settings, CAD improves most configurations. Classic CoT falling below the undistilled student is a known effect of long-CoT supervision on small models (L363-366).
      • Robustness to selection (Table H). 91% (66/72) of DFT + CAD configurations on BoolExp exceed the baseline's validation accuracy.
  • Experimental details.
    • Datasets. For GSM8K, we randomly split the official test set into validation (659) and test (660) sets. For BoolExp, we generate 7,000 training and 500 validation expressions following the BIG-bench grammar, and test on the 250 BBH problems.

    • Span-to-token mapping. We locate each extracted value in its CoT context, map its character span to all overlapping tokens, and average their hidden-state shifts.

    • Typo perturbation validity (Table I). A valid perturbation must (a) shift hidden states while (b) preserving query semantics. For (a), all models show clear shifts at both positions (Δr∗\Delta_{r^{\ast}} and Δy∗\Delta_{y^{\ast}} range from 2.25 to 8.90 with GPT-4.1-mini as teacher). For (b), manual inspection of 1,000 sampled perturbations finds that 99.8% preserve semantics.

    • Numerical stability. We prevent division by near-zero values by flooring Δanswer\Delta_{\mathrm{answer}} at 10−610^{-6}:

      ρ~=Δr∗max⁡(Δy∗,10−6).\tilde{\rho}=\frac{\Delta_{r^{\ast}}}{\max(\Delta_{y^{\ast}},10^{-6})}.

    • Training overhead (Table J). We report CAD's additional per-epoch costs. Table F. Accuracy (%) under matched tuning budgets (Basic Distillation Setting).

MethodλmαBase+ CADΔ
Classic CoT0.50.4628.23 ± 3.9141.46 ± 3.59+13.23 ± 6.31
DFT0.10.1438.23 ± 1.9840.30 ± 2.23+2.07 ± 0.87
GEM0.10.2625.15 ± 3.2138.89 ± 0.53+13.74 ± 2.73

Table G. Accuracy (%) under tuned method hyperparameters (Basic Distillation Setting).

MethodGSM8KSVAMPMultiArithMAWPSASDiv
Classic CoT20.05±1.1440.33±6.7542.04±9.4546.20±3.6144.82±4.99
+ CAD40.31±1.8155.50±4.0988.51±4.3276.38±5.7262.95±5.14
Δ+20.26±1.63+15.17±10.77+46.47±11.96+30.18±9.33+18.13±7.77
DFT38.23±1.1657.33±3.2183.89±2.5576.82±1.1364.19±0.96
+ CAD40.61±1.9357.17±5.2087.41±1.2878.86±1.5464.90±0.57
Δ+2.37±2.72−0.17±7.52+3.52±3.61+2.03±1.71+0.71±0.59
GEM18.59±1.5337.67±2.8442.04±7.2552.20±2.4646.22±1.15
+ CAD37.37±0.9857.00±2.1886.30±1.1674.47±2.7663.59±0.92
Δ+18.79±0.76+19.33±3.33+44.26±7.40+22.27±4.81+17.37±1.08

Table H. Validation accuracy (%) across the full DFT+CAD hyperparameter grid on BoolExp (baseline: 62.2%)

λ\lambdammα=0.25\alpha=0.25α=0.5\alpha=0.5α=2\alpha=2α=4\alpha=4
0.0010.180.0079.4090.0071.40
0.0010.270.6086.8045.6068.80
0.0010.480.4068.0081.2090.60
0.0030.184.2067.4042.8094.80
0.0030.280.8092.0093.4091.40
0.0030.483.0086.4087.0043.60
0.0100.176.6052.4088.8094.40
0.0100.277.6088.2088.6088.40
0.0100.474.8079.8091.4093.00
0.0300.183.4063.8085.8091.80
0.0300.286.4090.0093.2091.20
0.0300.490.4092.0085.4075.40
0.1000.190.8088.0094.8093.60
0.1000.288.6061.6093.6079.80
0.1000.479.2087.4092.6088.20
0.3000.175.8082.0091.4092.60
0.3000.288.0092.6091.8093.40
0.3000.466.6093.6015.4089.80

Table I. Hidden-state shifts under typo perturbations with GPT-4.1-mini as the teacher

Distillation MethodΔr∗\Delta_{r^*}Δy∗\Delta_{y^*}
No Distillation3.304.18
Base2.253.67
+ CAD8.248.90

Table J. Additional per-epoch overhead of CAD on Basic Distillation Setting

Training timePeak GPU memory
+0.0572 GPU-h (+64.22%)+4.28 GB (+22.31%)

Dear Reviewer,

Thank you for your positive assessment of our work. With the discussion period nearing its end, we would like to check whether our response has resolved your questions. If any point remains unclear, we would be happy to elaborate.

Thank you for your detailed and careful review.

Reviewer_Zg2FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that chain-of-thought (CoT) distillation aligns what a student writes with the teacher’s rationale but does not ensure that the student’s final answer is actually computed along the steps its CoT describes. It calls this missing property computation alignment. To characterize it mechanistically, the paper introduces multi-step circuit tracing (MCT), which recursively composes single-step transcoder-based attribution graphs across generated tokens (Sec. 3.1.1, Alg. 2). Computation alignment A(s)\mathcal{A}(s) is then defined as the Jaccard overlap between the node sets of two circuits: one traced from an LLM-selected reasoning-critical token r∗r^{\ast} and one traced from an answer-critical token y∗y^{\ast} (Eq. 9). MCT is too expensive to run during training, so the proposed Computation-Aligned Distillation (CAD) uses a proxy. The query is perturbed with character-level typos while the CoT and answer are kept fixed, and the last-layer hidden-state shifts Δr∗\Delta_{r^{\ast}} and Δy∗\Delta_{y^{\ast}} are measured. The deviation of ρ=Δr∗/Δy∗\rho=\Delta_{r^{\ast}}/\Delta_{y^{\ast}} from 1 is penalized by a cosh-shaped hinge loss that is added to any base distillation objective (Eqs. 10–13). The proxy is supported by a correlation of r=−0.90r=-0.90 between ∣ρ−1∣|\rho-1| and A(s)\mathcal{A}(s) on Llama-3.1-8B-Instruct (Fig. 3). In the experiments, Llama-3.3-70B, GPT-4.1-mini and Grok-3-mini are distilled into Qwen2.5-0.5B and Gemma-2-2B, with training on GSM8K or Boolean Expressions. CAD is added to classic CoT distillation, DFT and GEM, and the students are evaluated on eight benchmarks (Tables 1–3). A hyperparameter study (Fig. 5) and a typo-augmentation control (Fig. 6) complete the evaluation.

Critical strengths
  1. A well-posed question with a cheap, general-purpose regularizer. The paper separates two questions: whether the student writes the right steps, and whether the student’s answer is computed through those steps (p. 1, L36–44). It gives a mechanistic definition of the second property (Eq. 9) and a training-time proxy for it (Eqs. 10–13). The proxy needs extra forward passes on perturbed queries (three per sample; p. 6, L304–305) but no circuit tracing during training. Because the regularizer is simply added to an arbitrary base loss (Eq. 13), the paper can test it on three quite different objectives (classic CoT SFT, DFT, GEM), three teachers, two students and eight benchmarks (Tables 1–3). If the effect is real, this is a simple and broadly applicable way to bring interpretability-motivated constraints into distillation.

  2. Large reported gains, and a control for the most obvious confound. In Table 1 the CAD increment is positive in all 18 out-of-distribution cells, e.g., +25.50+25.50 on SVAMP and +34.42+34.42 on MultiArith for classic CoT distillation. CAD sees perturbed queries during training, so the natural first control is typo augmentation without the alignment loss. In Fig. 6, CAD (average 65.4) beats that control (average 45.8) on all five math benchmarks. These results suggest the direction deserves careful follow-up, despite the concerns below.

Critical weaknesses
  1. The central claim (CAD makes the student’s answer computation align with its CoT) is not tested, and the evidence that the proxy tracks circuit overlap is thin and partly inconsistent.

    • No measurement on students. Neither A(s)\mathcal{A}(s) nor ρ\rho is reported for any distilled model. Both the circuit-level validation (Fig. 3) and the analysis of correct versus incorrect predictions (Fig. 4) use Llama-3.1-8B-Instruct (p. 5, L256–257; p. 6, L316–317). That model is not one of the students (Qwen2.5-0.5B and Gemma-2-2B; p. 6, L295–296), and it was never trained with CAD. A correlation in a model that was not optimized for the proxy does not show that minimizing ∣ρ−1∣|\rho-1| increases circuit overlap. ρ\rho is a ratio of two norms of hidden-state shifts (Eqs. 10–11). It ignores the direction of the shifts, and in principle it could be driven to 1 by changes unrelated to shared circuitry. Two statements therefore go beyond the evidence: “steering the student toward reasoning–computation alignment” (p. 2, L66) and “the gains come more from computation alignment than from perturbation robustness” (p. 8, L423–424). Fig. 6 rules out plain typo augmentation. It does not rule out other explanations, such as consistency regularization at the answer position alone, or any effect that does not depend on the LLM-selected r∗r^{\ast}.

    • A small validation that rests on a few points. The paper does not say how many samples Fig. 3 uses or how they were chosen. Reading the 48 markers from the vector graphics of Fig. 3 recovers the reported Pearson r≈−0.90r\approx-0.90, but the Spearman correlation is only about −0.63-0.63. If the 7 points with ∣ρ−1∣>15|\rho-1|>15 are dropped, the Pearson correlation on the remaining 41 points falls to about −0.50-0.50, and their A(s)\mathcal{A}(s) values span only about 0.74–0.93.

    • Fig. 4 is hard to reconcile with ρ=Δr∗/Δy∗\rho=\Delta_{r^{\ast}}/\Delta_{y^{\ast}}. The two panels appear to be computed from the same samples, yet they disagree. In the vector data of Fig. 4(a), about 79 of the roughly 100 correct samples have ∣ρ−1∣≥3.8|\rho-1|\geq 3.8, which means ρ≥4.8\rho\geq 4.8 because ρ>0\rho>0. In Fig. 4(b), however, the outermost density contour of the correct samples lies within Δr∗∈[6.0,32.5]\Delta_{r^{\ast}}\in[6.0,32.5] and Δy∗∈[8.6,32.8]\Delta_{y^{\ast}}\in[8.6,32.8], where ρ≤3.8\rho\leq 3.8. The panels could be reconciled if ρ\rho were averaged over perturbations or tokens in a way dominated by near-zero values of Δy∗\Delta_{y^{\ast}}, but that would itself point to a heavy-tailed, unstable proxy. The margins in Table 6 (m∈[0.1,1.0]m\in[0.1,1.0]) are on the ratio scale, so it is unclear which quantity was validated, which was plotted, and which was optimized.

    This weakness is partly fixable. Public transcoders exist for Gemma-2-2B, one of the two students, so A(s)\mathcal{A}(s) could be measured before and after CAD. Without such evidence, the paper’s main conceptual claim is not supported.

  2. The quantitative evidence is not reliable enough to support “consistently improves” (Abstract; p. 2, L75).

    • Single seed and asymmetric tuning. All runs use seed 42 (Table 5), and no variance is reported. Each CAD variant is selected on the validation set from a grid of 3×4×3=363\times4\times3=36 (classic), 6×3×4=726\times3\times4=72 (DFT) or 4×3×5=604\times3\times5=60 (GEM) configurations (Table 6). No hyperparameter search is described for the baselines; only their checkpoints are selected on validation (p. 6, L307–310). Fig. 5 suggests large run-to-run variability. Read from its vector data, accuracy goes 27.1→18.1→24.227.1\to18.1\to24.2 for α=2,3,4\alpha=2,3,4, and 42.9→35.4→41.742.9\to35.4\to41.7 for λ=0.5,0.7,0.9\lambda=0.5,0.7,0.9. With variability of this size, comparing the best of 36–72 runs against a single baseline run can inflate the reported increments.

    • Under-performing baselines. On GSM8K, classic CoT distillation falls below the undistilled student (18.94 vs. 24.73), and GEM reaches only 19.55. On BoolExp, whose answers are binary, DFT reaches 60.40, while the other two base methods reach 91.60 and 92.40 (Table 1). The large increments in Table 1 occur for exactly these weak base runs. Classic CoT distillation and GEM gain between +16.70+16.70 and +40.56+40.56 on the five math sets, and DFT gains between +27.60+27.60 and +31.20+31.20 on the three Boolean sets. The base method that is not degraded in each domain gains much less: DFT gains between +0.94+0.94 and +7.00+7.00 on math, and CoT distillation and GEM change by between −0.40-0.40 and +2.70+2.70 on the Boolean sets. This pattern fits CAD repairing a weak base run at least as well as it fits a uniform benefit from computation alignment.

    • Inconsistent numbers.

      • Table 1, GSM8K/DFT: 42.47−38.48=3.9942.47-38.48=3.99, not the reported +3.63+3.63. Tables 2 and 3 report 42.2742.27 and +3.79+3.79 for the same configuration.

      • Table 2, Grok-3-mini/DFT: 29.39−27.73=1.6629.39-27.73=1.66, not +2.62+2.62. The text quotes the same +2.62+2.62 (p. 8, L393).

      • The “Baseline” bars in Fig. 6 (SVAMP 39.0, MultiArith 32.8, MAWPS 40.9, ASDiv 45.3) differ from Table 1’s CoT-distillation base (33.50, 52.78, 44.86, 39.06), although both are stated to use the Basic Distillation Setting (p. 8, L408). The text’s claim that CAD improves SVAMP by 20% (p. 8, L422–424) is computed against the Fig. 6 baseline.

      • The best configuration in Fig. 5 reaches about 42.9%, while Tables 1–3 report 38.50% for the same setting. Fig. 5 also sweeps λ∈{0.1,0.3,0.5,0.7,0.9}\lambda\in\lbrace 0.1,0.3,0.5,0.7,0.9\rbrace and α∈{1,…,6}\alpha\in\lbrace 1,\dots,6\rbrace, whereas Table 6 lists {0.1,0.5,0.9}\lbrace 0.1,0.5,0.9\rbrace and {2,4,6}\lbrace 2,4,6\rbrace.

    • Undocumented protocol. The validation set is never described, and GSM8K has no official validation split. The number of evaluated items per benchmark and the decoding settings are not given. App. C (p. 14, L752–753) says GSM8K’s test split is used for evaluation. The official test split has 1,319 problems, yet many of the GSM8K entries cannot arise from it. For example, 18.94% lies between 249/1319=18.88249/1319=18.88% and 250/1319=18.95250/1319=18.95%, but it equals 125/660125/660.

    Each of these problems can be fixed with multiple seeds, a matched tuning budget, and corrected tables and figures. As submitted, however, neither the size nor the consistency of the gains is established. Table 1 also contains a negative cell (BoolExp, classic CoT distillation, −0.40-0.40), which the text overlooks when it reports “consistent improvements across both datasets and all three distillation methods” (p. 7, L360–362).

Minor points.

  • Missing related work. SCOTT (Wang et al., ACL 2023) distills a student with a counterfactual objective that “prevents the student from ignoring the rationales”. FRODO (Paul et al., Findings of EMNLP 2024) trains small LMs to “faithfully reason over” their intermediate steps. Both target the gap stated on p. 1 (L36–38), and neither is cited. Their existence qualifies two claims: that computation alignment is “previously unstudied” (p. 2, L68), and that “Existing CoT distillation methods leave this gap unaddressed” (p. 9, L466–467).

  • Citations. Wang et al. (2022), which is the self-consistency paper, and Hao et al. (2023), which is RAP, are cited as CoT-distillation methods (p. 1, L32–33).

  • Transcoders. Sec. 2.1 describes cross-layer transcoders (p. 2, L88). According to the public circuit-tracer documentation, however, the CRV transcoders used for Fig. 3 are TopK per-layer transcoders.

  • Overstatements. “The largest gain across all experiments” (p. 8, L403) refers to +24.70+24.70, but Table 1 contains +40.56+40.56. The claim that “a weaker teacher limits CAD’s performance ceiling” (p. 8, L392–398) rests on no measurement of teacher quality. For DFT, moreover, the gain with the Llama-3.3-70B teacher (+3.79+3.79) is similar to the gain with Grok-3-mini.

  • MCT details. The values of kk, τ\tau and Φmax⁡\Phi_{\max} are not reported, and “cumulative influence” is not defined (p. 3, L160). Eq. 3 writes si=(x,c<t)s_i=(x,c_{<t}), whereas Alg. 2 writes (x,o<i)(x,o_{<i}). Eq. 5 composes weights across the discrete token-sampling step, so the “Chain rule” label (Alg. 2, line 12) has no derivative justification. MCT itself is never validated.

  • Cost. The training overhead is not reported, either for the perturbed forward passes or for running the 70B extractor on every training example.

Questions for authors
  1. [Clarification] Which quantity is plotted as ∣ρ−1∣|\rho-1| in Figs. 3–4, and is it the quantity used in Eq. 12? Please see Weakness 1 for the apparent mismatch between Fig. 4(a) and Fig. 4(b). How many samples do Figs. 3–4 use, from which dataset, whose generations are they, and how were they selected? Please also report rank correlations, and the correlation without the points with ∣ρ−1∣>15|\rho-1|>15. If the plotted quantity differs from Eq. 11, or if the association is driven by a few outliers, my concern about the proxy would grow. If corrected plots show a robust monotone relation, preferably on a student model, my soundness assessment would improve.

  2. [Clarification] Can you measure A(s)\mathcal{A}(s) directly on trained students? Gemma-2-2B is a natural candidate because public transcoders exist for it. The comparison should cover the base model, CAD and typo augmentation on held-out data, and should report Δr∗\Delta_{r^{\ast}}, Δy∗\Delta_{y^{\ast}} and ρ\rho separately. This is the most important missing evidence. If CAD raises circuit overlap beyond both controls, I would substantially raise my assessment of the central claim. If ρ\rho moves toward 1 while A(s)\mathcal{A}(s) does not change, the computation-alignment interpretation does not hold.

  3. [Clarification] Please report results over at least 3 seeds (mean ±\pm std) for the Basic Distillation Setting and for the cells with large increments (classic CoT distillation and GEM on GSM8K, DFT on BoolExp). The baselines should receive a comparable tuning budget, e.g., over learning rate and number of epochs. Please also give the selected (λ,m,α)(\lambda,m,\alpha) for each cell and the accuracies over the full grid. In particular, did DFT+CAD on BoolExp select a very small λ\lambda, such as 0.0010.001? And why does classic SFT degrade the student below its undistilled accuracy? If the gains survive matched tuning and seed variation, my rating would increase.

  4. [Clarification] Please reconcile the numbers listed in Weakness 2.

    • Does Fig. 5 show validation or test accuracy?

    • How was the validation set constructed, and how large is each evaluation set?

    • What decoding and answer-extraction protocol was used? The near-zero undistilled accuracy on binary BoolExp (0.40%) suggests that format failures are counted as errors.

    If these are transcription errors with a benign explanation, this concern would largely go away.

  5. [Clarification] Please clarify three aspects of the loss design. Clear answers would show whether the loss plausibly targets the claimed property.

    • Asymmetry and overflow. Because ρ>0\rho>0, ∣ρ−1∣<1|\rho-1|<1 whenever Δr∗<Δy∗\Delta_{r^{\ast}}<\Delta_{y^{\ast}}. With m=1.0m=1.0, which is in the classic grid, the loss is therefore zero for every ρ≤2\rho\leq 2, including ρ→0\rho\to0. With α=6\alpha=6 and m=0.4m=0.4, the penalty for ρ<1\rho<1 is at most cosh⁡(3.6)−1≈17.3\cosh(3.6)-1\approx17.3, while cosh⁡\cosh overflows fp32/bf16 once ∣ρ−1∣>15.3|\rho-1|>15.3. How are overflow and this asymmetry handled, and why not use a symmetric deviation such as ∣log⁡ρ∣|\log\rho|?

    • Teacher forcing. rr and yy are teacher-forced and identical in ss and s′s'. Could the student lower Lalign\mathcal{L}_{\text{align}} by making the answer position rely more on xx and less on the unchanged CoT tokens? That would be the opposite of the intended effect.

    • Aggregation. During training, how are multiple r∗r^{\ast} tokens and multi-token spans aggregated, and at which position is h(c)h(c) read?

  6. [Suggestion / future work] Two controls would isolate the coupling between r∗r^{\ast} and y∗y^{\ast}: (i) penalize only Δy∗\Delta_{y^{\ast}}, i.e., answer invariance to the perturbation; and (ii) replace the LLM-selected r∗r^{\ast} with random CoT tokens. Please also discuss SCOTT and FRODO, and ideally compare against them. Finally, in MCT the answer circuit expands r∗r^{\ast}’s own subgraph whenever r∗r^{\ast} survives pruning (Eq. 4). A(s)\mathcal{A}(s) may therefore largely reflect how much attribution flows from y∗y^{\ast} through the token r∗r^{\ast}. How does this differ from the faithfulness notion that footnote 1 calls orthogonal?

4 comments

We sincerely thank the reviewer for the constructive feedback, which has helped improve the quality and clarity of our manuscript. We will upload the revised manuscript shortly.

To major W1,Q1,and Q2:

  • Ablating the sensitivity-discrepancy signal (Table A). Penalizing Δy∗\Delta_{y^{\ast}} alone (removing Δr∗\Delta_{r^{\ast}}) or replacing r∗r^{\ast} with random CoT tokens lowers accuracy to 29.85% and 21.97%, respectively (vs. 38.50%).

  • Circuit-level validation (Table B, C, D and E). We trace all models on 100 GSM8K test examples (seed 42; Nmax⁡=2048N_{\max}=2048, τnode=95%\tau_{\mathrm{node}}=95\%, and τedge=98%\tau_{\mathrm{edge}}=98\%).

    • Proxy validity. Compared with the baseline, CAD reduces the mean ∣ρ−1∣|\rho-1| from 0.3519 to 0.1544, while increasing circuit overlap A(s)A(s) from 7.27% to 10.87%. This directly demonstrates that optimizing the proxy improves computation alignment in actual student models.
    • Performance attribution. Typo augmentation and CAD use the same perturbed inputs and differ only in the alignment loss. Adding this loss increases A(s)A(s) from 7.03% to 10.87% and accuracy from 25.80% to 38.50%, indicating that the additional performance gains are associated with increased circuit overlap rather than perturbation robustness alone.
  • Figure clarifications.

    • Fig. 3. Fig. 3 provides insufficient evidence for proxy validity and we remove it along with its associated discussion.
    • Fig. 4. To motivate computation alignment, we examine its association with prediction correctness before CAD optimization, using the first 100 correct and 100 incorrect Llama-3.1-8B-Instruct generations with valid reasoning and answer positions from the GSM8K. We clarify that the x-axis of Fig. 4(a) is ∣Δr∗−Δy∗∣\lvert\Delta_{r^{\ast}}-\Delta_{y^{\ast}}\rvert and the y-axis is probability density. Fig. 4(a) and (b) are consistent: smaller sensitivity differences in (a) correspond to points closer to the diagonal in (b).

Table A. Ablation study of the sensitivity-discrepancy signal on Basic Distillation Setting

Distillation MethodAcc (%)
Base18.94
+CAD38.50
Perturbation consistency(Δy∗\Delta_{y^*} only)29.85
Random CoT tokens21.97

Table B. Circuit-level validation under the Basic Distillation Setting

Distillation MethodA(s)A(s)Mean ∣ρ−1∣\lvert\rho-1\rvert
No Distillation8.35%0.2976
+CAD10.87%0.1544
Base7.27%0.3519
Typo Augmentation7.03%0.3377

Table C. Circuit-level validation with GPT-4.1-mini as the teacher

Distillation MethodA(s)A(s)Mean ∣ρ−1∣\lvert\rho-1\rvertMean Δr∗\Delta_{r^*}Mean Δy∗\Delta_{y^*}Mean ρ\rho
No Distillation7.70%0.30303.29924.18070.9203
+CAD10.63%0.10158.24278.89700.9247
Base5.66%0.33162.25273.67380.7518

Table D. Circuit-level validation with Grok-3-mini as the teacher

Distillation MethodA(s)A(s)Mean ∣ρ−1∣\lvert\rho-1\rvertMean Δr∗\Delta_{r^{\ast}}Mean Δy∗\Delta_{y^{\ast}}Mean ρ\rho
No Distillation8.79%0.28013.45194.15510.9127
+CAD9.63%0.15737.55458.41000.9346
Base8.12%0.29582.47803.58150.8259

Table E. Circuit-level validation with Gemma 2-2B as Student Model

Distillation MethodA(s)A(s)Mean ∣ρ−1∣\lvert\rho-1\rvertMean Δr∗\Delta_{r^{\ast}}Mean Δy∗\Delta_{y^{\ast}}Mean ρ\rho
No Distillation46.06%0.353833.409342.50160.9255
+CAD59.60%0.0112328.8216328.47560.9994
Base47.37%0.382422.699734.84190.7099

To major W2 ,Q3 and Q4:

  • Fairness and reliability. We evaluate two tuning protocols over three seeds (42, 456, 6), reporting mean ± std.

    • Tuned training hyperparameters & fixed method hyperparameters (Table F). Baseline and CAD independently tune learning rates {2e-5, 3e-5} and epochs {3, 4, 5}, with CAD's (λ,m,α)(\lambda,m,\alpha) fixed as shown below. CAD improves all three distillation methods.
    • Fixed training hyperparameters & tuned method hyperparameters (Table G, H and I).
      • Consistent gains (Table G). Under identical training settings, CAD improves most configurations. Classic CoT falling below the undistilled student is a known effect of long-CoT supervision on small models (L363-366).
      • Robustness to Selection (Table H and I). Over 97% (70/72) of DFT + CAD configurations on GSM8K and over 91% (66/72) on BoolExp exceed the baseline’s validation accuracy. Notably, DFT and GEM search smaller λλ ranges to match the magnitude of their base losses LKD\mathcal{L}_{\mathrm{KD}}.
  • Experimental details.

    • GSM8K has no official validation split, as following [1–3] we randomly split its test set (1,319) into validation (659) and test (660) sets.
    • Decoding is greedy with a maximum of 1,024 tokens.
    • Fig. 5 reports validation accuracy, and Table 6 specifies the hyperparameter ranges used in the main experiments.
    • We acknowledge the transcription errors and have corrected all inconsistencies.
    • The low BoolExp accuracy (0.40%) mainly reflects the undistilled model's inability to produce valid Boolean answers, often generating repeated questions, repetitive text, or incomplete reasoning instead.

[1] MCC-KD: Multi-CoT Consistent Knowledge Distillation.

[2] Keypoint-based Progressive Chain-of-Thought Distillation for LLMs.

[3] Sparta Alignment: Collectively Aligning Multiple Language Models through Combat.

Table F. Accuracy (%) under matched tuning budgets in the Basic Distillation Setting

Distillation Methodλ\lambdammα\alphaBase+ CADΔ\Delta
Classic CoT0.50.4628.23 ± 3.9141.46 ± 3.59+13.23 ± 6.31
DFT0.10.1438.23 ± 1.9840.30 ± 2.23+2.07 ± 0.87
GEM0.10.2625.15 ± 3.2138.89 ± 0.53+13.74 ± 2.73

Table G. Accuracy (%) under tuned method hyperparameters in the Basic Distillation Setting (mean ± std).

Distillation MethodGSM8KSVAMPMultiArithMAWPSASDiv
Classic CoT20.05±1.1440.33±6.7542.04±9.4546.20±3.6144.82±4.99
Classic +CAD40.31±1.8155.50±4.0988.51±4.3276.38±5.7262.95±5.14
Δ+20.26±1.63+15.17±10.77+46.47±11.96+30.18±9.33+18.13±7.77
DFT38.23±1.1657.33±3.2183.89±2.5576.82±1.1364.19±0.96
DFT +CAD40.61±1.9357.17±5.2087.41±1.2878.86±1.5464.90±0.57
Δ+2.37±2.72−0.17±7.52+3.52±3.61+2.03±1.71+0.71±0.59
GEM18.59±1.5337.67±2.8442.04±7.2552.20±2.4646.22±1.15
GEM +CAD37.37±0.9857.00±2.1886.30±1.1674.47±2.7663.59±0.92
Δ+18.79±0.76+19.33±3.33+44.26±7.40+22.27±4.81+17.37±1.08

Table H. Validation accuracy (%) across the full DFT+CAD hyperparameter grid on BoolExp (baseline: 62.2%)

λ\lambdammα=0.25\alpha=0.25α=0.5\alpha=0.5α=2\alpha=2α=4\alpha=4
0.0010.180.0079.4090.0071.40
0.0010.270.6086.8045.6068.80
0.0010.480.4068.0081.2090.60
0.0030.184.2067.4042.8094.80
0.0030.280.8092.0093.4091.40
0.0030.483.0086.4087.0043.60
0.0100.176.6052.4088.8094.40
0.0100.277.6088.2088.6088.40
0.0100.474.8079.8091.4093.00
0.0300.183.4063.8085.8091.80
0.0300.286.4090.0093.2091.20
0.0300.490.4092.0085.4075.40
0.1000.190.8088.0094.8093.60
0.1000.288.6061.6093.6079.80
0.1000.479.2087.4092.6088.20
0.3000.175.8082.0091.4092.60
0.3000.288.0092.6091.8093.40
0.3000.466.6093.6015.4089.80

Table I. Validation accuracy (%) across the full DFT+CAD hyperparameter grid on GSM8K (baseline: 39.45%)

λ\lambdammα=0.25\alpha=0.25α=0.5\alpha=0.5α=2\alpha=2α=4\alpha=4
0.0010.140.2140.6740.6740.97
0.0010.241.8841.4340.5241.12
0.0010.441.8841.7340.5241.58
0.0030.140.5240.6742.0342.49
0.0030.240.6742.1941.8840.21
0.0030.441.4340.3641.4341.27
0.0100.142.6442.1941.1242.03
0.0100.240.2142.3442.1940.82
0.0100.441.7340.0642.0340.52
0.0300.142.0340.8241.7342.19
0.0300.241.8840.3641.8840.52
0.0300.439.9140.9739.7639.76
0.1000.141.2741.1241.2744.31
0.1000.240.6740.6742.1942.64
0.1000.441.5841.7340.6738.54
0.3000.140.0641.4342.6440.21
0.3000.240.8242.3438.8541.27
0.3000.442.0340.5240.6740.67

To Q5 (loss construction):

  • Eq. (10). h(c)h(c) is the last-layer hidden state at each LLM-extracted critical token. The aggregation averages the hidden-state shifts across all r∗r^{\ast} tokens.

    Δc(s)=13∣c∣∑j=13∑ci∈c∥h(ci)−h(ci,j′)∥2,c∈{r∗,y∗}.\Delta_c(s)=\frac{1}{3\lvert c\rvert}\sum_{j=1}^{3}\sum_{c_i\in c}\left\lVert h(c_i)-h(c_{i,j}')\right\rVert_{2},\quad c\in\lbrace r^{\ast},y^{\ast}\rbrace.

  • Teacher forcing (Table J). The proposed shortcut requires Δy∗<Δr∗\Delta_{y^{\ast}}\lt\Delta_{r^{\ast}} and works by raising Δy∗\Delta_{y^{\ast}}. On teacher-forced training data (Basic Distillation Setting), we observe the opposite: Δy∗>Δr∗\Delta_{y^{\ast}}\gt\Delta_{r^{\ast}} before CAD, and CAD raises Δr∗\Delta_{r^{\ast}} while leaving Δy∗\Delta_{y^{\ast}} nearly unchanged. Circuit overlap also rises from 8.82% to 13.28%, contradicting the proposed shortcut.

  • Eq. (12).

    • Numerical overflow. All our experiments already use two numerical safeguards, a denominator floor and a cap on the cosh input, which the original manuscript omitted from Eq. (12). We now state them explicitly:

      ρ~=Δr∗max⁡(Δy∗,10−6),Lalign=cosh⁡(min⁡{αmax⁡(∣ρ~−1∣−m,0),10})−1.\tilde{\rho}=\frac{\Delta_{r^{\ast}}}{\max(\Delta_{y^{\ast}},10^{-6})},\qquad L_{\mathrm{align}}=\cosh\left(\min\left\lbrace\alpha\max\left(\lvert\tilde{\rho}-1\rvert-m,0\right),10\right\rbrace\right)-1.

    • Asymmetry (Table K). Eq. (12) is symmetric about ρ~=1\tilde{\rho}=1: ρ~=1+z\tilde{\rho}=1+z and ρ~=1−z\tilde{\rho}=1-z incur the same loss. ∣log⁡ρ~∣\lvert\log\tilde{\rho}\rvert is symmetric in ratio instead: ρ~=z\tilde{\rho}=z and ρ~=1/z\tilde{\rho}=1/z incur the same loss. Both are reasonable, as both satisfy our design principle: the loss is zero when Δr∗=Δy∗\Delta_{r^{\ast}}=\Delta_{y^{\ast}} and grows as the two sensitivities diverge. They also perform comparably.

Table J. Circuit-level validation under teacher forcing in the Basic Distillation Setting.

ModelA(s)A(s) (%) ↑Mean Δr∗\Delta_{r^{\ast}}Mean Δy∗\Delta_{y^{\ast}}
Before CAD8.822.98275.4608
After CAD13.284.58445.3495

Table K. Accuracy (%) of alternative LalignL_{\mathrm{align}} under the Basic Distillation Setting.

LossAccuracy (%)
Proposed (Eq. 12)38.50
∣log⁡ρ~∣\lvert\log\tilde{\rho}\rvert39.09

To Q6, minor W1, and minor W5:

Following the official circuit tracing (Ameisen et al., 2025), we standardize MCT. Starting from the target logit ℓ\ell, MCT iteratively attributes the unattributed node with the largest influence Φ(u;ℓ)\Phi(u;\ell) (Eq. 6) until Nmax⁡=2048N_{\max}=2048 features are attributed. It then keeps the smallest set of nodes (edges) whose summed influence reaches τnode=0.95\tau_{\mathrm{node}}=0.95 (τedge=0.98\tau_{\mathrm{edge}}=0.98) of the total. Weights across generation steps are combined by weight composition (formerly "chain rule"), and c<ic_{\lt i} is corrected to o<io_{\lt i}.

Under this procedure, faithfulness and computation alignment are logically independent.

  • Faithfulness does not imply alignment. Faithfulness requires r∗∈Ny∗r^{\ast}\in\mathcal N_{y^{\ast}}. However, the subgraph of Gy∗\mathcal G_{y^{\ast}} expanded from r∗r^{\ast} need not overlap with Gr∗\mathcal G_{r^{\ast}}: the former expands upstream nodes according to Φ(u;ℓy∗)\Phi(u;\ell_{y^{\ast}}), whereas the latter uses Φ(u;ℓr∗)\Phi(u;\ell_{r^{\ast}}).
  • Alignment does not imply faithfulness. Gy∗\mathcal G_{y^{\ast}} can share many nodes with Gr∗\mathcal G_{r^{\ast}} without attributing r∗r^{\ast}, since r∗r^{\ast} is selected only when it has the largest Φ(r∗;ℓy∗)\Phi(r^{\ast};\ell_{y^{\ast}}) among unattributed nodes.

Compared with faithfulness methods such as SCOTT and FRODO, our core contribution is identifying and optimizing computation alignment.

Table L. Accuracy (%) comparison with SCOTT (Gemma-2-2B as student).

Distillation MethodAccuracy (%)
CoT Distillation36.06
SCOTT39.39
CAD47.27

To minor W2:

We have corrected the citations for CoT distillation.

To minor W3:

Our circuit tracing supports both cross-layer transcoders (CLTs) and per-layer Top-K transcoders (PLTs). We have revised the corresponding statements accordingly.

To minor W4:

We have revised the overstated claims accordingly.

To minor W6:

Hyperparameter search is an optional extra cost, so we report only the necessary extra costs of CAD.

Table M. Necessary extra costs of CAD under the Basic Distillation Setting.

Extra costTimePeak GPU MemoryAPI Cost
Critical-token extraction (one-time)4.5 hN/AUSD 0.65
Training overhead (per epoch)+0.0572 GPU-h (+64.22%)+4.28 GB (+22.31%)N/A

Dear Reviewer,

As the discussion period is nearing its end, we wanted to follow up on our response. We would greatly appreciate knowing whether these address your concerns, and we are glad to clarify anything further.

Thank you for your detailed and careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.