- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The job try to propose an alternative paradigm to conventional knowledge distillation: instead of enforcing identical internal architectures between the teacher and student models, it identifies the critical computational circuits for each model separately, then aligns the impact of these circuits on model predictions. Nevertheless, the paper suffers from a fundamental flaw that persists throughout the manuscript: the authors prove and measure prediction shifts under specific interventions, yet repeatedly interpret such shifts as the models’ inherent reasoning mechanisms. They further equate the alignment of prediction shifts with the transfer of reasoning capabilities.
Critical strengths
S1: Well-organized layout and thoughtfully chosen color schemes for figures.
S2: The proposed idea is moderately novel: task-relevant circuits of teacher and student are identified independently, and distillation is achieved via aligning their predictive effects rather than direct matching of internal components.
S3: The method is compatible: RCD serves as an auxiliary loss and can be integrated with existing distillation methods (Naive KD, ABKD, etc.) with no architecture modification.
S4: Experimental validation is fairly comprehensive: comparisons against baselines over multiple model families on mathematical reasoning and code generation tasks, supplemented by circuit analysis and ablation studies.
Critical weaknesses
W1: The claim that we can perfectly recover the circuit depends on Assumption 2. This assumption directly states that all circuits found by EAP together equal the true faithfulness-optimal circuit. In other words, the assumption already takes for granted the core identifiability relation that the theorem is supposed to prove. The authors need to clarify: the theorem only guarantees consistent estimation if identifiability holds. It does not prove that identifiability is true in the first place.
W2: (Theorem 2,Lines 317–323; C.3,Lines 1208–1218) The generalization analysis assumes fixed model checkpoints and alignment samples that do not depend on training or model selection. In practice, though, the RCD loss is optimized over training data, and student parameters keep changing throughout training. Because of this, the bound they derive cannot directly prove generalization for the training objective or the learned student model. The authors need to clearly explain this difference. They also need to align the per-token selection for each example assumed in the proof with the batch-aware allocation used in Algorithm 1.
W3: A central motivation of RCD is to preserve the student’s intrinsic reasoning pattern, a claim that remains unverified in the paper (Lines 173–178, Lines 245–249, Appendix B, Algorithm 1, Line 937). While the method fixes selected edge indices during training, this does not guarantee that their computational functions stay invariant as the student parameters get updated. Direct circuit comparison before and after distillation, alongside baselines such as standard KD, is required to substantiate this claim.
W4: Interpreting Logit Variation Alignment as Reasoning Capability Transfer constitutes a conceptual leap (Lines 255–259, Page 5, Equation (6), Page 9, Conclusion, Lines 478–483). Though the proposed loss aligns circuit-attenuation-induced logit changes, it is unclear whether matching these predictive effects corresponds to reasoning capability transfer instead of general intervention sensitivity. The paper does not build a direct connection between effect alignment and the preservation or acquisition of specific reasoning operations. The authors are advised to provide targeted mechanistic evidence or limit their claim of RCD to intervention-based functional distillation.
Questions for authors
I suggest the authors conduct a unified ablation study on the underlying mechanism: compare the distillation performance of EAP Circuit, Random Circuit, and Bottom-K Circuit, and recover the student circuits after distillation. This experiment can simultaneously verify the two core claims: whether the identified circuits are genuinely important, and whether RCD indeed preserves the student’s internal computational patterns.