- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GRaph Attribution Distillation (GRAD), which supplements standard knowledge distillation with a Jensen–Shannon loss matching attribution-routing profiles between functionally aligned groups of MLP neurons. Task-specific ANOVA probes define shared supernodes across an 8B teacher and 1B/3B students. Experiments train on two-digit addition and evaluate several held-out arithmetic families, with representation-matching, scrambled-target, and compute controls. A detailed audit shows that the largest 1B multiplication gain arises from correcting a tendency to hedge under a BOS-token format mismatch; a simple commitment loss reproduces the improvement. The paper therefore contributes both a graph-based objective and a concrete investigation of what an apparent mechanism-transfer gain actually measures.
Critical strengths
The strongest contribution is the explicit causal investigation of the apparent generalization gain. Section 5 separates commitment from correctness, evaluates both BOS conventions, transplants layers, patches residual states, and tests a targeted commitment loss. Table 3 shows that this simple loss reproduces the multiplication improvement, and Table 8 reports the absence of the advantage under consistent BOS training. These controls expose a concrete alternative explanation and make the paper informative even where the original mechanism-transfer interpretation is weakened.
Critical weaknesses
The practical advantage of GRAD itself remains unestablished once the identified format confound is controlled. Under the standard BOS convention, Table 8 does not show a benefit over KD; under the mismatched convention, the much simpler commitment objective matches or exceeds GRAD in Table 3. Meanwhile, a warm-cache GRAD step costs about 14 times a KD step, with substantial additional teacher-cache construction cost. The 3B result is promising, but has not received the same commitment/format audit. Thus, the evidence currently supports the methodological audit more strongly than a broadly useful circuit-distillation method, and the title and conclusion should reflect that balance.
Questions for authors
- For the 3B student, do the operand-count gains persist when training and evaluation use the same BOS convention and when KD receives the commitment loss? This comparison would help establish whether any remaining benefit is specific to the graph objective.
- How do feature specificity and routing differ between GRAD and the commitment-only control? Do aligned supernode interventions show a teacher-like computational change beyond the commitment effect, and which exact arithmetic labels or masks are required to construct the training target?