- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the effects of Transformer components (blocks and attention heads) on model outputs, with a particular focus on second-order interactions between components. At the block-write level, the authors use per-context gradients (one backward pass) to measure the contribution of each write to the output after propagation through subsequent layers (equivalent to attribution patching). They use 6-node Gauss–Legendre quadrature to recover the magnitude of deletion effects (equivalent to integrated gradients along the ablation path) and propose second-order attribution patching: a single backward pass on the run with component deleted estimates its interactions with all other components. They also provide an error identity and a symmetrized formulation. Experiments across five models (0.36B–70B parameters) use actual ablations as ground truth. The per-context readout achieves a Spearman correlation of 0.69–0.82 with deletion effects (compared with 0.20–0.29 for logit lens/DLA). The second-order estimator achieves 0.65–0.76 for models up to 8B parameters and 0.56 for the 70B model in bf16. Additional experiments investigate sign, mediation, head-level IOI, comparisons with AtP*, and nine attempts to "turn attribution into capabilities."
Critical strengths
-
Explicitly models second-order interactions between components. It can identify component pairs whose individual effects are weak but whose joint effects are strong due to redundancy or self-repair—interactions that methods ranking components solely by their individual effects cannot discover.
-
Relatively low computational cost. Estimating all pairwise interactions requires approximately forward-pass equivalents, compared with forward passes for pairwise measurements. The advantage becomes increasingly significant as the number of components grows, such as when analyzing hundreds of attention heads.
Critical weaknesses
-
The baselines are too weak. The primary baselines for the first-order method are logit lens, DLA, and average transport (Jacobian lens and tuned lens), none of which were designed to predict deletion effects. The authors themselves acknowledge that this "does not test what lenses were built for," making it an unfair comparison to evaluate these methods against actual deletion effects. More relevant baselines (AtP*, EAP-IG, and Hessian–vector-product corrections) are only partially evaluated: AtP* is compared on Pythia-410M using just 10 prompt pairs, yielding nearly identical rankings (0.950 vs. 0.945). No quantitative comparisons with EAP-IG or HVP corrections are provided. For the second-order method, the comparisons are limited to an additive assumption and rankings based on individual component effects, with no comparisons against other budget-constrained search methods or second-order estimators.
-
The methodological novelty is relatively limited overall. The authors acknowledge in Appendix O that the first-order readout is simply attribution patching, path integration is equivalent to integrated gradients (EAP-IG), and response decomposition is standard causal mediation analysis. The primary novel contribution is therefore the second-order estimator, which is essentially a finite-difference approximation based on changes in gradients after ablation.
Questions for authors
-
Compared with prior methods, is the contribution of this paper limited primarily to modeling second-order effects? What are the precise differences between the first-order readout and AtP, and between the curvature correction and IG/EAP-IG?
-
Could the authors provide quantitative comparisons on the same metric and dataset against AtP* (including QK correction and GradDrop), EAP-IG (with varying numbers of integration steps), and the second-order HVP correction proposed by Zhang & Wang? These comparisons should report both attribution accuracy and computational cost.
-
Could the authors report the computational overhead across models of different sizes and at the attention-head level, where there may be hundreds of components, including setup and initialization costs?
Thank you for pressing on novelty and baselines; we answer your three questions in order.
Novelty and precise differences (W2, Q1). Methodologically, yes: the contribution is the second-order estimator. Gradient-times-write is AtP with a removal (zero) baseline, its removal-path integral is integrated gradients, and the mediation decomposition is standard; we claim none of these. What is new is that one backward pass on the component-removed run estimates that component's interaction with every other component, with an exact marginal-error identity and a symmetrised form.
The difference from EAP-IG is the path, not the principle. EAP-IG integrates along one interpolation of inputs or patched activations shared by all edges; our integral scales a single component's write from 1 to 0 with everything downstream recomputed, so it equals that component's removal effect. On that path the paper already reports a step sweep (Sec. 4.4, Fig. 9a, Qwen3-0.6B): pooled over blocks 1-6, median relative error 0.69 with one node (AtP), 0.47 with a local second-order correction (one extra gradient at s=0.1) and 0.38 with the trapezoid rule; six Gauss-Legendre nodes reach below 1e-4 over blocks 1-27, and Llama-3-8B repeats the ladder (first order 0.40, trapezoid 0.12, six nodes at most 0.003).
Relative to a second-order Taylor estimate, the finite step carries most of the gain. Against a clean-point gate-Hessian (HVP) baseline at full removal (Sec. 4.5, Table S.1, Fig. S.1), the symmetrised estimate reduces paired MAE by 0.056 [0.043, 0.069], 0.077 [0.049, 0.113] and 0.028 [0.015, 0.041] nats on pythia-410m, Qwen3-0.6B and SmolLM2-360M (40 documents each) and raises signed Spearman by 0.280, 0.279 and 0.263 (from 0.46-0.54 to 0.74-0.81); forward-only conditioning alone accounts for 0.21-0.25 of that gain. At doses up to 0.1 both are accurate, with the HVP's MAE 1.7-2.0 times ours, as a third-order expansion predicts; the ratio falls to 1.2-1.4 at full removal. This baseline is our implementation in removal-gate coordinates, not a native replication of Zhang and Wang's activation-patching HVP or multi-step HVP.
Baselines (W1, Q2). The new comparisons add a second-order estimator (the HVP) and further budget-constrained search strategies (AtP-product ordering, conditional row scans) to the additive and single-effect baselines you mention. AtP*: the existing pythia-410m comparison includes both the QK correction and GradDrop but covers only ten prompt pairs; verification counts for the true top-5/10/20 are 50/58/89 for AtP against 20/26/54 for AtP* (Table 20). We treat it as a limited comparison, not a replication. HVP and search: in a fixed pool of 128 pythia-410m heads (8,128 pairs, 40 IOI prompt families) with measured costs (Tables S.2-S.3, Fig. S.2), at 10% of the exhaustive time the symmetrised ranking recovers 0.617 of the exact top-10% interactions against 0.396 for single-effect ordering (+0.221 [0.203, 0.240]; 40/40 families), 0.380 for AtP-product ordering and 0.368/0.352 for single-/AtP-ordered row scans, with both rankings at 512 verified pairs (ceiling 0.630), i.e. at equal verification counts. The clean-point HVP ranks well (rank correlation 0.923 versus 0.972), but its longer setup delays its first batch to 10.3-10.6% of the exhaustive time, so it scores 0 at 10% and 0.825 at 20% against 0.981. EAP-IG: the step ladder above. Native expanded AtP*, input-level EAP-IG step sweeps, Zhang-Wang's activation-patching HVP/MS-HVP and Fisher-energy CoAx were not run; the paper lists them as limitations rather than presenting our adaptations as those benchmarks. At block level, measuring all pairs exactly costs only 2.6-3.2 estimator runs (Table S.1), so we claim accuracy per evaluation, not a speed-up.
Overhead (Q3). Per document, one estimator run costs 3.61, 6.48 and 5.63 s on pythia-410m, Qwen3-0.6B and SmolLM2-360M (HVP 3.85/6.75/5.88 s; exact all pairs 9.54/18.94/18.23 s). At head level the median setup per family is 1.62 s for the symmetrised estimate and 3.05 s for the HVP, 3.3% and 6.3% of the median exhaustive time of 48.5 s. Loading model and tokenizer takes 1.2-1.4 s per head worker and 1.2-2.2 s for the block models; the 83.5-84.3 s benchmark warm-up includes one full exact reference over all 8,128 pairs (48.7-49.2 s) plus ten policy warm-ups; cached singles cost 0.77 s per family (Appendix S.4). These are FP64 timings of our batched implementation on one GPU type, not generic cold-start latency.
All new measurements are FP64 runs on the same saved weights, with no new 8B or 70B runs; the head result is conditional on its restricted pool and the IOI template and name generator.