- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a second-order measurement method for bias-free gated (SiLU) Transformer MLPs in pretrained models. Such an MLP F satisfies F(0)=0 and its jacobian J_F(0)=0, so Taylor's theorem with integral remainder writes F(x) as a Hessian integrated along the segment from the origin to x. Splitting the normalised MLP input into upstream sources, x = Σ r_i, then gives an exact sum of pairwise terms, F(x) = Σ_{i≤j} ΔF(r_i, r_j). Projecting a term onto the gradient of a downstream scalar gives a score per pair of sources. The same expansion is applied to attention logits, which are already bilinear in query and key. Validation is on SmolLM2-135M, with one behaviour replicated on Qwen2.5-0.5B: numerical closure of the sum; agreement between the pair terms and an interaction measured by a 2×2 ablation (Pearson 0.95-0.97 over 1152 pairs at four layers). They do patching of pair writes at the MLP output, as well as factorial removal of sources on five prompt families of twelve prompts each. They also do an analysis of the cancellation between terms, whose norm sum to 724-2868 times the norm of the output.
Critical strengths
- A sound and well motivated method that applies to pretrained models as is. A bilinear MLP can be studied through its weights only, but this cannot be done anymore when an activation function is added, like SiLU. Such an MLP, having no biases, vanishes at the origin together with its Jacobian. The integrated Hessian of Janizek et al. (2021), taken from the origin, then writes the output exactly as a sum of one term per pair of input parts, with no first-order term and no reference input to choose. The contribution is to specialise it to this architecture, to apply it to internal sources (the outputs of earlier attention and MLP layers), and to give a closed form that makes it cheap. The derivation in Appendix A-C is clear and, as far as I checked, correct. Since gradient-based attribution methods are first-order, a principled second-order tool is a useful addition.
Critical weaknesses
- The paper cannot be read through the main text only. The paper cannot be understood without first reading the annexes, the pair term , which is the central object of the paper, is never defined outside of the appendix (Eqs. 12 and 15 are in Appendix A). F is used from the introduction but only written down in Eq. 2, after the decomposition. The sources are only "named residual writes", their passage through RMSNorm is left to Appendix D, and does not denote the same object in Section 3.1 and in Appendix D. Sections 7 and 9 contain the test on the sources and the whole behavioral study, yet they report their results in a few sentences, without a table, and with undefined symbols (m, , $k_j $and in Section 7). Figures 1, 2, 6 and 7 are never cited in the text, and several cannot be read. Fig. 2B shows 8 sources without saying how they were chosen, one of which ("H1") is not defined. Panels C and D of Fig. 6 carry claims in their titles ("Causal selectivity sharpens with position-conditioned modes", "Sequence readout is structural, not topical"), with axes such as "scaffold / ordinary prose KL ratio" and "high / low window motif ratio" that are explained nowhere. "Figure 7" is a table inserted as an image, with entries such as "margin about −0.085 per top pair" and "near zero", and a "Readout" column of labels ("schema binding", "phase and retrieval", "order and suffix") that no experiment in the text supports. Reviewers are not required to read the appendix, and fixing this means rewriting most of the main text, which I do not consider a minor revision.
- The reported results only show that the decomposition works correctly, not that the pair reading it provides can be interpreted in a clear mechanism, as claimed in section 9. The decomposition is a Taylor identity that holds for any additive split of the input, the numerical closure reported in Section 4 therefore only checks the implementation.. The output-patching slope of 0.98 does not test the pair structure either, as the paper itself explains in Section 6. The informative test is Section 5, where the pair term predicts an interaction measured by ablation. However, it is only compared with first-order estimates, which give zero for an interaction by construction. The natural baseline is missing: for a small removal, the second difference of Eq. 4 equals up to terms in , that is, the ordinary Hessian at x, with no path integral. Without this comparison, Section 5 does not show what integrating from the origin adds, neither at nor at full removal, where the correlation drops to 0.79-0.92. At the same time, the paper states that the method "identifies source-pair circuits" and names pairs in Fig. 7, and its own results limit how far a single pair can be trusted. First, the top-ranked pair changes in four of five behaviours when the other RMSNorm convention is used (Table 1). Appendix D says that the main text states the convention wherever a pair is named, but I could not find it there. Second, the pair terms mostly cancel: their norms sum to 724-2868 times the norm of the output, and still to about 80 times after the projection of Section 8, as I understand it. A pair said to carry 47.3% of the output is then one term among many of opposite signs.
Questions for authors
Clarifications:
- In Section 5, how is the prediction of obtained from ? In particular, how does it scale with α, and how are both quantities converted to the nats of Fig. 3?
- For Section 7, could the authors give, per behaviour and per locus, the interaction, its interval, the control value and the number of examples?
- In Section 8, is 80 the sum of the absolute projected attributions divided by the norm of the output?
- What are the cancellation ratios and the cosine of Section 8 under the frozen-denominator convention? I ask because the path-integrated convention seems to create cancellation by itself, we can look at a toy example: Let the residual stream be , with two sources that are orthogonal and of equal norm. Take the RMSNorm gain equal to one in every coordinate. As in Appendix D, is the mean square of the residual, is the stabiliser, and is the part of the normalised input assigned to source , with . I write for the factor that appears in Eq. 23, and assume . In the frozen denominator case, . The two parts keep the directions of the sources, so they are orthogonal, and . However in the Path-integrated case, we have in Eq. 22, , and Eq. 23 gives and . Therefore and symmetrically for . Each source is multiplied by compared with the frozen convention, and a multiple of the full residual is subtracted so that the parts still sum to . It follows that This means that for between 8 and 10 (which corresponds to between about 30 and 1000 for ), the cosine is between and and each part is 4.1 to 5.0 times the norm of their sum, although the two sources were orthogonal. Section 8 reports cosines of to and parts "about five times the size of their sum". Moreover, grows like , so these values depend on a numerical constant. How much of the reported cancellation comes from the convention and from , and how much from the model?
- How was the patch scale in Section 6 chosen, and how do the slope and change with , up to (removal of the whole pair write)?
- Is panel D of Fig. 10 up to date? If so, which reported results depend on the controls marked as needed?
Additional results that would affect my assessment:
- In Section 5, how does the local Hessian at x, , predict the measured interaction compared with the integrated Hessian, at and at full removal?
- Which pairs are selected in Fig. 7 under each RMSNorm convention, and do the held-out effects hold for both?
Suggestions for future work (not needed for my assessment): sparse dictionary features as sources; interactions of more than two sources.