acceptodds
Under review as a conference paper at ICLR 2027

Reading Multiplicative Circuits in Transformer MLPs

Abstract

Transformer MLPs are usually read through activations, sparse feature dictionaries, or logit-space projections. These views underuse the multiplicative structure of gated feed-forward blocks, where the native unit of computation is an interaction between two sources. Circuit methods built on activation patching estimate a patch’s effect with a gradient, which is linear in the patch, so whatever two sources contribute jointly is assigned zero. We measure that joint contribution with a 2 × 2 ablation and find it runs from 0.12 to 1.19 times the size of the main effects those methods recover accurately. We then introduce behavior-conditioned integrated-Hessian tomography, a weight-native method that reads which residual-stream sources interact inside a pretrained gated MLP. Decomposing the normalized MLP input into named upstream sources decomposes the output into pairwise source interactions, and the decomposition is exact: the terms sum back to the MLP’s true output to within one part in ten million. It predicts the measured interaction at Pearson correlation 0.95 to 0.97 over 1152 pairs at four layers. The same construction applies to attention scores, which are already exactly bilinear and need no path integral, so the method covers two of the three computational units of a transformer block. Patching a pair’s write moves the next-token log-probability by the predicted amount, at slope 0.98 with R2 = 0.91. The terms that compose the MLP output are about three orders of magnitude larger than the output itself, with output cancellation growing as the square of input cancellation, so ranking pairs by the magnitude of their write recovers which sources are large. Projecting each write onto the layer output gives an exact signed attribution that accounts for the cancellation and ranks pairs independently of their size.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, gated MLPs, bilinear layers, integrated Hessian, circuit discovery, causal intervention, weight-space analysis, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qDx6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a second-order measurement method for bias-free gated (SiLU) Transformer MLPs in pretrained models. Such an MLP F satisfies F(0)=0 and its jacobian J_F(0)=0, so Taylor's theorem with integral remainder writes F(x) as a Hessian integrated along the segment from the origin to x. Splitting the normalised MLP input into upstream sources, x = Σ r_i, then gives an exact sum of pairwise terms, F(x) = Σ_{i≤j} ΔF(r_i, r_j). Projecting a term onto the gradient of a downstream scalar gives a score per pair of sources. The same expansion is applied to attention logits, which are already bilinear in query and key. Validation is on SmolLM2-135M, with one behaviour replicated on Qwen2.5-0.5B: numerical closure of the sum; agreement between the pair terms and an interaction measured by a 2×2 ablation (Pearson 0.95-0.97 over 1152 pairs at four layers). They do patching of pair writes at the MLP output, as well as factorial removal of sources on five prompt families of twelve prompts each. They also do an analysis of the cancellation between terms, whose norm sum to 724-2868 times the norm of the output.

Critical strengths
  • A sound and well motivated method that applies to pretrained models as is. A bilinear MLP can be studied through its weights only, but this cannot be done anymore when an activation function is added, like SiLU. Such an MLP, having no biases, vanishes at the origin together with its Jacobian. The integrated Hessian of Janizek et al. (2021), taken from the origin, then writes the output exactly as a sum of one term per pair of input parts, with no first-order term and no reference input to choose. The contribution is to specialise it to this architecture, to apply it to internal sources (the outputs of earlier attention and MLP layers), and to give a closed form that makes it cheap. The derivation in Appendix A-C is clear and, as far as I checked, correct. Since gradient-based attribution methods are first-order, a principled second-order tool is a useful addition.
Critical weaknesses
  • The paper cannot be read through the main text only. The paper cannot be understood without first reading the annexes, the pair term ΔF\Delta F, which is the central object of the paper, is never defined outside of the appendix (Eqs. 12 and 15 are in Appendix A). F is used from the introduction but only written down in Eq. 2, after the decomposition. The sources are only "named residual writes", their passage through RMSNorm is left to Appendix D, and rir_i does not denote the same object in Section 3.1 and in Appendix D. Sections 7 and 9 contain the test on the sources and the whole behavioral study, yet they report their results in a few sentences, without a table, and with undefined symbols (m, qiq_i, $k_j $and IsI_s in Section 7). Figures 1, 2, 6 and 7 are never cited in the text, and several cannot be read. Fig. 2B shows 8 sources without saying how they were chosen, one of which ("H1") is not defined. Panels C and D of Fig. 6 carry claims in their titles ("Causal selectivity sharpens with position-conditioned modes", "Sequence readout is structural, not topical"), with axes such as "scaffold / ordinary prose KL ratio" and "high / low window motif ratio" that are explained nowhere. "Figure 7" is a table inserted as an image, with entries such as "margin about −0.085 per top pair" and "near zero", and a "Readout" column of labels ("schema binding", "phase and retrieval", "order and suffix") that no experiment in the text supports. Reviewers are not required to read the appendix, and fixing this means rewriting most of the main text, which I do not consider a minor revision.
  • The reported results only show that the decomposition works correctly, not that the pair reading it provides can be interpreted in a clear mechanism, as claimed in section 9. The decomposition is a Taylor identity that holds for any additive split of the input, the numerical closure reported in Section 4 therefore only checks the implementation.. The output-patching slope of 0.98 does not test the pair structure either, as the paper itself explains in Section 6. The informative test is Section 5, where the pair term predicts an interaction measured by ablation. However, it is only compared with first-order estimates, which give zero for an interaction by construction. The natural baseline is missing: for a small removal, the second difference of Eq. 4 equals α2HF(x)[ri,rj]\alpha^2 H_F(x)[r_i, r_j] up to terms in α3\alpha^3, that is, the ordinary Hessian at x, with no path integral. Without this comparison, Section 5 does not show what integrating from the origin adds, neither at α=0.1\alpha = 0.1 nor at full removal, where the correlation drops to 0.79-0.92. At the same time, the paper states that the method "identifies source-pair circuits" and names pairs in Fig. 7, and its own results limit how far a single pair can be trusted. First, the top-ranked pair changes in four of five behaviours when the other RMSNorm convention is used (Table 1). Appendix D says that the main text states the convention wherever a pair is named, but I could not find it there. Second, the pair terms mostly cancel: their norms sum to 724-2868 times the norm of the output, and still to about 80 times after the projection of Section 8, as I understand it. A pair said to carry 47.3% of the output is then one term among many of opposite signs.
Questions for authors

Clarifications:

  1. In Section 5, how is the prediction of Iij(α)I_ij(\alpha) obtained from ΔF(ri,rj)\Delta F(r_i, r_j)? In particular, how does it scale with α, and how are both quantities converted to the nats of Fig. 3?
  2. For Section 7, could the authors give, per behaviour and per locus, the interaction, its interval, the control value and the number of examples?
  3. In Section 8, is 80 the sum of the absolute projected attributions ∣aij∣|a_ij| divided by the norm of the output?
  4. What are the cancellation ratios and the cosine of Section 8 under the frozen-denominator convention? I ask because the path-integrated convention seems to create cancellation by itself, we can look at a toy example: Let the residual stream be r=r1+r2∈Rdr = r_1 + r_2 \in \mathbb{R}^d, with two sources that are orthogonal and of equal norm. Take the RMSNorm gain ww equal to one in every coordinate. As in Appendix D, M=mean(r⊙r)=∥r∥2/dM = \mathrm{mean}(r \odot r) = \|r\|^2 / d is the mean square of the residual, ϵ\epsilon is the stabiliser, and nin_i is the part of the normalised input N(r)N(r) assigned to source ii, with n1+n2=N(r)n_1 + n_2 = N(r). I write L=asinh(M/ϵ)L = \mathrm{asinh}(\sqrt{M/\epsilon}) for the factor that appears in Eq. 23, and assume M≫ϵM \gg \epsilon. In the frozen denominator case, ni=ri/M+ϵn_i = r_i / \sqrt{M + \epsilon}. The two parts keep the directions of the sources, so they are orthogonal, and ∥ni∥=∥n1+n2∥/2\|n_i\| = \|n_1 + n_2\| / \sqrt{2}. However in the Path-integrated case, we have in Eq. 22, mean(r⊙ri)=M/2\mathrm{mean}(r \odot r_i) = M/2, and Eq. 23 gives c1=L/Mc_1 = L / \sqrt{M} and c2M≈(L−1)/Mc_2 M \approx (L - 1)/\sqrt{M}. Therefore ni=1M[L ri−L−12 r],that is,n1=1M[L+12 r1−L−12 r2],n_i = \frac{1}{\sqrt{M}} \left[ L \, r_i - \frac{L-1}{2} \, r \right], \quad \text{that is,} \quad n_1 = \frac{1}{\sqrt{M}} \left[ \frac{L+1}{2} \, r_1 - \frac{L-1}{2} \, r_2 \right], and symmetrically for n2n_2. Each source is multiplied by LL compared with the frozen convention, and a multiple of the full residual rr is subtracted so that the parts still sum to N(r)N(r). It follows that cos⁡(n1,n2)=−L2−1L2+1,∥ni∥∥n1+n2∥=L2+12.\cos(n_1, n_2) = -\frac{L^2 - 1}{L^2 + 1}, \qquad \frac{\|n_i\|}{\|n_1 + n_2\|} = \frac{\sqrt{L^2 + 1}}{2}. This means that for LL between 8 and 10 (which corresponds to MM between about 30 and 1000 for ϵ=10−5\epsilon = 10^{-5}), the cosine is between −0.97-0.97 and −0.98-0.98 and each part is 4.1 to 5.0 times the norm of their sum, although the two sources were orthogonal. Section 8 reports cosines of −0.976-0.976 to −0.991-0.991 and parts "about five times the size of their sum". Moreover, LL grows like 12log⁡(1/ϵ)\tfrac{1}{2}\log(1/\epsilon), so these values depend on a numerical constant. How much of the reported cancellation comes from the convention and from ϵ\epsilon, and how much from the model?
  5. How was the patch scale α=0.2\alpha = 0.2 in Section 6 chosen, and how do the slope and R2R^2 change with α\alpha, up to α=1\alpha = 1 (removal of the whole pair write)?
  6. Is panel D of Fig. 10 up to date? If so, which reported results depend on the controls marked as needed?

Additional results that would affect my assessment:

  1. In Section 5, how does the local Hessian at x, α2HF(x)[ri,rj]\alpha^2 H_F(x)[r_i, r_j], predict the measured interaction compared with the integrated Hessian, at α=0.1\alpha = 0.1 and at full removal?
  2. Which pairs are selected in Fig. 7 under each RMSNorm convention, and do the held-out effects hold for both?

Suggestions for future work (not needed for my assessment): sparse dictionary features as sources; interactions of more than two sources.

Reviewer_CmwqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Modern LLMs use SwiGLU which is gated MLPs, where each input projected twice, which allow the interaction between parts of the input

The previous circuit-finding tools (gradient-based, first-order tools) cannot represent such interactions. This paper extends the previous tools to use a second-order (integrated-Hessian) decomposition, and it splits the output into pairwise source interactions. It shows the pairwise interactions is also as large as the first order output.

Critical strengths

I don't have any critical strengths for this paper.

Critical weaknesses
  1. The paper shows that first-order methods miss the large interaction term, but it doesn't show how important the missing is.
  2. The results that "The pairwise interactions as large as the main effects" is from a weird ablation, which by setting MLP input to zero. It is an unreal change to the model’s input
Questions for authors

The slope 0.98 results is true for any vector, so it says nothing about the decomposition. Can you explain this?

Reviewer_fGUZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a source-pair decomposition for bias-free gated Transformer MLPs. Using an integrated Hessian along the path from zero to the normalized input, the method expresses the MLP output as a sum of pairwise contributions from upstream sources. These contributions can be projected onto a downstream behavior gradient or the layer output direction for attribution. The paper also applies a direct bilinear decomposition to pre-softmax attention scores. Experiments on SmolLM2-135M, with a targeted replication on Qwen2.5-0.5B, examine numerical reconstruction, agreement with factorial source ablations, intervention calibration, and cancellation among pair contributions.

Critical strengths
  1. An exact decomposition that applies to pretrained gated MLPs. The method preserves the trained nonlinearity and decomposes the vector output directly, with reconstruction errors around 10−710^{-7}. The channel-wise formulation also makes the analysis practical without constructing a full output Hessian. This provides a useful way to study source interactions in existing models, beyond the constant-Hessian setting of purely bilinear MLPs.

  2. The validation distinguishes reconstruction, intervention prediction, and evidence for source interactions. Section 5 compares the decomposition with factorial source ablations across four layers, including randomly selected pairs. Section 6 explicitly acknowledges that accurate output-write patching does not by itself establish the proposed source mechanism. The analysis of cancellation further shows why ranking pairs by write magnitude can be misleading. Together, these results make the paper informative beyond the decomposition identity alone.

Critical weaknesses
  1. The identification of individual source pairs is sensitive to the RMSNorm attribution convention. Appendix D reports reasonably high overall ranking agreement between two conventions, but the highest-ranked pair is shared in only one of five behaviors, and top-five overlap is as low as two out of five. This matters because several conclusions identify particular source pairs as circuits. The sensitivity analysis is useful, but its implications should be clearer in the main text. It remains uncertain whether the reported source-level causal conclusions are stable across these conventions.

  2. The evidence for behavior-specific MLP mechanisms is narrower than the evidence for the decomposition itself. Section 7 reports clear MLP source-interaction evidence for capital facts, while the remaining behaviors mainly support a block-level interpretation. Each behavior family also contains only six discovery and six held-out prompts, and the Qwen replication focuses on capital facts. These results support the method as an interaction attribution tool, but provide more limited support for broadly identifying behavior-specific multiplicative circuits. A consolidated report of the source-level factorial effects, uncertainty intervals, and matched controls would make this distinction easier to assess.

Questions for authors
  1. Robustness of the circuit conclusions: When the leading pair changes between the two RMSNorm conventions, do the pairs selected under each convention retain comparable factorial effects on held-out examples? Do the main behavior-level conclusions remain the same?

  2. Definition of the interaction prediction: In Section 5, what exact quantity derived from ΔFij\Delta F_{ij} is compared with Iij(α)I_{ij}(\alpha), including its scaling with α\alpha, the off-diagonal factor, and any scalar projection? Please also clarify why agreement is expected, given that the integrated Hessian is evaluated along the origin-to-input path, whereas the factorial ablation probes a neighborhood of the endpoint.

  3. Strength of the source-level evidence: Could the source-level factorial results in Section 7 be reported in one table, with effect sizes, confidence intervals, and the corresponding matched controls for all five behaviors? In particular, which MLP conclusions are supported beyond a block-level effect?

  4. The manuscript has several formatting and readability issues. The title, abstract heading, and section headings do not follow the small-caps styling specified in the ICLR template. Several figures contain text that is difficult to read at normal viewing size, particularly Figures 1, 2, and 6. Figure 7 presents tabular results as an image and would be clearer as a properly formatted table. I also could not find the required AI use statement in the supplied PDF. These issues should be corrected, and template compliance should be checked.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.