- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
How would we even go about falsifying the common mech interp belief that "if we understand open source LLMs, that will help explain the behavior of closed LLMs too"? This paper notes that so long as a closed API gives access to log-odds, there are two API-compatible ways to operationalize "behavior": 1. The log odds themselves (referred to as "prediction-level") and 2. given some intervention, the projection of the difference between representations (which authors show is equal to the difference between log-odds: called "attribution-level").
The authors then study whether correlation between "mechanistic-level" metrics (e.g. attention patterns etc) of different models means that their "prediction-level" and "attribution-level" behavior also match. They claim that in general they do not: hence, ç
The validation is necessarily restricted to API-compatible interventions, so the authors focus on leave-one-out sentence ablations. These interventions and the corresponding cross-metric relationships are studied across a variety of open source and closed models (although notably not the latest GPT-5 or Claude models since they do not release log-probabilities).
Critical strengths
- I particularly appreciated the contribution of the "atttribution level": noting that the difference in log odds from two inputs is equal to the a projection of the difference-vector between the representations themselves. Section 2 and Figure 2 were very clear explainers of this. I was surprised that any representational information could be derived from the difference of log odds (even if it is far short of the information content of the representations themselves).
- While I have some hesitations about the negative "mechanistic-to-causal" results, I want to highlight a conclusion that holds regardless, paraphrased from line 428: Most mechanistic-level prediction of attributions are outperformed by API-compatible attribution metrics alone. In my own words: If the point of mech-interp is to understand model behavior, there are better ways to do so that require less model access! Which seems like a potentially large boon for public auditing of closed models.
Critical weaknesses
- Attention Fidelity is incorrectly treated as equivalent to Representational Fidelity.
Attention mechanisms alone are linear, and can't encode the nonlinear complexity of the MLP layers in transformers. Given this fact, it's surprising that the negative conclusions in this paper rely so heavily on the over-alignment of attention patterns in Tables 1 and 2! If they instead had chosen to study just the residual stream representations (F_mag and F_align in those tables) which encode the information from both the linear attention heads and the nonlinear MLPs, then the conclusion would not be "mechanistic info doesn't predict attributions" but just the unsurprising "only directional info predicts attributions" (unsurprising due to the magnitude rescaling of LayerNorm).
In short, the more appropriate thesis seems like "We can use surrogate fidelity to assess which mechanistic info is actually predictive of attributions": F_align is the only one which doesn't report an artificially high agreement. Not that "Most representation-level properties do not predict attributions well" (line 428).
Questions for authors
- Could you please clarify the so-what takeaways for Figure 3b and Figure 4? Even on reread, it's still quite unclear. These captions may benefit from a one-sentence takeway summary, to guide readers.
3 comments
Thank you for the edits to the paper, they look like a suitable improvement. I'm still a bit confused by Figure 4 but the caption seems better this time around.
How about "Mechanistic"? That would align nicely with your abstract and main-body prose. Or "Attention"?
This is a good suggestion. We have updated the paper to use "mechanistic." Thank you!
Thank you for your thoughtful comments. Your alternative thesis is helpful framing, and it's fairly close to where the analyses in the new appendices F and G ended up.
Attribution fidelity is not representation fidelity: We agree that attention pattern agreement does not mean agreement of residual stream representations, and by that metric only Fmag and Falign are true representation-level metrics. In our paper, we use "representation level" to mean "requires access to internals," as a distinct rung on the access hierarchy of Figure 1. We do not believe we treat any of our "representation-level" quantities as equivalent and only seek to test whether they (a) transfer across models, or (b) predict attributions.
We are open to suggestions for a more neutral term than "representation" to convey "internals-based."
Captions: We have expanded and clarified the captions for figures 3 and 4 in the revised draft.
Presentation: We have simplified the notation in Section 2, which now uses h for the residual stream and z for the normalized last-layer representation, and corrected Equation 2 accordingly.