- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to study a bridge between two different branches of model interpretability: mechanistic interpretability, which is based on causal analysis of fixed models, and singular learning theory, which is based on understanding the local geometry of the parameters of the learned models. The authors postulate and justify several connections between geometrical notions and interventional ones. They experiment with one inferred circuit for one task to demonstrate the framework.
Critical strengths
The connection between interpretability focused on activations (MI) and interpretability based on parameters (SLT) is quite interesting and could have potential for future work and applications. As elaborated below, I think the paper could frame geometrical considerations as a way to address potential difficulties in MI and contributes positively to the discussion.
Critical weaknesses
The stated main contribution over previous SLT work is that previously there was no reference mechanistic explanation at large scale. This paper supposedly provides the answer via a circuit validated by interventions. However, this is not enough; circuit discovery is not sufficient for validating a mechanistic explanation. It is widely documented that circuits are massively overdetermined in LLMs (see [1], [2], [3], [4] for example). A circuit component can have a high causal effect and just not be selected in the circuit because another redundant one next to it had a slightly higher effect. Similarly, low-causal-effect components can be incorporated to create a connected path between high-scoring ones. Generally, circuit discovery is not stable under hyperparameter and data variations.
In fact, I think the paper would benefit from framing itself more in terms of whether SLT aligns with intervention, maybe directly using the mediation analysis framework (see [5], [6]), and positioning its findings in the broader discussion about the MI overdetermination problem and the lack of identifiability of its target. Intervention and causal mediation analysis are more fundamental than circuits. I believe SLT could in fact help MI there by providing the geometrical signal.
Apart from being an interesting fact on its own, why is it beneficial to know that components of the Fisher information track causal effects? It seems it remains much more efficient to just perform the intervention than to estimate the Fisher information. I am not saying that there are no benefits (especially regarding the statement above), but I feel the paper does not sufficiently motivate this geometrical bridge.
In the end, the bridge between LLC and MI developed in Section 4 is mostly an intuition; it is neither made properly formal with theoretical results nor supported by more than limited experiments with one circuit for one task. I believe an ICLR paper would need either a strong formal framework or substantial empirical justification.
Overall, I would recommend focusing the experiments not on circuits, but on the causal mediation analysis step before the creation of the circuit. Not only would it be more fundamental, but it could also be even clearer and more impactful.
[1] Dead Salmons of AI interpretability. Meloux et al. [2] The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching. Vaidyanathan et al. [3] The Hydra Effect: Emergent Self-repair in Language Model Computations. McGrath et al. [4] Mechanistic interpretability as statistical estimation: A variance analysis. Meloux et al. [5] Locating and editing factual associations in gpt. Meng et al. [6] The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. Mueller et al.
Questions for authors
Independent of circuits, can you summarize how the geometrical quantities you consider relate to causal mediation analysis ones? What is the computational cost of estimating the various geometric quantities you consider in the experiments? What would be the downstream benefits of the SLT–MI bridge?
3 comments
(2/2)
Q1. Through one metric, to second order (Proposition 1). For a change in a component's output, the KL effect is plus higher-order terms, with (the submission's ) the Fisher metric the rest of the network induces on that output. Zero ablation (), interchange () and Gaussian weight noise () differ only in the second moment of , and the component's Fisher matrix is .
| Quantity | Relation to mediation | Agreement with effects |
|---|---|---|
| Second-order values | Taylor term of each intervention | rank the full effects at 0.96 or above |
| Trace | second-order value of weight noise | 0.27 to 0.51 with NIE over heads |
| Count | counts directions stiffer than | 0.70 to 0.90 with NIE, 0.67 to 0.90 with PIE |
In the terms of [2], NIE is our interchange (run on with the head's output from ), and PIE the denoising patch (run on with the output from ), each read as KL against its own run (Tables 2, 32; four tasks). The trace is dominated by value weights, which hold three quarters of it in RMS coordinates yet barely rank effects; the count saturates there. Among quantities that use neither an intervention nor , the count ranks interchange effects level with or ahead of all. Given zero-ablation KL, layer and output norm, it keeps a partial correlation of 0.41 to 0.68, confirmed in 16 further cells under a rule fixed in advance (Appendix C.2). On the contrast, where [2]'s interaction NIE PIE is defined, the count ranks its size at 0.47 to 0.69 (Table 32).
Q2. One run each on Pythia-1B, about 2,500 tokens (Table 28):
| Quantity | Passes | Time |
|---|---|---|
| Zero ablation, all components | one partial forward per component | 13 s |
| Interchange, all components | the same, plus one forward on | 13 s |
| Second-order values, all components | one backward per sampled output (4 per ranking, 16 per position) | 0.5 s each |
| Count, one head | 128 Fisher-vector products | 17 s |
| Schur fraction, one pair | truncated conjugate gradients | 32 s, a fifth of the retraining |
| Estimate on the true loss, one head | Metropolis-corrected chain, 16,384 tokens | 4,600 s |
Q3.
For MI:
- Which weights can take over a lost function: the Schur fraction predicts repair before any retraining (W2).
- A detector with no replacement value. The fixed-horizon response (the submission's nonlocal SGLD estimate) puts all eight layer-0 heads among its 23 largest of 144 components, while single zero ablation ranks them as low as 53rd. Mean replacement hides their joint role on synthetic strings (contrast drop 0.14, against 1.78 for zeroing) but not on natural bracket contexts (1.69; Table 52).
- Cheap effects at single positions: 16 sampled-output backward passes rank the interchange KL of single token positions at 0.96 to 0.99, where patching needs one forward pass per patched position.
- Two functions of one head. In three Pythia-14m runs where induction heads form, the value-output count at ranks interchange effects at 0.83 to 0.89, and the query-key count over to tracks attention entropy at 0.66 to 0.78, both fixed in advance (Appendix D.2).
For SLT, interventions supply known answers: a token lookup whose LLC follows from its mechanism (Proposition 4), and a singular point (, , zeroed) where the estimate follows . They also pick out the count, not the trace, as the summary of the spectrum that tracks mechanism.
Thank you for the detailed and substantive response. If integrated properly in the paper these could be useful improvements that strengthen the paper.
I still think that the proposed connection between SLT and MI is interesting and has potential. However, I am not yet convinced that the paper is mature for publication. The new theoretical and empirical material introduced in the revision would benefit from a systematic presentation and discussion and integration in a paper that was already fairly hard to foloow. In particular, it will be important to clarify the central contributions, their limitations, and the practical value of the proposed geometrical approach.
Overall, I would encourage the authors to consolidate these developments into a more coherent and thoroughly developed paper for a future submission. I will therefore maintain my overall score.
(1/2)
Section, table and proposition numbers refer to the revised PDF. Unless stated, effects are KL from the clean prediction at scored positions, at Pythia-1B's final checkpoint, and the count is read at T=105. We number your weaknesses in order.
W1. We agree, and measured it (Appendix E.1). A greedy search that restores components until 80% of the contrast (correct minus incorrect closer logit) is recovered picks nearly the same set when strings from our generator are resampled (Jaccard 0.96 over 100 bootstraps). It picks different sets when the replacement value, the number of bracket types or the output metric changes (Jaccard 0.16 to 0.53), and at least ten sets with pairwise Jaccard at most 0.5 each recover 80%. The revision therefore keeps the circuit only as a label (Section 5.3) and rests every main result on per-component mediation effects (W5).
W2. The revision adopts this framing. Following [5, 6], Section 5.2 compares geometry with causal mediation effects, with no circuit, and following [1] it treats a mediation estimand as identifiable where the locus of a mechanism is not. Overdetermination shows in the effects. On KL, the noising effects of all components sum to about twice the total effect, the interaction [2] warns of (Appendix C.2). A redundant group, the eight layer-0 heads, has a joint ablation larger than the sum of its single ablations (Section 5.4). It shows in the weights too. When one component's output is zeroed, retraining only another component's weights recovers a median 0.92 of the lost KL over 54 pairs. The Schur fraction, computed from the Fisher matrix before any retraining, ranks that recovery at 0.79, against 0.15 for the repairer's zero-ablation effect (Appendix F.1).
W3. For ranking whole components on one task, the intervention is the direct tool, and Section 6 says so. The bridge gives what interventions do not: a test of SLT's estimator against them, predictions about weights, a detector that needs no replacement value, and cheap effects at single positions (Q3), at the costs in Q2.
W4.
W5. Now, section 5.2 compares geometry with per-component mediation effects, without any circuit: across 128 heads and four tasks, and across 148 checkpoints for all 144 components. Section 5.3 then treats the circuit as a label.