acceptodds
Under review as a conference paper at ICLR 2027

Bridging Singular Learning Theory and Mechanistic Interpretability at the Billion-Parameter Scale

Abstract

Mechanistic interpretability (MI) explains a computation in a trained network by identifying the components and variables that implement it and testing them with interventions. Singular learning theory (SLT) describes the same network through the geometry of the loss around its parameters, summarized by the local learning coefficient, whose estimates track stages of training. In pretrained language models the two have not been compared: SLT's estimates at this scale concern whole models or components whose roles were not independently established, and the estimator, which averages the loss over parameter settings drawn near the trained ones, has had no reference for how far those draws may move the parameters. We show that both are functions of one object, the map from a component's parameters to the model's predictions under inputs and interventions, whose first derivatives determine the Fisher information, the learning coefficient, and intervention effects, and whose higher derivatives add a quartic correction and pairwise terms. We test this correspondence in Pythia-1B on Dyck bracket matching at all 154 training checkpoints, against a five-stage circuit identified by interventions. We measure the local geometry of all 144 components inside the region where predictions are preserved, with the Fisher spectrum at every checkpoint, which gives the quadratic part of the learning-coefficient estimate at every temperature. Changes of the Fisher trace track changes of causal effect (median partial rank correlation between adjacent checkpoints, replicated on independent inputs), largely as a trend shared across the model, whereas the gain Fisher of a component's output and the large-temperature count are specific to the component. The count follows causal change across the population but not the growth of a formed circuit component. The quartic part is what a sampler that leaves the region registers beyond curvature. Pairwise interaction curvature tracks joint-ablation interactions.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, developmental interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mbouOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to study a bridge between two different branches of model interpretability: mechanistic interpretability, which is based on causal analysis of fixed models, and singular learning theory, which is based on understanding the local geometry of the parameters of the learned models. The authors postulate and justify several connections between geometrical notions and interventional ones. They experiment with one inferred circuit for one task to demonstrate the framework.

Critical strengths

The connection between interpretability focused on activations (MI) and interpretability based on parameters (SLT) is quite interesting and could have potential for future work and applications. As elaborated below, I think the paper could frame geometrical considerations as a way to address potential difficulties in MI and contributes positively to the discussion.

Critical weaknesses

The stated main contribution over previous SLT work is that previously there was no reference mechanistic explanation at large scale. This paper supposedly provides the answer via a circuit validated by interventions. However, this is not enough; circuit discovery is not sufficient for validating a mechanistic explanation. It is widely documented that circuits are massively overdetermined in LLMs (see [1], [2], [3], [4] for example). A circuit component can have a high causal effect and just not be selected in the circuit because another redundant one next to it had a slightly higher effect. Similarly, low-causal-effect components can be incorporated to create a connected path between high-scoring ones. Generally, circuit discovery is not stable under hyperparameter and data variations.

In fact, I think the paper would benefit from framing itself more in terms of whether SLT aligns with intervention, maybe directly using the mediation analysis framework (see [5], [6]), and positioning its findings in the broader discussion about the MI overdetermination problem and the lack of identifiability of its target. Intervention and causal mediation analysis are more fundamental than circuits. I believe SLT could in fact help MI there by providing the geometrical signal.

Apart from being an interesting fact on its own, why is it beneficial to know that components of the Fisher information track causal effects? It seems it remains much more efficient to just perform the intervention than to estimate the Fisher information. I am not saying that there are no benefits (especially regarding the statement above), but I feel the paper does not sufficiently motivate this geometrical bridge.

In the end, the bridge between LLC and MI developed in Section 4 is mostly an intuition; it is neither made properly formal with theoretical results nor supported by more than limited experiments with one circuit for one task. I believe an ICLR paper would need either a strong formal framework or substantial empirical justification.

Overall, I would recommend focusing the experiments not on circuits, but on the causal mediation analysis step before the creation of the circuit. Not only would it be more fundamental, but it could also be even clearer and more impactful.

[1] Dead Salmons of AI interpretability. Meloux et al. [2] The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching. Vaidyanathan et al. [3] The Hydra Effect: Emergent Self-repair in Language Model Computations. McGrath et al. [4] Mechanistic interpretability as statistical estimation: A variance analysis. Meloux et al. [5] Locating and editing factual associations in gpt. Meng et al. [6] The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. Mueller et al.

Questions for authors

Independent of circuits, can you summarize how the geometrical quantities you consider relate to causal mediation analysis ones? What is the computational cost of estimating the various geometric quantities you consider in the experiments? What would be the downstream benefits of the SLT–MI bridge?

3 comments

(1/2)

Section, table and proposition numbers refer to the revised PDF. Unless stated, effects are KL from the clean prediction at scored positions, at Pythia-1B's final checkpoint, and the count is read at T=105T=10^5. We number your weaknesses in order.

W1. We agree, and measured it (Appendix E.1). A greedy search that restores components until 80% of the contrast (correct minus incorrect closer logit) is recovered picks nearly the same set when strings from our generator are resampled (Jaccard 0.96 over 100 bootstraps). It picks different sets when the replacement value, the number of bracket types or the output metric changes (Jaccard 0.16 to 0.53), and at least ten sets with pairwise Jaccard at most 0.5 each recover 80%. The revision therefore keeps the circuit only as a label (Section 5.3) and rests every main result on per-component mediation effects (W5).

W2. The revision adopts this framing. Following [5, 6], Section 5.2 compares geometry with causal mediation effects, with no circuit, and following [1] it treats a mediation estimand as identifiable where the locus of a mechanism is not. Overdetermination shows in the effects. On KL, the noising effects of all components sum to about twice the total effect, the interaction [2] warns of (Appendix C.2). A redundant group, the eight layer-0 heads, has a joint ablation larger than the sum of its single ablations (Section 5.4). It shows in the weights too. When one component's output is zeroed, retraining only another component's weights recovers a median 0.92 of the lost KL over 54 pairs. The Schur fraction, computed from the Fisher matrix before any retraining, ranks that recovery at 0.79, against 0.15 for the repairer's zero-ablation effect (Appendix F.1).

W3. For ranking whole components on one task, the intervention is the direct tool, and Section 6 says so. The bridge gives what interventions do not: a test of SLT's estimator against them, predictions about weights, a detector that needs no replacement value, and cheap effects at single positions (Q3), at the costs in Q2.

W4.

  • Formal. Seven propositions with proofs (Appendices A.2, A.3): (1) one metric for ablation, interchange and weight noise; (2) the count tends to the LLC when the Fisher rank is constant nearby; (3) an exact finite-TT formula for the estimate when curvature varies along flat directions; (4) a lookup component whose LLC follows from its mechanism; (5) for a quadratic loss, a pair's joint estimate is at most the sum of single ones; (6) pair interactions the Fisher matrix cannot see; (7) the first correction beyond the quadratic. The submission's Propositions 1, 2 and 3 are now 2, 6 and 7.
  • Empirical. No result needs the circuit. The count's lead over the trace was tested in 32 cases fixed in advance: zero-ablation and interchange KL on IOI, greater-than, induction and subject-verb agreement at three Pythia-1B checkpoints, and on four tasks in Pythia-160m. It held in direction in all 32 and cleared the preset bar in 27 (Appendix C.2). Over the 384 heads of Pythia-410m on IOI, the count ranks zero-ablation and interchange effects at 0.82 and 0.81, against 0.20 and 0.48 for the trace.

W5. Now, section 5.2 compares geometry with per-component mediation effects, without any circuit: across 128 heads and four tasks, and across 148 checkpoints for all 144 components. Section 5.3 then treats the circuit as a label.

(2/2)

Q1. Through one metric, to second order (Proposition 1). For a change δ\delta in a component's output, the KL effect is 12δ⊤Mδ\tfrac12\delta^\top M\delta plus higher-order terms, with MM (the submission's Σc\Sigma_c) the Fisher metric the rest of the network induces on that output. Zero ablation (δ=−o\delta=-o), interchange (δ=o(x′)−o(x)\delta=o(x')-o(x)) and Gaussian weight noise (δ=Jz\delta=Jz) differ only in the second moment of δ\delta, and the component's Fisher matrix is F=J⊤MJF=J^\top MJ.

QuantityRelation to mediationAgreement with effects
Second-order valuesTaylor term of each interventionrank the full effects at 0.96 or above
Trace Ac=12trFA_c=\tfrac12\mathrm{tr}Fsecond-order value of weight noise0.27 to 0.51 with NIE over heads
Count qQ=12∑iTμi/(1+Tμi)q_Q=\tfrac12\sum_i T\mu_i/(1+T\mu_i)counts directions stiffer than 1/T1/T0.70 to 0.90 with NIE, 0.67 to 0.90 with PIE

In the terms of [2], NIE is our interchange (run on xx with the head's output from x′x'), and PIE the denoising patch (run on x′x' with the output from xx), each read as KL against its own run (Tables 2, 32; four tasks). The trace is dominated by value weights, which hold three quarters of it in RMS coordinates yet barely rank effects; the count saturates there. Among quantities that use neither an intervention nor x′x', the count ranks interchange effects level with or ahead of all. Given zero-ablation KL, layer and output norm, it keeps a partial correlation of 0.41 to 0.68, confirmed in 16 further cells under a rule fixed in advance (Appendix C.2). On the contrast, where [2]'s interaction NIE −- PIE is defined, the count ranks its size at 0.47 to 0.69 (Table 32).

Q2. One run each on Pythia-1B, about 2,500 tokens (Table 28):

QuantityPassesTime
Zero ablation, all componentsone partial forward per component13 s
Interchange, all componentsthe same, plus one forward on x′x'13 s
Second-order values, all componentsone backward per sampled output (4 per ranking, 16 per position)0.5 s each
Count, one head128 Fisher-vector products17 s
Schur fraction, one pairtruncated conjugate gradients32 s, a fifth of the retraining
Estimate on the true loss, one headMetropolis-corrected chain, 16,384 tokens4,600 s

Q3.

For MI:

  • Which weights can take over a lost function: the Schur fraction predicts repair before any retraining (W2).
  • A detector with no replacement value. The fixed-horizon response (the submission's nonlocal SGLD estimate) puts all eight layer-0 heads among its 23 largest of 144 components, while single zero ablation ranks them as low as 53rd. Mean replacement hides their joint role on synthetic strings (contrast drop 0.14, against 1.78 for zeroing) but not on natural bracket contexts (1.69; Table 52).
  • Cheap effects at single positions: 16 sampled-output backward passes rank the interchange KL of single token positions at 0.96 to 0.99, where patching needs one forward pass per patched position.
  • Two functions of one head. In three Pythia-14m runs where induction heads form, the value-output count at T=104.5T=10^{4.5} ranks interchange effects at 0.83 to 0.89, and the query-key count over T=108T=10^8 to 101210^{12} tracks attention entropy at 0.66 to 0.78, both fixed in advance (Appendix D.2).

For SLT, interventions supply known answers: a token lookup whose LLC follows from its mechanism (Proposition 4), and a singular point (WVW_V, bVb_V, WOW_O zeroed) where the estimate follows τr4\tau r^4. They also pick out the count, not the trace, as the summary of the spectrum that tracks mechanism.

Thank you for the detailed and substantive response. If integrated properly in the paper these could be useful improvements that strengthen the paper.

I still think that the proposed connection between SLT and MI is interesting and has potential. However, I am not yet convinced that the paper is mature for publication. The new theoretical and empirical material introduced in the revision would benefit from a systematic presentation and discussion and integration in a paper that was already fairly hard to foloow. In particular, it will be important to clarify the central contributions, their limitations, and the practical value of the proposed geometrical approach.

Overall, I would encourage the authors to consolidate these developments into a more coherent and thoroughly developed paper for a future submission. I will therefore maintain my overall score.

Reviewer_eGEcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares a Dyck bracket-matching circuit in Pythia-1B across 154 training checkpoints with the local geometry of the loss landscape within a prediction-preserving region UU. It studies the Fisher information matrix trace, a Fisher soft rank, a quartic coefficient, and pairwise interaction curvature. The results suggest that the measured local component geometry largely reflects the Fisher spectrum and shared network-wide changes, rather than establishing a mechanism-specific singular invariant.

Critical strengths
  1. The paper demonstrates disciplined numerical hygiene through float64 Hessian–vector-product Lanczos calculations, antithetic radial contrasts, exact quadratic controls, and local MCMC. Together with its candid reporting of negative results, this makes the finite-scale geometric comparison informative even where the proposed LLC interpretation is not established. This methodological contribution is a substantive consideration in favor of acceptance.

  2. The pairwise interaction analysis finds a modest, replicated association with joint-ablation interactions beyond Fisher controls. Its linearized-logit control retains the exact softmax, separating nonlinear model responses from nonlinearity introduced by output normalization. This is consequential because it provides evidence that higher-order geometry carries mechanistic information beyond the Fisher matrix, even though it does not establish specifically singular structure.

Critical weaknesses
  1. At the smaller measured scales, the principal LLC estimates largely reduce to the Fisher trace. The paper has not established that these finite-scale measurements approximate the LLC: a theoretical connection in a limiting regime does not establish that the settings used here reach that regime. This is central to my assessment because the claimed connection between singular learning theory and mechanistic structure is stronger than the demonstrated connection between finite-scale geometry and mechanistic effects.

  2. The SGLD comparisons do not demonstrate adequately calibrated stationary sampling. Appendix K.2 tests a small hyperparameter grid but does not demonstrate tuning that produces healthy, stationary loss chains. In Appendices I and J, mean loss rises between the first and last halves of the retained draws in 90.5% of paired component–checkpoint cells, indicating residual drift. This materially weakens the comparison with prior LLC work: it leaves open whether the reported behavior reflects the underlying posterior geometry or the finite-time sampling protocol, and whether calibrated LLC estimates would yield a different comparison with mechanistic structure.

Questions for authors
  1. What evidence establishes that the reported finite-temperature estimates approximate the LLC at the measured scales, rather than primarily measuring Fisher geometry? Which conclusions depend on interpreting these quantities as LLC estimates rather than finite-scale geometric responses?
  2. Have you jointly tuned the SGLD step size, temperature, and localization strength to obtain stationary loss chains? If so, please provide the diagnostics.
2 comments

(1/2)

Section, table and proposition numbers refer to the revised PDF. Notation: τ\tau is the inverse temperature, rr the prior radius, T=τr2T=\tau r^2, and ±\pm one standard error.

W1. At the TT a sampler can reach they do not, and the revision now establishes what they measure instead (Section 5.1):

  • The estimate is the Fisher count. For trained components at r=0.2r=0.2, T=105T=10^5, it equals within 5% the submission's qQ=12∑iTμi/(1+Tμi)q_Q=\tfrac12\sum_i T\mu_i/(1+T\mu_i), the Fisher spectrum read at resolution 1/T1/T. This is past the trace regime: at T=105T=10^5 the count is a median 11% of TAcTA_c (Appendix B.4).
  • Metropolis-corrected samplers confirm it. HMC rejects states outside UU, and no state of the Fisher-preconditioned MALA chain leaves UU at r≤0.4r\le0.4. On L14H1, four HMC chains from dispersed starts give 492.7±1.0492.7\pm1.0 against a count of 491.1±2.1491.1\pm2.1 (Table 5); on 19 heads fixed in advance, 0.97 to 1.05 of the count (Tables 4, 7). Preconditioning removes the small step that stalled the submission's chains at τ≥105\tau\ge10^5. The submission's first-order quartic correction is valid only at r≤0.025r\le0.025 (Table 21); at r=0.2r=0.2 stiff directions absorb the nonquadratic part (Proposition 3).
  • Where theory predicts a departure, the samplers show it. With WVW_V, bVb_V and WOW_O zeroed, a singular point where the count is zero, the estimate scales as τr4\tau r^4: three radii agree within 6% at equal τr4\tau r^4 while TT varies 16-fold (Table 14). With only WOW_O zeroed, a Morse-Bott point, Proposition 3 predicts the WOW_O-block count averaged over the other weights; four chains give 0.36 to 0.38 of the starting count on L14H1, and the estimate is 1.97 times it on L14H0 (Tables 12, 13).
  • Known LLCs. On deep linear networks, HMC and MALA reproduce exact or blocked-Gibbs values, with RMS z-score at most 1.02 over the 1,255 estimates passing R^≤1.01\hat R\le1.01 and ESS ≥400\ge400 (Appendix B.3). L0.mlp's LLC follows from its mechanism, 11×2047/211\times2047/2 (Proposition 4); its count reaches it only at T=1012T=10^{12} (Table 15).
  • The LLC would not separate heads. Every Pythia-14m head, in 2,424 head-checkpoint cells on random-token and text inputs, has exactly 1,760 Fisher eigenvalues below 10−2010^{-20} of the largest, from a float64 QR factor: the dimension of its symmetry families. So each head is regular modulo symmetries, with the same LLC, 7,360. Two heads that stay regular on Dyck have counts of 213 and 615 at T=104T=10^4 and reach 99% of 7,360 only at T=5.4×1012T=5.4\times10^{12} and 2.2×10152.2\times10^{15} (Appendix D.1).

So the connection the revision claims is between finite-TT Fisher geometry and mechanism. Over the 128 heads of Pythia-1B, the count ranks zero-ablation KL at 0.75 to 0.88 and interchange KL at 0.70 to 0.90 on four tasks, where the trace gives 0.05 to 0.53 and 0.27 to 0.51 (Table 2).

(2/2)

W2.

  • Tuning (Appendix K.2). Step size, inverse temperature and localization were tuned jointly against eight criteria fixed before any chain (Table 16). On L14H1 three settings, each run for 52 relaxation times 2/(ηγ)2/(\eta\gamma), meet all eight (Table 17). The two inside UU agree with the count: 21.79±0.1221.79\pm0.12 against 21.77 at r=0.1r=0.1, T=103T=10^3 (loss R^=1.007\hat R=1.007, ESS 471), and 6.59±0.096.59\pm0.09 against 6.73. K.2 ran the reference implementation's nβn\beta, where a γ\gamma strong enough to stay in UU makes T=nβ/γT=n\beta/\gamma so small that only the prior acts; raising nβn\beta with γ\gamma lets the loss hold the chain.
  • The 90.5% drift (Appendices I, J). That sweep ran 160 updates per component, against a relaxation time of about 6.7×1066.7\times10^6. The revision treats it only as a weight-perturbation response, the fixed-horizon response, and nothing about what the estimate measures rests on it.
  • Prior LLC work. The third healthy setting is the one Urdshals et al. (2026) publish for language models (η=3×10−5\eta=3\times10^{-5}, γ=300\gamma=300, nβ=30n\beta=30). At stationarity it sits 2.3 RMS from the checkpoint, outside UU, and gives 2.8 times its count.
  • Would calibrated estimates change the comparison? No. At a trained component a calibrated sampler returns the count, and Section 5.2 computes the count from the spectrum, with no chain. Over 144 components and 148 checkpoints, the count's changes track zero-ablation changes at 0.65, against the trace's 0.51; given the trace the count keeps 0.51, while the trace given the count keeps 0.04 (Table 26).

Q1. They measure Fisher geometry, the count (W1). No conclusion depends on reading them as LLC estimates: Section 5.2 compares the count itself with causal effects. The LLC enters only as the count's large-TT limit where the Fisher rank is constant nearby (Proposition 2), and through the controls with known LLC.

Q2. Yes (W2). Table 16 lists the eight criteria. Among them: split R^≤1.1\hat R\le1.1 on the loss and ≤1.2\le1.2 on four fixed random projections and on ∥u∥2\Vert u\Vert^2; bulk ESS ≥100\ge100; four chains of at least five relaxation times; and Metropolis acceptance of the SGLD proposal ≥0.9\ge0.9, a diagnostic computed without rejection. The others cover stability, drift, the running mean, signal and sign. Table 17 gives every setting with its R^\hat R, ESS, acceptance, displacement and membership in UU. At the WOW_O-zeroed point, tuned SGLD meets all eight and gives 46.6±0.446.6\pm0.4, inside the spread of four Metropolis-corrected chains (45.7 to 48.2). The account of what the estimate measures rests on those Metropolis-corrected samplers.

Reviewer_ACHWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper draws a connection between singular learning theory (SLT) and mechanistic interpretability (MI). In a controlled setup (Pythia-1B + Dyck), the authors found that weight-space Fisher trace evolution largely reflects shared trends in causal effects, while gain Fisher tracks component-specific ablation effects. The quadratic estimate at large inverse temperature tracks ablation changes mainly for components outside the identified circuit. The SGLD-based local learning coefficient estimator adopted from prior work is more strongly associated with the quartic coefficient than with the Fisher trace in levels across training checkpoints, although its samples move outside the local neighborhood of the checkpoint parameters under the tested settings.

Critical strengths

(1) The paper offers meaningful insight into SLT, formulating some interesting intuitions about the abstract concept of LLC.

(2) Comprehensive experiments are conducted in the paper.

Critical weaknesses

(1) The writing is hard to follow (almost everywhere, but particularly in the Findings section). A lot of terminology is used before it is properly defined, and the experimental results are described in a way that makes it hard to extract the key information (too much listing of numbers that obscures the main message).

(2) The evidence for a concrete connection between SLT and MI is not sufficiently convincing. SLT studies how the output is sensitive to weights, and MI studies how the output is sensitive to activations. It is no surprise that the latter sensitivity can be affected by the former sensitivity via the chain rule. But the connection seems to stop there, and the rest are just moderate correlations that aren't very useful. The experimental results are not compelling enough for an MI researcher to use SLT, or vice versa. Perhaps the authors can answer my first question and address this concern.

(3) I feel that this paper attempts to cover too many different questions, but few questions are covered with substantial depth, which leads to the lack of usefulness of the current results, despite the fact that the questions themselves are promising. For example, the correlation between AcA_c and the ablation effect in Figure 4 A is not ideal (roughly 0.5 at best). We cannot confidently derive much from this result. It would be meaningful to study what circuit mechanisms SLT tends to pick up or miss. Moreover, AcA_c evolves during training (Figure 3B). What kind of learning progress does that reflect? Do increases or decreases in these values reveal the formation of certain mechanisms? Through such investigation, a more concrete and useful bridge can be built between SLT and MI. One other direction that warrants further discussion is whether the geometric insight (about the quadratic/quartic components) potentially leads to a better estimator of LLC.

I think this paper has great potential to be useful and impactful if it is developed properly in a focused direction. But, with the current status, I suggest rejection.

Questions for authors

(1) Can the authors propose a few scenarios in which using SLT can be concretely helpful for MI research, or the other way round? The current discussion section is too vague.

(2) One of the main results, as listed in the abstract and discussion, is that the gain Fisher is reflective of component differences and development, supported by the high correlation in Figure 4A. Nonetheless, isn't this the correlation between a function value and its dominant Taylor expansion term?

(3) The example at the beginning of Line 306 seems to work against P3. P3 states that joint estimates lie below the sum, but the example demonstrates a case where joint effects surpass the sum of individual effects.

(4) Several concepts should be better defined before being discussed:

  • Line 276: log-odds,
  • Proposition 1: the definition of Σ\Sigma is too inexact
  • Line 368: panel pairs
  • Line 477: replacement value.
3 comments

(1/2)

Section, table and proposition numbers refer to the revised PDF; its Propositions 2, 6 and 7 are the submission's Propositions 1, 2 and 3.

W1. Section 5 is rewritten so that each paragraph opens with its claim, with secondary numbers moved to tables. Section 3 now defines every term the findings use before they use it, and Appendix A.1 collects the notation. The appendices are regrouped into eight parts that follow Sections 4 and 5.

W2. The chain rule is Proposition 1: to second order, the KL effect of a change δ\delta in a component's output is 12δ⊤Mδ\tfrac12\delta^\top M\delta, with MM the submission's Σc\Sigma_c. Zero ablation, interchange and weight noise differ only in the δ\delta they apply. For weight noise the second-order value is the Fisher trace AcA_c, and that is why the submission's correlations were moderate: AcA_c measures how much output the weights could move, not how much the component uses.

The revision's main quantity is the Fisher count, the submission's qQ=12∑iTμi/(1+Tμi)q_Q=\tfrac12\sum_i T\mu_i/(1+T\mu_i) over the Fisher eigenvalues μi\mu_i, with T=τr2T=\tau r^2. It counts directions stiffer than 1/T1/T instead of summing curvature, so it is not dominated by the value weights, which hold three quarters of a head's trace in RMS coordinates yet barely rank its effects. It is not a Taylor term of any effect, and its correlations are not moderate. Interchange KL below is the effect of replacing a head's output by its value on the task's counterfactual (Dyck: bracket types permuted; IOI: names replaced; greater-than: start year 01; induction: first copy replaced).

128 heads, final checkpoint, T=105T=10^5 (Table 2)CountTrace AcA_c
Interchange KL, four tasks0.70 to 0.900.27 to 0.51
Zero-ablation KL, four tasks0.75 to 0.880.05 to 0.53

Given the traces of all seven weight and bias blocks and their sums, the count keeps a partial rank correlation of 0.37 to 0.55 with interchange KL (Table 35), so it is not a reweighted trace. Q1 gives concrete uses.

W3. The revision focuses on two: what SLT's estimator measures in Pythia-1B (Section 5.1), and which geometric quantity tracks which causal effect (Section 5.2). Your specific points:

  • Figure 4A, about 0.5. That 0.5 is the trace's (median partial rank correlation of adjacent changes, 144 components, 148 checkpoints). Under the same statistic the count gives 0.65 for zero-ablation and 0.73 for interchange effects, positive for every component; given the trace it keeps 0.51, while the trace given the count keeps 0.04 (Table 26). The submission read the count at T=106T=10^6, where its 32-step quadrature gives upper bounds up to twice too high (Appendix B.4); at T=105T=10^5 it has converged.
  • What SLT picks up or misses. The count's lead over the trace held in direction in all 32 cases fixed in advance (zero-ablation and interchange KL; four further tasks at three Pythia-1B checkpoints and four tasks in Pythia-160m) and cleared the preset bar in 27 (Appendix C.2). Read at two resolutions fixed in advance, it separates the two halves of a head: in three Pythia-14m runs where induction heads form, the value-output count at T=104.5T=10^{4.5} ranks interchange effects at 0.83 to 0.89, and the query-key count over T=108T=10^8 to 101210^{12} tracks attention entropy at 0.66 to 0.78 (Appendix D.2). It misses: no geometric quantity is best for MLPs on every task, the count cannot say which variable a head carries, and it gets the size but not the sign of a head's interaction with other components (Section 5.2).
  • What the evolution of AcA_c reflects. Mostly a trend shared across the model: its changes track the median effect of all components (0.54) about as well as the component's own (0.56), and four fifths of a change comes from weights upstream and downstream (Appendix C.3). The count follows each component's own effect, so its rise marks when that component's effect grows (Appendix C.1).
  • A better LLC estimator. In two senses. At the TT samplers reach (up to 10610^6), the estimate is the count, which then replaces the chain: on 19 heads at r=0.2r=0.2, T=105T=10^5, Metropolis-corrected samplers return 0.97 to 1.05 of it (Appendix B.1). And where a head's only Fisher null directions are its symmetries, one spectrum gives the LLC itself: every Pythia-14m head has exactly 1,760 null directions of 16,480, so its LLC is 7,360 (Appendix D.1).

(2/2)

Q1.

SLT for MI:

  1. Which component can take over a removed one's function. Zero one component's output and retrain only another's weights: over 54 pairs, the Schur fraction, computed from the Fisher matrix before any retraining, ranks the recovered KL at 0.79, against 0.15 for the repairer's own zero-ablation effect (Appendix F.1).
  2. A detector with no replacement value. The fixed-horizon response (the submission's nonlocal SGLD estimate) puts all eight layer-0 heads among its 23 largest components, while single zero ablation ranks them as low as 53rd. Their effect is joint, and mean replacement hides it on synthetic strings (contrast drop 0.14, against 1.78 for zeroing) but not on natural bracket contexts (1.69; Table 52).
  3. Cheap per-position effects. From 16 backward passes, second-order values rank the interchange KL of single token positions at 0.96 to 0.99, where patching needs one forward pass per position (Section 5.2).

MI for SLT:

  1. Known answers for the estimator. L0.mlp is a token lookup whose LLC follows from its mechanism, 11×2047/2=11,258.511\times2047/2=11{,}258.5 (Proposition 4); its count reaches it only at T=1012T=10^{12}. Zeroing a head's WVW_V, bVb_V and WOW_O gives a singular point where the count is zero and the estimate scales as τr4\tau r^4 (Section 5.1).

Q2. Yes. The revision treats the gain Fisher's ranking as a Taylor identity of Proposition 1 and no longer reports it as a finding. The main results rest on the count, which is not a Taylor term of any effect; the Schur fraction in Q1 predicts a separate experiment, retraining.

Q3. Both hold at once (Proposition 5). When components write the same variable, single ablations can be small while the joint ablation is large, so joint effects exceed their sum; for a quadratic loss the joint estimate is at most the sum of single estimates, because shared directions count once. The layer-0 heads show both: zeroing all eight lowers the contrast by 1.78, against 1.39 for the sum of single ablations, and their joint estimate lies below the sum of single estimates (Section 5.4).

Q4.

  • Log-odds: log⁡(py/py0)\log(p_y/p_{y_0}) against a fixed reference token y0y_0.
  • Σ\Sigma (now Proposition 2): block diagonal over the NN scored positions, with blocks 1N(diag(p)−pp⊤)\tfrac1N(\mathrm{diag}(p)-pp^\top), pp the probabilities of all tokens except y0y_0.
  • Panel pairs: the 5,082 component-checkpoint cells of the 33-component panel, now called cells.
  • Replacement value: what replaces a component's output in an intervention: zero, the position mean, another input's output, or a projection.

I thank the authors for their detailed responses. The rebuttal provides some interesting supplementary results. Nevertheless, I remain unconvinced of the paper's contribution.

To me, the paper primarily reports moderate correlations that are not particularly surprising. Moreover, many of the capabilities demonstrated by the proposed method are already available through established mechanistic interpretability methods. Particularly, in one of the responses, the authors highlight the computational efficiency of their method relative to activation patching. However, substantial work has already been devoted to developing computationally efficient alternatives, including EAP, EAP-IG, EAP-GP, CEAP, ...

More fundamentally, I had hoped to see a deeper and more natural connection between mechanistic interpretability and singular learning theory. Unfortunately, I do not think the current results establish such a connection convincingly. The implications for mechanistic understanding remain unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.