acceptodds
Under review as a conference paper at ICLR 2027

Predictive Mechanistic Interpretability of Post-Training: A Priori Forecasts of Circuit Localization and Sign from the Base Model

Abstract

Which attention heads a post-training update will change, and in which direction, can be forecast before training, from the base model alone. In the settings we study, only how much they change requires the training run. Mechanistic accounts of post-training have so far been retrospective: circuits are read off the model after fine-tuning, predictions are scalar quantities in output space, or reachable outputs are characterized without the direction of change. We instead seal a per-head, signed forecast from the base model, fine-tune, and confirm it. **Which** heads change, and the **sign** of each change, follow under measurable base-tangent conditions and an empirically tested pathwise sign-stability condition, without requiring the update to remain small. **How much** each head changes rests on a linearization, whose validity we quantify with a creation index (the deviation of the realized update from the base-tangent prediction). These magnitude forecasts are noisier. We pre-register the which-and-sign predictions on GPT-2 small (IOI, SFT) and confirm them (localization rank correlation 0.65, per-head sign agreement 1.00), with a held-out duality check at 0.88 and causal head ablations. The predictions generalize across ten models (124M–1.5B), three circuits, and three objectives (SFT, DPO, RLVR), against composition-matched chance. The sign forecast also outlives its motivating linearization, which we call the structural–metric dissociation: it survives a DPO update with creation index 0.87 and flips as predicted (0.90 per seed) under a sealed reversal fine-tune, while magnitude calibration degrades as the creation index grows. Recent output-space laws of post-training admit the same base-model reading.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, post-training, neural tangent kernel, circuit analysis, causal interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ne1hOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to predict the circuit graph of a model after post-training using only the base model. Its key assumption is that post-training doesn't really give the model brand-new abilities, but rather just activates existing ones. Because of this, parameter changes are assumed to be roughly in the same direction and linear, so they can be estimated directly by taking the inner product between the current model's local tangent vector ∇θah(θ0)\nabla_\theta a_h(\theta_0) and the task gradient direction J0⊤δJ_0^\top \delta.

Critical strengths
  1. The writing is clear.

  2. The paper has some interesting findings, such as all key heads in a circuit sharing the same scalar pattern in their tangent gradients, and the effect of parameter changes from post-training being almost linear.

  3. The proposed method is proactive/predictive to some degree, which is important for the circuit discovery field.

Critical weaknesses
  1. The tasks used (like IOI, Greater-Than, etc.) are very simple toy tasks, and the base model is already good at them. They don't really reflect what post-training actually changes. Although the authors tested a modified IOI task to show this, its output pattern is still fixed, and it's not clear whether a mechanistic circuit even needs to form for it. So it's still doubtful whether this method actually works in real-world post-training.

  2. The authors don't directly compare their method with circuit discovery methods run on the post-trained model. While they claim that such a comparison would be unfair, the current evaluation still feels biased and unconvincing. First, EAP already has the largest attribution score error among all circuit discovery methods and also relies on a first-order Taylor expansion, so methods like ACDC and Edge-Pruning should be included. Second, for circuit faithfulness, the authors only check how much performance drops after ablation instead of how much performance the circuit itself can retain, which is not convincing enough.

  3. The design of the creation index C\mathcal{C} is quite controversial. J0(x)ΔθJ_0(x)\Delta\theta itself is already a rough first-order approximation with noticeable error, and there isn't solid proof showing a clear relationship between non-linearity and training modes.

Questions for authors
  1. Can the authors show performance on more realistic post-training tasks, such as coding, math, instruction following, or CoT?

  2. Can the authors provide comparisons with circuits found by other methods on the post-trained model, like ACDC and Edge-Pruning, and evaluate them using standard faithfulness and completeness metrics?

  3. Can the authors show how the prediction error changes dynamically as the number of parameter update steps increases?

  4. Existing studies show that during post-training, circuit stability drops significantly because the underlying mechanisms are reframed to some extent (e.g., [1, 2]). How do the authors explain the tension between this observation and Proposition 2 proposed in the paper?

[1] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training

[2] Circuit Stability Characterizes Language Model Generalization

2 comments

We thank the reviewer for the constructive review, and for noting that the shared scalar pattern of the circuit heads' tangents and the predictive angle are of interest. Below, we provide a detailed response addressing each of the weaknesses [W#] and questions [Q#] raised: this comment covers the requested comparison with circuit discovery on the post-trained model [W2/Q2]; a follow-up comment covers realistic tasks [W1/Q1], the creation index [W3], prediction error versus update steps [Q3], and circuit stability [Q4].


[W2/Q2] We agree that this comparison was missing, and the revision adds it (new, Section 4.3 and App. G.4). On the fine-tuned GPT-2 IOI model we ran ACDC-inspired greedy head-level pruning (KL criterion, discovered on separate prompts, scored on held-out prompts, with the per-template ABC mean ablation of Wang et al.) and score circuits with the standard metrics. Faithfulness F(C)/F(M)F(C)/F(M) is what the circuit retains on its own, and incompleteness is ∣F(C∖K)−F(M∖K)∣/∣F(M)∣|F(C\setminus K)-F(M\setminus K)|/|F(M)| over random KK (lower is better).

circuitsize (base / FT)faithfulness (base / FT)incompleteness (base / FT)
Wang et al., 26 heads26 / 260.92 / 0.860.17 / 0.18
pruned, τ=0.003\tau=0.00343 / 20 (Jaccard 0.42)0.93 / 1.04- / 0.13
pruned, τ=0.01\tau=0.0126 / 16 (Jaccard 0.41)0.74 / 0.83- / 0.37

To locate the change itself, we patch the outputs of a 10-head set (fraction of the gain; sufficiency = fine-tuned outputs into the base model, necessity = base outputs into the fine-tuned model; 5 seeds):

10-head setchosensufficiencynecessity
forecast (base DLA)before training0.62 (0.60-0.63)0.52
post-hoc top-∥Δ\\\|\DeltaDLA$\|$ headsafter training0.64 (0.62-0.66)0.46
pruning ranking on FT modelafter training0.580.64
random heads-0.100.12

Per seed, the forecast is 0.02-0.04 below the post-hoc comparator at 10 heads, so we do not claim equivalence. With 20 heads: forecast 0.86 / 0.90, post-hoc 0.83 / 0.81, random 0.21 / 0.24. These are patching sufficiency and necessity scores, not a decomposition into parameter-only contributions, since head outputs also carry upstream changes.

On signs, direct-attribution signs are correct for all ten forecast heads. Individual patching effects agree on 9/10 in every seed; head 9.6 is the exception, consistent with interactions beyond the direct readout. Among the five heads with the largest base |DLA|, patching the three Name Movers recovers +0.27 of the gain and the two Negative Name Movers -0.39: the suppressors deepen, causally.

Where the methods disagree: at τ=0.003\tau=0.003 the pruned fine-tuned circuit keeps 7 of the 10 forecast heads, including all three Name Movers, but drops 10.7, 11.2 and 10.6. The fine-tuned model is near saturation, so removing a head that lowers the logit difference changes the output distribution little, which plausibly explains why a KL criterion discards it. Pruning measures what the output needs, patching where the change is, and Δ\DeltaDLA the change of a head's direct contribution.

Caveats (App. G.4): the pruning is node-level and single-pass, circuits are whole heads at all positions (so the numbers are not comparable to the 87% of Wang et al.), and MLPs are never ablated. We did not run edge-level discovery such as Edge Pruning, and the paper states this as a limitation. On EAP: our forecast is not a discovery method run on the fine-tuned model, and it is not EAP. It is the base model's DLA, computed before training, and attribution patching appears only as a cross-check.

We would appreciate reviewer ne1h's view on whether this comparison addresses the concern about the evaluation.

Additional details on realistic tasks, the creation index (now the tangent-residual ratio), update steps, and circuit stability.


[W1/Q1] We agree that this is the main limit of the paper's scope, and the revised limitations say so first. The method needs a per-component readout of the target behavior. The DLA provides one for tasks with an identifiable answer token, whereas open-ended generation (reasoning, coding, instruction following) would need another readout, such as a probe direction, and a validated circuit. Within that constraint, the lookup control has no imported head list and its update is far from linear (Rtan=0.87\mathcal R_{\mathrm{tan}}=0.87), yet the forecast heads carry the change (overlap 6.4/10) with the forecast signs (1.00, 10 seeds). The sign forecast also holds on bases up to 7B and on an instruction-tuned base (0.77-1.00); the clearest failure is a 7B-Instruct run whose fine-tune barely moves the task (gain +1.4, sign agreement 0.40). On the modified task: a fixed output pattern indeed does not show that a new circuit forms, and our claim does not need one to form. It predicts where the change lands among existing components. We did not attempt a reasoning or coding fine-tune in the discussion period, because doing it properly needs the readout and circuit validation described above.


[W3] We thank the reviewer for pointing out that the original name and interpretation overstated what this quantity measures. We have renamed the creation index C\mathcal C to the tangent-residual ratio (TRR, Rtan\mathcal R_{\mathrm{tan}}) throughout the revised manuscript (Section 3.4, Figure 5, tables and appendices). The new name follows its definition: the residual energy relative to the combined residual and base-tangent energies. It is not an estimator of capability creation, and we agree that nothing links it to training modes. The JVP computes the tangent prediction exactly, but that does not make the first-order approximation exact. Adding the quadratic Taylor term reduces the ratio for DPO from 0.87 to 0.21 (new, App. G.2), so much of its first-order residual is curvature. We therefore present it as a descriptive diagnostic of departure from the base tangent.


[Q3] Plain gradient descent, extended to five times the standard number of steps (new, App. G.3; 5 seeds):

step1510204080120160200
gain1.62.73.34.04.85.56.06.36.5
sign1.001.001.001.001.001.001.001.001.00

Overlap with the preselected top ten stays 8.2/10 at every step, and the rank correlation of change magnitudes declines slightly (0.68 -> 0.63) as Rtan\mathcal R_{\mathrm{tan}} rises (0.12 -> 0.18). A constant overlap does not by itself show that every checkpoint has an identical set of changing heads. With AdamW and LoRA (500 steps, 8 seeds per rate, Table 12), agreement is 1.00 for every observed gain up to +13.5, an observed range rather than a general threshold. Beyond it, agreement falls (AdamW 0.91 at +17.8 and 0.79 at +26.5, LoRA 0.86 at +20.2), and almost all failing heads are minority-sign heads (on GPT-2 IOI, the negative heads 10.7, 11.10 and 11.2). In a step-resolved AdamW run the break happens between step 10 (gain +8.8, sign 1.00) and step 25 (+21.0, 0.82). We conjecture that these runs violate the path conditions of the sign criterion (Eq. 3), but we did not measure κh\kappa_h for them, and the base model does not tell us where that boundary lies.


[Q4] [2] (Sun, ACL 2025) defines circuit stability as a model applying a consistent circuit across inputs; it does not study change under post-training, so we see no direct tension. [1] (Chen et al., 2026) finds that circuits evolve during SFT, so a circuit located on pre-update weights lags behind the model being updated. We agree, and our pruning finds base and fine-tuned circuits that differ substantially (Jaccard 0.41-0.42, Table 13). What we forecast is not circuit membership but the change itself: which heads it concentrates on and in which direction. Static circuit membership changes after tuning, whereas the base-DLA ranking retains predictive value for the location and sign of head-level attribution changes in our controlled tasks.

The initial sign alignment (old Prop. 2, now Eq. 2) is a statement at the base model. The sign criterion (Eq. 3) does not need the circuit to stay fixed; it needs the heads' rates to share one direction along training. The measured rate trajectories have high leading-mode energy (0.89-0.999), but only an endpoint margin proxy passes on 50 of 60 pairs (App. D), and the stronger integrated bound of Eq. 3 is not directly tested. This is compatible with large changes in which heads a pruning method keeps. The revision adds both papers: [1] in Section 4.3, next to our circuit-drift result, and both in the related work.

Reviewer_E3XXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper considers post-training updates on transformer architectures, specifically Supervised Fine-Tuning (SFT), Direct Preference Optimisation (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR) applied to three tasks: IOI, Greater Than and Successor. Using the Direct Logit Attribution (DLO) model, the paper aims to show that it is possible to forecast which attention head will change, and what the sign of this (DLA) change will be by using a first-order Taylor expansion of the model (Lemma 1 and Proposition 2). The author(s) also consider the question of the magnitude of the change. The authors present some theoretical results in sec 3 and some empirical results in sec 4 to validate their central claim.

Critical strengths

The simple linear model of Assumption 1 has the potential to provide some clear mathematical insight into the question. The list of experiments seems reasonable, although it would be nice to see how exactly SFT, DPO and RLVR exactly connect to the gradient J_0 for their updates, and thus if they really do something different from that standpoint.

Critical weaknesses
  1. The paper is very poorly written.

1.a) The vocabulary is really contrived - e.g. the word "seal" seems designed to obfuscate the simple concept of a forecast - leading to extraordinary sentences like "The predictions licensed by the linearization A1 fall into two tiers of different robustness, a structural–metric dissociation: the structural tier largely survives high-C updates, while the metric tier degrades." In other words, the first-order Taylor expansion makes predictions that work, and others that don't...a completely vacuous sentence.

1.b) Lots of notation is left unexplained (for example line 140 is almost unreadable) or explained in the wrong place (for example the notion of creation index is defined on line 201 but used several times from line 79). This is a problem even in the Appendix where space is not limited and proofs or extra material could be presented with more consideration for the reader in mind.

  1. The theoretical part (section 3) is problematic.

2.a) The presentation of 3.2 is really poor. The conditions C0, C1 and C2 are not stated properly, they introduced in a completely haphazard way in the middle of statements.

2.b) Proposition 1 is not a proposition and its proof is not a proof. The proof of Proposition 2 in the appendix is completely unreadable. I don't see what statement Proposition 3 is making, it looks more like two definitions (W^* and Cost). Corollary 1, theorem 1, Corollary 2, Proposition 4, and Proposition 5 do not have proofs in the Appendix. On the other hand, there are proofs in the appendix that do not correspond to any statements in the paper. Prop 6 and lemma 2 have very long statements and one-line proofs. This is evidence of bad maths, and bad presentation.

2.c) Sec 3 needs to be completely reworked. What exactly are you trying to prove, and under what assumptions?

  1. It is not clear that the question is important or that the answer is surprising.

3.a) Why is the task of forecasting these post-training stages important? Why is this research question justified in the first place?

3.b) Is it surprising that the gradient should predict the sign of a small change. On the face of it, not really. It's all in the Taylor expansion. What makes the forecasts made in this paper surprising?

3.c) Instead of using the creation index (l201), it would be more in the spirit of assumption 1 to perhaps look at the second order term of the Taylor expansion. This is, after all, what controls how quickly the linear regime stops being a good approximation. It might have more precise things to say about predicting the magnitudes. But again, this won't be terribly surprising as this is what Taylor expansions do.

Questions for authors

Do SFT, DPO and RLVR lead to significantly different usage of J_0 to perform their updates, or are they all doing variants of the same thing?

Question 3.a) and 3.b) above.

2 comments

We thank the reviewer for the detailed and rigorous reading. We especially appreciate the question on how SFT, DPO and RLVR use J0J_0, which led to a new measurement. Below, we provide a detailed response addressing each of the weaknesses [W#] and questions [Q#] raised, numbered in the order of the review ([W1]-[W8] = 1.a-3.c; [Q1] = the J0J_0 question; [Q2] = 3.a/3.b): this comment covers the writing and the theory [W1-W5]; a follow-up comment covers importance and surprise [W6, W7, Q2], the second-order term [W8], and J0J_0 [Q1].


[W1] (1.a) We have revised the organization, notation and presentation of the theory. "Seal" is no longer used, and the quoted sentence is replaced by plain statements (Section 4.2 opening): which heads change and their signs often survive substantial departure from the first-order approximation, while magnitude forecasts degrade.


[W2] (1.b) The definition of aha_h at the old line 140 is now explained in words next to the formula (opening of Section 3.2), and the creation index, now renamed the relative linearization error (Rtan\mathcal R_{\mathrm{tan}}, formerly C\mathcal C), is mentioned in the Introduction and defined in Section 3.4.


[W3]-[W5] (2.a-2.c) We have recast the original Proposition 1 as Eq. (1), identifying it as the path-integral identity given by the fundamental theorem of calculus. We have also streamlined Section 3 to separate this elementary identity from the additional conditions needed to predict the sign of a head's accumulated change. More broadly, we have removed the theorem, proposition, lemma and corollary labels throughout and present the results as equations with their derivations.

What we prove, and under which assumptions: Section 3.2 has three steps.

  1. The exact identity Δah=∫0Tκh(t),dt\Delta a_h=\int_0^T\kappa_h(t)\\,dt (Eq. 1), which by itself does not fix the sign.
  2. The initial sign alignment at the base model, κh(0)=sAgh+rh\kappa_h(0)=sAg_h+r_h with A>0A > 0 (Eq. 2), under two conditions that can be checked before training (initial error direction, collinear head gradients).
  3. One conditional sign criterion, sign(Δah)=sign(G),sign(DLAbase,h)\mathrm{sign}(\Delta a_h)=\mathrm{sign}(G)\\,\mathrm{sign}(\mathrm{DLA}_{\mathrm{base},h}) (Eq. 3), under path conditions stated explicitly: κh(t)=ghc(t)+rh(t)\kappa_h(t)=g_hc(t)+r_h(t) with ∫0T∣rh∣,dt<∣gh∣∣I∣\int_0^T|r_h|\\,dt < |g_h||I| and I=∫0Tc,dtI=\int_0^Tc\\,dt, circuit dominance ∑hgh>0\sum_hg_h > 0 and ∑h∫0T∣rh∣,dt+∣Δa⊥∣<(∑hgh)∣I∣\sum_h\int_0^T|r_h|\\,dt+|\Delta a_\perp| < (\sum_hg_h)|I|, and sign(gh)=sign(DLAbase,h)\mathrm{sign}(g_h)=\mathrm{sign}(\mathrm{DLA}_{\mathrm{base},h}). Pointwise sign stability is not assumed.

The derivation of Eq. 3 is short: the per-head bound gives sign(Δah)=sign(ghI)\mathrm{sign}(\Delta a_h)=\mathrm{sign}(g_hI), and summing over heads, G=(∑hgh)I+∑h∫0Trh,dt+Δa⊥G=(\sum_hg_h)I+\sum_h\int_0^Tr_h\\,dt+\Delta a_\perp, so circuit dominance gives sign(I)=sign(G)\mathrm{sign}(I)=\mathrm{sign}(G). Appendix D gives the details, including the difference between aha_h at the base LayerNorm scale and the measured DLA. Old numbering: Prop. 1 -> Eq. 1; Prop. 2 -> Eq. 2; Cor. 1, Prop. 6 and Lemma 2 -> Eq. 3.

On proofs. Appendices D and F give the derivations (old Lemma 1, Props. 1, 2 and 6, Lemma 2 and Thm. 2), each stated with the assumptions under which it holds:

  • complete derivations for the sign criterion (Eq. 3) in Appendix D, and in Appendix F for the minimum-norm cost, the effect on old inputs and the single-probe sign (Eqs. 4-6; old Prop. 3, Thm. 1 and Cor. 2), and the nonlinear counterexample (old Prop. 5, now constructive);
  • the scope of old Prop. 3 (Appendix F): the pseudo-inverse cost applies to targets in the range of the eNTK, and a target with a component in its kernel has cost +∞+\infty;
  • for the two exactly solvable examples (old Thm. 2), complete, self-contained derivations are in Appendix D, stating all assumptions, the exact dynamics and head-level sign relations, and the boundary cases, without theorem labels.

A limitation we now state: only the initial error direction and collinear head gradients can be checked at the base checkpoint. The path conditions require information from the realized trajectory, and Appendix D reports trajectory measurements and an endpoint proxy for the per-head bound (50/60) rather than a complete verification of those sufficient conditions. They are not a priori preconditions.

We would appreciate reviewer E3XX's view on whether the restructured Section 3 now states clearly what is proved and under which assumptions.

Additional details on importance, surprise, the second-order term, and J0J_0.


[W6/Q2] (3.a) Why forecast post-training? Most existing circuit-level studies of post-training are retrospective. A forecast names the few heads to audit before any compute is spent, says what to expect (e.g., suppressor heads coming out stronger rather than removed), and tests understanding, since a forecast that can fail is a stronger test than a post-hoc description.


[W7/Q2] (3.b) Is it surprising? For a small step, no: the sign is in the first-order Taylor expansion, and the revised Section 3.2 states it as a first-order relation rather than a lemma. The expansion does not give the following.

  • Which heads. The parameter-gradient localizer is also first-order and fails (Name Movers at zero-indexed ranks 77-117 of 144). The measured alignment of the circuit heads' tangents is what makes the DLA the predictor.
  • Large updates. For DPO the first-order relative error on the margin change is 0.74 (new, App. G.2), and κh(t)\kappa_h(t) changes sign along the path on all 12 heads, yet the final signs match on 11/12 (original, App. D). An expansion at θ0\theta_0 cannot fix the integrated sign once the rates change. The criterion (Eq. 3) adds a shared-rate decomposition and an integrated residual bound, and Appendix D reports partial diagnostics of them (an endpoint margin proxy holds on 50 of 60 head-run pairs).
  • The suppressor direction, and the observed 11.2 exception to the primary reversal rule, with which a separate initial-rate calculation on the base model is consistent in sign (see our reply to sCzo).

[W8] (3.c) We computed the second-order term QQ along the realized update by nested forward-mode JVPs (new, App. G.2, Table 10; 3 seeds). Rtan(2)\mathcal R^{(2)}_{\mathrm{tan}} (formerly C≤2\mathcal C_{\le2}) is the ratio after adding QQ, and err1, err2 are the first- and second-order relative errors.

configurationRtan\mathcal R_{\mathrm{tan}}Rtan(2)\mathcal R^{(2)}_{\mathrm{tan}}err1err2∥Q∥/∥R∥\\\|Q\\\|/\\\|R\\\|
SFT, lr 1e-40.140.110.330.431.6
SFT, lr 3e-4 (standard)0.140.150.350.562.0
RLVR (standard)0.160.180.360.702.3
SFT, lr 1e-30.150.370.471.914.3
DPO (standard)0.870.210.740.380.7
SFT, lr 3e-30.520.504.712427

You are right that it carries real information: for DPO it removes most of the residual (0.87 -> 0.21) and halves the magnitude error, so the high Rtan\mathcal R_{\mathrm{tan}} of DPO mostly reflects curvature. We now report it next to Rtan\mathcal R_{\mathrm{tan}}, and the renamed ratio is no longer read as capability creation. It does not replace Rtan\mathcal R_{\mathrm{tan}}: on low-Rtan\mathcal R_{\mathrm{tan}} runs QQ overshoots, so the second-order prediction is worse than the first-order one, and at lr 3e-3 it is far off. Neither quantity affects the sign result.


[Q1] Do SFT, DPO and RLVR use J0J_0 differently? At the base model each step is J0⊤δJ_0^\top\delta, and only the output-space error differs: δ=1IO−p\delta=1_{IO}-p (SFT), δ=β2(1IO−1S)\delta=\tfrac{\beta}{2}(1_{IO}-1_S) (DPO, base reference), and δ=pIO(1IO−p)\delta=p_{IO}(1_{IO}-p) (RLVR, expected-reward gradient). All three project positively on the IO-S readout, which is the initial-error-direction condition; our sampled REINFORCE update satisfies it only in expectation (Section 3.2, App. G.1). New measurement (App. G.1, Table 9; 5 seeds):

DPO-SFTDPO-RLVRRLVR-SFT
cosine of update at θ0\theta_00.450.470.99
cosine of realized total update0.620.660.98
correlation of per-head κh(0)\kappa_h(0)0.990.991.00
sign agreement of κh(0)\kappa_h(0)12/1212/1212/12

They use J0J_0 differently in parameter space, DPO especially, and end at different distances from the linear regime (Rtan\mathcal R_{\mathrm{tan}}: SFT 0.14, RLVR 0.17, DPO 0.87). Yet all three project onto the per-head readouts the same way, consistent with the circuit heads' readout gradients sharing one prominent direction. The per-head sign agreement was already in App. D of the original; the cosines and correlations are new.


Scope. The reported DLA uses each checkpoint's LayerNorm scale, whereas aha_h holds the base scale, and a positive scaling at each endpoint does not guarantee that the sign of their difference is preserved. We therefore checked agreement directly (SFT 12/12, DPO 11/12) and state the limitation. The criterion also uses the realized gain, while the forecast uses the intended direction ss. In the fine SGD sweep of Fig. 3b, sign agreement collapses where the realized gain turns negative, but runs at lr 10−210^{-2} also fail at positive gain; we did not verify the path conditions there and do not claim that the criterion guarantees their behavior.

Reviewer_sCzoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates if it is possible to inspect the base LLM before fine-tuning to forecast which attention heads will change and whether their contributions will increase or decrease as a result of the fine-tuning. The authors motivate this approach through the lazy or neural tangent kernel (NTK) regime. The authors find that task-improving training consistently amplifies the heads that initially favored the correct answer while driving suppressor heads to push back even more. Although calculating the exact magnitude of the changes becomes more challenging as the weight updates grow larger, the identity and direction of the specific components that change is still accurate across different training methods (SFT, DPO, RLVR).

Critical strengths

The problem statement is genuinely interesting; estimating which attention heads a post-training update will change, before that training occurs would be a useful contribution. The authors test their hypotheses on 10 different models of varying sizes as well as three different training adjectives. This thoroughness is appreciated.

Critical weaknesses

This paper has severe readability issues. It is overall hard to parse exactly what is studied and what the conclusions are.

W1. The writing overall could be much more polished. The Abstract in particular is very challenging to understand, with dense and jargon-heavy language introduced without explanation. It is not easy to understand what the authors are exactly trying to test, and what their conclusions are. Many terms are introduced and not explained or motivated, such as “NTK” (neural tangent kernel). The mathematical notation is much too verbose; it is unneeded to produce “lemmas” and “propositions” for what are essentially emprical results. Several lemmas and propositions formalize straightforward conditional statements, such as Proposition 1 which states that if a head’s attribution derivative keeps the same sign during training, its total change has that sign. This follows directly from integrating the derivative, but the real question that the paper needs to answer empirically is if this sign stability actually holds up in practice.

W2. Repeatedly emphasizing the “sealed” nature of the study is completely unnecessary…the authors seem to be borrowing terms from pre-registration of clinical trial results, but this is not at all needed for this type of study. I would hope that the authors did not bias their results by changing their initial hypotheses of circuit function in the base models. Since the method in the paper relies on automated metrics and ablations, such elaborate pre-registration implies additional rigour that is not actually present.

W3. The usefulness of the NTK framework here is unclear. The raw parameter-gradient localization score fails, whereas successful localization uses the base-model DLA instead. Although the authors attempt to connect DLA to this tangent framework, they do not clearly establish what the framework adds. Simply, the author’s findings are that heads with larger base-model DLA are predicted to change more during fine-tuning, and their changes are predicted to follow the base-model DLA sign for the updates that improve task performance. Therefore it is unclear what the elaborate NTK framework adds to these otherwise straightforward empirical results.

Questions for authors

Q1. What does the mathematical framework in Sections 2 & 3 allow you to predict that could not be obtained from simply measuring each head’s contribution before training?

Q2. What do the authors view to be the primary contribution of their work? What should we take away from these experiments?

4 comments

We thank the reviewer for the careful and candid review. We especially appreciate the recognition that forecasting which heads a post-training update will change, before training, is a useful problem. Below, we provide a detailed response addressing each of the weaknesses [W#] and questions [Q#] raised: this comment covers what the framework predicts beyond measuring the DLA [W3/Q1] and the presentation concerns [W1, W2]; a follow-up comment shows whether sign stability holds in practice [W1] and answers [Q2].


[W3/Q1] We agree that the submission did not make this clear. Measuring each head's DLA before training, together with the intuition that training builds on what is there, leaves two natural readings, and each gets part of the answer wrong. "Training removes what opposes the task" predicts that the Negative Name Movers weaken. "Training amplifies the existing circuit" predicts that every head keeps growing in its role even when the update degrades the task. The signed rule gives concrete, falsifiable tests that separate it from both readings, and we report its exception rather than counting it as a success:

Test (forecast fixed before training)OutcomeWhere
improving update: suppressor heads deepen16/16 head-seed pairs (10.7, 11.10; 8 seeds)original, App. B
reversed objective: forecast heads reverse9/10 in every seed; in seed 0 (per-head data) the exception is head 11.2 (amplify rule: 0.10)original, Sec. 4.1
SFT, DPO, RLVR: same per-head signs despite different updatesinitial rates κh(0)\kappa_h(0) agree in sign on 12/12 circuit heads (256-prompt training gradient; DPO 11/12 on a 64-prompt batch); realized amplify-sign 1.00 for all three on the five canonical heads; on the ten top-ranked heads, DPO is 0.90 (seed 0, head 11.2) (DPO-SFT update cosine 0.45)App. D; new, App. G.1

Head 11.2 is a failure of the primary DLA-sign rule under the reversal, which predicts that this negative-DLA head flips. A separate objective-specific calculation on the base model, the initial rate κh(0)\kappa_h(0) under the reversal gradient, does not reverse for 11.2 (App. D), because its tangent is dominated by an objective-independent softmax term that κh(0)\kappa_h(0) sees and the DLA does not. We treat this as an auxiliary diagnostic, not as a success of the pre-registered DLA-sign forecast. In addition (new, App. G.5), in a CPU re-run of the ten-model sweep, 22 of 23 distinct minority-sign heads (base DLA opposite to the majority of the top-10) move in their own forecast direction in every seed in which they occur (60/63 head-seed pairs).

The prospective rule predicts s⋅sign(DLAbase)s\cdot\mathrm{sign}(\mathrm{DLA}_{\mathrm{base}}) with the intended direction ss fixed before training. The conditional criterion (Eq. 3) relates the final sign to the realized gain GG, and the two agree when training moves in the intended direction. On which base quantity to read: the per-head parameter-gradient norm puts the Name Movers at zero-indexed ranks 77, 85 and 117 of 144. The path identity (Eq. 1) expresses the accumulated change in readout space, but its predictive use does not follow from the identity alone. The measured alignment of the circuit heads' tangents and the localization experiments provide that evidence.


[W1] We agree. The revised PDF rewrites the abstract and introduction in plain language: the abstract now opens with the claim (base-model measurements forecast which heads change during post-training, and in which direction), and NTK, eNTK and the tangent-residual ratio (TRR, formerly the creation index) are defined at first use. We also agree that the formal apparatus was inflated. Proposition 1 was the path-integral identity, and the revision states it as Eq. (1). More broadly, we removed the lemma, proposition, theorem and corollary labels throughout. Section 3.2 now gives the exact identity (Eq. 1), the initial sign alignment that can be checked on the base model (Eq. 2), and one conditional sign criterion whose assumptions are stated as inequalities (Eq. 3), with derivations in Appendix D. On the reviewer's central question, whether sign stability holds in practice, the measurement was already in Appendix D of the original submission. The follow-up comment summarizes it.


[W2] Agreed. "Sealed" is removed, and Section 4.1 says once that the forecasts were fixed and pre-registered before fine-tuning (by file times). The registered plan, what was run, and the deviations between them stay in Appendix E for readers who want to judge them, rather than serving as a rhetorical device in the main text.

We would appreciate reviewer sCzo's view on whether these tests and the revised presentation address the informativeness and readability concerns.

Additional details on sign stability in practice and the takeaway.


[W1] Does sign stability hold in practice? The requested analysis was already in Appendix D of the original submission. We measured the head-level attribution rates κh(t)\kappa_h(t) along the training trajectories of the 12 IOI circuit heads:

RunSign constancy of κh\kappa_h at sampled checkpoints (every 5 steps)Final sign matches forecast
SFT12/12 heads at every checkpoint12/12
Reversal (s=−1s=-1)12/12 keep one sign; 11/12 negated11/12 (11.2: exception to the primary rule)
DPO (Rtan=0.86\mathcal R_{\mathrm{tan}}=0.86)0/1211/12

Sign constancy at the sampled checkpoints therefore holds in the low-Rtan\mathcal R_{\mathrm{tan}} runs (a finite grid cannot rule out brief sign changes in between) and fails under DPO, yet the final signs hold. This is why the conditional criterion (Eq. 3) uses the integrated rate with a residual margin rather than assuming that every instantaneous rate keeps its sign. We agree that this evidence was not visible enough. The revision moves the summary into Section 3.2 ("Measured conditions"), together with what it does not show: an endpoint residual-margin diagnostic holds on 50 of 60 head-run pairs, but the stronger integrated condition of Eq. 3 has not been verified directly, so the criterion is a conditional explanation, not a certificate.

We also state where the forecast degrades (new, App. G.3). With AdamW or LoRA pushed to 3.7-5.5x the SFT gain, sign agreement falls to 0.79-0.91, and almost all failing heads are minority-sign heads (on GPT-2 IOI, the negative heads). We did not measure the pathwise decomposition in these runs, so we cannot say which condition fails, and the base model does not tell us where this regime begins.


[Q2] The contribution is a forecast, made from the base model alone before any training, of which attention heads a post-training update changes and in which direction. It is tested across ten models, three circuits and three objectives in partly overlapping experiments, together with the conditions under which it holds and the regimes where it fails.

The takeaway: when post-training improves a behavior the base model already partly has, the change concentrates on the heads that already write the answer. Each moves in the direction its base contribution points, including the heads that oppose the answer, which come out stronger rather than removed. Before training, one can name the few heads to audit (out of 144 in GPT-2 small) and expect the suppressors to deepen. Patching confirms that these heads matter (new, Section 4.3): the ten heads chosen before training recover 0.62 of the gain, compared with 0.64 for the ten heads chosen after training by their realized change.

We hope the clearer statement of the forecasting claim, the additional causal comparisons, and the explicit limits address the concerns behind the overall assessment.

The revised Abstract and Introduction are still incredibly hard to parse.

As well, the revised paper is altered so substantially that it as almost a different paper compared to the original submission. The authors extended the paper from 26 to 42 pages, and added several completely new and opaque figures (within 36 hours of reviews being released).

In my view, it is still not clear what the paper is trying to show. These statements in the rebuttal are not at all clarifying: "when post-training improves a behavior the base model already partly has, the change concentrates on the heads that already write the answer". What is the importance of this? In addition, when you say "Before training, one can name the few heads to audit (out of 144 in GPT-2 small) and expect the suppressors to deepen", what does it mean for "the suppressors to deepen"?

I retain my original score.

Thank you for your follow-up. We recognize that the paper was difficult to follow, and we are sorry that we did not communicate our main idea more clearly. In the final revision, we have rewritten the Abstract and Introduction in plain language. Our central question is whether measurements of the base model alone can tell us, before fine-tuning, which attention heads will change their contributions and in which direction. We sincerely appreciate the time and effort you have put into reviewing our work. Thank you.

Comments on the paper

1 comment

Dear all,

We would like to thank the reviewers for their careful reading and constructive suggestions. We summarize the reviews from our own perspective and outline how the revised PDF addresses them. We also acknowledge that the writing of the paper and of our earlier responses fell short in clarity. In trying to answer every comment as fully as we could, our responses and the revision grew long, and that process showed us how much the original submission had left unclear. We will take the reviewers' feedback on this to heart.

In the final revision, the Abstract and Introduction are rewritten in plainer terms, shorthand elsewhere is replaced by explicit statements, and the appendix states definitions and experimental details more fully. No experiments are added to the main text, and the figures are unchanged.

Strengths.

We are glad that the reviewers found the problem worthwhile. Reviewer sCzo wrote that "the problem statement is genuinely interesting; estimating which attention heads a post-training update will change, before that training occurs would be a useful contribution," and appreciated that we test "on 10 different models of varying sizes." Reviewer ne1h noted that "the proposed method is proactive/predictive to some degree, which is important for the circuit discovery field," and found "interesting findings, such as all key heads in a circuit sharing the same scalar pattern in their tangent gradients." Reviewer E3XX noted that the linear model "has the potential to provide some clear mathematical insight into the question" and that "the list of experiments seems reasonable."

Weaknesses.

The reviewers raised the following common concerns.

Readability and the presentation of the theory. (Rev. sCzo, E3XX, ne1h) The eNTK is defined in Section 2 and the relative linearization error Rtan\mathcal R_{\mathrm{tan}} in Section 3.4. Section 3 is rebuilt around an exact path identity (Eq. 1), an initial sign alignment that can be checked on the base model (Eq. 2), and one conditional sign criterion whose assumptions are stated explicitly (Eq. 3). We removed the lemma, proposition, theorem and corollary labels throughout, and the derivations remain in Appendices D and F. "Seal(ed)" is no longer used.

What the framework adds beyond measuring the DLA. (Rev. sCzo, E3XX) Beyond ranking heads, we test an explicit signed rule. Under an improving objective, negative-attribution heads become more negative (Negative Name Movers 10.7 and 11.10: 16/16 head-seed pairs). Under the reversed objective, the predicted sign reversal holds for 9 of the 10 top-ranked heads in every seed; in seed 0, the only seed with per-head data, the exception is head 11.2. A separate calculation on the base model, the objective-specific initial rate κh(0)\kappa_h(0), also yields a sign consistent with this exception (Appendix D); we treat it as an auxiliary diagnostic, not a success of the pre-registered forecast. SFT, DPO and RLVR have different parameter-update directions (DPO-SFT cosine 0.45), yet their initial circuit-head rates, computed with the 256-prompt training gradient, have matching signs on 12/12 heads (11/12 for DPO on a smaller 64-prompt batch). We also state what the theory does not do: the path conditions of Eq. 3 concern the realized trajectory, and our post-training diagnostics assess them only partially, so they explain the result conditionally rather than certifying the forecast in advance.

Comparison with post-hoc circuit discovery. (Rev. ne1h) New in the revision (Section 4.3, Appendix G.4): patching the ten heads chosen before training recovers 0.62 of the fine-tuning gain, compared with 0.64 for a post-hoc top-|ΔDLA| selection and 0.10 for random heads under the same intervention. We also report ACDC-inspired head-level pruning with faithfulness and incompleteness, including the cases where the methods disagree, and (new) the original edge-level ACDC and Edge Pruning, which give the same qualitative picture.

Training dynamics, optimizers, and the relative linearization error. (Rev. E3XX, ne1h) New: sign agreement over 200 SFT steps, AdamW and LoRA sweeps that show where sign agreement declines, and the second-order Taylor term. Following the reviewers' comments, we also renamed the creation index C\mathcal C to the relative linearization error Rtan\mathcal R_{\mathrm{tan}} (called the tangent-residual ratio, TRR, in the earlier revision and in our individual responses), which measures departure from the first-order approximation at the base model, not capability creation. The second-order term lowers it for DPO from 0.87 to 0.21.

For the details, please refer to the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.