- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to predict the circuit graph of a model after post-training using only the base model. Its key assumption is that post-training doesn't really give the model brand-new abilities, but rather just activates existing ones. Because of this, parameter changes are assumed to be roughly in the same direction and linear, so they can be estimated directly by taking the inner product between the current model's local tangent vector and the task gradient direction .
Critical strengths
-
The writing is clear.
-
The paper has some interesting findings, such as all key heads in a circuit sharing the same scalar pattern in their tangent gradients, and the effect of parameter changes from post-training being almost linear.
-
The proposed method is proactive/predictive to some degree, which is important for the circuit discovery field.
Critical weaknesses
-
The tasks used (like IOI, Greater-Than, etc.) are very simple toy tasks, and the base model is already good at them. They don't really reflect what post-training actually changes. Although the authors tested a modified IOI task to show this, its output pattern is still fixed, and it's not clear whether a mechanistic circuit even needs to form for it. So it's still doubtful whether this method actually works in real-world post-training.
-
The authors don't directly compare their method with circuit discovery methods run on the post-trained model. While they claim that such a comparison would be unfair, the current evaluation still feels biased and unconvincing. First, EAP already has the largest attribution score error among all circuit discovery methods and also relies on a first-order Taylor expansion, so methods like ACDC and Edge-Pruning should be included. Second, for circuit faithfulness, the authors only check how much performance drops after ablation instead of how much performance the circuit itself can retain, which is not convincing enough.
-
The design of the creation index is quite controversial. itself is already a rough first-order approximation with noticeable error, and there isn't solid proof showing a clear relationship between non-linearity and training modes.
Questions for authors
-
Can the authors show performance on more realistic post-training tasks, such as coding, math, instruction following, or CoT?
-
Can the authors provide comparisons with circuits found by other methods on the post-trained model, like ACDC and Edge-Pruning, and evaluate them using standard faithfulness and completeness metrics?
-
Can the authors show how the prediction error changes dynamically as the number of parameter update steps increases?
-
Existing studies show that during post-training, circuit stability drops significantly because the underlying mechanisms are reframed to some extent (e.g., [1, 2]). How do the authors explain the tension between this observation and Proposition 2 proposed in the paper?
[1] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
[2] Circuit Stability Characterizes Language Model Generalization
2 comments
Additional details on realistic tasks, the creation index (now the tangent-residual ratio), update steps, and circuit stability.
[W1/Q1] We agree that this is the main limit of the paper's scope, and the revised limitations say so first. The method needs a per-component readout of the target behavior. The DLA provides one for tasks with an identifiable answer token, whereas open-ended generation (reasoning, coding, instruction following) would need another readout, such as a probe direction, and a validated circuit. Within that constraint, the lookup control has no imported head list and its update is far from linear (), yet the forecast heads carry the change (overlap 6.4/10) with the forecast signs (1.00, 10 seeds). The sign forecast also holds on bases up to 7B and on an instruction-tuned base (0.77-1.00); the clearest failure is a 7B-Instruct run whose fine-tune barely moves the task (gain +1.4, sign agreement 0.40). On the modified task: a fixed output pattern indeed does not show that a new circuit forms, and our claim does not need one to form. It predicts where the change lands among existing components. We did not attempt a reasoning or coding fine-tune in the discussion period, because doing it properly needs the readout and circuit validation described above.
[W3] We thank the reviewer for pointing out that the original name and interpretation overstated what this quantity measures. We have renamed the creation index to the tangent-residual ratio (TRR, ) throughout the revised manuscript (Section 3.4, Figure 5, tables and appendices). The new name follows its definition: the residual energy relative to the combined residual and base-tangent energies. It is not an estimator of capability creation, and we agree that nothing links it to training modes. The JVP computes the tangent prediction exactly, but that does not make the first-order approximation exact. Adding the quadratic Taylor term reduces the ratio for DPO from 0.87 to 0.21 (new, App. G.2), so much of its first-order residual is curvature. We therefore present it as a descriptive diagnostic of departure from the base tangent.
[Q3] Plain gradient descent, extended to five times the standard number of steps (new, App. G.3; 5 seeds):
| step | 1 | 5 | 10 | 20 | 40 | 80 | 120 | 160 | 200 |
|---|---|---|---|---|---|---|---|---|---|
| gain | 1.6 | 2.7 | 3.3 | 4.0 | 4.8 | 5.5 | 6.0 | 6.3 | 6.5 |
| sign | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Overlap with the preselected top ten stays 8.2/10 at every step, and the rank correlation of change magnitudes declines slightly (0.68 -> 0.63) as rises (0.12 -> 0.18). A constant overlap does not by itself show that every checkpoint has an identical set of changing heads. With AdamW and LoRA (500 steps, 8 seeds per rate, Table 12), agreement is 1.00 for every observed gain up to +13.5, an observed range rather than a general threshold. Beyond it, agreement falls (AdamW 0.91 at +17.8 and 0.79 at +26.5, LoRA 0.86 at +20.2), and almost all failing heads are minority-sign heads (on GPT-2 IOI, the negative heads 10.7, 11.10 and 11.2). In a step-resolved AdamW run the break happens between step 10 (gain +8.8, sign 1.00) and step 25 (+21.0, 0.82). We conjecture that these runs violate the path conditions of the sign criterion (Eq. 3), but we did not measure for them, and the base model does not tell us where that boundary lies.
[Q4] [2] (Sun, ACL 2025) defines circuit stability as a model applying a consistent circuit across inputs; it does not study change under post-training, so we see no direct tension. [1] (Chen et al., 2026) finds that circuits evolve during SFT, so a circuit located on pre-update weights lags behind the model being updated. We agree, and our pruning finds base and fine-tuned circuits that differ substantially (Jaccard 0.41-0.42, Table 13). What we forecast is not circuit membership but the change itself: which heads it concentrates on and in which direction. Static circuit membership changes after tuning, whereas the base-DLA ranking retains predictive value for the location and sign of head-level attribution changes in our controlled tasks.
The initial sign alignment (old Prop. 2, now Eq. 2) is a statement at the base model. The sign criterion (Eq. 3) does not need the circuit to stay fixed; it needs the heads' rates to share one direction along training. The measured rate trajectories have high leading-mode energy (0.89-0.999), but only an endpoint margin proxy passes on 50 of 60 pairs (App. D), and the stronger integrated bound of Eq. 3 is not directly tested. This is compatible with large changes in which heads a pruning method keeps. The revision adds both papers: [1] in Section 4.3, next to our circuit-drift result, and both in the related work.
We thank the reviewer for the constructive review, and for noting that the shared scalar pattern of the circuit heads' tangents and the predictive angle are of interest. Below, we provide a detailed response addressing each of the weaknesses [W#] and questions [Q#] raised: this comment covers the requested comparison with circuit discovery on the post-trained model [W2/Q2]; a follow-up comment covers realistic tasks [W1/Q1], the creation index [W3], prediction error versus update steps [Q3], and circuit stability [Q4].
[W2/Q2] We agree that this comparison was missing, and the revision adds it (new, Section 4.3 and App. G.4). On the fine-tuned GPT-2 IOI model we ran ACDC-inspired greedy head-level pruning (KL criterion, discovered on separate prompts, scored on held-out prompts, with the per-template ABC mean ablation of Wang et al.) and score circuits with the standard metrics. Faithfulness F(C)/F(M) is what the circuit retains on its own, and incompleteness is ∣F(C∖K)−F(M∖K)∣/∣F(M)∣ over random K (lower is better).
To locate the change itself, we patch the outputs of a 10-head set (fraction of the gain; sufficiency = fine-tuned outputs into the base model, necessity = base outputs into the fine-tuned model; 5 seeds):
Per seed, the forecast is 0.02-0.04 below the post-hoc comparator at 10 heads, so we do not claim equivalence. With 20 heads: forecast 0.86 / 0.90, post-hoc 0.83 / 0.81, random 0.21 / 0.24. These are patching sufficiency and necessity scores, not a decomposition into parameter-only contributions, since head outputs also carry upstream changes.
On signs, direct-attribution signs are correct for all ten forecast heads. Individual patching effects agree on 9/10 in every seed; head 9.6 is the exception, consistent with interactions beyond the direct readout. Among the five heads with the largest base |DLA|, patching the three Name Movers recovers +0.27 of the gain and the two Negative Name Movers -0.39: the suppressors deepen, causally.
Where the methods disagree: at τ=0.003 the pruned fine-tuned circuit keeps 7 of the 10 forecast heads, including all three Name Movers, but drops 10.7, 11.2 and 10.6. The fine-tuned model is near saturation, so removing a head that lowers the logit difference changes the output distribution little, which plausibly explains why a KL criterion discards it. Pruning measures what the output needs, patching where the change is, and ΔDLA the change of a head's direct contribution.
Caveats (App. G.4): the pruning is node-level and single-pass, circuits are whole heads at all positions (so the numbers are not comparable to the 87% of Wang et al.), and MLPs are never ablated. We did not run edge-level discovery such as Edge Pruning, and the paper states this as a limitation. On EAP: our forecast is not a discovery method run on the fine-tuned model, and it is not EAP. It is the base model's DLA, computed before training, and attribution patching appears only as a cross-check.
We would appreciate reviewer ne1h's view on whether this comparison addresses the concern about the evaluation.