acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuning

Abstract

Mechanistic Localization bridges mechanistic interpretability and post-training optimization by isolating critical parameters via interpretative approaches and then guiding parameter-efficient Supervised Fine-Tuning (SFT) in a “locating-then-tuning” paradigm. However, due to the retrospective nature of mechanistic interpretability, directly interpreting pre-SFT models introduces misleading conclusions. Specifically for novel tasks, initially identified neurons differ drastically from those governing the final model, introducing biases that actively disrupt SFT. To address this, we propose a forward-looking localization framework that accurately estimates the post-SFT interpretability state using only pre-SFT parameters and the target dataset. Theoretically, we model SFT as a continuous parameter evolution, leveraging Taylor expansion to rigorously bridge the post-tuning mechanistic objective with the pre-SFT model's dynamic gradients. Practically, we design dual-granularity (neuron- and component-level) localization pipelines. Extensive experiments demonstrate that our approach not only provides superior SFT guidance but also exhibits robust performance and temporal scalability across increasing model sizes. This work transcends the fundamental limitation of traditional interpretability—its inability to identify task-critical mechanisms before they are trained—pioneering a predictive frontier that unites mechanistic interpretability with targeted optimization.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Large Language Models, SFT, Attribution Patching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_66JQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an improvement for "locating-then-tuning" paradigm. They propose to decompose the training process into two theoretical steps: isolating direction and extrapolating distance. Then, using these two properties, they define two locate-then-update pipelines (neuron level and component level) and test their effectiveness against several existing methods.

Their component-level method shows clear improvement over existing methods, which is evaluated across several tasks and models on both task improvement and retention of existing capabilities.

Overall, this is a nice and interesting paper. With some clarifications about my questions below and some improvements to the presentation, I would be happy to see it accepted. If addressed, I would be happy to further raise my score to a clear accept.

Critical strengths

The paper studies an interesting topic: changes in model mechanisms during FT, and how to identify components that will be influential after FT, in order to improve locate-then-update methods. These questions connect interpretability research with actionable training decisions, which show the potential of interpretability research, and specifically task and mechanism localization.

Their component-level method shows clear improvement over existing methods, which is evaluated across several tasks and models on both task improvement and retention of existing capabilities.

Critical weaknesses
  • The motivation of this paper is a bit unclear. The authors pose this question as their main research question: how can we predict the neurons responsible for a target task in an ideal post-SFT model using only the pre-SFT model parameters? However, this is not the question they later answer, as we see no comparison of empirical distance or component overlap between their predictions and the set of critical components that manifested after SFT. As far as I understand, the paper aims to find the best set of components to update in order to achieve the best task performance (TTA/PTA). As they compare against other locate-then-update methods. I suggest making this clearer in the intro and in the results section.

  • The authors cite Chen et al [1] for the claim that circuits change significantly across FT. However, the results for Chen et al are not reproduced in other cases [2], which show that FT largely preserve and enhance existing mechanisms. Did the authors verify this claim on their tasks and mechanisms? I would be happy to see this empirical result in the paper (even in an appendix with a brief mention in the main text), as this is an active discussion in the literature which has contradicting evidence. Moreover, citing Prakash et al [2] for “mechanisms undergo drastic transformations during SFT” seems like a wrong citation, since it does not match their main claims.

  • What are the base performances of the tested models on the tasks? This heavily influences FT results and how we view the findings, as there is a difference in what we expect FT to change in model mechanisms when the base performance is 30% or 99%.

  • The results section of this paper is a bit dense and difficult to follow, I would be happy to see it a bit more digested. Additionally, the captions in this section are short and missing some details: In table 1, what metric does it show?, In table 2: which task is shown? Is this an avg? There is a single line of text between tables 1 and 2 - I suggest rearranging this.

  • Some examples of the tested tasks would greatly help me understand the setup of the paper.

  • The tested tasks are not of fixed length and fixed positions, as is usually the case when using attribution patching. Could the authors elaborate on this point, and how they overcome it?

[1] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training. Chen et al. [2] FINE-TUNING ENHANCES EXISTING MECHANISMS: A CASE STUDY ON ENTITY TRACKING. Prakash et al.

Minor:

  • Typo in the capitalization of Llama-2-13B, line 77
  • The paper keeps mentioning “neurons/components” - this makes it unclear if the paper only accounts for neurons, attention heads, or both during the localization analysis. I suggest making this point clearer and removing the “/”.
Questions for authors

See section above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 66JQ (Q1)

We sincerely thank Reviewer 66JQ for recognizing our work, especially that it connects interpretability research with actionable training decisions. We also thank you for the many suggestions to improve the readability and theoretical robustness of our paper. Following your suggestions, we have carefully revised and supplemented the manuscript and verified the issues you raised (please see the updated PDF). Below, we respond to each point.


Q1: "The motivation of this paper is a bit unclear. The authors pose this question as their main research question: how can we predict the neurons responsible for a target task in an ideal post-SFT model using only the pre-SFT model parameters? However, this is not the question they later answer, as we see no comparison of empirical distance or component overlap between their predictions and the set of critical components that manifested after SFT. As far as I understand, the paper aims to find the best set of components to update in order to achieve the best task performance (TTA/PTA). As they compare against other locate-then-update methods. I suggest making this clearer in the intro and in the results section." (Weakness 1)

We sincerely thank you for this insightful comment, and we agree that our goal should be stated more clearly.

In short, we have revised the main research question to predicting the critical neurons for better SFT performance; the comparison between our prediction and the post-SFT circuit is provided in the appendix.

Following your suggestion, we have revised the main research question in the introduction from predicting the critical neurons of the ideal post-SFT model to predicting them for better SFT performance. Moreover, please allow us to note that Appendix F.4 and Appendix H compare our predicted circuit with the post-SFT circuit on several interpretability metrics: the ranking of component causal effects (Figure 10), Top@50 overlap and KL divergence (Tables 5–6), and Relative Hamming Distance (Figure 12). Due to the page limit, we placed them in the appendix, as we considered the main results (Table 1) and the ablation study (Table 2) more important for the main text.

To this end, we have made the following paper changes (please refer to our updated PDF):

  • Abstract: the framework is now motivated by better SFT performance, and the experiments attribute our gains to predicting more accurate localization.
  • Introduction: the research question now begins with "To this end and to achieve better SFT performance", and the contributions now claim state-of-the-art locating-then-tuning performance.
  • Section 2: our objective now includes tuning only the predicted parameter subset so that the resulting model approaches the ideal model.

AuthorsOpenReview ↗

Response to Reviewer 66JQ (Q2)


Q2: "The authors cite Chen et al [1] for the claim that circuits change significantly across FT. However, the results for Chen et al are not reproduced in other cases [2], which show that FT largely preserve and enhance existing mechanisms. Did the authors verify this claim on their tasks and mechanisms? I would be happy to see this empirical result in the paper (even in an appendix with a brief mention in the main text), as this is an active discussion in the literature which has contradicting evidence. Moreover, citing Prakash et al [2] for 'mechanisms undergo drastic transformations during SFT' seems like a wrong citation, since it does not match their main claims." (Weakness 2)

We sincerely thank you for pointing out this active discussion and for the valuable suggestion.

In short, we have verified this on 13 tasks, and both findings hold depending on adaptation difficulty: circuits are largely preserved for tasks the base model has mastered, as Prakash et al. found, but change substantially for harder tasks, as Chen et al. found; we have added this experiment to the paper as a new Discussion section.

To study this, we rank 13 tasks by adaptation difficulty, roughly measured by the base model's (1−accuracy)×shots(1-\text{accuracy}) \times \text{shots} and rescaled to [0,1][0, 1]: the hardest tasks are those that the base model still fails after many shots. On Mistral-7B, we tune each task and report five metrics:

  • CD (Circuit Distance, ×10−5\times10^{-5}): a continuous metric that mainly reflects changes in edge weights.
  • RHD (Relative Hamming Distance, %): a discrete metric that only counts added and removed edges.
  • Static / Ours / Full-Param: TTA (%) when tuning with the base-model circuit as the localization, with our extrapolated localization, and with full-parameter SFT, respectively.

The results are as follows:

TierTaskDifficultyCDRHDStaticOursFull-Param
1Docstring0.050.145100100100
1COPA0.080.124919191
1IOI0.160.189100100100
1Greater-than0.180.267949595
1Gender0.190.3911100100100
2BOOL0.210.93169810099
2RTE0.241.6318838786
2WinoGrande0.463.8724727674
2ARC0.664.3826849183
2Arithmetic0.844.54239810098
3TOFU0.915.9736939498
3Current Events0.956.3553413850
3SMolInstruct1.007.5644828188

As shown in the table, the tasks fall into three tiers:

  • Tier 1: mastered tasks. The base model already masters these tasks, so SFT barely changes their performance or circuit. Hence, the base-model circuit is already predictive, and Static performs as well as Ours. This agrees with findings that SFT barely changes the circuits of skills the base model has mastered [1–3].
  • Tier 2: nearly mastered tasks. The base model has the needed sub-skills but must learn a pattern or knowledge, so SFT quickly learns the output format or small skills and slightly updates stored knowledge. This is also the most common case of SFT in practice. Here, the circuit structure changes little, but its edge ranks change sharply, so our extrapolation clearly beats static prediction. This agrees with findings that typical SFT changes circuit weights substantially, making prediction from the base model inaccurate [4, 5].
  • Tier 3: novel tasks. The base model has never seen these tasks and lacks most of the domain knowledge, so SFT must form new components and functional heads. The circuit thus gains and loses many edges and nodes, and almost every prediction method fails. This agrees with findings that circuits change almost entirely when models acquire new knowledge or algorithms [6, 7].

Regarding the citation of Prakash et al., we agree that it does not support 'drastic transformations': their finding corresponds to our Tier 1.

Finally, we have made the following paper changes:

  • Section 6 (Discussion) now presents this experiment.
  • Section 2 now cites Prakash et al. for the preservation of mechanisms on mastered tasks.

References:

[1] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking.

[2] Neuroplasticity and Corruption in Model Mechanisms: A Case Study of Indirect Object Identification.

[3] Retrieval Head Mechanistically Explains Long-Context Factuality.

[4] Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis.

[5] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training.

[6] How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training.

[7] Progress Measures for Grokking via Mechanistic Interpretability.


AuthorsOpenReview ↗

Response to Reviewer 66JQ (Q3-Q7)


Q3: "What are the base performances of the tested models on the tasks? This heavily influences FT results and how we view the findings, as there is a difference in what we expect FT to change in model mechanisms when the base performance is 30% or 99%." (Weakness 3)

We sincerely thank you for and agree with this insightful question.

In short, the base performances are reported in Figure 7 (Figure 8 in the revision), and they indeed shape our findings: the lower the base performance, the more extrapolation (a larger KK) is needed, and the more the circuit changes during SFT.

Please allow us to note that Appendix F.1 reports the base performance of Mistral-7B on each task in the legend of Figure 7 (Figure 8 in the revision). We list it again below:

Base performance of Mistral-7B before SFT (TTA, %).

BOOLArithmeticInductionWinoGrandeSST-2GenderIOIDocstring
0.00.032.555.856.961.563.887.5

As shown in Appendix F.1, the base performance is correlated with the suitable range of KK. For tasks with near-zero performance (BOOL and Arithmetic), the overlap with the circuit from SFT model stays high over a broad range K∈[10,80]K \in [10, 80], whereas for tasks with higher performance (Docstring, IOI, and Gender), the suitable KK concentrates in [0,10][0, 10]. In other words, the lower the base performance, the more extrapolation is needed. Moreover, as shown in our response to Q2, the base performance, also somehow determines how much the circuit changes during SFT and how accurate our prediction is.


Q4: "The results section of this paper is a bit dense and difficult to follow, I would be happy to see it a bit more digested. Additionally, the captions in this section are short and missing some details: In table 1, what metric does it show?, In table 2: which task is shown? Is this an avg? There is a single line of text between tables 1 and 2 - I suggest rearranging this." (Weakness 4)

We sincerely thank you for these helpful suggestions.

We have revised the results section for readability: the captions of Tables 1–2 and Figures 2–4 now state the metrics, tasks, and settings, and the layout has been rearranged.

Moreover, we have made the following paper changes:

  • Table 1: the caption now states the model, evaluation, metrics, and tasks in each column, and that the values are mean ± std over 5 runs.
  • Table 2: the caption now states that all values are TTA (%), and that each value averages the tasks in its domain.
  • Figures 2–4: the captions now state the tasks, metrics, and settings.
  • Section 5: Section 5.2 is split into three headed paragraphs.

Q5: "Some examples of the tested tasks would greatly help me understand the setup of the paper." (Weakness 5)

We sincerely thank you for this suggestion. Please allow us to remind that Table 3 in Appendix D already lists the input prompt and an example for each task, and Figure 12 (Figure 13 in the revision) shows the circuit graphs of different localizations as examples. To make our setup even clearer, we have made the following paper change: Table 3 now also shows the corrupted input of each example, which replaces only one or two tokens of the clean input to change the answer.


Q6: "The tested tasks are not of fixed length and fixed positions, as is usually the case when using attribution patching. Could the authors elaborate on this point, and how they overcome it?" (Weakness 6)

We sincerely thank you for raising this important point. As stated in footnote 4 (Section 5.1) and Appendix D, we constrain the label of each dataset to a single token, and we reformulate the inputs into question-answering templates when necessary. Therefore, the metric of attribution patching is always computed at the final position, where all samples predict a label with the same semantics (e.g., Yes/No or an option letter). Nevertheless, we agree that this restriction limits the tasks we can study: current mechanistic interpretability mainly analyzes next-token prediction, which we discuss as a limitation in Section 7. We are already working on mechanistic interpretability for multi-token outputs.


Q7: "Typo in the capitalization of Llama-2-13B, line 77." "The paper keeps mentioning 'neurons/components' - this makes it unclear if the paper only accounts for neurons, attention heads, or both during the localization analysis. I suggest making this point clearer and removing the '/'." (Minor)

We sincerely thank you for pointing out these issues.We have proofread the paper again and corrected these expressions and typos; the changes are shown in red in the updated PDF. Thank you again for your careful review and valuable suggestions!


We hope these responses address your concerns. If you have any further questions, please feel free to discuss them with us at any time.

Reviewer_VGXLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes an interesting approach to identifying important parameters for SFT. The authors use a lightweight probing SFT step to estimate an initial parameter-update direction and extrapolate the importance of components in the anticipated post-SFT model, and the predicted importance scores are subsequently used to guide fine-tuning at two levels: selecting individual parameters at the neuron level and allocating LoRA ranks at the component level.

The authors evaluate their approach on several downstream tasks, including logical reasoning, natural language understanding, and mathematical reasoning. Mistral-7B serves as the primary evaluation model, while additional experiments on LLaMA-2-13B and Qwen3-30B investigate scalability. The reported results demonstrate promising improvements over several existing localization approaches.

Critical strengths
  1. Interesting motivation and a potentially useful perspective on parameter importance during SFT.

The paper addresses a meaningful limitation of static mechanistic localization: components that are important in a pre-SFT model may not remain the most important after fine-tuning.

The idea of anticipating future parameter importance using a small initial update is interesting and could be particularly useful when adapting pretrained models to tasks that require learning new capabilities. I consider this perspective a meaningful contribution to the relationship between mechanistic interpretability and model optimization.

  1. Promising empirical results and clear presentation.

The authors evaluate their approach across different tasks and model sizes, and provide comparisons with several existing localization methods. The neuron-level experiments show encouraging improvements over the probing baseline, particularly on logical and mathematical reasoning tasks.

The paper is also generally well organized, with clear figures and tables that make the proposed methodology and experimental comparisons relatively easy to follow.

Critical weaknesses
  1. Insufficient justification of the theoretical approximations.

The method relies on first-order Attribution Patching and extrapolation along an initial parameter-update direction. However, clean and corrupted activations may differ substantially, and the actual SFT trajectory may deviate from the initial direction. It remains unclear whether these approximations reliably preserve the rankings of important components.

  1. Limited evidence for the additional benefits of future-mechanism extrapolation.

Although the paper includes CircuitLoRA as a baseline, the component-level evaluation lacks comparisons with standard fixed-rank LoRA and other adaptive rank-allocation methods. Such comparisons under matched parameter and computational budgets would help establish whether future-mechanism extrapolation offers practical benefits beyond existing LoRA allocation strategies.

Questions for authors
  1. How well does the initial update direction approximate the actual SFT trajectory, particularly for tasks requiring substantial adaptation?
  2. Could the authors compare the component-level method with standard fixed-rank LoRA and representative adaptive rank-allocation methods, including more recent approaches, under matched parameter and computational budgets? This would help clarify whether future-mechanism extrapolation provides benefits beyond existing rank-allocation strategies.
  3. How can suitable K intervals be determined without access to a fully fine-tuned reference model? Were such references used to select K in the main experiments?
3 comments
AuthorsOpenReview ↗

Response to Reviewer VGXL (Q1)

We sincerely thank Reviewer VGXL for the positive assessment of our work. We also thank you for the valuable suggestions on our theoretical support and on the robustness and scalability of our experiments. Following your suggestions, we have supplemented and discussed the theoretical support and the related experiments, and revised the paper accordingly (please see the updated PDF). Below, we respond to each point.


Q1: "Insufficient justification of the theoretical approximations. The method relies on first-order Attribution Patching and extrapolation along an initial parameter-update direction. However, clean and corrupted activations may differ substantially, and the actual SFT trajectory may deviate from the initial direction. It remains unclear whether these approximations reliably preserve the rankings of important components." (Weakness 1)

We sincerely thank you for this comment. Below we will show from both theory and experiments, that linearly estimating post-SFT interpretability is justifiable.

In short, prior theory supports each step of our linear approximation, and our experiments confirm that it preserves the rankings of the top components. We have also added this theoretical basis to the introduction.

Theoretically, prior work supports each part of our approximation. First, training trajectories are locally well approximated by a first-order Taylor expansion [1–3]. Second, what ranks as important after fine-tuning, from mechanisms and parameters to training data, can be predicted from the base model [4–6]. Third, fine-tuning updates tend to keep their initial direction, along which the model can be extrapolated [7–9].

Experimentally, Appendix F.4 shows that, for the top-ranked components, our estimated ranking closely matches that of natural SFT without localization. Appendix H further shows that our estimated circuit overlaps more with the post-SFT circuit than the circuit of the base model does. These results provide evidence from inside the model: although linear extrapolation leaves some error relative to the true post-SFT circuit, it is accurate enough for the top-ranked components. Together with the improved task performance, they support that our approximation is reliable.

As for the gap between clean and corrupted activations, our derivation always treats their difference as a whole to avoid errors caused by a gap between two. Moreover, prior work shows that fine-tuning largely preserves the difference between clean and corrupted activations [10, 11].

To this end, we have made the following paper change (please refer to our updated PDF):

  • The introduction now includes the following paragraph: "This question builds on prior evidence that fine-tuning is largely predictable from the model before it. First, training trajectories are locally well approximated by a first-order Taylor expansion [1–3]. Second, what ranks as important after fine-tuning, from mechanisms and parameters to training data, can be predicted in advance [4–6]. Third, fine-tuning updates tend to keep their initial direction, along which the model can be extrapolated [7–9]."

References:

[1] Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. NeurIPS 2019.

[2] A Kernel-Based View of Language Model Fine-Tuning. ICML 2023.

[3] Learning Dynamics of LLM Finetuning. ICLR 2025.

[4] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024.

[5] Training Neural Networks with Fixed Sparse Masks. NeurIPS 2021.

[6] LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024.

[7] LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently. ICML 2025.

[8] Editing Models with Task Arithmetic. ICLR 2023.

[9] Model Extrapolation Expedites Alignment. ACL 2025.

[10] Mechanistically Analyzing the Effects of Fine-Tuning on Procedurally Defined Tasks. ICLR 2024.

[11] Retrieval Head Mechanistically Explains Long-Context Factuality. ICLR 2025.


AuthorsOpenReview ↗

Response to Reviewer VGXL (Q2)


Q2: "Limited evidence for the additional benefits of future-mechanism extrapolation. Although the paper includes CircuitLoRA as a baseline, the component-level evaluation lacks comparisons with standard fixed-rank LoRA and other adaptive rank-allocation methods. Such comparisons under matched parameter and computational budgets would help establish whether future-mechanism extrapolation offers practical benefits beyond existing LoRA allocation strategies." "Could the authors compare the component-level method with standard fixed-rank LoRA and representative adaptive rank-allocation methods, including more recent approaches, under matched parameter and computational budgets? This would help clarify whether future-mechanism extrapolation provides benefits beyond existing rank-allocation strategies." (Weakness 2 + Question 2)

We sincerely thank you for the attention and the constructive suggestion.

In short, fixed-rank LoRA with the same average rank is already the component-level w/o Localization baseline in Table 2, and following your suggestion, we have added two adaptive rank-allocation methods, AdaLoRA and EVA, to Table 2.

In our experiments, the baseline closest to our method in parameter count is LoRA with a fixed rank of 8, which is the component-level "Full-Param" row of Table 2. Since "Full-Param" may be misleading, we have renamed it "w/o Localization". Moreover, following your suggestion, we add two adaptive rank-allocation baselines, AdaLoRA [1] and EVA [2]: we replace the rank allocation of our component-level pipeline with theirs, keeping the average rank at 8. The results are as follows:

Table 2 (revised, component level). TTA (%), Mistral-7B.

Rank allocationLRNLUMR
w/o Localization (fixed rank 8)99.7386.3371.83
AdaLoRA [1]99.8686.4098.54
EVA [2]99.7986.3578.54
Ours100.0087.55100.00

Both adaptive methods fall behind ours: AdaLoRA drops slightly, and EVA drops more. We believe this is because AdaLoRA updates its rank allocation at every step, but early in SFT its allocation is closer to the Static localization than to the post-SFT circuit; EVA, like Static, determines its allocation from the pre-SFT model.

Therefore, we have made the following paper changes:

  • Table 2 in Section 5.3 renames Full-Param to w/o Localization and adds AdaLoRA and EVA at the component level.

References:

[1] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023.

[2] Parameter Efficient Fine-tuning via Explained Variance Adaptation. NeurIPS 2025.


AuthorsOpenReview ↗

Response to Reviewer VGXL (Q3-Q4)


Q3: "How well does the initial update direction approximate the actual SFT trajectory, particularly for tasks requiring substantial adaptation?" (Question 1)

We sincerely thank you for this important question.

In short, the initial direction approximates the SFT trajectory well unless the task is entirely new to the base model; we have added this analysis to the paper as a new Discussion section.

To study this, we rank 13 tasks by adaptation difficulty, roughly measured by the base model's (1−accuracy)×shots(1-\text{accuracy}) \times \text{shots} and rescaled to [0,1][0, 1]: the hardest tasks are those that the base model still fails after many shots. On Mistral-7B, we tune each task and report five metrics:

  • CD (Circuit Distance, ×10−5\times10^{-5}): a continuous metric that mainly reflects changes in edge weights.
  • RHD (Relative Hamming Distance, %): a discrete metric that only counts added and removed edges.
  • Static / Ours / Full-Param: TTA (%) when tuning with the base-model circuit as the localization, with our extrapolated localization, and with full-parameter SFT, respectively.

The results are as follows:

TierTaskDifficultyCDRHDStaticOursFull-Param
1Docstring0.050.145100100100
1COPA0.080.124919191
1IOI0.160.189100100100
1Greater-than0.180.267949595
1Gender0.190.3911100100100
2BOOL0.210.93169810099
2RTE0.241.6318838786
2WinoGrande0.463.8724727674
2ARC0.664.3826849183
2Arithmetic0.844.54239810098
3TOFU0.915.9736939498
3Current Events0.956.3553413850
3SMolInstruct1.007.5644828188

As shown in the table, the tasks fall into three tiers, which agree with prior findings:

  • Tier 1: mastered tasks. The base model already masters these tasks, so SFT barely changes their performance or circuit. Hence, the base-model circuit is already predictive, and Static performs as well as Ours. This agrees with findings that SFT barely changes the circuits of skills the base model has mastered [1–3].
  • Tier 2: nearly mastered tasks. The base model has the needed sub-skills but must learn a pattern, knowledge, or template, so SFT quickly learns the output format or small skills and slightly updates stored knowledge. This is also the most common case of SFT in practice. Here, the circuit structure changes little, but its edge ranks change sharply, so our extrapolation clearly beats static prediction. This agrees with findings that typical SFT changes circuit weights substantially, making direct prediction from the base model inaccurate [4, 5].
  • Tier 3: novel tasks. The base model has never seen these tasks and lacks most of the domain knowledge and sub-skills, so SFT must form new components and functional heads. The circuit thus gains and loses many edges and nodes, or the base model cannot form an effective circuit at all, and almost every prediction method fails. This agrees with findings that circuits change almost entirely when models acquire new knowledge or algorithms [6, 7].

Finally, we have made the following paper change:

  • Section 6 (Discussion) now presents this experiment.

References:

[1] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024.

[2] Neuroplasticity and Corruption in Model Mechanisms: A Case Study of Indirect Object Identification. Findings of NAACL 2025.

[3] Retrieval Head Mechanistically Explains Long-Context Factuality. ICLR 2025.

[4] Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis. ICML 2025.

[5] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training. NeurIPS 2026.

[6] How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training. Findings of ACL 2025.

[7] Progress Measures for Grokking via Mechanistic Interpretability. ICLR 2023.


Q4: "How can suitable KK intervals be determined without access to a fully fine-tuned reference model? Were such references used to select KK in the main experiments?" (Question 3)

We sincerely thank you for this question and apologize for the misunderstanding.

No reference model is used: in the main experiments, KK is randomly sampled from the fixed interval [0,100][0, 100] for all tasks, and we have stated this in Section 5.1.

We have made the following paper change :

  • Section 5.1 now states: "For all tasks, we randomly sample KK from [0,100][0, 100] ten times and average the resulting causal effects."

We hope these experimental and theoretical results address your concerns. If you have any further questions, please feel free to leave a comment at any time.

Reviewer_r6REOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles a limitation in locating-then-tuning: mechanistic localization performed strictly on a pre-SFT base model often misses subnetworks that emerge during fine-tuning. The authors propose estimating post-SFT attribution scores without running full SFT. They do this by fine-tuning a cheap "probing" model θ′\theta' (1 epoch on 1% data at 1% learning rate) and using a first-order Taylor expansion coupled with a trapezoidal rule to extrapolate scores along the update direction Δθ=θ′−θ\Delta\theta = \theta' - \theta out to θ+KΔθ\theta + K\Delta\theta.The method is deployed in two ways:Neuron-level tuning: Updating the top 20% ranked weights.Component-level tuning: Allocating LoRA ranks dynamically by importance (ranks 1–32, mean 8).Validation is conducted on Mistral-7B, LLaMA-2-13B, and Qwen3-30B across BOOL, GLUE, and multiple-choice Arithmetic tasks, comparing against baselines including Graft, WAGLE, CLUE, FLU, and CircuitLoRA.

Critical strengths
  1. This paper addressed the mismatch between pre-SFT localization and post-SFT representation drift is an important challenge for locating-then-tuning methods. Framing localization to anticipate parameter trajectories is conceptually sound, and testing it via Static vs. Probing vs. Extrapolation ablations directly targets the core hypothesis.

  2. When evaluated against existing localization baselines (e.g., Graft, WAGLE, CircuitLoRA), the proposed ranking delivers notable gains across diverse task benchmarks (BOOL, GLUE, Arithmetic) and scales consistently from 7B to 30B parameter models at low computation cost.

Critical weaknesses
  1. Theoretical over-formalization masking a linear weight extrapolation baseline: The translation-operator formulation in Appendix B relies fundamentally on assuming a linear training trajectory (θK=θ+KΔθ\theta_K = \theta + K\Delta\theta). Under this assumption, the path integral simplifies directly to attribution patching evaluated at the linearly extrapolated point θ+K(θ′−θ)\theta + K(\theta' - \theta). Because the authors do not evaluate attribution patching directly computed on these extrapolated weights, it remains unclear whether the Taylor series machinery offers any tangible benefit over standard linear weight extrapolation (e.g., ExPO-style probing). Furthermore, empirical results in Figure 11 indicate that the first-order approximation unexpectedly outperforms the exact second-order derivation, contradicting the theoretical premise.

  2. Information leakage from post-SFT checkpoints and inconsistent baselines: The claim of operating "using only pre-SFT parameters" is undermined by the hyperparameter selection in Appendix F.1 and F.2, where the extrapolation scalar KK is tuned or learned against the fully fine-tuned model. Additionally, the experimental setup exhibits inconsistencies: in Table 2, the full fine-tuning baseline achieves only 15.66% on Multi-digit Reasoning (near random guessing for a 7-choice task), while elsewhere the post-SFT model is treated as a near-perfect ground truth reference.

Questions for authors
  1. What are the empirical results if attribution patching is calculated directly on the extrapolated checkpoint θ+KΔθ\theta + K\Delta\theta using standard passes, bypassing the Taylor expansion/trapezoidal approximation?
  2. Were the task-specific KK intervals in Tables 1–2 chosen using metrics derived from the fully fine-tuned checkpoint (as described in App. F.1)? If so, how does the method perform when KK is fixed a priori or selected using only base/probing validation loss?
  3. Why does full fine-tuning achieve only 15.66% accuracy on Multi-digit Reasoning (MR) in Table 2? What training hyperparameters (learning rate, schedule) were used, and does the proposed method maintain its performance margin over a properly converged full-SFT baseline?
  4. Eq. 12 uses a finite difference at θ′\theta'. In practice, does evaluating the expectation over 10 sampled KK values require forward/backward passes at each θ+KΔθ\theta + K\Delta\theta? Please clarify the exact forward/backward pass count per candidate component.
4 comments
AuthorsOpenReview ↗

# Response to Reviewer r6RE (Q1)

We sincerely thank Reviewer r6RE for recognizing our motivations and contributions, as well as for the close attention to the accuracy of our extrapolation and to our experimental details and the valuable suggestions. We have organized your questions and answer them one by one below, and we have revised the paper accordingly (please see the uploaded PDF).


Q1: "Theoretical over-formalization masking a linear weight extrapolation baseline: The translation-operator formulation in Appendix B relies fundamentally on assuming a linear training trajectory (θK=θ+KΔθ\theta_K = \theta + K\Delta\theta). Under this assumption, the path integral simplifies directly to attribution patching evaluated at the linearly extrapolated point θ+K(θ′−θ)\theta + K(\theta' - \theta). Because the authors do not evaluate attribution patching directly computed on these extrapolated weights, it remains unclear whether the Taylor series machinery offers any tangible benefit over standard linear weight extrapolation (e.g., ExPO-style probing). Furthermore, empirical results in Figure 11 indicate that the first-order approximation unexpectedly outperforms the exact second-order derivation, contradicting the theoretical premise." (Weakness 1)

We sincerely thank you for your careful reading of Appendix B and for suggesting a linear extrapolation baseline.

In short, our method approximates the attribution gradient, S(θI)≈S(θ+KΔθ)S(\theta^{I}) \approx S(\theta + K\Delta\theta), which is not equivalent to attribution patching at the extrapolated point, ΔE(θI)≈ΔE(θ+KΔθ)\Delta E(\theta^{I}) \approx \Delta E(\theta + K\Delta\theta); following your suggestion, we have also added two extrapolation-like baselines to the main results (Table 1) for comparison.

Please allow us to clarify this misunderstanding with Eq. 11, which estimates the post-SFT attribution as

ΔEθI(a)≈ΔEθ(a)+12[S(θ0)+S(θ0+KΔθ)]⋅KΔθ.\Delta E^{\theta^{I}}(a) \approx \Delta E^{\theta}(a) + \frac{1}{2}\left[S(\theta^0) + S(\theta^0 + K\Delta\theta)\right] \cdot K\Delta\theta.

The extrapolated point θ+KΔθ\theta + K\Delta\theta appears only inside SS. Therefore, approximating S(θI)S(\theta^{I}) by S(θ+KΔθ)S(\theta + K\Delta\theta) does not imply ΔE(θI)≈ΔE(θ+KΔθ)\Delta E(\theta^{I}) \approx \Delta E(\theta + K\Delta\theta). In other words, we estimate the attribution of the ideal post-SFT model, not that of the checkpoint θ+KΔθ\theta + K\Delta\theta. Nor is θ+KΔθ\theta + K\Delta\theta an estimate of the post-SFT parameters: it only provides the point at which SS is evaluated.

Nevertheless, we agree that methods that also anticipate post-SFT importance should be compared. ExPO [1] does not fit this setting, because its authors report that extrapolating to its SFT model usually cause model collapse (Section 4.3 of [1]). We therefore add EarlyBERT [2] and NeFT [3], which also anticipate parameter importance from the early stage of training. The results are as follows:

Table 1 (revised, excerpt). Mistral-7B, zero-shot; mean ± std over 5 runs.

MethodLR TTALR PTANLU TTANLU PTAMR TTAMR PTA
EarlyBERT [2]96.37±0.2959.42±0.1185.01±0.0461.59±0.1295.84±0.2157.29±0.08
NeFT [3]97.62±0.2160.38±0.1683.27±0.2761.24±0.0997.95±0.0660.18±0.08
Ours (N)98.51±0.0960.53±0.0586.31±0.1362.02±0.0798.79±0.1460.13±0.10
Ours (C)100.00±0.0062.64±0.0887.55±0.0463.95±0.11100.00±0.0061.48±0.06

As shown in the table, EarlyBERT and NeFT also anticipate post-SFT importance, yet fall behind our methods. We believe this is because their extrapolation relies on learned gates or weight changes rather than interpretability, so it does not reach the neurons that truly matter for the target task.

Moreover, we apologize for the misunderstanding of Figure 11 (Figure 12 in the revision), and please allow us to explain it. The figure does not compare first- and second-order approximations: both curves include the second-order term and differ only in how it is computed. "w/o 2nd-order" uses the finite difference in Eq. 12, whereas "w/ 2nd-order" computes the exact derivative. Figure 11 shows that finite difference approximation does not lose much fidelity.

To this end, we have made the following paper changes (please refer to our updated PDF):

  • Table 1 now includes EarlyBERT and NeFT;
  • The caption of Figure 12 now explains that both curves include the second-order term.

References:

[1] Model Extrapolation Expedites Alignment. ACL 2025.

[2] EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets. ACL 2021.

[3] Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model. COLING 2025.


AuthorsOpenReview ↗

Response to Reviewer r6RE (Q2-Q3)


Q2: "Information leakage from post-SFT checkpoints and inconsistent baselines: The claim of operating 'using only pre-SFT parameters' is undermined by the hyperparameter selection in Appendix F.1 and F.2, where the extrapolation scalar KK is tuned or learned against the fully fine-tuned model." "Were the task-specific KK intervals in Tables 1–2 chosen using metrics derived from the fully fine-tuned checkpoint (as described in App. F.1)? If so, how does the method perform when KK is fixed a priori or selected using only base/probing validation loss?" (Weakness 2 + Question 2)

We sincerely thank you for raising this concern about potential leakage.

KK is randomly sampled from the fixed interval [0,100][0, 100], and Appendix F.1 and F.2 are post-hoc analyses. Specifically, Appendix F.1 and F.2 were conducted after SFT to analyze how the sampling interval of KK affects the estimation, and they are not used to choose KK in the main experiments.

To this end, we have made the following paper change (please refer to our updated PDF):

  • Section 5.1 now states: "For all tasks, we randomly sample KK from [0,100][0, 100] ten times and average the resulting causal effects."

Q3: "What are the empirical results if attribution patching is calculated directly on the extrapolated checkpoint θ+KΔθ\theta + K\Delta\theta using standard passes, bypassing the Taylor expansion/trapezoidal approximation?" (Question 1)

We sincerely thank you for this question.

As clarified in Q1, θ+KΔθ\theta + K\Delta\theta is not an estimate of the post-SFT model; attribution patching computed directly on it deviates greatly from the post-SFT ranking, as shown in the new Figure 10(d).

Please allow us to briefly recap why this computation is not meaningful for our method. First, it is not what we approximate: we approximate S(θI)S(\theta^{I}) by S(θ+KΔθ)S(\theta + K\Delta\theta), not ΔE(θI)\Delta E(\theta^{I}) by ΔE(θ+KΔθ)\Delta E(\theta + K\Delta\theta). Second, among the three terms of Eq. 11, ΔE(θ)\Delta E(\theta) and S(θ)S(\theta) are both computed on the actual pre-SFT model, and θ+KΔθ\theta + K\Delta\theta is only used to read the slope in the third term. As noted in footnote 2 (Section 3), extrapolating the post-SFT parameters along a fixed gradient direction is intractable, so θ+KΔθ\theta + K\Delta\theta cannot be treated as the post-SFT model.

Nevertheless, to verify this empirically, we apply attribution patching directly to θ+KΔθ\theta + K\Delta\theta and locate its Top-10 components in the ranking of the fully fine-tuned model (Figure 10(d) in Appendix F.4). About half of its Top-10 components fall hundreds of ranks lower. Therefore, θ+KΔθ\theta + K\Delta\theta cannot directly serve as the post-SFT model.

We summarize paper change as following: (please refer to our updated PDF):

  • Appendix F.4 adds Figure 10(d), which apply attribution patching directly to θ+KΔθ\theta + K\Delta\theta.

AuthorsOpenReview ↗

Response to Reviewer r6RE (Q4-Q5)


Q4: "Additionally, the experimental setup exhibits inconsistencies: in Table 2, the full fine-tuning baseline achieves only 15.66% on Multi-digit Reasoning (near random guessing for a 7-choice task), while elsewhere the post-SFT model is treated as a near-perfect ground truth reference." "Why does full fine-tuning achieve only 15.66% accuracy on Multi-digit Reasoning (MR) in Table 2? What training hyperparameters (learning rate, schedule) were used, and does the proposed method maintain its performance margin over a properly converged full-SFT baseline?" (Weakness 2 + Question 3)

We sincerely thank you for pointing out this result.

The 15.66% is the actual result of full fine-tuning, which most likely had not converged under our unified hyperparameters; trained longer, it reaches 65.59%, still below Probing (87.16%) and Ours (98.79%), and we have updated Table 2 accordingly.

Notably, w/o Localization reaches only 15.66% at the neuron level but 71.83% at the component level, because the neuron level updates all parameters, whereas the component level uses LoRA with a uniform rank of 8. To address your concern, we report the accuracy on each Arithmetic subtask:

w/o Localization on each Arithmetic subtask (TTA, %).

Level2-digit3-digit4-digit5-digitMR (avg.)
Neuron-level (full fine-tuning)23.1614.8315.339.3215.66
Component-level (rank-8 LoRA)81.6976.5068.5960.5471.83

As shown in the table, full fine-tuning performs poorly on every subtask. We suspect that, as you suggested, it has not converged: Mistral-7B has weak arithmetic ability before SFT, so full fine-tuning may need clearly more epochs or a larger learning rate. However, for fairness, all methods and tasks share the same hyperparameters (learning rate 1×10−51\times10^{-5}, 3 epochs; Section 5.1), under which LoRA and localized partial tuning converge faster.

To verify this, we retrain full fine-tuning for 10 epochs with a learning rate of 5×10−55\times10^{-5}. The results are as follows:

Neuron-level MR (TTA, %).

MethodEpochsLearning rateMR
w/o Localization (original)31×10−51\times10^{-5}15.66
w/o Localization (retrained)105×10−55\times10^{-5}65.59
Random31×10−51\times10^{-5}43.77
Static31×10−51\times10^{-5}55.83
Probing31×10−51\times10^{-5}87.16
Ours31×10−51\times10^{-5}98.79

As shown in the table, the retrained full fine-tuning surpasses Random and Static, but still falls behind Probing and Ours, which use fewer epochs and a smaller learning rate. Therefore, we believe that for tasks that are harder to adapt to, localization both accelerates convergence and improves SFT performance.

To this end, we have made the following paper change (please refer to our updated PDF):

  • In Table 2, the neuron-level MR of w/o Localization is updated from 15.66% to 65.59%, and the caption notes its training setting (10 epochs, learning rate 5×10−55\times10^{-5}).

Q5: "Eq. 12 uses a finite difference at θ′\theta'. In practice, does evaluating the expectation over 10 sampled KK values require forward/backward passes at each θ+KΔθ\theta + K\Delta\theta? Please clarify the exact forward/backward pass count per candidate component." (Question 4)

We sincerely thank you for this question on the computational cost.

Yes: each sampled KK takes 2 forward passes and 1 backward pass at θ+KΔθ\theta + K\Delta\theta, shared by all candidate components, and the finite difference in Eq. 12 keeps each KK under 100 s.

Please allow us to first explain Eq. 12. The finite difference avoids computing the second-order derivative and serves as an approximation for acceleration. Figure 11 (Figure 12 in the revision) confirms its effectiveness: compared with the exact derivative, it increases the Relative Hamming Distance to the fully fine-tuned circuit by only 3–10 points across all scales.

As for the pass count, each sampled KK requires 2 forward passes and 1 backward pass, which are shared by all candidate components. Therefore, taking Mistral-7B as an example, 10 sampled KK values take 20 forward and 10 backward passes in total. With the finite difference, the whole estimation takes less than 1000 (100 * 10) s, whereas computing the second-order derivative exactly takes more than 8000 (800 * 10) s. This is why Section 4.2 recommends the finite difference for practical SFT.

Moreover, Table 4 in Appendix F.2 shows that a single sampled KK or a KK predicted by probing can replace the 10-sample average with similar TTA. Since they compute KK only once, they need only 2 forward passes and 1 backward pass in total.


We hope these responses address your concerns. If you have any further questions, we would be glad to discuss them with you.

Dear Reviewer r6RE:

There are only two days left until the discussion ends, and we would really appreciate it if you could let us know whether your concerns have been addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response about Paper Revisions

Dear Chairs and Reviewer r6RE, VGXL, and 66JQ:

We sincerely thank all three reviewers for unanimously recognizing our substantive contribution to bridging mechanistic interpretability and LLM post-training, and for their valuable suggestions on the theoretical reliability, experimental clarity, and readability of our paper. Following these suggestions, we have thoroughly revised the paper (please see the updated PDF, where the main changes are marked in red). The main changes are as follows:

  1. Abstract and Introduction. We revised the research question to make clear that we predict the post-SFT circuit in order to improve SFT performance (Reviewer 66JQ Q1). We also added references showing that the post-SFT interpretability state is predictable under certain conditions (Reviewer VGXL Q1).
  2. Experiments. We added two extrapolation-like methods, EarlyBERT and NeFT, as baselines in the main results (Reviewer r6RE Q1), and two adaptive-rank LoRA methods, AdaLoRA and EVA, to the ablation study (Reviewer VGXL Q2). We also reorganized the results section and completed the experimental details, such as the sampling of KK and the metrics and settings in each caption (Reviewer r6RE Q2, Q4; Reviewer VGXL Q4; Reviewer 66JQ Q4).
  3. Discussion. We added a new Discussion section on how the circuit changes during SFT, and how accurate our prediction is, for tasks of different adaptation difficulty. This analysis unifies our work with prior studies and shows the practical value of our method (Reviewer VGXL Q3; Reviewer 66JQ Q2).
  4. Appendix. We added the corrupted input of each example (Reviewer 66JQ Q5), the prediction obtained by applying attribution patching directly to θ+KΔθ\theta + K\Delta\theta (Reviewer r6RE Q3), and clarifications of several experimental settings (Reviewer r6RE Q1).
  5. Writing. We proofread the whole paper, condensed the writing, and fixed typos (Reviewer 66JQ Q4, Q7).

To summarize, our work proposes a predictive localization method that overcomes the post-hoc nature of interpretability and bridges mechanistic interpretability with practical LLM post-training. It improves the performance of locating-then-tuning for SFT and scales from 7B to 30B models at a low computational cost. Moreover, the new Discussion shows that it applies to most SFT tasks (Tiers 1–2), confirming its practical value.

Once again, we sincerely thank all three reviewers for their constructive suggestions, which have substantially improved our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.