- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents an improvement for "locating-then-tuning" paradigm. They propose to decompose the training process into two theoretical steps: isolating direction and extrapolating distance. Then, using these two properties, they define two locate-then-update pipelines (neuron level and component level) and test their effectiveness against several existing methods.
Their component-level method shows clear improvement over existing methods, which is evaluated across several tasks and models on both task improvement and retention of existing capabilities.
Overall, this is a nice and interesting paper. With some clarifications about my questions below and some improvements to the presentation, I would be happy to see it accepted. If addressed, I would be happy to further raise my score to a clear accept.
Critical strengths
The paper studies an interesting topic: changes in model mechanisms during FT, and how to identify components that will be influential after FT, in order to improve locate-then-update methods. These questions connect interpretability research with actionable training decisions, which show the potential of interpretability research, and specifically task and mechanism localization.
Their component-level method shows clear improvement over existing methods, which is evaluated across several tasks and models on both task improvement and retention of existing capabilities.
Critical weaknesses
-
The motivation of this paper is a bit unclear. The authors pose this question as their main research question: how can we predict the neurons responsible for a target task in an ideal post-SFT model using only the pre-SFT model parameters? However, this is not the question they later answer, as we see no comparison of empirical distance or component overlap between their predictions and the set of critical components that manifested after SFT. As far as I understand, the paper aims to find the best set of components to update in order to achieve the best task performance (TTA/PTA). As they compare against other locate-then-update methods. I suggest making this clearer in the intro and in the results section.
-
The authors cite Chen et al [1] for the claim that circuits change significantly across FT. However, the results for Chen et al are not reproduced in other cases [2], which show that FT largely preserve and enhance existing mechanisms. Did the authors verify this claim on their tasks and mechanisms? I would be happy to see this empirical result in the paper (even in an appendix with a brief mention in the main text), as this is an active discussion in the literature which has contradicting evidence. Moreover, citing Prakash et al [2] for “mechanisms undergo drastic transformations during SFT” seems like a wrong citation, since it does not match their main claims.
-
What are the base performances of the tested models on the tasks? This heavily influences FT results and how we view the findings, as there is a difference in what we expect FT to change in model mechanisms when the base performance is 30% or 99%.
-
The results section of this paper is a bit dense and difficult to follow, I would be happy to see it a bit more digested. Additionally, the captions in this section are short and missing some details: In table 1, what metric does it show?, In table 2: which task is shown? Is this an avg? There is a single line of text between tables 1 and 2 - I suggest rearranging this.
-
Some examples of the tested tasks would greatly help me understand the setup of the paper.
-
The tested tasks are not of fixed length and fixed positions, as is usually the case when using attribution patching. Could the authors elaborate on this point, and how they overcome it?
[1] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training. Chen et al. [2] FINE-TUNING ENHANCES EXISTING MECHANISMS: A CASE STUDY ON ENTITY TRACKING. Prakash et al.
Minor:
- Typo in the capitalization of Llama-2-13B, line 77
- The paper keeps mentioning “neurons/components” - this makes it unclear if the paper only accounts for neurons, attention heads, or both during the localization analysis. I suggest making this point clearer and removing the “/”.
Questions for authors
See section above.
5 comments
Response to Reviewer 66JQ (Q2)
Q2: "The authors cite Chen et al [1] for the claim that circuits change significantly across FT. However, the results for Chen et al are not reproduced in other cases [2], which show that FT largely preserve and enhance existing mechanisms. Did the authors verify this claim on their tasks and mechanisms? I would be happy to see this empirical result in the paper (even in an appendix with a brief mention in the main text), as this is an active discussion in the literature which has contradicting evidence. Moreover, citing Prakash et al [2] for 'mechanisms undergo drastic transformations during SFT' seems like a wrong citation, since it does not match their main claims." (Weakness 2)
We sincerely thank you for pointing out this active discussion and for the valuable suggestion.
In short, we have verified this on 13 tasks, and both findings hold depending on adaptation difficulty: circuits are largely preserved for tasks the base model has mastered, as Prakash et al. found, but change substantially for harder tasks, as Chen et al. found; we have added this experiment to the paper as a new Discussion section.
To study this, we rank 13 tasks by adaptation difficulty, roughly measured by the base model's and rescaled to : the hardest tasks are those that the base model still fails after many shots. On Mistral-7B, we tune each task and report five metrics:
- CD (Circuit Distance, ): a continuous metric that mainly reflects changes in edge weights.
- RHD (Relative Hamming Distance, %): a discrete metric that only counts added and removed edges.
- Static / Ours / Full-Param: TTA (%) when tuning with the base-model circuit as the localization, with our extrapolated localization, and with full-parameter SFT, respectively.
The results are as follows:
| Tier | Task | Difficulty | CD | RHD | Static | Ours | Full-Param |
|---|---|---|---|---|---|---|---|
| 1 | Docstring | 0.05 | 0.14 | 5 | 100 | 100 | 100 |
| 1 | COPA | 0.08 | 0.12 | 4 | 91 | 91 | 91 |
| 1 | IOI | 0.16 | 0.18 | 9 | 100 | 100 | 100 |
| 1 | Greater-than | 0.18 | 0.26 | 7 | 94 | 95 | 95 |
| 1 | Gender | 0.19 | 0.39 | 11 | 100 | 100 | 100 |
| 2 | BOOL | 0.21 | 0.93 | 16 | 98 | 100 | 99 |
| 2 | RTE | 0.24 | 1.63 | 18 | 83 | 87 | 86 |
| 2 | WinoGrande | 0.46 | 3.87 | 24 | 72 | 76 | 74 |
| 2 | ARC | 0.66 | 4.38 | 26 | 84 | 91 | 83 |
| 2 | Arithmetic | 0.84 | 4.54 | 23 | 98 | 100 | 98 |
| 3 | TOFU | 0.91 | 5.97 | 36 | 93 | 94 | 98 |
| 3 | Current Events | 0.95 | 6.35 | 53 | 41 | 38 | 50 |
| 3 | SMolInstruct | 1.00 | 7.56 | 44 | 82 | 81 | 88 |
As shown in the table, the tasks fall into three tiers:
- Tier 1: mastered tasks. The base model already masters these tasks, so SFT barely changes their performance or circuit. Hence, the base-model circuit is already predictive, and Static performs as well as Ours. This agrees with findings that SFT barely changes the circuits of skills the base model has mastered [1–3].
- Tier 2: nearly mastered tasks. The base model has the needed sub-skills but must learn a pattern or knowledge, so SFT quickly learns the output format or small skills and slightly updates stored knowledge. This is also the most common case of SFT in practice. Here, the circuit structure changes little, but its edge ranks change sharply, so our extrapolation clearly beats static prediction. This agrees with findings that typical SFT changes circuit weights substantially, making prediction from the base model inaccurate [4, 5].
- Tier 3: novel tasks. The base model has never seen these tasks and lacks most of the domain knowledge, so SFT must form new components and functional heads. The circuit thus gains and loses many edges and nodes, and almost every prediction method fails. This agrees with findings that circuits change almost entirely when models acquire new knowledge or algorithms [6, 7].
Regarding the citation of Prakash et al., we agree that it does not support 'drastic transformations': their finding corresponds to our Tier 1.
Finally, we have made the following paper changes:
- Section 6 (Discussion) now presents this experiment.
- Section 2 now cites Prakash et al. for the preservation of mechanisms on mastered tasks.
References:
[1] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking.
[2] Neuroplasticity and Corruption in Model Mechanisms: A Case Study of Indirect Object Identification.
[3] Retrieval Head Mechanistically Explains Long-Context Factuality.
[4] Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis.
[5] Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training.
[6] How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training.
[7] Progress Measures for Grokking via Mechanistic Interpretability.
Response to Reviewer 66JQ (Q3-Q7)
Q3: "What are the base performances of the tested models on the tasks? This heavily influences FT results and how we view the findings, as there is a difference in what we expect FT to change in model mechanisms when the base performance is 30% or 99%." (Weakness 3)
We sincerely thank you for and agree with this insightful question.
In short, the base performances are reported in Figure 7 (Figure 8 in the revision), and they indeed shape our findings: the lower the base performance, the more extrapolation (a larger ) is needed, and the more the circuit changes during SFT.
Please allow us to note that Appendix F.1 reports the base performance of Mistral-7B on each task in the legend of Figure 7 (Figure 8 in the revision). We list it again below:
Base performance of Mistral-7B before SFT (TTA, %).
| BOOL | Arithmetic | Induction | WinoGrande | SST-2 | Gender | IOI | Docstring |
|---|---|---|---|---|---|---|---|
| 0.0 | 0.0 | 32.5 | 55.8 | 56.9 | 61.5 | 63.8 | 87.5 |
As shown in Appendix F.1, the base performance is correlated with the suitable range of . For tasks with near-zero performance (BOOL and Arithmetic), the overlap with the circuit from SFT model stays high over a broad range , whereas for tasks with higher performance (Docstring, IOI, and Gender), the suitable concentrates in . In other words, the lower the base performance, the more extrapolation is needed. Moreover, as shown in our response to Q2, the base performance, also somehow determines how much the circuit changes during SFT and how accurate our prediction is.
Q4: "The results section of this paper is a bit dense and difficult to follow, I would be happy to see it a bit more digested. Additionally, the captions in this section are short and missing some details: In table 1, what metric does it show?, In table 2: which task is shown? Is this an avg? There is a single line of text between tables 1 and 2 - I suggest rearranging this." (Weakness 4)
We sincerely thank you for these helpful suggestions.
We have revised the results section for readability: the captions of Tables 1–2 and Figures 2–4 now state the metrics, tasks, and settings, and the layout has been rearranged.
Moreover, we have made the following paper changes:
- Table 1: the caption now states the model, evaluation, metrics, and tasks in each column, and that the values are mean ± std over 5 runs.
- Table 2: the caption now states that all values are TTA (%), and that each value averages the tasks in its domain.
- Figures 2–4: the captions now state the tasks, metrics, and settings.
- Section 5: Section 5.2 is split into three headed paragraphs.
Q5: "Some examples of the tested tasks would greatly help me understand the setup of the paper." (Weakness 5)
We sincerely thank you for this suggestion. Please allow us to remind that Table 3 in Appendix D already lists the input prompt and an example for each task, and Figure 12 (Figure 13 in the revision) shows the circuit graphs of different localizations as examples. To make our setup even clearer, we have made the following paper change: Table 3 now also shows the corrupted input of each example, which replaces only one or two tokens of the clean input to change the answer.
Q6: "The tested tasks are not of fixed length and fixed positions, as is usually the case when using attribution patching. Could the authors elaborate on this point, and how they overcome it?" (Weakness 6)
We sincerely thank you for raising this important point. As stated in footnote 4 (Section 5.1) and Appendix D, we constrain the label of each dataset to a single token, and we reformulate the inputs into question-answering templates when necessary. Therefore, the metric of attribution patching is always computed at the final position, where all samples predict a label with the same semantics (e.g., Yes/No or an option letter). Nevertheless, we agree that this restriction limits the tasks we can study: current mechanistic interpretability mainly analyzes next-token prediction, which we discuss as a limitation in Section 7. We are already working on mechanistic interpretability for multi-token outputs.
Q7: "Typo in the capitalization of Llama-2-13B, line 77." "The paper keeps mentioning 'neurons/components' - this makes it unclear if the paper only accounts for neurons, attention heads, or both during the localization analysis. I suggest making this point clearer and removing the '/'." (Minor)
We sincerely thank you for pointing out these issues.We have proofread the paper again and corrected these expressions and typos; the changes are shown in red in the updated PDF. Thank you again for your careful review and valuable suggestions!
We hope these responses address your concerns. If you have any further questions, please feel free to discuss them with us at any time.
Response to Reviewer 66JQ (Q1)
We sincerely thank Reviewer 66JQ for recognizing our work, especially that it connects interpretability research with actionable training decisions. We also thank you for the many suggestions to improve the readability and theoretical robustness of our paper. Following your suggestions, we have carefully revised and supplemented the manuscript and verified the issues you raised (please see the updated PDF). Below, we respond to each point.
Q1: "The motivation of this paper is a bit unclear. The authors pose this question as their main research question: how can we predict the neurons responsible for a target task in an ideal post-SFT model using only the pre-SFT model parameters? However, this is not the question they later answer, as we see no comparison of empirical distance or component overlap between their predictions and the set of critical components that manifested after SFT. As far as I understand, the paper aims to find the best set of components to update in order to achieve the best task performance (TTA/PTA). As they compare against other locate-then-update methods. I suggest making this clearer in the intro and in the results section." (Weakness 1)
We sincerely thank you for this insightful comment, and we agree that our goal should be stated more clearly.
In short, we have revised the main research question to predicting the critical neurons for better SFT performance; the comparison between our prediction and the post-SFT circuit is provided in the appendix.
Following your suggestion, we have revised the main research question in the introduction from predicting the critical neurons of the ideal post-SFT model to predicting them for better SFT performance. Moreover, please allow us to note that Appendix F.4 and Appendix H compare our predicted circuit with the post-SFT circuit on several interpretability metrics: the ranking of component causal effects (Figure 10), Top@50 overlap and KL divergence (Tables 5–6), and Relative Hamming Distance (Figure 12). Due to the page limit, we placed them in the appendix, as we considered the main results (Table 1) and the ablation study (Table 2) more important for the main text.
To this end, we have made the following paper changes (please refer to our updated PDF):