- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates the effectiveness of supervised fine-tuning through AND–OR interaction decompositions of input-word effects on target-token confidence. It tracks interactions that are removed, continuously preserved, or newly emerged during fine-tuning, and characterizes them using interaction order, agreement with another fine-tuned model, and cancellation between positive and negative effects. Experiments use LoRA on five instruction/chat-tuned models across GoEmotions, Unilaw-R1-Data, and Databricks-Dolly-15k. The reported trajectories show early removal of many high-order, mutually cancelling interactions, followed by the accumulation of new interactions with similar properties. Preserved interactions tend to be lower-order and more consistent across models. Downstream evaluations in four model–dataset settings achieve their best reported performance at early checkpoints. The authors interpret these findings as a brief denoising phase followed by overfitting and suggest interaction-based training monitoring. The actual interaction analysis concerns the first response token, and an operational early-stopping criterion is not evaluated.
Critical strengths
- A systematic empirical observation across multiple settings. The paper examines interaction dynamics across five models and three datasets rather than relying on a single before/after example. The recurring distinction between early changes and later accumulation of interactions is potentially useful for developing hypotheses about adaptation dynamics. Tracking preserved, removed, and emerged interactions separately makes the observations more informative than aggregate interaction counts alone.
- Connection to task-level outcomes. Appendix D.3 evaluates downstream performance rather than relying exclusively on interaction-derived metrics. For example, Qwen2.5-7B-Instruct’s GoEmotions Micro-F1 rises from 0.267 before SFT to 0.654 at step 1000, then falls to 0.593 at step 4000. This correspondence gives the proposed explanation an empirical connection to model utility, although it does not establish that the interaction changes explain the performance changes.
Critical weaknesses
- The measurements do not sufficiently establish the “noise” and “inference backbone” interpretations. The generalizability measure in Eqs. (7)–(9) measures agreement with another model, not generalization to unseen examples; useful model-specific computations can fail this criterion. Similarly, cancellation in Eq. (10) does not establish that the corresponding interactions are noise: terms can cancel on the complete input while remaining necessary to describe behavior under different input conditions. Faithful reconstruction of masked-input outputs does not resolve these interpretation questions. There is also a concrete issue in Section 3.3: Eq. (2) and Appendix A include a baseline contribution, but Eq. (11) applies the inverse log-odds transformation to interaction sums without that baseline. Unless an unstated centering convention is used, these are not the reconstructed target probabilities. Consequently, Figure 4 does not yet provide convincing evidence that the preserved interactions are sufficient for inference.
- The experimental scope and controls do not support the breadth of the SFT and early-stopping conclusions. All experiments adapt already instruction/chat-tuned models using one reported LoRA training recipe, with rank 8 by default and 30 training epochs (Appendix E.1). Interaction analysis uses only the first response token and selected input words (Appendix E.4). This leaves open whether the observations concern response-opening or formatting behavior rather than the acquisition of useful answer content. Longer training on the same dataset also does not establish the value of adding more distinct training data. The downstream tables provide no uncertainty estimates or repeated-seed results; some differences are very small, such as Dolly ROUGE-L of 0.367 at step 300 versus 0.366 at step 1000. Finally, Appendix G explicitly leaves the stopping threshold and its generalization benefit to future work. The reported trajectories are therefore insufficient to establish the broad reconciliation of SFT’s effectiveness or a validated early-stopping method.
Questions for authors
- Calculation clarification: Was the baseline contribution (b=v(x_{\emptyset})) included when computing the probabilities and losses in Figure 4? Eq. (11) appears to omit it. How do the results compare with the full-model and baseline-only losses when the baseline is retained consistently?
- Extraction clarification and robustness: What exact significance threshold (\tau), masking baseline, input-word selection procedure, and reference-model checkpoint were used? Were the selected words fixed across training checkpoints? Evidence that the interaction categories are stable under these choices would materially strengthen the interpretation of emergence and removal.
- Evidence needed for the central interpretation: What evidence distinguishes low-agreement or strongly cancelling interactions from useful model-specific computations? In particular, do the main conclusions persist when analyzing later, task-relevant response tokens rather than only the first token?
- Evidence needed for the practical claim: How were the phase boundaries determined without using downstream test outcomes, and does interaction-based monitoring improve checkpoint selection over validation-loss-based early stopping? Appendix G treats this benefit as untested, whereas the conclusion suggests a method for identifying the stopping point.
4 comments
Response to Reviewer amFg (Part 2/4)
Q2: Ask for details about the threshold, masking baseline, word selection, reference checkpoint, and fixed variables
The experimental scope and controls do not support the breadth of the SFT ... selected input words (Appendix E.4).
Extraction clarification and robustness: What exact significance threshold ... the interpretation of emergence and removal.
A: Thank you for the question. We follow [cite1, cite2, cite3] to identify salient interactions using
where in the main experiments. An interaction is identified as salient when its absolute strength is larger than . Please see Appendix F and Algorithm 1 for the interaction extraction procedure.
For masking, we replace the embedding of a masked token with the mean embedding over the model vocabulary, which represents the no-information baseline described in Section 3.1. Each selected word is treated as one input variable. If a word contains multiple tokenizer pieces, all pieces are masked together. For each prompt, we randomly select ten valid words that can be mapped to tokenizer tokens and retain their original order. The same analysis samples and the same selected words are used at every checkpoint of the same training run. Please see Appendix E.4 for details.
For interaction generalizability, we use a fixed LLM as the baseline DNN throughout the training process. Please see Appendix E.2 for details.
We conducted new experiments using . The table reports the average number of interactions for each model on GoEmotions. The percentages show how removed or newly acquired interactions are distributed between the two stages.
| Model | Removed in the first stage | Acquired in the first stage | Removed in the second stage | Acquired in the second stage | |
|---|---|---|---|---|---|
| Llama-2-7B-Chat | 0.005 | 559.4 (87.3%) | 89.4 (14.6%) | 81.5 (12.7%) | 522.9 (85.4%) |
| Llama-2-7B-Chat | 0.010 | 453.8 (90.7%) | 53.0 (11.4%) | 46.8 (9.3%) | 412.2 (88.6%) |
| Llama-2-7B-Chat | 0.020 | 313.2 (93.6%) | 25.4 (8.9%) | 21.3 (6.4%) | 261.5 (91.1%) |
| Qwen2.5-3B-Instruct | 0.005 | 629.1 (88.2%) | 95.5 (11.8%) | 84.1 (11.8%) | 716.6 (88.2%) |
| Qwen2.5-3B-Instruct | 0.010 | 518.9 (91.8%) | 54.4 (8.3%) | 46.0 (8.2%) | 604.8 (91.7%) |
| Qwen2.5-3B-Instruct | 0.020 | 361.9 (95.0%) | 23.9 (5.2%) | 19.1 (5.0%) | 434.4 (94.8%) |
| Qwen2.5-7B-Instruct | 0.005 | 514.0 (91.3%) | 54.1 (10.0%) | 49.0 (8.7%) | 487.8 (90.0%) |
| Qwen2.5-7B-Instruct | 0.010 | 393.7 (93.6%) | 30.4 (7.8%) | 27.1 (6.4%) | 357.6 (92.2%) |
| Qwen2.5-7B-Instruct | 0.020 | 254.4 (95.7%) | 13.8 (6.2%) | 11.3 (4.3%) | 209.3 (93.8%) |
| Llama-3-8B-Instruct | 0.005 | 320.6 (85.0%) | 67.7 (18.6%) | 56.5 (15.0%) | 296.6 (81.4%) |
| Llama-3-8B-Instruct | 0.010 | 223.2 (86.4%) | 43.9 (17.2%) | 35.1 (13.6%) | 211.1 (82.8%) |
| Llama-3-8B-Instruct | 0.020 | 128.8 (87.9%) | 24.1 (16.5%) | 17.8 (12.1%) | 121.5 (83.5%) |
For every model and every value of , 85.0%--95.7% of the removed interactions are removed in the first stage, whereas 81.4%--94.8% of the newly acquired interactions are acquired in the second stage.
We also varied the number of selected words from 7 to 10 for Qwen2.5-3B-Instruct on GoEmotions. By step 1000, the model has removed 95.5%--98.9% of the interactions present before SFT, while newly acquired interactions account for only 2.3%--11.9% of the initial number. After model training, newly acquired interactions increase to 35.0%--77.7% of the initial number, while only 0.1%--2.4% of their effects remain after positive and negative cancellation. Therefore, changing either the significance threshold or the number of selected words does not change the main result: the first stage mainly removes interactions, whereas the second stage mainly acquires new interactions whose effects largely cancel each other. We will add these settings and robustness results to the revised appendix.
Response to Reviewer amFg (Part 3/4)
Q3: Ask whether low-agreement or cancelling interactions may be useful model-specific computation and whether the result holds for later tokens
The measurements do not sufficiently establish the “noise” ... resolve these interpretation questions.
Interaction analysis uses only the first response token and ... acquisition of useful answer content.
Evidence needed for the central interpretation: What evidence ... than only the first token?
A: Thank you for the question. We use target-token prediction loss to test whether different types of interactions help prediction. If a group of interactions helps the model predict the target token, using that group should yield lower loss.
First, Figure 4 and Section 3.3 provide this direct comparison. On GoEmotions, the losses obtained using Emerged-I versus Emerged-II interactions are 1.613 versus 2.444 for Llama-2-7B-Chat, 2.413 versus 4.095 for Qwen2.5-3B-Instruct, and 1.932 versus 4.808 for Qwen2.5-7B-Instruct. Preserved interactions also produce lower loss than Emerged-II interactions. In these examples, interactions retained throughout SFT or acquired early predict the target token better than interactions acquired late.
Second, Figures 2 and 3 in Section 3.2.2 show that late-acquired interactions are mainly high-order, are rarely learned by different LLMs, and have positive and negative effects that largely cancel each other. Among newly acquired interactions of order , only 0.08%--0.27% are learned by different LLMs, and the cancellation ratios of the remaining interactions reach 98.69%--99.97%. Their total strength at each order also grows approximately with the number of possible order- word combinations, .
Third, Appendix D.3 and Tables 1 and 2 show that the best downstream performance is reached during the early stage in all four reported cases. Once performance stops improving, high-order newly acquired interactions continue to increase, while fewer of them are shared by different LLMs and their positive and negative effects cancel more strongly.
Thus, late-acquired interactions yield higher prediction loss, are rarely learned by different LLMs, largely cancel each other, and continue to accumulate after test performance stops improving. We will therefore describe them as noise-like interactions associated with overfitting.
Our interaction analysis does not depend on the position of the target token. For a later response token, we include the preceding response tokens in the context and then compute how interactions between the input words affect the prediction of this target token. Thus, the same analysis can be applied to any task-relevant token in the response.
We also conducted experiments on later response tokens using Gemma-3-4B-IT on Databricks-Dolly-15k. For each response, we randomly skip some initial tokens and use a subsequent token as the prediction target. By the end of the denoising stage, the model has removed 317.9 of the 327.4 interactions present before SFT, while acquiring only 20.4 new interactions. During the subsequent overfitting stage, the number of newly acquired interactions increases to 351.8, and their total strength increases from 3.82 to 117.14. Meanwhile, the proportion of their effects remaining after positive and negative cancellation decreases from 14.5% to 0.3%. Therefore, the same phenomenon is observed for later response tokens: SFT first removes many interactions and later acquires many new interactions whose positive and negative effects largely cancel each other. We will add these results and clarify the target-token setting in the revised paper.
Response to Reviewer amFg (Part 4/4)
Q4: Ask how the phase boundary is determined without test outcomes and whether it improves on validation-loss early stopping
The downstream tables provide no uncertainty ... early-stopping method.
Evidence needed for the practical claim: How ... for identifying the stopping point.
A: Thank you for the question. Determining the phase boundary does not require downstream test outcomes. Before SFT, we sample a small fixed subset from the training data and periodically extract its interactions during training. The first stage ends when the rapid removal of pre-SFT interactions levels off. After this point, newly acquired interactions continue to increase, fewer of them are generalizable, and their positive and negative effects cancel more strongly. Thus, the phase boundary can be determined directly from the interaction changes on the sampled training data. The test set is used only for the final evaluation and is not used to determine the boundary or select the checkpoint.
We can use the proportion of generalizable interactions, , together with the validation loss to determine the stopping point.
In practice, early stopping can be triggered when drops below a given threshold, especially when the validation loss no longer improves. This indicates that SFT has learned many non-generalizable interactions, and stopping at this point can prevent further overfitting. In some cases, the validation loss may still decrease slowly even though has already dropped substantially. In this case, we recommend stopping the current training run and improving the training data, for example through data augmentation, before continuing SFT.
Specifically, we divide the training set into subgroups, with each subgroup representing one type of training sample. For GoEmotions, we construct the subgroups according to emotion categories and periodically compute for each subgroup during SFT. If becomes very small for one subgroup, this indicates that the model has learned many non-generalizable interactions from this subgroup, and training on this subgroup should be stopped before these interactions further accumulate.
Compared with monitoring validation loss over the entire dataset, monitoring within individual subgroups provides a more fine-grained signal for identifying overfitted samples. On GoEmotions, we divide the training set into 12 subgroups and apply this interaction-based early-stopping criterion. It reduces the training cost by approximately 30% while maintaining the test performance. We will add this early-stopping strategy and the corresponding experimental results to the revised paper.
Response to Reviewer amFg (Part 1/4)
Thank you for your detailed technical review. We respond to each question below.
If any answer is unclear or you have further questions, please let us know so we can respond before the deadline.
Q1: Ask whether Figure 4 includes b=v(x∅) and how it compares with full-model and bias-only losses
A: Thank you for the question. The baseline contribution b=v(x∅) was not included in the previous Figure 4. We followed your suggestion and conducted new experiments with the bias term added to Eq. (11), including the full model, full interaction reconstruction, and bias-only baseline. The table below shows the target-token prediction losses of Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on GoEmotions. Lower loss indicates better target-token prediction.
The full interaction reconstruction achieves almost the same loss as the full model. Moreover, preserved and Emerged-I interactions consistently achieve lower loss than Emerged-II interactions.
These results support our conclusion in Section 3.3: interactions preserved during SFT and interactions learned in the early stage contribute more to target-token prediction, whereas interactions learned in the later stage contribute much less. We will update Eq. (11) and Figure 4 with these new results.