acceptodds
Under review as a conference paper at ICLR 2027

Reconciling Contradictory Views on the Effectiveness of SFT in LLMs: An Interaction Perspective

Abstract

This paper explores a scientific question in supervised fine-tuning (SFT): why SFT is broadly effective for small-scale deep neural networks, yet can produce inconsistent or even detrimental effects when applied to large language models (LLMs). Recent advances in interaction-based explanations suggest that interactions between words/tokens provide a faithful metric for quantifying the inference patterns encoded by LLMs. We find that the evolution of interactions during SFT can effectively explain the inconsistent effectiveness of SFT for LLMs. Specifically, we find that (1) SFT primarily removes noise-like interactions, while rarely acquiring reliable new interactions. (2) This denoising stage is extremely brief, after which continued fine-tuning tends to introduce overfitted interactions. We validate these findings across multiple LLMs and datasets. Our findings provide new insights into early stopping and offer practical guidance for LLM training.

Primary area:
interpretability and explainable AI
Keywords:
Generalization mechanism, Theory, Hypothesis testing

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_amFgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the effectiveness of supervised fine-tuning through AND–OR interaction decompositions of input-word effects on target-token confidence. It tracks interactions that are removed, continuously preserved, or newly emerged during fine-tuning, and characterizes them using interaction order, agreement with another fine-tuned model, and cancellation between positive and negative effects. Experiments use LoRA on five instruction/chat-tuned models across GoEmotions, Unilaw-R1-Data, and Databricks-Dolly-15k. The reported trajectories show early removal of many high-order, mutually cancelling interactions, followed by the accumulation of new interactions with similar properties. Preserved interactions tend to be lower-order and more consistent across models. Downstream evaluations in four model–dataset settings achieve their best reported performance at early checkpoints. The authors interpret these findings as a brief denoising phase followed by overfitting and suggest interaction-based training monitoring. The actual interaction analysis concerns the first response token, and an operational early-stopping criterion is not evaluated.

Critical strengths
  1. A systematic empirical observation across multiple settings. The paper examines interaction dynamics across five models and three datasets rather than relying on a single before/after example. The recurring distinction between early changes and later accumulation of interactions is potentially useful for developing hypotheses about adaptation dynamics. Tracking preserved, removed, and emerged interactions separately makes the observations more informative than aggregate interaction counts alone.
  2. Connection to task-level outcomes. Appendix D.3 evaluates downstream performance rather than relying exclusively on interaction-derived metrics. For example, Qwen2.5-7B-Instruct’s GoEmotions Micro-F1 rises from 0.267 before SFT to 0.654 at step 1000, then falls to 0.593 at step 4000. This correspondence gives the proposed explanation an empirical connection to model utility, although it does not establish that the interaction changes explain the performance changes.
Critical weaknesses
  1. The measurements do not sufficiently establish the “noise” and “inference backbone” interpretations. The generalizability measure in Eqs. (7)–(9) measures agreement with another model, not generalization to unseen examples; useful model-specific computations can fail this criterion. Similarly, cancellation in Eq. (10) does not establish that the corresponding interactions are noise: terms can cancel on the complete input while remaining necessary to describe behavior under different input conditions. Faithful reconstruction of masked-input outputs does not resolve these interpretation questions. There is also a concrete issue in Section 3.3: Eq. (2) and Appendix A include a baseline contribution, but Eq. (11) applies the inverse log-odds transformation to interaction sums without that baseline. Unless an unstated centering convention is used, these are not the reconstructed target probabilities. Consequently, Figure 4 does not yet provide convincing evidence that the preserved interactions are sufficient for inference.
  2. The experimental scope and controls do not support the breadth of the SFT and early-stopping conclusions. All experiments adapt already instruction/chat-tuned models using one reported LoRA training recipe, with rank 8 by default and 30 training epochs (Appendix E.1). Interaction analysis uses only the first response token and selected input words (Appendix E.4). This leaves open whether the observations concern response-opening or formatting behavior rather than the acquisition of useful answer content. Longer training on the same dataset also does not establish the value of adding more distinct training data. The downstream tables provide no uncertainty estimates or repeated-seed results; some differences are very small, such as Dolly ROUGE-L of 0.367 at step 300 versus 0.366 at step 1000. Finally, Appendix G explicitly leaves the stopping threshold and its generalization benefit to future work. The reported trajectories are therefore insufficient to establish the broad reconciliation of SFT’s effectiveness or a validated early-stopping method.
Questions for authors
  1. Calculation clarification: Was the baseline contribution (b=v(x_{\emptyset})) included when computing the probabilities and losses in Figure 4? Eq. (11) appears to omit it. How do the results compare with the full-model and baseline-only losses when the baseline is retained consistently?
  2. Extraction clarification and robustness: What exact significance threshold (\tau), masking baseline, input-word selection procedure, and reference-model checkpoint were used? Were the selected words fixed across training checkpoints? Evidence that the interaction categories are stable under these choices would materially strengthen the interpretation of emergence and removal.
  3. Evidence needed for the central interpretation: What evidence distinguishes low-agreement or strongly cancelling interactions from useful model-specific computations? In particular, do the main conclusions persist when analyzing later, task-relevant response tokens rather than only the first token?
  4. Evidence needed for the practical claim: How were the phase boundaries determined without using downstream test outcomes, and does interaction-based monitoring improve checkpoint selection over validation-loss-based early stopping? Appendix G treats this benefit as untested, whereas the conclusion suggests a method for identifying the stopping point.
4 comments
AuthorsOpenReview ↗

Response to Reviewer amFg (Part 1/4)

Thank you for your detailed technical review. We respond to each question below.

If any answer is unclear or you have further questions, please let us know so we can respond before the deadline.


Q1: Ask whether Figure 4 includes b=v(x∅)b=v(x_\emptyset) and how it compares with full-model and bias-only losses

There is also a concrete issue in Section 3.3: Eq. (2) and Appendix A include a baseline ... are sufficient for inference.

Calculation clarification: Was the baseline contribution (b=v(x∅)b=v(x_{\emptyset})) ... the baseline is retained consistently?

A: Thank you for the question. The baseline contribution b=v(x∅)b=v(x_\emptyset) was not included in the previous Figure 4. We followed your suggestion and conducted new experiments with the bias term added to Eq. (11), including the full model, full interaction reconstruction, and bias-only baseline. The table below shows the target-token prediction losses of Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on GoEmotions. Lower loss indicates better target-token prediction.

Model and datasetFull modelFull interaction reconstructionBias onlyPreservedEmerged-IEmerged-II
Qwen2.5-3B-Instruct, GoEmotions1.2431.2604.2732.2422.4134.095
Qwen2.5-7B-Instruct, GoEmotions2.1212.1274.1692.8821.9324.808

The full interaction reconstruction achieves almost the same loss as the full model. Moreover, preserved and Emerged-I interactions consistently achieve lower loss than Emerged-II interactions.

These results support our conclusion in Section 3.3: interactions preserved during SFT and interactions learned in the early stage contribute more to target-token prediction, whereas interactions learned in the later stage contribute much less. We will update Eq. (11) and Figure 4 with these new results.


AuthorsOpenReview ↗

Response to Reviewer amFg (Part 2/4)

Q2: Ask for details about the threshold, masking baseline, word selection, reference checkpoint, and fixed variables

The experimental scope and controls do not support the breadth of the SFT ... selected input words (Appendix E.4).

Extraction clarification and robustness: What exact significance threshold ... the interpretation of emergence and removal.

A: Thank you for the question. We follow [cite1, cite2, cite3] to identify salient interactions using

τ=λ⋅1m∑j=1m∣vj(x)−vj(x∅)∣,\tau=\lambda\cdot\frac{1}{m}\sum_{j=1}^{m}|v_j(x)-v_j(x_\emptyset)|,

where λ=0.01\lambda=0.01 in the main experiments. An interaction is identified as salient when its absolute strength is larger than τ\tau. Please see Appendix F and Algorithm 1 for the interaction extraction procedure.

For masking, we replace the embedding of a masked token with the mean embedding over the model vocabulary, which represents the no-information baseline described in Section 3.1. Each selected word is treated as one input variable. If a word contains multiple tokenizer pieces, all pieces are masked together. For each prompt, we randomly select ten valid words that can be mapped to tokenizer tokens and retain their original order. The same analysis samples and the same selected words are used at every checkpoint of the same training run. Please see Appendix E.4 for details.

For interaction generalizability, we use a fixed LLM as the baseline DNN throughout the training process. Please see Appendix E.2 for details.

We conducted new experiments using λ∈{0.005,0.010,0.020}\lambda\in\{0.005,0.010,0.020\}. The table reports the average number of interactions for each model on GoEmotions. The percentages show how removed or newly acquired interactions are distributed between the two stages.

Modelλ\lambdaRemoved in the first stageAcquired in the first stageRemoved in the second stageAcquired in the second stage
Llama-2-7B-Chat0.005559.4 (87.3%)89.4 (14.6%)81.5 (12.7%)522.9 (85.4%)
Llama-2-7B-Chat0.010453.8 (90.7%)53.0 (11.4%)46.8 (9.3%)412.2 (88.6%)
Llama-2-7B-Chat0.020313.2 (93.6%)25.4 (8.9%)21.3 (6.4%)261.5 (91.1%)
Qwen2.5-3B-Instruct0.005629.1 (88.2%)95.5 (11.8%)84.1 (11.8%)716.6 (88.2%)
Qwen2.5-3B-Instruct0.010518.9 (91.8%)54.4 (8.3%)46.0 (8.2%)604.8 (91.7%)
Qwen2.5-3B-Instruct0.020361.9 (95.0%)23.9 (5.2%)19.1 (5.0%)434.4 (94.8%)
Qwen2.5-7B-Instruct0.005514.0 (91.3%)54.1 (10.0%)49.0 (8.7%)487.8 (90.0%)
Qwen2.5-7B-Instruct0.010393.7 (93.6%)30.4 (7.8%)27.1 (6.4%)357.6 (92.2%)
Qwen2.5-7B-Instruct0.020254.4 (95.7%)13.8 (6.2%)11.3 (4.3%)209.3 (93.8%)
Llama-3-8B-Instruct0.005320.6 (85.0%)67.7 (18.6%)56.5 (15.0%)296.6 (81.4%)
Llama-3-8B-Instruct0.010223.2 (86.4%)43.9 (17.2%)35.1 (13.6%)211.1 (82.8%)
Llama-3-8B-Instruct0.020128.8 (87.9%)24.1 (16.5%)17.8 (12.1%)121.5 (83.5%)

For every model and every value of λ\lambda, 85.0%--95.7% of the removed interactions are removed in the first stage, whereas 81.4%--94.8% of the newly acquired interactions are acquired in the second stage.

We also varied the number of selected words from 7 to 10 for Qwen2.5-3B-Instruct on GoEmotions. By step 1000, the model has removed 95.5%--98.9% of the interactions present before SFT, while newly acquired interactions account for only 2.3%--11.9% of the initial number. After model training, newly acquired interactions increase to 35.0%--77.7% of the initial number, while only 0.1%--2.4% of their effects remain after positive and negative cancellation. Therefore, changing either the significance threshold or the number of selected words does not change the main result: the first stage mainly removes interactions, whereas the second stage mainly acquires new interactions whose effects largely cancel each other. We will add these settings and robustness results to the revised appendix.


AuthorsOpenReview ↗

Response to Reviewer amFg (Part 3/4)

Q3: Ask whether low-agreement or cancelling interactions may be useful model-specific computation and whether the result holds for later tokens

The measurements do not sufficiently establish the “noise” ... resolve these interpretation questions.

Interaction analysis uses only the first response token and ... acquisition of useful answer content.

Evidence needed for the central interpretation: What evidence ... than only the first token?

A: Thank you for the question. We use target-token prediction loss to test whether different types of interactions help prediction. If a group of interactions helps the model predict the target token, using that group should yield lower loss.

First, Figure 4 and Section 3.3 provide this direct comparison. On GoEmotions, the losses obtained using Emerged-I versus Emerged-II interactions are 1.613 versus 2.444 for Llama-2-7B-Chat, 2.413 versus 4.095 for Qwen2.5-3B-Instruct, and 1.932 versus 4.808 for Qwen2.5-7B-Instruct. Preserved interactions also produce lower loss than Emerged-II interactions. In these examples, interactions retained throughout SFT or acquired early predict the target token better than interactions acquired late.

Second, Figures 2 and 3 in Section 3.2.2 show that late-acquired interactions are mainly high-order, are rarely learned by different LLMs, and have positive and negative effects that largely cancel each other. Among newly acquired interactions of order k≥4k\geq4, only 0.08%--0.27% are learned by different LLMs, and the cancellation ratios of the remaining interactions reach 98.69%--99.97%. Their total strength at each order also grows approximately with the number of possible order-kk word combinations, (nk)\binom{n}{k}.

Third, Appendix D.3 and Tables 1 and 2 show that the best downstream performance is reached during the early stage in all four reported cases. Once performance stops improving, high-order newly acquired interactions continue to increase, while fewer of them are shared by different LLMs and their positive and negative effects cancel more strongly.

Thus, late-acquired interactions yield higher prediction loss, are rarely learned by different LLMs, largely cancel each other, and continue to accumulate after test performance stops improving. We will therefore describe them as noise-like interactions associated with overfitting.

Our interaction analysis does not depend on the position of the target token. For a later response token, we include the preceding response tokens in the context and then compute how interactions between the input words affect the prediction of this target token. Thus, the same analysis can be applied to any task-relevant token in the response.

We also conducted experiments on later response tokens using Gemma-3-4B-IT on Databricks-Dolly-15k. For each response, we randomly skip some initial tokens and use a subsequent token as the prediction target. By the end of the denoising stage, the model has removed 317.9 of the 327.4 interactions present before SFT, while acquiring only 20.4 new interactions. During the subsequent overfitting stage, the number of newly acquired interactions increases to 351.8, and their total strength increases from 3.82 to 117.14. Meanwhile, the proportion of their effects remaining after positive and negative cancellation decreases from 14.5% to 0.3%. Therefore, the same phenomenon is observed for later response tokens: SFT first removes many interactions and later acquires many new interactions whose positive and negative effects largely cancel each other. We will add these results and clarify the target-token setting in the revised paper.


AuthorsOpenReview ↗

Response to Reviewer amFg (Part 4/4)

Q4: Ask how the phase boundary is determined without test outcomes and whether it improves on validation-loss early stopping

The downstream tables provide no uncertainty ... early-stopping method.

Evidence needed for the practical claim: How ... for identifying the stopping point.

A: Thank you for the question. Determining the phase boundary does not require downstream test outcomes. Before SFT, we sample a small fixed subset from the training data and periodically extract its interactions during training. The first stage ends when the rapid removal of pre-SFT interactions levels off. After this point, newly acquired interactions continue to increase, fewer of them are generalizable, and their positive and negative effects cancel more strongly. Thus, the phase boundary can be determined directly from the interaction changes on the sampled training data. The test set is used only for the final evaluation and is not used to determine the boundary or select the checkpoint.

We can use the proportion of generalizable interactions, γ\gamma, together with the validation loss to determine the stopping point.

In practice, early stopping can be triggered when γ\gamma drops below a given threshold, especially when the validation loss no longer improves. This indicates that SFT has learned many non-generalizable interactions, and stopping at this point can prevent further overfitting. In some cases, the validation loss may still decrease slowly even though γ\gamma has already dropped substantially. In this case, we recommend stopping the current training run and improving the training data, for example through data augmentation, before continuing SFT.

Specifically, we divide the training set into subgroups, with each subgroup representing one type of training sample. For GoEmotions, we construct the subgroups according to emotion categories and periodically compute γ\gamma for each subgroup during SFT. If γ\gamma becomes very small for one subgroup, this indicates that the model has learned many non-generalizable interactions from this subgroup, and training on this subgroup should be stopped before these interactions further accumulate.

Compared with monitoring validation loss over the entire dataset, monitoring γ\gamma within individual subgroups provides a more fine-grained signal for identifying overfitted samples. On GoEmotions, we divide the training set into 12 subgroups and apply this interaction-based early-stopping criterion. It reduces the training cost by approximately 30% while maintaining the test performance. We will add this early-stopping strategy and the corresponding experimental results to the revised paper.

Reviewer_sZebOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies what changes inside an LLM during SFT. The authors extract AND-OR interactions at several checkpoints and label them as preserved, removed, or newly emerged. They argue that early SFT mostly removes noisy interactions, while later SFT adds high-order, overfitted ones. This pattern is shown on several LLMs and datasets, alongside task metrics that often peak early in training. The intended takeaway is that interaction dynamics can explain why more SFT sometimes hurts and could guide early stopping.

Critical strengths
  1. This is a welcome attempt to study SFT mechanistically rather than propose another fine-tuning recipe. The authors track training over time and use several models and datasets.

  2. The decomposition gives a concrete language for asking where changes occur during SFT. The paper is also open about the computational cost of extracting interactions.

Critical weaknesses
  1. I do not think the current evidence supports calling the observed changes denoising or overfitting. “Generalizable” means that a second model has an interaction on the same word subset with the same sign. That is not generalization across examples or distributions. Likewise, high order is not evidence of unreliability, and a small signed sum only shows cancellation; it does not show that the individual interactions are noise or causally unimportant. These proxies carry the central claim of the paper.

  2. The early-stopping claim is retrospective. The paper first identifies a short “denoising stage” from the interaction curves and then notes that performance is best around that stage. It does not show that an interaction-based rule, fixed in advance, can select a good checkpoint on held-out tasks or runs. The performance tables also have no uncertainty over independent SFT runs. This is a temporal correlation, not yet a useful diagnostic.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer sZeb

Thank you for your clear review. We respond to each question below.

If any answer is unclear or you have further questions, please let us know so we can respond before the deadline.


Q1: Ask whether the current proxies support the denoising and overfitting interpretation

I do not think the current evidence supports calling the observed ... These proxies carry the central claim of the paper.

A: Thank you for the question. We use target-token prediction loss to test whether different types of interactions help prediction. If a group of interactions helps the model predict the target token, using that group should yield lower loss.

First, Figure 4 and Section 3.3 provide this direct comparison. On GoEmotions, the losses obtained using Emerged-I versus Emerged-II interactions are 1.613 versus 2.444 for Llama-2-7B-Chat, 2.413 versus 4.095 for Qwen2.5-3B-Instruct, and 1.932 versus 4.808 for Qwen2.5-7B-Instruct. Preserved interactions also produce lower loss than Emerged-II interactions. In these examples, interactions retained throughout SFT or acquired early predict the target token better than interactions acquired late.

Second, Figures 2 and 3 in Section 3.2.2 show that late-acquired interactions are mainly high-order, are rarely learned by different LLMs, and have positive and negative effects that largely cancel each other. Among newly acquired interactions of order k≥4k\geq4, only 0.08%--0.27% are learned by different LLMs, and the cancellation ratios of the remaining interactions reach 98.69%--99.97%. Their total strength at each order also grows approximately with the number of possible order-kk word combinations, (nk)\binom{n}{k}.

Third, Appendix D.3 and Tables 1 and 2 show that the best downstream performance is reached during the early stage in all four reported cases. Once performance stops improving, high-order newly acquired interactions continue to increase, while fewer of them are shared by different LLMs and their positive and negative effects cancel more strongly.

Thus, late-acquired interactions yield higher prediction loss, are rarely learned by different LLMs, largely cancel each other, and continue to accumulate after test performance stops improving. We will therefore describe them as noise-like interactions associated with overfitting.


Q2: How interaction-based early stopping performs across independent training runs

The early-stopping claim is retrospective. The paper first identifies ... a useful diagnostic.

A: Thank you for the question. We followed your suggestion and evaluated an early-stopping rule determined before training. We use the proportion of generalizable interactions, γ\gamma, together with the validation loss to determine when to stop SFT.

In practice, training is stopped when γ\gamma drops below a given threshold and the validation loss no longer improves. A low γ\gamma indicates that the model is learning many interactions that cannot be generalized to other LLMs. Stopping at this point prevents these non-generalizable interactions from further accumulating. In some cases, the validation loss may still decrease slowly even though γ\gamma has already dropped substantially. In this case, we recommend stopping the current training run and improving the training data, for example through data augmentation, before continuing SFT.

We apply this criterion to different subsets of the training data rather than only monitoring the entire dataset. Each subset represents one type of training sample. For GoEmotions, we divide the training set into 12 subsets according to emotion categories and periodically compute γ\gamma for each subset. If γ\gamma becomes very small for one subset, training on this subset is stopped before more non-generalizable interactions are learned from these samples.

Compared with monitoring validation loss over the entire dataset, this criterion can identify which types of training samples have begun to cause overfitting. On GoEmotions, it reduces the training cost by about 30% while maintaining the test performance.

We also repeated SFT of Qwen2.5-3B-Instruct on GoEmotions with three independent random seeds. At step 1000, the model has removed 411.1±0.8411.1\pm0.8 interactions and acquired only 12.2±3.112.2\pm3.1 new interactions. After model training, the number of newly acquired interactions increases to 186.0±35.6186.0\pm35.6, and their total strength increases from 3.30±0.543.30\pm0.54 to 83.53±12.7683.53\pm12.76. Meanwhile, the proportion of their effects that remains after positive and negative cancellation decreases from 33.9%±14.4%33.9\%\pm14.4\% to 1.5%±1.1%1.5\%\pm1.1\%. Therefore, all three independent runs show the same process: the model first removes many noise-like interactions and later learns many new interactions whose positive and negative effects largely cancel each other. We will add the stopping rule and the mean and standard deviation across independent runs to the revised paper.

Reviewer_iZG3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the training dynamics of supervised fine-tuning (SFT) using AND-OR interaction decomposition to track feature interactions that are removed, retained, or newly formed across checkpoints. The authors observe that early training removes numerous high-order, mutually canceling, and model-specific interactions, whereas prolonged training introduces more interactions of a similar nature. Based on these observations, the paper proposes a "brief denoising followed by prolonged overfitting" perspective on SFT, arguing that SFT primarily prunes noise rather than acquiring reliable new knowledge, and discusses its potential utility for early stopping.

Critical strengths
  • Rather than treating SFT as a static pre- vs. post-training comparison, the paper tracks how feature interactions evolve across checkpoints using AND–OR interaction decomposition. This is consequential to my assessment because it introduces a structured mechanistic lens to examine how fine-tuning alters model computations over time, which is a promising analytical direction if properly validated.
  • The paper connects its interaction dynamics to downstream evaluation curves across multiple task categories (e.g., showing an early performance surge followed by degradation on GoEmotions in Appendix D.3). This provides initial empirical grounding that links internal interaction shifts to observable generalization behavior.
Critical weaknesses
  • The paper's core thesis that SFT primarily denoises and rarely learns reliable new knowledge relies on unvalidated operational definitions and a flawed ablation. Specifically, it equates "cross-model sharing" (Eq. 7) with generalizable knowledge and full-input "positive-negative cancellation" (Eq. 10) with useless noise, even though model-unique interactions can encode valid domain adaptations and canceling interactions remain active on partial inputs. Furthermore, the "inference backbone" evaluation (Eq. 11, Fig. 4) omits the decomposition bias term bb and lacks a full-model baseline, rendering it an invalid ablation. Finally, the set definitions (Eqs. 5-6) conflate temporarily dipping interactions with newly learned ones under an unreported sensitivity threshold τ\tau. This is fatal to my assessment because the primary mechanistic conclusions are not supported by the mathematical measurements.
  • The experiments cannot support general claims about SFT for three reasons: (1) the evaluated model generations are outdated (with the newest being Qwen2.5), limiting the relevance of these findings to modern pre-training and post-training paradigms; (2) all runs start from already instruction-tuned (Instruct/Chat) checkpoints using low-rank LoRA (r=8r=8), testing continual low-rank adaptation rather than standard SFT from base models; and (3) interaction extraction is restricted solely to the first target token and a small subset of salient input words (Appendix E.4). Extrapolating general conclusions about SFT knowledge acquisition and multi-step reasoning from first-token continual LoRA adaptation on older models is a major overclaim.
Questions for authors
  • If you restore the bias term bb to Eq. (11) and plot the losses of the full model, full interaction reconstruction, and bias-only baseline alongside the retained interactions in Fig. 4, how much of the full model's predictive performance do the retained interactions actually recover?
  • What exact threshold τ\tau was used in Algorithm 1? If you vary τ\tau and separate genuinely new interactions from pre-existing interactions that temporarily dropped below τ\tau at an intermediate checkpoint before re-emerging (Eqs. 5-6), does the "brief denoising followed by prolonged overfitting" pattern still hold?
  • Does the same two-stage interaction dynamic hold when performing SFT starting from a Base (non-Instruct) checkpoint of a current-generation open-weight model, and when evaluating interactions across subsequent target tokens beyond the very first response token?
  • Can you demonstrate that interaction-based metrics (such as cross-model sharing) provide a more reliable or earlier stopping signal than simply monitoring standard validation loss, especially on generation tasks (Unilaw, Dolly) where post-peak ROUGE-L fluctuations are negligible (∼0.001\sim 0.001)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer iZG3 (Part 1/4)

Thank you for your detailed review. We respond to each question below.

If any answer is unclear or you have further questions, please let us know so we can respond before the deadline.


Q1: Ask whether Figure 4 omits the bias and how preserved interactions compare with full-model baselines

The paper's core thesis that SFT primarily denoises and rarely learns ... baseline, rendering it an invalid ablation.

If you restore the bias term bb to Eq. (11) and plot the losses ... do the retained interactions actually recover?

A: Thank you for the question. The baseline contribution b=v(x∅)b=v(x_\emptyset) was not included in the previous Figure 4. We followed your suggestion and conducted new experiments with the bias term added to Eq. (11), including the full model, full interaction reconstruction, and bias-only baseline. The table below shows the target-token prediction losses of Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on GoEmotions. Lower loss indicates better target-token prediction.

Model and datasetFull modelFull reconstructionBias onlyPreservedEmerged-IEmerged-II
Qwen2.5-3B-Instruct, GoEmotions1.2431.2604.2732.2422.4134.095
Qwen2.5-7B-Instruct, GoEmotions2.1212.1274.1692.8821.9324.808

The full interaction reconstruction achieves almost the same loss as the full model. Moreover, preserved and Emerged-I interactions consistently achieve lower loss than Emerged-II interactions.

These results support our conclusion in Section 3.3: interactions preserved during SFT and interactions learned in the early stage contribute more to target-token prediction, whereas interactions learned in the later stage contribute much less. We will update Eq. (11) and Figure 4 with these new results.


AuthorsOpenReview ↗

Response to Reviewer iZG3 (Part 2/4)

Q2: Ask about the significance threshold and robustness after separating reappearance

Finally, the set definitions (Eqs. 5-6) conflate temporarily ... are not supported by the mathematical measurements.

What exact threshold τ\tau was used in Algorithm 1 ... pattern still hold?

A: Thank you for the question. We followed [cite1, cite2, cite3] to identify salient interactions using the following threshold:

τ=λ⋅1m∑j=1m∣vj(x)−vj(x∅)∣,\tau=\lambda\cdot\frac{1}{m}\sum_{j=1}^{m}|v_j(x)-v_j(x_\emptyset)|,

where λ=0.01\lambda=0.01 in the main experiments. An interaction is identified as salient when its absolute strength is larger than τ\tau. Please see Appendix F and Algorithm 1 for details.

We also followed your suggestion and conducted new experiments using different values of λ\lambda. The results are shown below.

Modelλ\lambdaRemoved in the first stageAcquired in the first stageRemoved in the second stageAcquired in the second stage
Llama-2-7B-Chat0.005559.4 (87.3%)89.4 (14.6%)81.5 (12.7%)522.9 (85.4%)
Llama-2-7B-Chat0.010453.8 (90.7%)53.0 (11.4%)46.8 (9.3%)412.2 (88.6%)
Llama-2-7B-Chat0.020313.2 (93.6%)25.4 (8.9%)21.3 (6.4%)261.5 (91.1%)
Qwen2.5-3B-Instruct0.005629.1 (88.2%)95.5 (11.8%)84.1 (11.8%)716.6 (88.2%)
Qwen2.5-3B-Instruct0.010518.9 (91.8%)54.4 (8.3%)46.0 (8.2%)604.8 (91.7%)
Qwen2.5-3B-Instruct0.020361.9 (95.0%)23.9 (5.2%)19.1 (5.0%)434.4 (94.8%)
Qwen2.5-7B-Instruct0.005514.0 (91.3%)54.1 (10.0%)49.0 (8.7%)487.8 (90.0%)
Qwen2.5-7B-Instruct0.010393.7 (93.6%)30.4 (7.8%)27.1 (6.4%)357.6 (92.2%)
Qwen2.5-7B-Instruct0.020254.4 (95.7%)13.8 (6.2%)11.3 (4.3%)209.3 (93.8%)
Llama-3-8B-Instruct0.005320.6 (85.0%)67.7 (18.6%)56.5 (15.0%)296.6 (81.4%)
Llama-3-8B-Instruct0.010223.2 (86.4%)43.9 (17.2%)35.1 (13.6%)211.1 (82.8%)
Llama-3-8B-Instruct0.020128.8 (87.9%)24.1 (16.5%)17.8 (12.1%)121.5 (83.5%)

The table reports the average number of interactions for each model on GoEmotions. The percentages show how the removed or acquired interactions are distributed between the two stages. For every model and every value of λ\lambda, 85.0%--95.7% of the removed interactions are removed in the first stage, whereas 81.4%--94.8% of the newly acquired interactions are acquired in the second stage. Therefore, the first stage mainly removes interactions, whereas the second stage mainly acquires new interactions. The brief denoising stage followed by prolonged overfitting is consistently observed under different thresholds.

[cite1] Where We Have Arrived in Proving the Emergence of Sparse Interaction Primitives in DNNs. ICLR, 2024.

[cite2] Defining and Extracting Generalizable Interaction Primitives from DNNs. ICLR, 2024.

[cite3] Explaining Generalization Power of a DNN using Interactive Concepts. AAAI, 2024.


AuthorsOpenReview ↗

Response to Reviewer iZG3 (Part 3/4)

Q3: Ask whether the same dynamics hold for Base models, current-generation models, and later output tokens

The experiments cannot support general claims about SFT for three ... adaptation on older models is a major overclaim.

Does the same two-stage interaction dynamic ... tokens beyond the very first response token?

A: Thank you for the question. We conducted two new experiments to examine SFT starting from a Base model and interaction dynamics for later response tokens.

First, we use full-parameter fine-tuning on the Base model Qwen2.5-3B rather than LoRA. We fine-tune all model parameters on the MATH dataset [cite1], where the input is a mathematical problem and the output contains the reasoning process and final answer. The tables report the count, total strength, generalizability γ\gamma, and non-cancelled-effect ratio ρ\rho of removed, newly emerged, and preserved interactions.

Removed interactions

Training stageCountStrengthγ\gammaρ\rho
Before SFT0.000.00----
Denoising stage22.8519.440.0210.080
Overfitting stage23.0020.950.0410.068

Newly emerged interactions

Training stageCountStrengthγ\gammaρ\rho
Before SFT0.000.00----
Denoising stage2.901.950.1490.243
Overfitting stage16.5530.840.0500.135

Preserved interactions

Training stageCountStrengthγ\gammaρ\rho
Before SFT23.9522.250.0630.041
Denoising stage1.101.300.4270.325
Overfitting stage0.250.840.5470.606

By the end of the denoising stage, the count and strength of removed interactions have already reached 22.85 and 19.44, close to 23.00 and 20.95 in the overfitting stage. At the same time, only 2.90 new interactions with a total strength of 1.95 have emerged. During the subsequent overfitting stage, the count of newly emerged interactions increases to 16.55 and their total strength increases to 30.84, while their generalizability decreases from 0.149 to 0.050 and their non-cancelled-effect ratio decreases from 0.243 to 0.135. Therefore, full-parameter SFT of the Base model exhibits the same dynamics: the model first removes most noise-like interactions and then mainly acquires new interactions that are less generalizable and whose positive and negative effects cancel more strongly.

Second, our interaction analysis does not depend on the position of the target token. For a later response token, we include the preceding response tokens in the context and compute how interactions between the input words affect the prediction of this target token. Thus, the same analysis can be applied to any task-relevant token in the response.

We conducted experiments on later response tokens using Gemma-3-4B-IT on Databricks-Dolly-15k. For each response, we randomly skip some initial tokens and use a subsequent token as the prediction target. By the end of the denoising stage, the model has removed 317.9 of the 327.4 interactions present before SFT, while acquiring only 20.4 new interactions. During the subsequent overfitting stage, the number of newly acquired interactions increases to 351.8, and their total strength increases from 3.82 to 117.14. Meanwhile, the proportion of their effects remaining after positive and negative cancellation decreases from 14.5% to 0.3%. Therefore, the same phenomenon is observed for later response tokens: SFT first removes many interactions and later acquires many new interactions whose positive and negative effects largely cancel each other.

These results show that the two-stage interaction dynamics also hold for full-parameter SFT starting from a Base model and for later response tokens. We will add both experiments to the revised paper.

[cite1] Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS, 2021.


Reviewer_3NDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how input interactions evolve during supervised fine-tuning (SFT). Using an existing AND–OR interaction framework, the authors distinguish removed, preserved, and emerging interactions and analyze their order, cross-model agreement, and cancellation. Experiments across several instruction-tuned models and datasets using LoRA suggest an early phase dominated by interaction removal, followed by accumulating new interactions as downstream performance plateaus or declines. The authors interpret these observations as a short denoising phase followed by overfitting.

Critical strengths

1.The paper addresses a central question about SFT: why it improves downstream performance in some settings but impairs generalization in others. 2.The authors analyze interaction dynamics across multiple models and datasets, distinguishing three types of interactions and relating their evolution to downstream performance. This provides a potentially informative perspective beyond aggregate task scores.

Critical weaknesses

1.The main conclusions require stronger support. ●Definition of noise: Low cross-model agreement or cancellation does not necessarily indicate noise; such interactions may encode useful model-specific patterns or contribute to predictions on out-of-distribution (OOD) inputs. The paper needs a clearer definition of “noise-like interactions” and independent evidence that these interactions are unhelpful or harmful. ●Effects of the tracking definition: The preserved set is monotonically non-increasing by construction, while interactions that temporarily disappear and return are classified as emerging. These definitional effects should be distinguished from genuine removal and acquisition. ●Scope of the conclusions: All experiments use LoRA, so restricted adaptation capacity cannot be ruled out as an alternative explanation. A representative higher-rank comparison and, if feasible, full fine-tuning would help; otherwise, the conclusions should be narrowed. 2.The experimental protocol needs clarification. ●Checkpoint selection: The rationale for selecting checkpoints such as 1800 versus 2000 steps is unclear. Sparse early measurements also make it difficult to establish the boundary of the purported short denoising phase. ●Training configuration: Appendix E.1 reports BF16 training on V100 GPUs, which lack native BF16 support. It is unclear whether this is a reporting error or involves emulation or precision fallback. 3.The novelty should be more precisely articulated. ●Relationship to prior work: AND–OR decomposition, cross-model interaction analysis, and two-phase interaction dynamics have been studied previously. The inherited methodology should be clearly distinguished from this paper’s contributions.

Questions for authors

Please address the following questions in conjunction with the corresponding concerns in the Weaknesses section: 1.What distinguishes a noise-like interaction from a useful model-specific interaction? If possible, please provide a few concrete examples. 2.Do the main conclusions hold when first-time emergence is separated from reappearance? 3.Could the observed dynamics depend on LoRA capacity? If time and computational resources permit, please include a representative full fine-tuning comparison. 4.How were the checkpoints selected, and were the same analysis samples and input variables used throughout training?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3NDj (Part 1/3)

Thank you for your careful and constructive review. We appreciate your recognition of the value of tracking interaction dynamics beyond aggregate task scores. We respond to your questions below.

If any answer is unclear or you have further questions, please let us know so we can respond before the deadline.


Q1: Ask how noise-like interactions are distinguished from useful model-specific interactions

Definition of noise: Low cross-model agreement or ... independent evidence that these interactions are unhelpful or harmful.

What distinguishes a noise-like interaction from a ... examples.

A: Thank you for the question. Noise-like interactions often have two properties. First, they are non-generalizable, meaning that they are not consistently learned by different LLMs. Noise usually reflects accidental patterns fitted by one model and is therefore less likely to appear in other LLMs. Second, their positive and negative effects largely cancel each other. This means that many interactions are present, but together they make little net contribution to the prediction. Therefore, non-generalizable and mutually cancelling interactions are more likely to be noise-like.

We further tested Llama-2-7B-Chat, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Llama-3-8B-Instruct on GoEmotions. Among newly acquired interactions of order k≥4k\geq4, only 0.08%--0.27% are consistently learned by different LLMs. For the remaining non-generalizable interactions, positive and negative effects cancel by 98.69%--99.97%. Thus, these late-acquired high-order interactions are almost entirely non-generalizable, and their effects almost entirely cancel. This is why we describe them as noise-like interactions.


Q2: Ask whether the conclusions hold after separating first-time acquisition from reappearance

Effects of the tracking definition: The preserved set is monotonically non-increasing ... from genuine removal and acquisition.

Do the main conclusions hold when first-time emergence is separated from reappearance?

A: Thank you for the question. We introduce two definitions to distinguish these cases. A newly acquired interaction is one whose final effect is not present before SFT; this includes interactions that are absent before SFT and interactions whose effects change direction during SFT. A reappeared interaction is present before SFT with the same effect, disappears at an intermediate checkpoint, and appears again later.

Using these definitions, we tested Llama-2-7B-Chat, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Llama-3-8B-Instruct on GoEmotions. After model training, newly acquired interactions account for 86.7%--90.3% of EtE_t, while reappeared interactions account for only 9.8%--13.3%. These two categories cover all interactions in EtE_t. Therefore, most of the late-stage increase comes from interactions newly acquired during SFT, and our conclusion remains unchanged.


AuthorsOpenReview ↗

Response to Reviewer 3NDj (Part 2/3)

Q3: Ask whether the observed dynamics are caused by rank-8 LoRA capacity

Scope of the conclusions: All experiments use LoRA ... otherwise, the conclusions should be narrowed.

Could the observed dynamics depend on LoRA ... please include a representative full fine-tuning comparison.

A: Thank you for the suggestion. We conducted a new experiment using full-parameter fine-tuning rather than LoRA. Specifically, we fine-tune all parameters of Qwen2.5-3B on the MATH dataset [cite1], where the input is a mathematical problem and the output contains the reasoning process and final answer. We then analyze the interactions encoded by the model at different training stages.

The tables report the count, total strength, generalizability γ\gamma, and non-cancelled-effect ratio ρ\rho of removed, newly emerged, and preserved interactions.

Removed interactions

Training stageCountStrengthγ\gammaρ\rho
Before SFT0.000.00----
Denoising stage22.8519.440.0210.080
Overfitting stage23.0020.950.0410.068

Newly emerged interactions

Training stageCountStrengthγ\gammaρ\rho
Before SFT0.000.00----
Denoising stage2.901.950.1490.243
Overfitting stage16.5530.840.0500.135

Preserved interactions

Training stageCountStrengthγ\gammaρ\rho
Before SFT23.9522.250.0630.041
Denoising stage1.101.300.4270.325
Overfitting stage0.250.840.5470.606

First, full-parameter SFT removes most removable interactions during the denoising stage. By the end of this stage, the count and strength of removed interactions have already reached 22.85 and 19.44, close to 23.00 and 20.95 after model training. At the same time, only 2.90 new interactions with a total strength of 1.95 have emerged.

Second, continued training mainly increases newly emerged interactions rather than removing additional interactions. Their count increases from 2.90 to 16.55, and their total strength increases from 1.95 to 30.84. Their generalizability also decreases from 0.149 to 0.050, and their non-cancelled-effect ratio decreases from 0.243 to 0.135.

Therefore, full-parameter fine-tuning exhibits the same process observed in our LoRA experiments: SFT first removes most noise-like interactions within a short stage, while continued training mainly introduces new, less generalizable, and more strongly cancelling interactions. This result shows that the reported dynamics are not caused by the capacity constraint of rank-88 LoRA. We will add this full-parameter fine-tuning experiment to the revised paper.

[cite1] Hendrycks et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS, 2021.


AuthorsOpenReview ↗

Response to Reviewer 3NDj (Part 3/3)

Q4: Ask about checkpoint selection and whether samples and input variables are fixed across stages

Checkpoint selection: The rationale for selecting checkpoints ... the purported short denoising phase.

How were the checkpoints selected ... used throughout training?

A: We selected checkpoints to cover the entire SFT process: before SFT, the early period when downstream performance improves, and the later period when performance begins to decline. As shown in Appendix D.3 and Tables 1 and 2, downstream performance generally improves during the first stage. During this stage, the model mainly removes noise-like interactions. In the second stage, performance no longer improves and often declines, while the model starts learning noise-like interactions again. Therefore, the selected checkpoints cover both the effective stage of SFT and the subsequent overfitting stage.

We use the same analysis samples and the same ten selected input words at every checkpoint of a training run. As described in Section 3.1, each word is one input variable; if a word contains several tokenizer pieces, all of them are masked together. We will state this control explicitly in the experimental protocol.


Q5: Ask about the inconsistency between BF16 training and V100 GPUs

Training configuration: Appendix E.1 reports ... or precision fallback.

A: Thank you for pointing this out. The experiments were conducted on NVIDIA V100 GPUs using FP16. We will correct the precision in Appendix E.1.


Q6: Ask about novelty relative to prior interaction-based work

Relationship to prior work: AND–OR decomposition ... distinguished from this paper’s contributions.

A: Thank you for the question. The AND--OR decomposition and the cross-model comparison of interactions are adopted from prior work. Our contribution is to use these tools to explain what changes inside an LLM during SFT and why SFT first improves and later harms generalization.

First, Section 3.2 separates interactions into removed, preserved, and newly acquired interactions and follows their changes during SFT. This allows us to measure how many non-generalizable and mutually cancelling interactions are removed or learned at different training stages. Figures 2 and 3 show that the early stage mainly removes noise-like interactions, whereas continued training introduces many new noise-like interactions.

Second, Section 3.3 and Figure 4 directly test how much the different types of interactions help the model predict the target token. Appendix D.3 and Tables 1 and 2 further show that downstream performance improves while noise-like interactions are being removed, but stops improving and often declines when the model begins to learn such interactions again.

Therefore, our contribution is not a new interaction decomposition, but a quantitative explanation of the different effects of SFT: SFT first improves the model by removing noise-like interactions and later causes overfitting by learning them again. Because interactions are extracted from individual samples, this analysis can also identify which samples acquire these interactions and when they begin to appear during training. We will clarify this distinction from prior work in the introduction and related work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.