acceptodds
Under review as a conference paper at ICLR 2027

Not All Conflicts Are Equal: Forecasting Token Learning Trajectories for Supervised Fine-Tuning

Abstract

Supervised fine-tuning adapts pretrained language models to new domains, but often degrade their general capabilities. We study conflicting tokens, where the supervised target disagrees with a confident pretrained prediction. Recent token-level methods treat these tokens differently depending on the goal: suppressing them better preserves general capabilities but often believed to limit adaptation, while emphasizing them improves target-domain adaptation at the cost of retention. However, achieving strong target-domain adaptation while preserving general capabilities remains challenging. This motivates us to look more closely at conflicting tokens. We find that they do not behave uniformly during SFT: some resolve quickly but often revert, some are learned gradually, and others remain difficult to learn. Retaining gradually learned conflicts while masking other two groups gives the best adaptation-retention balance among the strategies we study. Based on this finding, we introduce ForeToken, an efficient pre-SFT forecasting method that estimates each conflict's resolution tendency from the initial optimization direction. This avoids an extra SFT run to observe token trajectories before constructing the mask. Across mathematical, medical, agentic, and code domains and three model families, ForeToken improves target-domain performance while better preserving general capabilities. On Qwen3-8B, it outperforms the strongest token-level baseline by 7.6 points on mathematical reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised fine-tuning, LLMs, Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_XiRJOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies conflicting tokens in supervised fine-tuning, where the supervised target disagrees with a confident pretrained prediction. The authors show that these conflicting tokens do not behave uniformly during SFT, but instead follow different learning trajectories: some are resolved early, some gradually, and some remain difficult to learn.

Based on this observation, the paper proposes ForeToken, which predicts the resolution tendency of conflicting tokens before SFT begins. It uses the initial SFT optimization direction and finite-difference estimates of token-level probability and entropy changes to construct a forecasting score. The method then retains predicted Intermediate conflicts while masking predicted Early and Late conflicts during training.

Experiments across math, medical, agent, and code adaptation, multiple model families, and continual fine-tuning settings show that ForeToken improves target-domain adaptation while better preserving general capabilities than standard SFT and several token-level baselines.

Critical strengths
  1. The paper provides an interesting analysis of conflicting tokens during SFT. Instead of treating all conflicting tokens uniformly, it identifies different learning trajectories and shows that these groups have different effects on the adaptation-retention trade-off.

  2. The proposed forecasting mechanism is simple and practical. It avoids the need for an additional full SFT run and only requires a one-time preprocessing step before training. The experimental evaluation is also broad, covering multiple domains, model families, and continual fine-tuning settings.

Critical weaknesses
  1. The trajectory forecasting signal is still relatively coarse. The paper itself shows substantial overlap between the forecasting-score distributions of Early, Intermediate, and Late conflicts. This suggests that the score captures a group-level tendency rather than reliably identifying the trajectory of each individual token.

  2. The method depends on the probability and entropy thresholds used to define conflicting tokens. The appendix shows noticeable performance variation across different threshold choices, especially on Pass@1, so it is unclear how robust the method is to these hyperparameters across models and domains.

Questions for authors
  1. How accurately can ForeToken distinguish Intermediate conflicts from Early and Late conflicts at the individual-token level? Since the score distributions overlap substantially, could the authors clarify how much misclassification occurs and how sensitive performance is to this prediction noise?

  2. How should the conflict thresholds be selected in a new domain or model without validation data? Are the default thresholds expected to transfer reliably across settings?

  3. Could the authors provide more intuition for why Early conflicts should be masked? Some Early conflicts are learned quickly, so it would be useful to better understand why retaining them is less helpful than retaining Intermediate conflicts.

4 comments

We thank the reviewer for their valuable suggestions and for recognizing the strengths of our work. We address each concern below and have updated the submitted PDF with additional results and clarifications.

W1 / Q1: Individual-Token Prediction Accuracy and Sensitivity to Prediction Errors

We evaluate how accurately ForeToken identifies the Intermediate group by comparing its predictions with the actual trajectories observed during SFT (Table 1). ForeToken correctly identifies 20,779 of 26,000 Intermediate tokens on Qwen3-8B Code and 19,952 of 28,951 on DeepSeek-R1-Qwen3-8B Math. We further compare ForeToken with simpler random, probability-based, and entropy-based predictors (Table 2), showing its advantage in identifying Intermediate conflicts.

Table 1. Counts of Intermediate conflicts identified and missed by ForeToken across models and domains.

MetricQwen3-8B (Code)DeepSeek-R1-Qwen3-8B (Math)
Actual Intermediate26,00028,951
Correctly identified20,77919,952
Missed5,2218,999

Table 2. Intermediate group F1 comparison with simpler predictors across models and domains.

MethodQwen3-8B (Code)DeepSeek-R1-Qwen3-8B (Math)
Random0.1250.089
Probability-only0.1660.095
Entropy-only0.2240.128
ForeToken0.6300.547

Importantly, ForeToken is not intended to perfectly classify every token's trajectory. Its objective is to estimate when conflicting tokens are likely to resolve during SFT and use this information to guide selective supervision. Thus, perfect classification is not necessary as long as the predictions help identify useful tokens.

Sensitivity to prediction errors. To examine how these prediction errors affect downstream performance, we compare ForeToken's predicted token selection with post-hoc selection using the actual token trajectories recorded during SFT (Table 3). Despite imperfect predictions, ForeToken achieves performance close to selection using actual trajectories, with only 1.6 points lower AMC and 0.6 points lower AIME P@1. This suggests that the observed prediction errors have a limited impact on downstream performance.

Table 3. Impact of trajectory prediction errors on downstream performance (Qwen3-8B Math, P@1, %).

MethodAMC ↑AIME ↑
Standard SFT29.42.5
ForeToken (Predicted)55.212.5
Post-hoc (Observed)56.813.1

W1 / Q2: Threshold Sensitivity and Transfer Across Models and Domains

To assess this concern, we conducted additional experiments on Qwen3-8B Math using five threshold configurations (Table 4; mean ± standard deviation over three evaluation runs). The results show limited sensitivity across the tested thresholds: AMC P@1 ranges from 54.45 to 56.61 and AIME P@1 from 10.10 to 12.29, while P@16 remains relatively stable (87.17–88.64 on AMC and 43.00–45.78 on AIME).

For models or domains without validation data, we propose percentile-based threshold selection. Our default thresholds (τp = τH = 0.30) correspond to the 14.81st and 88.49th percentiles, respectively. Applying these percentiles to a new model's pretrained distributions provides a validation-free way to adjust thresholds. While this offers a practical approach to threshold transfer, its reliability across different models and domains remains to be evaluated.

Table 4. Threshold sensitivity on Qwen3-8B Math (mean ± standard deviation over three runs, %).

τpτHAIME P@1 ↑AIME P@16 ↑AMC P@1 ↑AMC P@16 ↑
0.300.2510.10 ± 0.3244.44 ± 1.9454.45 ± 0.9587.17 ± 1.44
0.250.3010.83 ± 0.2143.00 ± 0.1255.95 ± 0.7087.50 ± 2.50
0.300.3512.29 ± 1.3044.11 ± 1.9256.61 ± 0.2187.67 ± 1.44
0.350.3011.04 ± 1.0844.78 ± 1.0955.51 ± 0.3388.33 ± 1.82
0.300.3011.12 ± 0.9445.78 ± 0.8555.53 ± 0.4388.64 ± 1.76

Q3: Why Mask Early Conflicts Rather Than Intermediate Conflicts?

Although Early conflicts resolve quickly, this does not necessarily mean they provide useful supervision. As shown in Figure 3, approximately 49% of Early conflicts later re-enter the conflict region, suggesting that their initial resolution is often unstable. To examine this further, we separate Early conflicts into Stable and Reverting groups (Table 5). Masking Reverting conflicts achieves 37.5 AMC and 5.0 AIME, compared with 30.2 AMC and 3.4 AIME when masking Stable conflicts. This supports our hypothesis that instability contributes to the limited usefulness of Early supervision. In contrast, Intermediate conflicts resolve gradually through sustained optimization, suggesting that they provide more useful learning signals.

Table 5. Effect of masking Stable vs. Reverting Early conflicts (Qwen3-8B Math).

Masking StrategyAMC P@1 (%) ↑AIME P@1 (%) ↑MMLU (%) ↑
No Mask (SFT)29.42.571.1
Early-Stable (Resolved)30.23.471.5
Early-Reverting (Re-entered)37.55.071.5

Consistent with this, Table 5 in the paper shows that masking Intermediate conflicts reduces AMC performance (28.6 vs. 29.4 for standard SFT), whereas masking Early conflicts improves it to 38.9. Together, these results support retaining Intermediate conflicts while masking Early conflicts as a more effective supervision strategy.

Dear Reviewer,

Thank you so much for your time and efforts! As the discussion period ends soon, we would appreciate it if you could review our responses and additional experimental results. We’re happy to clarify anything further.

Best regards, The Authors

Thank you for the additional experiments, which address some of my original concerns.

However, after reading the comments from reviewers QxEo, odjK, and g38u, as well as the authors' responses, I share their concerns about the experimental setup. In several domains, even standard SFT fails to improve over the base model, making it difficult to assess whether ForeToken achieves a meaningful adaptation–preservation trade-off rather than simply mitigating performance degradation.

I believe this is a significant limitation of the current evaluation. Therefore, I have decided to lower my rating from 3 to 2.

Reviewer_g38uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ForeToken, a token selection method for supervised fine-tuning that distinguishes confident conflicts according to their resolution behavior. Motivated by the observation that retaining gradually resolved conflicts can outperform retaining rapidly resolved or persistent conflicts, the method estimates resolution tendencies from the model’s initial optimization response and constructs a fixed token mask before fine-tuning.

Critical strengths
  1. The trajectory analysis and masking experiments motivate the proposed selection strategy. The observation that rapidly resolved conflicts are not necessarily the most beneficial training signals is interesting and provides a useful perspective on token selection.

  2. ForeToken estimates conflict-resolution tendencies before fine-tuning, avoiding the need to complete a separate training run to collect trajectories. The resulting fixed loss mask is straightforward to integrate into existing SFT pipelines, although the preprocessing overhead still warrants measurement.

Critical weaknesses
  1. The paper lacks adequate discussion and experimental comparisons with T3S, Token Cleaning, and ssToken, which also exploit training dynamics or model-update responses for token selection. These connections do not negate ForeToken’s novelty, but they make it difficult to establish the incremental value of its proposed conflict-resolution predictor.

  2. The strongest gains occur in mathematical reasoning, whereas the medical and agent results mainly demonstrate mitigation of SFT-induced degradation. The results therefore support an improved adaptation–preservation trade-off more clearly than a general improvement in domain capabilities.

  3. Standard SFT underperforms the starting model in several settings. Although using a common training configuration provides a controlled comparison, additional results with reasonably tuned SFT and competitive token-selection baselines would help determine whether ForeToken’s advantages persist under stronger training configurations.

Questions for authors

Please refer to weaknesses.

4 comments

We thank the reviewer for their valuable suggestions and for recognizing the strengths of our work. We address each concern below and have updated the submitted PDF with additional results and clarifications.

W1: Comparison with Related Token-Selection Methods

Although all three methods perform token-level selection, they address different research problems and use different approaches compared with ForeToken.

Token Cleaning aims to remove uninformative or redundant tokens that can weaken SFT. It scores each token by its loss reduction from the base model to a reference model fine-tuned on a small high-quality subset, and retains the tokens with the largest improvement.

ssToken addresses the limitation of relying only on loss to judge token importance. It measures each token’s loss reduction relative to a history model during training and combines this with an attention-based score, retaining tokens that are both learnable and semantically relevant to the task.

T3S targets continual reasoning distillation, where tokens the model fits quickly (Imitation-Anchor tokens) suppress the learning of other reasoning tokens. It identifies these tokens by their confidence increase between the start of training and the point where training accuracy drops the most, and masks them so the model can focus on the remaining tokens.

In contrast, ForeToken investigates whether all tokens that conflict with pretrained knowledge should be suppressed for capability preservation or retained for downstream adaptation. We show that neither treating all conflicts equally nor simply masking early-learned tokens is sufficient. Instead, we distinguish conflicts by their expected resolution stages and retain Intermediate conflicts while masking Early and Late ones. Unlike these methods, ForeToken forecasts this behavior before SFT begins, without relying on observed training trajectories.

Table 1. Comparison of ForeToken with ssToken and Token Cleaning.

MethodAIMEAMC
SFT2.529.4
ssToken8.650.2
Token Cleaning8.248.3
ForeToken12.555.2
T3SNo public implementation is available

ForeToken outperforms both ssToken and Token Cleaning (Table 1). ssToken and Token Cleaning were mainly evaluated on general instruction-tuning tasks with shorter responses in their original papers.

W2: Adaptation–Preservation Trade-off Across Domains

We agree that ForeToken's largest gains occur in mathematics. The effectiveness of SFT supervision varies across domains. In mathematics, the base model has considerable room for improvement (34.5 AMC), and ForeToken achieves large gains (55.2). In medical and agent tasks, the instruction-tuned base model is already strong, while standard SFT performs worse than the base model. This suggests that the available supervision may provide limited additional benefits while disrupting existing capabilities. Consequently, ForeToken's primary benefit in these settings is reducing harmful degradation, consistently outperforming other fine-tuning methods in capability preservation. Similar SFT-induced degradation has been observed in EAFT [1] and SDFT [2].

References

[1] Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting. Findings of EMNLP, 2026.

[2] Self-Distillation Enables Continual Learning. ICML, 2026.

W3: Robustness to Stronger SFT Configurations

We conducted additional experiments on Qwen3-8B Math, including learning-rate sweeps, validation-based early stopping, and three training seeds for SFT, TALR, and ForeToken.

Learning-rate robustness. To investigate whether our gains arise from an unfavorable SFT configuration, we evaluated SFT, TALR, and ForeToken across four learning rates on Qwen3-8B Math. ForeToken consistently outperforms both SFT and TALR on mathematical reasoning across all tested learning rates (Table 2).

Table 2. Learning-rate sensitivity results on Qwen3-8B Math (%).

Learning RateMetric ↑SFTTALRForeToken
2e-5AMC P@123.939.249.5
AIME P@12.97.313.8
MMLU72.8573.0272.92
5e-5AMC P@124.144.149.8
AIME P@14.06.012.9
MMLU72.1873.2272.57
1e-4AMC P@129.443.455.2
AIME P@12.56.712.5
MMLU72.1073.1073.50
2e-4AMC P@127.839.753.1
AIME P@12.57.111.3
MMLU71.6972.4172.17

Seed robustness. We further evaluated three training seeds (42, 43, 44), and ForeToken maintains its advantage across seeds. Together, the learning-rate sweep and seed experiments show that the mathematical reasoning improvements are not confined to a single learning-rate configuration or training seed (Table 3).

Table 3. Three-seed results at step 1250 (mean ± standard deviation, %).

MethodAMC P@1 ↑AMC P@16 ↑AIME P@1 ↑AIME P@16 ↑
SFT27.33 ± 0.4278.33 ± 3.823.20 ± 0.7520.00 ± 3.83
TALR43.53 ± 1.0783.33 ± 2.897.33 ± 1.1622.23 ± 2.08
ForeToken53.20 ± 2.0189.17 ± 2.8911.07 ± 1.3743.30 ± 3.54

Early stopping analysis. We additionally examined the effect of early stopping using held-out validation data. Stopping earlier slightly improves SFT performance (AMC: 29.4 → 29.7; AIME: 2.5 → 3.3). However, ForeToken still achieves substantially higher performance (AMC: 50.6 vs. 29.7; AIME: 11.0 vs. 3.3).

Overall, ForeToken maintains strong gains over SFT even with learning-rate tuning, validation-based early stopping, and multiple training seeds, demonstrating that its improvements are not simply due to an under-tuned SFT baseline.

Dear Reviewer,

Thank you so much for your time and efforts! As the discussion period ends soon, we would appreciate it if you could review our responses and additional experimental results. We’re happy to clarify anything further.

Best regards, The Authors

The rebuttal partially addresses my concerns. The additional experiments substantially alleviate W3 for Qwen3-8B mathematics, but W1 and W2 remain incompletely resolved. The new ssToken and Token Cleaning comparisons omit key implementation and tuning details and capability-retention results, while T3S remains unevaluated, leaving the claimed adaptation–preservation advantage insufficiently established against these methods. Medical and agent performance remain below the starting model, and the additional tuning experiments do not cover these domains, so neither broader capability improvements nor the adequacy of those baselines has been established.

Reviewer_odjKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies conflicting tokens in supervised fine-tuning, where the ground-truth next token disagrees with a confident prediction from the initial model. It classifies conflicting tokens into Early, Intermediate, and Late learning trajectories and finds that retaining Intermediate conflicts while masking the other two groups gives a better adaptation–retention trade-off. It then proposes FORETOKEN, which uses responses to the initial SFT gradient direction to estimate the trajectory group of each conflicting token before training. Experiments covering adaptation across different domains and sequential adaptation show improvements over standard SFT.

Critical strengths
  1. The analysis of conflicting tokens is interesting. Distinguishing their learning trajectories offers a useful perspective on why some conflicts should be retained while others can be suppressed.

  2. The paper is well organized and generally easy to follow. The progression from empirical observations to the proposed method makes the motivation clear.

  3. The proposed method shows promising results on mathematical reasoning, with substantial improvements over standard SFT and the token-level baselines on Qwen3-8B.

Critical weaknesses
  1. Section 3.2 does not clearly define Mask A–D, which makes Table 1 difficult to reproduce or connect to the later discussions.
  2. The adaptation setup is not fully convincing. In medical and agent adaptation, even FORETOKEN falls below the base model; after sequential fine-tuning, it also underperforms the base on every domain except math. These comparisons mainly show which method causes less damage, leaving successful adaptation untested in these settings. The evaluation should include settings where fine-tuning provides a clear target-domain gain, so that the adaptation–retention trade-off can be meaningfully assessed
  3. All methods use a learning rate of 10e-4, but a shared learning rate does not necessarily provide a fair comparison. Token masking and reweighting change the gradients and may shift the learning rate at which each method performs best. The paper should sweep learning rates for all methods under the same tuning budget and compare their adaptation–retention trade-offs.
Questions for authors
  1. The rationale for masking Early conflicts rests on their instability, yet this group identified in Section 4 also includes tokens that remain resolved. Separating stable and reverting Early tokens would help establish whether instability actually explains the gains.
  2. The post-hoc experiment in Table 10 only masks Late conflicts. Retraining with both Early and Late conflicts identified from a full SFT run would provide a useful reference for assessing the cost of using predicted trajectories.
  3. Section 5 mainly shows that high-scoring tokens resolve early, but the method relies on retaining Intermediate conflicts. How well the predicted Intermediate group matches the observed trajectories remains unclear.
4 comments

We thank the reviewer for their valuable suggestions and for recognizing the strengths of our work. We address each concern below and have updated the submitted PDF with additional results and clarifications.

W1: Definition of Masks A–D

We selected four checkpoints (steps 300, 600, 900, and 1200) during a standard SFT run, identified the conflicting tokens at each checkpoint, and separately trained models while masking the corresponding tokens. Masks A–D correspond to these four checkpoints, respectively.

W2: Adaptation Across Domains

We agree that ForeToken's largest gains occur in mathematics. The effectiveness of SFT supervision varies across domains. In mathematics, the base model has considerable room for improvement (34.5 AMC), and ForeToken achieves large gains (55.2). In medical and agent tasks, the instruction-tuned base model is already strong, while standard SFT performs worse than the base model. This suggests that the available supervision may provide limited additional benefits while disrupting existing capabilities. Consequently, ForeToken's primary benefit in these settings is reducing harmful degradation, consistently outperforming other fine-tuning methods in capability preservation. Similar SFT-induced degradation has been observed in recent works (EAFT [1] and SDFT [2]) in the medical and agent domains.

References

[1] Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting. Findings of EMNLP, 2026.

[2] Self-Distillation Enables Continual Learning. ICML, 2026.

W3: Learning-Rate Sensitivity

We evaluated SFT, TALR, and ForeToken across four learning rates (2e-5, 5e-5, 1e-4, and 2e-4) on Qwen3-8B Math. ForeToken consistently outperforms both baselines at every tested learning rate. Even when comparing each method's best results across the sweep, ForeToken achieves 55.2 vs. 44.1 AMC P@1 and 13.8 vs. 7.3 AIME P@1 against the competing baseline, TALR. These results show that ForeToken's gains are not due to an unfavorable learning-rate choice for the baselines (Table 1).

Table 1. Learning-rate sensitivity results for SFT, TALR, and ForeToken on Qwen3-8B Math.

Learning RateMetric ↑SFTTALRForeToken
2e-5AMC P@123.939.249.5
AIME P@12.97.313.8
MMLU72.8573.0272.92
5e-5AMC P@124.144.149.8
AIME P@14.06.012.9
MMLU72.1873.2272.57
1e-4AMC P@129.443.455.2
AIME P@12.56.712.5
MMLU72.1073.1073.50
2e-4AMC P@127.839.753.1
AIME P@12.57.111.3
MMLU71.6972.4172.17

Q1: Masking Early-Stable vs. Early-Reverting Conflicts

We divide Early conflicts into Stable tokens, which remain resolved, and Reverting tokens, which re-enter the conflict region after initially resolving. These groups represent approximately 51% and 49% of Early conflicts, respectively. Masking Reverting tokens achieves 37.5 AMC and 5.0 AIME, clearly outperforming masking Stable tokens (30.2 AMC and 3.4 AIME), while both achieve identical MMLU performance (71.5). These findings support our hypothesis that instability, rather than early resolution alone, contributes to the benefits of masking Early conflicts (Table 2).

Table 2. Effect of masking Stable vs. Reverting Early conflicts (Qwen3-8B Math).

Masking StrategyAMC P@1 (%) ↑AIME P@1 (%) ↑MMLU (%) ↑
No Mask (SFT)29.42.571.1
Early-Stable (Resolved)30.23.471.5
Early-Reverting (Re-entered)37.55.071.5

Q2: Post-hoc Early + Late Masking Comparison

We conducted an additional post-hoc Early + Late masking experiment, identifying both groups using their complete observed trajectories from a preceding SFT run (Table 3).

Table 3. Predicted vs. observed conflict masking (Pass@1, %).

MethodAMC ↑AIME ↑
Standard SFT29.42.5
ForeToken (Predicted)55.212.5
Post-hoc (Observed)56.813.1

Post-hoc masking achieves 56.8 AMC and 13.1 AIME, while ForeToken achieves 55.2 and 12.5, only 1.6 and 0.6 points lower, respectively. These results show that ForeToken captures most of the benefits of trajectory-aware masking without requiring an additional full SFT run to observe token trajectories.

Q3: Accuracy of Intermediate-Conflict Identification

To evaluate how well ForeToken's predicted Intermediate group matches the observed trajectories, we compare its predictions with the actual trajectories recorded during SFT (Table 4). ForeToken correctly identifies 20,779 of 26,000 Intermediate tokens on Qwen3-8B Code and 19,952 of 28,951 on DeepSeek-R1-Qwen3-8B Math. We further compare ForeToken with simpler random, probability-based, and entropy-based predictors (Table 5), showing its advantage in identifying Intermediate conflicts.

Table 4. Counts of Intermediate conflicts identified and missed by ForeToken across models and domains.

MetricQwen3-8B (Code)DeepSeek-R1-Qwen3-8B (Math)
Actual Intermediate26,00028,951
Correctly identified20,77919,952
Missed5,2218,999

Table 5. Intermediate group F1 comparison with simpler predictors across models and domains.

MethodQwen3-8B (Code)DeepSeek-R1-Qwen3-8B (Math)
Random0.1250.089
Probability-only0.1660.095
Entropy-only0.2240.128
ForeToken0.6300.547

Importantly, ForeToken is not intended to perfectly classify every token's trajectory. Its objective is to estimate when conflicting tokens are likely to resolve during SFT and use this information to guide selective supervision. Thus, perfect classification is not necessary as long as the predictions help identify useful tokens.

Dear Reviewer,

Thank you so much for your time and efforts! As the discussion period ends soon, we would appreciate it if you could review our responses and additional experimental results. We’re happy to clarify anything further.

Best regards, The Authors

Thank you for the authors' response. As my concern on performance degradation in medical and agent adaptation remains, I would like to keep my score.

Reviewer_QxEoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForeToken selects which conflicting tokens to learn during SFT. It estimates their learning trajectories from the initial optimization direction, then keeps predicted intermediate conflicts while masking early and late ones.

Critical strengths
  1. We find the observation that conflicting tokens behave differently useful. The masking ablations go beyond treating all conflicts the same way.

  2. The evaluation across multiple models and domains. The mathematical reasoning gains on Qwen3-8B are great.

Critical weaknesses
  1. We are not fully convinced by the trajectory interpretation. Figure 4 mainly shows that high scores identify early-resolving tokens, while Appendix D acknowledges substantial overlap between groups.Comparisons with simpler probability-, entropy-, or gradient-response selection at matched masking rates would help establish what trajectory forecasting adds.

  2. The broader adaptation claim is less convincing than the math results. In Table 4, medical and agent performance remain below the base model, so these results mainly show reduced degradation. The Llama results are also mixed, since standard SFT itself often underperforms the base model, we would like stronger evidence that the gains persist against well-tuned SFT rather than mainly correcting an unfavorable training setup.

Questions for authors
  1. How accurately does the forecast identify the intermediate group across models and domains

  2. Does it outperform simpler token-selection rules at the same masking rate?

  3. Do the gains hold with tuned learning rates, early stopping, and multiple training seeds?

3 comments

We thank the reviewer for their valuable suggestions and for recognizing the strengths of our work. We address each concern below and have updated the submitted PDF with additional results and clarifications.

Q1: Accuracy of Intermediate-Conflict Identification We evaluate how accurately ForeToken identifies the Intermediate group by comparing its predictions with the actual trajectories observed during SFT (Table 1). ForeToken correctly identifies 20,779 of 26,000 Intermediate tokens on Qwen3-8B Code and 19,952 of 28,951 on DeepSeek-R1-Qwen3-8B Math. We further compare ForeToken with simpler random, probability-based, and entropy-based predictors (Table 2), showing its advantage in identifying Intermediate conflicts.

Table 1. Counts of Intermediate conflicts identified and missed by ForeToken across models and domains.

MetricQwen3-8B (Code)DeepSeek-R1-Qwen3-8B (Math)
Actual Intermediate26,00028,951
Correctly identified20,77919,952
Missed5,2218,999

Table 2. Intermediate group F1 comparison with simpler predictors across models and domains.

MethodQwen3-8B (Code)DeepSeek-R1-Qwen3-8B (Math)
Random0.1250.089
Probability-only0.1660.095
Entropy-only0.2240.128
ForeToken0.6300.547

Importantly, ForeToken is not intended to perfectly classify every token's trajectory. Its objective is to estimate when conflicting tokens are likely to resolve during SFT and use this information to guide selective supervision. Thus, perfect classification is not necessary as long as the predictions help identify useful tokens.

W1 / Q2: Comparison with Simpler Token-Selection Methods

ForeToken aims to estimate when conflicting tokens are likely to resolve during SFT, rather than perfectly separate them into distinct classes. To establish what trajectory forecasting adds, we evaluate downstream performance against probability-based, entropy-based, and gradient-response selection at the same masking rates as ForeToken (Table 3).

Table 3. Downstream performance at matched masking rates (Qwen3-8B Math).

Token Selection MethodAMC Pass@1 (%) ↑AIME Pass@1 (%) ↑
Probability-only46.710.5
Entropy-only49.410.9
Gradient-response51.09.5
ForeToken55.212.5

ForeToken outperforms all simpler token-selection methods, achieving 55.2% AMC Pass@1 and 12.5% AIME Pass@1, compared with 46.7%/10.5% for probability-only and 49.4%/10.9% for entropy-only selection. As shown in Table 2, simpler methods are less accurate at identifying Intermediate conflicts, which helps explain their lower downstream performance.

W2: Broader Adaptation and Llama Performance

We agree that ForeToken's largest gains occur in mathematics. The effectiveness of SFT supervision varies across domains. In mathematics, the base model has considerable room for improvement (34.5 AMC), and ForeToken achieves large gains (55.2). In medical and agent tasks, the instruction-tuned base model is already strong, while standard SFT performs worse than the base model. This suggests that the available supervision may provide limited additional benefits while weakening existing capabilities. Consequently, ForeToken's primary benefit in these settings is reducing harmful degradation, consistently outperforming other fine-tuning methods in capability preservation. Similar SFT-induced degradation has been observed in recent works (EAFT [1], SDFT [2]) in the medical and agent domain.

Llama-specific performance. On Llama-3.1-8B-Instruct, ForeToken achieves mathematical performance close to the strongest baseline (35.9 vs. 36.4), while attaining the highest general-domain average among fine-tuned methods (69.3). This suggests an adaptation–preservation trade-off: ForeToken prioritizes selective supervision to limit harmful updates, which may provide greater benefits for capability preservation than for target-domain improvement on this backbone. Standard SFT also struggles to improve mathematical performance over the pretrained Llama model, indicating that effective adaptation remains challenging in this setting. Similar backbone-dependent behavior has been observed in LP-SFT [3] and TokenSeek [4].

Q3: Learning-Rate Tuning, Early Stopping, and Multiple Seeds

To address the reviewer’s concerns about SFT tuning and robustness, we conducted additional experiments involving learning-rate sweeps, early stopping, and multiple training seeds for SFT, TALR, and ForeToken.

Learning-rate robustness. To investigate whether our gains come from an unfavorable SFT configuration, we evaluated SFT, TALR, and ForeToken across four learning rates on Qwen3-8B Math. ForeToken consistently outperforms both SFT and TALR across all tested learning rates (Table 4).

Table 4. Learning-rate sensitivity results for SFT, TALR, and ForeToken on Qwen3-8B Math.

Learning RateMetric ↑SFTTALRForeToken
2e-5AMC P@123.939.249.5
AIME P@12.97.313.8
MMLU72.8573.0272.92
5e-5AMC P@124.144.149.8
AIME P@14.06.012.9
MMLU72.1873.2272.57
1e-4AMC P@129.443.455.2
AIME P@12.56.712.5
MMLU72.1073.1073.50
2e-4AMC P@127.839.753.1
AIME P@12.57.111.3
MMLU71.6972.4172.17

Seed robustness. We evaluated three training seeds (42, 43, 44), and ForeToken maintains its advantage across seeds. Together, the learning-rate sweep and seed experiments show that the mathematical reasoning improvements are not confined to a single learning-rate configuration or training seed (Table 5).

Table 5. Three-seed results (mean ± standard deviation, %).

MethodAMC P@1 ↑AMC P@16 ↑AIME P@1 ↑AIME P@16 ↑
SFT27.33 ± 0.4278.33 ± 3.823.20 ± 0.7520.00 ± 3.83
TALR43.53 ± 1.0783.33 ± 2.897.33 ± 1.1622.23 ± 2.08
ForeToken53.20 ± 2.0189.17 ± 2.8911.07 ± 1.3743.30 ± 3.54

Early stopping analysis. We additionally examined the effect of early stopping using held-out validation data. Stopping earlier slightly improves SFT performance (AMC: 29.4 → 29.7; AIME: 2.5 → 3.3). However, ForeToken still achieves considerably higher performance (AMC: 50.6 vs. 29.7; AIME: 11.0 vs. 3.3).

Overall, ForeToken maintains strong gains over SFT even with learning-rate tuning, early stopping, and multiple training seeds, demonstrating that its improvements are not simply due to an under-tuned SFT baseline.

References

[1] Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting. Findings of EMNLP, 2026.

[2] Self-Distillation Enables Continual Learning. ICML, 2026.

[3] LP-SFT: Local-Preserving Supervised Fine-Tuning via Multimodal Entropy Structure.

[4] TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching. ICLR, 2026.

Dear Reviewer,

Thank you so much for your time and efforts! As the discussion period ends soon, we would appreciate it if you could review our responses and additional experimental results. We’re happy to clarify anything further.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.