acceptodds
Under review as a conference paper at ICLR 2027

Today's Targets, Tomorrow's Context: Measuring Token-Level Future Influence in SFT

Abstract

Reasoning supervised fine-tuning (SFT) typically filters entire trajectories based on final-answer correctness, yet applies direct supervision to every response token in each retained trajectory. Consequently, trajectory-level verification cannot determine whether every token is a suitable training target. In an autoregressive model, the same token serves first as a prediction target and, once generated, as context for subsequent predictions. Directly supervising a token increases the model's tendency to generate it in the future. However, even in a trajectory that reaches the correct answer, some tokens may, as context, make the fixed demonstrated continuation harder for the current model to predict. Motivated by this observation, we introduce Future Influence, which measures the signed effect of each response token as context on subsequent prediction and uses this direction to select token-level supervision. Specifically, we attenuate the information a token transmits to later positions and determine the direction of its effect from the resulting change in prediction loss on the fixed demonstrated continuation. Experiments show that negatively directed tokens are common even in correct reasoning trajectories. Supervision selected by Future Influence consistently improves reasoning performance across model scales, training datasets, and evaluations. Our findings show that final-answer correctness does not imply that every token in a trajectory should be directly reinforced, and that accounting for tokens' roles as future context provides a new perspective on constructing more effective supervision for SFT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning SFT, Token-Level Supervision, Future Contextual Influence, Process Supervision, Token Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r95XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Future Influence, a token-selection criterion for reasoning SFT. It measures how decreasing a token’s information affects the prediction loss on the fixed teacher-forcing continuation. The scores are computed offline, and tokens with negative scores have their direct prediction losses masked while remaining available as context. Experiments across four models and two mathematical reasoning datasets report improvements over standard SFT and several token-level supervision baselines.

Critical strengths
  1. The idea and motivation are clear. Tokens have no contribution to later reasoning are masked out during SFT supervision.
  2. The computation of Future Influence is offline, and can be done in one forward pass and one backward pass, which is acceptable.
  3. The evaluation is broad and covers most of the important datasets for math reasoning, with avg@1 and pass@32.
Critical weaknesses
  1. A negative score does not necessarily mean a token should not be learned. Reasoning often includes trying an idea, finding a mistake, and changing direction. A token may help the current reasoning segment but make later segments harder to predict. Its total score could therefore be negative even when it belongs to a useful reasoning step.
  2. The novelty of the contextual contribution measure is unclear. Measuring how context affects model predictions is already an established research problem. Value Zeroing [1] measures a token’s contribution by zeroing its value vectors while keeping the input tokens unchanged. Attribution patching [2] further shows how gradients can efficiently approximate the effects of internal interventions. These methods already cover the main idea of measuring contextual influence through interventions. The paper’s potential contribution therefore appears to lie in its particular signed continuation-loss score and how this score is used to select SFT supervision. It should explain what this measure adds to existing context attribution methods and show why it provides a better signal for supervision selection.
  3. The ablations do not test the main design choices. Random, Low-NLL, and Reverse masking are useful controls, but they do not explain why the proposed attribution design is preferable. The paper does not compare scoring the full continuation against scoring only the final answer, or selecting individual tokens against selecting reasoning segments [3]. These comparisons would help show which choices produce the reported gains.

[1] Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. “Quantifying Context Mixing in Transformers.” EACL, 2023, pp. 3378–3400.
[2] Aaquib Syed, Can Rager, and Arthur Conmy. “Attribution Patching Outperforms Automated Circuit Discovery.” BlackboxNLP, 2024, pp. 407–416.
[3] Siyuan Wang, Yanchen Liu, and Xiang Ren. “Segment-Level Attribution for Selective Learning of Long Reasoning Traces.” ICLR, 2026.

Questions for authors
  1. Can you examine tokens that help their own reasoning segment but have a negative total influence on later segments? Do these tokens appear in useful exploration or self-correction, and does removing their supervision help? How do results change when scoring only the final answer, or selecting whole segments [3]?
  2. What does the gate-gradient estimator add beyond existing attribution methods [1, 2]?
  3. Please also report actual scoring time and total scoring-plus-training cost relative to standard SFT. Are they still acceptable for larger scale training?
6 comments

Response to Reviewer r95X (1/2)

Thank you for the review. We respond to each point below.

W1 / Q1 (first half): Tokens in exploration and self-correction

We computed whole-word masking rates from the masks of Qwen3-4B/8B/14B-Base on both corpora (6 settings, first 1,000 examples each). Words that turn the reasoning during exploration and self-correction are mostly kept, consistently across all 6 settings. Transition and hesitation words (however, hmm, maybe, alternatively, wait, ...) have a masking rate of 0.27–0.36, and words that open the next step (let me, another, ...) 0.27–0.32. In contrast, words that evaluate the previous step (easier, complicated, ...) have a masking rate of 0.73–0.86, and words that look back (earlier, already, result, ...) 0.73–0.85.

In the excerpt below, the reasoning abandons one approach and switches to another (Qwen3-14B-Base, OpenMathReasoning; masked tokens in [[ ]]):

Wait[[,]] maybe[[ that]]'s[[ getting complicated.]] Alternatively[[,]] maybe[[ there's]] a better formula[[ specific to polar]] coordinates[[.]]

Wait[[,]] another[[ approach]]: parametrize the curve[[ in Cartesian coordinates and]] then[[ use the surface]] of[[ revolution]] formula[[.]] Let me[[ try that]].

The turn itself ("Wait", "maybe", "Alternatively", "another", "Let me") is supervised; the comment on the previous step ("getting complicated") is not.

The case you describe can occur, since sis_i is the net effect on the whole continuation. We use the net effect because once the model generates the token, both the current segment and later segments are conditioned on it, and the SFT loss sums over all of them. Masking also does not take the token out of the context. The rest of its segment is still supervised with it as input, so its help to the current segment is kept in training; what is removed is only the signal that teaches the model to generate it. Given the masking rates above, at most 27–36% of turning words could fall into this case.

Removing the supervision of negative tokens does help. Our method changes SFT only in this one respect, and its avg@1 is above SFT in all 48 cells. Reverse supervises only the negative tokens and has the lowest mean avg@1 in all 8 settings (see W2).

W3 / Q1 (second half): Scoring only the final answer, or selecting by segment

We are running both ablations on Qwen3-8B-Base and Qwen3-14B-Base with OpenR1-Math-220k. Answer-only scoring replaces LrespL_\mathrm{resp} in scoring with the loss on the final-answer tokens only. Segment-level selection splits responses into paragraphs and keeps or masks each paragraph by the sign of its summed sis_i, matching the selection granularity of [3]. Each changes only the scoring or the mask, with training and evaluation unchanged, and we will post the results during the discussion period.

The answer loss is part of LrespL_\mathrm{resp}, so scoring the full continuation and scoring only the answer measure the same kind of effect; the difference is how reliable the estimate is. The final answer of a long CoT is only a few tokens, so with answer-only scoring each token's score rests on those few targets and has high variance. Under teacher forcing the answer depends mostly on the derivation right before it; the number in \boxed{} is often just the result of the previous line. An early token sits thousands of tokens before the answer, so its effect on the answer loss is close to zero and its sign is easily dominated by noise. With full-continuation scoring, each token's score sums the loss changes of all targets after it, so early tokens also have many nearby targets that provide signal.

We select per token because directions alternate heavily within segments. Splitting by paragraph (same 6 settings, segments of at least 5 tokens), the minority direction makes up at least 30% of tokens in 88.9–90.4% of segments, and deciding each segment by its summed sis_i would flip the decision for 45.4–45.6% of tokens. In the excerpt above, "Wait" is kept while the comma right after it is masked, and "another" is kept while "approach" is masked. Segment-level selection cannot tell these apart.

We will add [3] to Related Work and include both ablations in the revision.

(References are listed at the end of Part 2/2.)

Response to Reviewer r95X (2/2)

W2 / Q2: What this adds beyond Value Zeroing and Attribution Patching

We agree that measuring the role of context by intervention is an established problem. Our gate multiplies the value vectors, so gi=0g_i=0 is Value Zeroing [1] applied to token ii in all layers, and approximating the intervention with the gradient at g=1g=\mathbf{1} belongs to the same family as Attribution Patching [2]. We do not claim the estimator as a contribution.

[1,2] ask which context a trained model uses for a given prediction. We ask which tokens of a correct demonstration should be directly reinforced. Our contribution is turning the role of context into a supervision signal for SFT. First, attribution methods do not say why a token's role as context should decide whether it is trained as a target. Our basis is that under teacher forcing, today's target is tomorrow's context. Supervising a token makes the model generate it, after which it is context for all later predictions. If it makes the correct continuation harder to predict, reinforcing it teaches the model to enter a worse state. So we measure the signed effect on the continuation loss, which is the SFT objective itself.

Second, the training scheme handles the target role and the context role separately: negative tokens lose only their direct loss and remain as context. Scoring uses the same base checkpoint that is trained, needs no reference model, and runs once offline. Only the sign is used, so there is no retention ratio or threshold. Third, even in trajectories with correct answers, 50.21–51.83% of tokens are negative, spread roughly evenly over positions. A correct answer does not mean every token deserves direct reinforcement.

On whether it is a better supervision signal: under the same SFT setup, our method has the highest mean avg@1 in all 8 settings. Besides Random, Low-NLL, and Reverse, the baselines include four published methods built on different signals: AlphaToken (gradient alignment), Token Cleaning (loss differences), ssToken (training dynamics and attention), and DFT (target-probability weighting). Using the same scores in reverse (Reverse) is the worst in all 8 settings, 1.58–8.82 points below ours, so the useful information is in the sign.

We will add [1,2] to Related Work and rewrite the contributions in the introduction along these three points.

Q3: Scoring time and total cost

Measured times in hours on one 8×H100 node:

ModelCorpusScoringStandard SFTMasked SFTScoring / SFT(Scoring + Masked SFT) / SFT
Qwen3-4B-BaseOpenMath4.257.107.090.601.60
Qwen3-4B-BaseOpenR13.345.365.350.621.62
Qwen3-8B-BaseOpenMath4.7416.7717.820.281.35
Qwen3-8B-BaseOpenR13.0712.7612.760.241.24
Qwen3-14B-BaseOpenMath12.5516.1516.270.781.78
Qwen3-14B-BaseOpenR18.2612.0512.000.691.68

Scoring takes 0.24–0.78× one run of standard SFT, and scoring plus training takes 1.24–1.78×. Masked training takes about as long as standard SFT, because masked tokens still go through the forward and backward passes and are only left out of the loss. At larger scale, scoring is independent across examples and fully parallel, and its time grows linearly with data size, as SFT does. The mask is computed once per model and dataset and can be reused across epochs, hyperparameter tuning, and repeated runs. We train for only one epoch, so the table shows the least favorable case.

We will add this table to the appendix.

References

[1] Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. "Quantifying Context Mixing in Transformers." EACL, 2023, pp. 3378–3400.
[2] Aaquib Syed, Can Rager, and Arthur Conmy. "Attribution Patching Outperforms Automated Circuit Discovery." BlackboxNLP, 2024, pp. 407–416.
[3] Siyuan Wang, Yanchen Liu, and Xiang Ren. "Segment-Level Attribution for Selective Learning of Long Reasoning Traces." ICLR, 2026.

Thank you for the rebuttal.

W1. I am not convinced. My question was about tokens that help their own segment but get a negative net score because of later segments, for example in exploration or self-correction. The response does not identify such tokens and only reports masking rates for a few word types.

W2. Thank you for clarifying that the estimator is not a contribution. However, I am still not convinced by the novelty of the remaining contribution. Appendix C.2 describes AlphaToken as also scoring a token’s causal influence on later predictions through value propagation to select supervision. How does your criterion differ from this, beyond using only the sign and computing it offline?

W3. I look forward to the results for answer-only scoring and segment-level selection that you committed to posting.

Suggestion. I would suggest that the authors write the remaining responses themselves, without AI drafting or polishing, in clear and well-organized English that answers each question directly.

Reviewer_1JDsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional SFT overlooks the contextual state nature of tokens in autoregressive generation, assuming that every token in a trajectory with a correct final answer deserves direct supervision. This paper proposes Future Influence to address this issue. The method efficiently estimates the directional impact of each token on subsequent generation via a differentiable gating mechanism, and selectively masks the direct loss of negative tokens that hinder future prediction. Experiments show that this fine-grained filtering mechanism consistently outperforms standard SFT across different model scales and datasets.

Critical strengths

1 Fine-grained supervision selection. This paper moves beyond the coarse practice of full-token supervision based solely on final-answer correctness, and instead selects tokens that genuinely support the model's reasoning path from a contextual-state perspective. 2 Computational efficiency. The proposed gate-gradient approximation estimates the influence directions of all response tokens within a single forward and backward pass, avoiding the linear overhead that would otherwise make it impractical for long CoT trajectories. 3 Consistent performance gains. Experiments demonstrate that negatively influential tokens are common even in correct reasoning trajectories, and masking their direct supervision yields consistent improvements over standard SFT across model scales and datasets.

Critical weaknesses

1 The gate gradient is only a first-order approximation, and higher-order interactions are not sufficiently analyzed. If a token's contribution depends strongly on coordination with other tokens—for example, being negligible in isolation but critical in combination—the gate gradient may underestimate or misjudge its direction.

2 Comparison is insufficient. The absence of direct comparisons with methods such as CFT (ENHANCING LARGE LANGUAGE MODEL REASONING VIA SELECTIVE CRITICAL TOKEN FINE-TUNING) makes it difficult to fully establish the paper's novelty.

3 The relationship with token-level credit assignment in RL is not adequately discussed. Methods such as PPO and GRPO, as well as some token-level advantage allocation approaches, also attempt to decompose sequence-level signals into finer-grained supervision.

4 The writing needs improvement. Several citations appear to be missing, e.g., on lines 57, 451, and 452. Some sentences are also unclear; for example, "Even in a trajectory reaching the correct answer, some tokens, as context, may make the fixed demonstrated continuation harder for the current model to predict" could be expressed more directly.

Questions for authors

1 Since si is a first-order marginal sensitivity, can the authors show whether it stays reliable for tokens whose value is purely synergistic (near-useless alone, critical jointly)?

2 Could the authors add comparison with CFT and justify why gate-gradient direction and counterfactual criticality differ?

3 The gate-gradient sign is essentially a token-level credit signal. How do the authors distinguish it from token-level advantage schemes in PPO/GRPO along signal source and compute?

4 comments

Response to Reviewer 1JDs (1/2)

Thank you for the review. We respond to each point below.

W1 / Q1: First-order approximation and synergistic tokens

sis_i is the derivative at g=1g=\mathbf{1}, where every other gate is 1. It measures what happens when token ii is weakened while all of its partners are present, so no token is scored in isolation. If two tokens matter only together, the relevant part of −Lresp-L_\mathrm{resp} can be written as c gagbc\,g_a g_b. Then sa=c gbs_a = c\,g_b, which equals cc at g=1g=\mathbf{1}, the effect of the whole combination. This is also the quantity training needs. The mask removes only a token's direct loss and never takes the token out of the context, so its partners are always present during training.

Words split into several tokens are a typical synergistic case, since a single subword usually has no complete meaning, so we ran a new experiment on them. On the same setup as Table 2 (Qwen3-14B-Base and Gemma3-12B, 1,000 random examples per corpus), all tokens of a multi-token word share the same sign in 78% of such words (82% for 2-token words, 61% for 3 or more). If signs were independent, these figures would be about 50% and at most 25%. For these words, our method in effect makes one decision for the whole word. We removed all tokens of each such word from the sequence at once. The measured direction agrees with the shared sign for 96% of words, slightly above the 94.25–95.41% for single-token removal. The first-order direction still holds when a group of synergistic tokens is removed together.

Where a first-order estimate does fall short is redundancy. If either of two tokens is enough, the relevant part is c [1−(1−ga)(1−gb)]c\,[1-(1-g_a)(1-g_b)], and sa=c (1−gb)s_a = c\,(1-g_b) is 0 at g=1g=\mathbf{1}, so it misses their joint effect. We will state this in the Limitations and add the whole-word results to Appendix F.

W2 / Q2: Comparison with CFT

CFT [1] replaces a token in the model's own trajectory, regenerates from that position, and labels the token critical if the final answer becomes wrong. Only critical tokens are then supervised. We keep the demonstrated continuation fixed, weaken the information the token passes forward, and check whether the loss on this continuation goes up or down.

The two criteria ask different questions. CFT asks whether a token is necessary for the outcome. We ask whether, as context, it makes the demonstrated continuation easier or harder to predict. A token whose replacement leaves the answer unchanged gets no supervision in CFT, but keeps it under our criterion if it makes the continuation easier to predict. They also take different forms: CFT picks out a small set of critical tokens, while we use a two-sided sign and drop the direct loss only for negative tokens (50.21–51.83%).

We did not run CFT as a baseline because it has no faithful implementation in our setting. Its labels require a model to regenerate from its own trajectories; even in its offline transfer, the model that generated the data first labels its own responses. Our corpora are long CoT generated by external models. OpenR1-Math-220k, for example, comes from DeepSeek-R1, with responses up to 16k tokens. If the base model we train regenerates from a position in such a demonstration, it produces its own continuation, which no longer reflects the token's role in the demonstration. It is also computationally infeasible. CFT reports 0.5 hours on 8 A100s to label 1,000 GSM8K responses of about 240 tokens, per-position regeneration grows at least quadratically with length, and we have 100k trajectories of up to 16k tokens.

We will add CFT to Related Work with these differences.

(References are listed at the end of Part 2/2.)

Response to Reviewer 1JDs (2/2)

W3 / Q3: Relation to token-level credit assignment in RL

PPO [2], GRPO [3], and token-level advantage methods such as VinePPO [4] and PRIME [5] distribute differences in outcome reward over the tokens of trajectories sampled from the current policy. Like them, we break a sequence-level signal down to tokens, but the signal source and the compute are different.

On the signal source, RL needs sampled trajectories that receive different rewards. Our trajectories come from an external model and were all filtered for correct answers, so as RL data they would all get the same reward; GRPO's group advantages, for example, would all be zero. Our signal is the change in teacher-forced loss on the fixed demonstration when a token's forward information is weakened, and no reward is involved.

On compute, RL samples each problem many times and keeps querying value or reward models during training, so the signal changes as the policy updates. We run one forward and one backward pass per example with the frozen base model, once and offline, and the mask stays fixed during training.

We will add this discussion to Related Work.

W4: Writing

Thank you for pointing these out. The missing citations at lines 57, 451, and 452 are AlphaToken [6] and DFT [7], whose bib entries were left out of the submission; the revision restores them. We will also recheck the whole paper and polish any unclear sentences.

References

[1] Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. "Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning." arXiv:2510.10974, 2025.
[2] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. "Proximal Policy Optimization Algorithms." arXiv:1707.06347, 2017.
[3] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300, 2024.
[4] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. "VinePPO: Refining Credit Assignment in RL Training of LLMs." ICML, 2025.
[5] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, et al. "Process Reinforcement through Implicit Rewards." arXiv:2502.01456, 2025.
[6] Liu Qing, Ou Wu, and Yi Du. "AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training." arXiv:2606.01635, 2026.
[7] Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. "On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification." arXiv:2508.05629, 2026.

Thanks for the detailed response. The rebuttal clarifies the method substantially, especially the distinction from token-level credit assignment in RL.

I still have two remaining concerns. First, the multi-token-word experiment is helpful, but it does not fully address higher-order interactions, especially cases where constituent tokens have different signs. Second, the distinction from CFT is clearer now, but I still think a direct comparison on a smaller controlled subset would strengthen the paper.

Overall, several concerns are addressed, but these two points remain.

Thank you for the follow-up. We address the two remaining points below.

Constituent tokens with different signs and higher-order interactions. In the whole-word experiment, 22% of multi-token words contain tokens of different signs (18% of 2-token words and 39% of words with 3 or more tokens). For these words, our method does not make a word-level decision: it keeps the positive tokens and masks the negative ones. We agree that the first-order estimate sis_i does not capture interactions among these tokens. Such interactions are described by second-order terms. In the Hessian of LrespL_\mathrm{resp} with respect to the gates at g=1g=\mathbf{1}, the cross term ∂2L/∂gi∂gj\partial^2 L/\partial g_i\partial g_j measures the interaction between tokens ii and jj. This includes the redundancy case noted in our previous response: when two tokens can substitute for each other, each first-order derivative vanishes, but their cross term does not. For a word or any group of tokens, the second derivative along the group direction can be computed exactly with a single Hessian-vector product, without forming the Hessian, at a cost comparable to one gradient computation. In the revision, we will add this second-order correction, examine how it changes the criterion for mixed-sign words and cross-word interactions, and state the scope of the first-order criterion in the Limitations.

Direct comparison with CFT on a smaller controlled subset. We considered this comparison carefully, but we do not think it would be informative, for three reasons.

First, a faithful CFT [1] changes the training data as well as the method. CFT trains only on problems the model solves under greedy decoding, using the model's own responses. Our training data are offline long CoT generated by external models; OpenR1-Math-220k, for example, comes from DeepSeek-R1. Running CFT as specified would change both the problem set and the training trajectories, so the comparison would confound data with method and would no longer be controlled. If the base model instead regenerates from positions in our demonstrations, it produces its own continuations, which no longer measure the token's role in the demonstration.

Second, annotation is costly even on a small subset. For every position, CFT replaces the original token with its second- and third-most probable alternatives and greedily decodes to the end. Responses in our OpenR1 training set average about 4.4k tokens, so each example requires generating roughly 2×1072\times10^7 tokens. Even 200 examples require about 4×1094\times10^9 generated tokens, and to remain faithful to the demonstrations, the generator should be DeepSeek-R1.

Third, at an affordable subset size, the comparison lacks statistical power. With a few hundred SFT examples for an 8B or 14B model, differences between methods are hard to detect on benchmarks such as AIME, HMMT, and AMC, where a single problem accounts for 2.5–3.3 percentage points. At a scale large enough to detect differences, the annotation cost grows linearly with the number of examples.

We therefore did not run CFT on a subset. In the revised Related Work, we will state the differences between the two methods in signal source, applicability, and form of the criterion.

References

[1] Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, and Guanhua Chen. "Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning." arXiv:2510.10974, 2025.

Reviewer_PT9MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Future Influence, a criterion for deciding which tokens receive direct supervision in reasoning SFT. The authors attach a scalar gate to each token's value vector in attention and use the sign of the gradient with respect to this gate to judge whether the token, as context, helps predict the content that follows. Tokens with negative influence have their target loss masked but are kept as input. Qwen3 and Gemma3 models are trained on two math reasoning corpora and evaluated on six competition math benchmarks with pass@32 as the main metric; the method outperforms SFT and several token-selection baselines throughout.

Critical strengths
  • Simple to implement, with low overhead. Scores are obtained offline from a frozen model with a single gradient computation with respect to the scalar gates; training only changes the loss mask and leaves the pipeline untouched, so the method is easy to reproduce in existing SFT frameworks.
  • The controls are well designed. Reverse inverts the retention rule using the same scores and yields the lowest mean pass@32 in all eight model–corpus settings (Table 1).
  • The experimental coverage is broad: two model families, three scales from 4B to 14B, two training corpora, comparisons against four published token-level methods, and recent benchmarks such as AIME26 and HMMT Feb 2026, which reduces contamination concerns.
Critical weaknesses

1. The core assumption is not justified

sis_i measures the influence of yiy_i as context on the subsequent −log⁡p-\log p, whereas the mask acts on the loss of yiy_i as a target, i.e., on whether the model learns to generate yiy_i. The paper does not explain why the sign of the former should determine the latter.

2. The gains are not significant enough to be convincing

The margins in the main experiments are generally small. AIME has 30 problems per year, AMC23 has 40, and each HMMT set has about 30, so a single problem corresponds to 2.5 to 3.3 percentage points. Converting Table 1 accordingly, in 37 of the 48 combinations the method leads the strongest baseline by exactly one problem, and its lead over SFT is mostly two to three problems, especially considering that this is under pass@32. On evaluation sets of this size, the conclusions drawn in the main text are too strong.

3. The writing needs polishing

The writing as a whole could be polished further: a substantial part of the main text is spent restating experimental observations, leaving relatively little room for discussion. I encourage the authors to revise the presentation throughout. Specific points including:

  • The citations for AlphaToken and DFT appear as "(?)" in several places, and AlphaToken is missing from the references. According to Appendix C.2 it is the work closest to this paper, so it should be cited and the differences stated.
  • The main-text discussion of Table 1, Table 8, and Figure 3 largely restates the results ("best everywhere", "approximately uniform across positions"). Such descriptions could be moved to the appendix, leaving the main text for discussion.
Questions for authors
  • Could you show a few representative traces with the masked and retained tokens marked, and briefly describe what kinds of content they roughly correspond to?
  • Section 2.4 maps "does not support continuation as context" directly to "should not be supervised as a target". Could you further explain the basis for this correspondence, and under what circumstances it might not hold?
  • If compute allows, could you add a comparison on one model using non-math SFT data ?
4 comments

Response to Reviewer PT9M (1/2)

Thank you for the review. We respond to each point below.

W1 / Q2: Why the sign of the context role should decide target supervision

Under teacher forcing, today's target is tomorrow's context. Supervising yiy_i as a target makes the model more likely to generate it, and once generated, it becomes context for the predictions that follow. Every training example was filtered for a correct final answer, so the demonstrated continuation is a path known to reach that answer. If yiy_i as context makes this continuation harder to predict (si<0s_i<0), supervising it trains the model to enter a state from which the correct path is harder to follow. Dropping its direct loss only stops this reinforcement; yiy_i stays in the input as context. This is the basis of the criterion, not a claim that the two roles are equivalent. Whether it actually selects better supervision is tested by the training results in W2.

The correspondence may fail in three cases. First, the direction is estimated only against this one demonstrated continuation, not against other correct paths. For training on this example, though, masking still agrees with the objective: the target is this continuation, and the token stays in the input. Second, the mask is computed once by the frozen base model, and a token's direction may change as training proceeds. Our SFT runs for one epoch, so the model does not move far from the base, and periodic rescoring during training is a natural extension. Third, tokens near the end have short continuations, so their direction rests on only a few targets. Such tokens are a small fraction of a long CoT, and across position bins the negative fraction varies by at most 3.86 points, with no concentration at the end (Figure 3).

We will add this argument to Section 2.4 and these cases to the Limitations.

W2: Size of the gains

We agree that per-cell margins on pass@32 are limited: one problem is worth 2.5–3.3 points, and pass@32 only records whether any of the 32 samples is correct. avg@1, the mean accuracy over the 32 samples of each problem (Appendix H), has a resolution of 1/(32N)1/(32N) rather than 1/N1/N, so it is much less affected by benchmark size. On avg@1, our method has the highest six-benchmark mean in all 8 model–corpus settings, 0.33–2.82 points above SFT (1.23–2.82 in the 6 Qwen3 settings). Per benchmark, it is above SFT in all 48 cells: highest in 46, tied in 1, and below Random in 1.

Reverse uses the same scores and only flips the retention rule. In all 8 settings it has the lowest mean pass@32 and the lowest mean avg@1 of all methods, 1.58–8.82 avg@1 points below ours. Keeping the positive side or the negative side of the same scores puts the results at opposite ends, so the gain comes from the sign, not just from masking about half of the tokens.

We will tone down the claims in the main text, use avg@1 as the main evidence, and treat the per-cell pass@32 leads as supplementary.

W3: Writing

Thank you for catching this. The bib entries for AlphaToken and DFT were left out at submission, so the text shows "(?)" and AlphaToken is missing from the references. The revision restores both [1,2]. How the two methods differ from ours is already discussed in Related Work and Appendix C. We will also move the item-by-item descriptions of Table 1, Table 8, and Figure 3 to the appendix, and use the space for the argument in W1/Q2 and a new case-study section (see Q1).

(References are listed at the end of Part 2/2.)

Response to Reviewer PT9M (2/2)

Q1: Representative traces

We computed whole-word masking rates from the masks of Qwen3-4B/8B/14B-Base on both corpora (6 settings, first 1,000 examples each). In all 6 settings, words that decide where the reasoning goes next are mostly kept. Transition and hesitation words (however, hmm, maybe, actually, alternatively, wait, ...) have a masking rate of 0.27–0.36, and words that open the next step (let me, figure out, note, another, ...) 0.27–0.32. Words that evaluate or look back at the previous step are mostly masked: 0.73–0.86 for evaluations (easier, complicated, straightforward, ...) and 0.73–0.85 for words that look back (earlier, already, result, approach, ...).

An excerpt (Qwen3-14B-Base, OpenMathReasoning; masked tokens in [[ ]]):

Wait[[,]] maybe[[ that]]'s[[ getting complicated.]] Alternatively[[,]] maybe[[ there's]] a better formula[[ specific to polar]] coordinates[[.]]

Wait[[,]] another[[ approach]]: parametrize the curve[[ in Cartesian coordinates and]] then[[ use the surface]] of[[ revolution]] formula[[.]] Let me[[ try that]].

Here "Wait", "maybe", "Alternatively", "another", and "Let me" are kept, while evaluations such as "getting complicated" are masked. These word groups make up only about 7% of all words, and for most tokens the decision depends on context rather than on the word itself. The revision will add a case-study section with complete traces.

Q3: Non-math SFT data (new)

We trained Qwen3-8B-Base and Qwen3-14B-Base on a non-math subset of OpenThoughts (code, science, and other domains), with all other settings as in the paper. We evaluated on HumanEval, MBPP, and GPQA-Diamond with 4 samples per problem and report avg@4 and pass@4. For code, each cell gives the original EvalPlus tests / the extended tests (base / plus).

ModelMethodHumanEval avg@4HumanEval pass@4MBPP avg@4MBPP pass@4GPQA-D avg@4GPQA-D pass@4
Qwen3-8B-BaseSFT43.14 / 40.2485.37 / 80.4945.44 / 37.7684.39 / 71.1623.7447.98
Reverse18.29 / 16.9251.83 / 48.7819.91 / 16.0757.41 / 46.038.0826.26
Ours44.51 / 42.0785.98 / 81.7147.42 / 40.6787.30 / 75.9325.3851.52
Qwen3-14B-BaseSFT80.95 / 75.6196.34 / 92.6881.68 / 68.2594.44 / 81.4844.0769.19
Reverse78.05 / 72.7195.12 / 90.2477.25 / 65.0194.18 / 82.0142.6865.66
Ours84.76 / 79.4296.34 / 92.6884.52 / 70.6395.50 / 82.8049.1272.73

Our method is best on all 20 numbers, tied with SFT only on 14B HumanEval pass@4. Its avg@4 is 1.37–5.05 points above SFT. Reverse is lowest on 19 of the 20. At 8B, many of its generations do not terminate properly and get truncated, so the code cannot run. The criterion is therefore not specific to math data. We will add a section on these results to the experiments.

References

[1] Liu Qing, Ou Wu, and Yi Du. "AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training." arXiv:2606.01635, 2026.
[2] Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. "On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification." arXiv:2508.05629, 2026.

Thank you for the clarification and the additional experiments. Given that I still have doubts about the core motivation, and that the reported gains are not sufficient to convince me, I will maintain my score.

Thank you for the reply.

Core motivation. Our previous response gave the basis of the criterion and listed three cases in which it may not hold. The basis is as follows. Under teacher forcing, today's target is tomorrow's context. Every training example is filtered for a correct final answer, so the demonstrated continuation is a path known to reach that answer. A token judged negative loses only its direct loss and stays in the input as context. The reply does not say which of these steps remains in doubt. If the reviewer could point to it, for example the correspondence between the sign of the context role and target supervision, or the estimation of the direction against a single demonstrated continuation, we would be glad to address it directly.

The motivation also makes a testable prediction: with the same scores, keeping the positive side should outperform keeping the negative side. Reverse only flips the retention rule. In all 8 settings it has the lowest mean avg@1, 1.58–8.82 points below ours, and on non-math data it is lowest on 19 of the 20 numbers.

Size of the gains. We agree that per-cell pass@32 margins have limited resolution. For this reason, our previous response rested the evidence on avg@1 and on consistency across settings, which do not depend on per-cell margins:

  • In all 8 independently trained model–corpus settings, our method has the highest six-benchmark mean avg@1 among all methods, 0.33–2.82 points above SFT. If our method and SFT did not differ, the chance that all 8 settings point the same way would be 1/256.
  • Per benchmark, avg@1 is above SFT in all 48 cells.
  • On non-math data (code and science), our method is best on all 20 numbers across two models, tied with SFT only on 14B HumanEval pass@4, and its avg@4 is 1.37–5.05 points above SFT.

The gain in any single setting is modest, and the revision will tone down the main-text claims accordingly. The results, however, point in the same direction across two model families, three scales, two math corpora, and non-math data, while the same scores used in reverse place results at the other end. If the reviewer still finds this insufficient, we would appreciate knowing what kind of evidence, such as a particular significance test or evaluation setting, would address the concern.

Comments on the paper

1 comment
Area_Chair_3eSnOpenReview ↗

Please check the authors' rebuttal ASAP.

Dear reviewers,

As the discussion period will end very soon, please check the authors' rebuttal and see if your concerns have been addressed. Thank you.

-AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.