- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Future Influence, a token-selection criterion for reasoning SFT. It measures how decreasing a token’s information affects the prediction loss on the fixed teacher-forcing continuation. The scores are computed offline, and tokens with negative scores have their direct prediction losses masked while remaining available as context. Experiments across four models and two mathematical reasoning datasets report improvements over standard SFT and several token-level supervision baselines.
Critical strengths
- The idea and motivation are clear. Tokens have no contribution to later reasoning are masked out during SFT supervision.
- The computation of Future Influence is offline, and can be done in one forward pass and one backward pass, which is acceptable.
- The evaluation is broad and covers most of the important datasets for math reasoning, with avg@1 and pass@32.
Critical weaknesses
- A negative score does not necessarily mean a token should not be learned. Reasoning often includes trying an idea, finding a mistake, and changing direction. A token may help the current reasoning segment but make later segments harder to predict. Its total score could therefore be negative even when it belongs to a useful reasoning step.
- The novelty of the contextual contribution measure is unclear. Measuring how context affects model predictions is already an established research problem. Value Zeroing [1] measures a token’s contribution by zeroing its value vectors while keeping the input tokens unchanged. Attribution patching [2] further shows how gradients can efficiently approximate the effects of internal interventions. These methods already cover the main idea of measuring contextual influence through interventions. The paper’s potential contribution therefore appears to lie in its particular signed continuation-loss score and how this score is used to select SFT supervision. It should explain what this measure adds to existing context attribution methods and show why it provides a better signal for supervision selection.
- The ablations do not test the main design choices. Random, Low-NLL, and Reverse masking are useful controls, but they do not explain why the proposed attribution design is preferable. The paper does not compare scoring the full continuation against scoring only the final answer, or selecting individual tokens against selecting reasoning segments [3]. These comparisons would help show which choices produce the reported gains.
[1] Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. “Quantifying Context Mixing in Transformers.” EACL, 2023, pp. 3378–3400.
[2] Aaquib Syed, Can Rager, and Arthur Conmy. “Attribution Patching Outperforms Automated Circuit Discovery.” BlackboxNLP, 2024, pp. 407–416.
[3] Siyuan Wang, Yanchen Liu, and Xiang Ren. “Segment-Level Attribution for Selective Learning of Long Reasoning Traces.” ICLR, 2026.
Questions for authors
- Can you examine tokens that help their own reasoning segment but have a negative total influence on later segments? Do these tokens appear in useful exploration or self-correction, and does removing their supervision help? How do results change when scoring only the final answer, or selecting whole segments [3]?
- What does the gate-gradient estimator add beyond existing attribution methods [1, 2]?
- Please also report actual scoring time and total scoring-plus-training cost relative to standard SFT. Are they still acceptable for larger scale training?
6 comments
Response to Reviewer r95X (2/2)
W2 / Q2: What this adds beyond Value Zeroing and Attribution Patching
We agree that measuring the role of context by intervention is an established problem. Our gate multiplies the value vectors, so is Value Zeroing [1] applied to token in all layers, and approximating the intervention with the gradient at belongs to the same family as Attribution Patching [2]. We do not claim the estimator as a contribution.
[1,2] ask which context a trained model uses for a given prediction. We ask which tokens of a correct demonstration should be directly reinforced. Our contribution is turning the role of context into a supervision signal for SFT. First, attribution methods do not say why a token's role as context should decide whether it is trained as a target. Our basis is that under teacher forcing, today's target is tomorrow's context. Supervising a token makes the model generate it, after which it is context for all later predictions. If it makes the correct continuation harder to predict, reinforcing it teaches the model to enter a worse state. So we measure the signed effect on the continuation loss, which is the SFT objective itself.
Second, the training scheme handles the target role and the context role separately: negative tokens lose only their direct loss and remain as context. Scoring uses the same base checkpoint that is trained, needs no reference model, and runs once offline. Only the sign is used, so there is no retention ratio or threshold. Third, even in trajectories with correct answers, 50.21–51.83% of tokens are negative, spread roughly evenly over positions. A correct answer does not mean every token deserves direct reinforcement.
On whether it is a better supervision signal: under the same SFT setup, our method has the highest mean avg@1 in all 8 settings. Besides Random, Low-NLL, and Reverse, the baselines include four published methods built on different signals: AlphaToken (gradient alignment), Token Cleaning (loss differences), ssToken (training dynamics and attention), and DFT (target-probability weighting). Using the same scores in reverse (Reverse) is the worst in all 8 settings, 1.58–8.82 points below ours, so the useful information is in the sign.
We will add [1,2] to Related Work and rewrite the contributions in the introduction along these three points.
Q3: Scoring time and total cost
Measured times in hours on one 8×H100 node:
| Model | Corpus | Scoring | Standard SFT | Masked SFT | Scoring / SFT | (Scoring + Masked SFT) / SFT |
|---|---|---|---|---|---|---|
| Qwen3-4B-Base | OpenMath | 4.25 | 7.10 | 7.09 | 0.60 | 1.60 |
| Qwen3-4B-Base | OpenR1 | 3.34 | 5.36 | 5.35 | 0.62 | 1.62 |
| Qwen3-8B-Base | OpenMath | 4.74 | 16.77 | 17.82 | 0.28 | 1.35 |
| Qwen3-8B-Base | OpenR1 | 3.07 | 12.76 | 12.76 | 0.24 | 1.24 |
| Qwen3-14B-Base | OpenMath | 12.55 | 16.15 | 16.27 | 0.78 | 1.78 |
| Qwen3-14B-Base | OpenR1 | 8.26 | 12.05 | 12.00 | 0.69 | 1.68 |
Scoring takes 0.24–0.78× one run of standard SFT, and scoring plus training takes 1.24–1.78×. Masked training takes about as long as standard SFT, because masked tokens still go through the forward and backward passes and are only left out of the loss. At larger scale, scoring is independent across examples and fully parallel, and its time grows linearly with data size, as SFT does. The mask is computed once per model and dataset and can be reused across epochs, hyperparameter tuning, and repeated runs. We train for only one epoch, so the table shows the least favorable case.
We will add this table to the appendix.
References
[1] Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. "Quantifying Context Mixing in Transformers." EACL, 2023, pp. 3378–3400.
[2] Aaquib Syed, Can Rager, and Arthur Conmy. "Attribution Patching Outperforms Automated Circuit Discovery." BlackboxNLP, 2024, pp. 407–416.
[3] Siyuan Wang, Yanchen Liu, and Xiang Ren. "Segment-Level Attribution for Selective Learning of Long Reasoning Traces." ICLR, 2026.
Thank you for the rebuttal.
W1. I am not convinced. My question was about tokens that help their own segment but get a negative net score because of later segments, for example in exploration or self-correction. The response does not identify such tokens and only reports masking rates for a few word types.
W2. Thank you for clarifying that the estimator is not a contribution. However, I am still not convinced by the novelty of the remaining contribution. Appendix C.2 describes AlphaToken as also scoring a token’s causal influence on later predictions through value propagation to select supervision. How does your criterion differ from this, beyond using only the sign and computing it offline?
W3. I look forward to the results for answer-only scoring and segment-level selection that you committed to posting.
Suggestion. I would suggest that the authors write the remaining responses themselves, without AI drafting or polishing, in clear and well-organized English that answers each question directly.
Response to Reviewer r95X (1/2)
Thank you for the review. We respond to each point below.
W1 / Q1 (first half): Tokens in exploration and self-correction
We computed whole-word masking rates from the masks of Qwen3-4B/8B/14B-Base on both corpora (6 settings, first 1,000 examples each). Words that turn the reasoning during exploration and self-correction are mostly kept, consistently across all 6 settings. Transition and hesitation words (however, hmm, maybe, alternatively, wait, ...) have a masking rate of 0.27–0.36, and words that open the next step (let me, another, ...) 0.27–0.32. In contrast, words that evaluate the previous step (easier, complicated, ...) have a masking rate of 0.73–0.86, and words that look back (earlier, already, result, ...) 0.73–0.85.
In the excerpt below, the reasoning abandons one approach and switches to another (Qwen3-14B-Base, OpenMathReasoning; masked tokens in
[[ ]]):The turn itself ("Wait", "maybe", "Alternatively", "another", "Let me") is supervised; the comment on the previous step ("getting complicated") is not.
The case you describe can occur, since si is the net effect on the whole continuation. We use the net effect because once the model generates the token, both the current segment and later segments are conditioned on it, and the SFT loss sums over all of them. Masking also does not take the token out of the context. The rest of its segment is still supervised with it as input, so its help to the current segment is kept in training; what is removed is only the signal that teaches the model to generate it. Given the masking rates above, at most 27–36% of turning words could fall into this case.
Removing the supervision of negative tokens does help. Our method changes SFT only in this one respect, and its avg@1 is above SFT in all 48 cells. Reverse supervises only the negative tokens and has the lowest mean avg@1 in all 8 settings (see W2).
W3 / Q1 (second half): Scoring only the final answer, or selecting by segment
We are running both ablations on Qwen3-8B-Base and Qwen3-14B-Base with OpenR1-Math-220k. Answer-only scoring replaces Lresp in scoring with the loss on the final-answer tokens only. Segment-level selection splits responses into paragraphs and keeps or masks each paragraph by the sign of its summed si, matching the selection granularity of [3]. Each changes only the scoring or the mask, with training and evaluation unchanged, and we will post the results during the discussion period.
The answer loss is part of Lresp, so scoring the full continuation and scoring only the answer measure the same kind of effect; the difference is how reliable the estimate is. The final answer of a long CoT is only a few tokens, so with answer-only scoring each token's score rests on those few targets and has high variance. Under teacher forcing the answer depends mostly on the derivation right before it; the number in
\boxed{}is often just the result of the previous line. An early token sits thousands of tokens before the answer, so its effect on the answer loss is close to zero and its sign is easily dominated by noise. With full-continuation scoring, each token's score sums the loss changes of all targets after it, so early tokens also have many nearby targets that provide signal.We select per token because directions alternate heavily within segments. Splitting by paragraph (same 6 settings, segments of at least 5 tokens), the minority direction makes up at least 30% of tokens in 88.9–90.4% of segments, and deciding each segment by its summed si would flip the decision for 45.4–45.6% of tokens. In the excerpt above, "Wait" is kept while the comma right after it is masked, and "another" is kept while "approach" is masked. Segment-level selection cannot tell these apart.
We will add [3] to Related Work and include both ablations in the revision.
(References are listed at the end of Part 2/2.)