- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how fine-tuning data affects mathematical reasoning under irrelevant context. It examines distractor components, CoT-length-based data selection, and forced reasoning continuation, and reports robustness gains from training with distractors, including out-of-domain text.
Critical strengths
- The paper shows that retaining longer-CoT training examples can yield higher clean accuracy but worse performance under irrelevant context, revealing a useful trade-off in training-data selection.
- The experiments settings cover two main model families, multiple pruning ratios, and several distractor types. Extending GSM-IC to 1,319 problems, auditing distractor relevance, and reporting three-seed variability strengthen the empirical study.
- Out-of-domain distractor augmentation provides a simple, potentially useful training intervention. The reported gains with limited augmented samples and nearly unchanged clean accuracy in the Qwen3.5-4B setting are encouraging.
Critical weaknesses
- The shortcut claim in Section 3.1 goes beyond the evidence. Removing an irrelevant number makes the model more accurate, but this does not show that CoT signals during SFT stage taught the model to use that shortcut. The model may already have had this tendency before fine-tuning; the paper does not compare the behavior before and after SFT or directly change the proposed training correlation.
- The long-CoT and short-CoT training groups contain different questions. Their results could therefore differ because of what they learned, rather than how long the solutions were; the separate difficulty-pruning experiment does not resolve this.",
- The uptake metric counts a distractor number whenever it appears in the output, even in a sentence such as '10 is irrelevant.' It therefore does not reliably measure whether the model used irrelevant information incorrectly. The 'wait' experiment also lacks a clean-question comparison, so the accuracy drop could partly reflect a general problem with forced continuation.
- The claim of being the first to show that longer reasoning increases distraction is too broad. Inverse Scaling in Test-Time Compute (Gema et al., 2025) already reports this effect, although on different models and tasks.
Questions for authors
- Could you define the “shortcut correlations” in Contribution 1(line 48 - 50) more clearly, and demonstrate their causal effect on the model’s reliance on irrelevant information by training on datasets with and without these correlations?
- Could you train two models on the same questions, using short and long correct solutions that perform the same calculations? Does the model trained on longer solutions still suffer more from distractors?
- Does adding 'wait' also reduce accuracy on clean questions? Can you check whether the extra distractor numbers in the output actually enter an incorrect calculation, rather than merely being mentioned?
- What does your longer-generation experiment establish beyond Gema et al. (2025) [1]? Please state that difference explicitly and narrow the 'first work' claim.
- In page 6, line 312–313, the paper claim that: "We observe similar patterns, where long-CoT pruning increases uptake of irrelevant context, as shown in Fig. 5.". But in Figure 5, it indicates that "Long CoT Supervision(Shortest Pruned)increases uptake". Do you want to say "long-CoT supervision" rather than "long-CoT pruning"?
[1] Gema, Aryo Pradipta, et al. "Inverse scaling in test-time compute." arXiv preprint arXiv:2507.14417 (2025).
2 comments
Gentle Reminder
Dear Reviewer wUJv,
As we approach the end of the discussion period, we kindly as you to go through our rebuttal and let us know if we have addressed your concerns. Thank you for your time and effort!
We thank the reviewer for their comments. We address their concerns below:
We thank the reviewer for this question - we believe the experiment they are requesting for already exists within the paper - but we value their input. Traditionally in GSM8k, almost every number is correlated with an operation. This is true for most math reasoning benchmarks. This lets the model learn a shortcut that numbers must be correlated with operations. Our fine tuning with IC experimental setup does not hold the shortcut that “every number must be correlated with an operation”. Fine-tuning with such ICs helps overcome reliance on irrelevant context (numbers) - breaking that shortcut. We will also ensure to soften that narrative that this behavior is learned from SFT alone, thank you for pointing this out.
To assess this, we use GPT-6-Astra to modify the original chain of thought in the GSM8k train set by making them more verbose. In the verbose setting, we add additional explanations for why the arithmetics are being performed. Such explanations were added for every step. We train 2 models - one on the original and one on the verbose. We test on the clean and the Extended GSM-IC benchmarks. And observe the following performances:
We observe that for Verbose CoT, the drop in IC performance is significantly greater than for Original.
We note that synthetically making the train time chain of thought so much more verbose hurts the model’s overall performance. Thus, we repeated the experiment where we simply added 1 additional explanation sentence to the verbose setting. Here, the results are clearer:
Again, we notice that for Verbose CoT, the drop in IC performance is greater than for Original.
We show that for the exact same setup, the accuracy does not reduce on the clean dataset but reduces significantly on the setup with IC (as already shown in the paper):
With IC accuracy, we see a drop of up to 10-11%. We do not observe the same on clean accuracy.
In our paper, we present Unique Uptake values, which measures whether exact numeric tokens in the distractor appear within the generated response in those samples where the distractor does not contain the same numeric values as the question. We take it a step further and compute Unique Computational Uptake, which also ensures these numbers are used in arithmetic calculations. In GSM8k, all arithmetic computations occur within angular brackets <<>> - the model is instructed to do so. We parse for these, and observe if the number in the distractor appears within the left hand side of the operation. Such a search guarantees that the distractors are used within the arithmetic operations and not simply mentioned.
Unique Computational Uptake (%):
With an unpruned baseline of 5.05. We observe that long CoT supervision (Prune Shortest), still exacerbates IC reliance, as seen by an increase in Unique Computational Uptake.
Our work is focused primarily on fine-tuning data and how it impacts reliance on irrelevant context. This will be of benefit to data selection/pruning and data generation techniques. We note that the cited work does not study fine-tuning/training data at all and instead only studies varying inference time compute. We will articulate this more clearly in the manuscript.
Apologies for this typo. You are correct. We meant long-CoT supervision increases uptake. We will fix this in the revised version of the manuscript.