- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper looks at whether a correct answer from a model actually comes with a valid chain-of-thought trace. To test this, the authors use iGSM, a synthetic grade-school math benchmark from Ye et al. (2025a). Because every quantity and dependency in an iGSM problem is known, each step of a generated trace can be checked automatically.
They retrain the small GPT-2-style models from Ye et al. on problems up to 15 operations and test them on harder problems up to 23. On problems like the training ones, correct answers almost always come with valid traces. On harder problems the two drift apart: at 23 operations, about a third of correct answers (31.6%) have invalid traces, and more than half of those look fine on the surface and only fail when the dependencies are checked.
The second part of the paper changes what the models are trained on. Models trained on longer, non-minimal traces produce non-minimal traces themselves. When a problem is re-asked about a different quantity, some traces still include steps that only the original question needed, which the authors take as a sign that minimal traces are not enough to show the model is planning. Models trained on traces from unrelated problems still get about 82% of in-distribution problems right, and models trained with the first part of each trace scrambled do as well as the clean model, even though none of their traces pass the checker.
The results come from 4,096 test problems per difficulty level, mostly from one training run, with extra runs and seeds in the appendix, a no-trace baseline, and fifteen worked example traces. The authors present this as a warning for anyone reading model traces as evidence of reasoning or using them for safety monitoring, while noting they do not test whether traces are causally faithful.
Critical strengths
-
The setting makes the question checkable. Most work on CoT faithfulness has to judge natural-language traces by hand or with another model. Here every step can be checked exactly, and the paper reports the full counts (Tables 8 and 9), so the reader can see which failures drive the numbers. Testing this on iGSM, the benchmark originally used to argue for learned reasoning and planning, makes the result more relevant than it would be on a purpose-built task.
-
Some of the interventions are well designed. The shuffled-prefix models are the clearest case: with 30% of trace sentences scrambled, the model still reaches about 75% at op = 15, against about 25% with no trace at all, so the trace is clearly doing useful work while never being valid. The re-asking experiment is also a nice idea, since in standard iGSM the question always targets the last node, and it can't be told apart from simply computing toward the end. These are the parts of the paper I found most convincing.
Critical weaknesses
-
The clean model is much weaker than the one in Ye et al. out of distribution. At op = 23, Ye et al. report 76.8% with a valid trace, the model here gets 39.7% (Table 10). With Ye et al.'s numbers, 31.6% of correct answers having invalid traces is not even possible, so part of the effect may simply be undertraining rather than distribution shift. The 200k-step run also lowers this share at op = 20-22, which doesn't fit with the claim that doubling training doesn't change the picture.
-
The 31.6% looks less striking once broken down (Table 9, op = 23). Almost half of these traces (352/750) don't parse, so they are not plausible-looking at all. Most of the remaining ones use a wrong input that has the same value as the right one, or a wrong value that later gets multiplied by zero, so the answer comes out right anyway. The checker also seems strict in some cases: in the running example, using A instead of Q counts as an error, even though the problem says Q equals A. It would help to know how many of these cases have answer 0 or are just this kind of substitution.
Questions for authors
-
Can you report the main analysis on a model closer to Ye et al.'s performance (their checkpoints, or a longer/better-tuned run)? If the share of correct answers with invalid traces drops a lot there, the distribution-shift story becomes much weaker.
-
Of the 750 correct-but-invalid cases at op = 23, how many have answer 0, and how many same-value substitutions are definitional (like Q = A in the running example) rather than coincidental?
-
For the shuffled models, does the intact suffix pass the checker if the shuffled prefix is treated as given? This would show whether "no trace passes verification" reflects the reasoning or just the scrambled block.
-
The swapped models score below the no-trace baseline in distribution (81.7% vs 87.6%, Table 2). How does this fit with presenting them as evidence that invalid traces still help? Similarly, with 3.7% and 9.6% of re-asked traces padded (Table 5), most stay minimal, so why conclude that minimality fails?
Suggestion (future work): training with queries that do not always target the sink would separate a learned "compute toward the sink" habit from a lack of planning more cleanly than re-asking.
4 comments
Response to Reviewer rZHo (part 2 of 2): Q1 to Q4
Q3. Suffix validity of the shuffled models.
RESPONSE: Yes, for the 10% shuffled model the intact suffix passes the checker for most correct answers. We split each generated trace at the earliest point from which every sentence is well formed, bound every letter the suffix uses but never defines to any quantity of the problem with that value, trying all bindings, and run the full dependency, arithmetic and query check on the suffix. Share of correct answers whose suffix passes:
| op | 15 | 20 | 21 | 22 | 23 |
|---|---|---|---|---|---|
| clean model, whole trace valid | 99.5 | 92.5 | 87.5 | 78.6 | 68.4 |
| 10% shuffled, suffix valid given the block | 99.5 | 94.8 | 91.1 | 87.1 | 81.6 |
| 30% shuffled, suffix valid given the block | 82.3 | 57.9 | 48.2 | 40.3 | 34.5 |
So the reviewer's reading is correct. For the 10% shuffled model the shuffled block is almost always a single sentence, and what follows it is a valid derivation. The model writes the shuffled block and then a suffix. The suffix is valid at least as often as the clean model's whole trace at every level, and more often from op 20 onwards, 81.6% against 68.4% at op 23. The corruption stays in the block and does not spread into the derivation that follows. The suffix has exactly one letter whose definition was lost in the shuffled block in 2,454 of the 2,463 correct answers at op 23, so only one letter has to be matched to a quantity and the rest of the suffix is checked exactly. For the 30% model the block is about three sentences, accuracy falls, and the suffix is valid for only a third of correct answers at op=23. "No trace passes verification" therefore reflects the shuffled block, not the reasoning. We will add this analysis to the appendix.
Q4. Swapped traces and re-asking.
RESPONSE: The reviewer is right that 81.7% for the swapped traces model below 87.6% for the no-traces model does not show that "invalid traces help". The sentence in the introduction that a trace can help the model while being far from a valid derivation refers to the shuffled models, where the 30% shuffled model reaches 75% at op=15 against 25% with no trace. The swapped result shows something different, that a trace can be unrelated to the problem and the answer still be right, since accuracy survives when every training trace belongs to another problem. We will make the attribution to the shuffled models explicit.
For re-asking, 3.7% and 9.6% are indeed small and we do not claim most traces are padded. Our point is that the rate on the original queries is 0.4% and 1.3%; i.e. re-asking increases padding by 7 to 9 times. Further, nine in ten of the extra parameters lie on the original chain to the sink: 114 of 128 at op=15 and 142 of 157 at op=20. Parameters on that chain are only 24 to 28% of the unnecessary parameters available in these problems. If the model chose its unnecessary parameters at random, about one in four of them would be on that chain. In this case, nine in ten of the ones the model adds are on that chain. On the standard query the parameters the question needs are exactly the chain to the sink, so a minimal trace there cannot show that the question selected the parameters. The probing experiment run for Reviewer y8kb also supports our claim. At the end of a re-asked question, the probe on the clean model's hidden state still marks as necessary one in ten of the parameters on the original chain that the new question does not need. This is against 0.3% for unrelated ones, and when the model does write an unnecessary parameter, it is one the probe had already marked as necessary, in 65%, 50% and 58% of cases at op=15, 20 and 23.
Q1. Checkpoints.
RESPONSE: Please see our answer to weakness 1. Ye et al. have not released any checkpoints. We did evaluate the model checkpoints of Zhang, Neubig and Yue (2026), that show the same decoupling wherever their accuracy drops.
Q2. Answer-0 and definitional substitutions.
RESPONSE: Answered under weakness 2. Of the 750 correct answers with an invalid trace at op 23, 300 have answer 0, and 3 of the 234 same-value substitutions are definitional.
Suggestion (future work).
RESPONSE: Thanks for this suggestion, we will investigate this further.
We hope the new evidence on models trained by others, the recount under the reviewer's own accounting, and the validity of the shuffled models' traces after the shuffled block address the concerns behind the review.
Thank you for the detailed response and the new experiments.
My two main concerns are mostly answered. The Zhang et al. checkpoints show the decoupling is not only from your training run, and the op 23 breakdown is what I asked for.
Two things I would still fix in the abstract. It gives 31.6% alone, but the strict count is 16.7%. And for the 10% shuffled model the rest of the trace is valid in most cases (81.6% at op 23), so “no trace passes” comes from one shuffled sentence.
I am raising Soundness and my rating to 3.
Thank you for raising the score and the comments on the abstract. We will incorporate them in the next version of the paper.
Response to Reviewer rZHo (part 1 of 2): W1 and W2
We thank the reviewer for appreciating our contributions including that the setting makes every step of a trace checkable and reports the full counts, and the design of the shuffled-prefix and re-asking experiments. An overview of the new experiments is in our global comment. Our response is in two parts, W1 and W2 here, and Q1 to Q4 in part 2.
W1. The gap between Ye et al.'s and our models in accuracy.
RESPONSE: Our clean model was trained with Ye et al.'s published recipe, and we could not close the gap despite repeated training runs. Doubling the training to 200k steps raises the share of correct answers with a valid trace at op 23 from 39.7 to 41.3%, and a 4x larger Qwen3-architecture model trained with the same recipe reaches 45.5%. The 200k run moves the share of correct answers with invalid traces by 2 to 5 points at op 20 to 23 and leaves the pattern unchanged.
The main question is whether this decoupling of correct answers and trace validity is an artifact of our training. Ye et al. did not release their checkpoints, so we could not examine them. However, Zhang, Neubig and Yue (ICML 2026) have released checkpoints of 100M parameter Qwen2.5 style models that were pre-trained from scratch for 10B tokens on GSM-Infinite, a close variant of iGSM with the same step format, together with a trace validity checker. We evaluated their two pre-trained models, trained on op=2 to 10 and on op=2 to 20, on their test problems using their checker. The table gives, per op, their answer accuracy and P(trace valid | correct). Averaged over op as in their paper, our evaluation reproduces their Figures 3 and 12.
Where accuracy is above 95, P(valid | correct) is 88 to 99. Where accuracy falls to between 50 and 90, it drops to between 48 and 68, including inside the op=2 to 20 model's own training range. Models trained by others, on a related benchmark and scored with their own checker, show the same decoupling, so it is not an artifact of our training.
References: Zhang, C., Neubig, G. and Yue, X. (2026). On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models. ICML 2026.
W2. The op 23 breakdown.
RESPONSE: We agree the headline needs the breakdown next to it, and we expand Table 9 further. At op 23 there are 750 correct answers with an invalid trace.
Case when Answer is 0. 300 of the 750 have answer 0, against 22.7% of all correct answers. Removing every problem whose answer is 0 leaves 450 of 1,836 correct answers with an invalid trace i.e., 24.5%.
Same-value substitutions. Of the 234 traces in which a step uses a wrong input with the same value as the right one, 3 use a quantity the problem defines as equal to the right one, 171 use an unrelated quantity that happens to have the same value, 109 of them a shared 0, and 60 differ in more than one input. The case of the running example is therefore rare.
Further, we do not agree that such a substitution is valid. That the problem defines Q as equal to A does not make A a substitute for Q. The step has to use the quantity that the problem's sentence names, and using another quantity that happens to have the same value is a different computation. Moreover, the training traces never make such a substitution, so a model that makes one at inference has deviated from its training.
Traces that do not parse. These traces are not malformed text. In 338 of the 352 a letter is bound twice, 194 at a Define statement, 127 as a clause-local symbol and 17 both, and the other 14 use a letter that was never defined. They read like valid traces, which is why we count them as invalid rather than as noise. One trace at op 23, for instance, defines Parrot Paradise's Shark as w at its second step, with w = 6, then at step 14 writes "Define Trout's Bone as w; so w = Z = 7." and two steps later computes the answer as "K = E * w = 7 * 7 = 3". The inputs and the arithmetic are right and the answer is correct, but w names two quantities, so the trace does not parse.
Counting only failures that no value coincidence can explain leaves 397 of 2,376 correct answers at op 23, i.e. 16.7%. In distribution, at op 15, 0.5% of correct answers have an invalid trace under the paper's full count. The strictest count out of distribution is therefore still more than thirty times the loosest count in distribution.
Finally, note that the cases this strict count excludes are not noise. A trace in which a step uses a wrong input of the same value, or whose error a multiplication by zero removes, reaches the right answer by coincidence and is still semantically invalid.