- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Step Dropout (SeDo), which randomly omits complete reasoning steps from chain-of-thought (CoT) training examples and trains on both complete and partial CoTs. The goal is to teach models to predict across multiple reasoning steps, thereby improving implicit computation and generalization to problems with greater reasoning complexity. The paper also studies SeDo-R, which replaces omitted steps with mask tokens and adds prediction targets for the masked steps. The evaluation includes direct-answer accuracy on the DEPO multi-hop graph task, complexity generalization on Maze, Quickselect, and GSM-∞, and analyses using complete-CoT prompts and linear probes. The reported results show improvements from SeDo on several tasks and model families, although the evidence for the proposed internal mechanism and the behavior of SeDo-R on GPT2-RoPE remain unclear.
Critical strengths
- Simple, general, and practical. SeDo requires no architectural changes or auxiliary prediction heads and can be applied during post-training of pretrained models. It is lighter than methods such as MTP and FSP that require auxiliary layers.
- Broad evaluation with a relatively complete body of evidence. The paper evaluates implicit multi-hop reasoning on controlled DEPO and OOD complexity generalization on Maze, Quickselect, and GSM-∞. It also includes RL, self-generated CoT, linear-probe, and looped-Transformer experiments.
- Clear presentation and detailed experimental setup. The appendix describes data generation, training details, the SeDo-R implementation, RL, and self-generated-data settings, supporting reproducibility.
Critical weaknesses
-
The probe and looped-Transformer results are correlational, not causal. The probe shows omitted values are more decodable in SeDo states, not that the model uses them. The looped-Transformer result is also correlational and warrants more cautious interpretation.
-
SeDo's gains may come from structured data augmentation. SeDo creates step-dropped partial-CoT views of each training example, but current experiments do not distinguish the contribution of this augmentation from implicit computation of omitted steps. If the main gains come from data augmentation, the paper's motivation and contribution regarding implicit reasoning would be fundamentally weakened. A matched control is needed to distinguish these explanations.
-
Important performance differences remain unexplained. The paper attributes SeDo's weaker small-data results on randomized Maze to the need for more training, but does not explain the large / gap on hard Quickselect or the GPT2-RoPE SeDo-R collapse on DEPO.
Questions for authors
Please address the concerns raised in W1–W3 above.
3 comments
Response to Reviewer rS3X (2/3)
-
“SeDo's gains may come from structured data augmentation.”; “If the main gains come from data augmentation, the paper's motivation and contribution regarding implicit reasoning would be fundamentally weakened. A matched control is needed to distinguish these explanations.”
Thank you for raising this point. We agree that clear baselines are needed to distinguish the benefits of implicit reasoning from those of structured data augmentation. We address this in two ways. First, we compare SeDo with token-level augmentation baselines, and with a baseline trained on additional new problems. Second, we show that SeDo models compute intermediate results that are not written in the CoT, and use them to predict later steps.
Comparison with augmentation baselines. To test whether SeDo's gains come from trace augmentation alone, in the paper we compare SeDo with two trace augmentation baselines, Token Drop and MFT (Table 2), which perturb the same CoTs by dropping or masking tokens instead of whole steps. SeDo outperforms both in nearly every OOD range across models and datasets. For example, on GSM- with , SeDo with Qwen3-1.7B reaches 49.2% / 77.1% (p@1 / p@16), compared to 34.2% / 62.1% for Token Drop and 39.2% / 69.6% for MFT.
Since data augmentation approximates additional data, we further compare SeDo with an upper bound: a Full-fresh baseline on Qwen3-1.7B, which replaces SeDo's partial CoTs with complete CoTs of new problems. Concretely, Full-fresh is trained on four to five times as many distinct problems as SeDo. We report OOD p@1 / p@16, with OOD ranges merged:
| Task | OOD range | Full | Full-fresh | SeDo |
|---|---|---|---|---|
| Maze | [200, 650) | 35.96 / 50.80 | 44.68 / 61.86 | 45.53 / 70.74 |
| Quickselect | [140, 380) | 41.75 / 76.75 | 46.42 / 77.42 | 59.67 / 96.83 |
| GSM- | [26, 36) | 47.71 / 67.92 | 61.88 / 83.96 | 55.00 / 80.83 |
SeDo outperforms Full-fresh on Maze and Quickselect (e.g., 96.83% vs. 77.42% p@16 on Quickselect). On these tasks, SeDo's gains thus exceed those from five times as many new problems, suggesting that they do not come from data augmentation alone. On GSM-, Full-fresh is higher, but SeDo still achieves about half (p@1) to four fifths (p@16) of Full-fresh's gain over Full. We have added this baseline to Table 2 and Section 4.1.
Evidence of implicit computation. Our behavioral, probing, and intervention results in point 1 further suggest that SeDo models compute omitted values implicitly and use them in subsequent predictions. In particular, without 's defining line, SeDo predicts the dependent value with 86.4% accuracy, compared with 2.6% for Full, and erasing 's value with LEACE reduces this accuracy to 78.0%.
Together, these controls, behavioral tests, and interventions suggest that learning to infer omitted computations contributes to SeDo's gains.
Response to Reviewer rS3X (3/3)
-
“The paper ... does not explain the large p@1/p@16 gap on hard Quickselect or the GPT2-RoPE SeDo-R collapse on DEPO.”
Thank you for this comment. We explain both results below.
Quickselect. To understand the large p@1/p@16 gap on the hardest range (), we inspected the sampled solutions of SeDo with Qwen3-1.7B. These problems require long CoTs with hundreds of steps, so a small error in one step can make the answer off by a few ranks. Indeed, many incorrect samples return an element close to the target rank, while other samples of the same problem give the correct answer. Among the incorrect samples, 63.5% return an element one rank away from the target, and 89.8% are within three ranks. For example, on a problem asking for the 39th-smallest element, greedy decoding stops partitioning one step too early and returns the 38th-smallest element, whereas 10 of 16 samples continue with the remaining candidates and return the correct answer. This helps explain why repeated sampling substantially increases success. A similar gap also appears for Full (Table 2). We have added this analysis to Appendix E.
DEPO. SeDo-R's low K12 and K16 accuracy on GPT2-RoPE is due to the lower learning efficiency of both the architecture and the method.
First, GPT2-RoPE learns less efficiently than Canon-Llama. We report the earliest checkpoint from which validation accuracy stays above 80% at every remaining evaluation through 150k updates (evaluated every 5k updates). Each entry shows Canon-Llama / GPT2-RoPE.
| Method | K4 Canon-Llama | K4 GPT2-RoPE | K8 Canon-Llama | K8 GPT2-RoPE |
|---|---|---|---|---|
| SeDo | 10k | 30k | 20k | 105k |
| SeDo-R | 20k | 25k | 55k | 150k |
| MTP | 15k | 40k | 50k | 105k |
| iCoT-SI | 10k | 15k | 15k | >150k |
GPT2-RoPE reaches this threshold later, or does not maintain it through the end of training. As a result, within our 150k-update budget, only SeDo and MTP achieve non-trivial K12 and K16 accuracy on GPT2-RoPE. In particular, SeDo-R reaches 80% at K8 only at the final 150k checkpoint, leaving no updates to learn K12 and K16.
Second, SeDo-R learns long chains less efficiently than SeDo. It needs more updates than SeDo at K8 on both architectures (table above). On Canon-Llama, SeDo-R still has 0% K16 accuracy after 50k updates, while SeDo already reaches 46.09%, although both eventually reach near-perfect K16 accuracy.
We attribute the slower learning of SeDo-R to a mismatch between its explicit and implicit reasoning data. In SeDo, the more steps are removed from a partial CoT, the closer it becomes to an answer-only example. In SeDo-R, however, even a fully masked CoT still contains one mask token per step, unlike answer-only examples, which contain none. To test this, we train and evaluate SeDo-R on GPT2-RoPE with every answer-only example replaced by a fully masked CoT (one mask token per intermediate node), so that its implicit reasoning data match its partial CoTs. SeDo-R learns substantially faster here: its K8 accuracy stays above 80% from 85k updates, compared with 150k for standard SeDo-R (table above). It thus has more updates left to learn K12, reaching 76.56% after 150k updates, compared with 2.34%. We have added these learning curves to Appendix F.3 (Figure 9).
References
- Belrose et al. (2023). LEACE: Perfect linear concept erasure in closed form.
- Sinha et al. (2026). The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs.
Response to Reviewer rS3X (1/3)
Thank you for your detailed and helpful review. We have revised the paper, with changes marked in blue.
“The probe and looped-Transformer results are correlational, not causal. The probe shows omitted values are more decodable in SeDo states, not that the model uses them. The looped-Transformer result is also correlational and warrants more cautious interpretation.”
Thank you for this comment. We agree that our probes alone do not show that models use the decoded values, and we have conducted an intervention experiment to test this.
Probes and erasure. We test from three aspects whether SeDo models compute and use omitted values implicitly, with behavioral, probing, and intervention experiments on GSM-∞. Specifically, we omit the line that calculates a variable V, and ask the model to predict a dependent value U, e.g., complete
U = V * 2 =. We use the 500 held-out probing problems from Section 5.1 with Qwen3-1.7B.First, SeDo models can predict future steps without the omitted dependencies provided in the CoT context: without V's defining line, SeDo predicts U with 86.4% accuracy, compared with 2.6% for Full.
Second, V's value is represented in the hidden states: linear probes at the final
=decode V's value much more accurately from SeDo than from Full (Section 5.1, Figure 5).Third, SeDo models use this representation to predict U, which the probes alone do not show. We erase V's value at the same position with LEACE (Belrose et al., 2023), jointly in all layers of Qwen3-1.7B, and measure whether the model still predicts U correctly. As a baseline, we instead remove as many random directions as LEACE does (averaged over five random seeds). We report the accuracy (%) of predicting U:
With V's defining line omitted, LEACE reduces SeDo's accuracy from 86.4% to 78.0%, while random erasure has no effect. In contrast, with the line present, where the model can read V's value, LEACE has little effect. This suggests that SeDo models use the implicitly computed values to predict subsequent steps. The effect is partial, likely because LEACE does not remove all information about V: after erasure, probes still decode V with 16.0% accuracy (48.4% before erasure; 5% by chance). V may also be represented at earlier positions or nonlinearly. We have added this experiment to Section 5.1 and Appendix H (Table 15).
Looped Transformers. We agree that this result is correlational, and have interpreted it more cautiously in the revision. However, looping is a simple approach to scale implicit computation without changing the model parameters. We show that more implicit computation improves both implicit reasoning and single-step CoT reasoning, even on calculations the models already solve perfectly without a CoT. This parallels the finding of Sinha et al. (2026) that larger models execute far more steps correctly, even when smaller models have near-perfect single-step accuracy; here, we increase computation through looping instead of model size. We include this experiment as a demonstration beyond SeDo: in models not trained with SeDo, stronger implicit reasoning also correlates with better explicit reasoning. We have revised Section 5.3 accordingly.