acceptodds
Under review as a conference paper at ICLR 2027

Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization

Abstract

Large Language Models can perform multi-step reasoning and improve task performance through different forms of intermediate computation, from token-based traces to computation carried out in latent space. However, a question remains open: do these different forms of thinking rely on the same underlying mechanism? To address this, we train and compare five variants of the same GPTNeoX backbone from scratch on an extended multi-hop reasoning task (ProsQA-Ext): a vanilla model, a Chain-of-Thought (CoT) model, a Pause Token model, and two latent-reasoning models that are optimized end-to-end without intermediate reasoning traces. We find that, strong in-distribution (ID) performance does not guarantee depth generalization. Vanilla, CoT, and Pause Token models solve ID problems well, but rely largely on local graph features and generalize poorly to out-of-distribution (OOD) problems with longer hops. In contrast, latent variants generalize better and show internal dynamics consistent with forward reachability propagation on the graph. Causal interventions and circuit analysis localize this computation to a sparse recurrent search circuit in the bottleneck latent model: an attention head retrieves graph relations, an MLP and the residual stream update the reachability state across recurrent steps, while multiple attention heads together then do the candidate matching. Together, these results show that different thinking mechanisms can learn distinct computational solutions, even at similar ID performance. In this setting, latent recurrence supports a reusable forward-search algorithm that generalizes beyond the training depth.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, latent reasoning, multi-hop reasoning, length generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FavzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what mechanism models with different internal reasoning (CoT, pause token, latent reasoning) use for multi-step reasoning, especially for OOD generalization. They use five model variants on one synthetic task and perform an extensive mechanistic study. They find that the bottleneck-latent models have the best generalization capabilities, possibly due to relying less on shortcuts.

Critical strengths
  1. The experimental setup is strong and the analysis is very detailed. I think the task is clearly related to the problem at hand, and the authors perform a very extensive study of the five models on this task. It's clear how the study progresses and what question the authors are investigating (due to subsection names), and its clear the study provides a very full understanding of the models and the task.
Critical weaknesses

My primary concerns are listed in the discussion section of the paper by the authors.

  1. Lack of important baseline: looped transformer. The authors mention on lines 472-473: "One family that is commonly used but not included here is the looped transformer". Since goal of the paper is to study how different types of LLMs perform multi-step reasoning, the claim in the title about a "recurrent search algorithm for depth generalization", and the study itself analyzes different types of internal reasoning, the study seems incomplete without including looped transformers. Looped transformers are by definition recurrent (in depth), which seems very relevant when studying depth generalization. There are a number of works studying looped transformers in similar contexts (e.g. Saunshi et al. 2025 studying reasoning in looped transformer, showing an inductive bias towards reasoning and how they "implicitly generate latent thoughts", or Yang et al. 2024 learning learning algorithms, Fan et al. 2025 length generalization).

  2. Generalizability of results. Currently, the study is performed at one mode scale on one synthetic task. The authors say the following: "However, the use of small models and a synthetic task limits how far the current conclusions can extend." I agree with this claim; currently, the scale of the study seems limited.

Some suggestions that would strengthen the paper and address my concerns:

a) Include a looped transformer in the study. The authors cite two looped transformers (Dehghani et al., 2019; Giannou et al., 2023), and recently there is also (Prairie et al. 2026, Jeddi et al. 2026, and many more). Any of these looped models or the ones mentioned in 1) would strengthen the study.

b) Utilize various scales of models instead of just one

c) Use more synthetic tasks OR include a component studying pretrained language models

Questions for authors
  1. Is there a reason the looped transformer was not included in the study? If I am misunderstanding a difficulty in comparing it to the models currently in the study, please clarify.
3 comments

We thank the reviewer for the suggestions. We have prepared a revised version in which updated text is marked in cyan. The anonymized revised Introduction, Discussion and new Appendix can be viewed at https://images4.imgbox.com/f9/4a/SN2O8vgK_o.png, https://images4.imgbox.com/e4/18/QsAb8udI_o.png and https://images4.imgbox.com/42/bf/yuKKytzs_o.png.

W1, (a) and Q1: Looped Transformer We have added a looped Transformer test to the study (Appendix A.9) and revised the Introduction and Discussion ( see above).

Following 2502.17416, we use the same four-layer backbone and apply the two middle layers 6 times, with the same evaluation sets. The looped model has the same parameters as the other variants but uses more computation as it updates all positions in 14 layer passes, while latent models use 4 layer passes over the prompt and one new position per thinking step, so the looped model uses ~ x3 FLOPs. On the main OOD evaluation, the looped model reaches around 95%. Strong depth generalization is expected for the looped model, given its recurrence over depth and its larger computation. Second, it is not affected by the two degree manipulations. Third, when the candidates have child nodes (they are all leaf nodes in training), its performance drops and like CoT, it more often selects a child of the correct candidate, while the latent models are still the best. This suggests that with a fixed number of iterations and no explicit training on when to stop, the looped model does not develop a stopping mechanism automatically, but we haven't tested adaptive halting.

Q1 there is no difficulty in comparing its behavior, and we should have included it. We did not include it initially because our focus was on models that reason step by step over sequence positions, like CoT. We were especially interested in this because in the Coconut paper the variant trained without the curriculum is no better than the no-CoT baseline, and many follow-up studies use CoT compression or RL to get latent reasoning. We first found that Coconut-style latent reasoning can work without RL or extra guidance like CoT traces (Full-latent), and introduced Bottleneck-latent to decouple the intermediate computation from the question and study its mechanism. Looped Transformers recur over depth instead of over sequence positions, so we left them out. They are now included in the revised version.

The difficulty is only in the circuit analysis. Our method transplants the state between steps, which is a single vector in Bottleneck-latent but is spread over all positions in a looped model. So the circuit analysis stays on Bottleneck-latent, and how the looped model works internally needs newer methods, which we are exploring actively.

We think the new result supports the main point in a broader form. Both the looped model and our latent models do recurrent computation in latent space, over depth or over sequence positions, and both generalize to longer paths better than Direct, CoT and Pause, requiring no extra information like proof during training. The Discussion now says this, and the Introduction now discusses related looped transformer works mentioned.

W2 and (b): model scale We have not been able to run this in the rebuttal period. We plan to train a larger CoT and a larger Bottleneck-latent and test whether they still differ on OOD and on the degree tests, and will report it in the final version.

W2 and (c): more tasks or pretrained models We tested the models on another task, modulo-pp (p=11p=11) arithmetic (Appendix A.10). The results are preliminary. On the ID test set, CoT and Full-latent are the best (99.9% and 98.1%). On longer expressions all variants drop a lot, but Full-latent is still the best (36.6%, vs 23.4% for CoT and 21.1% for Direct; chance level is 9.1%). However, bottleneck-latent has not converged on this task under the same training budget. We are working on this (we think this is related to weak supervision signal as in this task only one token is used for answer and bottleneck latent has a a decoupled structure) and will update once finished.

We agree that showing the same effect with pretrained language models would be even better. Originally we tried GPT-2 as Coconut did, and Full-latent works better on ProsQA-Ext(not shown in the manuscript) than the direct variant. For the circuit analysis purpose and the consideration to eliminate potential interference from knowledge in pretraining, we later focused on the smaller architecture that works and trained everything from scratch.

I thank the reviewer for the changes and inclusion of looped transformer. I think my concerns around mostly b) but also c) still stand, so I will maintain my score.

AuthorsOpenReview ↗

Second round response

We thank the reviewer for recognizing that we have addressed the concern about the looped transformer.

We would like to further clarify the two remaining points.

For W2 and (c), we believe the new results do support our claim. Please let us further clarify this. The Full-latent model and CoT model are at comparable in-distribution accuracy, but the Full-latent model achieves about 20% higher OOD accuracy than CoT. The Bottleneck-latent model reaches OOD accuracy similar to CoT despite substantially lower in-distribution accuracy. Together, these results suggest that latent reasoning generalizes better than CoT models. We apologize that this was unclear.

Regarding the concern about the lack of experiments on larger models, we agree that the results can vary across model sizes and architectures. However, how results change with model size and architecture is a broad question that is actively studied in the field. We are sincerely sorry that we cannot cover all of these in this study.

But we do believe this work makes a significant contribution to the field. It demonstrates that latent models generalize better than CoT models, and it further identifies the underlying circuits in the bottleneck-latent models. We hope the reviewer can kindly improve the score to reflect the contribution of this work.

Reviewer_3ZMUOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper compares five reasoning interfaces—Direct, CoT, Pause Token, Full-latent, and Bottleneck-latent—using the same small GPTNeoX backbone on a synthetic graph-reachability task. Through representational analysis, counterfactual state transplantation, and circuit pruning, the authors argue that the bottleneck model learns a recurrent forward-search algorithm.

Critical strengths
  1. The included models of this paper use the same backbone, tokenizer, dataset, and training budget and are trained from scratch, reducing confounds from pretraining.

  2. The forward-search interpretation is supported by multiple complementary tests. RSA shows stepwise alignment between latent states and graph frontiers, while connectivity-swap interventions produce the predicted depth-dependent state changes.

Critical weaknesses
  1. The training data still contain systematic degree shortcuts. This conflicts with the earlier claim that superficial features cannot predict the answer.

  2. The circuit analysis concerns a four-layer model on one synthetic task and may depend on the specific training settings.

  3. The interfaces differ in more than intermediate computation. CoT receives explicit proof supervision, while Bottleneck-latent removes access to the prompt cache during answer generation.

Questions for authors
  1. How does performance change over a wider range of recurrent steps and on paths substantially deeper than 12 hops?

  2. Do the latent models retain their advantage when training and OOD graphs are produced by the same generator and differ only in path length?

3 comments

We thank the reviewer for the comments and have prepared a revised version, in which updated test is marked in cyan. The anonymized revised intro, discussion and new appendix can be viewed at https://imgbox.com/SN2O8vgK https://images4.imgbox.com/e4/18/QsAb8udI_o.png and https://images4.imgbox.com/42/bf/yuKKytzs_o.png.

W1 Sorry for the confusion and thank the reviewer for pointing this out, the reviewer is right that the two statements conflict, and section 3.2.2 was the wrong one. Our generation procedure checked a set of heuristics, including candidate in-degree and out-degree, candidate position and label statistics, alone and in combination. On the 400k training set, choosing the candidate with lower in-degree gives 49.9% accuracy, and a logistic regression and a depth-3 decision tree over the audited features give 49.9% and 51.6%. So candidate in-degree is not predictive of the answer in the training set, and the sentence "correct candidates tend to have lower in-degree" should not have been written. What Fig. 3 shows is that Direct and Pause are still sensitive to candidate in-degree when we change it in matched pairs. We now describe this as sensitivity to local graph features, not as a shortcut that is available in the training data. The degree of the query root's successors was not part of the audit, and we are computing its statistics on the training set now. We have also changed the Methods sentence to correct that the heuristics we checked are near chance, since we cannot guarantee that all possible shortcuts are removed.

W2 We agree that the circuit analysis is for one four-layer model on one task, and we do not claim that the same components would be found under other settings. We now emphasize more this in the new Discussion. To test whether the behavioral results hold on another task, we ran experiments on a modulo-pp arithmetic task (A.10). Full-latent has an advantage on the OOD test set (36.6%, vs 23.4% for CoT and 21.1% for Direct while chance level is 9.1%), though no variant solves the longer expressions. Bottleneck-latent has not converged on this task under the same training budget and we're working on this currently. We will keep it updated once finished.

W3 These differences are part of the model designs we want to compare. CoT receives explicit proof supervision. Direct, CoT, Pause, Full-latent and Looped (added) retain access to the prompt during answer generation. Only Bottleneck-latent accesses the prompt during intermediate computation but not when generating the answer. Our goal is to understand how these designs lead to different learned solutions to the same task. Two comparisons differ in only one factor: Full-latent and Pause have the same number of thinking positions, the same prompt access and no proof supervision, and differ only in whether the input at each thinking step is the feed-back state or a fixed embedding; Full-latent and Bottleneck-latent differ only in prompt access at readout. We agree that for CoT, proof supervision and token-level steps are not separated.

Q1 As we replied to Reviewer jHQD, we require the same correct and incorrect chain length in our OOD evaluation, and 26 different symbols were used during training, so H=12H=12 is the largest depth we can test. The models are trained on H=3H=3-66, so this still tests generalization well beyond the training range. For larger depth, we expect the models to fail gradually, given to the fixed budget at evaluation t9ke. For recurrent steps, Fig. 4C tests K=5,6,7K=5,6,7 at evaluation (trained with K=6K=6): Bottleneck-latent keeps its accuracy, while Full-latent is sensitive to K=5K=5 and Pause to K=7K=7. We have not tested a wider range.

Q2 Not exactly. The training graphs are generated independently to match ProsQA's graph-complexity statistics, while the 7-12-hop OOD graphs are generated by Algorithm 1 (Appendix A.1), where the two candidates are the endpoints of two chains of the same length. So the training and OOD sets differ in the generator as well as in path length. The degree tests (Fig. 3) use matched pairs with the same construction and show that the latent models are less affected at both short and long depths.

Thank you for the response. While some of my concerns have been addressed, my initial concern regarding the experimental evaluation protocol remains unresolved. Nevertheless, I will adjust my score to 2 in recognition of the authors' efforts to address the feedback.

AuthorsOpenReview ↗

Second round response

We thank the reviewer for recognizing the improvements to our manuscript. Please let us know what concerns regarding the experimental evaluation protocol remain, so that we can further clarify them.

If this is the only remaining concern and you are satisfied with our results, we hope you can further improve the score to reflect the contribution of this work. Thank you.

Reviewer_jHQDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper aims to understand the differences in thinking mechanisms used by 5 different forms of thinking: 1. direct (no CoT), 2. CoT, 3. Pause tokens, 4. Coconut style unsupervised latent thinking, 5. Latent thinking with a bottleneck introduced to prevent answer generation from attending to the prompt and it only has to rely on the latent tokens as the bottleneck. They choose a synthetic multi-hop reasoning task (ProsQA-Ext) and observe that while all 5 modes of reasoning can provide strong in-distribution performance, only the latent forms provide depth generalization OOD. Direct, CoT and Pause tokens tend to use shortcuts or statistical properties of the train data which allows them to generalize in-distribution well but they dont learn to properly reason.

On the ProsQa task the authors train on, all modes get in-distribution performance of over 75% with latent modes performing the strongest. Out of distribution, the performance of Direct, CoT and Pause token modes deteriorates with CoT observing the most severe deterioration followed by Direct and then Pause. The two latent modes retain the performance.

To understand this further, the authors perform a mechanistic interpretability analysis. First they want to test whether the model in any mode is performing a forward propagation reasoning from the query root node to the destination node. They perform a representational similarity analysis of intermediate representations with the algorithmic steps for forward propagation solution (without assuming a 1-1 correspondence between the steps), and observe that only the latent methods exhibit signs of performing forward propagation.

The above observation raises the question of how Direct, Cot and Pause were doing well in-distribution. The authors show that they are doing this by exploiting shortcuts present as local graph features from the data generating process. In particular, correct candidates tend to have lower in-degree than incorrect ones. By intervening to remove this specific shortcut, the authors observe that performance drops severely even within in-distribution problem sizes for Direct, Cot and pause. Performance also drops but less severely for the latent approaches suggesting that they learn to exploit the shortcuts to some degree as well.

Finally the authors perform a deeper circuit analysis into which of the latent modes are actually performing a recurrent reasoning algorithm and observe that only the bottleneck latent mode has a circuit which does this.

Critical strengths
  • The bottleneck latent design is interesting and seems to confer very interesting inductive biase in the toy problem the authors studied. A more thorough study of bottleneck reasoning would be exciting to see.

  • Disambiguating how different ways of training for reasoning capabilities in LLMs actually confer deep reasoning to different degrees is a very important research direction which the paper aims to pursue.

Critical weaknesses
  • Shortcut learning pitfalls are a known phenomenon and today's LLMs largely are able to circumvent it due to the diversity of tasks they are trained on. Focusing on a single toy task to make general claims about CoT or pause tokens does not provide a complete picture.

  • While the choice of small values for H is understandable from a perspective of tractability for mechanistic interpretability, it unfortunately puts into question whether the findings will transfer to larger scales (this is unfortunately a critic that applies to many mechanistic interpretability works)

  • The paper focuses only on a single toy problem and I remain skeptical that the conclusions will apply to more realistic settings. To make the main point of the paper robust, a broader exploration across wider set of toy tasks (combined with a deep dive mechanistic analysis of one task) might make a more compelling argument.

Questions for authors
  1. It is strange to see CoT underperform pause tokens in-distribution and also strange to see if underperform direct and pause OOD. Why is CoT OOD so bad compared to direct or pause?
  2. Have you ablated different choices of H ranges for train vs OOD?
  3. Part of your rationale for claiming that latent methods do not exploit shortcuts is OOD generalization. However, this alone doesnt necessarily imply shortcuts arent being exploited.
  4. Can we better test the hypothesis that only direct, cot are exploiting shortcuts? You do present analysis of two specific shortcuts but its unclear if latent is using a different shortcut. perhaps by generating more diverse data shortcuts go away. this could be an evidence in favor of this hypothesis?
3 comments

We thank the reviewer for the comments and have prepared a revised version, in which updated test is marked in cyan. The anonymized revised intro, discussion and new appendix can be viewed at https://imgbox.com/SN2O8vgK https://images4.imgbox.com/e4/18/QsAb8udI_o.png and https://images4.imgbox.com/42/bf/yuKKytzs_o.png.

W1 We agree a single task cannot give a full picture of reasoning in LLMs. To clarify, we claim that latent models generalize better than CoT models under the same controlled setting without extra guidance like proofs, not that CoT cannot generalize with further training or more diverse datasets. We aim to understand how these models naturally learn different ways to solve the same task. We have revised the Discussion to acknowledge this and to note that whether our findings hold for larger models and a wider range of tasks still needs to be tested.

W2 We used 26 different symbols in training, and in the main OOD evaluation we require the correct and incorrect chains to have the same length. Under these constraints, H=12H=12 is the largest depth we can test. Since the models are trained on H=3H=3-66, the OOD evaluation still tests generalization well beyond the training range. For greater depth, we expect the models to fail gradually, because the number of latent steps is fixed at evaluation, even if a model emulates a for-loop execution, it would fail eventually.

W3 We agree that more tasks are needed for a more general principle, so we added a modulo-pp (p=11p=11) arithmetic task (Appendix A.10), though the results are preliminary. On the ID test set, CoT and Full-latent are the best (99.9% and 98.1%). On the OOD set with longer expressions, all variants drop a lot, but Full-latent is the best (36.6%, vs. 23.4% for CoT, 21.1% for Direct and 19.3% for Pause; chance is 9.1%). So on another task, latent thinking without human guidance or RL again generalizes further than the CoT and pause variants, although no variant solves the longer expressions. Bottleneck-latent has not converged under the same training budget (78.4% on ID), so we report its numbers only for completeness. This may be because its separate answer pass gets a weaker training signal when there is only one output token, but further probing is needed for this.

Q1 On the ID test set both CoT and Pause are accurate, and CoT declines as HH increases. For OOD, a CoT LM must generate a reasoning trace and a final-answer statement, so we studied the two parts separately (Appendix A.8). In 55.2 ±\pm 12.7% of OOD trials, CoT generates a valid proof but no correct final-answer statement. We then removed the last complete statement from each output, provided the prefix "ROOT is", and scored the next token. On the OOD set (7-12 hops), accuracy increased from 14.0 ±\pm 1.7% to 70.6 ±\pm 11.3% (mean ±\pm SEM across four seeds), which is still lower than the two latent models. With the prefix, CoT is still sensitive to the in-degree minus out-degree of the query root's successor. So the major reason why CoT fails on longer-hop problems is the failure to generate the final-answer statement, not choosing the incorrect candidate.

Q2 We have not run this yet, as our resources are focused on the arithmetic task above. We are currently shifting the HH border between ID and OOD, finding optimal hyperparameter for each H and will update the full results in the final version.

Q3 and Q4 We agree that OOD generalization alone does not imply that shortcuts are not used, and that our two shortcut tests cannot rule out all other shortcuts in the latent models. We do not claim that the latent models use no shortcuts. in the degree tests they are less affected, not completely unaffected. Two other results support our claim. First, in the connectivity-swap experiments (sec 3.3), the paired graphs have the same node degrees and differ only in which candidate is reachable, and transplanting the latent state from the swapped graph changes the answer of Bottleneck-latent variant. A degree shortcut cannot explain this.

Second, as suggested, we built an additional test set in which candidates have nonzero out-degree (they are leaf nodes in training and in our previous tests). Accuracy drops for all variants, the latent models are still the best, and CoT and the newly added looped variants more often select a child of the correct candidate (Appendix A.9). We have not yet retrained all the models on data without the degree shortcuts, which would be the more direct test.

Reviewer_jHQDOpenReview ↗

Thanks the authors for their response

I thank the authors for their response but unfortunately my concerns largely remain.

  • W2: The restriction of 26 symbols dictating the constraint on the depth H is artificial. One can always use a larger set of symbols to allow for deeper H.

  • W3: Thanks for adding these additional experiments. The different conclusions reached regarding some of the methods further justifies my concerns on the claims requiring stronger experimental evidence to make. You yourselves mention "further probing is needed for this.".

I do believe the paper has potential and urge the authors to consider developing it into a more thoroughly fleshed out exploration in future work. I will maintain my original rating.

AuthorsOpenReview ↗

Second round response.

We thank the reviewer for recognizing that most of the concerns have been addressed and for acknowledging the potential of our work.

We would like to further clarify the two remaining points.
W2. We agree that the 26-symbol is a design choice, not a limit. However, our OOD evaluation already extends the hop from 3–6 (in-distribution) to 7–12, up to twice the maximum training depth. In addition, all models will fail eventually as depth keeps growing. If the reviewer has a specific hypothesis that deeper H would change, we would be happy to test it. We respectfully believe this limitation does not affect the robustness of our main findings or the significance of the work.

W3. We would like to clarify our new results: The new results support our central claim. At comparable in-distribution accuracy, the Full-latent model achieves about 20% higher OOD accuracy than CoT. The Bottleneck-latent model reaches OOD accuracy similar to CoT despite substantially lower in-distribution accuracy. Together, these results suggest that latent reasoning generalizes better than CoT models. We apologize that this was unclear.

We fully agree that additional experiments would further strengthen the manuscript. However, we believe these further explorations are beyond the scope of our current work. In light of these clarifications and the new results, we would be grateful if the reviewer could consider updating the score to reflect the contributions of the current work. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.