acceptodds
Under review as a conference paper at ICLR 2027

Does Latent Chain-of-Thought Actually Reason? A Single, Unifying Taxonomy for Latent Reasoning

Abstract

Latent chain-of-thought replaces written reasoning steps with continuous hidden states, giving models additional steps to compute before answering. It remains unclear what computation these steps actually perform. High task accuracy and decodable intermediate answers do not establish that a model uses its latent chain to reach an answer. We study this question across twenty-one model-task configurations covering three training recipes, three reasoning tasks, and two backbones. We combine chain deletion and content replacement with probes and trajectory analysis to test whether the chain affects the answer, what it encodes, and how its states change. On logical tasks, COCONUT often carries an answer that is already readable before the chain begins. Its states settle early, and deleting the chain leaves predictions almost unchanged. Per-step supervision produces an ordered layout of reasoning steps and can make the model depend on the chain. Yet even an ordered chain can act as a placeholder, with deletion hurting performance while content replacement leaves answers unchanged. On arithmetic tasks, all tested recipes show dependence on chain content. These results show why readable latent states alone are insufficient evidence of reasoning. We introduce a taxonomy that separates what a chain represents from how the model uses it. Seen this way, earlier reports no longer compete: placeholder tokens, collapsed superposition and step-dependent causal roles are each consistent with a different kind of chain, each obtained on a different task, training recipe or measurement. Once a study says which property it tested, whether the model uses the chain, what the chain holds, or how its state moves, claims about latent reasoning can be compared.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Reasoning, chain of thought, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_DbvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper taxonomizes latent reasoning traces into carriers, empty, bare placeholder, laid-out placeholder, assembling, and unstructured using a series of diagnostic tests that measure causal use, encoded content, and state dynamics. The paper resolves disagreements between prior work that completed a subset of these tests. The paper makes a few surprising observations, including that readable final answers and apparent sequential reasoning in the latent reasoning trace do not mean the model needs the trace to get its final answer.

Critical strengths

S1) The ProsQA + Sim-CoT-Coconut result is interesting. It is surprising that you can decode the 'step k written into slot k' and yet you can replace them with noise/mean vector/donor vector and still get the same accuracy. This clearly demonstrates that readability does not imply usage.

S2) The paper does a good job of expanding diagnostic tests from prior work, and demonstrating that multiple diagnostic tests are needed to understand a latent reasoning trace.

Critical weaknesses

W1) The 'unstructured' label for the GSM8k results are likely due to the probe construction. The probe for ProsQA looks for intermediate results, but for GSM8k it only looks for the final answer. The paper says "GSM8k has no candidate list", but why not try the gold label solution for GSM8k? Of course that gold label solution is not unique; there are many valid solutions to a given GSM8k problem, but it seems like a good starting point. If you do find the 'step k written into slot k' result for the GSM8k gold solutions, then the 'unstructured' label should be changed to 'assembling'. Why do you think the unstructured label is correct given the current probe setup?

W2) Limited models, which have been studied by prior work (though less extensively). The paper studies Coconut, CODI, and SIM-CoT Coconut only; the first two have been studied by prior work with similar findings in their Section 4. How would the diagnostic tests and taxonomy apply to more recent latent reasoning architectures, such as full-bandwidth transformers or looped architectures?

Questions for authors

Suggestion 1) Figure 5 should be in the main paper.

Suggestion 2) "Where prior causal tests apply a single corruption (deletion (Dilgren & Wiegreffe, 2026)or noise(Cui et al., 2026)) each, we pair deletion with three replacements (noise, the dataset mean, and another problem’s vectors)" is not true; Dilgren & Wiegreffe also do the replace with dataset mean test.

Reviewer_5BUzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether latent chains contribute to answers, what information they encode, and how their states evolve. It studies 21 checkpoint-task configurations across COCONUT, CODI, and SIM-COT-COCONUT, using GPT-2 and Llama-3.2-1B-Instruct on ProsQA, PrOntoQA, and GSM8k. Finally the authors organize these observations into six operational chain types.

Critical strengths
  1. Through joint analysis of representation and behavioral dependence, the paper goes beyond examining what can be decoded from latent states to investigate whether those states affect the model’s answers.

2.The controlled comparisons and complementary diagnostics strengthen the empirical contribution. A matched six-slot latent budget facilitates comparisons across the three ways of latent reasoning.

Critical weaknesses

1.Several key verdicts depend on interventions whose interpretation is not fully pinned down, so the main text relies heavily on appendix to establish its verdicts. Even Figure 5, which presents the central taxonomy, appears in the appendix.

2.Two of the three tasks are near ceiling (ProsQA/PrOntoQA C0 ≈ 0.97–1.00) and are plausibly solvable in a single forward pass over the question. Consequently, limited degradation under corruption could reflect task simplicity, rather than establish that latent computation is generally unnecessary.

3.Finding 8 is weakly supported. With only two backbones that differ in scale, family and instruction tuning, attributing the effect to "the backbone, not the mechanism or the supervision" is weakly supported.

4.There are several broken references ("App. ??" in Table 5's caption, C.3, C.4, D.4, D.5), and Section A.8 is empty.

Questions for authors

1.How stable are the leaf assignments across training seeds for the checkpoints you trained? Would any cell change leaf?

2.Precisely which tensors are overwritten during substitution, and which key/value states and question-attention pathways remain available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.