acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Good Latent Thought for Reasoning?

Abstract

Latent reasoning promises shorter generation by replacing token-level traces with compact continuous thoughts, but these representations must both preserve the reasoning trace and be reliably generated from the input. We ask which properties of latent thoughts support reliable reasoning under compression, using controlled permutation state-tracking tasks and varying compression and representation structure. Our findings are threefold. (i) At moderate compression, autoregressive (AR) generation remains highly accurate with far fewer emissions: 20 latent emissions recover a 160-token trace with near-perfect final-state accuracy, matching or exceeding a parameter-matched token-AR baseline with \(8\times\) fewer emissions. (ii) At higher compression, reconstruction remains accurate while generation fails. Factorization or blockwise encoding restores accuracy, showing that latent structure strongly affects the ability to generate. With further compression, errors persist even with correct prefixes and on a simpler affine composition task, revealing a composition-capacity limit on how many state transitions the generator can realize within one emission. (iii) Iterative refinement is not a universal remedy. Diffusion does not recover highly compressed composition. On context-dependent graph walking, however, compression reverses the generator ordering: AR falls from perfect to chance, while diffusion improves. Together, these results show that usable compression depends on both latent structure and the computation packed into each emission. Effective latent reasoning therefore requires the right computational granularity for the representation, generator, and task.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N4fyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks, in a fully controlled synthetic setting, what makes compressed "latent thoughts" usable. A frozen β-VAE codec compresses every R tokens of a reasoning trace into one latent, optionally split into K sub-slots or encoded blockwise. A generator, either latent AR with a flow head or latent diffusion, then learns to emit these latents from the prompt. The main task is S5 permutation tracking (5 tokens per state, up to 32 steps, starting from the identity with four generators), with an affine task and a layered graph walk as contrasts. The findings are:

  1. At moderate compression, latent AR stays near perfect with far fewer emissions: 20 latents for a 160-token trace give 0.994, versus 0.962 for a parameter-matched token AR.
  2. At R=16, reconstruction is perfect but generation collapses (0.003). Factorized (0.93) or blockwise (0.95) codes fix this, while a 5× larger generator does not (0.006).
  3. At R=32, errors remain even with a correct prefix, which the authors call a "composition-capacity limit".
  4. Continuous latent diffusion does not rescue S5 (discrete code-token diffusion does better, but changes several things at once). On the graph walk, compression "reverses" the ordering: AR falls to 0.20, described as chance, while diffusion improves from 0.40 to 0.60.
Critical strengths
  1. The design separates "can the trace be encoded" from "can the latents be generated". It does this with a frozen codec whose reconstruction ceiling is measured, an exact oracle, and correct-prefix probes. Finding 2 comes straight out of this design and is large and robust: at fixed compression, code structure moves accuracy from 0.003 to 0.93–0.95, while 5× scale does nothing.

  2. The reporting is transparent. It gives three generator seeds with min–max ranges, says which pre-registered hypotheses failed, and lists caveats openly in the appendix.

Critical weaknesses
  1. The two most novel conclusions, Finding 3 and the graph-walk reversal, leave obvious alternatives open.

    Composition-capacity limit. The paper blames the R=32 failure on how many state updates one emission must carry, but its own probe shows strong position effects at a fixed load. With a correct prefix (Fig. 5c, one checkpoint), the first latent gets all six states nearly right (1.00 … 0.93). The second gets three right and then drops to 0.15, and the third and fourth get their first contained state right only 58% and 19% of the time. Similarly, with the blockwise codec, diffusion reaches 0.946 at D=16 but 0.027 at D=32 (Table 8), with the same load per latent. Coverage is one candidate. With the start fixed at the identity and four generators, the argument of Lemma 1 bounds the first R=32 block at 4^7 = 16,384 possible contents, about 39 exposures each among the 640K length-32 programs, while later blocks can take up to 0.5–2M contents. The affine task has few contents and still fails, so coverage is probably not the whole story. Either way, a single-checkpoint probe does not establish a per-emission capacity limit.

    Graph-walk reversal. Each layer is a random function that need not be a permutation (C.4), so much of the answer can be read off the prompt without tracking the walk. In a quick simulation of the sampling in C.4, guessing the final node from the last layer's edges alone is right 46% of the time, and from the last two layers 58%. So 0.20 is the level of blind guessing, not of a model that reads the prompt but does not track, and diffusion's 0.40–0.60 is within reach of such a shortcut. AR's collapse also uses a global (entangled) walker codec at three states per latent, the same kind of code that failed on S5 at a similar load (R=16). Finally, the reported diffusion seeds (3–5) replaced an earlier cohort after inspection (D.4).

  2. The scope does not support the general prescriptions in the title, abstract and Section 4. Everything is synthetic, with models of at most 319M parameters trained from scratch and one codec per configuration. No existing latent-reasoning method (e.g., Coconut, CODI, CoLaR) and no language task is tested. The continuous diffusion baseline also looks undertrained. It gets 0.019 at R=4, D=32, where AR gets 1.000, and 0.721 ± 0.476 even at R=1. It is trained for 25k updates, while the discrete code-token diffusion that does much better (0.64 vs 0.01 at R=16, K=4, D=32) gets 75k. The paper's own controls show that masked diffusion is very budget-sensitive (0.14 at 8k vs 1.00 at 24k updates for a small token-level model, Table 13). Finally, the main lesson, that a code can be easy to decode yet hard to generate, is close to CALM (Shao et al., 2025). CALM reports that reconstruction-only autoencoders give brittle latent spaces and regularizes them. The paper's β-VAE codec follows a similar recipe, but CALM is not discussed.

Questions for authors
  1. Why is the first latent near perfect over six states while the third and fourth often miss their first contained state? Please report correct-prefix accuracy per latent, including split states, over several seeds.
  2. To separate per-emission capacity from position and coverage effects, please try state-aligned compression (e.g., R=30 with K=6, so each sub-slot is exactly one state) and an R=32 run with more training programs.
  3. For the graph walk, please report a baseline that reads the prompt but does not track the walk, or rerun with bijective layers. Please also add a blockwise walker codec for AR and report the earlier diffusion cohort.
  4. Please train continuous diffusion with the same 75k budget as the code-token models and show its training curves.
1 comment

Dear Reviewer N4fy, thank you for reviewing the paper and taking the time to read our response and new results. Since the rebuttal deadline is approaching, we just wanted to let you know that we are happy to address any of your further comments. If our reply addressed your comments properly, we would greatly appreciate your consideration of a higher score. We appreciate your time and fully understand that you may need more time to review the results!

Reviewer_ExmzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what makes continuous latent thoughts usable for reasoning, by separating what a latent can encode from what a generator can produce from the prompt. It trains a frozen encoder-decoder that compresses every R trace tokens into one latent, and then trains latent AR and latent diffusion generators in this space on synthetic permutation tracking, affine composition, and graph walking tasks. Experiments show that moderate compression keeps AR accurate with 4 to 8 times fewer emissions, and that factorized or blockwise latents recover accuracy at a compression ratio where the entangled latent fails. At higher compression a generation bottleneck remains, and on graph walking compression makes AR collapse while diffusion improves.

Critical strengths

The paper studies a meaningful question for latent reasoning. Separating the codec stage from the generator stage and reporting a reconstruction ceiling for each codec makes it possible to tell whether a failure comes from information loss or from generation.

The idea of comparing latent AR and latent diffusion in the same frozen latent space is reasonable. Since diffusion adds iterative computation without adding latent positions, it gives a natural way to test whether more computation can make up for stronger compression.

The paper includes several useful controls, such as token AR baselines matched in both parameter count and backbone depth, a lossless-prompt control for diffusion, and a leakage check on grading.

Critical weaknesses

All experiments are conducted on three synthetic tasks with very small vocabularies (e.g., only four operations for the main S5 task), and there is no evaluation on any natural-language reasoning benchmark. In contrast, the cited latent reasoning works such as Coconut and CCoT are evaluated on GSM8K or ProsQA. However, the title and conclusion give general design advice for latent reasoning, e.g., “Effective latent reasoning therefore requires the right computational granularity”. It is not clear whether findings such as the factorization gain or the ordering of AR and diffusion still hold when traces are in natural language. Please add at least one natural-language reasoning setting, e.g., GSM8K or ProsQA with a small model, or narrow the claims to the tested tasks. Several conclusions are drawn from only three seeds with large variance. On S5, the final-state accuracy of K=2 at D=32 ranges from 0.236 to 0.834 across seeds (Table 7). On graph walking, the paper claims that diffusion performs better than AR under compression (“compression reverses the generator ordering”), but with R=3 at D=8 latent AR ranges from 0.22 to 0.99 and diffusion from 0.50 to 0.91 (Table 9). With such overlapping ranges, comparing the means is not reliable, and the advantage of diffusion is only clear at D=12. In addition, AR and diffusion use different seeds (0, 1, 2 vs. 3, 4, 5). Could the authors use the same seeds for both generators, and report more seeds or a paired comparison on the shared test programs (e.g., how often each generator wins)? It would also be better to analyze why diffusion clearly outperforms AR at D=12, and whether this advantage still holds for longer chains (e.g., D>12). The codec training recipe changes with the compression ratio, but the paper does not explain how it is chosen. As shown in Table 1, R=1 and R=4 use 4/4 encoder/decoder layers, 15k updates, and cycled chain lengths, while R=8 and R=16 use 6/6 layers, 45k updates, and length-biased sampling, and some R=32 codecs use a 12-layer decoder with 55k updates. Therefore, comparisons across R also change the codec size and training. For example, R=1 latent AR is worse than R=4 and R=8 (0.917 vs. 1.000 and 0.994 at D=32, Table 3), which may partly come from its smaller codec. Could the authors describe the rule used to select these recipes, and explain whether the conclusions across R are affected by them?

Questions for authors

The tasks in this paper are very small in scale. For example, the S5 task has only about 14 token types, and each state can take only 120 values. In natural-language reasoning, the vocabulary has tens or hundreds of thousands of tokens, so the latent space is likely to be much more complex. Is it possible that a compression scheme that works well in this small setting, such as factorization or blockwise encoding, no longer works at that scale? How do the authors make sure that the current experiments are representative of real latent reasoning? If the authors could add an experiment with a larger model in a more general setting, e.g., natural-language reasoning, and show that the findings still hold, it would be very helpful. Several results vary a lot across the three seeds, e.g., K=2 on S5 (0.236 to 0.834 at D=32) and both AR and diffusion with R=3 on graph walking (0.22 to 0.99 and 0.50 to 0.91 at D=8). Why are these settings so sensitive to the seed? Given only three seeds and such large ranges, how do the authors make sure that the conclusions drawn from the means, e.g., the comparison between AR and diffusion, are not caused by seed choice? Could the authors report results with more seeds? On graph walking, AR and diffusion use different seeds (0, 1, 2 vs. 3, 4, 5). Could the authors compare the two generators with the same seeds, or give a paired comparison on the shared test programs, e.g., how often each generator wins? Please also explain why diffusion clearly outperforms AR at D=12 but not at D=4 and D=8, and whether this advantage still holds for longer chains (e.g., D>12). The codec recipe in Table 1 changes with the compression ratio, including the number of encoder/decoder layers, the training updates, and the chain-length sampling. What rule was used to choose these recipes for each R? Since comparisons across R also change the codec, how do the authors make sure that the conclusions across R, e.g., R=1 latent AR being worse than R=4 and R=8, come from the compression ratio rather than from the different codec recipes?

3 comments

Dear Reviewer Exmz, thank you for reviewing the paper and taking the time to interact with us and read our new response/results. Since the rebuttal deadline is approaching, we just wanted to let you know that we are happy to address any of your further comments. If our reply addressed your comments properly, we would greatly appreciate your consideration of a higher score. We appreciate your time and fully understand that you may need more time to review the results!

Thanks for the clarification. Now the ProntoQA setting is more clear to me.

Although the two codes pool the same 16-token blocks, I think they are not equivalent. The paper itself mentions in Appendix A that “the encoder contextualizes the full trace before pooling, so identical local blocks can receive different latent representations”, while the blockwise codec “makes each latent a function of its own block”. So for the global code, the space that each latent need to encode is not limited to the at most 7,680 contents of its block, and can be much larger. In my side, its failure suggests that the generation breaks down when this space becomes large. Similarly, the “10× more possible local contents” only counts the token states that each latent spans. But the entangled R=8 code also uses the global encoder, suggesting the latent may carry information beyond the "two or three" states, which make real space larger than authors' estimation. So this comparison cannot show that the factorized latent has a larger space to encode.

For the Last point, what I mean is not that the two-stage structure is a toy, but the experiments are too small to prove the efficiency of the method. CALM and LaDiR also train an autoencoder first, but the experiment setting are quite different. For example, CoLaR uses compression factor like 2 or 5 on math reasoning sequences with the vocabulary of more than a hundred thousand tokens. Therefore latents need to encode a quite larger space. In the paper, the author claims “The loss arises in generation, not reconstruction.”. This maybe because each latent only need to encode within a small space, and the paper never tests on a large space encoding setting. As revised paper itself also says the conclusions are not extended to CoLaR. The response says this guidance is “testable”, but it is still not tested in such systems. I think the most direct way to prove this claim is to do the same reconstruction–generation analysis in a similar setting, for example, on GSM8K with the same base model as CoLaR.

If the method only works on the limited synthetic setting, but not in real settings like reasoning with pretrained LLMs, I think the contribution to the practical use and the academic value are both limited. Especially the paper is motivated by latent reasoning in language models but now narrows the claim and does not extend the conclusions to pretrained models in the revision.

Overall, I will keep my current rating.

Thank you for the careful follow-up. We appreciate your engagement.

On the two codes. We agree that they are not equivalent, and our phrase “exactly the same 16-token blocks” referred only to token content. The global codec contextualizes the whole trace, so its latents can occupy a much larger space than the block contents. Token-level counts therefore do not bound that space, for the R=8 vs. R=16 comparison either. Your reading, that generation breaks down when the space each latent must encode becomes large, is in fact the mechanism the paper reports. At fixed compression and token content, how the code organizes information (global contextualization, factorization, block locality) changes the space the generator must produce, and with it how much training generation needs. The blockwise and factorized codecs are interventions on exactly this variable.

On scope. This is an mechanistic analysis paper, not a method paper: we do not propose a latent-reasoning method whose efficiency needs to be demonstrated. Our aim is to identify mechanisms: whether a compression failure comes from reconstruction or from generation, and which properties of a latent make generation learnable. Answering this requires several controls at once:

  • varying compression, code structure, per-emission load, training budget and task family independently;
  • checking every intermediate state against an exact oracle;
  • holding reconstruction fixed with a gate. GSM8K with a pretrained model does not allow this. Its intermediate states cannot be checked exactly, and reconstruction, vocabulary and pretraining are confounded.

We agree with you, however, that “the loss arises in generation, not reconstruction” is a property of our gated setting. With a vocabulary of more than 100k tokens at CoLaR’s compression factors, reconstruction may itself become limiting. A reconstruction–generation analysis on GSM8K with CoLaR’s base model is the natural test of whether our mechanisms carry over, and we view this as a follow-up work and state it as such in the limitations.

Thank you again. This discussion has sharpened how we state the paper’s scope.

Reviewer_D3j8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Latent reasoning replaces the explicit chain-of-thought (CoT) of language models with continuous vectors, enabling reasoning with shorter sequences. However, this paper points out that such approaches rest on the assumption that latent representations properly support reliable reasoning, and explores structures that can shorten reasoning traces without sacrificing the performance of reliable reasoning generation. Specifically, by varying the compression rate, latent structure, and codec context, it investigates when generation fails even though reconstruction succeeds. As a result, the paper reports two limits of latent compression (i.e., the code-structure limit and the composition-capacity limit), and shows that AR and diffusion models each have distinct reasoning tasks on which they are stronger.

Critical strengths
  • Thorough Analysis. Through a thorough analysis based on latent reasoning models, the paper carefully reports the degree of compression of latent representations required for reasoning traces, along with their other properties. It will be a good reference for researchers working on related topics.
  • Good Writing. The paper is very easy to read, and its organization, figures, and experimental results are well polished.
Critical weaknesses
  • While the paper conducts various experiments to answer an interesting question, it was unfortunate that the benefit of latent reasoning is mainly framed as shorter reasoning traces. The information gain from passing continuous vectors instead of discrete tokens, or the reasoning benefit from the geometry of the latent space itself, is also important, but beyond accuracy-level comparisons (Table 3, Table 12) these are not analyzed.
  • Also, the main questions in this paper are limited to latent reasoning methods that obtain latent representations through autoencoding, e.g., “reconstruction succeeds but generation fails” and the recovery via factorization/blockwise encoding. If the paper aims to make general claims about latent reasoning, these phenomena should have been systematically observed in other setups as well. The findings from this paper’s experimental setup alone cannot be generalized to latent reasoning in general.
Questions for authors
  • Would the reconstruction–generation gap persist if the codec and generator were trained jointly? More broadly, how do the authors expect these findings to transfer to latent reasoning without a frozen codec (e.g., Coconut, CoLaR)?

Comments on the paper

1 comment

Hi Reviewers,

The authors have provided responses. Please check if they have addressed your concerns.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.