- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks, in a fully controlled synthetic setting, what makes compressed "latent thoughts" usable. A frozen β-VAE codec compresses every R tokens of a reasoning trace into one latent, optionally split into K sub-slots or encoded blockwise. A generator, either latent AR with a flow head or latent diffusion, then learns to emit these latents from the prompt. The main task is S5 permutation tracking (5 tokens per state, up to 32 steps, starting from the identity with four generators), with an affine task and a layered graph walk as contrasts. The findings are:
- At moderate compression, latent AR stays near perfect with far fewer emissions: 20 latents for a 160-token trace give 0.994, versus 0.962 for a parameter-matched token AR.
- At R=16, reconstruction is perfect but generation collapses (0.003). Factorized (0.93) or blockwise (0.95) codes fix this, while a 5× larger generator does not (0.006).
- At R=32, errors remain even with a correct prefix, which the authors call a "composition-capacity limit".
- Continuous latent diffusion does not rescue S5 (discrete code-token diffusion does better, but changes several things at once). On the graph walk, compression "reverses" the ordering: AR falls to 0.20, described as chance, while diffusion improves from 0.40 to 0.60.
Critical strengths
-
The design separates "can the trace be encoded" from "can the latents be generated". It does this with a frozen codec whose reconstruction ceiling is measured, an exact oracle, and correct-prefix probes. Finding 2 comes straight out of this design and is large and robust: at fixed compression, code structure moves accuracy from 0.003 to 0.93–0.95, while 5× scale does nothing.
-
The reporting is transparent. It gives three generator seeds with min–max ranges, says which pre-registered hypotheses failed, and lists caveats openly in the appendix.
Critical weaknesses
-
The two most novel conclusions, Finding 3 and the graph-walk reversal, leave obvious alternatives open.
Composition-capacity limit. The paper blames the R=32 failure on how many state updates one emission must carry, but its own probe shows strong position effects at a fixed load. With a correct prefix (Fig. 5c, one checkpoint), the first latent gets all six states nearly right (1.00 … 0.93). The second gets three right and then drops to 0.15, and the third and fourth get their first contained state right only 58% and 19% of the time. Similarly, with the blockwise codec, diffusion reaches 0.946 at D=16 but 0.027 at D=32 (Table 8), with the same load per latent. Coverage is one candidate. With the start fixed at the identity and four generators, the argument of Lemma 1 bounds the first R=32 block at 4^7 = 16,384 possible contents, about 39 exposures each among the 640K length-32 programs, while later blocks can take up to 0.5–2M contents. The affine task has few contents and still fails, so coverage is probably not the whole story. Either way, a single-checkpoint probe does not establish a per-emission capacity limit.
Graph-walk reversal. Each layer is a random function that need not be a permutation (C.4), so much of the answer can be read off the prompt without tracking the walk. In a quick simulation of the sampling in C.4, guessing the final node from the last layer's edges alone is right 46% of the time, and from the last two layers 58%. So 0.20 is the level of blind guessing, not of a model that reads the prompt but does not track, and diffusion's 0.40–0.60 is within reach of such a shortcut. AR's collapse also uses a global (entangled) walker codec at three states per latent, the same kind of code that failed on S5 at a similar load (R=16). Finally, the reported diffusion seeds (3–5) replaced an earlier cohort after inspection (D.4).
-
The scope does not support the general prescriptions in the title, abstract and Section 4. Everything is synthetic, with models of at most 319M parameters trained from scratch and one codec per configuration. No existing latent-reasoning method (e.g., Coconut, CODI, CoLaR) and no language task is tested. The continuous diffusion baseline also looks undertrained. It gets 0.019 at R=4, D=32, where AR gets 1.000, and 0.721 ± 0.476 even at R=1. It is trained for 25k updates, while the discrete code-token diffusion that does much better (0.64 vs 0.01 at R=16, K=4, D=32) gets 75k. The paper's own controls show that masked diffusion is very budget-sensitive (0.14 at 8k vs 1.00 at 24k updates for a small token-level model, Table 13). Finally, the main lesson, that a code can be easy to decode yet hard to generate, is close to CALM (Shao et al., 2025). CALM reports that reconstruction-only autoencoders give brittle latent spaces and regularizes them. The paper's β-VAE codec follows a similar recipe, but CALM is not discussed.
Questions for authors
- Why is the first latent near perfect over six states while the third and fourth often miss their first contained state? Please report correct-prefix accuracy per latent, including split states, over several seeds.
- To separate per-emission capacity from position and coverage effects, please try state-aligned compression (e.g., R=30 with K=6, so each sub-slot is exactly one state) and an R=32 run with more training programs.
- For the graph walk, please report a baseline that reads the prompt but does not track the walk, or rerun with bijective layers. Please also add a blockwise walker codec for AR and report the earlier diffusion cohort.
- Please train continuous diffusion with the same 75k budget as the code-token models and show its training curves.
Dear Reviewer N4fy, thank you for reviewing the paper and taking the time to read our response and new results. Since the rebuttal deadline is approaching, we just wanted to let you know that we are happy to address any of your further comments. If our reply addressed your comments properly, we would greatly appreciate your consideration of a higher score. We appreciate your time and fully understand that you may need more time to review the results!