- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This work investigates which information should be preserved in a latent state. The authors compare three different choices:
- a frontier state which keeps only active states at a current step
- a cumulative state which keeps all states reached so far with equal weight
- a weighted state which does the same as the previous, but weighing the values.
The theory treats these choices as abstract coefficient vectors over graph nodes (eq. 6) and measures the hidden width a single normalized latent vector needs for a later attention readout to succeed; a recurrent two-block transformer with a markovian assumption on the latent thinking (each thought depends only on the previous thought and the prompt tokens, Sec. 5.3) is then shown to implement these states.
They show that:
- If the question draws on multiple reached nodes (no node receives more than C/√m of the query weight, eq. 13), uniform cumulative is best and needs less width than frontier
- If the query family is known in advance, the ideal state is a weighted state with weights matched to the query; uniform cumulative is within a factor C² of this ideal
- If the query is not known in advance, uniform cumulative state is best (minimax over nonnegative weightings)
- Although not stated by the authors, the same algebra implies that if the final question only needs the current frontier, the frontier state is best
The authors validate their claims empirically on ProsQA with a lightweight router and a 2-layer GPT-2 block, where the models are trained to reproduce each prescribed state and the required width is measured, and show that these models exhibit the predicted ordering.
All in all, I believe the work is interesting but rather narrow. I would raise my score if the authors
- provide experiments on a coconut-style model end to end to study which state strategy is learned without any supervision
- rewrite the intro/problem setting sections in a way that better scopes their claims? Concretely I would want here
- A better explanation of the considered model architecture
- A justification of the assumptions made in the theory
- An explanation of how these claims could carry over to more realistic settings
Critical strengths
The result is interesting and novel. I believe that the space of theoretical results in latent reasoning methods (especially coconut-style decoding) is rather underdeveloped. I believe that this work tries to fill a gap which is quite real in the literature.
Critical weaknesses
- Main weakness 1: the authors assume (it seems to me?) a markovian Transformer model . This limits the theoretical model vastly in my opinion: access to all previous latent thoughts written to the KV cache implies a growing computation buffer. The results presented here would not hold in this case as a model could easily diffuse frontier/state information across the KV cache, which could easily be retrieved using a J-L embedding scheme or something of the sort.
- Main weakness 2: I found the overall paper hard to understand. It seems to me many details which would be crucial to me to better understand the breadth of the results are hard to find (see questions for more details).
- Main weakness 3: I find that some of the main theoretical results are a bit tautological. The tasks for which certain frontier types are best (e.g. frontier is best for final query on frontier) seem fairly obvious
Questions for authors
- It seems the model of the transformer assumed in the theoretical section is markovian and only assumes access to the previous latent though + the input tokens from the prompt. Is my assessment on this correct?
- Related to previous point. Do the authors have any empirical or intuitive justification for this specific definition of latent reasoning?
- Can the authors give more details on the training setup for the GPT-2 experiments? How many latent thoughts are given to the model (it seems only 1)? What is the training process (similar or not to the one of the "reasoning by superposition)?
- Can the authors comment on the third weakness? Is there something I am not seeing? Is there a way in which the results are not self evident?
- There is concurrent work about latent thinking (https://arxiv.org/abs/2604.06374, https://arxiv.org/abs/2604.04902) which I believe to be quite relevant which is not cited in your work. Namely in the first paper by Rizvi-Martel et al., there is an analysis of how state size is related to ability to maintain superposition of the frontier. I believe this analysis of state size is very related to your work.
- Would it be possible to provide experiments in which a model (coconut style) is trained end-to-end on ProsQA with only the answer loss and inspect what state type was learned? I would be interested to see what strategy models converge to without additional supervision
8 comments
Requested revision (Weakness 2) — What architecture, assumptions and applications does the paper cover?
We revised the abstract, introduction and problem setting to state the single-state interface. One normalized thought carries graph-search history between steps, and the final head reads that thought against a common set of candidate keys.
In the revised paper. The abstract identifies the single thought carried between steps. In §1, “Cumulative-superposition hypothesis,” we ask for the minimum hidden width at which all routing steps and the final selection succeed together with probability at least for each fixed query in the family. gives the scale of that minimum. §1, “Contributions,” distinguishes the terminal readout advantage, its realization across Transformer steps, and the guarantee for equal weights when future queries are unknown. §3 specifies what each block and the final head can access and interprets the query constraint in Eq. (13).
- Architecture (§§1 and 3). Section 3 follows one update from input to output. The first block uses the thought and source embeddings to identify eligible edges. The second writes their destinations into the next thought. This locates the choice between frontier and cumulative in the update itself. Section 3 then specifies what the final head can access. If each past frontier occupies a separate cache slot, a head can sum those slots at readout. The single-state comparison applies when historical information reaches that head through one vector.
- Why the assumptions enter (§3 and Appendix C). Independent Gaussian node embeddings and orthogonal competing directions in coefficient space let us derive the distribution of the largest competing score. Nonnegative weights model a query that pools supporting results, so a node retained by the state can contribute to the correct answer. Section 3 now interprets the concentration bound in terms of that query: if it gives equal weight to of reached nodes, its smallest admissible is . Thus the reader can tell how broadly the query draws on history. Appendix C gives the corresponding readout probability for a fixed correlated candidate bank under Gaussian embeddings.
- What transfers to learned models (§§6.2 and 6.4 and Appendix E). We test the predicted ordering of widths while holding the graph, query and competing keys fixed between policies. For Qwen we embed the carried thought in the native backbone with a fixed orthonormal basis and project the thought output back to the tested dimension before the recurrent update. At width 32 on both EntailmentBank and FinQA, the two-layer Transformer and GPT-2 each exceeded 90% with cumulative and stayed below 90% with frontier (one seed per task and router, §6.2, Table 1). With the 14B pretrained router on FinQA program-dependency graphs, cumulative and frontier reached 90% candidate-selection accuracy at latent widths 23.3 and 57.5, respectively (Table 1). We also tested the embedding assumption directly on 340 EntailmentBank proofs. Distinct reached statements encoded with their question had mean cosine similarity 0.937. With competing keys matched in size and weights, cumulative reached 90% at a mean projected width of 593.1, while frontier remained below 90% through width 1,536. Figure 5(b) shows how the retrieval curves shift with learned representations while preserving this ordering. Appendix E.2 separately reports program execution on the original FinQA questions.
Weakness 1 and Question 1 — Which part of the result requires a Markov interface?
Yes. Section 5.3 carries one thought across steps. Your proposed Johnson–Lindenstrauss (JL) retrieval scheme is viable when the final head can address earlier slots. Our contribution is a sharp width law and an optimal weighting rule for the case where history reaches that head through one thought. If a fixed query picks out a cached key by a margin , a projection that perturbs each score by less than preserves the winner. A reader can also add the cached sums of disjoint frontiers to recover the cumulative sum exactly, provided it retains the scales if those sums are normalized. Thus a cache can move accumulation from writing to reading.
What the width law adds. Applying JL to a fixed bank gives a sufficient dimension once a distortion tolerance is chosen. Even substituting the decision margin for that tolerance gives only a sufficient bound by this argument. Our readout ranks the same keys for any upstream process producing the same thought. Under our independent Gaussian embedding and orthogonal candidate-bank assumptions, Theorem 4.1 gives the minimum terminal width for the intended state as for fixed target success probability as grows. Here is the least alignment with an admissible correct query. The leading constant comes from the maximum of the competing scores. Comparing that maximum with the correct score establishes necessity as well as sufficiency. For a complete trace, the minimum width for the intended states also has matching upper and lower bounds up to constants. It scales as the larger of this terminal term and the routing term set by the weakest reusable node (§4, Eq. (17)).
Why the weighting matters. Suppose either of two reached nodes may be needed for routing, but the final query weights them in the ratio . Matching that query raises final alignment from to , while the weaker node's coefficient falls from to . Even an exact isometry preserves both consequences of the write. Theorem 5.1 uses the joint width law to choose weights before the future query is known. When every reached node must remain reusable, uniform cumulative weights minimize worst-case joint width scale over the full admissible family. For a known family they cost at most a sharp factor over the best fixed nonnegative weights. Section 5.3 then gives a sufficient width for a recurrent Transformer to realize the updates. A growing cache changes the number of readable slots, and the optimal width–slot trade-off remains open.
In the revised paper. §3 specifies which history the final head can address and explains the cached-frontier example. §4, Eq. (17) and §5.1, Theorem 5.1 give the joint width law and weighting guarantee. §5.3 shows how the recurrent two-block Transformer actually writes these states.
Question 2 — Why study this interface?
On ProsQA, restricting Coconut to the latest thought during thought generation preserved all 1,500 paired answers when the answer decoder retained full history.
In the revised paper. §2, “Recurrent computation and latent states,” discusses depth-recurrent models that update a fixed-shape state. Appendix E.1, Table 11 tests historical access during generation and decoding separately. §6.4 reports the change in paired predictions when both phases use only the latest thought.
This gives an empirical reason to study a single carried thought during generation even for a model trained with a full cache.
Depth-recurrent language models provide an architectural example. Geiping et al. update a fixed-shape state by , where is the input representation. Their state is a matrix spanning input positions. Our abstraction studies how one normalized thought in such an interface allocates capacity across reached objects. Geiping et al., §3.1
We also tested how much a trained full-cache model relies on access to earlier thought positions. After training standard Coconut with full history, we separately restricted historical thought access during thought generation and during answer decoding. Prompt access was preserved. We measured greedy exact match on all 500 ProsQA test questions. For each seed, all four interventions use the same checkpoint.
| Thought access during generation | Thought access during decoding | Seed 0 | Seed 1 | Seed 2 | Mean EM |
|---|---|---|---|---|---|
| All | All | 77.4% | 76.4% | 76.8% | 76.87% |
| Latest only | All | 77.4% | 76.4% | 76.8% | 76.87% |
| All | Latest only | 76.6% | 76.6% | 76.8% | 76.67% |
| Latest only | Latest only | 76.6% | 76.4% | 76.8% | 76.60% |
Restricting both phases changed 12 predictions and reduced exact match by 0.27 percentage points.
Scope of the Single-State Theory, Access to Past Thoughts, and End-to-End COCONUT Training
Thank you for finding the result interesting and novel, and for identifying a gap it addresses. We have few theoretical accounts of what a continuous thought can retain, particularly in Coconut-style decoding. We appreciate your distinction between the state a model can implement and the state it actually learns from answer supervision. We begin with the end-to-end experiment and the revisions to the introduction and problem setting, the two changes you identified as most important.
Requested experiment (Question 6) — What does end-to-end Coconut learn from answers alone?
With the paper's ProsQA trace annotations, the answer-trained thoughts showed little preference for frontier over past nodes at every step where both groups could be compared. At the third thought, the probe recovered neither prescribed state's ranking pattern.
In the revised paper. §1 asks how a thought organizes history when training uses only the final answer. §6.4 reports answer exact match and the node rankings measured at the third of six thoughts. Appendix E.1 gives the training and probe protocols. Figure 6 and Table 12 compare all three pairwise rankings with the two prescribed-state controls.
Starting from a pretrained 12-layer GPT-2, we trained Coconut end to end with six continuous thoughts on all 17,886 ProsQA training questions. Only final-answer tokens contributed to the loss. We used AdamW at 10−4, effective batch size 128 and 50 epochs, selecting the checkpoint by validation answer accuracy. Across three seeds, mean answer exact match was 76.87% on all 500 test questions. We used graph labels only after training. We fitted a ridge-regression membership probe for each node on 257 annotated validation questions and evaluated them on 419 annotated test questions. We excluded the root and gave ties half credit. The table gives the probability that the third thought ranks a node in the first group above one in the second (chance 0.5). Past nodes were reached before the current frontier.
The paper's trace annotations cover steps 1–4. At steps 2, 3 and 4, the probabilities of ranking frontier above past were 0.519, 0.514 and 0.511, and those of ranking past above unseen were 0.452, 0.459 and 0.472 (419, 419 and 217 questions). Step 1 has no past node after excluding the root. We also evaluated all six thoughts using breadth-first levels of the full input graph. Under those labels, past nodes ranked above unseen nodes with probabilities 0.550, 0.580, 0.582 and 0.597 at steps 3–6 (500, 345, 103 and 17 questions). Thus some reached-node information is decodable under the full-graph labels. The deeper steps use progressively fewer questions, and these rankings alone do not establish equal weighting.
The controls explain how to read the near-chance scores. A frontier state singles out the current frontier (1.000 in the first column). A cumulative state keeps earlier nodes accessible even after the frontier moves on (1.000 in the second column). The answer-trained thought showed neither signature under the same probe. The width comparison in the paper therefore answers which prescribed state supports a given readout more efficiently.