acceptodds
Under review as a conference paper at ICLR 2027

How Should Reasoning be Organized in a Transformer's Latent Space?

Abstract

Continuous reasoning has emerged as a promising way to improve reasoning in large language models (LLMs). Yet we still lack a clear principle for deciding what a latent state should preserve. Reasoning by superposition shows that a single latent state can encode several search alternatives and expand them in parallel. We ask how those states should be weighted as reasoning proceeds. A natural choice is to preserve only the states active at the frontier step, since keeping every reached state appears to spread a limited hidden width too thin. We show that the opposite can hold. When later computation draws on several reached states, a cumulative state can guide attention correctly at a smaller hidden width than a frontier state that stores fewer states. At the same width, the cumulative state therefore keeps more intermediate states available for later reasoning. More generally, equal cumulative weights are optimal when future queries are unknown and remain close to the best task-specific weights when those queries are known. Experiments with two-layer and GPT-2 Transformers reproduce the predicted width advantage and show that unequal weights fail first on the states that receive the least weight. This suggests a important principle: keep reached states equally weighted, and restore equal weights as computation proceeds.

Primary area:
learning theory
Keywords:
continuous latent reasoning, cumulative superposition, representation capacity, Transformer theory, latent memory, minimax optimality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Cq9kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This work investigates which information should be preserved in a latent state. The authors compare three different choices:

  • a frontier state which keeps only active states at a current step
  • a cumulative state which keeps all states reached so far with equal weight
  • a weighted state which does the same as the previous, but weighing the values.

The theory treats these choices as abstract coefficient vectors over graph nodes (eq. 6) and measures the hidden width a single normalized latent vector needs for a later attention readout to succeed; a recurrent two-block transformer with a markovian assumption on the latent thinking (each thought depends only on the previous thought and the prompt tokens, Sec. 5.3) is then shown to implement these states.
They show that:

  • If the question draws on multiple reached nodes (no node receives more than C/√m of the query weight, eq. 13), uniform cumulative is best and needs less width than frontier
  • If the query family is known in advance, the ideal state is a weighted state with weights matched to the query; uniform cumulative is within a factor C² of this ideal
  • If the query is not known in advance, uniform cumulative state is best (minimax over nonnegative weightings)
  • Although not stated by the authors, the same algebra implies that if the final question only needs the current frontier, the frontier state is best

The authors validate their claims empirically on ProsQA with a lightweight router and a 2-layer GPT-2 block, where the models are trained to reproduce each prescribed state and the required width is measured, and show that these models exhibit the predicted ordering.

All in all, I believe the work is interesting but rather narrow. I would raise my score if the authors

  1. provide experiments on a coconut-style model end to end to study which state strategy is learned without any supervision
  2. rewrite the intro/problem setting sections in a way that better scopes their claims? Concretely I would want here
    • A better explanation of the considered model architecture
    • A justification of the assumptions made in the theory
    • An explanation of how these claims could carry over to more realistic settings
Critical strengths

The result is interesting and novel. I believe that the space of theoretical results in latent reasoning methods (especially coconut-style decoding) is rather underdeveloped. I believe that this work tries to fill a gap which is quite real in the literature.

Critical weaknesses
  • Main weakness 1: the authors assume (it seems to me?) a markovian Transformer model q~c+1=f(q~c,edge tokens)q̃_{c+1} = f(q̃_c, \text{edge tokens}). This limits the theoretical model vastly in my opinion: access to all previous latent thoughts written to the KV cache implies a growing computation buffer. The results presented here would not hold in this case as a model could easily diffuse frontier/state information across the KV cache, which could easily be retrieved using a J-L embedding scheme or something of the sort.
  • Main weakness 2: I found the overall paper hard to understand. It seems to me many details which would be crucial to me to better understand the breadth of the results are hard to find (see questions for more details).
  • Main weakness 3: I find that some of the main theoretical results are a bit tautological. The tasks for which certain frontier types are best (e.g. frontier is best for final query on frontier) seem fairly obvious
Questions for authors
  • It seems the model of the transformer assumed in the theoretical section is markovian and only assumes access to the previous latent though + the input tokens from the prompt. Is my assessment on this correct?
  • Related to previous point. Do the authors have any empirical or intuitive justification for this specific definition of latent reasoning?
  • Can the authors give more details on the training setup for the GPT-2 experiments? How many latent thoughts are given to the model (it seems only 1)? What is the training process (similar or not to the one of the "reasoning by superposition)?
  • Can the authors comment on the third weakness? Is there something I am not seeing? Is there a way in which the results are not self evident?
  • There is concurrent work about latent thinking (https://arxiv.org/abs/2604.06374, https://arxiv.org/abs/2604.04902) which I believe to be quite relevant which is not cited in your work. Namely in the first paper by Rizvi-Martel et al., there is an analysis of how state size is related to ability to maintain superposition of the frontier. I believe this analysis of state size is very related to your work.
  • Would it be possible to provide experiments in which a model (coconut style) is trained end-to-end on ProsQA with only the answer loss and inspect what state type was learned? I would be interested to see what strategy models converge to without additional supervision
8 comments
AuthorsOpenReview ↗

Scope of the Single-State Theory, Access to Past Thoughts, and End-to-End COCONUT Training

Thank you for finding the result interesting and novel, and for identifying a gap it addresses. We have few theoretical accounts of what a continuous thought can retain, particularly in Coconut-style decoding. We appreciate your distinction between the state a model can implement and the state it actually learns from answer supervision. We begin with the end-to-end experiment and the revisions to the introduction and problem setting, the two changes you identified as most important.

Requested experiment (Question 6) — What does end-to-end Coconut learn from answers alone?

With the paper's ProsQA trace annotations, the answer-trained thoughts showed little preference for frontier over past nodes at every step where both groups could be compared. At the third thought, the probe recovered neither prescribed state's ranking pattern.

In the revised paper. §1 asks how a thought organizes history when training uses only the final answer. §6.4 reports answer exact match and the node rankings measured at the third of six thoughts. Appendix E.1 gives the training and probe protocols. Figure 6 and Table 12 compare all three pairwise rankings with the two prescribed-state controls.

Starting from a pretrained 12-layer GPT-2, we trained Coconut end to end with six continuous thoughts on all 17,886 ProsQA training questions. Only final-answer tokens contributed to the loss. We used AdamW at 10−410^{-4}, effective batch size 128 and 50 epochs, selecting the checkpoint by validation answer accuracy. Across three seeds, mean answer exact match was 76.87% on all 500 test questions. We used graph labels only after training. We fitted a ridge-regression membership probe for each node on 257 annotated validation questions and evaluated them on 419 annotated test questions. We excluded the root and gave ties half credit. The table gives the probability that the third thought ranks a node in the first group above one in the second (chance 0.5). Past nodes were reached before the current frontier.

StateFrontier above pastPast above unseenFrontier above unseen
Explicit frontier1.0000.4760.999
Explicit cumulative0.4921.0001.000
Answer-only Coconut0.5140.4590.468

The paper's trace annotations cover steps 1–4. At steps 2, 3 and 4, the probabilities of ranking frontier above past were 0.519, 0.514 and 0.511, and those of ranking past above unseen were 0.452, 0.459 and 0.472 (419, 419 and 217 questions). Step 1 has no past node after excluding the root. We also evaluated all six thoughts using breadth-first levels of the full input graph. Under those labels, past nodes ranked above unseen nodes with probabilities 0.550, 0.580, 0.582 and 0.597 at steps 3–6 (500, 345, 103 and 17 questions). Thus some reached-node information is decodable under the full-graph labels. The deeper steps use progressively fewer questions, and these rankings alone do not establish equal weighting.

The controls explain how to read the near-chance scores. A frontier state singles out the current frontier (1.000 in the first column). A cumulative state keeps earlier nodes accessible even after the frontier moves on (1.000 in the second column). The answer-trained thought showed neither signature under the same probe. The width comparison in the paper therefore answers which prescribed state supports a given readout more efficiently.

Requested revision (Weakness 2) — What architecture, assumptions and applications does the paper cover?

We revised the abstract, introduction and problem setting to state the single-state interface. One normalized thought carries graph-search history between steps, and the final head reads that thought against a common set of candidate keys.

In the revised paper. The abstract identifies the single thought carried between steps. In §1, “Cumulative-superposition hypothesis,” we ask for the minimum hidden width at which all DD routing steps and the final selection succeed together with probability at least 1−δ1-\delta for each fixed query in the family. D\mathfrak D gives the scale of that minimum. §1, “Contributions,” distinguishes the terminal readout advantage, its realization across DD Transformer steps, and the guarantee for equal weights when future queries are unknown. §3 specifies what each block and the final head can access and interprets the query constraint in Eq. (13).

  • Architecture (§§1 and 3). Section 3 follows one update from input to output. The first block uses the thought and source embeddings to identify eligible edges. The second writes their destinations into the next thought. This locates the choice between frontier and cumulative in the update itself. Section 3 then specifies what the final head can access. If each past frontier occupies a separate cache slot, a head can sum those slots at readout. The single-state comparison applies when historical information reaches that head through one vector.
  • Why the assumptions enter (§3 and Appendix C). Independent Gaussian node embeddings and orthogonal competing directions in coefficient space let us derive the distribution of the largest competing score. Nonnegative weights model a query that pools supporting results, so a node retained by the state can contribute to the correct answer. Section 3 now interprets the concentration bound in terms of that query: if it gives equal weight to ss of mm reached nodes, its smallest admissible CC is m/s\sqrt{m/s}. Thus the reader can tell how broadly the query draws on history. Appendix C gives the corresponding readout probability for a fixed correlated candidate bank under Gaussian embeddings.
  • What transfers to learned models (§§6.2 and 6.4 and Appendix E). We test the predicted ordering of widths while holding the graph, query and competing keys fixed between policies. For Qwen we embed the carried thought in the native backbone with a fixed orthonormal basis and project the thought output back to the tested dimension before the recurrent update. At width 32 on both EntailmentBank and FinQA, the two-layer Transformer and GPT-2 each exceeded 90% with cumulative and stayed below 90% with frontier (one seed per task and router, §6.2, Table 1). With the 14B pretrained router on FinQA program-dependency graphs, cumulative and frontier reached 90% candidate-selection accuracy at latent widths 23.3 and 57.5, respectively (Table 1). We also tested the embedding assumption directly on 340 EntailmentBank proofs. Distinct reached statements encoded with their question had mean cosine similarity 0.937. With competing keys matched in size and weights, cumulative reached 90% at a mean projected width of 593.1, while frontier remained below 90% through width 1,536. Figure 5(b) shows how the retrieval curves shift with learned representations while preserving this ordering. Appendix E.2 separately reports program execution on the original FinQA questions.

Weakness 1 and Question 1 — Which part of the result requires a Markov interface?

Yes. Section 5.3 carries one thought across steps. Your proposed Johnson–Lindenstrauss (JL) retrieval scheme is viable when the final head can address earlier slots. Our contribution is a sharp width law and an optimal weighting rule for the case where history reaches that head through one thought. If a fixed query picks out a cached key by a margin γ>0\gamma>0, a projection that perturbs each score by less than γ/2\gamma/2 preserves the winner. A reader can also add the cached sums of disjoint frontiers to recover the cumulative sum exactly, provided it retains the scales if those sums are normalized. Thus a cache can move accumulation from writing to reading.

What the width law adds. Applying JL to a fixed bank gives a sufficient dimension once a distortion tolerance is chosen. Even substituting the decision margin for that tolerance gives only a sufficient bound by this argument. Our readout ranks the same keys for any upstream process producing the same thought. Under our independent Gaussian embedding and orthogonal candidate-bank assumptions, Theorem 4.1 gives the minimum terminal width for the intended state as dans=(2+o(1))log⁡M/Γ2d_{\mathrm{ans}}=(2+o(1))\log M/\Gamma^2 for fixed target success probability as MM grows. Here Γ>0\Gamma>0 is the least alignment with an admissible correct query. The leading constant comes from the maximum of the MM competing scores. Comparing that maximum with the correct score establishes necessity as well as sufficiency. For a complete trace, the minimum width for the intended states also has matching upper and lower bounds up to constants. It scales as the larger of this terminal term and the routing term set by the weakest reusable node (§4, Eq. (17)).

Why the weighting matters. Suppose either of two reached nodes may be needed for routing, but the final query weights them in the ratio 1:21:2. Matching that query raises final alignment from 3/103/\sqrt{10} to 11, while the weaker node's coefficient falls from 1/21/\sqrt{2} to 1/51/\sqrt{5}. Even an exact isometry preserves both consequences of the write. Theorem 5.1 uses the joint width law to choose weights before the future query is known. When every reached node must remain reusable, uniform cumulative weights minimize worst-case joint width scale over the full admissible family. For a known family they cost at most a sharp factor C2C^2 over the best fixed nonnegative weights. Section 5.3 then gives a sufficient width for a recurrent Transformer to realize the updates. A growing cache changes the number of readable slots, and the optimal width–slot trade-off remains open.

In the revised paper. §3 specifies which history the final head can address and explains the cached-frontier example. §4, Eq. (17) and §5.1, Theorem 5.1 give the joint width law and weighting guarantee. §5.3 shows how the recurrent two-block Transformer actually writes these states.

Question 2 — Why study this interface?

On ProsQA, restricting Coconut to the latest thought during thought generation preserved all 1,500 paired answers when the answer decoder retained full history.

In the revised paper. §2, “Recurrent computation and latent states,” discusses depth-recurrent models that update a fixed-shape state. Appendix E.1, Table 11 tests historical access during generation and decoding separately. §6.4 reports the change in paired predictions when both phases use only the latest thought.

This gives an empirical reason to study a single carried thought during generation even for a model trained with a full cache.

Depth-recurrent language models provide an architectural example. Geiping et al. update a fixed-shape state by si=R(e,si−1)s_i=R(e,s_{i-1}), where ee is the input representation. Their state is a matrix spanning input positions. Our abstraction studies how one normalized thought in such an interface allocates capacity across reached objects. Geiping et al., §3.1

We also tested how much a trained full-cache model relies on access to earlier thought positions. After training standard Coconut with full history, we separately restricted historical thought access during thought generation and during answer decoding. Prompt access was preserved. We measured greedy exact match on all 500 ProsQA test questions. For each seed, all four interventions use the same checkpoint.

Thought access during generationThought access during decodingSeed 0Seed 1Seed 2Mean EM
AllAll77.4%76.4%76.8%76.87%
Latest onlyAll77.4%76.4%76.8%76.87%
AllLatest only76.6%76.6%76.8%76.67%
Latest onlyLatest only76.6%76.4%76.8%76.60%

Restricting both phases changed 12 predictions and reduced exact match by 0.27 percentage points.

Reviewer_rCW4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how intermediate reasoning states should be organized in the latent space of Transformers. The authors propose a cumulative-state representation that preserves all previously reached states with equal weights, rather than retaining only the current reasoning frontier. Through theoretical analysis, the paper shows that cumulative representations can reduce the hidden width required for reliable reasoning under certain assumptions. Experiments with lightweight Transformers and GPT-2-based models further support the theoretical findings and demonstrate the advantages of uniform cumulative weighting.

Critical strengths

1.The paper addresses an interesting and important research question concerning how intermediate information should be preserved during latent reasoning.

2.The proposed cumulative-state representation is intuitive and interesting, providing a new perspective on organizing intermediate reasoning states.

3.The theoretical analysis is relatively comprehensive and provides useful insights into the relationship between state weighting, hidden dimensions, and reasoning reliability.

4.The experiments include different state-weighting strategies and Transformer architectures, providing empirical support for the theoretical results.

Critical weaknesses

1.The theoretical analysis relies on relatively strong assumptions, including independent Gaussian embeddings, nonnegative state weights, and specific query distributions. It remains unclear whether these assumptions hold in practical latent reasoning models.

2.The experiments mainly focus on synthetic graph-based reasoning tasks and relatively small Transformer architectures. Evaluation on larger pretrained language models and more complex reasoning benchmarks would strengthen the conclusions.

3.The proposed equal-weight cumulative strategy may not be optimal when different historical states have varying importance. More analysis of task-dependent or adaptive weighting strategies would be valuable.

4.The paper primarily evaluates controlled state representations rather than demonstrating how the proposed organization can be learned naturally in existing latent reasoning models.

5.Although the theoretical results are promising, the practical benefits for real-world reasoning tasks remain unclear. Additional experiments on reasoning accuracy and computational efficiency would improve the significance of the work.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Assumptions Behind Equal Weighting, Learned State Strategies, and Practical Reasoning Performance

Thank you for highlighting the importance of deciding which intermediate results a latent reasoning model should keep. We appreciate that you found cumulative retention intuitive and that the analysis clarified how state weights and hidden width together affect reliability. You also noted that comparing several weighting strategies and Transformer architectures gave empirical support to the theory. Your questions led us to examine how the comparison changes when weights are learned and when success is measured on the final answer.

Weakness 1 — Gaussian embeddings, nonnegative weights, and the query distribution

Cumulative retained its width advantage on EntailmentBank with strongly non-orthogonal learned representations.

In the revised paper. §3 explains what a nonnegative query represents and how its concentration bounds the weight on one reached node. Appendix D.1 specifies how the evaluation queries were constructed. §6.4, Figure 5(b) tests the embedding assumption on learned proof representations, and Appendix C, Theorem C.1 treats fixed correlated candidate directions under Gaussian embeddings.

How closely the representations align depends on the task and on how its statements are encoded. The main analysis uses independent Gaussian node embeddings and orthogonal competing candidate directions in coefficient space to make the width dependence explicit. Appendix C also analyzes fixed, possibly correlated candidate directions under Gaussian embeddings and positive margins. Nonnegative state weights describe pooling supporting evidence, and the query concentration bound specifies how much weight any one reached node can receive.

For all 340 EntailmentBank test proofs, we encoded each statement with its question using frozen Qwen2.5-1.5B-Instruct. Distinct reached nodes in the same proof had mean cosine similarity 0.937. At width 640, cumulative selected the correct candidate on 91.16% of trials and frontier on 79.35% (paired difference 11.81 percentage points, 95% interval 8.84–14.63). The two states used the same nonnegative queries, and competing candidates used the same number of nodes and weights as the correct candidate. With single-node alternatives both states reached 100% by width 32. W4 examines separately what a model learns from answer supervision alone. For each of five orthonormal projections, we take successive coordinates in the same basis, rank candidates by dot product and interpolate the first 90% crossing when it occurs (Appendix E).

Weakness 2 — Larger pretrained models and more complex tasks

On EntailmentBank and FinQA, cumulative outperformed frontier at widths 32, 64 and 96 with both 7B and 14B pretrained Qwen routers.

In the revised paper. §6.2, Table 1 reports accuracy at matched state widths across tasks for the two small routers and the three pretrained model sizes. The paragraph on FinQA in §6.2 compares accuracy by reached-set size. Appendix D.1, Table 2 gives the splits and explains how proofs and programs supply the reasoning traces.

These tasks supply science proof graphs and financial program dependencies. At each width the two policies share the router, examples, queries and distractors, isolating the effect of what the recurrent state retains.

For each Qwen state width dd, we start from the same pretrained checkpoint at its native hidden size HH. For d<Hd<H, the first dd columns of a fixed orthonormal basis map the edge and thought vectors into the backbone. We project the output at the thought position back to dd coordinates before the message projection and normalization. Frontier and cumulative share the trained router, graph, query and competing keys at each width. Qwen training uses a learning rate of 10−410^{-4}, gradient accumulation over 16 examples and at most 40 epochs (Appendix E).

On EntailmentBank and FinQA we also train the two-layer and GPT-2 routers at widths 16, 24, 32, 64, 96 and 128, using the full splits and one seed per configuration. The two-layer router trains for four epochs and GPT-2 for at most 40. Each test graph is evaluated with four fixed queries and two embedding draws against 4,096 distractors.

Accuracy at d=32d=32 is the percentage of held-out trials in which the 32-dimensional thought ranks the correct candidate first. We compute d0.9d_{0.9} by linear interpolation at the first upward crossing of 90% across all tested widths. An entry of >192>192 means the target was unreached through width 192. The ProsQA Qwen-7B and 14B entries are projections from FinQA.

DatasetRouterStateAcc. (%) at d=32d=32Acc. (%) at d=64d=64Acc. (%) at d=96d=96d0.9d_{0.9}
ProsQA1.5BFrontier16.652.380.2122.0
ProsQA1.5BCumulative99.4100.0100.023.2
ProsQA7BFrontier26.449.446.0270.4
ProsQA7BCumulative97.690.485.330.5
ProsQA14BFrontier17.763.579.3124.4
ProsQA14BCumulative98.9100.0100.023.4
EntailmentBankTwo-layer TransformerFrontier44.374.887.8108.9
EntailmentBankTwo-layer TransformerCumulative97.6100.0100.025.4
EntailmentBankGPT-2Frontier65.989.095.469.1
EntailmentBankGPT-2Cumulative91.999.8100.031.4
EntailmentBank1.5BFrontier44.974.988.0108.7
EntailmentBank1.5BCumulative98.2100.0100.024.0
EntailmentBank7BFrontier1.11.10.1>192
EntailmentBank7BCumulative12.625.639.9>192
EntailmentBank14BFrontier0.214.60.0>192
EntailmentBank14BCumulative7.573.130.6>192
FinQATwo-layer TransformerFrontier68.295.599.457.5
FinQATwo-layer TransformerCumulative99.5100.0100.022.8
FinQAGPT-2Frontier87.599.7100.038.5
FinQAGPT-2Cumulative97.2100.0100.027.2
FinQA1.5BFrontier70.696.099.556.4
FinQA1.5BCumulative99.5100.0100.023.1
FinQA7BFrontier64.217.21.3125.0
FinQA7BCumulative98.390.485.330.4
FinQA14BFrontier69.195.499.157.5
FinQA14BCumulative99.0100.0100.023.3

With GPT-2, cumulative reached 90% at widths 31.4 on EntailmentBank and 27.2 on FinQA, compared with 69.1 and 38.5 for frontier. The two-layer Transformer showed the same ordering on both tasks. These four comparisons use one seed each (§6.2, Table 1).

On FinQA, cumulative reached 98.33% (7B) and 99.04% (14B) at width 32, compared with 64.23% and 69.10% for frontier. On 14B the interpolated 90% widths were 23.3 for cumulative and 57.5 for frontier.

FinQA contains 490 test graphs with at least four reached nodes and 657 smaller graphs. At width 32 on 1.5B, frontier/cumulative accuracy was 0.467/0.990 on the larger graphs and 0.884/0.998 on the smaller ones. W5 reports program execution on the original questions.

Weakness 3 — Unequal importance and adaptive weights

Fitting weights to ProsQA queries slightly reduced the width needed for 90% readout success. Uniform cumulative weights required less width on each weighting’s hardest admissible query. Theorem 5.1 quantifies the value of knowing the query family. When every reached node must remain available for routing, the joint width scale for uniform cumulative weights is at most C2C^2 times that of the best fixed nonnegative weights for a known family with concentration bound CC. Over the full family allowed by that bound, uniform cumulative weights minimize worst-case joint width.

In the revised paper. §6.4, Figure 5(a) shows how much width the learned write gates save on ProsQA. The same section reports the reversal at width 32 when terminal coefficients are fitted to familiar queries and then tested on the least-aligned allowed query. Appendix E, “Learning to retain results,” gives the fitting procedure. §5.1, Theorem 5.1 states the guarantee for an unknown future query.

The comparison uses the same test graphs and changes which query must be served. It isolates the value of knowing which historical states a future query is likely to use.

In the recurrent experiment, learning a multiplier for each candidate write reduced the width for 90% success from 40.3 to 35.1 on lightweight ProsQA, improving on uniform cumulative in all five paired seeds.

Figure 4(a) measures both widths as ratios to uniform cumulative, one for routing and one for the hardest allowed final query. Panels (b,c) show what happens to individual nodes. With the same routing head at d=192d=192, recency weighting gives the oldest group 0.81 times its uniform weight, and its mean score margin over the strongest inactive key falls from 5.7 to 4.5. A routing step succeeds only if every reached node is accepted and every inactive key rejected (§6.3, Appendix D.5).

Entries are means ± 95% Student-tt intervals across seeds.

StateAcc. @ 32Acc. @ 64Acc. @ 96Empirical d0.9d_{0.9}
Frontier0.210 ± 0.0070.622 ± 0.0080.847 ± 0.005110.6 ± 1.6
Uniform cumulative0.793 ± 0.0150.984 ± 0.0030.997 ± 0.00140.3 ± 1.1
Learned gate per step0.845 ± 0.0110.990 ± 0.0030.998 ± 0.00137.1 ± 1.0
Learned gate per candidate0.869 ± 0.0120.993 ± 0.0030.999 ± 0.00135.1 ± 1.1

We also fitted the terminal coefficients of Eq. (6) directly on training queries from five random positive initializations. At test time the coefficients were fixed. The table reports the probability of a correct readout under the paper’s Gaussian embedding model and the interpolated 90% width, averaged across the five initializations. For the least-aligned allowed query, each policy faces its own worst admissible query on each graph.

Query familyTerminal coefficientsSuccess @ 32Success @ 64Success @ 96Ideal d0.9d_{0.9}
Original test bankUniform0.9471.0001.00029.6
Original test bankRandom positive0.8180.9730.98842.8
Original test bankFitted from random0.9541.0001.00029.0
Least-aligned allowedUniform0.7920.9971.00038.9
Least-aligned allowedRandom positive0.2990.7070.864d0.9≤128d_{0.9}\leq128 for 4 of 5 initializations
Least-aligned allowedFitted from random0.6590.9861.00046.5

For terminal readout, fitting reduced the 90% width from 29.6 to 29.0 on the original queries. Each weighting then faced its own least-aligned allowed query, for which uniform needed 38.9 dimensions and the fitted weights needed 46.5.

Reviewer_HPakOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares frontier (keep only current-step states) vs. cumulative (equal-weight all reached states) strategies for organizing intermediate reasoning results in a Transformer's latent space. The main theoretical result is counterintuitive: cumulative requires less hidden width than frontier, because useful components reinforce at attention readout while irrelevant candidates contribute only noise. The authors prove a C²-factor optimality guarantee for equal weights and validate on ProsQA graph reasoning with a two-block toy model and GPT-2 (2-layer, 124M). The predicted width ratios match empirical measurements.

Critical strengths
  • The core finding genuinely challenges standard superposition intuition. The argument is clean, and the connection between the abstract theory and actual Transformer operations (attention, residual, RMSNorm) is carefully worked out.

  • Theory-experiment alignment is tight. Predicted width ratios match across both architectures, and the position-level failure pattern (least-weighted states fail first) tests the mechanism directly.

  • The paper is well-written for a theory piece. Figure 1 makes the hypothesis intuitive before the formalism arrives.

Critical weaknesses
  • The experiments are my main concerns. The entire validation is on a toy two-block model and GPT-2 with 2 layers. The paper frames itself around "reasoning in large language models" (the abstract opens with LLMs), but experiments are not done on LLMs. ProsQA is a synthetic graph task. Without at least one experiment at 1B+ scale on a less contrived task, the framing is unsupported.

  • The theory assumes independent Gaussian embeddings and orthogonal candidate directions. Real models have correlated, learned embeddings. The paper should address how robust the predictions are when these assumptions break.

  • Only frontier vs. uniform cumulative is compared. Learned selective retention, where the model decides per-step which states to keep is the natural competitor, and it is not discussed.

  • Practical actionability is unclear. "Preserve reached states equally weighted" is a principle, but the paper doesn't show how to implement it as a training recipe that ports to continuous CoT at scale.

Questions for authors
  • Have you attempted any experiment at 1B+ scale, or on a task beyond ProsQA? Even a preliminary result would help.

  • Do you have evidence the predictions hold with learned, correlated embeddings?

  • How do you expect cumulative to compare against a learned selective-retention policy?

  • What are the realized values of C in your experiments, and how sensitive is the C² factor to task complexity?

5 comments
AuthorsOpenReview ↗

Cumulative States Across Models and Tasks, Robustness to Correlated Embeddings, and Learned Retention

Thank you for pointing out why the width result is surprising. A state that retains more of the search history can need fewer dimensions. We appreciate your assessment that Figure 1 makes the hypothesis intuitive and that the analysis connects it to attention, the residual update and RMSNorm. You also identified two distinct empirical tests of the theory. The width ratios agree with its predictions on both architectures, and lightly weighted states are the first to fail. Your questions prompted us to test those predictions with larger routers, learned embeddings and learned retention policies.

Weakness 1 and Question 1 — Larger models and tasks beyond ProsQA

Yes. With Qwen2.5-14B on FinQA, cumulative reached 90% retrieval accuracy at an interpolated width of 23.3, compared with 57.5 for frontier.

In the revised paper. §6.2, Table 1 now compares candidate-selection accuracy and 90% widths across router sizes on ProsQA, EntailmentBank and FinQA. Appendix D.1, Table 2 identifies the data splits. Appendix E, “Comparing frontier and cumulative at matched widths,” explains how we vary the carried width inside each pretrained router.

The 7B and 14B comparisons also favor cumulative on EntailmentBank science proofs and FinQA program-dependency graphs at each displayed width.

For each Qwen state width dd, we start from the same pretrained checkpoint at its native hidden size HH. For d<Hd<H, the first dd columns of a fixed orthonormal basis map the edge and thought vectors into the backbone. We project the output at the thought position back to dd coordinates before the message projection and normalization. Frontier and cumulative share the trained router, graph, query and competing keys at each width. Qwen training uses a learning rate of 10−410^{-4}, gradient accumulation over 16 examples and at most 40 epochs (Appendix E).

On EntailmentBank and FinQA we also train the two-layer and GPT-2 routers at widths 16, 24, 32, 64, 96 and 128, using the full splits and one seed per configuration. The two-layer router trains for four epochs and GPT-2 for at most 40. Each test graph is evaluated with four fixed queries and two embedding draws against 4,096 distractors.

Accuracy at d=32d=32 is the percentage of held-out trials in which the 32-dimensional thought ranks the correct candidate first. We compute d0.9d_{0.9} by linear interpolation at the first upward crossing of 90% across all tested widths. An entry of >192>192 means the target was unreached through width 192. The ProsQA Qwen-7B and 14B entries are projections from FinQA. For example, the 14B FinQA cumulative score rises from 0.562 at 16 to 0.932 at 24, giving d0.9=23.3d_{0.9}=23.3.

DatasetRouterStateAcc. (%) at d=32d=32Acc. (%) at d=64d=64Acc. (%) at d=96d=96d0.9d_{0.9}
ProsQA1.5BFrontier16.652.380.2122.0
ProsQA1.5BCumulative99.4100.0100.023.2
ProsQA7BFrontier26.449.446.0270.4
ProsQA7BCumulative97.690.485.330.5
ProsQA14BFrontier17.763.579.3124.4
ProsQA14BCumulative98.9100.0100.023.4
EntailmentBankTwo-layer TransformerFrontier44.374.887.8108.9
EntailmentBankTwo-layer TransformerCumulative97.6100.0100.025.4
EntailmentBankGPT-2Frontier65.989.095.469.1
EntailmentBankGPT-2Cumulative91.999.8100.031.4
EntailmentBank1.5BFrontier44.974.988.0108.7
EntailmentBank1.5BCumulative98.2100.0100.024.0
EntailmentBank7BFrontier1.11.10.1>192
EntailmentBank7BCumulative12.625.639.9>192
EntailmentBank14BFrontier0.214.60.0>192
EntailmentBank14BCumulative7.573.130.6>192
FinQATwo-layer TransformerFrontier68.295.599.457.5
FinQATwo-layer TransformerCumulative99.5100.0100.022.8
FinQAGPT-2Frontier87.599.7100.038.5
FinQAGPT-2Cumulative97.2100.0100.027.2
FinQA1.5BFrontier70.696.099.556.4
FinQA1.5BCumulative99.5100.0100.023.1
FinQA7BFrontier64.217.21.3125.0
FinQA7BCumulative98.390.485.330.4
FinQA14BFrontier69.195.499.157.5
FinQA14BCumulative99.0100.0100.023.3

At width 32 on both EntailmentBank and FinQA, cumulative exceeded 90% with the two-layer Transformer and GPT-2. Frontier stayed below 90% in all four comparisons. These four comparisons use one seed each (§6.2, Table 1).

W4 tests whether the same design principle also helps answer the original FinQA questions.

Weakness 2 and Question 2 — Learned, correlated embeddings and candidate directions

Yes. On EntailmentBank, cumulative required less width than frontier even when the learned node representations were strongly non-orthogonal.

In the revised paper. §6.4, Figure 5(b) shows how replacing Gaussian embeddings with learned statement embeddings shifts the two width curves on the same proofs. Appendix E, “Correlated node representations,” gives the matched-candidate construction, the single-node control and the candidate norms. Appendix C, Theorem C.1 gives the Gaussian readout probability for fixed correlated candidate directions.

How closely two nodes align depends on the task, the statements they represent, and the model used to encode them. In the main analysis we use independent Gaussian node embeddings and assume orthogonality of the competing candidate directions in coefficient space. This gives a simple expression for how much width is needed for the correct candidate to win.

We tested the width prediction on all 340 EntailmentBank test proofs. We used a frozen Qwen2.5-1.5B-Instruct model to encode each statement together with its question. Distinct reached nodes in the same proof had mean cosine similarity 0.937. We formed frontier and cumulative from the same node representations and compared the correct candidate with 339 alternatives constructed using the same number of nodes and the same weights. The table reports accuracy at each projected width and the mean 90% width across five projections. For each of five orthonormal projections, we take successive coordinates in the same basis, rank candidates by dot product and interpolate the first 90% crossing when it occurs (Appendix E).

Node representationsStateAcc. @ 32Acc. @ 64Acc. @ 96Empirical d0.9d_{0.9}
Independent Gaussian embeddingsFrontier0.6490.8840.95671.1
Independent Gaussian embeddingsCumulative1.0001.0001.00015.9
Learned statement representationsFrontier0.0820.1650.247>1,536
Learned statement representationsCumulative0.0910.1780.295593.1

With independent Gaussian embeddings, cumulative reached 90% at width 15.9 and frontier at 71.1. With learned representations, cumulative reached 90% at a mean width of 593.1, while frontier remained below 90% through width 1,536. Thus the absolute width changed, but the predicted advantage of retaining all reached nodes persisted on this task. At width 640 the paired difference was 11.81 percentage points (91.16% versus 79.35%, question-bootstrap 95% interval 8.84–14.63).

Matching how the candidates were constructed mattered. With single-node alternatives both states reached 100% by width 32. At width 384 the correct candidate had mean norm 2.60, single-node alternatives 1.00, and alternatives constructed with the same weights and number of nodes 2.58.

Weakness 3 and Question 3 — Learned selective retention

Fitting weights to ProsQA queries slightly reduced the width needed for 90% readout success. Uniform cumulative weights required less width on each weighting’s hardest admissible query. Theorem 5.1 quantifies the value of knowing the query family. When every reached node must remain available for routing, the joint width scale for uniform cumulative weights is at most C2C^2 times that of the best fixed nonnegative weights for a known family with concentration bound CC. Over the full family allowed by that bound, uniform cumulative weights minimize worst-case joint width.

In the revised paper. §6.4, Figure 5(a) compares the two learned gates against frontier and uniform cumulative across widths. Appendix E, “Learning to retain results,” describes gate training, reports the EntailmentBank comparison, and gives the separate test of randomly initialized terminal coefficients on the original and least-aligned queries at width 32. The table below reports readout probabilities at each width and the interpolated thresholds.

Training a gate for each candidate write reduced the recurrent 90% width from 40.3 to 35.1. Fitting the terminal coefficients directly let us examine the cost of specialization when the future query changes.

Learning what to write. On the original lightweight ProsQA task, we froze each validation-selected router and trained gates for four additional epochs using the final query loss. The gates see the current state and candidate before the future query is revealed. Both initialize at the uniform update. Retrieval success and the interpolated width for 90% success are averaged over five paired seeds.

Entries are means ± 95% Student-tt intervals across seeds.

StateAcc. @ 32Acc. @ 64Acc. @ 96Empirical d0.9d_{0.9}
Frontier0.210 ± 0.0070.622 ± 0.0080.847 ± 0.005110.6 ± 1.6
Uniform cumulative0.793 ± 0.0150.984 ± 0.0030.997 ± 0.00140.3 ± 1.1
Learned gate per step0.845 ± 0.0110.990 ± 0.0030.998 ± 0.00137.1 ± 1.0
Learned gate per candidate0.869 ± 0.0120.993 ± 0.0030.999 ± 0.00135.1 ± 1.1

The largest reduction on lightweight ProsQA comes from retaining the history at all (d0.9d_{0.9} falls from 110.6 to 40.3). Selecting writes individually reduces it further to 35.1. On Qwen2.5-1.5B/EntailmentBank, uniform cumulative and the step gate both first exceeded 90% at the same tested width, 24 (90.15% and 90.04%).

Figure 4(a) measures both widths as ratios to uniform cumulative, one for routing and one for the hardest allowed final query. Panels (b,c) show what happens to individual nodes. With the same routing head at d=192d=192, recency weighting gives the oldest group 0.81 times its uniform weight, and its mean score margin over the strongest inactive key falls from 5.7 to 4.5. A routing step succeeds only if every reached node is accepted and every inactive key rejected (§6.3, Appendix D.5).

Learning the coefficients directly. In the ideal terminal-state model of Eq. (6), we fitted nonnegative coefficients by maximizing alignment with training queries from five random positive initializations. Graphs of the same reached-set size shared coefficients by first-arrival position. We fixed the fitted coefficients at test time. The table reports the probability of a correct readout under the paper’s Gaussian embedding model and the interpolated 90% width, averaged across the five initializations. For the least-aligned allowed query, each policy faces its own worst admissible query on each graph.

Query familyTerminal coefficientsSuccess @ 32Success @ 64Success @ 96Ideal d0.9d_{0.9}
Original test bankUniform0.9471.0001.00029.6
Original test bankRandom positive0.8180.9730.98842.8
Original test bankFitted from random0.9541.0001.00029.0
Least-aligned allowedUniform0.7920.9971.00038.9
Least-aligned allowedRandom positive0.2990.7070.864d0.9≤128d_{0.9}\leq128 for 4 of 5 initializations
Least-aligned allowedFitted from random0.6590.9861.00046.5

The original ProsQA queries always include the terminal frontier. Fitting can exploit that regularity, but reduces success on the least-aligned allowed queries relative to uniform weighting.

This coefficient-fitting experiment isolates the terminal readout component of the joint width comparison in Theorem 5.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.