acceptodds
Under review as a conference paper at ICLR 2027

Superposition and Distributed Memory in Continuous Chain-of-Thought Reasoning

Abstract

Continuous chain-of-thought reasoning feeds hidden states back into a language model while retaining a persistent key–value (KV) history. In this work, we observe a distinction between latent generation and answer readout in Coconut: restricting latent-history access to recent positions during generation preserves most answers, whereas restricting it to the final position at readout changes many answers. Restoring earlier cached values also recovers much of the lost accuracy after history corruption while the final latent input remains fixed. To understand how this historical information can support reasoning, we develop a frontier–history model for graph reachability. We show that a two-block Transformer can implement exact frontier updates and answer readout by superposing the current frontier in feedback and retaining earlier frontiers in the KV cache. Further experiments on a trained two-layer Coconut model reveal depth-aligned feedback, depth-selective historical readout, and attenuation of revisited nodes. However, paired interventions show that feedback also retains update-relevant history beyond the current frontier. These results offer a distributed account of continuous reasoning: superposition need not entail cumulative storage in every feedback vector.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Reasoning, Continuous Cot, Expressiveness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bG6bOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims at understanding that in continuous chain of thought, how the current continuous token and the previous KV cache help reasoning respectively. First, by careful intervention on a already trained model, this paper shows that the previous KV cache history is still important for correct generation in continuous chain of thought reasoning. Next, this paper shows a expressiveness result: on graph reachability task, there exists a Transformer that use both continuous token and previous KV cache to find the correct answer. Empirical result on this task show the some consistent features between the constructed model and real trained model.

Critical strengths
  1. This paper analyzes an interesting problem in continuous chain of thought. By careful controlled experiments, this paper aims at showing the role of continuous token itself and previous KV cache in continuous CoT.
Critical weaknesses

My soundness concern is mainly about the theoretical results comparing to [1] as both are considering the same task.

  1. The theoretical result is not as strong as that in [1]. In Theorem 1 of [1], the number of required continuous tokens is DD, where DD is the diameter of the graph. However, in the Theorem 1 of this paper, the number of required continuous tokens scales as NN where NN is the number of the nodes. It is counter-intuitive that after using KV cache in construction, the required tokens increases.

[1] Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought. https://arxiv.org/abs/2505.12514

Questions for authors

Please refer to the weaknesses part.

  1. Could the authors explain if N−1N-1 continuous tokens is necessary in this construction, or can we improve it to DD continuous tokens, where DD is the diameter of the graph?
3 comments
AuthorsOpenReview ↗

Response to Reviewer bG6b

We sincerely thank you for your thoughtful feedback. Our responses follow.

You point out that, since both papers study the same task, our theoretical result is weaker than that of [1]: Theorem 1 of [1] requires a number of continuous tokens scaling with the graph diameter, whereas our Theorem 1 requires a number scaling with the number of nodes. You find it counter-intuitive that using the KV cache in the construction increases the required number of tokens, and ask whether this number is necessary or can be improved to a diameter-dependent bound.

Response. Thank you for raising this important point. We agree that the token bound in Theorem 1 can be sharpened to depend on the graph diameter. The original N−1N-1 bound is a worst-case sufficient budget over graphs with NN nodes, rather than a requirement for every graph.

Specifically, each continuous step performs one level of BFS expansion: the feedback represents the current frontier FcF_c, while the current and cached frontier records collectively recover the visited set (Rc=⋃j=0cFj)(R_c=\bigcup_{j=0}^{c}F_j). The next frontier is therefore Fc+1=N(Fc)∖Rc.F_{c+1}=\mathcal N(F_c)\setminus R_c. After CC continuous steps, answer readout succeeds whenever the reachable candidate belongs to RCR_C, equivalently whenever C≥dist⁡(r,c⋆)C\geq\operatorname{dist}(r,c_\star). Thus, DD continuous tokens are enough to obtain a correct answer, where DD is the maximum shortest-path distance between any reachable pair of nodes.

Accordingly, using cached history does not increase the required number of continuous tokens in our construction: it achieves the same diameter-dependent token bound as [1]. The distinction is how information is stored: [1] maintains the cumulative reachable set in each continuous thought, whereas our construction represents the current frontier in feedback and retains earlier frontiers in the KV cache.

For your convenience, we display our revised Theorem 1 below:

Let zcz_c be the input at position ic=3M+6+ci_c=3M+6+c, and let iA=3M+C+7i_A=3M+C+7 be the answer position.

Theorem (frontier-history). We consider integer budgets CC satisfying C≥dist⁡(r,c⋆)C\geq\operatorname{dist}(r,c_\star) and 3M+C+7≤Pmax⁡3M+C+7\leq P_{\max}. For any fixed NN, Mmax⁡M_{\max}, and Pmax⁡P_{\max}, there exists a two-block Transformer with the architecture above and parameters depending only on these quantities. (In particular, DD continuous tokens suffice for correct answer readout, and any larger budget also yields the correct answer, subject to the same context bound.)

For every legal prompt and every budget CC satisfying the conditions above, initializing z0=erz_0=e_r and running CC additional feedback steps before ⟨A⟩\langle A\rangle gives:

i. Frontier updates. For 0≤c≤C0\leq c\leq C, hic(2)=ϕ(Fc+1)h_{i_c}^{(2)}=\phi(F_{c+1}). Feeding each output into the next step gives zc=ϕ(Fc)z_c=\phi(F_c) for 1≤c≤C1\leq c\leq C.

ii. Visited-set recovery. For 0≤c≤C0\leq c\leq C, Block 1 retains zcz_c in frontier coordinates; a uniform causal head in Block 2 returns bc∈RNb_c\in\mathbb R^N with supp⁡(bc)=Rc\operatorname{supp}(b_c)=R_c, where supp⁡\operatorname{supp} denotes the nonzero node coordinates.

iii. Answer readout. The answer output is hiA(2)=ec⋆h_{i_A}^{(2)}=e_{c_\star}, giving the reachable candidate a unit logit margin over every other node.

We have revised the paper to make this guarantee explicit:

  1. Problem setup: We define the finite directed diameter DD and clarify that the context bound Pmax⁡P_{\max} covers the complete sequence, including the continuous tokens and the answer token.
  2. Theorem 1: We replace the fixed N−1N-1 step requirement with C≥dist⁡(r,c⋆)C\geq\operatorname{dist}(r,c_\star), subject to 3M+C+7≤Pmax⁡3M+C+7\leq P_{\max}, and explicitly state that DD continuous tokens suffice. We also make the dependence of the model parameters on Pmax⁡P_{\max} explicit.
  3. Appendix proof: We extend the induction to arbitrary admissible budgets CC, establish that the answer-position history signal has support RCR_C, and verify that the same readout thresholds remain valid throughout the fixed context bound, including after the frontier becomes empty.

Thanks for the response. I would increase my score to 3 to support acceptance.

We would like to thank you again for your insightful feedback, which indeed helped us strengthen the paper's theoretical results. We also appreciate your support for acceptance, and we would be happy to address any further questions or discuss any remaining concerns.

Reviewer_6hacOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how continuous CoT reasoning methods (Coconut in particular) distribute information between current latent state and the KV representations of earlier latent positions. The authors propose that the current latent state primarily captures the current stage of reasoning, while the KV cache retains information from earlier steps for use in subsequent computation and answer generation. I understand the authors' contribution to be as follows:

  • They show that earlier latent KV history continues to support the final answer generation even after the latent trajectory is complete, which is validated by hiding the previous latent KVs from the model during the answer generation.
  • They construct a two-block Transformer and formally prove that it solves directed graph reachability under the stated assumptions. In this construction, the current latent state encodes only the current search frontier, while the KV cache retains earlier frontiers. This establishes the computational sufficiency of this organization without requiring every latent state vector to store all previously reached nodes.
  • They investigate whether a similar organization emerges in trained models. While some similarities are observed in the trained two-layer model, the authors find that the current latent state also retains history relevant to subsequent updates, indicating only a partial correspondence with the theoretical construction.
Critical strengths

The paper combines causal evidence with a formal construction to clarify the role of persistent memory in continuous reasoning. Interventions across several models and datasets show that earlier latent KV entries remain useful for answer generation after the latent trajectory is complete. Complementing this evidence, the authors formally establish that a Transformer can solve graph reachability by distributing information between a latent state encoding the current search frontier and the cached history. Together, these contributions motivate studying the recurrent state and KV cache jointly, although they do not establish that trained models implement the exact theoretical mechanism.

Critical weaknesses

The connection between the theoretical construction and the mechanism learned by trained models is only partially supported. The current latent vector retains update-relevant historical information, while replacing the second latent step’s cached KVs with those from a paired graph where a node has a different visitation status produces only a small effect, opposite to the expected direction (Appendix G, Table 13). Moreover, the clear depth alignment observed in the two-layer checkpoint is not reproduced by the six-layer checkpoint (Appendix E), whose lower task accuracy makes this comparison inconclusive. These findings limit both the construction’s explanatory value and the generality of the empirical conclusions.

Questions for authors
  1. Are results available from additional training seeds or a six-layer checkpoint with task accuracy comparable to the two-layer model? Such evidence would help determine whether frontier alignment is reproducible and whether its absence in the six-layer checkpoint reflects architectural depth or unsuccessful training.
  2. Which aspects of the frontier–history mechanism can be generalized beyond graph reachability? In particular, do the experiments on arithmetic benchmarks provide evidence for a similar organization of intermediate information, or do they establish only that answer generation depends on earlier KV entries?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6hac

We sincerely thank you for your thoughtful feedback. Our responses follow.

The connection between the theoretical construction and the mechanism learned by trained models is only partially supported. The current latent vector retains update-relevant historical information, while replacing the second latent step’s cached KVs with those from a paired graph where a node has a different visitation status produces only a small effect, opposite to the expected direction (Appendix G, Table 13).

Regarding the experiment on 2-layer Transformer in Appendix G, we agree that learned feedback is not determined by frontier membership alone. Although our theorem constructs an exact frontier representation and trained models matched most of its properties, practical training may not necessarily discover exactly this representation.

In Appendix G, we use 59 answer-matched graph pairs that preserve F3F_3 and its outgoing neighborhood but make a node vv first reachable at depth two or four. We transplant either historical P2P_2 KV or feedback z3z_3 between paired runs and measure the resulting change in cos(z4,ev)cos(z_4,e_v). To examine whether the original result persists across model configurations, we additionally evaluated four- and six-layer models with tuned hyperparameters on the same pairs.

Intervention2-layer: Signed cosine shift4-layer: Signed cosine shift6-layer: Signed cosine shift
P2P_2 KV, all layers−0.001049+0.001060+0.000889
Donor z3z_3 only+0.053862+0.038684+0.044453

Positive shifts follow the direction predicted by donor history, and the four- and six-layer checkpoints show small positive KV-transplant effects. These differences may reflect changes in practical optimization or model configuration.

To interpret this stronger feedback effect, we directly compare the paired z3z_3 states before transplantation. Although the graphs have identical F3F_3, their feedback vectors assign different scores to vv:

Cosine similarity2-layer4-layer6-layer
Original: cos(z3,ev)cos(z_3,e_v)−0.05915−0.02650−0.04811
Counterfactual: cos(z3,ev)cos(z_3,e_v)−0.03491−0.00200−0.00816

Across all three models, z3z_3 gives vv a lower cosine score when vv has already been visited, which, to some extent, is aligned with our construction (we clear visited nodes from the current frontier, but it may not be so clean). Replacing the original z3z_3 with the counterfactual z3z_3 introduces a state in which vv has a higher score. This helps explain why vv’s score also increases in z4z_4 after the replacement. The reverse transplantation supplies feedback with a lower score.

Consequently, the results support frontier-selective feedback and depth-selective historical readout, while revealing a less clean separation in practical training than the theoretical construction, namely update-relevant context can pass through feedback as well as cached history. We have revised Appendix G accordingly.

AuthorsOpenReview ↗

Response to Reviewer 6hac (Conti.)

You mention that the clear depth alignment observed in the two-layer checkpoint is not reproduced by the six-layer checkpoint (Appendix E), whose lower task accuracy makes this comparison inconclusive.

Thank you for pointing out this concern. After carefully tuning the training hyperparameters, we obtained additional four- and six-layer models with task accuracy comparable to the two-layer model. On the standard test set, using three latent steps for three-hop questions and four latent steps for four-hop questions, their overall accuracies are:

ModelStandard test accuracy
2-layer97.97%
4-layer98.45%
6-layer98.57%

We then repeated the depth-alignment analysis on these checkpoints. The following table reports cosine-score AUROC for distinguishing nodes in each frontier FdF_d from nodes at other reachable depths, excluding the root.

ModelFeedback inputF1F_1F2F_2F3F_3F4F_4
4-layerz1z_10.98740.44320.27440.3113
4-layerz2z_20.24870.95490.38730.2834
4-layerz3z_30.26650.27630.91500.5961
4-layerz4z_40.35650.39130.48480.9387
6-layerz1z_10.98800.43340.27320.3562
6-layerz2z_20.28080.93090.37380.3177
6-layerz3z_30.25330.30410.87930.6248
6-layerz4z_40.35050.39110.49200.9312

Both models exhibit a clear diagonal pattern: z1,…,z4z_1,\ldots,z_4 preferentially represent their corresponding frontiers F1,…,F4F_1,\ldots,F_4, reproducing the depth-alignment pattern observed in the two-layer model.

For historical readout, we generate four latent steps normally and then restrict answer-generation access to a single latent KV position, keeping prompt KV.

ModelTarget depthP1P_1 onlyP2P_2 onlyP3P_3 onlyP4P_4 only
4-layer141.5464.1755.4339.21
4-layer247.2991.4176.4457.18
4-layer339.5166.2095.8468.24
4-layer447.7852.2586.3797.30
6-layer136.6241.1645.5842.05
6-layer228.7689.6868.3561.91
6-layer329.1646.1495.5772.57
6-layer427.0638.3477.2198.02

These results reproduce the pattern observed in the two-layer model: feedback representations align with the current frontier, while earlier cache positions support depth-selective readout after the trajectory is complete. Thus, the depth-aligned organization also emerges in four- and six-layer models with task accuracy comparable to the original two-layer checkpoint.

We have revised Appendix E accordingly in the revision.

AuthorsOpenReview ↗

Response to Reviewer 6hac (Conti.)

Which aspects of the frontier–history mechanism can be generalized beyond graph reachability? In particular, do the experiments on arithmetic benchmarks provide evidence for a similar organization of intermediate information, or do they establish only that answer generation depends on earlier KV entries?

Response. Our arithmetic experiments indeed provide evidence for stage-selective intermediate representations as well as historical KV dependence. The transferable principle is that feedback emphasizes information associated with the current reasoning stage, while earlier cached states remain useful for readout.

In addition, we analyzed 363 GSM8K problems with exactly three parseable reference equations. Our GPT-2 Coconut model uses six latent positions, two per reasoning step. Without training a probe, we applied the native LM head directly to each feedback input. The table reports how often each equation’s right-hand-side result appears among the top-three vocabulary predictions.

Feedback inputStep 1 RHSStep 2 RHSStep 3 RHS
z1z_115.43%4.96%6.34%
z2z_262.26%9.37%4.41%
z3z_32.48%4.96%3.58%
z4z_49.37%50.14%6.89%
z5z_512.12%2.48%2.48%
z6z_68.26%7.44%24.79%

The three intermediate results are most readily exposed at z2z_2, z4z_4, and z6z_6, respectively, resembling the depth-selective feedback observed in graph reachability.

On the same subset, we separately tested historical access. Readout interventions preserve the generated trajectory and mask the specified latent KV positions across all layers, retaining prompt KV. "Thinking prev-1" restricts generation to the preceding historical latent position, followed by unrestricted readout.

ConditionAnswer accuracy
Full latent KV history38.57%
Readout: mask P1,P2P_1,P_2 KV33.61%
Readout: mask P3,P4P_3,P_4 KV21.49%
Readout: mask P5,P6P_5,P_6 KV34.44%
Thinking prev-134.71%

Masking the middle KV group reduces accuracy by 17.08%, compared with 3.86 points under prev-1 thinking. Thus, arithmetic exhibits both stage-selective intermediate content and continued use of earlier history for answer generation. These findings support the broader information-organization principle, without establishing that arithmetic implements the construction's graph-specific frontier updates or visited-node exclusion.

We have added these addtional results in the revised version(Section 2 , Observation 4).

Reviewer_sXdEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper builds on a line of work studying language models with continuous chain of thought (COT). Here, the COT input vector at a step is the undecoded output from previous step, unlike usual COT, where the input tokens are decoded versions of the previous step output. Since the continuous COT inputs do not correspond to a single token, in principle, they can store significantly more information than a single token. In the limit, one can imagine that the input at time step t stores all necessary information about the past context, making the model operate like a recurrent model. The main message of this paper is refuting this simplistic picture, showing that information needed for future generation is accessed through both the latest cot vector as well as through the attention mechanism operating on the past context KV. In other words, the "memory" of the model is distributed and not localized to the most recent COT vector.

First, the paper investigates continuous COT models on the GSM8K task, showing that masking intermediate CoT vectors can hurt accuracy. Interestingly, it is shown that the accuracy-hit is minimal if masking is only done during COT generation but not during final answer generation. The implication being that the final answer read-out involves more global computation while COT generation mostly relies on recent continuous vectors.

Second, on a synthetic graph reachability task, a Transformer construction is given, showing how it can simulate BFS search using continuous COT. Here, the most recent COT vector encodes the frontier nodes of BFS search while the information about other already visited nodes is stored in the older KV. This construction aligns with the main message and is in contrast with construction in a prior work that stores all visited node information in the most recent COT vector.

Finally, it is empirically shown that a small Transformer trained on the graph reachability task demonstrates elements of this construction. For example, the most recent COT vector is shown to have higher cosine similarity with the nodes at the frontier of the BFS search compared to other visited nodes. However, this correspondence between the theoretical construction and the empirical results is only partial, and the separation of information between the recent COT vector and past KV is not as clean in practice.

Overall, the experiments build towards the main message suggestion that the "memory" of a continuous COT model is distributed across past KV and the most recent COT vector.

Critical strengths

The individual mechanistic experiments are well-done and stress-test the associated hypotheses from several angles. I found it interesting that on GSM8k style tasks, the continuous COT relies mostly on local information while the final answer relies on more globally spread information.

Critical weaknesses

My main concern with the paper is the lack of sufficiently general, non-obvious takeaways. As also discussed in the paper summary, the main takeaway that seem to generalize across different settings considered is that continuous COT models do not rely solely on the most recent COT vector and the KV of earlier COT also matters. While the experiments clearly demonstrate this, this behavior seems very much in line with how transformers are expected to work with attention mechanism on the past kv playing a crucial role.

The individual settings and experiments raise many other interesting hypothesis, but it remains unclear to what extent they generalize even to other settings considered within the paper. For example, the results on GSM8k show that intermediate COT vectors rely more on local past vectors while the final answer relies more broadly on globally spread information, but its unclear how general of a phenomenon it is beyond arithmetic datasets. Even in the graph reachability setup, many times, the final answer seems to be decodable from a single index in the COT. Another example: the theoretical construction provides a nice intuition about the two memory channels responsible for carrying information about the BFS frontier and the past visited nodes respectively but the experimental findings about it seem mixed, with results implying past visited node information still present in the recent COT vector.

Questions for authors

Many of the individual experiments raise interesting non-obvious findings. Can the authors shed some light on the generality of these findings? The paper would be much stronger if some of them can be shown to generalize across different settings (e.g. beyond just arithmetic to different kind of tasks), or if they don't uniformly generalize, some understanding is provided on the factors that control their generalizability. Also, if these conclusions can go beyond GSM8k and graph-reachability tasks with short COT trajectories to slightly more complex tasks with longer trajectories (e.g. MATH500), that would also make the paper more compelling.

6 comments
AuthorsOpenReview ↗

Response to Reviewer sXdE

We sincerely thank you for your thoughtful feedback. Our responses follow.

My main concern with the paper is the lack of sufficiently general, non-obvious takeaways. As also discussed in the paper summary, the main takeaway that seem to generalize across different settings considered is that continuous COT models do not rely solely on the most recent COT vector and the KV of earlier COT also matters. While the experiments clearly demonstrate this, this behavior seems very much in line with how transformers are expected to work with attention mechanism on the past kv playing a crucial role.

Response: While Transformers are expected to access historical KV, our work shows that its role differs substantially across stages of continuous CoT. That is, restricting historical access during thinking has a relatively small effect on accuracy, whereas restricting it during answer generation causes a much larger decline. Note that this dependence differs from explicit CoT, for example, on GSM8K, retaining only the final reasoning position at readout reduces Coconut's accuracy by 17.89% but leaves explicit CoT accuracy unchanged, as shown below.

ModelClean Accuracy (%)Thinking prev-1 ΔAcc. (pp)Readout last-one ΔAcc. (pp)Readout none ΔAcc. (pp)
GPT-2 Coconut34.42−2.96−17.89−28.73
Explicit CoT42.25—0.00−39.42

Moreover, Zhu et al. (2025) construct a continuous reasoner that accumulates all visited nodes in every feedback vector. Our theoretical construction shows that such accumulation is unnecessary, namely superposition can support reasoning with information distributed across feedback and persistent history. Our takeaway therefore goes beyond establishing that historical KV matters, we demonstrate that Coconut preserves much of its performance with limited historical access during thinking, while its trained answer reader remains strongly dependent on earlier latent history.

AuthorsOpenReview ↗

Response to Reviewer sXdE (Conti.)

For example, the results on GSM8k show that intermediate COT vectors rely more on local past vectors while the final answer relies more broadly on globally spread information, but its unclear how general of a phenomenon it is beyond arithmetic datasets. Even in the graph reachability setup, many times, the final answer seems to be decodable from a single index in the COT.

Response. We would like to clarify that our graph rexperiments indeed demonstrate that correct answer readout relies on information distributed across the reasoning history. While a single position can suffice for a graph query, the relevant finding here is that which position suffices depends on the queried depth, and earlier positions remain useful after all latent steps have finished.

Specifically, we fix the budget at four latent steps. "Feedback-only" removes access to past latent KV during thinking while preserving recurrent feedback and prompt access, followed by unrestricted readout. All other conditions use normal latent generation and modify only the answer’s access to latent KV. Prompt KV remains visible throughout.

Thinking conditionReadout access to latent KV1-hop2-hop3-hop4-hop
NormalAll positions50.7365.8259.7698.22
Feedback-onlyAll positions51.8966.9560.8398.60
NormalP1 only38.3649.4147.0042.99
NormalP2 only64.9892.7546.1943.16
NormalP3 only64.0773.4299.1449.38
NormalP4 only50.4864.7557.5698.64
NormalNone17.9115.4311.056.83

Feedback-only thinking remains close to the baseline at all four depths, while readout depends strongly on the accessible position. In particular, P2P_2 supports 2-hop queries well but performs poorly on 3- and 4-hop queries, while P3 supports 3-hop queries well but performs poorly on 4-hop queries , showing that available historical information is not always used effectively and answer-relevant information is distributed across the reasoning history, and answering queries at different depths requires access to different parts of that history.. Hence, single-position success therefore reveals selective historical access, and it does not make earlier positions dispensable.

We also measure frontier separability on unmodified questions. Each entry below is the AUROC for distinguishing nodes in FdF_d from nodes at other reachable depths using their cosine similarity to feedback ztz_t.

Feedback inputFrontier F1F_1Frontier F2F_2Frontier F3F_3Frontier F4F_4
z1z_10.9970.3760.3340.302
z2z_20.3190.9880.2310.336
z3z_30.3530.2220.9740.448
z4z_40.4450.4830.2770.975

The frontier AUROCs show that feedback preferentially represents the corresponding reasoning depth. Meanwhile, selective readout at P2 and P3 outperforms unrestricted readout, suggesting that the trained reader does not always use available information effectively. Together, these findings motivate studying latent computation and answer readout separately.

The cross-task takeaway is therefore stage-dependent and selective historical access, not that every answer requires all past positions. We have clarified this in Section 4.1 in the revised version.

AuthorsOpenReview ↗

Response to Reviewer sXdE (Conti.)

The experimental findings about it seem mixed, with results implying past visited node information still present in the recent COT vector.

Regarding the experiment on 2-layer Transformer in Appendix G, we agree that learned feedback is not determined by frontier membership alone. Although our theorem constructs an exact frontier representation and trained models matched most of its properties, practical training may not necessarily discover exactly this representation.

In Appendix G, we use 59 answer-matched graph pairs that preserve F3F_3 and its outgoing neighborhood but make a node vv first reachable at depth two or four. We transplant either P2 KV or feedback z3z_3 and measure the resulting change in cos⁡(z4,ev)\cos(z_4,e_v). We additionally trained four- and six-layer models (model's detail are in Appendix D) with tuned hyperparameters and evaluated the same pairs.

Intervention2-layer: Signed cosine shift4-layer: Signed cosine shift6-layer: Signed cosine shift
P2P_2 KV, all layers−0.001049+0.001060+0.000889
Donor z3z_3 only+0.053862+0.038684+0.044453

Positive shifts follow the direction predicted by donor history, and the four- and six-layer checkpoints show small positive KV-transplant effects. These differences may reflect changes in practical optimization or model configuration.

The strong effect of replacing z3z_3 also has an explanation. Although the paired graphs have identical F3F_3, their feedback vectors differ in their response to the previously visited node vv:

Cosine similarity2-layer4-layer6-layer
Original: cos(z3,ev)cos(z_3,e_v)−0.05915−0.02650−0.04811
Counterfactual: cos(z3,ev)cos(z_3,e_v)−0.03491−0.00200−0.00816

Across all three models, z3z_3 gives vv a lower cosine score when vv has already been visited, which, to some extent, is aligned with our construction (we clear visited nodes from the current frontier, but it may not be so clean). Replacing the original z3z_3 with the counterfactual z3z_3 introduces a state in which vv has a higher score. This helps explain why vv’s score also increases in z4z_4 after the replacement. The reverse replacement is expected to have the opposite effect.

Consequently, the results support frontier-selective feedback and depth-selective historical readout, while revealing a less clean separation in practical training than the theoretical construction, namely update-relevant context can pass through feedback as well as cached history. We have revised Appendix G accordingly.

Comments on the paper

1 comment

We would like to thank the AC for overseeing the review process and all three reviewers for their thoughtful feedback and active engagement throughout the discussion. Below, we summarize the reviewers' main concerns and how we addressed them during the rebuttal and discussion. The corresponding changes are highlighted in blue in the latest revision.

1. Difference between our findings and normal transformer (Reviewers sXdE). We clarified that our finding is not simply that Transformers can attend to earlier KV states, but that latent generation and answer readout depend on them differently. Zhu et al. [1] showed that accumulating all visited nodes in each feedback vector suffices for graph reachability; our construction shows that this is not necessary: feedback can represent only the current frontier while the KV cache retains earlier frontiers.

2. Generality across tasks (Reviewers sXdE and 6hac). Across 15 of 16 arithmetic model–dataset settings, restricting historical access during thinking caused a smaller accuracy loss than restricting readout to the final latent position. We also added an analysis of 363 GSM8K problems with three reference equations. Their successive results were most readily decoded at latent steps two, four, and six, respectively, providing evidence for stage-specific feedback beyond graph reachability. Following Reviewer sXdE’s suggestion, we added MATH500 and longer reasoning trajectories to the Concluding Remarks as future work. We plan to train and validate a Coconut checkpoint on a stronger backbone than the GPT-2 model used in this analysis so that both interventions can be meaningfully compared on harder problems.

3. Correspondence between the construction and trained models (Reviewers sXdE and 6hac). We added four- and six-layer graph models with task accuracy comparable to the two-layer model. All three showed frontier-selective feedback and depth-selective historical readout. In matched graph pairs with identical current frontiers, replacing feedback affected the revisited node’s next-step score more than replacing the earlier KV entry. The revised discussion explains how learned feedback can emphasize the current frontier while retaining additional context that influences the next update.

4. Reasoning budget and the role of cached history (Reviewers bG6b and sXdE). In response to Reviewer bG6b, we strengthened Theorem 1 to show that any admissible budget covering the shortest-path distance to the reachable candidate suffices, yielding a diameter-dependent guarantee. In response to Reviewer sXdE, we clarified that masking earlier latent positions during thinking can allow revisits while preserving the correct answer when the reader has access to the full history.

We appreciate the reviewers’ suggestions, which strengthened both the theorem and its empirical interpretation, and are grateful that all three reviewers now lean toward acceptance.

[1] Hanlin Zhu, et al. "Reasoning by superposition: A theoretical perspective on chain of continuous thought." NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.