- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how explicit chain-of-thought is internalized into continuous latent computation through a framework connecting training dynamics with inference-time function. The results show positional specialization and training-time parity reversals, partial semantic alignment with explicit reasoning despite distinct intermediate representations, and recurrent dependence between latent states. The explicit–latent reliability gap increases with solution depth, while failed trajectories exhibit weaker semantic ordering and more diffuse positional influence.
Critical strengths
-
The checkpoint analyses provide consistent evidence of positional specialization and parity reversals across multiple measurements. These findings offer valuable insights into how latent computation is reorganized during training.
-
Comparing fixed and regenerated continuations, together with restoration experiments, provides convincing evidence of inter-state dependence, and the careful intervention design helps explain how latent states work together during reasoning.
The paper presents a solid empirical study with systematic analyses and carefully designed interventions.
Critical weaknesses
The experiments are restricted to mathematical word problems on GSM8K, leaving the generality of the reported organizational patterns uncertain. Evaluation on non-mathematical benchmarks, such as PrOntoQA for logical reasoning and CommonsenseQA for commonsense reasoning, would help establish whether positional specialization, recurrent dependence, and the relationship between latent organization and reliability extend beyond arithmetic reasoning.
Questions for authors
-
How can the positional, alignment, and intervention analyses guide practical improvements to existing methods such as CODI and Coconut, for example in latent-state supervision or latent-budget allocation?
-
Can the authors evaluate whether the main positional patterns and outcome-conditioned organizational differences also appear on non-mathematical tasks such as PrOntoQA and CommonsenseQA?
3 comments
Part 2
W1+Q2 (continue)
The reliability gap retains its depth dependence (§4.3). On GSM-Hard, which spans multiple reasoning depths, the conditional latent failure rate again increases sharply with solution depth:
| Explicit solution depth (GSM-Hard) | ≤2 | 3 | 4 |
|---|---|---|---|
| CODI GPT-2 | 0.11 | 0.26 | 0.53 |
| CODI Llama-1B | 0.13 | 0.17 | 0.36 |
| Coconut GPT-2 | 0.23 | 0.40 | 0.55 |
Together, these findings highlight a fundamental limitation of current implicit reasoning paradigms: latent computation is not consistently and reliably learned and organized into a coherent reasoning process. Models may either bypass latent states altogether or exhibit pronounced positional biases and insufficient coordination when using them. This motivates stronger relational process supervision to promote meaningful latent computation and coherent coordination across states.
[1] Are Latent Reasoning Models Easily Interpretable? COLM 2026
[2] The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models. 2026
[3] Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought. EMNLP 2026
Q1
Our mechanistic findings provide concrete directions for improving CODI and Coconut, building on the discussion in §5
- Relational process supervision. Our analyses highlight the importance of meaningfully utilizing latent states and coordinating their computation into a coherent reasoning process, as reflected in differences in causal reliance across tasks and weaker semantic ordering and more diffuse causal influence in failed trajectories. These findings motivate stronger supervision of intermediate computational content and inter-state coordination, promoting meaningful and well-organized latent computation without imposing rigid step-to-position alignment. This principle could guide improvements to CODI and Coconut, and potentially extend to iterative architectures such as looped models and latent diffusion models.
We hope the above replies meaningfully address your core concerns.
Thank you for the additional cross-task experiments. These results help clarify the scope of the findings, and I encourage you to include them in the revised paper.
Part 1
Thank you for the constructive review and positive assessment. We address your questions and concerns below and will incorporate your feedback in the updated version.
We extend our experiments to additional benchmarks, including ProsQA, CommonsenseQA, SVAMP, MultiArith and GSM-Hard. The new evidence further strengthens the paper’s central finding: current implicit reasoning paradigms have yet to reliably learn to meaningfully utilize and coordinate latent states into a coherent, well-organized reasoning process. On ProsQA and CommonsenseQA, latent states are unused for computation, and thus no pronounced organization emerges; on SVAMP, MultiArith, and GSM-Hard, where latent computation is actively used, the core organizational patterns reappear, including positional specialization, training-time reversals, and weaker causal organization in failed trajectories.
Removing latent tokens leaves accuracy virtually unchanged on ProsQA and CommonsenseQA, but reduces GSM8K accuracy by 22–28 points. Causal interventions confirm this contrast: donor substitutions change no ProsQA answers and at most 1.2% on CommonsenseQA, versus 94–96% on GSM8K. This negligible causal reliance persists across all analyzed checkpoints.
("Donor" replaces the whole latent trajectory with that of a different-answer problem (as in Fig. 5c), and total causal load is the summed effect of erasing each latent input in turn (Σ_i |N_{dir,i}|), in nats of gold-answer log-probability.)
Correspondingly, no pronounced positional specialization emerges on ProsQA or CommonsenseQA. As the model itself does not properly learn to utilize latent states for reasoning, there's naturally not any prominent organization within the latent states emerges.
Training-time formation and reversals replicate. Across 10 CODI GPT-2 checkpoints, the even–odd selectivity of erasure effects,
(even − odd)/(even + odd), follows the same dynamics:Positional and causal specialization persist at convergence. At the final checkpoints:
Layers: GSM8K parity-index magnitude >0.5. Erasure profiles retain the GSM8K-dominant parity for both CODI models on all benchmarks, and Coconut on MultiArith/GSM-Hard (S=0.45/0.17). GSM-Hard CODI GPT-2 also retains its erasure-profile shape (u0–u5: 1.26/.13/.22/.08/.37/.04 nats; Fig. 1d).