acceptodds
Under review as a conference paper at ICLR 2027

Learning to Echo Chain of Thought in Latent Space via Self-Distillation

Abstract

Chain-of-thought (CoT) reasoning has driven much of the recent progress of language models on complex problems, but generating reasoning traces autoregressively makes inference costly. Latent reasoning instead carries out intermediate computation in continuous hidden states, which can compress reasoning into far fewer steps. Supervising these states is the open problem: a text trace does not say what each state should encode, so existing methods rely on sparse answer supervision or on traces with clear reasoning-step boundaries, which most domains lack. We introduce Echo, which teaches a model to reason in latent space by *echoing* its own explicit reasoning. Our key insight is that as a model reads the CoT, its hidden states already trace the progression of its reasoning. Thus, randomly sampling and sorting them yields dense targets for latent states, without a separate teacher, a curriculum, and step annotations. Echo has two branches: an explicit branch that sees the question and CoT and a latent branch that sees only the question and learns to reason with a small set of latent slots. At each training step, the slots are supervised with freshly sampled, chronologically ordered hidden states from the explicit branch. With GPT-2 and Llama-3.2-1B-Instruct, Echo is the most competitive latent method in 9 of 12 settings across math, commonsense reasoning, and coding, even surpassing Explicit CoT in some cases. Since it refines a compact set of slots in parallel, it reduces inference latency by up to relative to Explicit CoT. These results suggest that models can effectively self-distill from the CoT hidden states and reason compactly in the latent space during inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, chain-of-thought, large language models, looped transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EzHfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose Echo, a method to train a model to reason with continuous vectors without needing per step supervision. The paper presents a simple approach on using sampled hidden states of explicit CoT to supervise the latent representations. Experiments using GPT-2, LLaMA-1B on math, commonsense and coding tasks establish that the method is competitive with existing methods. Authors present detailed ablations to show the importance of various components of the approach.

Critical strengths
  • Importance of the Problem and Simplicity of the approach. The core problem of requiring annotated steps for latent supervision is neatly sidestepped by using sampled hidden states of CoT. Ablations on different methods of sampling are helpful and interesting. Authors try out multiple possible options but find that the simplest Random Sorted approach works the best.
Critical weaknesses
  • Inference Latency Comparison. The paper uses GFLOPs as the main measure of inference compute but it is not specified how it is calculated for different methods. Since the final model to be used for inference is similar for all the considered methods (a backbone -> hidden state looped -> answer) it is unclear why there is a variation among the methods. Accuracy numbers should be accompanied by allocated latent steps during training. Moreover, Inference Latency comparison is only presented with explicit CoT, when compared to other latent reasoning baselines it remains similar. Authors are requested to give a detailed account of inference compute and latency when compared to other latent reasoning methods.

  • Seed Variance and Evaluation Datasets. Considering that the core approach involves training on randomly sampled hidden states, authors should provide details on(or lack of) training seed variance on evals, as some selections might result in sub-optimal training. Further, this concern would be more applicable on longer problems. It is well established by prior works that latent reasoning methods work well (even better than CoT) on non-math tasks [CODI, CoCoNUT]. Therefore, atleast one evaluation on a slightly longer math dataset (GSM8k-Aug-NL for example) would remove any suspected regression on those tasks.

Score Update

Inference Latency Comparison.

Authors explain the Inference-time algorithms for methods and present the required table containing number of recurrent steps, latent slots and accuracy. The inference gains result from an improved source of supervision on LOTUS's inference scheme. I encourage the authors to include a more detailed description of the method and add Table R4 to the appendix.

Seed Variance and Evaluation datasets

Authors provide results with multiple seeds and a larger model and the improvements are consistent and the seed variance is also low, suggesting potential reliability of the supervision signal. Experiments on GSM8K-Aug-NL also show improvement w.r.t baselines showing that the method can work harder and longer traces.

As my major concerns are resolved I have raised my score to 3.

Questions for authors
  • Are you using different latent reasoning budget for different methods?
    • If yes, could you provide a detailed table with columns as (method, number of latent slots, number of recurrent steps, acc on gsm, acc on csqa, acc on pyx).
    • if not, why are flops and inference latencies so different for different methods?

Update

Authors provide the answers and my concerns regarding these questions are appropriately resolved.

12 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their thoughtful and constructive feedback. We were glad the reviewer appreciated how our simple approach neatly sidesteps the need for annotated reasoning steps, and the comprehensive ablations on sampling strategies.

The reviewer raised a few questions asking to compare Echo’s inference latency with other latent reasoning methods, explain how FLOPs are computed and how we set latent budgets across methods, and report seed variance and results on longer math data. We address each of these points below.

1. Inference compute and latency across latent methods

Echo lies on the accuracy–efficiency Pareto frontier in every panel of Fig. 3 (GPT 2) and Fig. S3 (Llama 1B) except GSM8K FLOPs. Echo's efficiency advantage grows with reasoning length. Going from GSM8K to PyX, the explicit-CoT reasoning trace grows from ~17 to ~460 tokens (Table S2). Explicit CoT latency grows 27× (114 → 3,032 ms), while Echo’s stays flat (49 → 48 ms), so its speed-up grows from 2.3× to 63×. Echo is also faster than LOTUS on both datasets while being more accurate. Coconut and CODI are faster on GSM8K but around 7–20pp less accurate. On PyX, Echo is faster than CODI (47.8 vs. 56.6 ms) with similar accuracy.

We updated the manuscript to add this more detailed comparison to other latent reasoning models in Section 4.3.

Table R1: Inference latency (Llama-1B-Instruct, batch size 1, one H200).

MethodGSM8K latency (ms)Speed-up vs. CoTPyX latency (ms)Speed-up vs. CoT
Explicit CoT113.81–3032.14–
Coconut39.672.87×40.1075.6×
CODI33.103.44×56.6253.6×
LOTUS52.862.15×50.4460.1×
Echo48.932.33×47.8063.4×

2. How we measure cost

GFLOPs are computed analytically for the full inference pass: question prefill, latent computation, and answer decoding. Latency is wall-clock time for the same pass. Both are averaged over the test set with batch size 1 on one H200, so padding does not affect either. Fig. S2 (Appendix D.1) shows that Echo’s speed-up grows further with larger batch sizes.

3. Seed variance and larger model

As per your recommendation, we ran our experiments and updated the manuscript with:

  1. Three random seeds for every main experiment.
  2. A larger model, Llama-3B-Instruct.

Also, we’d like to note that CommonsenseQA and PyX are also longer reasoning datasets, whose reasoning is ~67 and ~460 tokens respectively (compared to GSM8K at ~17 tokens).

Below, we report the average performance across 3 seeds, comparing Echo with the top-scoring baseline on each dataset. Echo is competitive on 8 of the 9 settings shown below, and maintains generalization to OOD math datasets (full results in Table 2).

Table R2: In-distribution accuracy (mean ± std over 3 seeds), Echo vs. the top-scoring baseline.

ModelGSM8K-AugCSQA (free-form)PyX (free-form)
GPT-246.75 ± 0.34 vs. 44.55 ± 0.56 (LOTUS)45.07 ± 0.31 vs. 41.36 ± 0.49 (LOTUS)42.88 ± 1.16 vs. 43.13 ± 0.38 (CODI)
Llama-1B54.33 ± 0.69 vs. 53.15 ± 1.24 (CODI)74.64 ± 0.90 vs. 73.90 ± 0.41 (CODI)64.75 ± 0.82 vs. 63.28 ± 0.32 (LOTUS)
Llama-3B67.17 ± 0.39 vs. 65.25 ± 0.73 (LOTUS)81.65 ± 0.65 vs. 81.08 ± 0.43 (CODI)70.86 ± 1.33 vs. 70.57 ± 0.24 (LOTUS)

4. Longer math data (GSM8K-Aug-NL) with more baselines

We also added GSM8K-Aug-NL and two new baselines PCCoT and MUX (suggested by Reviewer nrEQ), for Llama-1B. We are also running Llama-3B experiments and will update the reviewer in a couple of days. GSM8K-Aug-NL contains the same problems as GSM8K-Aug but with longer language-based reasoning. We draw two observations:

a) Natural-language traces are harder for every method at 1B. Moving from GSM8K-Aug to GSM8K-Aug-NL lowers accuracy across the board, including Explicit CoT (−5.6pp).

b) Importantly, Echo remains the strongest latent method on GSM8K-Aug-NL, outperforming both the newly added baselines by 2-4pp.

Table R3: GSM8K test accuracy (%) after training on GSM8K-Aug or GSM8K-Aug-NL. Bold marks the best latent method.

ModelMethodGSM8K-AugGSM8K-Aug-NL
Llama-1BExplicit CoT59.7754.21
Coconut34.7227.22
CODI53.1542.23
LOTUS49.7628.7
PCCoT52.9247.08
MUX54.9745.56
Echo57.4749.28
AuthorsOpenReview ↗

Continued response

5. Are latent budgets matched?

In latent reasoning models, there are two axes to consider for the latent budget: the number of latents generated / processed for each forward pass (KK) and the number of sequential forward passes (RR). The number of sequential forward passes decides the computational depth of the model. Thus, following LOTUS, we standardize the number of sequential forward passes and use the optimal configuration for KK recommended in each paper (Table R3).

  • Coconut, CODI: generate one continuous token per step, sequentially (6 steps).
  • LOTUS: 6 latent blocks of width c = 25 or 30 (150–180 latents in total), refined in parallel over R = 6 passes.
  • Echo: K = 6–10 slots, refined in parallel over R = 6 passes.

Table R4: Latent budget and accuracy (Llama-1B-Instruct). Best latent reasoning model in bold.

MethodTotal latent slots (GSM8K / CSQA / PyX)Sequential forward passesGSM8KCSQAPyX
Explicit CoT–~18 / 85 / 461 (one per token)59.7770.4367.93
Coconut6634.7267.4955.42
CODI6653.1573.9063.00
LOTUS150 / 180 / 150649.7672.1063.28
Echo10 / 10 / 6654.3374.6464.75

6. Why do FLOPs and latency differ across methods?

Although all latent methods share the same backbone, they process different numbers of latent slots. FLOPs scale with the total number of slots processed across all steps, while latency scales mostly with the number of sequential forward passes: parallel latent slots (LOTUS, Echo) are computed in the same forward call, so they add FLOPs but relatively little latency. This is why LOTUS (up to 180 latents) uses up to 12× more FLOPs than Echo (6 - 10 latents) at similar latency. It is also why Coconut and CODI, which process a single slot per step, use the fewest FLOPs.

Thank you for your detailed response and additional experiments. My concerns regarding variation with random seeds (3) and potential degradation on GSM8k-Aug-NL(4) are now resolved. Authors' explanation regarding GFLOPs(2) and flops/latency connection(6) also clarifies a lot of details.

I still have concerns regarding (1) and (5) which I outline below.

As I understand, ECHO's core contribution is training the latents by self-distilling randomly sampled, chronologically ordered hidden states from model's own CoT into its latent slots. This should only improve improve latent quality and downstream performance. Inference GFLOPs and latency, are set by the number of latent slots and sequential forward pass the algorithm uses.

Could you describe the inference difference (algo, GFLOPs and/or latency) between LOTUS(c=1, K=6) and ECHO (K=6, R=6)? I think a short explanation here would confirm the source of efficiency gains.

Reviewer_nrEQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Echo, a latent reasoning method trained from an explicit chain of thought. During training, the teacher receives the question, CoT, and answer, while the student receives the question and K latent positions. Echo samples K hidden states from the explicit CoT at each update, keeps their CoT order, and uses them as targets for the latent positions. The training loss combines explicit-CoT loss, answer loss, and a normalized Smooth L1 hidden-state loss. At inference, Echo uses the latent pass and does not generate a CoT.

The method uses the same basic parallel recurrent latent computation as PCCoT. Its main change is to replace PCCoT’s pre-answer target with K sampled hidden-state targets from across the CoT. The paper evaluates GPT-2 (124M) and Llama-3.2-1B-Instruct trained on GSM8K-Aug and CommonsenseQA.

Critical strengths

The method replaces the sparse, single-position supervision used by CODI and PCCoT with dense supervision across the latent trajectory. This directly targets what the intermediate latent positions should represent.

Critical weaknesses
  1. The related-work framing and baseline comparison are incomplete. Echo is close to PCCoT [1] in its parallel recurrent latent computation, but replaces the single pre-answer distillation target with K sampled targets from across the CoT. Tables 2 and 3 do not include PCCoT [1], KaVa [2], or MUX [3], even though KaVa and MUX also use dense supervision from explicit reasoning traces. The paper also claims that Echo needs no separate teacher, curriculum, or step boundaries, “all of which prior work relies on.” This statement is too broad and is contradicted by the paper's own Table 1, which describes CODI [4] as one-stage, self-distilled, and not dependent on a curriculum or CoT boundaries. PCCoT also uses CODI's shared-backbone training objective. The paper should state the narrower differences from PCCoT and CODI and compare against PCCoT and the dense-supervision methods.

  2. The novelty and method description are not well separated. Most components of Echo, including parallel recurrent latent positions and joint explicit/latent training, build directly on previous methods. The main new component appears to be how the distillation targets are selected and resampled. However, the method section devotes substantial space to reused components without clearly isolating the actual contribution which distinguish their method from previous works.

  3. The evaluation is limited.

    • The models are GPT-2 (124M) and Llama-3.2-1B-Instruct only.

    • For mathematical reasoning, the training data are limited to GSM8K-Aug, whose traces are short and contain only equations. It does not include GSM8K-Aug-NL or a more challenging math dataset with longer traces, such as MetaMathQA.

    • The paper does not report multiple seeds, standard deviations, or confidence intervals. This matters for the small differences between Echo, LOTUS, and explicit CoT.

Questions for authors
  1. How does Echo differ from PCCoT [1] beyond replacing the single pre-answer distillation target with K sampled, ordered hidden states from the CoT? Why should this supervision signal be more effective than other dense supervision signals in the literature? The paper's positioning relative to prior work should be clearer.

  2. Why are PCCoT [1], KaVa [2], and MUX [3] not included in Tables 2 and 3? Please compare against them using the same backbones and data protocols.

  3. Could the authors provide results on GSM8K-Aug-NL and MetaMathQA [5], including comparisons with the dense-supervision baselines?

  4. Could the authors run the experiments with multiple seeds and report the mean and standard deviation?

[1] Wu, Haoyi, Zhihao Teng, and Kewei Tu. "Parallel continuous chain-of-thought with jacobi iteration." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[2] Kuzina, Anna, Maciej Pióro, and Babak Ehteshami Bejnordi. "Kava: Latent reasoning via compressed KV-cache distillation." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Suleymanzade, Ayhan, et al. "MUX: Continuous Reasoning via Multiplexed Tokens." arXiv preprint arXiv:2607.18264 (2026).

[4] Shen, Zhenyi, et al. "Codi: Compressing chain-of-thought into continuous space via self-distillation." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[5] Yu, Longhui, et al. "Metamath: Bootstrap your own mathematical questions for large language models." International Conference on Learning Representations. Vol. 2024. 2024.

2 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their thoughtful and constructive feedback.

We were glad the reviewer appreciated that Echo’s dense supervision, unlike the single-position supervision of CODI and PCCoT, directly targets what intermediate latents should represent.

The reviewer raised a few questions asking to expand the scope of the evaluations (model size, dataset, and baselines) and to clarify Echo’s contribution in the context of prior work. We address each of these points below.

1. Multiple seeds

We now report all results in Tables 2 and 3 over 3 random seeds and add Llama-3B-Instruct as a larger model. Echo is competitive on 8 of the 9 settings below and maintains generalization to the OOD math datasets in Table 2. The table below summarizes the in-distribution results, juxtaposing Echo’s score against the top-scoring latent reasoning baseline on each dataset. The full table is in the updated manuscript.

Table R1: In-distribution accuracy (mean ± std over 3 seeds), Echo vs. the top-scoring baseline.

ModelGSM8K-AugCSQA (free-form)PyX (free-form)
GPT-246.75 ± 0.34 vs. 44.55 ± 0.56 (LOTUS)45.07 ± 0.31 vs. 41.36 ± 0.49 (LOTUS)42.88 ± 1.16 vs. 43.13 ± 0.38 (CODI)
Llama-1B54.33 ± 0.69 vs. 53.15 ± 1.24 (CODI)74.64 ± 0.90 vs. 73.90 ± 0.41 (CODI)64.75 ± 0.82 vs. 63.28 ± 0.32 (LOTUS)
Llama-3B67.17 ± 0.39 vs. 65.25 ± 0.73 (LOTUS)81.65 ± 0.65 vs. 81.08 ± 0.43 (CODI)70.86 ± 1.33 vs. 70.57 ± 0.24 (LOTUS)

2. Experiments with a new dataset (GSM8K-Aug-NL) and additional baselines (PCCoT, MUX)

As the reviewer suggested, we added GSM8K-Aug-NL and two new baselines, PCCoT and MUX, for Llama-1B. We are also running Llama-3B experiments and will update the reviewer in a couple of days. GSM8K-Aug-NL contains the same problems as GSM8K-Aug but with longer language-based reasoning. We do not include KaVa, as there is no public official implementation. We draw two observations:

a) Natural-language traces are harder for every method at 1B. Moving from GSM8K-Aug to GSM8K-Aug-NL lowers accuracy across the board, including Explicit CoT (−5.6pp).

b) Importantly, Echo remains the strongest latent method on GSM8K-Aug-NL, outperforming both the newly added baselines by 2-4pp.

Table R2: GSM8K test accuracy (%) after training on GSM8K-Aug or GSM8K-Aug-NL. Bold marks the best latent method.

ModelMethodGSM8K-AugGSM8K-Aug-NL
Llama-1BExplicit CoT59.7754.21
Coconut34.7227.22
CODI53.1542.23
LOTUS49.7628.7
PCCoT52.9247.08
MUX54.9745.56
Echo57.4749.28

3. Comparison to related work

Supervision methods for latent reasoning fall broadly into two groups.

  • Sparse methods:
    • CODI & PCCoT: supervise only the hidden state right before the answer (e.g., at the “:” token). The intermediate latents get no target of their own and learn only indirectly through the answer loss. PCCoT adopts CODI’s objective and enables computing the latents in parallel via Jacobi iteration.
    • LoRi: matches the low-rank trajectory statistics of the teacher’s hidden state trajectory.
  • Dense methods:
    • Text (Coconut, LOTUS, SIM-CoT): latents are supervised with the text of reasoning steps and assume access to step-wise boundaries in the reasoning data. LOTUS and Coconut progressively swap the reasoning text with latents through a curriculum. SIM-CoT uses an auxiliary decoder to rebuild each step from its latent.
    • KV Cache (KaVa): latents are distilled from the teacher’s KV cache, compressed with a hand-designed importance/redundancy score.
    • Token-embedding / Vocabulary space (MUX, CoLaR): targets are built from CoT token embeddings. MUX uses vocabulary-space superpositions of CoT spans. CoLaR predicts compressed embeddings made by merging consecutive CoT tokens. These also assume access to step-wise boundaries in the reasoning data.
    • Hidden State (Echo): targets are the teacher’s internal hidden states. Echo matches hidden states from the same model’s explicit CoT, resampled during training while keeping their order.

In the revision, we reorganized the related work section accordingly and clarified how Echo differs from other latent reasoning supervision methods.

AuthorsOpenReview ↗

Continued response

4. Echo’s novelty should be isolated

We agree the novelty should be isolated. Our contribution is the supervision target: at every training step, each latent slot is matched to a hidden state from a freshly resampled, chronologically ordered subset of the model’s own CoT. Unlike prior dense methods, this needs no reasoning-step boundaries (Coconut, LOTUS, SIM-CoT) and no hand-designed score for choosing positions (KaVa).

Our proposed supervision target is what drives Echo’s gains. Our ablations in Fig. 4 (GPT-2, GSM8K) isolate each part of the target and illustrate that all three of its characteristics are important:

  • Dense targets: removing our dense supervision drops the performance 47.1% → 32.5% (−14.6pp).
  • Order: shuffling the target order and removing temporal consistency drops accuracy to 44.2% (−2.9pp).
  • Resampling: disabling re-sampling and fixing the target positions throughout training also degrades performance by 2.4pp.

Why this target is effective (Intuition): Echo’s objective can be viewed as masked modeling (BERT [1], data2vec [2], I-JEPA [3], MAE [4]), where the entire CoT is withheld from the latent branch and the latents must predict a subset of the model’s own CoT hidden states. So, the three components (dense targets, ordering, and resampling) result in: Causal and ordered targets give dense, cumulative signal. Each target is a causal hidden state, so it teaches the model to recover the reasoning up to that position Resampled targets act as augmentation that prevents the latents from overfitting to particular positions

In the revision, we took the reviewer’s feedback to heart and made the following changes:

  1. Introduction. We removed the overly broad claim about prior work from the contribution statement.
  2. Method section. We restructured Section 3 to clearly delineate what we adopt and what we contribute. In Section 3.2, we added details for our target selection policy (previously in the Appendix) and in Section 3.4, we explain the intuition behind why it works.

[1] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019. [2] Baevski, Alexei, et al. "Data2vec: A general framework for self-supervised learning in speech, vision and language." International conference on machine learning. PMLR, 2022. [3] Assran, Mahmoud, et al. "Self-supervised learning from images with a joint-embedding predictive architecture." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023. [4] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2022.

Reviewer_KRrqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ECHO, a method that uses randomly sampled hidden states from Explicit CoT as targets for latent reasoning. Experiments with GPT-2 and Llama-3.2-1B-Instruct show that the proposed method is the most competitive latent reasoning approach in 9 out of 12 settings.

Critical strengths
  • The paper is easy to follow, and the core idea is straightforward.

  • The method outperforms other latent reasoning baselines on most tasks.

  • The authors provide comprehensive ablations to study the different components of the method.

Critical weaknesses
  • The experiments are limited to relatively small models. It remains unclear whether the method generalizes to larger models, where reducing inference cost could be particularly useful.

  • The training cost appears relatively high, as the method requires maintaining both an Explicit CoT teacher and a latent reasoning student during joint training.

Questions for authors
  • Could the authors evaluate the method on more models?

  • Have the authors considered a two-stage variant where the Explicit CoT teacher is trained first and then frozen before distilling its hidden states into the latent student? This would help clarify the benefits of joint training.

2 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their thoughtful and constructive feedback. We were glad the reviewer appreciated the paper’s clarity, the strong performance of Echo over latent reasoning baselines, and the comprehensive ablations.

The reviewer raised a few questions about Echo’s scalability on larger models and the cost of Echo’s joint training framework. We address each of these points below.

1. Scalability of Echo to larger models

We added Llama-3.2-3B-Instruct to all six benchmarks and ran three random seeds for each run. At 3B, Echo is the strongest latent method, with the highest accuracy on 5 out of 6 benchmarks.

We updated Tables 2 and 3 in the manuscript and provide a compact view of results below.

Table R1: Accuracy (%) with Llama-3.2-3B-Instruct, averaged over three random seeds. Bold marks the best latent method.

MethodGSM8KCSQAPyXAvg
Explicit CoT71.1778.9279.7876.62
Answer-only SFT38.6480.9470.5063.36
Coconut46.7578.9266.2263.96
CODI60.4081.0869.7470.41
LOTUS65.2579.9970.5771.94
Echo (ours)67.1781.6570.8673.23

2. Training cost of Echo

We’d like to clarify a point here. Echo does not keep a separate teacher model. The explicit and latent branches use the same weights, run in two “modes” (explicit vs. latent reasoning).

Echo is usually cheaper to train end-to-end than the baselines, because it trains in a single stage:

Table R2: Comparison of Training Pipeline and Cost.

MethodTraining pipelineGPU Hours with 2 x H200 GPUs
LOTUSMulti-stage: Train explicit CoT teacher, then a progressive curriculum for the student replacing the CoT steps with latent slots126
CoconutMulti-stage: Progressively replaces the CoT steps with latent slots31
CODISingle stage: trains a single model (2 forward passes)40
EchoSingle stage: trains a single model (1 forward pass)28

The GPU hours are estimated from the same 2 x H200 GPUs training setup that we used for the 1B model.

3. Have the authors considered a two-stage variant where the Explicit CoT teacher is trained first and then frozen before distilling its hidden states into the latent student?

This is an insightful suggestion that we explored. The frozen-teacher variant works well, but does not improve upon our proposed joint training paradigm. Thus, we propose to jointly train because it is simpler and cheaper:

  • It needs one training run instead of two.
  • It does not require holding two copies (student & teacher) of the weights during distillation.

Dear Reviewer KRrq,

As the discussion period approaches its end, we would like to kindly check whether you have any remaining questions or comments. We hope that our clarifications have addressed your concerns satisfactorily. Please don’t hesitate to reach out if further discussion would be helpful. We would be happy to engage.

Thank you again for your response and thoughtful engagement!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback, and the AC for their time and consideration. We were glad that reviewers appreciated “Echo’s core contribution is training the latents by self-distilling randomly sampled, chronologically ordered hidden states from [the] model’s own CoT into its latent slots,” (EzHf, nrEQ), “neatly sidestep[ing] the need for annotated reasoning steps (EzHf). Reviewers also appreciated the paper's clarity (KRrq), the strong performance of Echo over latent reasoning baselines (KRrq), and the comprehensive ablations, including those on sampling strategies (KRrq, EzHf).

During the response phase, we ran new experiments and revised the paper to address each major question. We summarize the results and takeaways below, and we welcome further discussion.

1. Echo scales to larger models (KRrq, nrEQ)

We extended all six benchmarks to Llama-3.2-3B-Instruct, with three random seeds per configuration. At 3B as well, Echo is the strongest latent method in all three domains and leads on five of six benchmarks (Tables 2 and 3).

2. Echo’s gains hold across seeds (nrEQ, EzHf)

We report all main results with 3 random seeds and Echo achieves the highest mean accuracy among latent methods in 8 of 9 in-distribution settings. **The gains also carry over out of distribution on GSM-Hard, SVAMP, and MultiArith (See updated Tables 2 and 3).

3. New dataset with longer reasoning traces (GSM8K-Aug-NL) and additional baselines (PCCoT, MUX) (nrEQ, EzHf)

Following the reviewers' suggestions, we trained on GSM8K-Aug-NL, which has the same problems as GSM8K-Aug but longer, natural-language traces. We also added PCCoT and MUX as baselines on Llama-3.2-1B. Longer traces lower accuracy for every method, including Explicit CoT (−5.6pp). Echo remains the strongest latent method (49.28), beating PCCoT (47.08) and MUX (45.56) by 2–4pp. CSQA and PyX use even longer, free-form rationales than GSM8K, and Echo’s advantage is largest on exactly these tasks: it surpasses Explicit CoT in four of six settings.

4. Echo’s contribution is its supervision target (nrEQ, EzHf)

Echo's contribution lies in how the latent slots are supervised: at each training step, the slots are matched to a newly sampled, chronologically ordered subset of the hidden states along the model's own CoT. We detail how this supervision differs from that of prior latent reasoning methods in our response to Reviewer nrEQ ("Comparison to related work"). The ablations in Fig. 4 (GPT-2, GSM8K) empirically support that each property of this target matters. We took the reviewer’s feedback to re-organize the related work and our method section to clarify our contributions from what we adopted from prior work.

5. Clarification on Training and inference cost (KRrq, EzHf)

  • Training. Echo needs no separate teacher and trains in a single stage, making it cheaper end-to-end than baseline latent reasoning methods (Table R2 in our response to Reviewer KRrq).
  • Inference. Echo's speed-up over Explicit CoT grows with reasoning length, from 2.3× on GSM8K to 63× on PyX, and Echo lies on the accuracy–efficiency Pareto frontier in every panel of Fig. 3 and Fig. S3 except GSM8K FLOPs. Echo’s dense supervision is effective: it allows Echo to achieve strong performance with a substantially smaller latent workspace than LOTUS (6–10 vs. 150–180), lowering latency by 5.2–16.2% and GFLOPs by up to 12× (Section 4.3; Tables R1 and R4 in our response to Reviewer EzHf).

Could you please let us know whether these explanations and new experiments address your concerns? We are happy to answer any further questions and, if they are already addressed, we would be grateful if you would consider updating your assessment.

Again, thank you to all reviewers for their time and constructive feedback.

AuthorsOpenReview ↗

Summary of Discussion and Additional Experiments

We sincerely thank all reviewers for their thoughtful feedback and engagement throughout the discussion. Building on our earlier general response, we would like to briefly highlight the main outcomes of the additional experiments and discussions.

Robustness, generalization, and scalability (KRrq, nrEQ, EzHf).

We substantially expanded our evaluation with 3 random seeds for the main experiments, 4 datasets (including the new GSM8K-Aug-NL with longer natural-language reasoning traces), and models up to 3B. Echo consistently achieves strong results across these settings. Notably, on GSM8K, Echo's improvement over CODI grows from +1.2pp at 1B to +6.8pp at 3B, providing encouraging evidence of scalability.

Broader baseline comparisons (nrEQ).

Following reviewer suggestions, we added PCCoT and MUX. Echo remains the strongest latent method on GSM8K-Aug-NL, further supporting the effectiveness of its hidden-state supervision.

Understanding the supervision signal (EzHf).

We conducted additional controlled comparisons and reduced-data experiments to better isolate the contribution of Echo's supervision. On GSM8K, Echo maintains improvements over CODI under reduced training data, with larger gains as the dataset grows. This is consistent with the intuition that increased data provides broader coverage of the CoT hidden-state distribution, suggesting an interesting direction for future theoretical investigation.

We are also grateful that Reviewer EzHf raised their score from 2 (Weak Reject) to 3 (Weak Accept) following our initial responses, and continued to provide insightful questions that helped us further examine the method's strengths and limitations.

We sincerely appreciate all reviewers' time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.