- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors propose Echo, a method to train a model to reason with continuous vectors without needing per step supervision. The paper presents a simple approach on using sampled hidden states of explicit CoT to supervise the latent representations. Experiments using GPT-2, LLaMA-1B on math, commonsense and coding tasks establish that the method is competitive with existing methods. Authors present detailed ablations to show the importance of various components of the approach.
Critical strengths
- Importance of the Problem and Simplicity of the approach.
The core problem of requiring annotated steps for latent supervision is neatly sidestepped by using sampled hidden states of CoT. Ablations on different methods of sampling are helpful and interesting. Authors try out multiple possible options but find that the simplest
Random Sortedapproach works the best.
Critical weaknesses
-
Inference Latency Comparison. The paper uses GFLOPs as the main measure of inference compute but it is not specified how it is calculated for different methods. Since the final model to be used for inference is similar for all the considered methods (a backbone -> hidden state looped -> answer) it is unclear why there is a variation among the methods. Accuracy numbers should be accompanied by allocated latent steps during training. Moreover, Inference Latency comparison is only presented with explicit CoT, when compared to other latent reasoning baselines it remains similar. Authors are requested to give a detailed account of inference compute and latency when compared to other latent reasoning methods.
-
Seed Variance and Evaluation Datasets. Considering that the core approach involves training on randomly sampled hidden states, authors should provide details on(or lack of) training seed variance on evals, as some selections might result in sub-optimal training. Further, this concern would be more applicable on longer problems. It is well established by prior works that latent reasoning methods work well (even better than CoT) on non-math tasks [CODI, CoCoNUT]. Therefore, atleast one evaluation on a slightly longer math dataset (GSM8k-Aug-NL for example) would remove any suspected regression on those tasks.
Score Update
Inference Latency Comparison.
Authors explain the Inference-time algorithms for methods and present the required table containing number of recurrent steps, latent slots and accuracy. The inference gains result from an improved source of supervision on LOTUS's inference scheme. I encourage the authors to include a more detailed description of the method and add Table R4 to the appendix.
Seed Variance and Evaluation datasets
Authors provide results with multiple seeds and a larger model and the improvements are consistent and the seed variance is also low, suggesting potential reliability of the supervision signal. Experiments on GSM8K-Aug-NL also show improvement w.r.t baselines showing that the method can work harder and longer traces.
As my major concerns are resolved I have raised my score to 3.
Questions for authors
- Are you using different latent reasoning budget for different methods?
- If yes, could you provide a detailed table with columns as (method, number of latent slots, number of recurrent steps, acc on gsm, acc on csqa, acc on pyx).
- if not, why are flops and inference latencies so different for different methods?
Update
Authors provide the answers and my concerns regarding these questions are appropriately resolved.
12 comments
Continued response
5. Are latent budgets matched?
In latent reasoning models, there are two axes to consider for the latent budget: the number of latents generated / processed for each forward pass () and the number of sequential forward passes (). The number of sequential forward passes decides the computational depth of the model. Thus, following LOTUS, we standardize the number of sequential forward passes and use the optimal configuration for recommended in each paper (Table R3).
- Coconut, CODI: generate one continuous token per step, sequentially (6 steps).
- LOTUS: 6 latent blocks of width c = 25 or 30 (150–180 latents in total), refined in parallel over R = 6 passes.
- Echo: K = 6–10 slots, refined in parallel over R = 6 passes.
Table R4: Latent budget and accuracy (Llama-1B-Instruct). Best latent reasoning model in bold.
| Method | Total latent slots (GSM8K / CSQA / PyX) | Sequential forward passes | GSM8K | CSQA | PyX |
|---|---|---|---|---|---|
| Explicit CoT | – | ~18 / 85 / 461 (one per token) | 59.77 | 70.43 | 67.93 |
| Coconut | 6 | 6 | 34.72 | 67.49 | 55.42 |
| CODI | 6 | 6 | 53.15 | 73.90 | 63.00 |
| LOTUS | 150 / 180 / 150 | 6 | 49.76 | 72.10 | 63.28 |
| Echo | 10 / 10 / 6 | 6 | 54.33 | 74.64 | 64.75 |
6. Why do FLOPs and latency differ across methods?
Although all latent methods share the same backbone, they process different numbers of latent slots. FLOPs scale with the total number of slots processed across all steps, while latency scales mostly with the number of sequential forward passes: parallel latent slots (LOTUS, Echo) are computed in the same forward call, so they add FLOPs but relatively little latency. This is why LOTUS (up to 180 latents) uses up to 12× more FLOPs than Echo (6 - 10 latents) at similar latency. It is also why Coconut and CODI, which process a single slot per step, use the fewest FLOPs.
Thank you for your detailed response and additional experiments. My concerns regarding variation with random seeds (3) and potential degradation on GSM8k-Aug-NL(4) are now resolved. Authors' explanation regarding GFLOPs(2) and flops/latency connection(6) also clarifies a lot of details.
I still have concerns regarding (1) and (5) which I outline below.
As I understand, ECHO's core contribution is training the latents by self-distilling randomly sampled, chronologically ordered hidden states from model's own CoT into its latent slots. This should only improve improve latent quality and downstream performance. Inference GFLOPs and latency, are set by the number of latent slots and sequential forward pass the algorithm uses.
Could you describe the inference difference (algo, GFLOPs and/or latency) between LOTUS(c=1, K=6) and ECHO (K=6, R=6)? I think a short explanation here would confirm the source of efficiency gains.
Response to reviewer
We thank the reviewer for their thoughtful and constructive feedback. We were glad the reviewer appreciated how our simple approach neatly sidesteps the need for annotated reasoning steps, and the comprehensive ablations on sampling strategies.
The reviewer raised a few questions asking to compare Echo’s inference latency with other latent reasoning methods, explain how FLOPs are computed and how we set latent budgets across methods, and report seed variance and results on longer math data. We address each of these points below.
Echo lies on the accuracy–efficiency Pareto frontier in every panel of Fig. 3 (GPT 2) and Fig. S3 (Llama 1B) except GSM8K FLOPs. Echo's efficiency advantage grows with reasoning length. Going from GSM8K to PyX, the explicit-CoT reasoning trace grows from ~17 to ~460 tokens (Table S2). Explicit CoT latency grows 27× (114 → 3,032 ms), while Echo’s stays flat (49 → 48 ms), so its speed-up grows from 2.3× to 63×. Echo is also faster than LOTUS on both datasets while being more accurate. Coconut and CODI are faster on GSM8K but around 7–20pp less accurate. On PyX, Echo is faster than CODI (47.8 vs. 56.6 ms) with similar accuracy.
We updated the manuscript to add this more detailed comparison to other latent reasoning models in Section 4.3.
Table R1: Inference latency (Llama-1B-Instruct, batch size 1, one H200).
GFLOPs are computed analytically for the full inference pass: question prefill, latent computation, and answer decoding. Latency is wall-clock time for the same pass. Both are averaged over the test set with batch size 1 on one H200, so padding does not affect either. Fig. S2 (Appendix D.1) shows that Echo’s speed-up grows further with larger batch sizes.
As per your recommendation, we ran our experiments and updated the manuscript with:
Also, we’d like to note that CommonsenseQA and PyX are also longer reasoning datasets, whose reasoning is ~67 and ~460 tokens respectively (compared to GSM8K at ~17 tokens).
Below, we report the average performance across 3 seeds, comparing Echo with the top-scoring baseline on each dataset. Echo is competitive on 8 of the 9 settings shown below, and maintains generalization to OOD math datasets (full results in Table 2).
Table R2: In-distribution accuracy (mean ± std over 3 seeds), Echo vs. the top-scoring baseline.
We also added GSM8K-Aug-NL and two new baselines PCCoT and MUX (suggested by Reviewer nrEQ), for Llama-1B. We are also running Llama-3B experiments and will update the reviewer in a couple of days. GSM8K-Aug-NL contains the same problems as GSM8K-Aug but with longer language-based reasoning. We draw two observations:
a) Natural-language traces are harder for every method at 1B. Moving from GSM8K-Aug to GSM8K-Aug-NL lowers accuracy across the board, including Explicit CoT (−5.6pp).
b) Importantly, Echo remains the strongest latent method on GSM8K-Aug-NL, outperforming both the newly added baselines by 2-4pp.
Table R3: GSM8K test accuracy (%) after training on GSM8K-Aug or GSM8K-Aug-NL. Bold marks the best latent method.