- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
This paper proposes a geometry-aware training method for latent reasoning LLMs to enhance their latent chains. Using SVD, the authors decompose a latent chain into a principal direction and an orthogonal subspace. They suggest that the two capture different aspects of reasoning: the principal direction relates to the main logical form, and the orthogonal subspace to semantic transitions. Based on this decomposition, they propose an auxiliary objective that aligns the latent chain with the principal direction of the language CoT and uniformly expands it in the orthogonal subspace. Experiments across different models and benchmarks show that the proposed training method outperforms prior latent SFT methods.
Critical strengths
The observations provide compelling evidence that the latent chains of current methods converge toward a single direction.
The proposed method is clean, adds no inference cost, and can be employed in many latent reasoning models.
Critical weaknesses
- The paper lacks comparisons with other commonly used latent reasoning methods, such as Coconut [1] and PCCoT [2], although both are cited.
- On the harder settings, the performance is still worse than explicit CoT. For Qwen3-8B, even in non-thinking mode, the accuracy should be well above the 22.46 reported in this paper.
- The principal direction alignment only matches a single scalar (the proportion of the student’s variance explained by v_t), rather than directly aligning the directions themselves. As a result, the student’s principal direction is not guaranteed to match the teacher’s.
[1] Hao et al. Training Large Language Models to Reason in a Continuous Latent Space. COLM 2025.
[2] Wu et al. Parallel Continuous Chain-of-Thought with Jacobi Iteration. EMNLP 2025.
Questions for authors
N/A
3 comments
Response to Reviewer CrhC (2/2)
W2. Explicit CoT performance (continued)
Table. Revised CoT-SFT results on MATH and NuminaMath-CoT. Non-thinking mode is used for Qwen 3.
| CoT-SFT | MATH | NuminaMath-CoT | ||||
|---|---|---|---|---|---|---|
| Pass@1$\uparrow$ | Avg #{T} | EFF$\uparrow$ | Pass@1$\uparrow$ | Avg #{T} | EFF$\uparrow$ | |
| Llama 3.2-1B-Instruct (old) | 7.10 | 109.2 | 0.07 | 7.69 | 103.5 | 0.07 |
| Llama 3.2-1B-Instruct (revised) | 12.18 | 231.6 | 0.05 | 6.41 | 130.8 | 0.05 |
| Llama 3.1-8B-Instruct (old) | 23.80 | 157.7 | 0.15 | 20.51 | 143.7 | 0.14 |
| Llama 3.1-8B-Instruct (revised) | 36.04 | 498.1 | 0.07 | 19.23 | 565.2 | 0.03 |
| Qwen 3-0.6B (old) | 17.50 | 185.5 | 0.09 | 12.82 | 227.7 | 0.06 |
| Qwen 3-0.6B (revised) | 19.66 | 287.3 | 0.07 | 25.64 | 345.3 | 0.07 |
| Qwen 3-8B (old) | 22.46 | 205.0 | 0.11 | 15.38 | 236.1 | 0.07 |
| Qwen 3-8B (revised) | 54.10 | 398.1 | 0.14 | 33.33 | 442.5 | 0.08 |
We update the revised results in Table 2 and the detailed CoT-SFT settings in Appendix C.1, with text marked in blue, and explicitly states the performance gap between latent reasoning and CoT-SFT in the results analysis, instead of claiming a competitive accuracy.
W3. principal direction alignment design
Your observation is consistent with our intention: our aim is NOT to fully align the student's principal direction with the teacher's. Principal direction alignment matches only the student's spectral mass along the teacher's principal direction. This encourages the student latent states toward the direction associated with the main reasoning thread, while leaving room for exploration in orthogonal directions. Figure 5 shows its effect: in outcome-rewarding models, the student and teacher principal directions can be nearly orthogonal, with absolute cosine similarity close to zero. Our method substantially increases their absolute cosine similarity, indicating an effective principal direction alignment. We further clarify this purpose under the 1st aim in Section 3.2.
Follow-up on Our Revisions
Dear Reviewer,
Thank you again for your careful review. We have posted our responses and revised the paper to address the concerns you raised. As the discussion period draws to a close, we would value your thoughts on whether the revisions address your main concerns. We would be happy to clarify any remaining points.
Thank you for your time and consideration.
Response to Reviewer CrhC (1/2)
We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below. All section, figure, and table numbers below refer to our revised PDF.
We do not include a direct comparison with Coconut as it uses a different curriculum learning protocol rather than self-distillation, so it cannot be directly used to compare student losses in Table 2. Additionally, newer baselines, including CODI and SIM-CoT, consistently report better performance than Coconut.
The table below lists Coconut accuracies reported by different papers with similar training setups. We also include CODI results as a reference for accuracy variation across papers. Our results come from Tables 8 and 9; see Appendix C.2 for detailed analysis.
Table. Accuracy of Coconut on GSM8K and GSM-Hard. Best results are in bold.
PCCoT is a self-distillation-based latent reasoning model that uses Jacobi iteration to update all latent states in parallel at each iteration. We apply LGD to PCCoT to explore its effectiveness in parallel latent reasoning. Following PCCoT's implementation details, we do not use a projection MLP and use the Smooth-L1 loss for answer distillation. We use 24 latent states and 3 extra Jacobi iterations for both settings. Results are shown in Table 10, Appendix C.3. It shows that adding LGD improves Pass@1 and EFF on all three models.
Table 10. PCCoT results on MATH. Best results are in bold.
For latent vs. explicit CoT: we acknowledge that the difficulty of keeping latent reasoning (reasoning entirely through hidden states without generating any explicit reasoning text) competitive with explicit CoT is a major and widespread challenge in the field. Very few existing latent reasoning approaches outperform explicit CoT across settings [1]. Such a challenge may stem from the substantial compression of reasoning chains, as well as differences in the scope, depth and technical maturity of the two research fields. Therefore, our contribution is that the proposed geometric decomposition provides an interpretable way to improve latent reasoning models; we do not claim that it surpasses explicit CoT.
For Qwen 3-8B results: thank you for pointing this out. It prompts us to revisit our CoT-SFT evaluation settings, where we find an important oversight: CODI's default
max_new_tokens=256was kept when we introduced CoT-SFT. As a result, the reasoning text for some long and difficult math problems was truncated at 256 tokens. We also used the same default CoT-SFT settings for Qwen 3 and Llama 3, while some of these settings did not match the recommended format for Qwen 3's mathematical reasoning. Specifically, (1) the question, CoT and answer was directly concatenated, whereas Qwen prefers a template in Chat Markup Language; (2) the prompt template encourages to directly output the answer while Qwen prefers the\boxed{}tag. Together, these settings led to underreported CoT-SFT results.We correct these issues and report the revised CoT-SFT results below. After correcting the evaluation, Qwen3-8B CoT-SFT in non-thinking mode reaches 54.10% Pass@1 on MATH and 33.33% on NuminaMath-CoT, up from 22.46% and 15.38%; Llama models also show Pass@1 improvements in most cases. However, CoT-SFT's token efficiency remains very low across models, indicating that its inference still requires many forward passes. So this does not affect our central conclusions or the importance of our method.
(to be continued in Part 2)
[1] Chen et al. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. EMNLP 2026