acceptodds
Under review as a conference paper at ICLR 2027

Geometric Decomposition of Reasoning Logic in Latent Spaces

Abstract

Latent chain-of-thought (CoT) reasoning methods train LLMs to reason with continuous latent states, reducing language CoT tokens to a small fixed budget. However, many latent reasoning methods rely solely on *outcome-rewarding objectives* which mainly supervise whether the latent chain reaches an answer state while giving little guidance to the reasoning space itself. In this paper, we study latent reasoning from the perspective of *global reasoning geometry*. Based on compact singular value decomposition, we introduce a **geometric decomposition** of the latent reasoning chain, separating the reasoning space into the principal direction and the orthogonal subspace. We empirically connect the two geometric objects with different aspects of reasoning logic: the principal direction is associated with the core logical form, while the orthogonal subspace is associated with semantic transitional logic in reasoning. Furthermore, we propose a **geometric decomposition-based auxiliary objective** for distillation-based latent reasoning. It aligns the latent reasoning chain with the language CoT's principal direction, and uniformly expands the latent reasoning chain in the orthogonal subspace. We theoretically and empirically show that outcome-rewarding models can lead to ill-conditioned reasoning as intermediate states converge to an "ideal exit" in the reasoning space, which harms the reasoning ability on difficult problems. In contrast, our geometric decomposition-based objective alleviates the latent chain convergence, expands the orthogonal subspace and provides richer reasoning information for answer decoding. Experiments on Llama 3 and Qwen 3 models show 12.7%-35.5% average Pass@1 relative improvements over outcome-rewarding methods on difficult mathematical reasoning settings, validating its effectiveness and geometric interpretability. Code is available at https://anonymous.4open.science/r/gdF451 .

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, SFT, latent reasoning, representational geometry, interpretability, logic

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CrhCOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper proposes a geometry-aware training method for latent reasoning LLMs to enhance their latent chains. Using SVD, the authors decompose a latent chain into a principal direction and an orthogonal subspace. They suggest that the two capture different aspects of reasoning: the principal direction relates to the main logical form, and the orthogonal subspace to semantic transitions. Based on this decomposition, they propose an auxiliary objective that aligns the latent chain with the principal direction of the language CoT and uniformly expands it in the orthogonal subspace. Experiments across different models and benchmarks show that the proposed training method outperforms prior latent SFT methods.

Critical strengths

The observations provide compelling evidence that the latent chains of current methods converge toward a single direction.

The proposed method is clean, adds no inference cost, and can be employed in many latent reasoning models.

Critical weaknesses
  1. The paper lacks comparisons with other commonly used latent reasoning methods, such as Coconut [1] and PCCoT [2], although both are cited.
  2. On the harder settings, the performance is still worse than explicit CoT. For Qwen3-8B, even in non-thinking mode, the accuracy should be well above the 22.46 reported in this paper.
  3. The principal direction alignment only matches a single scalar (the proportion of the student’s variance explained by v_t), rather than directly aligning the directions themselves. As a result, the student’s principal direction is not guaranteed to match the teacher’s.

[1] Hao et al. Training Large Language Models to Reason in a Continuous Latent Space. COLM 2025.

[2] Wu et al. Parallel Continuous Chain-of-Thought with Jacobi Iteration. EMNLP 2025.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer CrhC (1/2)

We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below. All section, figure, and table numbers below refer to our revised PDF.

W1. Comparison with Coconut and PCCoT

We do not include a direct comparison with Coconut as it uses a different curriculum learning protocol rather than self-distillation, so it cannot be directly used to compare student losses in Table 2. Additionally, newer baselines, including CODI and SIM-CoT, consistently report better performance than Coconut.

The table below lists Coconut accuracies reported by different papers with similar training setups. We also include CODI results as a reference for accuracy variation across papers. Our results come from Tables 8 and 9; see Appendix C.2 for detailed analysis.

Table. Accuracy of Coconut on GSM8K and GSM-Hard. Best results are in bold.

ModelSource paperGSM8K, Llama 3.2-1B-InstructGSM-Hard (OOD), Llama 3.2-1B-InstructGSM8K, Llama 3.1-8B-Instruct
CoconutCoconut----43.6
CoconutCODI45.39.9--
CoconutSIM-CoT33.27.0--
CODICODI55.612.8--
CODISIM-CoT52.711.961.1
CODIOurs52.612.759.0
LGD\mathcal{L}_{\text{GD}}Ours34.6--44.5
CODI + LGD\mathcal{L}_{\text{GD}}Ours55.613.062.8

PCCoT is a self-distillation-based latent reasoning model that uses Jacobi iteration to update all latent states in parallel at each iteration. We apply LGD\mathcal{L}_{\text{GD}} to PCCoT to explore its effectiveness in parallel latent reasoning. Following PCCoT's implementation details, we do not use a projection MLP and use the Smooth-L1 loss for answer distillation. We use 24 latent states and 3 extra Jacobi iterations for both settings. Results are shown in Table 10, Appendix C.3. It shows that adding LGD\mathcal{L}_{\text{GD}} improves Pass@1 and EFF on all three models.

Table 10. PCCoT results on MATH. Best results are in bold.

Llama 3.2-1B-Instruct⟶\longrightarrowLlama 3.1-8B-Instruct⟶\longrightarrowQwen 3-8B⟶\longrightarrow
Pass@1$\uparrow$EFF$\uparrow$Pass@1$\uparrow$EFF$\uparrow$Pass@1$\uparrow$EFF$\uparrow$
PCCoT + answer distill7.700.3214.040.5826.241.09
PCCoT + LGD\mathcal{L}_\text{GD}9.86 ($\uparrow$2.16)0.41 ($\uparrow$0.09)15.74 ($\uparrow$1.70)0.66 ($\uparrow$0.08)27.30 ($\uparrow$1.06)1.14 ($\uparrow$0.05)

W2. Explicit CoT performance

For latent vs. explicit CoT: we acknowledge that the difficulty of keeping latent reasoning (reasoning entirely through hidden states without generating any explicit reasoning text) competitive with explicit CoT is a major and widespread challenge in the field. Very few existing latent reasoning approaches outperform explicit CoT across settings [1]. Such a challenge may stem from the substantial compression of reasoning chains, as well as differences in the scope, depth and technical maturity of the two research fields. Therefore, our contribution is that the proposed geometric decomposition provides an interpretable way to improve latent reasoning models; we do not claim that it surpasses explicit CoT.

For Qwen 3-8B results: thank you for pointing this out. It prompts us to revisit our CoT-SFT evaluation settings, where we find an important oversight: CODI's default max_new_tokens=256 was kept when we introduced CoT-SFT. As a result, the reasoning text for some long and difficult math problems was truncated at 256 tokens. We also used the same default CoT-SFT settings for Qwen 3 and Llama 3, while some of these settings did not match the recommended format for Qwen 3's mathematical reasoning. Specifically, (1) the question, CoT and answer was directly concatenated, whereas Qwen prefers a template in Chat Markup Language; (2) the prompt template encourages to directly output the answer while Qwen prefers the \boxed{} tag. Together, these settings led to underreported CoT-SFT results.

We correct these issues and report the revised CoT-SFT results below. After correcting the evaluation, Qwen3-8B CoT-SFT in non-thinking mode reaches 54.10% Pass@1 on MATH and 33.33% on NuminaMath-CoT, up from 22.46% and 15.38%; Llama models also show Pass@1 improvements in most cases. However, CoT-SFT's token efficiency remains very low across models, indicating that its inference still requires many forward passes. So this does not affect our central conclusions or the importance of our method.

(to be continued in Part 2)

[1] Chen et al. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. EMNLP 2026

AuthorsOpenReview ↗

Response to Reviewer CrhC (2/2)

W2. Explicit CoT performance (continued)

Table. Revised CoT-SFT results on MATH and NuminaMath-CoT. Non-thinking mode is used for Qwen 3.

CoT-SFT⟵\longleftarrowMATH⟶\longrightarrow⟵\longleftarrowNuminaMath-CoT⟶\longrightarrow
Pass@1$\uparrow$Avg #{T}↓\downarrowEFF$\uparrow$Pass@1$\uparrow$Avg #{T}↓\downarrowEFF$\uparrow$
Llama 3.2-1B-Instruct (old)7.10109.20.077.69103.50.07
Llama 3.2-1B-Instruct (revised)12.18231.60.056.41130.80.05
Llama 3.1-8B-Instruct (old)23.80157.70.1520.51143.70.14
Llama 3.1-8B-Instruct (revised)36.04498.10.0719.23565.20.03
Qwen 3-0.6B (old)17.50185.50.0912.82227.70.06
Qwen 3-0.6B (revised)19.66287.30.0725.64345.30.07
Qwen 3-8B (old)22.46205.00.1115.38236.10.07
Qwen 3-8B (revised)54.10398.10.1433.33442.50.08

We update the revised results in Table 2 and the detailed CoT-SFT settings in Appendix C.1, with text marked in blue, and explicitly states the performance gap between latent reasoning and CoT-SFT in the results analysis, instead of claiming a competitive accuracy.

W3. principal direction alignment design

Your observation is consistent with our intention: our aim is NOT to fully align the student's principal direction with the teacher's. Principal direction alignment matches only the student's spectral mass along the teacher's principal direction. This encourages the student latent states toward the direction associated with the main reasoning thread, while leaving room for exploration in orthogonal directions. Figure 5 shows its effect: in outcome-rewarding models, the student and teacher principal directions can be nearly orthogonal, with absolute cosine similarity close to zero. Our method substantially increases their absolute cosine similarity, indicating an effective principal direction alignment. We further clarify this purpose under the 1st aim in Section 3.2.

AuthorsOpenReview ↗

Follow-up on Our Revisions

Dear Reviewer,

Thank you again for your careful review. We have posted our responses and revised the paper to address the concerns you raised. As the discussion period draws to a close, we would value your thoughts on whether the revisions address your main concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Reviewer_H9opOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies latent and text-based reasoning through the lens of SVD to the generated reasoning chains, finding that the first singular vector is less sensitive to semantic changes in the reasoning chain, which is then defined as the "main reasoning thread", compared to its orthogonal complement. Motivated by these observations, the paper proposes an auxiliary loss for training latent reasoning models that aligns latent student's latent reasoning trace with that of teacher's first singular vector and encourages uniform singular values for the student when projected to the teacher's orthogonal complement. Evaluated on math and commonsense reasoning tasks the auxillary loss leads to competitive performance compared to standard CoT while maintaining better efficiency.

Critical strengths
  1. The empirical results are strong, each term in the auxiliary loss seems helpful and the final method is often competitive with the CoT baseline while being more efficient.
  2. The SVD-based analysis is interesting and provides a solid motivation for the auxiliary loss
Critical weaknesses
  1. I found the paper hard to read and understand, while the motivation is largely understandable from the text, the related figures 1-4 are very information dense and it is not entirely clear what to read from them and how they support the claims. The suggested loss term, which seems to be the core essence of the work is detailed in a single condense page which is hard to read, while motivating sections span pages 3-6.
  2. The competitive empirical results, which I found surprising, motivate additional analysis and ablations which are missing.
Questions for authors
  1. It seems that the latent reasoning models, which all rely either on curriculum learning from the text or on various forms of distillation from a CoT-based teacher, seem to often surpass the performance of the teacher. e.g. The Latent LM method (not stated which method this is) achieves 27% accuracy compared to the teacher's 22.46% while using a fraction of the compute. This sounds improbable and warrants a suitable explanation that is not provided.
  2. The additional loss term does seem to help performance, did you consider if it may address existing limitations of latent reasoning methods? to name a few: allow adaptive computation instead of a fixed number of reasoning steps, reduce training duration, allow training without back propagation through time
6 comments
AuthorsOpenReview ↗

Response to Reviewer H9op (1/3)

We sincerely thank the reviewer for the comments and suggestions. We use "W" for weaknesses and "Q" for questions below. All section, figure, and table numbers below refer to our revised PDF.

W1. Presentation and readability

In the revised version, we have improved the figures, substantially expanded the captions and added detail to Section 3.2. These changes are marked in green, including:

  • Figure 1 (overview): The revised layout uses color and bold text to highlight the key contributions, and the caption links each part of the figure to its corresponding section.
  • Figure 2 (principal direction study): We add red annotations to highlight the key observations. The expanded caption explains that "the principal directions (green arrows) form nearly parallel clusters with high mean resultant lengths". Section 2.1 further explains how this observation supports our claim: "the principal vectors concentrate around a common direction, which can be connected with the reasoning logic shared within each group".
  • Figure 3 (orthogonal subspace study): We significantly expand the caption to read the statistics, e.g., "From the top 10% to the bottom 10%, transition pivots decrease from 40.6% to 3.3%, while inference pivots increase from 6.9% to 32.2%".
  • Figure 4 (latent chain convergence): The expanded caption explains that the insets show magnified views at the final epoch and that "the spectral masses are dominated by λ1\lambda_1 while λ2\lambda_2--λ5\lambda_5 are strongly suppressed".
  • Section 3.2 (geometric decomposition loss): We give more detail on the two aims and their design purposes, e.g., principal direction alignment "encourages the student latent states toward the direction associated with the main reasoning thread", while orthogonal subspace uniformization "encourages the student chain to explore more dimensions and counteracts spectral collapse". A new SFT procedure paragraph explains how LGD\mathcal L_{\mathrm{GD}} operates and how gradients are backpropagated during latent SFT.

Regarding the length of the motivation section: although our experiments mainly tell how the proposed geometric decomposition improves latent reasoning, we hope to highlight it as a tool for reasoning interpretation. We therefore regard the geometric decomposition (and its empirical connection to the reasoning logic) as a core contribution and devote the most important part of the paper to it.

W2. Additional analyses and ablations

We already provide several analyses and ablations of LGD\mathcal L_{\mathrm{GD}}, including: (1) Table 2 ablates its two aims separately and shows the performance benefit of combining them. (2) Figures 5, 14 and 15 show its positive effects on reasoning geometry to explain its better accuracy. (3) Figure 6 shows how it expands the orthogonal subspace of the latent chain and mitigates latent chain convergence. (4) Figure 12 examines how LGD\mathcal{L}_\mathrm{GD}'s performance vary with the loss coefficient and latent-chain length.

We further add the following analyses and ablations:

  1. We add a discussion in Appendix B.1 of two types of perturbed reasoning steps that may affect the geometric decomposition. (1) If a teacher CoT contains filler sentences unrelated to the main reasoning thread, such as "Okay, let's see." or "I think that's correct.", their latent states may lie far from the original reasoning subspace and alter the geometry of the chain. (2) Mathematical CoTs may contain numerical errors like "1+1=3" that leave the reasoning logic unchanged and may therefore be less apparent in the principal direction. We examine both cases in the geometric case study in Figure 9 of Appendix B.1. These observations motivate using concise and correct CoTs as supervision signals for our geometric decomposition objective.
  2. We add a new setting for getting latent states of a reasoning step: latent state of the last token in the sentence. We compare last token with mean pooling in terms of the principal direction and model performance in Table 11, Appendix C.5.

Table 11. Pass@1 on MATH series under different settings for the latent states of a reasoning step: mean pooling (MP) and last token (LT).

Llama 3.2-1B-Instruct⟶\longrightarrowLlama 3.1-8B-Instruct⟶\longrightarrow
MPLTMPLT
Latent-LM + answer distill8.027.58 ($\downarrow$0.44)6.406.20 ($\downarrow$0.20)
Latent-LM + LGD\mathcal{L}_{\mathrm{GD}}9.749.44 ($\downarrow$0.30)12.9212.58 ($\downarrow$0.34)

The last token setting performs worse in all four settings, with Pass@1 drops ranging from 0.20 to 0.44. To examine the reason in reasoning geometry, we also repeat the experiment in Figure 7(b) of Appendix B.1 using the last token setting. Figure 13 in Appendix C.5 shows that the last token setting produces less distinguishable latent chains and principal-direction clusters.

(to be continued in Part 2)

AuthorsOpenReview ↗

Response to Reviewer H9op (2/3)

W2. Additional analyses and ablations (continued)

  1. We add an experiment adapting LGD\mathcal L_{\mathrm{GD}} to the parallel latent reasoning architecture PCCoT in Appendix C.3, discussed in our response below.
  2. We add a principal direction study using Qwen3-8B in Figure 8 of Appendix B.1.

Q1. Teacher vs student performance

Thank you for pointing this out. It prompts us to revisit our CoT-SFT evaluation settings, where we find an important oversight: CODI's default max_new_tokens=256 was kept when we introduced CoT-SFT. As a result, the reasoning text for some long and difficult math problems was truncated at 256 tokens. We also used the same default CoT-SFT settings for Qwen 3 and Llama 3, while some of these settings did not match the recommended format for Qwen 3's mathematical reasoning. Together, these settings led to underreported CoT-SFT results.

We correct these issues and report the revised CoT-SFT results below. After correcting the evaluation, Qwen3-8B CoT-SFT in non-thinking mode reaches 54.10% Pass@1 on MATH and 33.33% on NuminaMath-CoT, up from 22.46% and 15.38%; Llama models also show Pass@1 improvements in most cases. However, CoT-SFT's token efficiency remains very low across models, indicating that its inference still requires many forward passes. So this does not affect our central conclusions or the importance of our method. Also, there are still cases when our model is competitive with the CoT baseline, e.g., on NuminaMath-CoT and CommonsenseQA.

Table. Revised CoT-SFT results on MATH and NuminaMath-CoT. Non-thinking mode is used for Qwen 3.

CoT-SFT⟵\longleftarrowMATH⟶\longrightarrow⟵\longleftarrowNuminaMath-CoT⟶\longrightarrow
Pass@1$\uparrow$Avg #{T}↓\downarrowEFF$\uparrow$Pass@1$\uparrow$Avg #{T}↓\downarrowEFF$\uparrow$
Llama 3.2-1B-Instruct (old)7.10109.20.077.69103.50.07
Llama 3.2-1B-Instruct (revised)12.18231.60.056.41130.80.05
Llama 3.1-8B-Instruct (old)23.80157.70.1520.51143.70.14
Llama 3.1-8B-Instruct (revised)36.04498.10.0719.23565.20.03
Qwen 3-0.6B (old)17.50185.50.0912.82227.70.06
Qwen 3-0.6B (revised)19.66287.30.0725.64345.30.07
Qwen 3-8B (old)22.46205.00.1115.38236.10.07
Qwen 3-8B (revised)54.10398.10.1433.33442.50.08

We update the revised results in Table 2 and the detailed CoT-SFT settings in Appendix D.1, with text marked in blue, and explicitly states the performance gap between latent reasoning and CoT-SFT in the results analysis, instead of claiming a competitive accuracy. The Latent-LM in Table 2 refers to the self-distillation-based latent reasoning model with textual LM loss for the teacher and Latent-LM loss for the student, as mentioned in Section 3.1; we further add this to the caption of Table 2.

We also note that, although almost all students underperform the teacher now, CoT-SFT is not a performance upper bound for Latent-LM in the self-distillation setting, since the student and teacher share parameters. We believe the main reasons latent reasoning models often perform worse are the substantial compression of the reasoning chain and the mismatch between explicit CoT pre-training and latent SFT, rather than the distillation architecture itself.

(to be continued in Part 3)

AuthorsOpenReview ↗

Response to Reviewer H9op (3/3)

Q2. Addressing other limitations

Thank you for these insights. Our primary aim of the auxiliary loss is to improve latent reasoning performance in a more interpretable way. To our knowledge, two major limitations of latent reasoning are (1) low performance, especially on long and difficult problems; and (2) limited interpretability of latent thoughts.

The limitations you raise are also very important. To offer some insights, we adapt LGD\mathcal{L}_{\mathrm{GD}} to PCCoT [1], a self-distillation-based latent reasoning model that uses Jacobi iteration to update all latent states in parallel at each iteration. The results are shown in Table 10 of Appendix C.3.

Table 10. PCCoT results on MATH. Best results are in bold.

Llama 3.2-1B-Instruct⟶\longrightarrowLlama 3.1-8B-Instruct⟶\longrightarrowQwen 3-8B⟶\longrightarrow
Pass@1$\uparrow$EFF$\uparrow$Pass@1$\uparrow$EFF$\uparrow$Pass@1$\uparrow$EFF$\uparrow$
PCCoT + answer distill7.700.3214.040.5826.241.09
PCCoT + LGD\mathcal{L}_\text{GD}9.86 ($\uparrow$2.16)0.41 ($\uparrow$0.09)15.74 ($\uparrow$1.70)0.66 ($\uparrow$0.08)27.30 ($\uparrow$1.06)1.14 ($\uparrow$0.05)

Adding LGD\mathcal{L}_{\text{GD}} improves Pass@1 and EFF on all three models. This adaptation also provides insights into the three limitations. (1) Although PCCoT does not implement adaptive computation, a meaningful future direction is to introduce an early-exit mechanism from looped models, e.g., using a cumulative distribution function to calculate exit probability [2] or distances between successive steps [3] to decide when to stop. As PCCoT decouples the number of latent states from that of iterations, it better fits the early exit mechanism for adaptive computation. (2) PCCoT reports a 45% training duration reduction with respect to sequential latent reasoning [1]. (3) Backpropagation through time is still used, but each iteration processes all latent states in parallel, so the backpropagation depth depends on the number of Jacobi iterations rather than the number of latent states.

[1] Wu et al. Parallel continuous chain-of-thought with Jacobi iteration. EMNLP 2025

[2] Zhu et al. Scaling latent reasoning via looped language models. arXiv:2510.25741

[3] Geiping et al. Scaling up test-time compute with latent reasoning: A recurrent depth approach. NeurIPS 2025

Reviewer_CgcfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies latent chain-of-thought reasoning, where text-based reasoning steps are replaced by a small number of continuous latent states. It proposes to decompose latent reasoning trajectories into a principal direction and an orthogonal subspace using compact SVD, and introduces an auxiliary training objective that aligns latent chains with language CoT geometry while expanding the orthogonal subspace. The method is evaluated on MATH, NuminaMath-CoT, CommonsenseQA, and GSM8K variants with Llama 3 and Qwen 3 models, using Pass@1, token count, token efficiency, and geometric analyses.

Critical strengths
  1. A useful part of the paper is that it makes latent reasoning trajectories measurable beyond final-answer accuracy. The proposed decomposition gives interpretable quantities such as principal-direction alignment and orthogonal-spectrum expansion.

  2. The training objective follows naturally from the paper’s geometric analysis. The two components of the objective correspond to preserving the main direction of language CoT and enriching the orthogonal subspace, and the ablation results help isolate their roles.

  3. The experiments cover several models and reasoning datasets. The results on Llama 3 and Qwen 3 models across mathematical and commonsense reasoning tasks provide reasonable evidence that the method can improve existing latent-reasoning baselines.

Critical weaknesses
  1. The link between the principal direction and “reasoning logic” still needs stronger evidence. The current experiments show a correlation with reasoning templates, but they do not fully establish that this direction captures the underlying logical structure in a necessary or causal sense.

  2. The gains are sometimes stronger in token efficiency than in absolute accuracy. In several settings, latent methods use far fewer reasoning tokens, but their Pass@1 remains below language CoT-SFT, so the paper should state more clearly whether the main contribution is compression, accuracy improvement, or both.

  3. The geometric findings may depend on analysis choices. The decomposition could be sensitive to latent-chain length, normalization, SVD/PCA choices, teacher CoT quality, or random seeds; more stability analysis would make the conclusions more convincing.

Questions for authors
  1. How stable are the principal-direction and orthogonal-subspace measurements across random seeds, latent-chain lengths, and alternative normalization or decomposition choices?

  2. In cases where LGD improves token efficiency but does not outperform language CoT-SFT in Pass@1, should the result be interpreted mainly as efficient compression of reasoning, or as improved reasoning ability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Cgcf (1/2)

We sincerely thank the reviewer for the positive feedback. We use "W" for weaknesses and "Q" for questions below. All section, figure, and table numbers below refer to our revised PDF.

W1. Link between the principal direction and reasoning logic

Our experiments support an association between the principal direction and reasoning templates. When questions share a template, changes to numbers, entities, and scenarios preserve the template's inference structure, which aligns with the logical form defined in logic [1]. We also compare different solution templates for the same questions: in our newly added Qwen3-8B results in Figure 8, the average angle separating the principal directions of the two template groups is approximately 20 degrees across the three question groups. We interpret these results as evidence of an association within the underlying logical structure. Similar inductive designs are also widely used in influential interpretability studies, such as constructing true/false sentences and examining whether their hidden states are linearly separable [2], and replacing sentiment tokens to identify sentiment directions [3]. Establishing a causal role would require intervention on the direction during reasoning, which is out of our research scope.

[1] Stanford Encyclopedia of Philosophy. Logical Form. 1999. https://plato.stanford.edu/entries/logical-form/

[2] Marks & Tegmark. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. COLM 2024

[3] Tigges et al. Linear Representations of Sentiment in Large Language Models. arXiv:2310.15154

W2 & Q2. Main contribution: compression vs accuracy

We further clarify this matter with Llama 3.2-1B-Instruct in Table 2 as an example. LGD\mathcal{L}_{\text{GD}} improves Pass@1 (from 8.30 to 9.74) over Latent-LM with only 6 latent states, and the Pass@1 of CoT-SFT is 12.18 with 231.6 reasoning tokens on average. The first comparison measures the gain from LGD\mathcal{L}_{\text{GD}} at a fixed latent budget. The second shows the accuracy cost of compressing the reasoning trace on this benchmark. We report Pass@1 alongside reasoning-token efficiency (EFF) to make this trade-off explicit.

W3 & Q1. Stability of geometric decomposition

We respond to your suggestions point-by-point. Relevant additions in the revised PDF are marked in purple.

Latent-chain length. Appendix B.1 presents our principal direction study results on MATH, where latent chains are longer than those on GSM8K (8–12 vs. 3–6 states). We observe a consistent pattern: principal directions are only mildly affected by shallow perturbations but change clearly with the main reasoning thread. For the orthogonal subspace study, Figures 3 and 10 already report statistics from a large number of CoT samples from different models, with text lengths ranging from 157 to 1,462 tokens. Figure 12(b) in Appendix C.4 further shows how model performance varies with latent-chain length.

Alternative normalization. We add a new normalization setting for latent states of a reasoning step: latent state of the last token in the sentence. We compare last token with mean pooling in terms of the principal direction and model performance in Table 11, Appendix C.5.

Table 11. Pass@1 on MATH series under different settings for the latent states of a reasoning step: mean pooling (MP) and last token (LT).

Llama 3.2-1B-Instruct⟶\longrightarrowLlama 3.1-8B-Instruct⟶\longrightarrow
MPLTMPLT
Latent-LM + answer distill8.027.58 ($\downarrow$0.44)6.406.20 ($\downarrow$0.20)
Latent-LM + LGD\mathcal{L}_{\mathrm{GD}}9.749.44 ($\downarrow$0.30)12.9212.58 ($\downarrow$0.34)

The last token setting performs worse in all four settings, with Pass@1 drops ranging from 0.20 to 0.44. To examine the reason in reasoning geometry, we repeat the experiment in Figure 7(b) of Appendix B.1 using the last token setting. Figure 13 in Appendix C.5 shows that the last token setting produces less distinguishable latent chains and principal-direction clusters.

SVD/PCA choices. Appendix E.1 has discussed the uniqueness of the SVD-derived principal direction in the majority of cases. Our conclusions are supported by mean resultant length (MRL) and cosine similarity measured directly in the high-dimensional space. PCA is used only for illustration of the space, and its choices do not affect these measurements.

(to be continued in Part 2)

AuthorsOpenReview ↗

Response to Reviewer Cgcf (2/2)

W3 & Q1. Stability of geometric decomposition (continued)

Teacher CoT quality. We add a discussion in Appendix B.1 of two types of perturbed reasoning steps that may affect the geometric decomposition: filler sentences and numerical errors. (1) If a teacher CoT contains filler sentences unrelated to the main reasoning thread, such as "Okay, let's see." or "I think that's correct.", their latent states may lie far from the original reasoning subspace and alter the geometry of the chain. (2) Mathematical CoTs may contain numerical errors like "1+1=3" that leave the reasoning logic (e.g., the symbolic structure of the calculation) unchanged and may therefore be less apparent in the principal direction. We examine both cases in the geometric case study in Figure 9 of Appendix B.1. These observations motivate using concise and correct CoTs as supervision signals for our geometric decomposition objective.

Random seeds. The geometric findings in Section 2 are measured directly on official pretrained models and do not require stochastic decoding. So our quantitative measurements including MRL and orthogonal displacement are insensitive to the random seed. We further add a principal direction study using Qwen3-8B in Figure 8 of Appendix B.1 as an additional reliability check of our findings.

Thanks for the response. The new Qwen3-8B results, the Table 2 clarification on the compression-vs-accuracy trade-off, and the added stability analyses (length, pooling, teacher CoT quality, seeds) cover my main concerns. I'll keep my positive score.

AuthorsOpenReview ↗

Response to Reviewer Cgcf

Thank you again for your positive feedback! We very much appreciate that.

Comments on the paper

2 comments
Area_Chair_o5s9OpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. The rebuttal is now available in the discussion forum. I would like to kindly remind you to check the rebuttal and leave comments for the authors, which would be very helpful.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

AuthorsOpenReview ↗

Summary of Revisions and Responses

Dear Area Chair and Reviewers,

We thank the reviewers for their thoughtful questions, constructive suggestions and positive comments. We appreciate their recognition including:

  • Our decomposition "makes latent reasoning trajectories measurable beyond final-answer accuracy" (Reviewer Cgcf);
  • Our "analysis is interesting and provides a solid motivation for the auxiliary loss" and the revised results are "indeed insightful as it can potentially provide additional supervision to latent reasoning methods" (Reviewer H9op);
  • Our method is "clean, adds no inference cost, and can be employed in many latent reasoning models" (Reviewer CrhC).

In response, we made the following main changes:

  1. Geometry and evidence (Reviewer Cgcf). We clarify that the connection between principal directions and reasoning logic is an empirical association. We add analyses of the Qwen 3-8B model, step pooling and perturbed teacher CoTs.
  2. Presentation and objective (Reviewers H9op and CrhC). We revise Figures 1–4 and their captions, expand Section 3.2 and explain the SFT procedure. We also clarify that principal-direction alignment matches spectral mass along the teacher's direction instead of fully aligning the two principal directions.
  3. Additional validation (Reviewers CrhC). We extend our LGD\mathcal{L}_{\text{GD}} method to the parallel training framework PCCoT and observe apparent and consistent improvements.
  4. CoT-SFT evaluation (Reviewers H9op and CrhC). We correct the CoT-SFT evaluation and update Table 2, which reveals a larger accuracy gap to explicit CoT. We have answered the remaining questions from reviewers' further response, and make the reproduction code and original result files available. The correction leaves LGD\mathcal{L}_{\text{GD}}'s gains over latent baselines at the same latent budget unchanged, thus this does not affect our central conclusions or the importance of our method. As also noted in the reviewer response: "Requiring latent reasoning to outperform slow, explicit thinking would be unreasonable".

We appreciate all reviewers' suggestions, and we hope our summary highlights the value of our paper and helps support favorable consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.