acceptodds
Under review as a conference paper at ICLR 2027

LDIR: A Local Direction-Informed Riemannian Metric for LLM Reasoning Trajectories

Abstract

Large language models (LLMs) generate reasoning as trajectories through high-dimensional hidden-state spaces, while these trajectories are commonly analyzed using fixed Euclidean geometry. Such a fixed geometry may fail to capture the nonlinear structure of learned representations and the directional organization of reasoning dynamics. We propose LDIR, a Local Direction-Informed Riemannian framework for analyzing chain-of-thought (CoT) trajectories under a geometry adapted to both representation structure and local reasoning directions. We first map hidden states to a latent space using a geometry-preserving variational autoencoder (VAE) and use its decoder to induce a pullback metric. We then refine this geometry using the local directional statistics of reasoning transitions, yielding a post-pullback anisotropic metric that captures local reasoning dynamics. We evaluate the resulting geometric features on controlled matched valid–invalid reasoning pairs from MATH500 across multiple language models and model families. LDIR consistently provides additional discriminative information over the corresponding pullback geometry induced by the Euclidean metric in hidden-state space, with particularly clear improvements across several Qwen models and additional gains in cross-model settings. Feature-family ablations and trajectory-length controls further indicate that the improvement is not attributable to a single geometric statistic or to reasoning-step count alone. Matched-pair analysis shows that invalid reasoning generally exhibits reduced trajectory movement, lower local speed, and reduced cumulative turning, suggesting that reasoning validity is reflected in the local geometry of hidden-state dynamics. Our code will be made publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Reasoning, chain-of-thought, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Dx5TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LDIR, a local direction-informed Riemannian metric for analyzing LLM reasoning trajectories. It combines VAE-based latent representations with local token-transition statistics to extract geometric features for predicting reasoning correctness. Experiments across multiple models on MATH500 show modest improvements over existing geometric baselines, with additional evaluation on MGSM.

Critical strengths

Interesting perspective. The paper provides an interesting geometric perspective on LLM reasoning trajectories by incorporating local transition directions into the analysis of hidden states.

Critical weaknesses
  1. Limited practical significance. LDIR achieves only modest improvements over existing geometric baselines, and the gain on MGSM is not statistically significant. Given the complexity of the proposed metric, its practical benefits remain unclear.

  2. Limited evaluation on realistic reasoning errors. The experiments mainly rely on artificially perturbed reasoning trajectories rather than naturally occurring errors. It is unclear whether the identified geometric patterns generalize to real-world reasoning failures.

Questions for authors
  1. Given the relatively small performance gains over existing geometric baselines, could the authors provide further evidence that the added complexity of LDIR is justified?

  2. How well does LDIR generalize to naturally occurring reasoning errors rather than artificially perturbed trajectories? Could the authors evaluate it on unmodified model-generated reasoning traces?

Reviewer_Q1DFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigate how to tell if a CoT ends in the right answer by measuring the “shape” of the model’s hidden-state path, token by token. To this end, It proposes LDIR, a metric on a VAE latent space that starts from the decoder-induced pullback metric and reweights it locally. The main experiment is done like this. On MATH500 problems, each paired with a valid trajectory and an invalid one created by injecting a localized error, each metric is assessed with AUROC, treating the correctness prediction as a binary prediction problem.

Critical strengths
  1. The method works on frozen hidden states without fine-tuning the LLM, and the metric itself is fit without validity labels.

  2. It needs only a single forward pass per trace, with no activation intervention or repeated sampling, unlike self-consistency or multi-sample uncertainty methods.

Critical weaknesses
  1. The paper is badly written. It's full of vague descriptions accompanied with unnecessary defensive claims. E.g. "LDIR achieves the highest mean AUROC among the compared geometries in all evaluated settings, with the clearest improvements in several Qwen settings; its small numerical gain on MGSM is statistically inconclusive." and "Overall, LDIR shows a consistent numerical advantage, although the gain varies across settings." ...

  2. LDIR assigns lower metric costs to frequently observed transition directions, but the paper provides no justification for why this transformation should reveal reasoning errors. Furthermore, the sign-invariant directional covariance captures unoriented axes rather than actual flow. Since the metric directly reweights displacements according to their alignment with these axes, improvements in speed-based features may reflect the imposed anisotropic scaling rather than a meaningful geometric signature of reasoning correctness.

  3. Turning and curvature treat differences between adjacent latent states as tangent vectors, despite having no evidence that token-level transitions are sufficiently small for this approximation to be reliable. Moreover, the turning calculation neglects the manifold connection, while endpoint efficiency uses a straight coordinate path rather than a geodesic. Consequently, the efficiency ratio can exceed one, and these features are not invariant under nonlinear changes of latent coordinates. Their values therefore depend on the arbitrary GeoVAE parameterization, undermining the paper's motivation for introducing Riemannian geometry in the first place.

  4. Every comparison is between three variants of their own pipeline, and HAP is nearly identical to EP throughout.

Questions for authors
Reviewer_68rHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LDIR, a local direction-informed Riemannian metric for analyzing token-level LLM reasoning trajectories. Hidden states are compressed using a geometry-regularized VAE; the decoder induces a pullback metric, which is then adjusted using local statistics of adjacent-token transition directions. Eleven geometric features describing movement, turning, curvature, and endpoint efficiency are used to predict final-answer correctness. On matched valid–invalid MATH500 trajectories, LDIR consistently outperforms the Euclidean pullback baseline by 0.49–2.16 AUROC points across several models. However, its external MGSM improvement is only 0.29 points and is not statistically significant.

Critical strengths
  1. The metric construction is technically principled. LDIR combines decoder-induced geometry with local transition anisotropy while preserving smoothness and positive definiteness.
  2. All learned components are fitted within grouped training folds, and the paper uses paired bootstrapping, multiple-comparison correction, and token-count controls.
  3. LDIR obtains the highest mean AUROC in all ten MATH500 settings i.e., constant gains, including different model families and trajectory sources.
Critical weaknesses
  1. The dataset is very small and artificially constructed. The primary evaluation contains only 121 problems, with invalid traces created through manually designed perturbations and retained only when they change the final answer.
  2. LDIR improves over EP by only 0.49–2.16 AUROC points, and four of ten comparisons are not significant after correction.
  3. I feel the external generalization is unconvincing. On MGSM, the gain is only 0.29 AUROC points and statistically insignificant. Moreover, all components are retrained on MGSM, so this does not test cross-dataset transfer.
  4. LDIR introduces several choices: GeoVAE compression, 64 prototypes, shrinkage, interpolation, spectral clipping, and conditioning strength, but sensitivity and component ablations are limited.
  5. The paper does not compare against strong controls using question-only states, initial hidden states, endpoint representations, or conventional hidden-state probes. Thus, it is unclear whether trajectory geometry adds information beyond simpler representations.
Questions for authors
  1. Does LDIR transfer across datasets without retraining the GeoVAE, prototypes, metric, or classifier—for example, when trained on MATH500 and directly evaluated on MGSM?
  2. How does LDIR compare with classifiers using the final hidden state, mean-pooled hidden states, question-only activations, or the same trajectory features computed directly in standardized hidden space?
  3. Does the advantage remain on naturally occurring incorrect generations, rather than invalid traces produced by controlled perturbations and outcome-based filtering?
  4. How sensitive are the results to latent dimension, number of prototypes, conditioning strength γ\gamma, shrinkage, and interpolation bandwidth?
1 comment
Reviewer_68rHOpenReview ↗

Official Review by Reviewer 68rH

The authors didn't provide a rebuttal, so I will stay with my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.