- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes LDIR, a local direction-informed Riemannian metric for analyzing LLM reasoning trajectories. It combines VAE-based latent representations with local token-transition statistics to extract geometric features for predicting reasoning correctness. Experiments across multiple models on MATH500 show modest improvements over existing geometric baselines, with additional evaluation on MGSM.
Critical strengths
Interesting perspective. The paper provides an interesting geometric perspective on LLM reasoning trajectories by incorporating local transition directions into the analysis of hidden states.
Critical weaknesses
-
Limited practical significance. LDIR achieves only modest improvements over existing geometric baselines, and the gain on MGSM is not statistically significant. Given the complexity of the proposed metric, its practical benefits remain unclear.
-
Limited evaluation on realistic reasoning errors. The experiments mainly rely on artificially perturbed reasoning trajectories rather than naturally occurring errors. It is unclear whether the identified geometric patterns generalize to real-world reasoning failures.
Questions for authors
-
Given the relatively small performance gains over existing geometric baselines, could the authors provide further evidence that the added complexity of LDIR is justified?
-
How well does LDIR generalize to naturally occurring reasoning errors rather than artificially perturbed trajectories? Could the authors evaluate it on unmodified model-generated reasoning traces?
Official Review by Reviewer 68rH
The authors didn't provide a rebuttal, so I will stay with my original score.