- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a region-based view of LLM reasoning. Instead of modeling reasoning behaviors with a single global linear direction, the authors partition the latent space into local regions using a Mixture of Factor Analyzers (MFA) and represent each reasoning trace as a trajectory through these regions. Across multiple reasoning models and benchmarks, they find that different reasoning behaviors tend to occupy different regions, while local behavior directions can differ substantially from global ones. Causal interventions further show that some regions are critical for correct reasoning, and promoting selected regions can improve accuracy and reduce reasoning length.
Critical strengths
-
The paper challenges the common assumption that complex reasoning behaviors are represented by a single global linear direction, and provides evidence that local regions and local directions better capture the geometry of reasoning activations. This is a conceptual contribution to mechanistic interpretability.
-
Causal interventions show that region traversal matters for reasoning performance. The paper goes beyond correlation by rerouting and promoting latent regions, identifying load-bearing regions whose removal substantially hurts accuracy and beneficial regions whose promotion can improve accuracy and reduce reasoning length. This strengthens the practical relevance of the proposed framework.
Critical weaknesses
-
The evidence does not clearly isolate behavior-specific local structure. Figure 3 compares different classification settings, and most gains against the global rest persist after shuffling episode labels within regions. It is therefore unclear whether higher AUROC reflects better behavioral separation or simply region membership.
-
The reported performance gains are difficult to interpret because some baselines are very weak under the chosen inference setting. The experiments impose a 2048-token thinking budget, and, e.g., Qwen3-32B achieves only 25.7% accuracy on MATH before intervention. The evaluation leaves the source of performance gains and the reliability of baseline comparisons unclear.
Questions for authors
-
Can global and local directions in Figure 3 be evaluated on identical held-out examples to separate behavior-specific discrimination from region membership?
-
Why are some baseline accuracies substantially lower than commonly reported numbers for the same models, e.g., Qwen3-32B on MATH and GPQA? Since all experiments use a 2048-token thinking budget, could the authors report whether the intervention gains persist under a larger reasoning budget?
-
Given the limited behavioral specificity of interventions, what operational meaning is assigned to “semantic units,” and what evidence supports this interpretation beyond statistical association?
Dear Reviewers,
As the rebuttal period is drawing to a close, we kindly ask everyone to engage with our comments. We have addressed the questions and critiques and would appreciate any feedback going forward. Thanks for your time and attention,
The Authors