- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Reasoning Change-Points (RCP), a training-free approach for identifying the first erroneous step in LLM reasoning by detecting shifts in hidden-state representations. The authors demonstrate that artificially injected errors yield more distinct hidden-state transitions than naturally occurring errors, and introduce verification conditioning to amplify these signals. Experiments across multiple models and mathematical reasoning benchmarks show improved first-error localization, with confidence-based selection and targeted regeneration explored as downstream applications.
Critical strengths
- The proposed approach is training-free and is relatively straightforward to understand and implement.
Critical weaknesses
-
Limited novelty and missing baselines. Prior work has already explored hidden-state-based step-level correctness detection (e.g. https://arxiv.org/abs/2511.06209), as well as step-level uncertainty quantification (https://arxiv.org/pdf/2502.11250), and resampling at detected error positions (https://arxiv.org/abs/2507.14958) for reasoning models. The proposed approach seems to primarily combine existing ideas, but lacks direct comparisons with these related methods. Without such comparisons, it is difficult to assess its practical advantages over potentially simpler alternatives. Comparisons with some standard Process Reward Models (PRMs) would also provide useful reference points.
-
Limited experimental evaluation. Experiments rely on off-policy reasoning traces (e.g., ProcessBench). It remains unclear whether the method generalizes to other model families, domains, or errors in the model's own generations.
Questions for authors
-
Most importantly, could the authors compare against relevant hidden-state probing and uncertainty-based verification methods mentioned in the Weaknesses section? I see that the proposed method is training-free, but I would argue that requiring a small amount of training data is not necessarily a major practical limitation, so a comparison with change-point detection based on hidden-state probing would be beneficial.
-
Since the method already uses verification conditioning, could the authors report how accurately the backbone LLM judges the correctness of its own reasoning steps? This would provide another useful baseline for the method.
-
What motivates the specific weighting 0.4P+0.3L+0.3D? The choice seems somewhat arbitrary.
-
It would be valuable to include on-policy evaluations, where models detect errors in their own reasoning, as well as experiments across additional domains. Detecting errors in another model's reasoning may differ from detecting one's own mistakes.