- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces FORCE and REMOVE sanity checks to examine whether reasoning-based hallucination detectors rely on reasoning traces or final-answer cues. It also proposes TRACT, a lightweight detector using lexical and structural trajectory features. Experiments across multiple models and benchmarks reveal substantial endpoint sensitivity in existing detectors and demonstrate TRACT's competitive performance.
Critical strengths
The paper raises a meaningful evaluation question by investigating whether reasoning-based hallucination detectors genuinely leverage intermediate reasoning traces or rely primarily on final-answer signals. The proposed sanity checks offer a simple and interpretable framework for examining the sources of detector performance.
Critical weaknesses
-
Insufficient Motivation and Questionable Problem Importance. The paper assumes that reliance on final-answer information constitutes an undesirable shortcut, but does not establish why such reliance is problematic for hallucination detection. Final-answer consistency is a legitimate and informative signal, and the paper fails to demonstrate the practical benefits of excluding it.
-
Unfair Evaluation of Existing Methods. Several baselines, particularly RACE, explicitly incorporate answer uncertainty and reasoning-answer alignment. Removing or modifying final answers necessarily disrupts their intended functionality. The authors should conduct component-level ablations to distinguish legitimate answer-based signals from superficial shortcuts.
-
Flawed Experimental Design. FORCE replaces generated answers with ground-truth answers while preserving the original reasoning and correctness labels. This introduces artificial inconsistencies and creates an unrealistic evaluation setting. Consequently, performance changes cannot be reliably attributed to dependence on superficial answer cues.
-
Limited Methodological Novelty and Weak Feature Justification. TRACT relies on manually designed lexical and structural features, such as hedging words, step lengths, and punctuation patterns. The paper provides insufficient theoretical or empirical justification that these features genuinely reflect reasoning correctness rather than writing style.
-
Insufficient Evidence of Reasoning Faithfulness. Stability under FORCE and REMOVE does not demonstrate that a detector evaluates logical reasoning. TRACT is designed to ignore final-answer announcements, making its invariance largely guaranteed by construction. The authors should introduce controlled logical-error interventions and process-level correctness annotations.
-
Limited Generalization Evidence. TRACT's handcrafted features may depend heavily on model-specific writing styles, prompt templates, and task formats. The paper lacks systematic cross-prompt and out-of-distribution evaluations to establish whether its findings generalize beyond the tested configurations.
Questions for authors
NA