acceptodds
Withdrawn from ICLR 2027

Sanity Checks for Reasoning Based Hallucination Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hallucination detection methods for large language models increasingly operate on chain-of-thought reasoning traces, yet it remains unclear whether they evaluate the reasoning itself or merely exploit surface correlates of the final answer. We introduce a controlled-invariance methodology that exposes this distinction through two oracle tests: Force, which replaces each response's final answer with the ground truth while preserving the reasoning trace, and Remove, which strips answer-announcement steps while leaving the trajectory intact. This reveals if their predictive power derives from answer-level artifacts rather than from the structure or validity of intermediate reasoning. We further show that once these artifacts are controlled for, effective discrimative power does still exist within the trace itself. To verify, we introduce TRACT, a lightweight scorer built on explainable lexical trajectory features (hedging trends, step-length dynamics, and cross-response vocabulary convergence), verifying that traces still contain important discrimative signatures, while remaining competitive with or outperforming existing baselines on unperturbed traces. These findings suggest that the current central challenge in reasoning-aware hallucination detection is not the absence of signal in the trace, but the failure to isolate it from endpoint cues.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Language, text generation, summarization, VQA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nzsCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FORCE and REMOVE sanity checks to examine whether reasoning-based hallucination detectors rely on reasoning traces or final-answer cues. It also proposes TRACT, a lightweight detector using lexical and structural trajectory features. Experiments across multiple models and benchmarks reveal substantial endpoint sensitivity in existing detectors and demonstrate TRACT's competitive performance.

Critical strengths

The paper raises a meaningful evaluation question by investigating whether reasoning-based hallucination detectors genuinely leverage intermediate reasoning traces or rely primarily on final-answer signals. The proposed sanity checks offer a simple and interpretable framework for examining the sources of detector performance.

Critical weaknesses
  1. Insufficient Motivation and Questionable Problem Importance. The paper assumes that reliance on final-answer information constitutes an undesirable shortcut, but does not establish why such reliance is problematic for hallucination detection. Final-answer consistency is a legitimate and informative signal, and the paper fails to demonstrate the practical benefits of excluding it.

  2. Unfair Evaluation of Existing Methods. Several baselines, particularly RACE, explicitly incorporate answer uncertainty and reasoning-answer alignment. Removing or modifying final answers necessarily disrupts their intended functionality. The authors should conduct component-level ablations to distinguish legitimate answer-based signals from superficial shortcuts.

  3. Flawed Experimental Design. FORCE replaces generated answers with ground-truth answers while preserving the original reasoning and correctness labels. This introduces artificial inconsistencies and creates an unrealistic evaluation setting. Consequently, performance changes cannot be reliably attributed to dependence on superficial answer cues.

  4. Limited Methodological Novelty and Weak Feature Justification. TRACT relies on manually designed lexical and structural features, such as hedging words, step lengths, and punctuation patterns. The paper provides insufficient theoretical or empirical justification that these features genuinely reflect reasoning correctness rather than writing style.

  5. Insufficient Evidence of Reasoning Faithfulness. Stability under FORCE and REMOVE does not demonstrate that a detector evaluates logical reasoning. TRACT is designed to ignore final-answer announcements, making its invariance largely guaranteed by construction. The authors should introduce controlled logical-error interventions and process-level correctness annotations.

  6. Limited Generalization Evidence. TRACT's handcrafted features may depend heavily on model-specific writing styles, prompt templates, and task formats. The paper lacks systematic cross-prompt and out-of-distribution evaluations to establish whether its findings generalize beyond the tested configurations.

Questions for authors

NA

Reviewer_hVyQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tackles the problem of hallucination detection for (long) reasoning in LLMs by analysing the role played by the reasoning trace itself. The contribution is twofold:

  1. it introduces two sanity-check operators, claimed to verify whether a hallucination-detection method actually exploits the trace itself: FORCE replaces the final answer with the corresponding ground truth, whilst REMOVE deletes the answer-announcement steps;
  2. it introduces TRACT, a lightweight scorer built on common-sense heuristics (intuitions), partially drawn from the specialised literature. FORCE and REMOVE are used to test TRACT and seven other baselines, supporting the claims that (i) existing detectors shift substantially under FORCE and REMOVE; (ii) TRACT remains stable under both, whilst being competitive on unmodified traces; and (iii) TRACT adds information when combined with the baselines (Appendix D). As such, TRACT would improve over the other baselines in both performance and reliability.
Critical strengths

The main strengths of the paper lie in the ideas it brings to the field of hallucination detection.

  1. A timely perspective on a real and under-explored problem. The paper introduces an interesting perspective on hallucination detection, which is timely and relevant for the field, focusing on trace analysis, a topic comparatively neglected in the literature. In doing so, it exposes a real potential problem worth investigating, namely whether state-of-the-art detectors claiming to analyse the reasoning trace actually do so. Relatedly, one of the paper's take-home messages, which in the words of the abstract reads "the current central challenge in reasoning-aware hallucination detection is not the absence of signal in the trace, but the failure to isolate it from endpoint cues", is relevant and worth pursuing.
  2. An interpretable feature inventory. The formalisation of the TRACT feature inventory (Table 1) and its design are interesting and, albeit based on previous literature, offer an interpretable and clear view.

However, the claims are currently not substantiated in a convincing manner (see Critical Weaknesses).

Critical weaknesses

The reviewer leans towards a strong rejection, as the claims do not appear to be substantiated by how the components are actually designed, and the tests seem tailored to favour TRACT's reading. In particular, TRACT's invariance appears to hold by construction rather than as a confirmation, and FORCE injects a signal tied to the labels, which might interfere with the final classification rather than abstract the answer away. Hence, the paper's main claim is unsupported, and cannot be repaired without redesigning the tests. The two main identified weaknesses follows.


TRACT's invariance under FORCE and REMOVE holds by construction, and partly breaks where the tests are non-trivial.

Specifically, under FORCE every response has the same final answer, so answer-agreement methods such as exact-match repetition drop to chance AUC by definition; although this is disclosed by the authors, these methods are still presented as failures overall (Fig. 2, EMR and NSN; §4.1, NSN). Furthermore, FORCE and REMOVE seem to confirm TRACT's construction rather than prove that it captures the actual reasoning behind the answer. Indeed, on BBH-Tracking and CausalT5K-D4, TRACT's scores are unchanged between unmodified traces and either intervention, as expected. However, the invariance breaks on the two datasets where the answer plausibly also appears within the reasoning body: Table 3 reports a mismatch between FORCE and REMOVE on GPQA Diamond and MATH-500, contradicting the claim that TRACT's "FORCE and REMOVE inputs coincide" (§4). Moreover, on MATH-500 TRACT drops substantially for some of the models with respect to unmodified traces (T column of Tables 11–15 vs Table 3; e.g., GPT-OSS-120B from 81.3 to 69.5, and Nemotron-30B from 76.4 to 62.5 under REMOVE). By the paper's own diagonal criterion (Fig. 2), TRACT therefore also relies on answer-level content wherever such content is not confined to the announcement step. In the reviewer's opinion, this is consequential, and it is so since the evidence offered does not establish that FORCE and REMOVE separate reasoning-faithful from answer-reliant detectors: where TRACT passes, it does so by construction, and where the tests are non-trivial, it partly fails them as well.


FORCE injects a new, label-correlated artefact.

In fact, replacing the final answer with the gold label creates a mismatch between the reasoning and the answer, which might be captured mostly (but not only) in incorrect traces (e.g., in Fig. 1, step 3 uses "DE=10" whilst the answer reads "DE=√51"). This mismatch is a new artefact injected by FORCE, which detectors checking reasoning–answer agreement would naturally exploit. Table 3 seems consistent with this reading, with a number of AUCs falling well below chance under FORCE but not under REMOVE, mostly for RACE, the only baseline with an explicit reasoning–answer alignment component: e.g., 24.5 vs 59.0 (Gemma, GPQA Diamond), 28.0 vs 76.0 (Nemotron, BBH-Tracking), and 36.5 vs 70.0 (Gemma, MATH-500). Notably, an AUC of 24.5 does not indicate an absence of signal, but rather a strong, sign-inverted one; conversely, §4.1 reads it as a failure of the detector. The same problem might also affect correct traces, particularly on MATH-500, where a correct answer under sound reasoning might in practice be scored as a hallucination if the reasoning reaches the answer through a different path (or perhaps under a different variable name) with respect to the gold label. Clearly, the latter is only a possibility, which the reviewer cannot verify, as the code is not released and the "canonicalization" step of FORCE is never specified (Appendix B.5 documents only REMOVE). A score shift under FORCE hence conflates dependence on the answer with sensitivity to the injected mismatch, and cannot be read as evidence of answer reliance alone. In the reviewer's opinion, this calls for a redesign of the intervention rather than a more detailed discussion.

Questions for authors

The reviewer main questions are regards the critical weaknesses highlighted above.

W1. TRACT's invariance under FORCE and REMOVE holds by construction, and partly breaks where the tests are non-trivial.

Can the authors provide formal or empirical evidence that FORCE and REMOVE separate reasoning-faithful from answer-reliant detectors? How do you reconcile the highlighted results on TRACT?

W2. FORCE injects a new, label-correlated artefact. How do the author reconciliate the current design of FORCE, or which redesign might avoid the highlighted problem, and how this changes the results presented in the paper?


Further questions follows

Clarification requests

  1. Unit of evaluation. Every TRACT feature aggregates over the K samples, so, per the reviewer's reading, TRACT yields one score per prompt rather than one score per response; yet §4 states that "responses maintain their original label" and that "performance is based on the initial response", which suggests that the statistics are evaluated per response. Could the authors clarify the unit of evaluation?

  2. Selection of the evaluated traces. §4 states: "We restrict evaluation to sets of sampled traces where the final answer is no longer a valid separator between incorrect and correct responses, ensuring that variation in scores arises from differences in reasoning structure, rhetorical coherence, and cross-trace consistency rather than answer disagreement." This is a key point on how the data are chosen, yet the impact of this choice remains mostly unexplored; as such, the reviewer is not able to determine whether the comparison is really fair. Relatedly, neither the hallucination rate per prompt nor the size of the overall set is clearly presented. Specifying these details and analysing how the chosen criteria might affect a real deployment would help the overall understanding of the contribution.

  3. Specification of FORCE. The "canonicalization" step of FORCE is never specified, as Appendix B.5 documents only REMOVE (cf. Critical Weakness 2). The reviewer advice specifying this as well.


Suggestions

  1. Overconfident hallucinations. TRACT, like part of the literature presented in §2, is not discussed with respect to the problem posed by overconfident hallucinations, which might hinder many of the heuristics/intuitions presented in Table 1 (particularly for the Content features). This problem is particularly relevant at low sampling temperature (e.g., T=0.3), where the responses tend to be more correlated. Analysing the results in light of the hallucination rates and of the coherence of the responses would greatly increase the reliability of the method.

  2. Cost/gain analysis in K. TRACT requires multiple samples over the same prompt, which might hinder its applicability. The authors should perform a more complete scalability analysis of the cost/gain in the parameter K, also comparing with the other baselines. The current framing at K=3 and K=5 does not provide a full view of the issue, with no confidence intervals, no per-benchmark breakdown, and no baselines at the same K (per the reviewer's reading, only K=10 is run for the other baselines). Furthermore, offering a view also on greater values of K would give a better picture of the overall behaviour of the proposed model, including whether a plateau in performance is reached and/or whether more response might support/hinder the methodology.


Further remarks

  1. Positioning of TRACT. The paper claims that "TRACT is not meant to be a state of the art detector, but to validate that true reasoning discrimination is possible; the challenge is therefore not only to design stronger detectors, but to evaluate whether their strength comes from the reasoning process they are meant to assess." Yet the remainder of the paper misleads on this point, as the main message conveyed is that TRACT is indeed evaluated against such competitors (e.g., "[TRACT is] competitive with or outperforming existing baselines" in the abstract, "TRACT has the highest number of faithful settings under both interventions" in Fig. 2, and "TRACT outperforms the best baseline on both metrics" in Appendix C.2). The authors should choose a line of presentation, either posing tract as a competitor, or building it as a (non-competitive) tool to enhance other methods performance (i.e. making the results from appendix D more central).

  2. Reproducibility. No reproducibility artefact is provided, hence the reviewer could not verify any of the reported numbers or data.

  3. Novelty. The contribution appears mostly incremental, since all the intuitions behind TRACT are already present, albeit scattered, in the literature (cf., e.g., Vanhoyweghen et al. (2025), cited by the authors). A more detailed discussion of when the adopted heuristic are gathered from the literature (and from where), would help the fair assessment of the novelty.

Reviewer_Qpb5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether hallucination detectors that work on chain-of-thought traces really evaluate the reasoning, or mainly use the final answer. The authors propose two oracle tests: FORCE replaces the final answer of every response with the ground truth, and REMOVE deletes the answer-announcement steps, both keeping the reasoning body. They apply them to seven black-box uncertainty scorers (including RACE and semantic entropy variants) on BBH-Tracking, GPQA Diamond, MATH-500 and CausalT5K with five LLMs, and show that the AUC of several scorers changes a lot. They also propose TRACT, a lightweight scorer built from lexical trajectory features (hedging trends, step-length dynamics and cross-sample vocabulary divergence) that does not use the answer, and show that it is stable under both tests and competitive in AUC. A supervised fusion of TRACT with each baseline improves over TRACT alone.

Critical strengths
  1. The question is reasonable and the tests are simple. Checking whether the score of a detector survives when the answer is controlled is a useful sanity check, and Table 3 and Figure 4 give some clear examples (e.g., RACE drops from 76.0 to 55.0 on BBH-Tracking with Gemma-3, and NSN is at chance under FORCE but reaches 77.5 under REMOVE on MATH-500 with GPT-OSS-120B).
  2. The fusion experiment (Table 4) is a good addition. It shows that TRACT carries information that the existing scorers do not have, which supports the claim that the trace itself contains signal. The paper is generally well written.
Critical weaknesses
  1. Part of the comparison is not fair to the baselines. The authors say the tests "should be interpreted in light of each detector's intended objective", but then use them to say that detectors are not "trace-faithful". Methods such as RACE are designed to use both the reasoning trace and the answer (reasoning-answer alignment), so removing or replacing the answer breaks the signal they were built to use, and for such methods a drop under FORCE does not show a flaw. For answer-based baselines like EMR, FORCE is a guaranteed failure (the paper says they are "not expected to pass"). At the same time, TRACT removes the answer-announcement steps in its own preprocessing, so its FORCE and REMOVE columns are identical by design (stated in Section 4), and "TRACT has the highest number of faithful settings" (Figure 2) follows from the construction. FORCE also puts the ground-truth answer next to a reasoning body that may reach a different conclusion for wrong responses, so for alignment-based scorers it is not label-neutral. The tests are fine as diagnostics, but the conclusion that current detectors are "less trace-faithful than standard evaluations suggest" is stronger than what they show.
  2. The practical value of a trace-only detector and the evaluation setting are unclear. In real use, the answer is available and is a legitimate signal for hallucination detection, so a trace-only score matters mostly if it adds something on top of answer-based signals; the fusion in Table 4 is the right experiment for this, but it reports the best of seven partners for each model and the gain over TRACT alone, not over the best baseline or the baselines fused together. The benchmarks (BBH-Tracking, GPQA, MATH-500, CausalT5K-D4) are also different from the ones used in the hallucination detection papers it compares with (e.g., RACE), which makes the numbers hard to compare. Many entries in Table 3 have standard errors of 5 to 20 points (e.g., Nova-Pro on BBH-Tracking), so a part of the differences may not be significant, and the parameters of TRACT (µ = 28, σ = 50, the feature signs and the equal weights) seem to have been chosen with these data, which I could not confirm in the main text.

Overall, the paper is reasonable and the sanity-check idea is useful, but some experiment settings are not fully fair and the claims should be toned down. I lean toward weak acceptance.

Questions for authors
  • How were µ, σ and the feature signs in TRACT chosen, and on which data? Are the 10 folds in Table 3 used for anything other than computing the standard error?
  • Can you report the fusion gain over the best baseline alone and over the fusion of the baselines without TRACT?
  • Would the conclusions hold on the standard hallucination detection benchmarks used by RACE?
  • For FORCE, how do alignment-based detectors behave when the inserted answer contradicts the reasoning body, and is this effect separated from the "answer artifact" effect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.