acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Endpoint: Hallucination Detection via Predictive-Path Geometry

Abstract

Large language models (LLMs) are increasingly deployed where factual reliability matters, making hallucination detection a pressing need. Existing detectors draw on output confidence, response consistency, and internal representations, yet the geometry of ordered intermediate predictions remains underexplored. We introduce Hierarchical Path Excess (HiPEx), a post-hoc white-box detector that characterizes predictive-path non-directness. HiPEx projects intermediate residual states into a shared vocabulary probability space and measures path excess—the two-step path length minus the direct endpoint distance. The same measure is applied across neighboring decoder blocks and within their attention–feed-forward transitions, yielding complementary layer-wise profiles. A lightweight probe scores these profiles, extracted in one teacher-forced pass through the frozen model without auxiliary generations. Across two instruction-tuned models and four question-answering benchmarks under BLEURT-20 labels, HiPEx outperforms all eight baselines in AUROC in every setting, with an average gain of 2.32 percentage points over the strongest baseline in each setting. Controlled ablations demonstrate the benefits of subtracting endpoint distance and combining both scales. Cross-benchmark transfer and alternative-label evaluations further support its robustness. These results highlight the value of predictive-path geometry for hallucination detection: the route to a prediction matters alongside its endpoint.

Primary area:
interpretability and explainable AI
Keywords:
hallucination detection, large language models, white-box detection, model interpretability, internal representations, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RaFSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hierarchical Path Excess (HiPEx), a supervised, post-hoc white-box method for response-level hallucination detection. It projects intermediate residual-stream states through the model’s final normalization and output head into a shared vocabulary probability space. Using normalized square-root Jensen–Shannon distance, it defines path excess as the two-step distance through an intermediate state minus the direct endpoint distance. This quantity is computed at two scales: across neighboring decoder-block outputs and within each block’s cumulative pre-attention, post-attention, and post-FFN states. The resulting layer-wise features are averaged over answer tokens and scored by a lightweight MLP. Feature extraction requires one teacher-forced pass over the completed response, without auxiliary generations.

The authors evaluate HiPEx on Llama-3-8B-Instruct and Gemma-2-9B-it across four QA benchmarks. Under BLEURT-20-derived answer-correctness labels, HiPEx achieves the highest AUROC in all eight model–benchmark settings, with a macro-average of 87.79%. The paper further investigates endpoint subtraction, scale complementarity, matched-budget and pooling controls, source-only cross-benchmark transfer, and alternative automatic and human answer–reference agreement labels.

Critical strengths
  1. A well-defined geometric feature supported by targeted ablations. HiPEx applies the same three-point path-excess functional to neighboring block outputs and to cumulative residual states along the executed attention–FFN computation. This provides a concrete computational interpretation and yields a compact representation that preserves layer coordinates at both scales. Section 4.3 reports a mean improvement of 1.63 AUROC percentage points over the stronger path-length or endpoint-distance control in each setting. Appendix E.2 additionally compares explicit subtraction with jointly supplied component features. The two-scale construction improves over the stronger parameter-matched single-scale variant by 2.10 points on average. These controls substantiate the particular feature design rather than merely showing that internal activations contain information about answer correctness.

  2. Consistent main-protocol gains and substantial experimental controls. HiPEx achieves the highest AUROC in all eight model–benchmark settings under BLEURT-20 labels. The evaluation uses shared candidate answers, training-only preprocessing, and paragraph-disjoint partitions for passage-grounded datasets. The authors also provide a matched-budget comparison with ICR Probe and an answer-token mean-pooling control for TriLens. HiPEx remains ahead of mean-pooled TriLens in all eight settings, by 2.44 AUROC percentage points on average. Appendix L reports significant paired improvements over ICR Probe and HaloScope in all 16 comparisons, while explicitly restricting the inference to the fitted models. The substantial human annotation study provides further positive evidence, with HiPEx leading three of four settings and achieving the highest macro-average AUROC among the four evaluated methods.

Critical weaknesses
  1. The relationship between the operational labels and substantive hallucination errors remains insufficiently characterized. Appendix B explicitly states that the primary BLEURT-20 labels measure answer–reference mismatch. This is a reasonable proxy for QA correctness, and its use does not invalidate the main comparison. However, it can combine different failure types. The human protocol in Appendix H.2 also measures answer–reference agreement and includes refusals, non-answers, and names without textual evidence of alias equivalence in the incorrect class. Consequently, the current evaluation does not reveal whether the advantage primarily concerns factual or contextual errors, failure to answer, or discrepancies arising from reference coverage.

    The human study is valuable: it retains 13,621 annotated examples and gives HiPEx a macro-average AUROC of 75.59%, with the best mean in three of four settings. Nevertheless, automatic–human label agreement, annotator agreement before majority voting, and performance by error type are not reported. The probes are also refitted on the smaller human-labeled subsets. This is a valid evaluation of the representation under another supervision target, but it does not isolate label effects from sample-size effects or directly assess the original BLEURT-trained detectors against human judgments. I therefore remain uncertain about which kinds of errors account for the reported gains. This uncertainty is consequential because hallucination detection is the paper’s central application.

  2. The comparative robustness evidence does not include the stronger configuration of the closest logit-lens baseline. Appendix M.2 shows that mean pooling improves TriLens by 6.26 AUROC percentage points on average. HiPEx still wins all eight BLEURT-20 comparisons, providing meaningful evidence for its additional value within that protocol. However, the human-label comparison in Appendix H.2 retains first-token TriLens, and the source-only transfer experiment in Section 4.5 compares only ICR Probe and HaloScope. Thus, the advantage over mean-pooled TriLens has not been established in either setting, even though these evaluations are important evidence for robustness beyond the main automated-label protocol.

    Transfer results also depend on the backbone: HiPEx leads 16 of 24 directions overall, while its mean off-diagonal AUROC on Llama-3 trails HaloScope by 1.12 points. The authors appropriately disclose this limitation; my concern is the remaining scope of comparative validation, rather than inaccurate reporting. Likewise, the paired significance analysis in Appendix L supports the specified ICR Probe and HaloScope comparisons, but does not establish the corresponding advantage over mean-pooled TriLens. Given that the contribution builds on existing internal-dynamics and logit-lens detection approaches, establishing its additional value against this stronger nearby configuration under human labels and transfer is important to my assessment of its significance.

Questions for authors
  1. Label agreement and error composition: On the human-annotated subsets, could you report the BLEURT–human confusion matrices and annotator agreement before majority voting? A targeted audit distinguishing factual/contextual errors from refusals, non-answers, and reference/alias discrepancies would also be helpful. Does HiPEx retain its comparative advantage when refusals and non-answers are analyzed separately from substantive answer errors?

  2. Separating label effects from sample-size effects: Could you evaluate the frozen BLEURT-trained detectors against human labels on the corresponding annotated test examples? For the refitted experiments, a comparison using the same retained examples, partitions, and training-set sizes under the two labeling schemes would help isolate the effect of the labels. I regard the existing refitting protocol as valid; these diagnostics would clarify what its robustness results establish.

  3. Comparative robustness: Could you evaluate mean-pooled TriLens under the human-label protocol and source-only cross-benchmark transfer, using the same partitions and comparable probe/training settings? Paired AUROC differences and confidence intervals for the human-label comparisons would be particularly informative. These results could materially change my assessment of whether the geometric representation provides a robust improvement over the closest existing approach.

4 comments
AuthorsOpenReview ↗

Label Reliability and Error Analysis (W1, Q1)

We sincerely thank you for your thoughtful review. Your feedback helped us examine label reliability and strengthen the empirical evidence for HiPEx under human-labeled and cross-benchmark settings.

1.Label Agreement and Error Composition (W1, Q1)

We appreciate your emphasis on the importance of characterizing both label reliability and error composition. Following your suggestion, we conducted additional analyses to examine the agreement between automatic and human labels, assess inter-annotator consistency, and evaluate HiPEx after excluding refusals and non-answers.

1.1 BLEURT–Human Label Agreement

We constructed confusion matrices by comparing BLEURT-20 labels with majority-vote human labels on the same 13,621 retained examples. As shown in table below, the overall agreement rate is 87.96% (11,981/13,621). Agreement varies across settings and is lower on Llama-3 TyDiQA (68.03%). This may be because TyDiQA is a passage-based QA task, where answers need to be matched against passage-level evidence. Longer, more context-dependent responses may introduce greater ambiguity in semantic matching, thereby leading to lower agreement between BLEURT and human judgments.

ModelDatasetBoth CorrectBLEURT Correct / Human IncorrectBLEURT Incorrect / Human CorrectBoth IncorrectTotal SamplesAgreement Rate
Llama-3TriviaQA3,4001502801,1484,97891.36%
Llama-3TyDiQA1,120505371291,83668.03%
Gemma-2TriviaQA3,5201102311,1184,97993.15%
Gemma-2TyDiQA1,380452371661,82884.57%
Total—9,4203551,2852,56113,62187.96%

1.2 Inter-Annotator Agreement

The unanimous agreement rates for the three annotators independently labeling each example are as follows.

SettingAll Three AgreeTwo Agree, One DisagreesUnanimous Agreement
Llama-3 TriviaQA4,55542391.5026%
Llama-3 TyDiQA1,39444275.9259%
Gemma-2 TriviaQA4,53144891.0022%
Gemma-2 TyDiQA1,48134781.0175%
总计11,9611,66087.8129%

1.3 Targeted Error-Type Audit

Due to time constraints, we were unable to verify and categorize all the data. To examine whether HiPEx relies primarily on refusal and non-answer signals, we randomly sampled 150 originally incorrect answers from each of the four model–benchmark settings with human labels (600 in total). We identified 268 factual errors, 30 contextual errors, 154 refusals/non-answers, and 148 reference/alias mismatches. After excluding the 154 refusals and non-answers, 446 candidate errors remained. For each seed, we evaluated the frozen detectors on a test subset comprising all correct answers in the original test partition and the intersection of these 446 non-refusal candidate errors with that same partition. As shown in the table below, HiPEx achieves the highest mean AUROC (65.94%).

MethodBefore ExclusionAfter Exclusion
ICR Probe0.66170.5946
HaloScope0.67170.6327
TriLens Mean0.69870.6289
HiPEx0.72780.6594

We are grateful for your suggestion, which prompted us to examine this potential confound more closely.

AuthorsOpenReview ↗

Label Controls(W1, Q2) and Comparative Robustness(W2, W3, Q3)

2.Label Effects from Sample-Size Effects (W1, Q2)

We appreciate your thoughtful suggestion to disentangle the effects of supervision labels from those of sample size. We agree that this distinction is important for interpreting the robustness of human-label evaluations. In response to your concern, we conducted two additional controlled experiments.

2.1 Evaluating Frozen BLEURT-Trained Detectors on Human Labels

We directly evaluated the detectors originally trained under BLEURT-20 supervision on the corresponding human-labeled test examples, without retraining or modifying their parameters. HiPEx achieves the highest AUROC of 69.22%.

MethodAUROC
HaloScope65.17%
ICR Probe64.98%
TriLens (First-token)66.83%
TriLens (Mean-token)68.67%
HiPEx69.22%

2.2 Matched-Sample Comparison under Alternative Supervision Labels To disentangle the effect of supervision labels from differences in sample composition and training-set size, we retrained all detectors on the same retained human-annotated subset, using identical train/validation/test partitions and matched training-set sizes. We evaluated two settings: training and testing with BLEURT-20 labels, and training and testing with human labels. As shown in the table below, HiPEx maintains the highest AUROC under both supervision schemes.

MethodBLEURT-trained / BLEURT-labeledHuman-trained / Human-labeled
HaloScope80.16%69.80%
ICR Probe82.34%70.94%
TriLens (First-token)78.56%71.39%
TriLens (Mean-token)84.55%74.88%
HiPEx85.17%75.59%

3.Comparative Robustness under Human Labels and Cross-Benchmark Transfer (W2, W3, Q3)

We appreciate your suggestion to evaluate HiPEx against the stronger mean-pooled TriLens configuration under both human-labeled and cross-benchmark settings. We agree that these comparisons are important for assessing the additional value and generalizability of our geometric representation. In response to your comments, we extended the evaluation to include mean-pooled TriLens in both settings and conducted paired statistical analyses.

3.1 Human-Label Evaluation with Mean-Pooled TriLens

We evaluated mean-pooled TriLens on the same five random seeds, data partitions, and human-annotated subsets used in Appendix H.2. As shown in the table, HiPEx achieves the highest macro-average AUROC of 75.59%, compared with 74.88% for mean-pooled TriLens, a gain of 0.71 percentage points. HiPEx achieves the highest mean AUROC in three of the four settings. These results demonstrate a modest overall advantage for HiPEx under human labels.

Model / BenchmarkTriLens (First)TriLens (Mean)HiPEx
Llama-3 / TriviaQA74.08 ± 1.5475.51 ± 2.8077.10 ± 1.24
Llama-3 / TyDiQA73.06 ± 5.4573.78 ± 6.0572.43 ± 6.43
Gemma-2 / TriviaQA67.21 ± 1.3774.91 ± 0.8576.16 ± 1.37
Gemma-2 / TyDiQA71.22 ± 5.2975.35 ± 2.8976.69 ± 4.89
Macro-average71.39 ± 1.4674.88 ± 1.9975.59 ± 2.25

3.2 BLEURT-Labeled Source-Only Cross-Benchmark Transfer

We extended Section 4.5 to include mean-pooled TriLens while retaining the same source-only protocol: each detector is fitted using source-benchmark training data and evaluated on target test data without target adaptation. On Gemma-2, HiPEx achieves an average off-diagonal AUROC of 80.44%, outperforming mean-pooled TriLens (79.31%) by 1.13 percentage points. However, on Llama-3, mean-pooled TriLens achieves 80.40%, exceeding HiPEx (77.16%) by 3.24 percentage points. Therefore, we acknowledge that HiPEx does not have a universal transfer advantage over mean-pooled TriLens.

ModelMethodIn-domainOff-diagonal
Llama-3HiPEx88.8777.16
Llama-3ICR Probe86.2271.49
Llama-3HaloScope86.0978.28
Llama-3TriLens (Mean)86.5080.40
Gemma-2HiPEx86.7280.44
Gemma-2ICR Probe82.9170.82
Gemma-2HaloScope75.9071.52
Gemma-2TriLens (Mean)84.2079.31
AuthorsOpenReview ↗

Comparative Robustness(W2, W3, Q3) and Summary of Revisions

3.Comparative Robustness under Human Labels and Cross-Benchmark Transfer (W2, W3, Q3)

3.3 Source-Only Transfer under Human-Label Supervision

We further evaluated cross-benchmark transfer using human answer–reference agreement labels on TriviaQA and TyDiQA across both backbones. We retained the same human-labeled partitions and used comparable probe-training protocols. For each transfer direction, detectors were trained exclusively on the source benchmark and evaluated on the target test partition. As shown in the table, HiPEx outperforms mean-pooled TriLens in three of the four cross-benchmark directions.We hope this additional evaluation addresses your concern regarding cross-benchmark generalization under human-label supervision.

ModelTransfer DirectionHiPExICR ProbeHaloScopeTriLens (Mean)
Llama-3TriviaQA → TyDiQA67.80 ± 4.8655.33 ± 11.5473.88 ± 2.7170.65 ± 4.19
TyDiQA → TriviaQA63.23 ± 2.8661.25 ± 4.1260.03 ± 2.4357.82 ± 1.56
Gemma-2TriviaQA → TyDiQA74.86 ± 5.2564.62 ± 7.1562.22 ± 8.3170.85 ± 3.67
TyDiQA → TriviaQA63.83 ± 4.2858.23 ± 2.7860.24 ± 9.2055.15 ± 1.66

3.4 Paired Statistical Comparison under Human Labels

To assess the reliability of the human-label comparison, we conducted paired bootstrap analyses of AUROC differences between HiPEx and mean-pooled TriLens on identical test examples. The table below reports the paired differences, 95% confidence intervals, and Holm-adjusted p-values across the four settings. HiPEx achieves higher mean AUROC in three settings and wins in three to five of the five seed-level comparisons, depending on the setting. Although none of the Holm-adjusted p-values reach the 0.05 significance level, HiPEx maintains a modest numerical advantage in three of the four settings. We therefore interpret these results cautiously and thank you for encouraging a more rigorous statistical comparison.

SettingHiPExTriLens (Mean)Δ95% CIpp_HolmWins
L-TriviaQA77.1075.51+1.59[-0.41, +3.63]0.12030.48124/5
L-TyDiQA72.4373.78-1.35[-6.25, +3.38]0.57761.00003/5
G-TriviaQA76.1674.91+1.25[-1.01, +3.55]0.27160.81475/5
G-TyDiQA76.6975.35+1.34[-3.31, +6.12]0.57621.00004/5

Summary of Revisions

No.Rebuttal ItemTarget Section
1W1/Q1 (1.1–1.2): Agreement between BLEURT-20 and human labels, and inter-annotator consistencyAppendix H.2
2W1/Q1 (1.3): Potential reliance on refusal and non-answer signalsAppendix H.2
3W1/Q2 (2.1): Effects of supervision labels independent of retrainingAppendix H.2
4W1/Q2 (2.2): Effects of supervision labels versus sample-size differencesAppendix H.2
5W2/W3/Q3 (3.1): Comparison with mean-pooled TriLens under human labelsAppendix H.2, Table 9
6W2/W3/Q3 (3.4): Statistical reliability of the human-label comparisonAppendix H.2
7W2/W3/Q3 (3.2): Comparison with mean-pooled TriLens under cross-benchmark transferAppendix I.2; Main text §4.5
8W2/W3/Q3 (3.3): Cross-benchmark transfer under human-label supervisionAppendix I.3

We are currently working to finalize the revised manuscript and will upload the updated version shortly. We sincerely appreciate the time and effort you have devoted to reviewing our responses, as well as your thoughtful consideration of our work.

Dear Reviewer,

Thank you for your time and thoughtful review. When you have a chance, we would be grateful if you could take a look at our response. Please let us know if any questions remain.

Best regards,

The Authors

Reviewer_kLdZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Path Excess (HiPEx), a hallucination detection method that captures the geometric properties of predictive trajectories across LLM layers. HiPEx projects intermediate hidden states into a shared vocabulary probability space and measures path excess by subtracting the direct endpoint distance from the two-step path length. It extracts both inter-block and intra-block features, which are combined and fed into an MLP classifier to predict hallucination risk. Experiments on two LLMs and four QA benchmarks demonstrate consistent improvements over existing baselines.

Critical strengths

1. Novel Approach Using Predictive-Path Geometry

Unlike existing methods that primarily rely on hidden states, entropy, or layer-wise prediction differences, this paper introduces predictive-path non-directness as a signal for hallucination detection. Combining inter-block and intra-block information provides a novel perspective on capturing internal prediction dynamics.

2. Comprehensive Ablation Studies

The paper conducts extensive ablation studies comparing path excess with path length and endpoint distance, examining the complementary effects of inter-block and intra-block features, and controlling for feature dimensionality and probe capacity. These experiments systematically demonstrate the effectiveness of the proposed feature design.

3. Consistent Performance Improvements and Robustness Evaluations

HiPEx achieves the highest AUROC across all eight model–benchmark settings. The paper further evaluates cross-benchmark transfer, alternative labeling criteria, and human-annotated labels, providing additional evidence of the method's effectiveness and robustness.

Critical weaknesses

1. Limited Explanation of the Relationship Between Path Excess and Hallucination

  • Although Figure 2 and Table 5 demonstrate that path excess is more effective than path length or endpoint distance, the paper provides limited insight into why predictive-path non-directness is associated with hallucination.

  • In Figure 3, the differences between hallucinated and correct answers appear relatively small, and their directions vary across layers.

  • Further analysis or qualitative examples would help clarify what characteristics the proposed features capture and how they distinguish hallucinated from correct answers.

2. Limited Validation of Logit Lens-Based Features

  • HiPEx relies on Logit Lens to project intermediate hidden states into vocabulary probability distributions. However, intermediate-layer projections may not faithfully reflect the model's actual predictive states, potentially introducing noise into the extracted features.

  • The paper does not sufficiently investigate how such projection uncertainty affects path excess or compare its approach with alternatives such as Tuned Lens.

  • Additionally, comparisons with features derived directly from layer-wise probability distributions could help establish the advantages of the proposed representation.

3. Lack of Distinction Between Different Sources of Incorrect Answers

  • The paper defines hallucination primarily based on the mismatch between generated and reference answers, without distinguishing whether the model produces an incorrect answer because it lacks the relevant knowledge or because it fails to use knowledge it already possesses.

  • Although alternative labeling criteria and human annotations are evaluated, these experiments do not explicitly account for the model's underlying knowledge state.

  • Consequently, it remains unclear whether HiPEx captures a general signal of incorrectness or specifically identifies hallucinations arising from failures in knowledge utilization.

Questions for authors

Motivation Behind Predictive-Path Non-directness

The paper argues that predictive-path non-directness provides useful information beyond endpoint distance. However, the motivation for why such non-directness should be associated with hallucination is not sufficiently clear. Could the authors provide more intuition or theoretical justification for this hypothesis? In particular, why should the predictive trajectories of hallucinated answers differ from those of correct answers?

7 comments
AuthorsOpenReview ↗

Geometric Motivation, Interpretation, and Empirical Evidence (W1, Q1)

We sincerely thank you for your careful review and thoughtful feedback. We are grateful for the opportunity to address your concerns.

1.Motivation and Interpretation of Predictive-Path Non-directness (Q1, W1)

We appreciate your thoughtful questions regarding why predictive-path non-directness may be associated with hallucination. We have therefore expanded our theoretical motivation and conducted additional analyses to clarify the nature and distribution of the discriminative information captured by HiPEx.

1.1 Theoretical Motivation

Existing studies have shown that hallucination-related information is reflected in intermediate computational dynamics, rather than solely in final predictions. Jiang et al. [1] identified distinct layer-wise prediction trajectories between correct and hallucinated responses, while Zhang et al. [2] demonstrated that residual-stream update dynamics provide informative signals for hallucination detection. Sun et al. [3] further linked reasoning fluctuations and error backtracking to reasoning hallucinations. More broadly, Damirchi et al. [4] showed that inter-layer geometric displacements can capture structural information beyond static representations.

These findings motivate our hypothesis that the distinction between correct and hallucinated answers may lie not only in how much their predictions change, but also in how consecutive predictive updates are geometrically coordinated. Specifically, two answers may exhibit similar net predictive changes while following different intermediate trajectories. As established in Appendix E.1, for fixed step lengths, Path Excess increases with directional disagreement between consecutive updates in the metric's implicit Hilbert-space embedding. Therefore, it provides a principled geometric descriptor of predictive-path non-directness, beyond prediction movement magnitude or endpoint displacement alone.

1.2 Interpreting Layer and Scale Dependent Differences

Regarding your observation about Figure 3, we emphasize that Path Excess should not be interpreted as a layer-wise hallucination score. Figure 3 reveals that class differences are dependent on both computational depth and scale, with their directions varying across layers. These patterns suggest that hallucination-related geometric information is not expressed through a uniform increase across layers. Instead, different computational stages exhibit distinct local geometric contrasts. This motivates preserving layer coordinates and combining both computational scales, rather than collapsing them into a single global magnitude.

1.3 Additional Empirical Evidence

Figure 2 and Appendix E.2 show that Path Excess consistently outperforms both path length and endpoint distance, including under matched dimensionality. Additionally, we conducted layer-wise classification and layer-count ablation experiments using logistic regression on Llama-3 TriviaQA. Single-layer features achieve limited performance (best single-layer AUROC: 0.6992), whereas the full 62-dimensional representation reaches 0.8348. Repeating the best single-layer or same-layer features to 62 dimensions does not improve performance. Furthermore, AUROC increases systematically as more genuine layer-wise features are incorporated under both random and contiguous layer selection, reaching 0.8348 with 30 selected layers. Together, these results indicate that discriminative information is distributed across multiple computational depths, rather than concentrated in a single layer or arising merely from redundant dimensional expansion.

Single-layer and redundancy controls

FeatureOriginal AUROCRepeated to 62 dimensions
Best single layer0.69920.6992
Layer-21 Inter0.66860.6686
Layer-17 Combined0.69740.6974
Full HiPEx0.8348—

Layer-count ablation

LayersRandom-kContiguous-k
10.62560.6256
20.68070.6655
40.72650.7061
80.76790.7614
160.80770.8157
300.83480.8348
Full (32)0.83480.8348

Overall, our results suggest that hallucination-related information is distributed across the layer-wise organization of predictive-path geometry. HiPEx exploits this information by preserving computational depth and both inter- and intra-block structures. We emphasize that these findings establish an empirically supported geometric association, while identifying the precise computational mechanisms underlying these differences remains an important direction for future work.

AuthorsOpenReview ↗

Logit-Lens Features and Knowledge Availability (W2–W3)

2.Validation of Logit-Lens-Based Features (W2)

Thank you for raising these concerns. We address Logit-Lens reliability and HiPEx’s design choices from three perspectives.

2.1 The objective of HiPEx does not require exact intermediate predictions

HiPEx does not assume that Logit Lens faithfully recovers intermediate predictive states. We apply shared final normalization and unembedding to intermediate residual states to characterize their predictive tendencies and relative geometry in a common vocabulary space, rather than seek the most accurate approximation of the final prediction.

2.2 Why Logit Lens rather than Tuned Lens?

We thank the reviewer for suggesting that we consider alternative projection methods. Our choice of Logit Lens is deliberate and aligns with the geometric objective of HiPEx.

Specifically, Logit Lens applies a shared frozen readout to each intermediate state:

qℓLL=softmax⁡(WULN⁡(hℓ)).q_\ell^{LL}=\operatorname{softmax}(W_U\operatorname{LN}(h_\ell)).

In contrast, Tuned Lens introduces layer-specific learnable affine transformations:

qℓTL=softmax⁡(WULN⁡(Aℓhℓ+bℓ)),q_\ell^{TL}=\operatorname{softmax}(W_U\operatorname{LN}(A_\ell h_\ell+b_\ell)),

which are trained to align intermediate predictions with the final output distribution but may alter the path geometry measured by HiPEx.

HiPEx applies the same path-excess functional to both inter-block outputs and intra-block post-attention states. Standard Tuned Lens targets block-level states and would require position-specific adaptation for intra-block states. Logit Lens therefore provides a consistent readout across both scales without additional learnable parameters.Our choice reflects this alignment with HiPEx’s geometric objective, rather than a claim of universally superior predictive accuracy.

2.3 Comparison with direct statistics of intermediate probability distributions

On TriviaQA with Llama-3 and Gemma-2, we compare HiPEx with entropy, maximum probability, and top-1 probability margin extracted at the same computational positions. We also evaluate their concatenation (Combined) and a PCA variant matched to HiPEx’s dimensionality (Combined-PCA).

MethodLlama-3 Input DimLlama-3 AUROCGemma-2 Input DimGemma-2 AUROC
Entropy6585.30±0.328582.05±0.57
MaxProb6584.16±0.228581.32±0.16
Margin6583.74±0.358580.68±0.38
Combined19586.16±0.7825583.29±0.47
Combined-PCA6284.84±0.248281.55±0.39
HiPEx6286.44±0.848282.91±0.43

The results show that HiPEx achieves comparable AUROC to the uncompressed Combined baseline while using only approximately one-third of its feature dimensions.

3.Distinguishing Different Sources of Incorrect Answers (W3)

Thank you for noting that answer–reference mismatch alone cannot distinguish knowledge unavailability from failures to use available knowledge.

To investigate this issue, we conduct an additional knowledge-availability-stratified evaluation using Llama-3 on TriviaQA and NQ-Open. Specifically, we generate ten auxiliary responses per question using stochastic decoding, following the original prompt template and the model's native chat format. A question is classified as Known if at least one auxiliary response achieves a BLEURT-20 score above 0.5 against a reference; otherwise, it is classified as Unknown. These auxiliary generations are used exclusively to determine observed knowledge availability, while the original frozen answers and their correctness labels remain unchanged.

This evaluation allows us to examine a particularly relevant subset: questions for which the model has independently demonstrated the ability to produce a correct answer, but whose original responses may still be incorrect. We compare HiPEx with ICR Probe, TriLens, and HaloScope on this Known subset under the same evaluation protocol.

Results. As shown below, HiPEx achieves the highest AUROC on both benchmarks, reaching 0.8904 on TriviaQA and 0.9163 on NQ-Open.

MethodTriviaQANQ-OpenAverage
ICR Probe0.8741 ± 0.01040.9097 ± 0.00840.8919
TriLens0.8183 ± 0.01850.7948 ± 0.02400.8066
HaloScope0.8390 ± 0.04050.9152 ± 0.00520.8771
HiPEx (Ours)0.8904 ± 0.01630.9163 ± 0.01160.9034

These results suggest that HiPEx is not limited to detecting incorrect answers associated with low observed knowledge availability. It also retains strong discriminative ability when the model has independently demonstrated the capacity to produce a correct answer, providing evidence that predictive-path geometry captures signals relevant to errors in answer generation and knowledge utilization.

AuthorsOpenReview ↗

References and Summary of Revisions

[1] Jiang, C., Qi, B., Hong, X., Fu, D., Cheng, Y., Meng, F., Yu, M., Zhou, B., and Zhou, J. (2024). On Large Language Models' Hallucination with Regard to Known Facts. NAACL 2024.

[2] Zhang, Z., Hu, X., Zhang, H., Zhang, J., and Wan, X. (2025). ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs. ACL 2025.

[3] Sun, Z., Wang, Q., Wang, H., Zhang, X., and Xu, J. (2026). Mechanistic Detection and Mitigation of Hallucination in Large Reasoning Models. ICLR 2026.

[4] Damirchi, H., Meza De la Jara, I., Abbasnejad, E., Shamsi, A., Zhang, Z., and Shi, J. Q. (2026). Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning. ACL 2026.

Summary of Revisions

No.Addressed Rebuttal Question(s)Target Section(s)
1W1/Q1(1.1): Theoretical motivation for predictive-path non-directnessMain text §1 (Introduction)
2W1/Q1(1.2): Interpretation of layer- and scale-dependent differencesMain text §4.4.1
3W1/Q1(1.3): Distribution of discriminative information across computational depthsMain text §4.4.2; Appendix G.3
4W2/Q2(2.3): Comparison with direct statistics of intermediate probability distributionsAppendix O
5W3/Q3: Distinguishing knowledge unavailability from knowledge utilization failuresAppendix P
6W3/Q3: Detection performance on questions with demonstrated answerabilityMain text §4.6
7Supporting literature for predictive-path geometryReferences

We are currently working to finalize the revised manuscript and will upload the updated version shortly. We sincerely appreciate the time and effort you have devoted to reviewing our responses, as well as your thoughtful consideration of our work.

Reviewer_KHjLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiPEx, a post-hoc white-box detector for hallucination in LLM answers. HiPEx projects every residual state of an answer token into the shared vocabulary probability space via the logit lens, and measures path excess. On two instruction-tuned models and four short-answer QA benchmarks with BLEURT-20-based labels, HiPEx achieves SOTA performance.

Critical strengths
  • The geometric idea is simple and well-motivated.

  • The paper is well-written.

  • The evaluations are extensive.

Critical weaknesses
  • Scope is narrow relative to the framing. All four benchmarks are short-answer QA, and only two 8-9B models are tested.

  • The labels are proxy labels, and the generation regime is stylized. The human audit is agreement-based and covers a subset. Answers are generated with deterministic five-beam decoding, which does not match deployment decoding.

  • Baseline fairness is questionable. TriLens is evaluated in the main table with the authors' own first-token readout, while the appendix shows that the mean-pool variant improves it substantially and shrinks HiPEx's headline margin from 4.76 to roughly 2.44 points.

  • Computational cost is unreported.

  • The paper shows that path geometry correlates with hallucination but offers no account of why

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Evaluation Coverage, Robustness, and Fair Comparisons (W1–W3, Q1–Q3)

We sincerely thank you for your thoughtful review and constructive feedback. We address each of your comments below.

1.Limited Model and Task Coverage(W1, Q1)

Thank you for pointing out the limited coverage of models and tasks. Regarding model diversity, we extended our evaluation to Qwen2.5-7B-Instruct. Under the same experimental protocol, HiPEx achieves the highest AUROC on three of the four benchmarks. These results provide additional evidence of HiPEx's effectiveness across different model families.We will incorporate the Qwen2.5-7B-Instruct results into the main results table.

MethodTriviaQANQ-OpenSQuADTyDiQA
Perplexity60.1870.1458.6554.92
LLM-Check79.3676.2958.4348.29
LN-Entropy†64.9174.8860.0755.66
Semantic Entropy†73.5979.1958.4554.23
SEP78.8984.6271.6155.06
HaloScope89.4081.1681.4066.52
ICR Probe90.9986.3782.6465.81
TriLens (first-token)81.2178.1075.2265.95
TriLens (mean-token)91.3290.7883.7874.68
HiPEx (Ours)92.3893.2283.1479.39

Regarding benchmark selection, for the hallucination detection task, we chose four commonly used datasets covering two complementary factual QA settings: closed-book QA (TriviaQA and NQ-Open), where models primarily rely on parametric knowledge, and passage-grounded QA (SQuAD 1.1 and TyDiQA-GoldP English), where answers are supported by explicitly provided contexts. This design allows us to evaluate HiPEx under both knowledge-driven and evidence-grounded conditions.

We acknowledge that long-form generalization remains worth investigating. However, the focus on short-form factual QA is consistent with the evaluation settings widely adopted in prior hallucination detection studies. The absence of long-form evaluation does not undermine the validity of our experiments within the current short-form QA setting.We sincerely thank you again for raising this important concern.

2.Proxy Labels, Human Validation, and Generation Settings(W2, Q2)

We appreciate your concerns regarding label reliability, human validation, and the generation protocol.

Regarding proxy labels, following established protocols such as HaloScope (Du et al., 2024), we used BLEURT-20 with a threshold of 0.5. To examine metric dependence, we additionally evaluated ROUGE-L F1 labels on Llama-3 TriviaQA and TyDiQA. On both benchmarks, HiPEx achieves the highest AUROC among all nine methods, reaching 86.04% and 85.96%, respectively (Table 3).

Regarding human validation, we conducted a three-annotator study on 13,656 answers from Llama-3 and Gemma-2 across TriviaQA and TyDiQA. After excluding 35 indeterminate cases, 13,621 annotations were retained. We acknowledge that these annotations assess human judgments of answer-reference agreement. Under these labels, however, HiPEx achieves the highest macro-average AUROC of 75.59%, ranking first in three of the four settings (Appendix H.2), which suffices to demonstrate the robustness of our method.

Regarding generation settings, we follow HaloScope's five-beam decoding protocol to construct a reproducible candidate-answer set shared by all detectors, thereby controlling for generation-related variations in method comparisons. To further examine the sensitivity to decoding strategies, we conducted additional experiments on Llama-3 TriviaQA using Beam(5), Greedy, and Sampling decoding, retraining each detector under the corresponding setting. As shown in the table, HiPEx achieves the highest AUROC under Beam(5) and Sampling, and the second-highest under Greedy, remaining competitive across all three strategies.

MethodBeam(5)GreedySampling
ICR0.8453 ± 0.00850.7830 ± 0.00870.7870 ± 0.0089
HaloScope0.8057 ± 0.03750.6979 ± 0.02030.6911 ± 0.0303
TriLens-mean0.8544 ± 0.00990.8262 ± 0.00640.8237 ± 0.0037
HiPEx0.8644 ± 0.00840.8162 ± 0.00500.8329 ± 0.0021

3.Fairness of the TriLens Comparison(W3, Q3)

We appreciate you highlighting the potential impact of answer-token aggregation on the TriLens comparison. In our main experiments, we follow the first-answer-token readout in the authors' released implementation without modifying its original configuration.

Nevertheless, we agree that a pooling-matched comparison is important. We therefore additionally evaluate a mean-pooled TriLens variant using the same candidate answers and data splits, while keeping its feature construction and probe training protocol unchanged (Appendix M.2). HiPEx still outperforms this stronger variant in all eight settings, with a macro-average advantage of 2.44 percentage points (87.79% vs. 85.35%).

To improve transparency, we will move the mean-pooled TriLens comparison from the appendix to the main paper.

AuthorsOpenReview ↗

Computational Cost and Geometric Motivation (W4–W5, Q4–Q5)

4.Computational Cost(W4, Q4)

We thank you for pointing out the lack of computational cost analysis. We have added a comparison with three competitive white-box baselines across 12 model–benchmark combinations, covering three models and four benchmarks.

MethodAvg. AUROCAvg. Latency (ms/answer)Avg. Throughput (answers/s)Avg. Peak GPU Memory (GiB)
HaloScope80.5428.794536.828415.6597
ICR Probe83.5367.466815.380515.7082
TriLens mean-token85.28347.68393.320618.1519
HiPEx87.54191.63236.281517.5506

HiPEx achieves the highest average AUROC (87.54%). In terms of efficiency, it is approximately 1.81× faster than TriLens with lower memory consumption. Although it incurs higher latency than HaloScope and ICR Probe, this demonstrates a favorable trade-off between detection performance and computational cost.

5.Why Is Predictive-Path Geometry Related to Hallucination?(W5, Q5)

Thank you for highlighting this gap. We have expanded the theoretical motivation connecting predictive-path geometry to hallucination and added analyses of the nature and distribution of HiPEx’s discriminative information.

Theoretical Motivation. Existing work has shown that hallucination-related information is reflected in intermediate computational dynamics, rather than solely in the final predictions. Jiang et al. [1] identified that correct and hallucinated responses exhibit distinct layer-wise prediction trajectories, while Zhang et al. [2] demonstrated that residual stream update dynamics can provide informative signals for hallucination detection. Sun et al. [3] further linked reasoning fluctuations and error backtracking to reasoning hallucinations. More broadly, Damirchi et al. [4] demonstrated that inter-layer geometric displacements can capture structural information beyond static representations.

These findings motivate our core hypothesis: the difference between hallucinated and correct responses may lie not only in how much their predictions change, but also in whether the consecutive updates are directionally coordinated. The path excess captures this distinction by measuring the extra distance introduced by the intermediate predictions relative to the direct endpoint displacement. Consequently, it provides a geometric signal of the non-straightness of the prediction path, which cannot be captured by endpoint distance alone.

Additional Empirical Evidence. We conducted three complementary analyses:

First, Our geometric ablation (Fig. 2 and Appendix E.2) shows that Path Excess consistently outperforms path length and endpoint distance, supporting their difference as an effective geometric inductive bias.

Second, our layer-wise analysis (Fig. 3) reveals that class differences are depth- and scale-dependent, with their direction varying across layers. This explains why no single layer-wise magnitude should be interpreted as a universal hallucination indicator. Instead, preserving depth coordinates enables the detector to integrate distinct local geometric contrasts across computational stages.

Third, we conducted additional layer-wise classification and layer-count ablations using logistic regression on Llama-3 TriviaQA. Single-layer features achieve limited performance (best single-layer AUROC: 0.6992), whereas the full 62-dimensional representation reaches 0.8348. Repeating the best single-layer or same-layer features to 62 dimensions does not improve performance. Furthermore, AUROC increases systematically as more genuine layer-wise features are incorporated under both random and contiguous layer selection, reaching 0.8348 with 30 selected layers.

Single-layer and redundancy controls

FeatureOriginal AUROCRepeated to 62 dimensions
Best single layer0.69920.6992
Layer-21 Inter0.66860.6686
Layer-17 Combined0.69740.6974
Full HiPEx0.8348—

Layer-count ablation

LayersRandom-kContiguous-k
10.62560.6256
20.68070.6655
40.72650.7061
80.76790.7614
160.80770.8157
300.83480.8348
Full (32)0.83480.8348

Together, these findings support the complementary contributions of predictive-path features across computational depths, rather than gains arising merely from redundant dimensional expansion.

Overall, our results suggest that hallucination-related information is distributed across the layer-wise organization of predictive-path geometry. HiPEx exploits this information by preserving computational depth and both inter- and intra-block structures. We emphasize that these findings establish an empirically supported geometric association, while identifying the precise computational mechanisms underlying these differences remains an important direction for future work.

AuthorsOpenReview ↗

References and Summary of Revisions

[1] Jiang, C., Qi, B., Hong, X., Fu, D., Cheng, Y., Meng, F., Yu, M., Zhou, B., and Zhou, J. (2024). On Large Language Models' Hallucination with Regard to Known Facts. NAACL 2024

[2] Zhang, Z., Hu, X., Zhang, H., Zhang, J., and Wan, X. (2025). ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs. ACL 2025

[3] Sun, Z., Wang, Q., Wang, H., Zhang, X., and Xu, J. (2026). Mechanistic Detection and Mitigation of Hallucination in Large Reasoning Models. ICLR 2026.

[4] Damirchi, H., Meza De la Jara, I., Abbasnejad, E., Shamsi, A., Zhang, Z., and Shi, J. Q. (2026). Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning. ACL 2026

Summary of Revisions

No.Addressed Rebuttal Question(s)Target Section(s)
1W1/Q1: Limited evaluation scopeMain text §4.1, §4.2, Table 1
2W2/Q2: Sensitivity to different decoding strategiesAppendix A.1
3W3/Q3: Fair comparison with matched TriLens poolingMain text §4.2, Table 1
4W4/Q4: Trade-off between computational cost and performanceAppendix N
5W5/Q5: Theoretical motivation for predictive-path geometryMain text §1 (Introduction)
6W5/Q5: Layer- and scale-dependent predictive differencesMain text §4.4.1
7W5/Q5: Complementary information across computational depthsMain text §4.4.2; Appendix G.3
8W1/Q1, W2/Q2, W4/Q4: Updates to overall experimental resultsAbstract; Main text §§1, 5
9Supporting literature for predictive-path geometryReferences

We are currently working to finalize the revised manuscript and will upload the updated version shortly. We sincerely appreciate the time and effort you have devoted to reviewing our responses, as well as your thoughtful consideration of our work.

Dear Reviewer,

Thank you for your time and thoughtful review. When you have a chance, we would be grateful if you could take a look at our response. Please let us know if any questions remain.

Best regards,

The Authors

Comments on the paper

1 comment

We sincerely thank the reviewers for their time and constructive feedback. We have revised the manuscript, with the main additions highlighted in blue. The main revisions are summarized below:

  • Strengthened the theoretical motivation for predictive-path geometry in Section 1.
  • Added Qwen2.5-7B-Instruct results across four benchmarks and mean-token TriLens comparisons in Sections 4.1–4.2 and Table 1.
  • Expanded the interpretation of layer-wise differences and complementarity across computational depths in Section 4.4 and Appendix G.3.
  • Updated the cross-benchmark transfer comparisons in Section 4.5 and Appendix I.
  • Added evaluation on questions with demonstrated answerability in Section 4.6 and Appendix P.
  • Updated the abstract, introduction, and conclusion to reflect the revised results.

The appendices additionally report decoding-strategy comparisons, computational costs, direct probability-statistic baselines, and expanded human-label analyses covering annotation agreement, label and sample-size controls, refusal exclusion, and statistical reliability.

We appreciate the reviewers’ valuable suggestions and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.