- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hierarchical Path Excess (HiPEx), a supervised, post-hoc white-box method for response-level hallucination detection. It projects intermediate residual-stream states through the model’s final normalization and output head into a shared vocabulary probability space. Using normalized square-root Jensen–Shannon distance, it defines path excess as the two-step distance through an intermediate state minus the direct endpoint distance. This quantity is computed at two scales: across neighboring decoder-block outputs and within each block’s cumulative pre-attention, post-attention, and post-FFN states. The resulting layer-wise features are averaged over answer tokens and scored by a lightweight MLP. Feature extraction requires one teacher-forced pass over the completed response, without auxiliary generations.
The authors evaluate HiPEx on Llama-3-8B-Instruct and Gemma-2-9B-it across four QA benchmarks. Under BLEURT-20-derived answer-correctness labels, HiPEx achieves the highest AUROC in all eight model–benchmark settings, with a macro-average of 87.79%. The paper further investigates endpoint subtraction, scale complementarity, matched-budget and pooling controls, source-only cross-benchmark transfer, and alternative automatic and human answer–reference agreement labels.
Critical strengths
-
A well-defined geometric feature supported by targeted ablations. HiPEx applies the same three-point path-excess functional to neighboring block outputs and to cumulative residual states along the executed attention–FFN computation. This provides a concrete computational interpretation and yields a compact representation that preserves layer coordinates at both scales. Section 4.3 reports a mean improvement of 1.63 AUROC percentage points over the stronger path-length or endpoint-distance control in each setting. Appendix E.2 additionally compares explicit subtraction with jointly supplied component features. The two-scale construction improves over the stronger parameter-matched single-scale variant by 2.10 points on average. These controls substantiate the particular feature design rather than merely showing that internal activations contain information about answer correctness.
-
Consistent main-protocol gains and substantial experimental controls. HiPEx achieves the highest AUROC in all eight model–benchmark settings under BLEURT-20 labels. The evaluation uses shared candidate answers, training-only preprocessing, and paragraph-disjoint partitions for passage-grounded datasets. The authors also provide a matched-budget comparison with ICR Probe and an answer-token mean-pooling control for TriLens. HiPEx remains ahead of mean-pooled TriLens in all eight settings, by 2.44 AUROC percentage points on average. Appendix L reports significant paired improvements over ICR Probe and HaloScope in all 16 comparisons, while explicitly restricting the inference to the fitted models. The substantial human annotation study provides further positive evidence, with HiPEx leading three of four settings and achieving the highest macro-average AUROC among the four evaluated methods.
Critical weaknesses
-
The relationship between the operational labels and substantive hallucination errors remains insufficiently characterized. Appendix B explicitly states that the primary BLEURT-20 labels measure answer–reference mismatch. This is a reasonable proxy for QA correctness, and its use does not invalidate the main comparison. However, it can combine different failure types. The human protocol in Appendix H.2 also measures answer–reference agreement and includes refusals, non-answers, and names without textual evidence of alias equivalence in the incorrect class. Consequently, the current evaluation does not reveal whether the advantage primarily concerns factual or contextual errors, failure to answer, or discrepancies arising from reference coverage.
The human study is valuable: it retains 13,621 annotated examples and gives HiPEx a macro-average AUROC of 75.59%, with the best mean in three of four settings. Nevertheless, automatic–human label agreement, annotator agreement before majority voting, and performance by error type are not reported. The probes are also refitted on the smaller human-labeled subsets. This is a valid evaluation of the representation under another supervision target, but it does not isolate label effects from sample-size effects or directly assess the original BLEURT-trained detectors against human judgments. I therefore remain uncertain about which kinds of errors account for the reported gains. This uncertainty is consequential because hallucination detection is the paper’s central application.
-
The comparative robustness evidence does not include the stronger configuration of the closest logit-lens baseline. Appendix M.2 shows that mean pooling improves TriLens by 6.26 AUROC percentage points on average. HiPEx still wins all eight BLEURT-20 comparisons, providing meaningful evidence for its additional value within that protocol. However, the human-label comparison in Appendix H.2 retains first-token TriLens, and the source-only transfer experiment in Section 4.5 compares only ICR Probe and HaloScope. Thus, the advantage over mean-pooled TriLens has not been established in either setting, even though these evaluations are important evidence for robustness beyond the main automated-label protocol.
Transfer results also depend on the backbone: HiPEx leads 16 of 24 directions overall, while its mean off-diagonal AUROC on Llama-3 trails HaloScope by 1.12 points. The authors appropriately disclose this limitation; my concern is the remaining scope of comparative validation, rather than inaccurate reporting. Likewise, the paired significance analysis in Appendix L supports the specified ICR Probe and HaloScope comparisons, but does not establish the corresponding advantage over mean-pooled TriLens. Given that the contribution builds on existing internal-dynamics and logit-lens detection approaches, establishing its additional value against this stronger nearby configuration under human labels and transfer is important to my assessment of its significance.
Questions for authors
-
Label agreement and error composition: On the human-annotated subsets, could you report the BLEURT–human confusion matrices and annotator agreement before majority voting? A targeted audit distinguishing factual/contextual errors from refusals, non-answers, and reference/alias discrepancies would also be helpful. Does HiPEx retain its comparative advantage when refusals and non-answers are analyzed separately from substantive answer errors?
-
Separating label effects from sample-size effects: Could you evaluate the frozen BLEURT-trained detectors against human labels on the corresponding annotated test examples? For the refitted experiments, a comparison using the same retained examples, partitions, and training-set sizes under the two labeling schemes would help isolate the effect of the labels. I regard the existing refitting protocol as valid; these diagnostics would clarify what its robustness results establish.
-
Comparative robustness: Could you evaluate mean-pooled TriLens under the human-label protocol and source-only cross-benchmark transfer, using the same partitions and comparable probe/training settings? Paired AUROC differences and confidence intervals for the human-label comparisons would be particularly informative. These results could materially change my assessment of whether the geometric representation provides a robust improvement over the closest existing approach.
4 comments
Label Controls(W1, Q2) and Comparative Robustness(W2, W3, Q3)
2.Label Effects from Sample-Size Effects (W1, Q2)
We appreciate your thoughtful suggestion to disentangle the effects of supervision labels from those of sample size. We agree that this distinction is important for interpreting the robustness of human-label evaluations. In response to your concern, we conducted two additional controlled experiments.
2.1 Evaluating Frozen BLEURT-Trained Detectors on Human Labels
We directly evaluated the detectors originally trained under BLEURT-20 supervision on the corresponding human-labeled test examples, without retraining or modifying their parameters. HiPEx achieves the highest AUROC of 69.22%.
| Method | AUROC |
|---|---|
| HaloScope | 65.17% |
| ICR Probe | 64.98% |
| TriLens (First-token) | 66.83% |
| TriLens (Mean-token) | 68.67% |
| HiPEx | 69.22% |
2.2 Matched-Sample Comparison under Alternative Supervision Labels To disentangle the effect of supervision labels from differences in sample composition and training-set size, we retrained all detectors on the same retained human-annotated subset, using identical train/validation/test partitions and matched training-set sizes. We evaluated two settings: training and testing with BLEURT-20 labels, and training and testing with human labels. As shown in the table below, HiPEx maintains the highest AUROC under both supervision schemes.
| Method | BLEURT-trained / BLEURT-labeled | Human-trained / Human-labeled |
|---|---|---|
| HaloScope | 80.16% | 69.80% |
| ICR Probe | 82.34% | 70.94% |
| TriLens (First-token) | 78.56% | 71.39% |
| TriLens (Mean-token) | 84.55% | 74.88% |
| HiPEx | 85.17% | 75.59% |
3.Comparative Robustness under Human Labels and Cross-Benchmark Transfer (W2, W3, Q3)
We appreciate your suggestion to evaluate HiPEx against the stronger mean-pooled TriLens configuration under both human-labeled and cross-benchmark settings. We agree that these comparisons are important for assessing the additional value and generalizability of our geometric representation. In response to your comments, we extended the evaluation to include mean-pooled TriLens in both settings and conducted paired statistical analyses.
3.1 Human-Label Evaluation with Mean-Pooled TriLens
We evaluated mean-pooled TriLens on the same five random seeds, data partitions, and human-annotated subsets used in Appendix H.2. As shown in the table, HiPEx achieves the highest macro-average AUROC of 75.59%, compared with 74.88% for mean-pooled TriLens, a gain of 0.71 percentage points. HiPEx achieves the highest mean AUROC in three of the four settings. These results demonstrate a modest overall advantage for HiPEx under human labels.
| Model / Benchmark | TriLens (First) | TriLens (Mean) | HiPEx |
|---|---|---|---|
| Llama-3 / TriviaQA | 74.08 ± 1.54 | 75.51 ± 2.80 | 77.10 ± 1.24 |
| Llama-3 / TyDiQA | 73.06 ± 5.45 | 73.78 ± 6.05 | 72.43 ± 6.43 |
| Gemma-2 / TriviaQA | 67.21 ± 1.37 | 74.91 ± 0.85 | 76.16 ± 1.37 |
| Gemma-2 / TyDiQA | 71.22 ± 5.29 | 75.35 ± 2.89 | 76.69 ± 4.89 |
| Macro-average | 71.39 ± 1.46 | 74.88 ± 1.99 | 75.59 ± 2.25 |
3.2 BLEURT-Labeled Source-Only Cross-Benchmark Transfer
We extended Section 4.5 to include mean-pooled TriLens while retaining the same source-only protocol: each detector is fitted using source-benchmark training data and evaluated on target test data without target adaptation. On Gemma-2, HiPEx achieves an average off-diagonal AUROC of 80.44%, outperforming mean-pooled TriLens (79.31%) by 1.13 percentage points. However, on Llama-3, mean-pooled TriLens achieves 80.40%, exceeding HiPEx (77.16%) by 3.24 percentage points. Therefore, we acknowledge that HiPEx does not have a universal transfer advantage over mean-pooled TriLens.
| Model | Method | In-domain | Off-diagonal |
|---|---|---|---|
| Llama-3 | HiPEx | 88.87 | 77.16 |
| Llama-3 | ICR Probe | 86.22 | 71.49 |
| Llama-3 | HaloScope | 86.09 | 78.28 |
| Llama-3 | TriLens (Mean) | 86.50 | 80.40 |
| Gemma-2 | HiPEx | 86.72 | 80.44 |
| Gemma-2 | ICR Probe | 82.91 | 70.82 |
| Gemma-2 | HaloScope | 75.90 | 71.52 |
| Gemma-2 | TriLens (Mean) | 84.20 | 79.31 |
Comparative Robustness(W2, W3, Q3) and Summary of Revisions
3.Comparative Robustness under Human Labels and Cross-Benchmark Transfer (W2, W3, Q3)
3.3 Source-Only Transfer under Human-Label Supervision
We further evaluated cross-benchmark transfer using human answer–reference agreement labels on TriviaQA and TyDiQA across both backbones. We retained the same human-labeled partitions and used comparable probe-training protocols. For each transfer direction, detectors were trained exclusively on the source benchmark and evaluated on the target test partition. As shown in the table, HiPEx outperforms mean-pooled TriLens in three of the four cross-benchmark directions.We hope this additional evaluation addresses your concern regarding cross-benchmark generalization under human-label supervision.
| Model | Transfer Direction | HiPEx | ICR Probe | HaloScope | TriLens (Mean) |
|---|---|---|---|---|---|
| Llama-3 | TriviaQA → TyDiQA | 67.80 ± 4.86 | 55.33 ± 11.54 | 73.88 ± 2.71 | 70.65 ± 4.19 |
| TyDiQA → TriviaQA | 63.23 ± 2.86 | 61.25 ± 4.12 | 60.03 ± 2.43 | 57.82 ± 1.56 | |
| Gemma-2 | TriviaQA → TyDiQA | 74.86 ± 5.25 | 64.62 ± 7.15 | 62.22 ± 8.31 | 70.85 ± 3.67 |
| TyDiQA → TriviaQA | 63.83 ± 4.28 | 58.23 ± 2.78 | 60.24 ± 9.20 | 55.15 ± 1.66 |
3.4 Paired Statistical Comparison under Human Labels
To assess the reliability of the human-label comparison, we conducted paired bootstrap analyses of AUROC differences between HiPEx and mean-pooled TriLens on identical test examples. The table below reports the paired differences, 95% confidence intervals, and Holm-adjusted p-values across the four settings. HiPEx achieves higher mean AUROC in three settings and wins in three to five of the five seed-level comparisons, depending on the setting. Although none of the Holm-adjusted p-values reach the 0.05 significance level, HiPEx maintains a modest numerical advantage in three of the four settings. We therefore interpret these results cautiously and thank you for encouraging a more rigorous statistical comparison.
| Setting | HiPEx | TriLens (Mean) | Δ | 95% CI | p | p_Holm | Wins |
|---|---|---|---|---|---|---|---|
| L-TriviaQA | 77.10 | 75.51 | +1.59 | [-0.41, +3.63] | 0.1203 | 0.4812 | 4/5 |
| L-TyDiQA | 72.43 | 73.78 | -1.35 | [-6.25, +3.38] | 0.5776 | 1.0000 | 3/5 |
| G-TriviaQA | 76.16 | 74.91 | +1.25 | [-1.01, +3.55] | 0.2716 | 0.8147 | 5/5 |
| G-TyDiQA | 76.69 | 75.35 | +1.34 | [-3.31, +6.12] | 0.5762 | 1.0000 | 4/5 |
Summary of Revisions
| No. | Rebuttal Item | Target Section |
|---|---|---|
| 1 | W1/Q1 (1.1–1.2): Agreement between BLEURT-20 and human labels, and inter-annotator consistency | Appendix H.2 |
| 2 | W1/Q1 (1.3): Potential reliance on refusal and non-answer signals | Appendix H.2 |
| 3 | W1/Q2 (2.1): Effects of supervision labels independent of retraining | Appendix H.2 |
| 4 | W1/Q2 (2.2): Effects of supervision labels versus sample-size differences | Appendix H.2 |
| 5 | W2/W3/Q3 (3.1): Comparison with mean-pooled TriLens under human labels | Appendix H.2, Table 9 |
| 6 | W2/W3/Q3 (3.4): Statistical reliability of the human-label comparison | Appendix H.2 |
| 7 | W2/W3/Q3 (3.2): Comparison with mean-pooled TriLens under cross-benchmark transfer | Appendix I.2; Main text §4.5 |
| 8 | W2/W3/Q3 (3.3): Cross-benchmark transfer under human-label supervision | Appendix I.3 |
We are currently working to finalize the revised manuscript and will upload the updated version shortly. We sincerely appreciate the time and effort you have devoted to reviewing our responses, as well as your thoughtful consideration of our work.
Dear Reviewer,
Thank you for your time and thoughtful review. When you have a chance, we would be grateful if you could take a look at our response. Please let us know if any questions remain.
Best regards,
The Authors
Label Reliability and Error Analysis (W1, Q1)
We sincerely thank you for your thoughtful review. Your feedback helped us examine label reliability and strengthen the empirical evidence for HiPEx under human-labeled and cross-benchmark settings.
1.Label Agreement and Error Composition (W1, Q1)
We appreciate your emphasis on the importance of characterizing both label reliability and error composition. Following your suggestion, we conducted additional analyses to examine the agreement between automatic and human labels, assess inter-annotator consistency, and evaluate HiPEx after excluding refusals and non-answers.
1.1 BLEURT–Human Label Agreement
We constructed confusion matrices by comparing BLEURT-20 labels with majority-vote human labels on the same 13,621 retained examples. As shown in table below, the overall agreement rate is 87.96% (11,981/13,621). Agreement varies across settings and is lower on Llama-3 TyDiQA (68.03%). This may be because TyDiQA is a passage-based QA task, where answers need to be matched against passage-level evidence. Longer, more context-dependent responses may introduce greater ambiguity in semantic matching, thereby leading to lower agreement between BLEURT and human judgments.
1.2 Inter-Annotator Agreement
The unanimous agreement rates for the three annotators independently labeling each example are as follows.
1.3 Targeted Error-Type Audit
Due to time constraints, we were unable to verify and categorize all the data. To examine whether HiPEx relies primarily on refusal and non-answer signals, we randomly sampled 150 originally incorrect answers from each of the four model–benchmark settings with human labels (600 in total). We identified 268 factual errors, 30 contextual errors, 154 refusals/non-answers, and 148 reference/alias mismatches. After excluding the 154 refusals and non-answers, 446 candidate errors remained. For each seed, we evaluated the frozen detectors on a test subset comprising all correct answers in the original test partition and the intersection of these 446 non-refusal candidate errors with that same partition. As shown in the table below, HiPEx achieves the highest mean AUROC (65.94%).
We are grateful for your suggestion, which prompted us to examine this potential confound more closely.