acceptodds
Under review as a conference paper at ICLR 2027

Detecting Contextual Hallucinations by Pooling and Probing Attention-Head Outputs

Abstract

Retrieval-augmented language models can produce claims that are unsupported by their supplied context, motivating methods for detecting contextual hallucinations. Prior work has shown that truthfulness and hallucination can be predicted from internal model representations, suggesting that these failures leave detectable signatures within the model. Building on linear probing methods for truthfulness, we pool each attention head's current-value contributions across generated positions into a response-level representation and train a linear probe to learn a hallucination-discriminating axis. Across multiple RAG hallucination benchmarks and model families, HeadAxis consistently matches or exceeds strong white-box hallucination detectors. Analysis shows that the axes capture a hallucination signal that generalizes across datasets and can be decomposed across response tokens without additional supervision, while ablations show that the representation is robust across alternative design choices.

Primary area:
interpretability and explainable AI
Keywords:
contextual hallucination detection, retrieval-augmented generation, mechanistic interpretability, attention heads, activation probing, linear probing, internal representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tSuFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HeadAxis, a hallucination detection method that operates on pooled attention head value contributions across generated positions. HeadAxis matches or exceeds prior methods in white-box hallucination detection on four hallucination detection benchmarks over three backbone models across model families.

Critical strengths

S1. The flow of the language is consistent.
S2. This work operationalize the value contributions of attention heads as a richer subspace for hallucination detection, which is mechanistically sound.

Critical weaknesses

W1. The proposed method improves the detection performance by a small margin to competing methods like ITI and RAGLens.
W2. The distinction between eq. 4 and eq. 5 is not empirically demonstrated. I wonder if just pool the value vectors would constitute a good detector.
W3. Experiments do not include larger models from 13B-70B. The universality of the advantage is not shown.

Questions for authors

Please address W1-3.

2 comments

Thank you for your review and for the constructive feedback. All three points led to additions that we believe strengthen the paper.

  1. Small margins over ITI and RAGLens: HeadAxis’s strength is not only that it outperforms baselines, but its consistency. HeadAxis is better on AUROC for all 12 settings, while the ranking of the other baselines varies considerably from setting to setting. To quantify this, we added a pooled test over all 12 settings (Appendix E). Compared to the next-best method ITI-Detect, HeadAxis wins by 2.0 AUROC percentage points, and the margins over the remaining baselines, including RAGLens, are even higher. All comparisons are strongly significant (p < 0.001).

In addition, HeadAxis is conceptually simpler than RAGLens: it needs no sparse autoencoder trained for each scoring model, only linear probes on activations from a single forward pass. Detection performance is only one part of its advantage over such baselines.

  1. Eq. 4 versus Eq. 5: You are right that we did not empirically demonstrate this result. We ran the comparison and added it as Appendix G.1. Normalizing each position before pooling (Eq. 5) performs clearly worse than pooling before normalizing (Eq. 4): the difference is 2.2 AUROC points and is significant (p < 0.01). Regarding your question, pooling the value vectors alone does give a reasonable detector (.883 AUROC, see Appendix G.2), but attention weighting adds a further 2.3 percentage points.

  2. Larger models: We now evaluate LLaMA-2-13B and LLaMA-2-70B on RAGTruth and RAGognize’s LLaMA-2-7B responses (Appendix M). At 13B, HeadAxis achieves the highest AUROC and AUPRC on both datasets, ahead of ITI-Detect, Lookback Lens and RAGLens by at least 1.5 AUROC points on RAGTruth and 3.7 on RAGognize; these margins are larger than at 7B. At 70B, HeadAxis again achieves the highest scores on both datasets, ahead of all three baselines by at least 3.2 AUROC points on RAGTruth and 3.1 on RAGognize.

We hope these additions address your concerns and that you will consider revising your score. We are happy to run any further experiments you think would be useful.

Thank you very much for your response. My concerns are resolved, and I would like to increase the rating.

Reviewer_AuQ8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HeadAxis detects response-level contextual hallucinations using attention-head representations. For each head, it averages diagonal-attention-weighted current-position value vectors over response positions, normalizes the pooled vector, and fits a linear probe. A sparse logistic classifier combines head scores, using grouped out-of-fold predictions for fitting. Experiments cover four contextual-hallucination datasets and three scoring-model families, including teacher-forced scoring of externally generated responses. The method reports strong in-domain AUROC/AUPRC, variable cross-dataset transfer, and improvements over matched terminal-head representations. Further analyses examine alternative projections, attribution to response positions, TruthfulQA detection, and exploratory steering.

Critical strengths
  1. The empirical evaluation supports a useful improvement to a simple detector. Table 1 covers twelve model-dataset combinations with numerous white-box baselines, and Appendix B uses grouped out-of-fold probe scores to fit the final classifier without reusing in-sample predictions. Appendix E distinguishes statistically resolved gains from numerical wins: the RAGBench gains survive correction for all three models, as do the LLaMA and Qwen RAGognize comparisons. The additional matched-pipeline comparison in Appendix G.5 helps attribute improvements to the representation rather than only to the sparse aggregation method.

  2. The analysis is unusually explicit about what the representation does and does not establish. The decomposition explains why pooling complete head outputs repeatedly includes earlier values and prompt contributions, motivating the proposed alternative. The weighting and projection ablations test that rationale, while Section 6 acknowledges that key, query, and PCA projections can perform similarly. The position-level decomposition and Appendix H.3 provide quantitative localization evidence without token-level supervision, and the negative or mixed steering results appropriately distinguish detection from causal control.

Critical weaknesses
  1. The evidence for transferable contextual-faithfulness detection is less convincing than the in-domain results. Table 2 shows substantial source dependence: RAGTruth-trained detectors achieve only .562 and .525 AUROC on RAGBench for LLaMA and Mistral. There is no matched cross-dataset baseline table to show whether the proposed representation transfers better than terminal probing or other detectors. Appendix G.6's fixed-answer context test is helpful, but it uses a claim-level projection rather than the actual final response detector. Consequently, the extent to which the deployed detector learns transferable evidence support, instead of source-specific response regularities, remains uncertain.

  2. The practical cost advantage is not quantified. Appendix F lists model-pass counts and fitting requirements but provides no measured runtime or peak memory. HeadAxis needs value representations and diagonal attention weights across response positions and layers, plus many probe fits; obtaining these quantities can have different costs depending on the attention implementation. A single pass therefore does not establish low incremental inference overhead. Since the performance gains are modest or statistically unresolved in several settings, the accuracy-cost tradeoff is important to judging the method's practical significance.

Questions for authors
  1. Under the same fixed source-training and target-evaluation protocol as Table 2, how do matched terminal probing and the strongest relevant detector compare? A transfer advantage would strengthen the contribution beyond in-domain pooling improvements.
  2. Does the frozen final response-level HeadAxis detector respond appropriately when supporting evidence is removed or contradicted while the answer is held fixed, relative to irrelevant-context changes? Appendix G.6 evaluates a different claim-level projection; testing the deployed detector would directly address the grounding interpretation.
  3. What are extraction/scoring latency and peak memory versus target-model inference and matched terminal probing at representative response and context lengths? Please include whether attention weights are obtained without materializing full attention matrices, and report probe-fitting cost separately.
1 comment

Thank you for your careful and detailed review. We ran all three requested experiments; full tables are in the revision.

  1. Matched cross-dataset transfer: We ran the comparison under the same fixed source-training and target-evaluation protocol as Table 2 (27 transfers, all fitted components frozen), against matched terminal probing, ITI-Detect, Lookback Lens and RAGLens (Appendix O, Table 27). HeadAxis has the highest mean AUROC and AUPRC (.777 / .713). HeadAxis’s advantage is significant against all four on both metrics (Holm-adjusted p < .001, paired source-cluster bootstrap). We agree that transfer remains source-dependent, and now state this in Section 5.1.

  2. Grounding with the deployed detector: We repeated the fixed-answer test with the frozen final response-level detector on 200 faithful RAGognize/Mistral responses (Appendix L; the claim-level test formerly in G.6 is now L.1). Removing a supporting passage shifts the HeadAxis logit by 1.63 more than removing an irrelevant one (95% CI [1.51, 1.74]); marking it false instead of true shifts it by 0.88 [0.81, 0.95]. The matched terminal detector also responds (1.33 and 0.78).

  3. Latency and memory: On 12 RAGTruth/Mistral examples (median 789 tokens) on one L40S GPU, plain SDPA inference takes 83 ms per response with 13.7 GiB peak memory, matched terminal probing 115 ms and 14.6 GiB, and HeadAxis 173 ms and 15.0 GiB (Appendix F). Times include tokenization, the forward pass and feature extraction; detector scoring adds under 1 ms per response. To answer your question: the code behind all reported results does materialize full attention matrices. Prompted by your question, we implemented a diagonal-only extractor that builds no full attention matrix; it takes 87 ms and 13.7 GiB, about 4% over plain inference, and its detector scores differ from the original code by at most 0.001.

  4. On modest or unresolved gains: A pooled test over all 12 in-domain settings shows that HeadAxis improves on each of the nine baselines in both AUROC and AUPRC (p < .001, Table 9). With LLaMA-2-13B and LLaMA-2-70B as scoring models, HeadAxis is also ahead of ITI-Detect, Lookback Lens and RAGLens on RAGTruth and RAGognize (Table 26).

We hope these results address your questions, and we are happy to provide any further detail.

Reviewer_97TeOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new hallucination detection technique for contextual hallucinations, using a linear probe on the representation layer attention outputs, and selectively focusing on only the augmented context (and not the question itself).

The results show that the method proposed in the paper is able to outperform other baselines. In addition, the learned probes even transfer to other datasets, and the training even works for settings with standard hallucination detection (and not just contextual hallucinations). Finally, the hallucination detection technique can also be used to perform hallucination localization, shown through qualitative examples.

Critical strengths
  1. The hallucination detection is simple and seems to work better than other baselines. Moreover, the cross-dataset transfer is extremely promising. Both of these properties make it a method that is actually likely to be deployed and can have impact.
  2. The results are shown across a wide range of models, datasets, RAG techniques, and baselines, giving good support to the generalizability of the claims, although I do have some concerns about the statistical significance of the results.
Critical weaknesses
  1. It is unclear to me if there is technical novelty in the hallucination technique beyond simply targeting a different application (contextual hallucinations instead of hallucinations) and a pooling choice. If there is, I would recommend the authors highlight it appropriately.
  2. The statistical significance tests of the results are confusing to me. The paper points to Appendix E for statistical significance tests, but Appendix E doesn’t contain any results. It contains text explaining how ‘some’ statistical tests showed ‘most consistent differences’ (what does that mean? what are the numbers?) while others are not statistically resolved after correction (again, what does that mean? and why isn’t this explicitly highlighted in the main text if certain results are not statistically significant?)
  3. The qualitative analysis seems extremely limited. Are there only two examples in the entire paper, one in the main text and one in the appendix?
Questions for authors

Please see weaknesses, they contain both weaknesses and my questions.

2 comments

Thank you for your review and for the constructive feedback. You raise valid points about the paper’s framing and presentation, which we have addressed in the revision.

One clarification on your summary: HeadAxis does not target the retrieved context, or any part of the prompt. It pools only over the tokens of the generated response.

  1. Novelty: You ask whether there is technical novelty beyond a new application and a pooling choice. There is: in contrast to ITI, which probes each head’s output at a single terminal position, and RAGLens, which relies on sparse-autoencoder features, HeadAxis pools diagonal-attention-weighted current-position value contributions across response positions and normalizes after pooling. This construction is not trivial, and we are not simply porting a probing method from truthfulness to contextual faithfulness. The obvious way to extend terminal-output probing to long RAG responses is to average each head’s output over the response, and we discover that this fails; in fact it performs worse than not pooling at all. We then decompose the head output to identify the cause, fix it, and verify the fix with strong performance and numerous ablations. Following your recommendation, we now state these findings explicitly in the introduction’s list of contributions.

Beyond the method itself, we see the insights from the decomposition and analysis as a contribution in their own right: they explain why the natural extension fails and lead to a conceptually simple yet high-performing detector. We believe this is new knowledge of value to the community, and we would be very grateful if you would consider the sum of our contributions in your evaluation.

  1. Statistical significance: We apologize for our oversight; the significance values were present in a table in the paper, but it had moved well past the section and we neglected to reference it in the section text. This has now been remedied, and Table 8 contains the significance results. As you suggest, the main text now also states and marks which gains are significant (RAGBench and RAGognize). We further strengthen the analysis with a pooled test over all 12 settings: HeadAxis improves on each of the nine baselines in both AUROC and AUPRC, all with Holm-adjusted p < .001 (Appendix E, Table 9).

  2. Qualitative analysis: We now include ten more examples in Appendix N, covering each dataset and model evaluated and comparing HeadAxis to ITI-Detect at the token level, including two failure cases (Figures 7 and 8).

We hope these changes address your concerns and that you will consider revising your score. We are happy to answer any further questions.

Thank you for the response, and for updating the paper with the fixes. I had earlier missed the contributions of decomposition.

I'm happy to increase my scores.

Comments on the paper

1 comment

Dear reviewers and area chair,

Thank you again for your time on our submission. As the discussion period closes soon, we would like to briefly point to our responses and the revised PDF, in case they are useful for your final assessments:

  • Reviewer 97Te: the significance results are now in Appendix E (Tables 8 and 9) and marked in Table 1, the introduction states the pooling analysis as a contribution, and Appendix N adds ten qualitative examples.
  • Reviewer AuQ8: the three requested experiments are in the revision: matched transfer baselines (Appendix O), evidence tests with the deployed detector (Appendix L), and measured latency and memory (Appendix F).
  • Reviewer tSuF: we added the Eq. 4 versus Eq. 5 comparison (Appendix G.1) and evaluations with 13B and 70B scoring models against ITI-Detect, Lookback Lens and RAGLens (Appendix M).

We would be glad to answer any remaining questions in the time left.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.