- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on hallucination detection in LLMs for question-answering tasks. The authors introduce a new approach that removes the question-aligned component from answer representations. Additionally, this representation can be stacked with the question representation to improve in-domain performance. The authors also propose filtering layers and neurons by a Fisher discriminant criterion to retain the most discriminative signals. Finally, the paper provides an experimental evaluation across several QA tasks and LLMs in both in-domain and out-of-domain setups.
Critical strengths
- An important research direction focusing on the generalization of supervised hallucination detection methods.
- The proposed approach is clearly explained and shows superior results, both in-domain and out-of-domain, across the considered baselines. Importantly, orthogonal projection shows quite stable performance without substantial degradation in the OOD setup.
Critical weaknesses
- The paper overlooks several related works and strong baselines: Semantic Density [1], CoCoA [2], and RAUQ [3] (unsupervised), or TAD [4], LookBackLens [5], and UHead 6.
- The results section is poorly structured. Table 1 appears after Figure 2 but is referenced before it (and without a link). This makes the paper hard to follow.
- The method uses only the hidden state of the final final token. While this works empirically, an ablation study with all tokens should be presented to clearly identify the improvements.
- The evaluation benchmark is quite concerning. First, it considers only QA datasets, while other tasks and more complex QA datasets are overlooked. Second, some of the considered LLMs are outdated (Llama-2, Gemma-2), while the others represent only the Qwen3 family, so testing on more modern LLMs is crucial. Third, the baselines are weak (see Weakness 1). Fourth, hallucination detection papers usually consider PR-AUC/PRR as primary evaluation metrics, so I think they should be presented here, at least as additional results [7].
- While the paper claims strong generalization, its evaluation is quite limited. Generalization is tested only for a single setup across all models, or for a single model across various setups. Since this is one of the paper’s main contributions, it should be investigated in more detail.
- Minor issues:
- Figure 1 is low resolution and looks blurry.
- In Section 3.3.1, is not defined in Equation 2.
- Line 170 talks about sample means without saying which axis is being averaged: tokens in a sequence, dataset instances, or random seeds.
- is not a sampling consistency method. It usually refers to single-forward-pass methods that extract the logit of the token “false” (or “true”, respectively), and this should be clarified here.
[1] Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space. (Qiu et al., NeurIPS 2024)
[2] Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency (Vashurin et al., NeurIPS 2025)
[3] Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads. (A. Vazhentsev et al., ICML 2026)
[4] Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (Vazhentsev et al., EMNLP 2025)
[5] Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (Chuang et al., EMNLP 2024)
[6] A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (Shelmanov et al., EMNLP 2025)
[7] Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph (Vashurin et al., TACL 2025)
Questions for authors
See weaknesses
6 comments
3. Additional cross-domain experiments
We have expanded the cross-domain evaluation on Gemma-2-2B, Qwen3-14B, and Qwen3-30B-A3B to cover three additional leave-one-domain-out QA settings and two transfer paths to GSM8K. These results support QAOD’s transfer advantage across a broader range of models and datasets.
| Model | Method | SQuAD+NQ+BioASQ → TriviaQA | TriviaQA+NQ+BioASQ → SQuAD | TriviaQA+SQuAD+BioASQ → NQ | TriviaQA+SQuAD+NQ → GSM8K | BioASQ → GSM8K |
|---|---|---|---|---|---|---|
| Gemma-2-2B | P_ik | 65.02 | 65.22 | 60.58 | 56.14 | 48.67 |
| MIND | 59.46 | 59.90 | 63.08 | 54.82 | 56.07 | |
| SAPLMA | 72.83 | 71.95 | 71.87 | 55.05 | 54.99 | |
| ICR Probe | 51.02 | 60.56 | 65.33 | 49.31 | 48.31 | |
| QAOD V⊥ | 79.36 | 78.34 | 73.67 | 59.47 | 56.91 | |
| QAOD Q⊕V⊥ | 78.13 | 76.02 | 72.62 | 56.99 | 54.89 | |
| Qwen3-14B | P_ik | 65.61 | 65.20 | 59.12 | 64.76 | 54.81 |
| MIND | 64.89 | 63.21 | 66.07 | 72.62 | 71.07 | |
| SAPLMA | 68.97 | 73.33 | 76.55 | 65.35 | 57.19 | |
| ICR Probe | 66.16 | 61.75 | 59.98 | 42.49 | 36.14 | |
| QAOD V⊥ | 83.56 | 81.18 | 79.48 | 80.63 | 71.24 | |
| QAOD Q⊕V⊥ | 83.33 | 80.55 | 79.37 | 78.93 | 71.29 | |
| Qwen3-30B-A3B | P_ik | 78.44 | 73.69 | 67.80 | 67.90 | 58.51 |
| MIND | 69.71 | 70.10 | 65.76 | 68.29 | 69.12 | |
| SAPLMA | 74.08 | 74.94 | 75.47 | 65.29 | 60.18 | |
| ICR Probe | 70.23 | 73.46 | 68.32 | 48.28 | 53.85 | |
| QAOD V⊥ | 84.78 | 84.88 | 80.69 | 71.74 | 69.07 | |
| QAOD Q⊕V⊥ | 84.99 | 83.63 | 81.78 | 72.90 | 70.31 |
4. Scope of the claims and an additional model
Thank you for pointing out the limitations in task and model coverage.
Task coverage. As noted in the Limitations section, our current evaluation focuses on knowledge-intensive QA. We agree that broader task coverage is needed to assess QAOD’s applicability beyond this setting, and we plan to explore its extension to open-ended generation in future work.
Model coverage. We also agree that including more recent non-Qwen models would make the evaluation more complete. We have therefore added Gemma4 E2B, a recent model from the Gemma family. On this model, QAOD Q⊕V⊥ achieves the highest AUROC, PR-AUC, and PRR₅₀ on all four in-domain datasets. In the six transfer settings, a QAOD variant also achieves the highest AUROC in every setting. These additional results provide further evidence for QAOD’s effectiveness beyond the originally evaluated models.
Each cell below reports AUROC (%) / PR-AUC (%) / PRR₅₀ (dimensionless).
| Method | TriviaQA | SQuAD | NQ | BioASQ |
|---|---|---|---|---|
| PPL | 74.61 / 43.97 / 0.413 | 80.08 / 37.89 / 0.532 | 74.85 / 89.39 / 0.585 | 78.68 / 76.29 / 0.552 |
| Logit Entropy | 72.72 / 43.51 / 0.385 | 75.00 / 25.47 / 0.435 | 79.04 / 92.64 / 0.777 | 82.62 / 85.28 / 0.695 |
| SINdex | 66.16 / 36.11 / 0.318 | 66.48 / 22.09 / 0.325 | 74.37 / 90.31 / 0.681 | 79.60 / 81.71 / 0.614 |
| P_false | 71.27 / 38.25 / 0.360 | 69.30 / 26.00 / 0.342 | 41.08 / 71.06 / -0.384 | 69.35 / 64.30 / 0.303 |
| LLM-Check | 62.53 / 31.40 / 0.227 | 62.72 / 18.14 / 0.197 | 58.95 / 81.45 / 0.244 | 67.85 / 68.48 / 0.335 |
| P_ik | 80.70 / 56.39 / 0.566 | 77.79 / 38.97 / 0.506 | 79.29 / 92.03 / 0.749 | 77.50 / 81.40 / 0.636 |
| MIND | 77.24 / 57.59 / 0.536 | 76.33 / 40.36 / 0.513 | 82.23 / 93.51 / 0.802 | 79.39 / 82.54 / 0.653 |
| SAPLMA | 88.05 / 71.44 / 0.711 | 86.21 / 54.26 / 0.672 | 84.61 / 94.93 / 0.879 | 85.78 / 87.80 / 0.770 |
| ICR Probe | 75.39 / 49.97 / 0.454 | 74.81 / 35.25 / 0.462 | 80.76 / 93.83 / 0.854 | 79.85 / 81.66 / 0.605 |
| QAOD V⊥ | 89.51 / 74.61 / 0.749 | 85.61 / 53.54 / 0.668 | 84.83 / 95.16 / 0.907 | 83.05 / 86.04 / 0.758 |
| QAOD Q⊕V⊥ | 90.00 / 74.81 / 0.757 | 87.28 / 54.33 / 0.697 | 86.26 / 95.69 / 0.924 | 86.14 / 88.05 / 0.777 |
| Method | SQuAD+NQ+BioASQ → TriviaQA | TriviaQA+NQ+BioASQ → SQuAD | TriviaQA+SQuAD+BioASQ → NQ | TriviaQA+SQuAD+NQ → BioASQ | TriviaQA+SQuAD+NQ → GSM8K | BioASQ → GSM8K |
|---|---|---|---|---|---|---|
| P_ik | 59.62 / 29.62 / 0.167 | 64.02 / 18.91 / 0.254 | 53.85 / 78.62 / 0.081 | 45.35 / 50.95 / -0.113 | 47.08 / 84.79 / -0.159 | 54.50 / 87.72 / 0.154 |
| MIND | 62.38 / 27.51 / 0.214 | 58.09 / 20.34 / 0.182 | 56.52 / 79.29 / 0.151 | 44.94 / 51.19 / -0.196 | 51.67 / 86.23 / 0.000 | 46.60 / 85.19 / -0.099 |
| SAPLMA | 66.59 / 35.07 / 0.301 | 70.88 / 31.61 / 0.386 | 66.37 / 82.84 / 0.451 | 57.08 / 62.93 / 0.082 | 62.35 / 90.82 / 0.440 | 46.66 / 85.00 / -0.137 |
| ICR Probe | 61.32 / 26.83 / 0.179 | 63.25 / 18.76 / 0.230 | 54.86 / 81.11 / 0.187 | 62.66 / 64.50 / 0.233 | 45.64 / 85.35 / -0.108 | 39.69 / 82.46 / -0.366 |
| QAOD V⊥ | 81.44 / 55.49 / 0.563 | 74.35 / 35.75 / 0.445 | 68.36 / 85.46 / 0.471 | 68.23 / 74.47 / 0.411 | 64.04 / 91.42 / 0.457 | 57.45 / 88.74 / 0.179 |
| QAOD Q⊕V⊥ | 79.61 / 52.28 / 0.531 | 74.00 / 33.61 / 0.417 | 66.85 / 83.25 / 0.456 | 55.12 / 60.79 / 0.016 | 64.66 / 91.74 / 0.474 | 63.69 / 90.56 / 0.419 |
These results provide further support for QAOD's effectiveness. Together, the original and additional experiments now cover three model families: Gemma, Llama, and Qwen. Qwen3-14B and Qwen3-30B-A3B also provide a comparison between dense and mixture-of-experts architectures. This coverage offers evidence for QAOD's effectiveness across the evaluated models.
5. Why we use the final token
We chose final-token representations because they can incorporate preceding context, have been effective in prior work, and are simple to extract. Our ablation further supports this choice: the final-token readout performs best for the joint probe, while mean pooling does not provide a consistent improvement across both probes despite requiring aggregation over the full sequence.
We compared the readout choices on Qwen3-14B / TriviaQA using the same questions as the main table, with answers from a separate generation run. AUROC (%) is reported as the mean ± standard deviation across seeds.
| Question representation Q | Answer representation A | QAOD V⊥ | QAOD Q⊕V⊥ |
|---|---|---|---|
| Final token | Final token | 86.64 ± 0.25 | 90.42 ± 0.19 |
| Final token | Token at 1/2 of the answer length | 72.95 ± 0.51 | 87.48 ± 0.53 |
| Final token | Token at 2/3 of the answer length | 85.12 ± 0.89 | 89.10 ± 0.20 |
| Final token | Mean over answer tokens | 86.78 ± 0.67 | 87.97 ± 0.41 |
| Mean over question tokens | Final token | 86.78 ± 0.67 | 90.17 ± 0.24 |
| Mean over question tokens | Mean over answer tokens | 89.01 ± 0.22 | 88.96 ± 0.47 |
Mean pooling improves QAOD V⊥, whereas the final-token readout remains best for QAOD Q⊕V⊥, supporting its use as a simple default. Given the broadly comparable performance of the two choices, we use the final-token readout to reduce computation.
6. Paper organization and presentation
Thank you for your suggestions on the paper's organization and presentation. In the revision, we will add the missing cross-reference to Table 1, improve the clarity of Figure 1, define the notation before Equation (2), clarify that the mean on line 170 is taken over training samples, and clarify the description of P_false.
Thank you for your detailed review and for recognizing the stable cross-domain performance of the orthogonal probe. In response to your concerns about the breadth of the evaluation, we have added comparisons with recent baselines and expanded the experiments across models and transfer settings. We also report PR-AUC/PRR results and token-pooling ablations.
1. Additional baselines
We have now evaluated RAUQ, CoCoA, TAD, Semantic Density, and UHead on all four datasets with Qwen3-14B. QAOD Q⊕V⊥ retains the highest in-domain AUROC on all four datasets after adding these baselines. We have also added cross-domain experiments for TAD and UHead, both of which require training. These additional comparisons show that QAOD remains competitive against the newly evaluated baselines under domain shift.
These results continue to support QAOD's in-domain advantage. CoCoA and Semantic Density also require multiple sampled answers, so QAOD has lower detection overhead.
We also evaluated cross-domain transfer for TAD and UHead, both of which require training. AUROC (%) is reported below.
QAOD V⊥ achieves the highest AUROC when transferring to TriviaQA, SQuAD, and NQ. On BioASQ, it exceeds the joint readout by 5.11 percentage points, showing the value of the orthogonal readout in these transfer settings. TAD performs slightly better than QAOD V⊥ on BioASQ; QAOD nevertheless shows clear gains on the other three target datasets. The V⊥ representation also exhibits a smaller normalized centroid shift and lower CKA alignment with domain labels. After Fisher-based layer and neuron selection, it retains useful information for correctness prediction. These observations support the interpretation that question–answer orthogonal decomposition attenuates some domain-related variation while preserving discriminative signals.
LookBackLens targets generation with an explicit input context and assesses whether an answer departs from that context. Our main evaluation instead judges short-answer correctness against reference answers, and the four QA datasets do not share a uniform supporting-context setup. We therefore have not included it in the comparison at this stage.
2. Additional PR-AUC and PRR results
Thank you for suggesting additional metrics for class imbalance. We report PR-AUC and PRR₅₀ for Qwen3-14B, treating incorrect answers as the positive class. The additional PR-AUC and PRR₅₀ results continue to support QAOD’s advantage: both variants outperform the listed baselines on both metrics across all four Qwen3-14B datasets. These results complement the AUROC comparison by assessing precision–recall performance and the benefit of rejecting high-risk answers.
PR-AUC is computed by trapezoidal integration over the full precision–recall curve and is reported as a percentage. PRR₅₀ measures the accuracy gain from rejecting up to 50% of answers according to their risk scores. A value of 0 corresponds to random ranking, 1 to ideal ranking, and a negative value to performance below random ranking. PRR₅₀ is reported as a dimensionless decimal.
Each cell reports AUROC (%) / PR-AUC (%) / PRR₅₀.