acceptodds
Under review as a conference paper at ICLR 2027

When Answers Stray from Questions: Hallucination Detection via Question-Answer Orthogonal Decomposition

Abstract

Hallucination detection in large language models (LLMs) requires balancing accuracy, efficiency, and robustness to distribution shift. Black-box consistency methods are effective but demand repeated inference; single-pass white-box probes are efficient yet treat answer representations in isolation, often degrading sharply under domain shift. We propose **QAoD** (**Q**uestion-**A**nswer **O**rthogonal **D**ecomposition), a single-pass framework that projects away the question-aligned direction from the answer representation to obtain a question-orthogonal component that suppresses domain-conditioned variation. To identify informative signals, QAoD further selects layers via diversity-penalized Fisher scoring and discriminative neurons via Fisher importance. To address both in-domain detection and cross-domain generalization, we design two complementary probing strategies: pairing the orthogonal component with question context yields a joint probe that maximizes in-domain discriminability, while using the orthogonal component alone supports robust transfer in our cross-domain evaluations. QAoD's joint probe achieves the best in-domain AUROC across all evaluated model-dataset pairs, while the orthogonal-only probe delivers the strongest OOD transfer, surpassing the strongest evaluated white-box baseline for each model by 6.27-14.22 AUROC percentage points on BioASQ at under 2% of generation cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, hallucination detection, white-box probing, hidden-state representations, orthogonal decomposition, Fisher discriminant analysis, out-of-distribution generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mKH3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on hallucination detection in LLMs for question-answering tasks. The authors introduce a new approach that removes the question-aligned component from answer representations. Additionally, this representation can be stacked with the question representation to improve in-domain performance. The authors also propose filtering layers and neurons by a Fisher discriminant criterion to retain the most discriminative signals. Finally, the paper provides an experimental evaluation across several QA tasks and LLMs in both in-domain and out-of-domain setups.

Critical strengths
  1. An important research direction focusing on the generalization of supervised hallucination detection methods.
  2. The proposed approach is clearly explained and shows superior results, both in-domain and out-of-domain, across the considered baselines. Importantly, orthogonal projection shows quite stable performance without substantial degradation in the OOD setup.
Critical weaknesses
  1. The paper overlooks several related works and strong baselines: Semantic Density [1], CoCoA [2], and RAUQ [3] (unsupervised), or TAD [4], LookBackLens [5], and UHead 6.
  2. The results section is poorly structured. Table 1 appears after Figure 2 but is referenced before it (and without a link). This makes the paper hard to follow.
  3. The method uses only the hidden state of the final final token. While this works empirically, an ablation study with all tokens should be presented to clearly identify the improvements.
  4. The evaluation benchmark is quite concerning. First, it considers only QA datasets, while other tasks and more complex QA datasets are overlooked. Second, some of the considered LLMs are outdated (Llama-2, Gemma-2), while the others represent only the Qwen3 family, so testing on more modern LLMs is crucial. Third, the baselines are weak (see Weakness 1). Fourth, hallucination detection papers usually consider PR-AUC/PRR as primary evaluation metrics, so I think they should be presented here, at least as additional results [7].
  5. While the paper claims strong generalization, its evaluation is quite limited. Generalization is tested only for a single setup across all models, or for a single model across various setups. Since this is one of the paper’s main contributions, it should be investigated in more detail.
  6. Minor issues:
    1. Figure 1 is low resolution and looks blurry.
    2. In Section 3.3.1, yy is not defined in Equation 2.
    3. Line 170 talks about sample means without saying which axis is being averaged: tokens in a sequence, dataset instances, or random seeds.
    4. pfalsep_\text{false} is not a sampling consistency method. It usually refers to single-forward-pass methods that extract the logit of the token “false” (or “true”, respectively), and this should be clarified here.

[1] Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space. (Qiu et al., NeurIPS 2024)
[2] Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency (Vashurin et al., NeurIPS 2025)
[3] Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads. (A. Vazhentsev et al., ICML 2026)
[4] Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (Vazhentsev et al., EMNLP 2025)
[5] Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps (Chuang et al., EMNLP 2024)
[6] A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (Shelmanov et al., EMNLP 2025)
[7] Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph (Vashurin et al., TACL 2025)

Questions for authors

See weaknesses

6 comments

Thank you for your detailed review and for recognizing the stable cross-domain performance of the orthogonal probe. In response to your concerns about the breadth of the evaluation, we have added comparisons with recent baselines and expanded the experiments across models and transfer settings. We also report PR-AUC/PRR results and token-pooling ablations.

1. Additional baselines

We have now evaluated RAUQ, CoCoA, TAD, Semantic Density, and UHead on all four datasets with Qwen3-14B. QAOD Q⊕V⊥ retains the highest in-domain AUROC on all four datasets after adding these baselines. We have also added cross-domain experiments for TAD and UHead, both of which require training. These additional comparisons show that QAOD remains competitive against the newly evaluated baselines under domain shift.

MethodTriviaQASQuADNQBioASQ
RAUQ63.6466.8974.4377.72
Semantic Density60.6669.1874.2074.92
CoCoA-PPL75.1077.9379.2379.53
CoCoA-SP75.0778.7480.4580.82
TAD80.8582.8381.2481.59
UHead80.8275.6479.8180.74
QAOD V⊥87.4281.3479.9878.39
QAOD Q⊕V⊥90.1385.1782.6985.69

These results continue to support QAOD's in-domain advantage. CoCoA and Semantic Density also require multiple sampled answers, so QAOD has lower detection overhead.

We also evaluated cross-domain transfer for TAD and UHead, both of which require training. AUROC (%) is reported below.

MethodSQuAD+NQ+BioASQ → TriviaQATriviaQA+NQ+BioASQ → SQuADTriviaQA+SQuAD+BioASQ → NQTriviaQA+SQuAD+NQ → BioASQ
TAD76.1273.4878.1478.59
UHead71.4173.1573.4562.15
QAOD V⊥83.5681.1879.4876.73
QAOD Q⊕V⊥83.3380.5579.3771.62

QAOD V⊥ achieves the highest AUROC when transferring to TriviaQA, SQuAD, and NQ. On BioASQ, it exceeds the joint readout by 5.11 percentage points, showing the value of the orthogonal readout in these transfer settings. TAD performs slightly better than QAOD V⊥ on BioASQ; QAOD nevertheless shows clear gains on the other three target datasets. The V⊥ representation also exhibits a smaller normalized centroid shift and lower CKA alignment with domain labels. After Fisher-based layer and neuron selection, it retains useful information for correctness prediction. These observations support the interpretation that question–answer orthogonal decomposition attenuates some domain-related variation while preserving discriminative signals.

LookBackLens targets generation with an explicit input context and assesses whether an answer departs from that context. Our main evaluation instead judges short-answer correctness against reference answers, and the four QA datasets do not share a uniform supporting-context setup. We therefore have not included it in the comparison at this stage.

2. Additional PR-AUC and PRR results

Thank you for suggesting additional metrics for class imbalance. We report PR-AUC and PRR₅₀ for Qwen3-14B, treating incorrect answers as the positive class. The additional PR-AUC and PRR₅₀ results continue to support QAOD’s advantage: both variants outperform the listed baselines on both metrics across all four Qwen3-14B datasets. These results complement the AUROC comparison by assessing precision–recall performance and the benefit of rejecting high-risk answers.

PR-AUC is computed by trapezoidal integration over the full precision–recall curve and is reported as a percentage. PRR₅₀ measures the accuracy gain from rejecting up to 50% of answers according to their risk scores. A value of 0 corresponds to random ranking, 1 to ideal ranking, and a negative value to performance below random ranking. PRR₅₀ is reported as a dimensionless decimal.

Each cell reports AUROC (%) / PR-AUC (%) / PRR₅₀.

MethodTriviaQASQuADNQBioASQ
PPL53.25 / 22.95 / 0.06554.89 / 10.91 / 0.09861.81 / 76.88 / 0.36961.33 / 52.17 / 0.243
Logit Entropy53.25 / 23.61 / 0.06554.92 / 11.39 / 0.09862.06 / 78.44 / 0.37861.88 / 56.49 / 0.257
SINdex65.73 / 29.85 / 0.31174.02 / 19.72 / 0.45178.30 / 86.14 / 0.69880.50 / 71.78 / 0.540
P_false78.04 / 39.47 / 0.54958.33 / 6.96 / 0.13379.05 / 85.06 / 0.63679.17 / 69.07 / 0.503
LLM-Check61.32 / 14.05 / 0.19762.64 / 7.44 / 0.16953.16 / 65.60 / 0.07759.67 / 44.08 / 0.130
P_ik79.14 / 44.75 / 0.56469.76 / 15.61 / 0.35867.73 / 76.50 / 0.43273.22 / 62.98 / 0.454
MIND81.35 / 46.14 / 0.61972.12 / 14.89 / 0.40073.69 / 81.99 / 0.56276.81 / 69.90 / 0.521
SAPLMA85.45 / 53.89 / 0.71883.57 / 23.57 / 0.59978.93 / 87.02 / 0.69581.66 / 73.46 / 0.552
ICR Probe73.00 / 35.31 / 0.53473.26 / 20.75 / 0.46269.01 / 82.85 / 0.56874.14 / 70.52 / 0.502
QAOD V⊥87.42 / 58.77 / 0.75781.34 / 31.07 / 0.63879.98 / 88.77 / 0.74678.39 / 79.18 / 0.665
QAOD Q⊕V⊥90.13 / 57.52 / 0.74985.17 / 28.31 / 0.65182.69 / 89.16 / 0.76385.69 / 83.14 / 0.713

3. Additional cross-domain experiments

We have expanded the cross-domain evaluation on Gemma-2-2B, Qwen3-14B, and Qwen3-30B-A3B to cover three additional leave-one-domain-out QA settings and two transfer paths to GSM8K. These results support QAOD’s transfer advantage across a broader range of models and datasets.

ModelMethodSQuAD+NQ+BioASQ → TriviaQATriviaQA+NQ+BioASQ → SQuADTriviaQA+SQuAD+BioASQ → NQTriviaQA+SQuAD+NQ → GSM8KBioASQ → GSM8K
Gemma-2-2BP_ik65.0265.2260.5856.1448.67
MIND59.4659.9063.0854.8256.07
SAPLMA72.8371.9571.8755.0554.99
ICR Probe51.0260.5665.3349.3148.31
QAOD V⊥79.3678.3473.6759.4756.91
QAOD Q⊕V⊥78.1376.0272.6256.9954.89
Qwen3-14BP_ik65.6165.2059.1264.7654.81
MIND64.8963.2166.0772.6271.07
SAPLMA68.9773.3376.5565.3557.19
ICR Probe66.1661.7559.9842.4936.14
QAOD V⊥83.5681.1879.4880.6371.24
QAOD Q⊕V⊥83.3380.5579.3778.9371.29
Qwen3-30B-A3BP_ik78.4473.6967.8067.9058.51
MIND69.7170.1065.7668.2969.12
SAPLMA74.0874.9475.4765.2960.18
ICR Probe70.2373.4668.3248.2853.85
QAOD V⊥84.7884.8880.6971.7469.07
QAOD Q⊕V⊥84.9983.6381.7872.9070.31

4. Scope of the claims and an additional model

Thank you for pointing out the limitations in task and model coverage.

Task coverage. As noted in the Limitations section, our current evaluation focuses on knowledge-intensive QA. We agree that broader task coverage is needed to assess QAOD’s applicability beyond this setting, and we plan to explore its extension to open-ended generation in future work.

Model coverage. We also agree that including more recent non-Qwen models would make the evaluation more complete. We have therefore added Gemma4 E2B, a recent model from the Gemma family. On this model, QAOD Q⊕V⊥ achieves the highest AUROC, PR-AUC, and PRR₅₀ on all four in-domain datasets. In the six transfer settings, a QAOD variant also achieves the highest AUROC in every setting. These additional results provide further evidence for QAOD’s effectiveness beyond the originally evaluated models.

Each cell below reports AUROC (%) / PR-AUC (%) / PRR₅₀ (dimensionless).

MethodTriviaQASQuADNQBioASQ
PPL74.61 / 43.97 / 0.41380.08 / 37.89 / 0.53274.85 / 89.39 / 0.58578.68 / 76.29 / 0.552
Logit Entropy72.72 / 43.51 / 0.38575.00 / 25.47 / 0.43579.04 / 92.64 / 0.77782.62 / 85.28 / 0.695
SINdex66.16 / 36.11 / 0.31866.48 / 22.09 / 0.32574.37 / 90.31 / 0.68179.60 / 81.71 / 0.614
P_false71.27 / 38.25 / 0.36069.30 / 26.00 / 0.34241.08 / 71.06 / -0.38469.35 / 64.30 / 0.303
LLM-Check62.53 / 31.40 / 0.22762.72 / 18.14 / 0.19758.95 / 81.45 / 0.24467.85 / 68.48 / 0.335
P_ik80.70 / 56.39 / 0.56677.79 / 38.97 / 0.50679.29 / 92.03 / 0.74977.50 / 81.40 / 0.636
MIND77.24 / 57.59 / 0.53676.33 / 40.36 / 0.51382.23 / 93.51 / 0.80279.39 / 82.54 / 0.653
SAPLMA88.05 / 71.44 / 0.71186.21 / 54.26 / 0.67284.61 / 94.93 / 0.87985.78 / 87.80 / 0.770
ICR Probe75.39 / 49.97 / 0.45474.81 / 35.25 / 0.46280.76 / 93.83 / 0.85479.85 / 81.66 / 0.605
QAOD V⊥89.51 / 74.61 / 0.74985.61 / 53.54 / 0.66884.83 / 95.16 / 0.90783.05 / 86.04 / 0.758
QAOD Q⊕V⊥90.00 / 74.81 / 0.75787.28 / 54.33 / 0.69786.26 / 95.69 / 0.92486.14 / 88.05 / 0.777
MethodSQuAD+NQ+BioASQ → TriviaQATriviaQA+NQ+BioASQ → SQuADTriviaQA+SQuAD+BioASQ → NQTriviaQA+SQuAD+NQ → BioASQTriviaQA+SQuAD+NQ → GSM8KBioASQ → GSM8K
P_ik59.62 / 29.62 / 0.16764.02 / 18.91 / 0.25453.85 / 78.62 / 0.08145.35 / 50.95 / -0.11347.08 / 84.79 / -0.15954.50 / 87.72 / 0.154
MIND62.38 / 27.51 / 0.21458.09 / 20.34 / 0.18256.52 / 79.29 / 0.15144.94 / 51.19 / -0.19651.67 / 86.23 / 0.00046.60 / 85.19 / -0.099
SAPLMA66.59 / 35.07 / 0.30170.88 / 31.61 / 0.38666.37 / 82.84 / 0.45157.08 / 62.93 / 0.08262.35 / 90.82 / 0.44046.66 / 85.00 / -0.137
ICR Probe61.32 / 26.83 / 0.17963.25 / 18.76 / 0.23054.86 / 81.11 / 0.18762.66 / 64.50 / 0.23345.64 / 85.35 / -0.10839.69 / 82.46 / -0.366
QAOD V⊥81.44 / 55.49 / 0.56374.35 / 35.75 / 0.44568.36 / 85.46 / 0.47168.23 / 74.47 / 0.41164.04 / 91.42 / 0.45757.45 / 88.74 / 0.179
QAOD Q⊕V⊥79.61 / 52.28 / 0.53174.00 / 33.61 / 0.41766.85 / 83.25 / 0.45655.12 / 60.79 / 0.01664.66 / 91.74 / 0.47463.69 / 90.56 / 0.419

These results provide further support for QAOD's effectiveness. Together, the original and additional experiments now cover three model families: Gemma, Llama, and Qwen. Qwen3-14B and Qwen3-30B-A3B also provide a comparison between dense and mixture-of-experts architectures. This coverage offers evidence for QAOD's effectiveness across the evaluated models.

5. Why we use the final token

We chose final-token representations because they can incorporate preceding context, have been effective in prior work, and are simple to extract. Our ablation further supports this choice: the final-token readout performs best for the joint probe, while mean pooling does not provide a consistent improvement across both probes despite requiring aggregation over the full sequence.

We compared the readout choices on Qwen3-14B / TriviaQA using the same questions as the main table, with answers from a separate generation run. AUROC (%) is reported as the mean ± standard deviation across seeds.

Question representation QAnswer representation AQAOD V⊥QAOD Q⊕V⊥
Final tokenFinal token86.64 ± 0.2590.42 ± 0.19
Final tokenToken at 1/2 of the answer length72.95 ± 0.5187.48 ± 0.53
Final tokenToken at 2/3 of the answer length85.12 ± 0.8989.10 ± 0.20
Final tokenMean over answer tokens86.78 ± 0.6787.97 ± 0.41
Mean over question tokensFinal token86.78 ± 0.6790.17 ± 0.24
Mean over question tokensMean over answer tokens89.01 ± 0.2288.96 ± 0.47

Mean pooling improves QAOD V⊥, whereas the final-token readout remains best for QAOD Q⊕V⊥, supporting its use as a simple default. Given the broadly comparable performance of the two choices, we use the final-token readout to reduce computation.

6. Paper organization and presentation

Thank you for your suggestions on the paper's organization and presentation. In the revision, we will add the missing cross-reference to Table 1, improve the clarity of Figure 1, define the notation before Equation (2), clarify that the mean on line 170 is taken over training samples, and clarify the description of P_false.

Reviewer_N9r1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces QAOD, a single-pass, white-box hallucination detector for short-form question answering (in LLMs). The main idea introduced in the paper is, given a prompt and its corresponding main direction in the embedding space (of each hidden layer), one can them elide such direction from the answers to study the question-agnostic features and hence to efficiently detect hallucinations in OOD settings. To do so, the final-token hidden state of the generated answer (per layer) and of the prompt are analysed and compared, forming a question-orthogonal residual v⊥v_\perp. After filtering the most informative layers/neurons under a diversity-penalised Fisher scorer/Fisher importance, a two-hidden-layer MLP is used to analyse the OOD vector (created by the orthogonal residual) or the in-domain vector (formed by OOD vector + the question features). Validation covers four LLMs (gemma-2-2b, Llama-2-7B-chat, Qwen3-14B, Qwen3-30B-A3B) and four QA benchmarks (TriviaQA, SQuAD, NQ, BioASQ) with responses labelled with an LLM-as-a-judge (DeepSeek-V3) against the reference answers (token-F1 ≥\geq 50 for SQuAD), audited manually on 300 samples per dataset (Appendix A.3). The in-domain vector is reported to achieve the best in-domain AUROC on all 16 model–dataset pairs (Table 1), while the OOD vector is claimed to preserve transferability, offering the best zero-shot AUROC among the white-box detectors evaluated with a margin of 6.27–14.22 points over the strongest of them (and the smallest drop from in-domain to OOD). These results are complemented by leave-one-domain-out and GSM8K transfer (Table 2), centroid-shift and CKA analyses (Fig. 3, Table 3), ablations (Tables 4–5), and a detection overhead of about 2% of generation time (Fig. 4). Code for the QAOD pipeline is provided.

Critical strengths
  1. The idea is simple, novel and inexpensive. Projecting the question direction out of the answer representation (Eq. (1)) is conceptually minimal, requires no additional generation, and costs about 0.0126 s per query against 0.64 s of generation (Fig. 4). This is consequential to the assessment because, were the reported gains confirmed, the method would be deployable at negligible cost, unlike sampling-based detectors.
  2. The focus is explicitly centered on out-of-distribution detection. The paper targets the known weakness of white-box probes under domain shift with a dedicated orthogonal-only probe, a source-to-target protocol (Section 4.3, Fig. 2, Table 2), and representational evidence for the reduced domain sensitivity of v⊥v_\perp (Fig. 3, Table 3). The released code confirms that cross-domain runs use a fixed final epoch rather than model selection on the target domain (train_qaod.py, L952–954); to the best of the reviewer's understanding, the OOD AUROCs are therefore free of target-label leakage. This is the most distinctive contribution of the paper, and the main reason the reviewer regards the work as publishable, provided the results are confirmed.
Critical weaknesses
  1. The evaluation protocol does not allow the reported margins to be taken at face value. Three main issues compound.

    • The class imbalance is severe, varies in direction across datasets, and is never reported. The positive class is the hallucination class (train_qaod.py, L65 and L92). Since, under the protocol of Appendix A.4, the F1 threshold is optimised on the evaluated split and the sweep includes the "flag everything" threshold (L663), every F1 in Table 1 is at least 2π/(1+π)2\pi/(1+\pi), where π\pi is the hallucination rate; the smallest F1 per column therefore should bound π\pi from above at most about 16% on TriviaQA and below 17% on SQuAD, whereas NQ and BioASQ admit rates of up to 70% and 53%, respectively. Under skewed distributions, AUROC says little about operating performance, and F1 on the hallucination class mixes detection quality with prevalence. On gemma-2-2b/NQ, all eleven methods fall within 82.08–85.68 F1 whilst their AUROC spans 49.90–81.42; on Qwen3-14B/SQuAD, the joint probe's F1 of 32.09 at an AUROC of 85.17 implies that precision or recall is at most 32%.
    • No confusion matrices or imbalance-aware metrics are reported, making it impossible to assess whether the probe leans towards flagging hallucinations or genuine answers, although the pipeline already records the confusion-matrix counts and the positive-class ratios of both splits (train_qaod.py, L637–651 and L1388–1389).
    • No independent test set is adopted, and the F1 threshold is selected on the same split on which it is reported (Appendix A.4; train_qaod.py, L1271–1273, where the report itself records "independent_test": False, L1391), which does not appear completely fair to the reviewer (even if the same methodology were applied to all baselines). In addition, in-domain runs select the training epoch with the highest AUROC on that same split (checkpoint_selection="auto", L1249–1257 and L1488), a step not disclosed in Appendix A.4; the in-domain AUROCs of Table 1 are therefore also maxima over 30 epochs on the evaluation data, whilst some in-domain margins are as narrow as 1.60–1.79 points (Qwen3-14B and Qwen3-30B-A3B on SQuAD).

    As such, the central in-domain claim (best AUROC on all 16 pairs) rests on data whose meaning under imbalance and selection bias cannot be judged in the present state. The reviewer regards as necessary the reporting of per-split hallucination rates, confusion matrices and imbalance-aware metrics (e.g., AUPRC), together with the selection of thresholds and epochs on a development split held out from the training data, and the reporting of all metrics on an untouched test split.

  2. The comparison with the state of the art is neither transparent nor complete.

    • The baseline protocol is undocumented and cannot be verified. Section 4.1 lists nine baselines, and Appendix A.4 states only that their original settings were used. Assuming the trained baselines (P$_{ik}$, MIND, SAPLMA, ICR Probe) were retrained on the same data as QAOD, it remains unclear how these were trained (layers, epochs, checkpoint selection), and whether all baselines received the same threshold optimisation as QAOD. The released code covers QAOD only (which is clearly acceptable), yet no script produces any baseline number, and no further explanation is provided.
    • Table 1 is inconsistent with a uniform F1 protocol. On Qwen3-14B/NQ, the F1 of PPL (49.47) should bound the hallucination rate at 32.9%; at that rate, the F1 of LLM-Check (77.03) requires an AUROC of at least 62.6, since AUROC ≥TPR⋅(1−FPR)\geq \mathrm{TPR}\cdot(1-\mathrm{FPR}) at any operating point, yet 53.16 is reported. Both rows cannot result from F1-optimal thresholds on the same labels.
    • Recent alternatives are omitted or relegated to the appendices. Advanced internal-state probes (PRISM, ReDeEP) and gray-box hybrids (INSIDE with its EigenScore, cited as Chen et al., 2024a, but not compared) are absent from the main comparison. PRISM is evaluated only in Appendix B, on Llama-2-7B-chat with a fixed verification template (Table 8), and MHAD only as an appendix reproduction (Table 9).
    • The OOD claim is framed against trained white-box probes only. SINdex, however, should require no training, so its BioASQ AUROC should effectively be zero-shot; and it attains 79.19, 80.50 and 81.37 on Llama-2-7B-chat, Qwen3-14B and Qwen3-30B-A3B (Table 1), above the OOD AUROC of QAOD_{V_\perp} (75.63, Table 2; 77.68 and 77.21, Table 7). Fig. 2, whose caption states that QAOD_{V_\perp} "achieves best OOD AUROC", omits all training-free methods. The reviewer acknowledges the cost gap (5.15 s against 0.0126 s per query, Fig. 4), but this accuracy–cost trade-off should be stated explicitly in the text and in the caption.

    Clarifying these points is essential because the contribution is positioned as an improvement over the state of the art, which cannot be confirmed without a transparent and complete comparison. The baseline protocols and code should be documented (if not released), the same threshold and model-selection procedure should be applied to all methods, and the recent alternatives above should enter the main comparison.

Questions for authors

Clarification requests (answers could materially affect the assessment):

  1. What is the hallucination rate of the training and evaluation splits for each model–dataset pair, and what are the confusion matrices at the reported operating points?
  2. For each baseline, on which data was it trained, with which layer(s), number of epochs and checkpoint-selection rule, and was its F1 threshold optimised on the evaluation split as for QAOD?
  3. Why were the F1 threshold and the in-domain training epoch selected on the evaluation split rather than on a held-out development split, and how do the in-domain results of Table 1 change when both are selected on such a split and all metrics are reported on an untouched test set?
  4. Why were PRISM (with its prompt-selection procedure), ReDeEP and INSIDE/EigenScore not included in the main comparison?
  5. How does QAOD${V\perp}$ compare with training-free detectors such as SINdex in the zero-shot BioASQ setting, once detection cost is accounted for? Or shall they be excluded from the comparison and, if so, why?

Suggestions (for a revision, not future work):

  • Report imbalance-aware metrics (e.g., AUPRC) and confusion matrices alongside AUROC and F1.
  • Adopt a train/development/test protocol applied identically to QAOD and to all baselines.
9 comments

Thank you for checking the code line by line and raising concerns about several reported results. We have checked the run reports for each method and conducted additional experiments. We address your questions below.

1. Correcting F1 and reporting hallucination rates

The inconsistent F1 values resulted from different threshold-selection rules across methods. We have now standardized F1 reporting by selecting each method’s threshold to maximize hallucination-class F1 on the training set and holding it fixed during evaluation. When reproducing the baselines, we retained their training and scoring settings wherever possible. As a result, some baseline F1 scores used their original setting like Youden's criterion or a fixed threshold, whereas QAOD used a threshold that maximized F1. For example, the PPL F1 of 49.47% on Qwen3-14B / NQ that you noted was computed using a Youden threshold. We have corrected this inconsistency: the revised table will use the maximum-F1 rule for all methods, with thresholds selected on the training set. For Qwen3-14B / NQ, the thresholds and TP, FP, TN, and FN counts under this unified protocol are given below.

MethodThreshold selected on the training setTP / FP / TN / FNF1 with the threshold held fixed (%)
PPL1.02,257 / 1,353 / 0 / 076.94
Logit Entropy0.02,257 / 1,353 / 0 / 076.94
SINdex0.0285388554815725061,982 / 745 / 608 / 27579.53
P_false1.12817895412445072,094 / 822 / 531 / 16380.96
LLM-Check-184.539520263671882,255 / 1,348 / 5 / 276.96
P_ik0.44074360167864271,620 / 613 / 740 / 63772.16
MIND0.438570439815521241,700 / 556 / 797 / 55775.34
SAPLMA0.47407075762748721,763 / 553 / 800 / 49477.10
ICR Probe0.44921338558197021,825 / 709 / 644 / 43276.18
QAOD V⊥0.370013356208801271,718 / 498 / 855 / 53976.82
QAOD Q⊕V⊥0.5097414851188661,847 / 577 / 776 / 41078.91

We also report the requested hallucination rates for each model–dataset pair.

ModelDatasetTraining split: incorrect / total (rate)Validation split: incorrect / total (rate)
Gemma-2-2BTriviaQA1,257/10,000 (12.57%)416/3,074 (13.53%)
SQuAD1,754/10,000 (17.54%)850/5,928 (14.34%)
NQ7,233/10,000 (72.33%)2,594/3,610 (71.86%)
BioASQ1,577/3,054 (51.64%)532/1,027 (51.80%)
Llama-2-7B-chatTriviaQA1,221/10,000 (12.21%)411/3,074 (13.37%)
SQuAD1,724/10,000 (17.24%)755/5,928 (12.74%)
NQ6,532/10,000 (65.32%)2,354/3,610 (65.21%)
BioASQ1,590/3,054 (52.06%)522/1,027 (50.83%)
Qwen3-14BTriviaQA788/10,000 (7.88%)273/3,074 (8.88%)
SQuAD1,001/10,000 (10.01%)305/5,928 (5.15%)
NQ6,192/10,000 (61.92%)2,257/3,610 (62.52%)
BioASQ1,190/3,054 (38.97%)386/1,027 (37.59%)
Qwen3-30B-A3BTriviaQA805/10,000 (8.05%)282/3,074 (9.17%)
SQuAD1,131/10,000 (11.31%)395/5,928 (6.66%)
NQ6,257/10,000 (62.57%)2,286/3,610 (63.32%)
BioASQ1,124/3,054 (36.80%)364/1,027 (35.44%)

2. Baseline configurations

Thank you for pointing out that the baseline descriptions in Appendix A.4 were too brief. We provide the details below. We retained each baseline's features and followed the parameter settings in its official implementation or paper.

  • PPL: Exponentiate the mean token negative log-probability of the original generated answer.

  • Logit Entropy: Compute token entropy from the language model's output distribution and take the maximum window-averaged entropy. The window size in this run was 1.

  • SINdex: Sample 10 answers per question at temperature 1.0, embed them with all-MiniLM-L6-v2, cluster them using a cosine-similarity threshold of 0.95, and compute the score.

  • P_false: Construct a self-evaluation prompt using the candidate answer and few-shot examples from the training set, with risk score sfalse=1−logP(mathrmTrue)s_{\mathrm{false}}=1-log P(mathrm{True}). This score is not a probability, so its threshold may exceed 1.

  • P_ik: Extract the final-layer hidden vector at the answer position, fit logistic regression using answer-correctness labels from the training set, and output risk scores for validation answers.

  • MIND: Read out the last token's representation from the final layer, train for 20 epochs, and retain the checkpoint with the highest validation AUROC.

  • SAPLMA: Evaluate five candidate layers at 50%, 62.5%, 75%, 87.5%, and 100% of the model's depth. Train a feedforward probe at each layer with Adam for 5 epochs, and select the checkpoint with the highest validation AUROC.

  • LLM-Check: Score answers using the mean log-determinant of the generation model's self-attention matrices.

  • ICR Probe: Average the ICR scores over answer tokens within each layer and use the scores from all layers as probe inputs. Train with Adam for 50 epochs, with a batch size of 32 and a learning rate of 5×10⁻⁴, and select the checkpoint with the highest validation AUROC.

3. Validation-set use for threshold and checkpoint selection

QAOD’s AUROC and AUPRC advantage remains on all four Qwen3-14B datasets when checkpoint and threshold selection are moved to a separate development set. In the original in-domain experiments, both QAOD and the baselines that require training used the validation set for checkpoint selection. The original table therefore compares methods under this shared protocol. We agree, however, that using the same validation set does not remove the optimistic bias introduced by model selection. To assess whether this issue affects the paper's main comparative conclusions, we conducted additional experiments on all four datasets with Qwen3-14B, separating training, development, and evaluation.

Specifically, we held out 10% of each dataset's original training split as a development set and used the remaining 90% for training. For QAOD and all baselines that require training, we selected checkpoints by development-set AUROC. Methods with candidate-layer selection also used only the development set for that choice. We then selected the threshold that maximized F1 for the hallucination class on the development set, froze both the checkpoint and threshold, and reported AUROC, AUPRC, and F1 on the original evaluation set. The evaluation set was not used for model or threshold selection in these additional experiments.

Each cell reports AUROC (%) / AUPRC (%) / F1 (%) with the threshold held fixed.

MethodTriviaQASQuADNQBioASQ
PPL53.25 / 22.95 / 13.1054.89 / 10.91 / 14.6561.81 / 76.88 / 76.9461.33 / 52.17 / 54.64
Logit Entropy53.25 / 23.61 / 12.9454.92 / 11.39 / 14.6562.06 / 78.44 / 76.9461.88 / 56.49 / 54.64
SINdex65.73 / 29.85 / 32.0874.02 / 19.72 / 22.1978.30 / 86.14 / 79.5980.50 / 71.78 / 66.51
P_false78.84 / 36.93 / 38.8458.33 / 6.96 / 11.8079.05 / 85.05 / 81.0378.66 / 67.51 / 65.32
LLM-Check61.32 / 14.05 / 17.9562.64 / 7.44 / 12.5053.16 / 65.60 / 77.0059.67 / 44.08 / 55.47
P_ik79.02 / 45.12 / 47.5470.39 / 16.04 / 18.9967.99 / 76.51 / 76.9073.72 / 61.49 / 62.30
MIND81.60 / 45.77 / 48.0473.81 / 15.08 / 20.3276.65 / 85.18 / 78.5878.76 / 68.92 / 66.26
SAPLMA83.38 / 45.44 / 46.3083.68 / 24.51 / 30.5279.45 / 86.63 / 79.8480.28 / 69.76 / 67.29
ICR Probe79.07 / 35.44 / 37.0175.49 / 20.27 / 23.2373.63 / 82.47 / 78.2178.48 / 67.34 / 65.44
QAOD V⊥86.54 / 57.23 / 55.3881.56 / 19.55 / 29.8079.30 / 86.57 / 80.2276.14 / 68.33 / 64.14
QAOD Q⊕V⊥89.74 / 64.03 / 60.4484.99 / 25.80 / 29.9482.42 / 88.52 / 81.2684.50 / 78.09/ 70.05

Under this selection protocol, the joint QAOD probe retains the highest AUROC and AUPRC on all four datasets. Its AUROC exceeds that of the strongest baseline on each dataset by 6.36, 1.31, 2.97, and 4.00 percentage points, respectively. Thus, in these four settings, QAOD's advantage on ranking metrics remains when model and threshold selection are moved to a held-out development set. This provides further support for its relative advantage and for the analyses based on the original Table 1.

4. Why PRISM, ReDeEP, and INSIDE/EigenScore were not included as baselines

  • PRISM. Appendix B of the original submission already discusses prompt selection and reports related experiments. We did not include PRISM as a main baseline because its results depend strongly on the candidate templates, whose generation is stochastic. A selected template is optimal within that candidate pool under the chosen metric, but this does not establish that it remains optimal for other datasets or models. We therefore evaluated the fixed template selected for Llama-2-7B-chat in the original PRISM paper. These experiments also favor QAOD.

  • ReDeEP. ReDeEP is designed for retrieval-augmented generation and detects hallucinations by examining how the model uses retrieved evidence. Our main comparison evaluates short-answer correctness against reference answers, and the original inputs across the four datasets do not provide a uniform set of retrieved evidence. The original ReDeEP setting therefore differs from our main four-domain comparison.

  • INSIDE/EigenScore. We have now added EigenScore to the comparison. QAOD Q⊕V⊥ achieves higher AUROC on all four Qwen3-14B datasets using a single generated answer per question, whereas EigenScore requires multiple sampled answers. EigenScore measures disagreement across multiple generations for the same question using the covariance spectrum of their last-token representations at an intermediate layer. In the original main table, SINdex served as the baseline for semantic consistency across multiple sampled outputs, while SAPLMA served as the baseline using a single answer's internal representation. We therefore did not initially include EigenScore. To provide a more direct comparison, we have now evaluated it on Qwen3-14B. AUROC (%) is reported below.

MethodTriviaQASQuADNQBioASQ
EigenScore60.9272.9077.5378.99
QAOD V⊥87.4281.3479.9878.39
QAOD Q⊕V⊥90.1385.1782.6985.69

5. Comparison with SINdex, which requires no detector training

SINdex requires no detector training, so its BioASQ AUROC can serve directly as a zero-shot OOD baseline. Figure 2 in the original submission focused on single-pass white-box methods that require detector training and therefore did not include SINdex. We agree that SINdex outperforms QAOD V⊥ in some model–dataset settings. For example, on Qwen3-14B / BioASQ, SINdex achieves 80.50% AUROC, compared with 76.73% for QAOD V⊥. However, SINdex requires repeated sampling and substantially more detection time, as does EigenScore discussed above. In Figure 4, on Qwen3-14B / TriviaQA with an A100 and a batch size of 256, QAOD and SINdex with 10 samples incur additional costs of 0.0126 and 5.15 seconds per question, respectively, a roughly 400-fold difference. Considering both detection performance and online overhead, QAOD may therefore be a better choice.

Reviewer_zMqcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Question-Answer Orthogonal Decomposition (QAOD), a framework that projects away the question-aligned direction from the answer representation to obtain a question-orthogonal component for cross-domain correctness prediction. It uses diversity-penalized Fisher scoring to select layers and uses Fisher importance to select discriminative neurons. Based on the selected informative signals, it utilizes two complementary probing strategies to ensure in-domain detection and cross-domain generalization. The QAOD is evaluated on five benchmarks (TriviaQA, SQuAD, NQ, BioASQ, and GSM8K) with four LLMs.

Critical strengths
  1. The cross-domain generalization of hallucination detection models is a topic worthy of investigation. The hypothesis that removing domain-specific knowledge from the representation could improve cross-domain transfer performance is reasonable.
  2. The experiments cover several LLMs of varying sizes. Evaluations are conducted in-domain using general QA datasets (TriviaQA, SQuAD, and NQ) as well as the domain-specific dataset BioASQ; additionally, GSM8K is utilized for cross-domain evaluation. Compared to the baselines, QAOD achieved the best performance in both in-domain and cross-domain scenarios. Furthermore, the experimental results validate the motivation for the model design: in in-domain evaluation, the Q⊕V⊥Q \oplus V_\perp consistently outperforms V⊥V_\perp alone, whereas in cross-domain settings—whether transferring from a general domain to a specific domain or between different specific domains—relying solely on V⊥V_\perp achieves better results.
Critical weaknesses
  1. The method’s implementation is relatively simple and straightforward: it first removes the question-aligned component from the answer representations, then identifies informative layers and neurons, and finally feeds the concatenated feature vector into a lightweight MLP for hallucination detection. While the idea of considering question-answer orthogonal decomposition is novel, the implementation lacks technical novelty.
  2. Although the experiments cover two distinct domains—BioASQ and GSM8K—the results reported to support the claim of superior cross-domain transferability remain somewhat narrow. Furthermore, as the authors acknowledge in the limitations section, the current evaluation focuses on knowledge-intensive QA tasks, where factual errors can be traced to specific misapplied entities or relationships. Extending QAOD to tasks such as open-ended generation would enhance the soundness; however, leaving this for future work is acceptable.
Questions for authors

None

2 comments

1. Implementation compactness and effectiveness

QAOD is based on the novel question–answer orthogonal decomposition and achieves effective hallucination detection through a compact architecture with low computational overhead. The experiments show that this compact and simple architecture brings practical gains. Table 4 shows that question-orthogonal residuals outperform both the original answer representations and residuals obtained by removing a random direction on both Qwen models. Table 5 further supports the layer-selection strategy: with the same five-layer budget on Qwen3-14B / TriviaQA, adding a diversity penalty based on residual similarity improves AUROC from 84.92% with pure Fisher ranking to 87.27%. The simple implementation also keeps detection overhead low: as shown in Figure 4, the additional detection time is approximately 2% of generation time on the same model and dataset.

2. Cross-domain evidence

Thank you for pointing out the limited coverage of the original cross-domain experiments. We have expanded the evaluation to cover multiple transfer directions on each of Gemma-2-2B, Qwen3-14B, and Qwen3-30B-A3B, and the expanded results support QAOD’s transfer advantage. The original Table 2 reported six transfer directions on Llama-2-7B-chat. In addition to the BioASQ results already reported in Figure 2, the new experiments cover the other three leave-one-domain-out QA settings and two transfer paths to GSM8K. AUROC % is reported below.

MethodSQuAD+NQ+BioASQ →\rightarrow TriviaQATriviaQA+NQ+BioASQ →\rightarrow SQuADTriviaQA+SQuAD+BioASQ →\rightarrow NQTriviaQA+SQuAD+NQ →\rightarrow GSM8KBioASQ →\rightarrowGSM8K
Gemma-2-2B
P_ik65.0265.2260.5856.1448.67
MIND59.4659.9063.0854.8256.07
SAPLMA72.8371.9571.8755.0554.99
ICR Probe51.0260.5665.3349.3148.31
QAOD V⊥79.3678.3473.6759.4756.91
QAOD Q⊕V⊥78.1376.0272.6256.9954.89
Qwen3-14B
P_ik65.6165.2059.1264.7654.81
MIND64.8963.2166.0772.6271.07
SAPLMA68.9773.3376.5565.3557.19
ICR Probe66.1661.7559.9842.4936.14
QAOD V⊥83.5681.1879.4880.6371.24
QAOD Q⊕V⊥83.3380.5579.3778.9371.29
Qwen3-30B-A3B
P_ik78.4473.6967.8067.9058.51
MIND69.7170.1065.7668.2969.12
SAPLMA74.0874.9475.4765.2960.18
ICR Probe70.2373.4668.3248.2853.85
QAOD V⊥84.7884.8880.6971.7469.07
QAOD Q⊕V⊥84.9983.6381.7872.9070.31

3. Limitations in task coverage

Our current evaluation supports QAOD’s effectiveness for answer-correctness detection across knowledge-intensive QA and mathematical reasoning settings. These experiments examine transfer across both knowledge domains and task types, providing evidence for the value of question–answer orthogonal decomposition within the evaluated scope. As discussed in the Limitations section, extending QAOD to more complex tasks, including open-ended generation, remains a direction for future work.

Thanks for your response. I have no further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.