acceptodds
Under review as a conference paper at ICLR 2027

DualTrace: Probing Evidence Integration for RAG Hallucination Detection

Abstract

Retrieval-augmented language models can rely on retrieved text yet still misrepresent it: context dependence alone does not establish faithfulness. We introduce DualTrace, a supervised detector built around two complementary views of evidence integration: how removing context changes a fixed response computation, and whether predictions decoded from hidden states agree across depth. DualTrace replays the same response with and without context, measuring output changes, hidden-state movement, and intermediate-to-final prediction disagreement, then pools these signals for response-level classification. On RAGTruth and HalluRAG, DualTrace improves on the evaluated final-layer controls and transfers across datasets and models. Averaging feature-family detector scores also outperforms selecting a single family, supporting their complementary predictive value. Across settings, hallucinated responses exhibit lower early uncertainty but greater late-layer disagreement. Context sensitivity further correlates with fixed-response log-probability changes under activation replacement. Together, these results connect a practical detection representation with measurable changes in evidence-sensitive computation and prediction agreement across depth.

Primary area:
interpretability and explainable AI
Keywords:
retrieval-augmented generation, hallucination detection, hidden states, internal representations, mechanistic analysis, model interpretability, faithfulness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NpxLOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces DualTrace, a method to detect hallucination in RAG systems by measuring how an LLM's internal signals change when it generates the same answer with/without context. The paper measures such signals in three aspects, including output changes, hidden state movement, and intermediate-to-final prediction disagreement, computes their statistics, and aggregates them over the generation index into a feature. At the end, an XGBoost model is trained on top of the feature and predicts hallucination. The authors experiment on 2 datasets and compare DualTrace against 9 baselines. Experiments show that DualTrace outperforms baselines in most settings, suggesting its effectiveness. Extensive ablations and analyses are conducted to justify the design choice.

Critical strengths

None

Critical weaknesses
  1. The paper is hard to read.
    • The authors introduce three feature families; each of them has multiple measurements. However, those measurements are introduced in a highly unstructured way, making it hard for readers to understand what measurements are actually used as a feature. In addition, many features are introduced in vague language; it is difficult to understand how those features are obtained. Please clarify questions 1-4. With this many measurements per feature family, it is also hard to understand what measurements are effective and what are redundant. Please also justify the selection of each measurement. Currently, they seem to be chosen arbitrarily.
    • The authors tried to show many experiments and analyses. However, the motivation and experimental setup of some experiments are not clearly introduced, making it hard to understand and interpret the results. Please clarify questions 5-12. In addition, the presentation of Tables 2 and 3 is not well-organized (and tables in appendix as well), further increase the difficulty of understanding the experiments.
  2. The novelty and motivation are unclear to me. DualTrace follows prior work on hallucination detection in RAG systems and proposes different features and measurements. Some of the features are similar to those introduced in prior work, e.g., conflict and instability in LUMINA [1]. However, there is no enough discussion on the limitation of prior work. Thus, the approach seems incremental. It is unclear what technical problem DualTrace wants to solve and why their solution is novel.
  3. The paper didn't justify how many performance gain is come from the constructed feature and how many is come from XGBoost.

[1]: Yeh et al. 2026. Detecting Hallucinations in RAG System with Context–Knowledge Signals.

Questions for authors
  1. Lines 151-152: "Cosine alignment with document states, final-state differences, and path-specific norms complete the family." Could the authors provide formulas for these measurements?
  2. Lines 161-163: "Within-path summaries retain the maximum, late-layer mean, total, and variance of movement magnitudes, together with changes in direction. Additional features project states onto subspaces derived from the output head, following Hu et al. (2026)" How is late-layer defined? How are changes in direction measured? Also, the authors should describe the additional features with more details in the main paper, instead of putting them only in appendix.
  3. Lines 171-172: "We retain early, middle, and late KL, early, middle, and final entropy, and top-prediction agreement." The early, middle, and late is defined in appendix, which is hard to find. The authors should mention it explicitly in the main paper. Also, how is top-prediction agreement measured?
  4. Line 187: "Four statistics for 36 token features plus response length yield a 145-dimensional vector." Why do the authors use response length as a feature along with those internal measurements? In addition, why use four statistics? How does each statistic contribute to the performance?
  5. Lines 261-262: "Output-gap uses nine final-layer differences, including target logit, log probability, entropy, rank, JSD, and symmetric KL. Output-all adds ten path-specific values such as target likelihood and entropy." The authors didn't show the full list of features in the main paper, but put that information in appendix, increasing the burden of understanding this text. In addition, it is unclear to me why the authors choose to evaluate these specific features.
  6. What ranking features are used in full DualTrace but not used in Output-all and Output-gap?
  7. What's the motivation of matched-method comparison?
  8. Lines353-356 and 363-369: Which LLM and dataset are used in these two experiments?
  9. Lines357-361: Which dataset is used in this experiment?
  10. Line 364: "Across six settings with responses shared across conditions..." What are the six settings?
  11. Line 377: "The Compact subset, with 11 rather than 36 features, reaches 0.9078" The authors didn't show the full list of features in the main paper, but put that information in appendix, increasing the burden of understanding this text. In addition, it is unclear to me why the authors choose to evaluate these specific features.
  12. Eq 7: What are Cˉ\bar{C}, Dˉ\bar{D}, and Iˉ\bar{I}?
Reviewer_x2b4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DualTrace, a supervised white-box detector for response-level hallucination in retrieval-augmented generation. The same observed response is replayed under teacher forcing with the retrieved documents (CTX) and without them (NOCTX), so the two paths differ only in the evidence supplied. Three feature families are extracted from these paired computations. Conflict measures CTX–NOCTX output differences and document alignment. Drift measures differences in layer-wise hidden-state displacement plus output-head subspace projections. Instability measures logit-lens disagreement between intermediate and final layers on the CTX path. The 36 token-level features are pooled into a 145-dimensional vector and classified with XGBoost. On RAGTruth and HalluRAG, DualTrace achieves the highest AUROC/AUPRC among nine baselines across eight broad-benchmark settings, improves on final-layer output controls by about 0.045 macro AUROC, and transfers across datasets and models. Further analyses report that hallucinated responses show lower early-layer uncertainty but larger late-layer disagreement, and that context sensitivity correlates with log-probability changes under activation replacement.

Critical strengths
  1. The statistical protocol is careful: source-grouped cross-validation and bootstrap, Holm correction, thresholds calibrated only on training OOF scores, and five-seed ensembles.

  2. The controls are extensive: output-only controls under an identical pipeline, random-context paths, replay-artifact matching, false-positive analysis on faithful responses, transfer, and cost.

  3. The authors report results candidly, e.g., stating that the matched comparisons show competitive rather than uniformly superior performance.

  4. Replaying a fixed response on both paths is a sensible design that isolates the effect of evidence from differences in generated content.

Critical weaknesses
  1. Limited novelty, and no clear win over the closest methods. On the matched RAGTruth responses in Table 3, DualTrace only beats native ReDeEP significantly. It ties LUMINA+XGB (0.8924 vs. 0.8930), and CORTEX (0.9185) and native RAGLens actually have higher point estimates. The method mostly takes signals that ReDeEP, LUMINA, CORTEX, HARP, and logit-lens work already use and bundles them into a hand-engineered feature set for XGBoost. I suspect the big margins in Table 1 come largely from comparing a supervised classifier against mostly unsupervised or calibration-only baselines, not from a better way of capturing evidence integration.

  2. The numbers move a lot across evaluation sets. For Llama3-8B/RAGTruth alone, I count three different AUROCs: 0.9354 in Table 1, 0.9884 on the control subset in Table 2, and 0.8924 on the matched set in Table 3. The control subsets keep only responses where every feature is available, under a 4,096-token cap and hardware limits, which leaves around 780 training responses instead of roughly 2,500. I’d like the authors to tell us what they dropped and show that the filtering doesn’t make the task easier.

  3. The gains over output-only controls are mostly within noise, and the random-context results cut against the main story. In Table 2, only 3 of the 12 Full-minus-Output-all intervals exclude zero. In Table 11, the CTX/NOCTX gain on Llama3-8B/RAGTruth is about 0.006. On Llama3-8B/HalluRAG, Random/NOCTX (0.9794) actually beats the real CTX/NOCTX pairing (0.9587). If random documents work as well as the retrieved ones, the detector may be picking up a generic “context changed the computation” signal rather than anything about evidence integration.

  4. Replay artifacts seem to drive much of the signal. When you teacher-force a response that was written with documents in view and then take the documents away, many of its tokens naturally look unlikely. The authors’ own likelihood-only Artifact detector already does well in several settings (e.g., 0.9278 vs. 0.9345 for Full on Qwen3-14B/RAGTruth). After artifact matching, Full falls sharply in some cases (Mistral-7B/RAGTruth goes from 0.8529 to 0.7350). The false-positive rate on faithful responses also swings from 1.2% to 47.1% across settings, which makes me doubt how reliable the detector would be in practice.

  5. The ablations and analyses don’t support the “two axes” framing. The paper presents context sensitivity and cross-layer agreement as two complementary axes, but dropping Instability barely matters (macro AUROC goes from 0.9279 to 0.9245). The E/I quadrant patterns don’t hold up across settings: Llama3-8B/HalluRAG reverses the RAGTruth pattern, and only one of the four contrasts survives Holm correction. The lens uses layers 8/20/28 for every model regardless of depth. I also find the activation-replacement result partly circular. E already includes the CTX–NOCTX output JSD, and swapping CTX states into the NOCTX path will push its outputs toward CTX almost by construction, at least at the final block. At the non-final blocks, Qwen3-8B even shows a negative correlation.

Questions for authors
  1. How were responses and labels obtained for Qwen3-8B/14B on RAGTruth and HalluRAG? Are these the models’ own generations?

  2. Why does Random/NOCTX match or exceed CTX/NOCTX in several settings, and how is this consistent with the claim that DualTrace measures evidence integration?

Reviewer_xCf9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DualTrace proposes a well-motivated approach to RAG hallucination detection by comparing model computations with and without retrieved context and combining three complementary signals: Conflict, Drift, and Instability. Its main strengths are strong performance over final-layer baselines, comprehensive ablation studies, and promising cross-model and cross-dataset generalization. However, its superiority over recent internal-state methods is less convincing, as most matched comparisons show no statistically significant improvements. Additionally, the relationship between the proposed signals and hallucination varies across models and datasets, while activation-replacement experiments demonstrate changes in response probability rather than factual faithfulness.

Critical strengths
  • The paper has a well-motivated methodology with complementary signals. DualTrace introduces a principled approach that compares the same response with and without retrieved context, isolating the effect of evidence from variations in generated text. Combining Conflict, Drift, and Instability captures information beyond final-layer predictions. This is a meaningful contribution because context dependence alone does not guarantee factual faithfulness.

  • The paper shows strong empirical evaluation and generalization. There are multiple models and datasets being evaluated, with ablation studies, output-level controls, and transfer experiments. DualTrace achieves a macro AUROC of 0.9279 versus 0.8825 for the strongest output-only control and demonstrates promising cross-model and cross-dataset transfer. These results provide evidence for the practical value of the proposed representation.

Critical weaknesses
  • Although DualTrace outperforms output-level baselines, its advantages over recent internal-state methods are less convincing. In the matched comparisons, only the improvement over native ReDeEP is statistically significant after correction. LUMINA+XGBoost performs comparably, while CORTEX and RAGLens achieve higher point estimates in some settings. This weakens the case for a substantial empirical advance over the strongest existing approaches.

  • The relationship between context sensitivity and hallucination varies across models and datasets, making the signals difficult to interpret consistently. Furthermore, activation-replacement experiments demonstrate changes in fixed-response probability rather than improvements in factual faithfulness. The experiments support the predictive usefulness of these signals more strongly than their proposed interpretation as indicators of faithful evidence integration.

Questions for authors
  • The output-only baselines use positions predicting the observed tokens, whereas DualTrace extracts hidden-state features after consuming those tokens. Could the authors clarify how this difference affects comparison fairness and whether the performance improvements persist with position-aligned output controls?

  • In Table 3, DualTrace does not demonstrate statistically significant improvements over most competing internal-state methods, and some achieve higher AUROC. Could the authors clarify what evidence supports DualTrace's advantage over these methods, particularly when accounting for differences in supervision and classifier protocols?

1 comment
Reviewer_xCf9OpenReview ↗

Response to Authors

I appreciate the authors' rebuttal. The new position-aligned controls have addressed my concern regarding comparison fairness. However, my concern regarding the limited statistically significant improvements over closely related internal-state methods remains only partially addressed. I'm more inclined to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.