- Rating
- 2(+1)
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces DualTrace, a method to detect hallucination in RAG systems by measuring how an LLM's internal signals change when it generates the same answer with/without context. The paper measures such signals in three aspects, including output changes, hidden state movement, and intermediate-to-final prediction disagreement, computes their statistics, and aggregates them over the generation index into a feature. At the end, an XGBoost model is trained on top of the feature and predicts hallucination. The authors experiment on 2 datasets and compare DualTrace against 9 baselines. Experiments show that DualTrace outperforms baselines in most settings, suggesting its effectiveness. Extensive ablations and analyses are conducted to justify the design choice.
Critical strengths
None
Critical weaknesses
- The paper is hard to read.
- The authors introduce three feature families; each of them has multiple measurements. However, those measurements are introduced in a highly unstructured way, making it hard for readers to understand what measurements are actually used as a feature. In addition, many features are introduced in vague language; it is difficult to understand how those features are obtained. Please clarify questions 1-4. With this many measurements per feature family, it is also hard to understand what measurements are effective and what are redundant. Please also justify the selection of each measurement. Currently, they seem to be chosen arbitrarily.
- The authors tried to show many experiments and analyses. However, the motivation and experimental setup of some experiments are not clearly introduced, making it hard to understand and interpret the results. Please clarify questions 5-12. In addition, the presentation of Tables 2 and 3 is not well-organized (and tables in appendix as well), further increase the difficulty of understanding the experiments.
- The novelty and motivation are unclear to me. DualTrace follows prior work on hallucination detection in RAG systems and proposes different features and measurements. Some of the features are similar to those introduced in prior work, e.g., conflict and instability in LUMINA [1]. However, there is no enough discussion on the limitation of prior work. Thus, the approach seems incremental. It is unclear what technical problem DualTrace wants to solve and why their solution is novel.
- The paper didn't justify how many performance gain is come from the constructed feature and how many is come from XGBoost.
[1]: Yeh et al. 2026. Detecting Hallucinations in RAG System with Context–Knowledge Signals.
Questions for authors
- Lines 151-152: "Cosine alignment with document states, final-state differences, and path-specific norms complete the family." Could the authors provide formulas for these measurements?
- Lines 161-163: "Within-path summaries retain the maximum, late-layer mean, total, and variance of movement magnitudes, together with changes in direction. Additional features project states onto subspaces derived from the output head, following Hu et al. (2026)" How is late-layer defined? How are changes in direction measured? Also, the authors should describe the additional features with more details in the main paper, instead of putting them only in appendix.
- Lines 171-172: "We retain early, middle, and late KL, early, middle, and final entropy, and top-prediction agreement." The early, middle, and late is defined in appendix, which is hard to find. The authors should mention it explicitly in the main paper. Also, how is top-prediction agreement measured?
- Line 187: "Four statistics for 36 token features plus response length yield a 145-dimensional vector." Why do the authors use response length as a feature along with those internal measurements? In addition, why use four statistics? How does each statistic contribute to the performance?
- Lines 261-262: "Output-gap uses nine final-layer differences, including target logit, log probability, entropy, rank, JSD, and symmetric KL. Output-all adds ten path-specific values such as target likelihood and entropy." The authors didn't show the full list of features in the main paper, but put that information in appendix, increasing the burden of understanding this text. In addition, it is unclear to me why the authors choose to evaluate these specific features.
- What ranking features are used in full DualTrace but not used in Output-all and Output-gap?
- What's the motivation of matched-method comparison?
- Lines353-356 and 363-369: Which LLM and dataset are used in these two experiments?
- Lines357-361: Which dataset is used in this experiment?
- Line 364: "Across six settings with responses shared across conditions..." What are the six settings?
- Line 377: "The Compact subset, with 11 rather than 36 features, reaches 0.9078" The authors didn't show the full list of features in the main paper, but put that information in appendix, increasing the burden of understanding this text. In addition, it is unclear to me why the authors choose to evaluate these specific features.
- Eq 7: What are , , and ?
Response to Authors
I appreciate the authors' rebuttal. The new position-aligned controls have addressed my concern regarding comparison fairness. However, my concern regarding the limited statistically significant improvements over closely related internal-state methods remains only partially addressed. I'm more inclined to keep my score.