- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on real-time hallucination detection, aiming to detect hallucinations right after the hallucinated sentence is produced. The authors propose ProbeTok, a supervised learning method that learns an embedding and a linear head to turn the generator LLM into a detector. As a result, the system is easy to switch between generation and hallucination detection with a minor computational overhead. The authors conduct experiments on two datasets and compare ProbeTok against 6 baselines. The results show that ProbeTok outperform baselines in most cases and with small latency, suggesting its effectiveness.
Critical strengths
- The motivation is clear and sound. Real-time hallucination detecting during generation is important, which saves cost and prevents error accumulation.
- The paper proposes a simple solution by training an embedding and a linear head to turn the generator LLM into a detector.
- ProbeTok can identify hallucinations with a high AUROC and low computational overhead, which is good for real-world scenarios.
Critical weaknesses
- ProbeTok is trained with sentence-level labels, where high-quality annotations are hard to obtain in real-world scenarios. It would be helpful to know how much data is needed to achieve good performance.
- If sentence is a hallucination, it is also passed as input when detecting hallucination of sentence . It is unclear to me how ProbeTok differentiate hallucination of the target sentence and former generated sentences.
- ProbeTok detects hallucination based on partial generation. The performance may vary across different numbers of sentences per output and the positions of the hallucinated sentence (e.g., the first or the last sentence). Such analyses are missing in the paper.
- ProbeTok is similar to TSV [1], in which a representation is learnt and added to a certain transformer layer to make the output representation of truthful and hallucinated generation more separable. This paper overlooks this related work and should discuss it and/or compare with it.
[1]: Park et al. 2025. Steer LLM Latents for Hallucination Detection.
Questions for authors
- Figure 2 is not very intuitive. A line plot is not suitable when the x-axis is categorical, and the meaning of the matched line is not easy to understand at first glance. Overall, it requires readers to put extra effort into understanding the message of this figure.
- Presentation of the appendix can be improved. Since the appendix is long, a table of content would be helpful to find a certain content. Also, there are many cases where one table contains multiple settings (e.g., A5-12, A14, and A19-22), making it hard to understand and interpret the results. The authors can selectively show results or separate results into different tables to increase readability.
- It is unclear to me why the authors focus only on hallucination detection in RAG systems. The learnt embedding does not depend on input context and should also work for hallucination detection without retrieved documents. I wonder how well does ProbeTok work on other hallucination detection datasets.
8 comments
W2: If sentence i is a hallucination, it is also passed as input when detecting hallucination of sentence j. It is unclear to me how ProbeTok differentiate hallucination of the target sentence and former generated sentences.
Response to W2: In short, during sentence-level supervised learning, to minimize the prediction error, the probe token and readout head learn on their own to detect only whether the immediately preceding sentence is hallucinated.
To verify this empirically, we sample 3.5K test target sentences from responses with an earlier hallucinated sentence. These target sentences are randomly picked from any position after the hallucinated sentence. We keep each target sentence unchanged and evaluate three prefix settings in the table below.
Table R7. Target-sentence AUROC under three prefix settings on 3.5K test target sentences. We report 8-condition averages (mean ± std over 3 seeds), where Std is the standard deviation across the three prefix settings.
| Detector | Original Prefix AUROC | Remove Hallucinated Sentence AUROC | Remove Non-Hallucinated Sentence AUROC | Std |
|---|---|---|---|---|
| ProbeTok-last | 0.715 ± 0.015 | 0.711 ± 0.024 | 0.704 ± 0.018 | 0.006 |
| ProbeTok-sel | 0.756 ± 0.005 | 0.761 ± 0.002 | 0.753 ± 0.007 | 0.004 |
-
Original Prefix: We keep all sentences in the prefix unchanged.
-
Remove Hallucinated Sentence: We remove the hallucinated sentence nearest to the target sentence in the prefix.
-
Remove Non-Hallucinated Sentence: We remove the non-hallucinated sentence nearest to that hallucinated sentence in the prefix.
If earlier hallucinations misled ProbeTok, AUROC on the target sentence would increase noticeably once we remove them. Instead, across all three prefix settings, Std is only 0.006 for ProbeTok-last and 0.004 for ProbeTok-sel. In particular, when we remove an earlier hallucinated sentence, we find that AUROC changes very little, from 0.715 to 0.711 for ProbeTok-last and from 0.756 to 0.761 for ProbeTok-sel, and both changes are statistically insignificant under 95% paired bootstrap intervals. This confirms that ProbeTok focuses on the target sentence rather than earlier hallucinations in the prefix.
W3: ProbeTok detects hallucination based on partial generation. The performance may vary across different numbers of sentences per output and the positions of the hallucinated sentence (e.g., the first or the last sentence). Such analyses are missing in the paper.
Response to W3: To address the reviewer's concern, we evaluate ProbeTok-last at different sentence positions and output lengths across all 8 conditions. As a result, we find that its AUROC does vary. However, this variation could come from two sources: either the dataset itself has different difficulty across positions and lengths, or our learned probe token is sensitive to position and length.
To further analyze what causes this variation, we use Last-H from our paper as a reference. Since Last-H does not insert a probe token and only applies a linear head to each sentence's final hidden state, its mechanism is not affected by sentence position or output length. In other words, if Last-H also shows variation, the cause should be the dataset itself.
Therefore, we compute the difference between ProbeTok-last and Last-H and test whether this difference changes significantly across sentence positions and output lengths.
1. Performance across sentence positions:
Table R8. Sentence-level AUROC across sentence positions in the response. We report 8-condition averages over 3 seeds.
| Detector | Position 1 | Position 2 | Position 3 | Position 4 | Position 5 | Position 6+ | Slope | 95% CI |
|---|---|---|---|---|---|---|---|---|
| Last-H | 0.733 | 0.778 | 0.744 | 0.762 | 0.731 | 0.711 | −0.0066 | [−0.0094, −0.0039] |
| ProbeTok-last | 0.780 | 0.794 | 0.776 | 0.771 | 0.749 | 0.740 | −0.0067 | [−0.0093, −0.0042] |
| ProbeTok-last − Last-H | +0.046 | +0.016 | +0.033 | +0.010 | +0.018 | +0.029 | −0.0001 | [−0.0023, +0.0020] |
-
Slope: We fit a linear regression to measure the average AUROC change per 1-step increase in sentence position or output length.
-
95% CI: This is the 95% bootstrap confidence interval of Slope. If this interval includes 0, there is no statistically significant trend.
The table above shows that Last-H and ProbeTok-last decrease at nearly the same rate at later positions, with slopes of −0.0066 and −0.0067. In contrast, the difference between ProbeTok-last and Last-H has a slope of −0.0001, and its 95% CI [−0.0023, +0.0020] indicating insignificant. Therefore, the lower AUROC at later positions comes from dataset difficulty rather than sensitivity of our probe token.
2. Performance across numbers of sentences per output:
Table R9. Sentence-level AUROC across different numbers of sentences per output. We report 8-condition averages over 3 seeds.
| Detector | 1–3 Sentences | 4 Sentences | 5 Sentences | 6 Sentences | 7 Sentences | 8+ Sentences | Slope | 95% CI |
|---|---|---|---|---|---|---|---|---|
| Last-H | 0.680 | 0.754 | 0.704 | 0.740 | 0.738 | 0.744 | +0.0049 | [+0.0021, +0.0077] |
| ProbeTok-last | 0.699 | 0.744 | 0.741 | 0.769 | 0.775 | 0.775 | +0.0072 | [+0.0046, +0.0095] |
| ProbeTok-last − Last-H | +0.019 | −0.010 | +0.037 | +0.029 | +0.037 | +0.032 | +0.0023 | [−0.0001, +0.0047] |
We group outputs of 1 to 3 sentences together because our datasets have relatively few such short responses. The table above shows that both Last-H and ProbeTok-last achieve higher AUROC on longer outputs, with slopes of +0.0049 and +0.0072. Meanwhile, the difference between ProbeTok-last and Last-H has a slope of +0.0023, and its 95% CI [−0.0001, +0.0047] includes 0. Therefore, our probe token has no statistically significant sensitivity to output length. We have added both analyses to the Appendix of our revised paper.
Response to W1: How much training data does ProbeTok need? First, the training set used in our paper contains only about 2K samples per condition, which is already small. However, to further address the reviewer's question, we randomly subsample 10%, 25%, 50%, and 100% of this training set to retrain ProbeTok. The evaluation results are shown in the table below.
Table R6. Test sentence-level AUROC across training set fractions vs. the two best baselines. We report 8-condition averages (mean ± std over 3 seeds for ProbeTok). Best results in each column are in bold.
ProbeTok-last: We read the probe token from the last layer, which is the version in our paper.
ProbeTok-sel: This is our further improved version of ProbeTok. Since last-layer hidden states are specialized for next-token prediction rather than hallucination detection, we select the readout layer on the validation set.
In the table above, both Lettuce-R and Lettuce-P were pretrained on about 15K human-labeled samples, where Lettuce-R runs in real time while Lettuce-P is post-hoc and requires the complete response after generation. Even though they have the advantage of much more training data and Lettuce-P also has the advantage of post-hoc detection, ProbeTok-sel reaches 0.716 and already matches real-time Lettuce-R at 0.717 when we use only 10% of our training set, which is about 200 samples per condition. Moreover, when we use 25% of our training set, which is about 500 samples per condition, ProbeTok-sel reaches 0.736 and outperforms both Lettuce-R and post-hoc Lettuce-P at 0.732.
Second, in our paper, we use LLM-as-a-judge (GPT-5.6 SOL) to generate sentence-level labels. To verify whether these LLM-as-a-judge labels are accurate, we audited 71 sentences in our paper and found 57/71 initial human-LLM agreement. After we carefully re-read the retrieved documents, we found that 12 of the 14 disagreements were human reading errors, and agreement rose to 69/71 once we corrected them.
To further verify label quality at full scale, we also used a stronger independent model, GPT-6 Astra, to blindly relabel all 28.5K test sentences across all 8 conditions. As a result, we found that GPT-6 Astra achieved 93.8% sentence agreement and 0.745 Cohen's κ with GPT-5.6 SOL. Therefore, we can collect high-quality sentence-level labels reliably at scale without large-scale human annotation.