acceptodds
Under review as a conference paper at ICLR 2027

ProbeTok: Real-Time Hallucination Detection in RAG with One Shared Probe Token

Abstract

Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination: a response may still contain sentences that the retrieved documents do not support. Such a sentence should be detected in real time since LLMs tend to build on their earlier mistakes. However, most detectors check the complete response and therefore run only after it has been generated, and most of those that can check each sentence in real time need a separate model to read the documents. In this paper, we rethink real-time detection from the perspective of generation and find that, at the moment a sentence is generated, the generator has already read everything a detector needs and holds it in its cache, which makes it a natural basis for a real-time detector. However, its hidden states are computed to generate the next token, not to detect hallucination. Intuitively, asking the generator whether the sentence is supported by the documents would switch it from generation to detection, but asking after every sentence is costly. We therefore propose ProbeTok, which uses a single learned token, the probe token, instead of a handwritten question, and in our experiments the learned token is on average more accurate than the handwritten one. After each sentence is generated, ProbeTok feeds the probe token to the frozen generator. The embedding of the probe token is learned so that the generator computes a hidden state for detection, from which a linear head reads the hallucination score. ProbeTok then removes the probe token and restores the cache, so the generator continues the response as if no check had happened. On the RAGTruth and TRIVIA+ datasets with responses from four of the latest LLMs, ProbeTok achieves the highest average sentence-level AUROC compared with state-of-the-art detectors, and adds only 4.2% to the generation time. Moreover, a strong open LLM can use our method to check the responses of the other generators without access to their hidden states. In our experiments the best such detector is on average only 0.03 AUROC below ProbeTok running on the generator itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hallucination detection, RAG, probing, hidden states, real-time detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2tVdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on real-time hallucination detection, aiming to detect hallucinations right after the hallucinated sentence is produced. The authors propose ProbeTok, a supervised learning method that learns an embedding and a linear head to turn the generator LLM into a detector. As a result, the system is easy to switch between generation and hallucination detection with a minor computational overhead. The authors conduct experiments on two datasets and compare ProbeTok against 6 baselines. The results show that ProbeTok outperform baselines in most cases and with small latency, suggesting its effectiveness.

Critical strengths
  1. The motivation is clear and sound. Real-time hallucination detecting during generation is important, which saves cost and prevents error accumulation.
  2. The paper proposes a simple solution by training an embedding and a linear head to turn the generator LLM into a detector.
  3. ProbeTok can identify hallucinations with a high AUROC and low computational overhead, which is good for real-world scenarios.
Critical weaknesses
  1. ProbeTok is trained with sentence-level labels, where high-quality annotations are hard to obtain in real-world scenarios. It would be helpful to know how much data is needed to achieve good performance.
  2. If sentence tt is a hallucination, it is also passed as input when detecting hallucination of sentence t+1t+1. It is unclear to me how ProbeTok differentiate hallucination of the target sentence and former generated sentences.
  3. ProbeTok detects hallucination based on partial generation. The performance may vary across different numbers of sentences per output and the positions of the hallucinated sentence (e.g., the first or the last sentence). Such analyses are missing in the paper.
  4. ProbeTok is similar to TSV [1], in which a representation is learnt and added to a certain transformer layer to make the output representation of truthful and hallucinated generation more separable. This paper overlooks this related work and should discuss it and/or compare with it.

[1]: Park et al. 2025. Steer LLM Latents for Hallucination Detection.

Questions for authors
  1. Figure 2 is not very intuitive. A line plot is not suitable when the x-axis is categorical, and the meaning of the matched line is not easy to understand at first glance. Overall, it requires readers to put extra effort into understanding the message of this figure.
  2. Presentation of the appendix can be improved. Since the appendix is long, a table of content would be helpful to find a certain content. Also, there are many cases where one table contains multiple settings (e.g., A5-12, A14, and A19-22), making it hard to understand and interpret the results. The authors can selectively show results or separate results into different tables to increase readability.
  3. It is unclear to me why the authors focus only on hallucination detection in RAG systems. The learnt embedding does not depend on input context and should also work for hallucination detection without retrieved documents. I wonder how well does ProbeTok work on other hallucination detection datasets.
8 comments

W1: ProbeTok is trained with sentence-level labels, where high-quality annotations are hard to obtain in real-world scenarios. It would be helpful to know how much data is needed to achieve good performance.

Response to W1: How much training data does ProbeTok need? First, the training set used in our paper contains only about 2K samples per condition, which is already small. However, to further address the reviewer's question, we randomly subsample 10%, 25%, 50%, and 100% of this training set to retrain ProbeTok. The evaluation results are shown in the table below.

Table R6. Test sentence-level AUROC across training set fractions vs. the two best baselines. We report 8-condition averages (mean ± std over 3 seeds for ProbeTok). Best results in each column are in bold.

Detector10% Samples25% Samples50% Samples100% Samples
Lettuce-R———0.717
Lettuce-P———0.732
ProbeTok-last0.678 ± 0.0040.703 ± 0.0120.734 ± 0.0120.763 ± 0.006
ProbeTok-sel0.716 ± 0.0060.736 ± 0.0120.762 ± 0.0080.790 ± 0.002
  • ProbeTok-last: We read the probe token from the last layer, which is the version in our paper.

  • ProbeTok-sel: This is our further improved version of ProbeTok. Since last-layer hidden states are specialized for next-token prediction rather than hallucination detection, we select the readout layer on the validation set.

In the table above, both Lettuce-R and Lettuce-P were pretrained on about 15K human-labeled samples, where Lettuce-R runs in real time while Lettuce-P is post-hoc and requires the complete response after generation. Even though they have the advantage of much more training data and Lettuce-P also has the advantage of post-hoc detection, ProbeTok-sel reaches 0.716 and already matches real-time Lettuce-R at 0.717 when we use only 10% of our training set, which is about 200 samples per condition. Moreover, when we use 25% of our training set, which is about 500 samples per condition, ProbeTok-sel reaches 0.736 and outperforms both Lettuce-R and post-hoc Lettuce-P at 0.732.

Second, in our paper, we use LLM-as-a-judge (GPT-5.6 SOL) to generate sentence-level labels. To verify whether these LLM-as-a-judge labels are accurate, we audited 71 sentences in our paper and found 57/71 initial human-LLM agreement. After we carefully re-read the retrieved documents, we found that 12 of the 14 disagreements were human reading errors, and agreement rose to 69/71 once we corrected them.

To further verify label quality at full scale, we also used a stronger independent model, GPT-6 Astra, to blindly relabel all 28.5K test sentences across all 8 conditions. As a result, we found that GPT-6 Astra achieved 93.8% sentence agreement and 0.745 Cohen's κ with GPT-5.6 SOL. Therefore, we can collect high-quality sentence-level labels reliably at scale without large-scale human annotation.

W2: If sentence i is a hallucination, it is also passed as input when detecting hallucination of sentence j. It is unclear to me how ProbeTok differentiate hallucination of the target sentence and former generated sentences.

Response to W2: In short, during sentence-level supervised learning, to minimize the prediction error, the probe token and readout head learn on their own to detect only whether the immediately preceding sentence is hallucinated.

To verify this empirically, we sample 3.5K test target sentences from responses with an earlier hallucinated sentence. These target sentences are randomly picked from any position after the hallucinated sentence. We keep each target sentence unchanged and evaluate three prefix settings in the table below.

Table R7. Target-sentence AUROC under three prefix settings on 3.5K test target sentences. We report 8-condition averages (mean ± std over 3 seeds), where Std is the standard deviation across the three prefix settings.

DetectorOriginal Prefix AUROCRemove Hallucinated Sentence AUROCRemove Non-Hallucinated Sentence AUROCStd
ProbeTok-last0.715 ± 0.0150.711 ± 0.0240.704 ± 0.0180.006
ProbeTok-sel0.756 ± 0.0050.761 ± 0.0020.753 ± 0.0070.004
  1. Original Prefix: We keep all sentences in the prefix unchanged.

  2. Remove Hallucinated Sentence: We remove the hallucinated sentence nearest to the target sentence in the prefix.

  3. Remove Non-Hallucinated Sentence: We remove the non-hallucinated sentence nearest to that hallucinated sentence in the prefix.

If earlier hallucinations misled ProbeTok, AUROC on the target sentence would increase noticeably once we remove them. Instead, across all three prefix settings, Std is only 0.006 for ProbeTok-last and 0.004 for ProbeTok-sel. In particular, when we remove an earlier hallucinated sentence, we find that AUROC changes very little, from 0.715 to 0.711 for ProbeTok-last and from 0.756 to 0.761 for ProbeTok-sel, and both changes are statistically insignificant under 95% paired bootstrap intervals. This confirms that ProbeTok focuses on the target sentence rather than earlier hallucinations in the prefix.

W3: ProbeTok detects hallucination based on partial generation. The performance may vary across different numbers of sentences per output and the positions of the hallucinated sentence (e.g., the first or the last sentence). Such analyses are missing in the paper.

Response to W3: To address the reviewer's concern, we evaluate ProbeTok-last at different sentence positions and output lengths across all 8 conditions. As a result, we find that its AUROC does vary. However, this variation could come from two sources: either the dataset itself has different difficulty across positions and lengths, or our learned probe token is sensitive to position and length.

To further analyze what causes this variation, we use Last-H from our paper as a reference. Since Last-H does not insert a probe token and only applies a linear head to each sentence's final hidden state, its mechanism is not affected by sentence position or output length. In other words, if Last-H also shows variation, the cause should be the dataset itself.

Therefore, we compute the difference between ProbeTok-last and Last-H and test whether this difference changes significantly across sentence positions and output lengths.

1. Performance across sentence positions:

Table R8. Sentence-level AUROC across sentence positions in the response. We report 8-condition averages over 3 seeds.

DetectorPosition 1Position 2Position 3Position 4Position 5Position 6+Slope95% CI
Last-H0.7330.7780.7440.7620.7310.711−0.0066[−0.0094, −0.0039]
ProbeTok-last0.7800.7940.7760.7710.7490.740−0.0067[−0.0093, −0.0042]
ProbeTok-last − Last-H+0.046+0.016+0.033+0.010+0.018+0.029−0.0001[−0.0023, +0.0020]
  • Slope: We fit a linear regression to measure the average AUROC change per 1-step increase in sentence position or output length.

  • 95% CI: This is the 95% bootstrap confidence interval of Slope. If this interval includes 0, there is no statistically significant trend.

The table above shows that Last-H and ProbeTok-last decrease at nearly the same rate at later positions, with slopes of −0.0066 and −0.0067. In contrast, the difference between ProbeTok-last and Last-H has a slope of −0.0001, and its 95% CI [−0.0023, +0.0020] indicating insignificant. Therefore, the lower AUROC at later positions comes from dataset difficulty rather than sensitivity of our probe token.

2. Performance across numbers of sentences per output:

Table R9. Sentence-level AUROC across different numbers of sentences per output. We report 8-condition averages over 3 seeds.

Detector1–3 Sentences4 Sentences5 Sentences6 Sentences7 Sentences8+ SentencesSlope95% CI
Last-H0.6800.7540.7040.7400.7380.744+0.0049[+0.0021, +0.0077]
ProbeTok-last0.6990.7440.7410.7690.7750.775+0.0072[+0.0046, +0.0095]
ProbeTok-last − Last-H+0.019−0.010+0.037+0.029+0.037+0.032+0.0023[−0.0001, +0.0047]

We group outputs of 1 to 3 sentences together because our datasets have relatively few such short responses. The table above shows that both Last-H and ProbeTok-last achieve higher AUROC on longer outputs, with slopes of +0.0049 and +0.0072. Meanwhile, the difference between ProbeTok-last and Last-H has a slope of +0.0023, and its 95% CI [−0.0001, +0.0047] includes 0. Therefore, our probe token has no statistically significant sensitivity to output length. We have added both analyses to the Appendix of our revised paper.

Reviewer_3BcaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ProbeTok, a real-time sentence-level hallucination detector for RAG. After each generated sentence, a single learned token is fed to the frozen generator on top of its KV cache. A linear head reads the hidden state of this token and predicts whether the sentence is supported by the retrieved documents. The token is then removed and the cache restored, so the generated response is unchanged. Only the token embedding and the linear head are trained. The method is evaluated on prompts from RAGTruth and TRIVIA+ with responses regenerated by four open LLMs (Qwen3.5-9B, Gemma-4-E4B, Ministral-3-8B, Granite-4.2-8B) and labeled by GPT-5.6. ProbeTok reaches the highest average sentence-level AUROC (0.763 against 0.732 for the strongest baseline) with 4.2% extra generation time. The paper also shows a black-box setting in which one LLM uses its own probe token to check the responses of the others.

Critical strengths
  1. The method is cheap and does not change the generated response. Each check costs one forward pass on one input position, adds 4.2% to generation time on average, and needs no second model. The cache restoration is implemented for full-attention, sliding-window and recurrent layers, and the paper verifies on 675 checks that the responses are identical and the cache is restored tensor by tensor. Apart from the trivial Min-Prob and Last-H controls, it is the only real-time detector in the comparison that runs without an extra language model, which is what makes the approach practical.
  2. The evaluation is careful. Results are averaged over three seeds with paired bootstrap intervals, the controls are well chosen (a fixed random token, a probe on the last hidden state, and a handwritten self-check question), and the appendix analyzes in detail the one condition where the method loses instead of leaving it out.
Critical weaknesses
  1. The gains over the baselines are not shown to come from the probe token. ProbeTok is trained on labels for each dataset and generator, while the strongest baselines (Lettuce, MiniCheck) are released models that were never trained on TRIVIA+ or on responses of these generators. The improvement over the strongest baseline is statistically significant in only 3 of 8 conditions (Table A5), and in one condition (RAGTruth with Ministral), ProbeTok is significantly worse. All three significant wins are on TRIVIA+, where the baselines are out of domain and where the simple Last-H probe trained on the same labels already beats every baseline in all four conditions (Table 3 against Table 2). There, the probe token adds only 0.5 to 2.8 points over Last-H with no interval excluding zero (Table A6). On RAGTruth, where the probe token does improve over Last-H, the gain over the strongest baseline is not significant in any of the four conditions. So the two parts of the claim, that the method beats the state of the art and that the learned token is responsible, never hold in the same setting. A baseline fine-tuned on the same labels and a stronger probing baseline than the last layer at the last token are needed to attribute the gain to the learned token.
  2. The learned detector does not generalize across datasets. The probe token and head are trained separately for each dataset and generator pair. In the transfer experiment (Table A22), a detector trained on RAGTruth and tested on TRIVIA+ loses 7.4 to 17.3 AUROC points and falls below Lettuce-P, which was also trained on RAGTruth (original responses) but transfers better, for three of the four generators. Since the intended use is a deployed system, needing new labeled responses for every dataset and generator limits the contribution, and it suggests that in-domain labels rather than the probe token explain much of the headline result.
Questions for authors
  1. How does Lettuce (or MiniCheck) perform when fine-tuned on the same per-condition training labels as ProbeTok? This is the comparison that would show whether the probe token itself is the source of the gain.
  2. How does the Last-H control perform with layer selection, multi-layer input, or mean pooling over the sentence? The last layer at the last token is a weak form of probing.
  3. The test labels come from one LLM annotator, checked on 71 sentences by one reviewer, with 57 of 71 agreement before the annotator’s judgment was shown. Can you report blind agreement with a second annotator on a larger sample?
  4. How many responses were excluded per condition because the annotator abstained?
10 comments

W1: The gains over the baselines are not shown to come from the probe token. ProbeTok is trained on labels for each dataset and generator, while the strongest baselines (Lettuce, MiniCheck) are released models that were never trained on TRIVIA+ or on responses of these generators. The improvement over the strongest baseline is statistically significant in only 3 of 8 conditions (Table A5), and in one condition (RAGTruth with Ministral), ProbeTok is significantly worse. All three significant wins are on TRIVIA+, where the baselines are out of domain and where the simple Last-H probe trained on the same labels already beats every baseline in all four conditions (Table 3 against Table 2). There, the probe token adds only 0.5 to 2.8 points over Last-H with no interval excluding zero (Table A6). On RAGTruth, where the probe token does improve over Last-H, the gain over the strongest baseline is not significant in any of the four conditions. So the two parts of the claim, that the method beats the state of the art and that the learned token is responsible, never hold in the same setting. A baseline fine-tuned on the same labels and a stronger probing baseline than the last layer at the last token are needed to attribute the gain to the learned token.

Q1: How does Lettuce (or MiniCheck) perform when fine-tuned on the same per-condition training labels as ProbeTok? This is the comparison that would show whether the probe token itself is the source of the gain.

Response: We agree that in our paper, ProbeTok did not significantly outperform both the external detectors and Last-H on the same dataset. However, this is because the comparisons were not fair. Once we make the comparisons fair, ProbeTok can significantly outperform them. We show the fair comparison with external detectors in Table R1 in our General Response above, and we evaluate Last-H across additional readout methods in Q2 right below.

First, we do fine-tuning for MiniCheck and Lettuce-R. However, during fine-tuning, MiniCheck and Lettuce-R have 3.5B and 0.39B trainable parameters, while ProbeTok only has 8K. Since this parameter gap is unfair, we also build MiniCheck-7B-head and Lettuce-R-head by freezing both models and training only an 8K head.

Moreover, before fine-tuning, Lettuce was already pretrained on about 15K original RAGTruth samples, while ProbeTok only saw about 2K samples per condition in our Base Train Set. That data gap is why ProbeTok did not significantly outperform Lettuce on RAGTruth in our paper. To make the training data fair as well, we build an Expanded Train Set by adding about 13.5K non-overlapping original RAGTruth samples.

In addition, we further improved ProbeTok beyond the version in our paper. Since last-layer hidden states are specialized for next-token prediction rather than hallucination detection, we select the readout layer on the validation set and call this improved version ProbeTok-sel, while ProbeTok-last is the version in our paper reading from the last layer.

The fair comparison are reported in Table R1 of our General Response above, and they show two clear conclusions:

  • Even though MiniCheck-7B-head and Lettuce-R-head have the advantage of running an extra detector model, ProbeTok requires no extra model and still outperforms both at the same 8K trainable parameter size. On the Expanded Train Set, ProbeTok-last reaches 0.812 and ProbeTok-sel reaches 0.828, both clearly outperforming MiniCheck-7B-head at 0.785 and Lettuce-R-head at 0.787.

  • MiniCheck-7B and Lettuce-R match or outperform ProbeTok due to they have 0.39B to 3.5B trainable parameters. On the Expanded Train Set, ProbeTok-sel at 0.828 already matches fine-tuned Lettuce-R at 0.825. Fine-tuned MiniCheck-7B reaches 0.850, which is expected since it has 3.5B trainable parameters compared with only 8K in ProbeTok. As for efficiency, MiniCheck-7B and Lettuce-R require an extra model that costs 1.6 GB to 15.5 GB VRAM and 7.1% to 18.7% extra time, whereas ProbeTok-sel needs 0 GB extra VRAM and only 1.4% extra time. We have updated our paper's claims to state this trade-off clearly.

Q2: How does the Last-H control perform with layer selection, multi-layer input, or mean pooling over the sentence? The last layer at the last token is a weak form of probing.

Response: To make a fair and comprehensive comparison between Last-H and ProbeTok, we systematically evaluate layer selection, multi-layer input, and sentence mean pooling on both methods across both our Base Train Set and Expanded Train Set, as shown in the table below.

Table R3. Comprehensive comparison between Last-H and ProbeTok across four matched readout methods. We report test sentence-level AUROC (mean ± std over 3 seeds) averaged across all 8 conditions. Within each training set, bold indicates that one method significantly outperforms the other under 95% paired bootstrap intervals.

Readout MethodBase Train Set: Last-HBase Train Set: ProbeTokExpanded Train Set: Last-HExpanded Train Set: ProbeTok
Last layer0.733 ± 0.0040.763 ± 0.0060.737 ± 0.0030.812 ± 0.006
Selected layer0.787 ± 0.0020.790 ± 0.0020.791 ± 0.0000.828 ± 0.003
Multi-layer0.769 ± 0.0020.769 ± 0.0070.775 ± 0.0040.813 ± 0.008
Mean pooling0.769 ± 0.0050.781 ± 0.0030.780 ± 0.0050.813 ± 0.001
  • Selected layer: For Last-H, we train a linear head on every layer and select the layer on the validation set. For ProbeTok, to avoid training a probe token on every layer (which is time-consuming), we select on the validation set among three candidates: the layer chosen by Last-H, the middle layer, and the three-quarter layer.

  • Multi-layer: We concatenate hidden states from the middle, three-quarter, and last layers.

  • Mean pooling: At the last layer, Last-H averages hidden states across all tokens in the sentence, while ProbeTok concatenates its probe token hidden state with that sentence average.

The table above shows two clear conclusions:

  • ProbeTok outperforms or matches Last-H across all settings, and it significantly outperforms Last-H in all four readout methods on the Expanded Train Set. Even on the Base Train Set, ProbeTok significantly outperforms Last-H in two readout methods and matches it in the other two.

  • As training data grows, the improvement of Last-H is very small, whereas ProbeTok improves substantially. For example, at the selected layer, Last-H only improves from 0.787 on the Base Train Set to 0.791 on the Expanded Train Set, whereas ProbeTok improves from 0.790 to 0.828. This is because Last-H only reads hidden states computed for next-token prediction, whereas our probe token learns from more samples to compute hidden states dedicated to hallucination detection.

W2: The learned detector does not generalize across datasets. The probe token and head are trained separately for each dataset and generator pair. In the transfer experiment (Table A22), a detector trained on RAGTruth and tested on TRIVIA+ loses 7.4 to 17.3 AUROC points and falls below Lettuce-P, which was also trained on RAGTruth (original responses) but transfers better, for three of the four generators. Since the intended use is a deployed system, needing new labeled responses for every dataset and generator limits the contribution, and it suggests that in-domain labels rather than the probe token explain much of the headline result.

Q3: The test labels come from one LLM annotator, checked on 71 sentences by one reviewer, with 57 of 71 agreement before the annotator’s judgment was shown. Can you report blind agreement with a second annotator on a larger sample?

Response to W2 & Q3: We address cross-dataset generalization, cross-generator deployment, and LLM-as-a-judge label reliability (Q3) in three points below:

  1. Cross-dataset generalization: In our paper, ProbeTok fell below Lettuce-P on RAGTruth → TRIVIA+ because Lettuce-P was pretrained on about 15K original RAGTruth samples, while ProbeTok only saw about 2K samples per generator in our Base Train Set. For a fair comparison, we train ProbeTok on our Expanded Train Set and also re-train Lettuce-R from scratch on the same Expanded Train Set, called Lettuce-R-retrain, as shown in the table below.

Table R4. Out-of-distribution transfer from RAGTruth to TRIVIA+. We report test sentence-level AUROC (mean ± std over 3 seeds) on TRIVIA+. Best results in each column are in bold.

DetectorQwenGemmaMinistralGraniteAverage AUROC
Lettuce-P0.7110.7640.7790.6810.734
Lettuce-R-retrain0.713 ± 0.0660.778 ± 0.0090.834 ± 0.0140.739 ± 0.0190.766 ± 0.020
ProbeTok-last0.798 ± 0.0300.768 ± 0.0130.866 ± 0.0120.788 ± 0.0140.805 ± 0.010
ProbeTok-sel0.814 ± 0.0110.784 ± 0.0210.858 ± 0.0050.786 ± 0.0140.810 ± 0.006

In the table above, even though Lettuce-P has the advantage of post-hoc detection and Lettuce-R-retrain trains 0.39B parameters in an extra model, both ProbeTok-last and ProbeTok-sel outperform Lettuce-P on all four generators and significantly outperform Lettuce-R-retrain on average. This shows that our probe token generalizes well to an unseen dataset without target-domain labels.

  1. Zero-label cross-generator deployment: As shown in Section 4.6 of our paper, ProbeTok does not require new labels for every generator. A single open LLM (such as Qwen3.5-9B) can apply its own trained ProbeTok to score responses from unseen or black-box generators. This costs only 0.03 AUROC on average compared with training ProbeTok directly on the target generator.

  2. Reliable and scalable labeling via LLM-as-a-judge (Q3): In our paper, we use LLM-as-a-judge (GPT-5.6 SOL) to generate sentence-level labels. To verify whether these labels are accurate, we audited 71 sentences in our paper and found 57/71 initial human-LLM agreement. After we carefully re-read the retrieved documents, we found that 12 of the 14 disagreements were human reading errors, and agreement rose to 69/71 once we corrected them. Therefore, we can collect abundant reliable labels in practice without large-scale human annotation.

To further verify blind agreement on a large sample (Q3), we use a stronger independent model, GPT-6 Astra, to blindly relabel all 28.5K test sentences across all 8 conditions. As a result, we find that GPT-6 Astra achieves 93.8% sentence agreement and 0.745 Cohen's κ with GPT-5.6 SOL. We then re-evaluate all 8K-parameter detectors under GPT-6 Astra's blind labels and on the 26.7K agreed sentences, as shown in the table below.

Table R5. Comparison of 8K-parameter detectors under independent blind test labels. All detectors are trained on the Expanded Train Set, and we report test sentence-level AUROC (mean ± std over 3 seeds) averaged across all 8 conditions. Best results in each column are in bold.

DetectorOriginal Test Labels AUROCGPT-6 Astra Labels AUROCAgreed Sentences AUROC
MiniCheck-7B-head0.785 ± 0.0040.779 ± 0.0050.812 ± 0.004
Lettuce-R-head0.787 ± 0.0020.780 ± 0.0020.814 ± 0.003
Last-H (last layer)0.737 ± 0.0030.720 ± 0.0030.753 ± 0.003
Last-H (selected layer)0.791 ± 0.0000.778 ± 0.0000.815 ± 0.000
ProbeTok-last0.812 ± 0.0060.801 ± 0.0070.836 ± 0.008
ProbeTok-sel0.828 ± 0.0030.818 ± 0.0030.853 ± 0.003

Across all three test label sets, both ProbeTok-last and ProbeTok-sel clearly outperform MiniCheck-7B-head, Lettuce-R-head, and both Last-H variants.

Reviewer_4vtGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProbeTok, a real-time hallucination detector for RAG on sentence level which runs on frozen generator. After each sentence, a single probe token is fed on the top of the generator’s cache. A linear head reads the last-layer hidden state and outputs an unsupported score. Then the cache is restored exactly to make the response unchanged. The authors regenerate RAGTruth and TRIVIA+ responses using four more advanced LLMs than what were used when they were released, with at most 9B and label them with GPT-5.6 Sol. They train ProbeTok separately on each dataset-generator condition. ProbeTok reaches the highest average sentence-level AUROC: 0.763, comparing with Lettuce-P 0.732 and Last-H 0.733, with a very small amount of extra time (4.2%).

Critical strengths

A practical real-time detector design at low cost. Each check is just one forward pass on one token and reuses the cache. The exact cache restoration is handled and verified. It is a strength that the real-time self-checking without a second model or a second pass.

Critical weaknesses
  1. The setting is asymmetric. ProbeTok is trained on each condition’s labels and tested under the same dataset. However the released baseline detectors (Lettuce, MiniCheck, ReDeEP, LUMINA) were trained before these response sets existed. So for every test set is out of distribution for the baseline detectors.
  2. Comparing to the prior work IntroLM, the novelty seems limited here.
Questions for authors
  1. How does ProbeTok compare against those baselines (such as Lettuce-R and MiniCheck) if they are fine-tuned on each condition’s training labels? Will the ProbeTok’s advantage still hold?
  2. Equal out-of-distribution comparison. Can the authors report ProbeTok trained on RAGTruth and tested on TRIVIA+, against Lettuce on the same responses? And in the reverse direction, trained on TRIVIA+ and tested on RAGTruth.
5 comments

W1: The setting is asymmetric. ProbeTok is trained on each condition’s labels and tested under the same dataset. However the released baseline detectors (Lettuce, MiniCheck, ReDeEP, LUMINA) were trained before these response sets existed. So for every test set is out of distribution for the baseline detectors.

Q1: How does ProbeTok compare against those baselines (such as Lettuce-R and MiniCheck) if they are fine-tuned on each condition’s training labels? Will the ProbeTok’s advantage still hold?

Response: To address the reviewer's concern, we re-evaluate the baselines mentioned by the reviewer for fair comparison.

First, ReDeEP and LUMINA do not need retrain, as both compute scores directly from the generator's own internal states. As shown in our paper, we already tune the hyper parameters of both methods on validation set.

For MiniCheck-7B and Lettuce-R, we fine-tune both models on our training set and report all 8-condition averages in Table R1 of our General Response above. Notice that, during fine-tuning, MiniCheck-7B and Lettuce-R have 3.5B and 0.39B trainable parameters, while ProbeTok only has 8K. Since this parameter gap is unfair, we also build MiniCheck-7B-head and Lettuce-R-head by freezing both models and training only an 8K head for fair comparison with our method.

Moreover, before fine-tuning, Lettuce was already pretrained on about 15K original RAGTruth samples, while ProbeTok only saw about 2K samples per condition in our Base Train Set. To make the training data fair as well, we build an Expanded Train Set by adding about 13.5K non-overlapping original RAGTruth samples to the Base Train Set.

In addition, we further improved ProbeTok beyond the version in our paper. Since last-layer hidden states are specialized for next-token prediction rather than hallucination detection, we select the readout layer on the validation set and call this improved version ProbeTok-sel, while ProbeTok-last is the version in our paper reading from the last layer.

All results for MiniCheck-7B and Lettuce-R are reported in Table R1 of our General Response above, and they show two clear conclusions:

  • Even though MiniCheck-7B-head and Lettuce-R-head have the advantage of running an extra detector model, ProbeTok requires no extra model and still outperforms both at the same 8K trainable parameter size. On the Expanded Train Set, ProbeTok-last reaches 0.812 and ProbeTok-sel reaches 0.828, both clearly outperforming MiniCheck-7B-head at 0.785 and Lettuce-R-head at 0.787.

  • MiniCheck-7B and Lettuce-R match or outperform ProbeTok due to they have 0.39B to 3.5B trainable parameters. On the Expanded Train Set, ProbeTok-sel at 0.828 already matches fine-tuned Lettuce-R at 0.825. Fine-tuned MiniCheck-7B reaches 0.850, which is expected since it has 3.5B trainable parameters compared with only 8K in ProbeTok. As for efficiency, MiniCheck-7B and Lettuce-R require an extra model that costs 1.6 GB to 15.5 GB VRAM and 7.1% to 18.7% extra time, whereas ProbeTok-sel needs 0 GB extra VRAM and only 1.4% extra time. We have updated our paper's claims to state this trade-off clearly.

W2: Comparing to the prior work IntroLM, the novelty seems limited here.

Response to W2: IntroLM and ProbeTok differ in three ways:

1. Task: IntroLM performs LLM routing by deciding which model should handle each query and cannot detect hallucination, whereas ProbeTok detects sentence-level hallucinations in real time.

2. Mechanism: IntroLM appends tokens to the prompt before generation, so it never sees the generated response and naturally cannot tell which sentence is hallucinated. In contrast, ProbeTok inserts its probe token after each generated sentence to check that sentence in real time. Intuitively, IntroLM uses the appended tokens to predict, before answering, whether the model can answer the query well. In contrast, our method uses the probe token to ask the generator to recheck whether the previous sentence is a hallucination based on the query, the retrieved documents and the previously generated sentences. Although both methods insert tokens, their underlying motivations are completely different.

3. Trainable parameters: IntroLM trains millions of LoRA parameters on the generator. In contrast, ProbeTok freezes the generator and only trains 8K parameters.

We have moved this comparison from Appendix A into the Related Work section of our paper.

Q2: Equal out-of-distribution comparison. Can the authors report ProbeTok trained on RAGTruth and tested on TRIVIA+, against Lettuce on the same responses? And in the reverse direction, trained on TRIVIA+ and tested on RAGTruth.

Response to Q2: In our paper, the cross-dataset comparison was unequal because Lettuce was pretrained on about 15K original RAGTruth samples, while ProbeTok only trained on about 2K samples per generator in our Base Train Set. To match Lettuce's RAGTruth data scale, we add about 13.5K non-overlapping original RAGTruth samples to build an Expanded Train Set.

Moreover, since Lettuce was already pretrained on RAGTruth, if we train it on TRIVIA+ and test it on RAGTruth, the test set is not out-of-distribution. We therefore build Lettuce-R-retrain by retraining Lettuce-R from scratch. Both Lettuce-R-retrain and ProbeTok use the Expanded Train Set for OOD comparison as shown in the table below.

Table R2. Out-of-distribution transfer across datasets. We report test sentence-level AUROC (mean ± std over 3 seeds) averaged across all 4 generators. Best results in each column are in bold.

DetectorRAGTruth → TRIVIA+ AUROCTRIVIA+ → RAGTruth AUROCAverage AUROC
Lettuce-R-retrain0.766 ± 0.0200.619 ± 0.0200.692 ± 0.020
ProbeTok-last0.805 ± 0.0100.599 ± 0.0230.702 ± 0.008
ProbeTok-sel0.810 ± 0.0060.616 ± 0.0100.713 ± 0.002

The result shows, even though Lettuce-R-retrain trains 0.39B parameters in an extra model while ProbeTok only trains 8K parameters with no extra model, both ProbeTok-last and ProbeTok-sel outperform Lettuce-R-retrain on RAGTruth → TRIVIA+. When trained on TRIVIA+ and tested on RAGTruth, ProbeTok-sel matches Lettuce-R-retrain. Averaged across both directions, ProbeTok-sel outperforms Lettuce-R-retrain at 0.713 vs. 0.692.

We sincerely thank the reviewer again for the careful and constructive review, which has helped us substantially improve our paper.

In the responses above, we have addressed each weakness and question with new experiments and clarifications, and all of them are included in our revised paper and marked in blue. If these responses have resolved the reviewer's concerns, we would be grateful if the reviewer could consider raising the score.

If any concern remains, we would be glad to discuss it further and to run additional experiments during the discussion period.

Comments on the paper

2 comments

We thank the three reviewers for their detailed and constructive feedback. We appreciate the reviewers' recognition of our real-time design without an extra model, our KV-cache rollback (4vtG, 3Bca), our multi-seed bootstrap evaluation (3Bca), and our clear motivation (2tVd).

Summary. The main concern shared by Reviewers 4vtG and 3Bca is that ProbeTok was trained on our training set, while MiniCheck-7B and Lettuce-R were tested without fine-tuning. We have run this comparison, and ProbeTok's advantage holds once the comparison is fair: with the same training data and the same 8K trainable parameters, ProbeTok clearly outperforms both detectors, with 5× to 13× lower extra time cost and without relying on any extra model. All new results are included in our revised paper and marked in blue.

We make the comparison fair in three steps:

  1. Same training labels: We fine-tune MiniCheck-7B and Lettuce-R on our training set.

  2. Same trainable parameters: During fine-tuning, MiniCheck-7B and Lettuce-R train 3.5B and 0.39B parameters, while ProbeTok only trains 8K. Since this parameter gap is unfair, we also build MiniCheck-7B-head and Lettuce-R-head, where we freeze the models and only train an 8K head.

  3. Same training data: Lettuce was already pretrained on about 15K original RAGTruth samples, while ProbeTok only saw about 2K samples per condition in our Base Train Set. To make the training data fair as well, we build an Expanded Train Set by adding about 13.5K non-overlapping original RAGTruth samples.

Table R1. Comparison with external real-time detectors on the same training labels. We report test sentence-level AUROC (mean ± std over 3 seeds) averaged across all 8 conditions. Best results in each column are in bold.

DetectorRequire Extra Model (VRAM)Trained ParametersExtra Time CostBase Train Set AUROCExpanded Train Set AUROC
MiniCheck-7B, fine-tuned7B LLM (15.5 GB)3.5B18.7%0.837 ± 0.0030.850 ± 0.003
Lettuce-R, fine-tunedEncoder (1.6 GB)0.39B7.1%0.816 ± 0.0020.825 ± 0.001
MiniCheck-7B-head7B LLM (15.5 GB)8K18.7%0.775 ± 0.0050.785 ± 0.004
Lettuce-R-headEncoder (1.6 GB)8K7.1%0.794 ± 0.0050.787 ± 0.002
ProbeTok-lastNo (0 GB)8K4.2% †0.763 ± 0.0060.812 ± 0.006
ProbeTok-selNo (0 GB)8K1.4% ‡0.790 ± 0.0020.828 ± 0.003
  • Base Train Set: We use the training set from our paper, with about 2K samples per condition.

  • Expanded Train Set: We add about 13.5K non-overlapping original RAGTruth samples to our RAGTruth conditions, since Lettuce already used this part data during pretraining.

  • MiniCheck-7B, fine-tuned: We fine-tune the last 16 layers of MiniCheck model. We did not fine-tune all 32 layers since it causes OOM.

  • Lettuce-R, fine-tuned: We fine-tune all 0.39B parameters of Lettuce-R.

  • MiniCheck-7B-head: We freeze MiniCheck-7B and only train an 8K MLP head.

  • Lettuce-R-head: We freeze Lettuce-R and only train an 8K MLP head.

  • ProbeTok-last: We read the probe token from the last layer, which is the version in our paper.

  • ProbeTok-sel: This is our further improved version of ProbeTok. The motivation is: we notice the last-layer hidden states are specialized for next-token prediction and may not suit hallucination detection task. Therefore, we select the readout layer on the validation set.

  • Extra Time Cost: We measure the extra time required for hallucination detection relative to response generation time, averaged over all 8 conditions.

    • † In our paper, we ran a separate 1-token forward pass after each sentence, which added 4.2% to generation time.

    • ‡ In our follow-up improvement, we feed the probe token together with the next generated token up to the readout layer, cutting the extra time to 1.4% without affecting detection performance.

The table above shows two clear conclusions:

  • With the same 8K trainable parameters, ProbeTok wins. Even though MiniCheck-7B-head and Lettuce-R-head have the advantage of running an extra detector model, both ProbeTok-last and ProbeTok-sel clearly outperform them on the Expanded Train Set.

  • Even compared with the fully fine-tuned Lettuce-R, which has 0.39B trainable parameters, ProbeTok-sel already matches it with only 8K trainable parameters. Although the fine-tuned MiniCheck-7B achieves a higher AUROC, this is because it trains 3.5B parameters. But for the efficiency comparison, the extra time cost of ProbeTok-sel is 5× to 13× lower than that of both detectors, and ProbeTok does not need any extra model for detection.

We resolve the remaining concerns as follows (details in each reviewer's thread):

  1. Do the gains come from the probe token? (3Bca) Yes. We further conduct a comprehensive analysis of layer selection, multi-layer input, and sentence mean pooling to fully evaluate the no-probe Last-H baseline. The results show that ProbeTok significantly outperforms Last-H under all four readout methods on the Expanded Train Set.

  2. Does ProbeTok generalize to an unseen dataset? (4vtG, 3Bca) Yes. We build a fair comparison setting, where we train ProbeTok and retrain Lettuce-R from scratch on the same RAGTruth training data and test both on TRIVIA+. The results show that ProbeTok significantly outperforms the retrained Lettuce-R, and it also outperforms post hoc Lettuce-P on all four generators.

  3. Are the test labels reliable? (3Bca) Yes. We use a stronger independent model, GPT-6 Astra, to blindly relabel all 28.5K test sentences. It reaches 93.8% agreement with our labels, and all main conclusions remain unchanged under its labels.

  4. How much labeled data does ProbeTok need? (2tVd) We retrain ProbeTok on 10%, 25%, 50%, and 100% of our training set. The results show that ProbeTok-sel already outperforms post hoc Lettuce-P with only 25% of our training set, which is about 500 samples per condition.

  5. Do earlier hallucinated sentences, sentence positions, or output lengths affect ProbeTok? (2tVd) No. We conduct ablation experiments that remove earlier hallucinated sentences from the prefix and that compare ProbeTok with Last-H across sentence positions and output lengths. The results show that none of them significantly affects ProbeTok.

  6. Does ProbeTok work without retrieved documents? (2tVd) Yes. We build a sentence-level benchmark on closed-book TriviaQA (ACL 2017) [1], where no retrieved document is available to the detectors. We compare ProbeTok with three state-of-the-art detectors that do not rely on retrieved documents either, i.e., SIVR (ACL 2026) [2], SinkProbe (ICML 2026) [3], and TSV (ICML 2025) [4]. The results show that ProbeTok-sel matches the best of them in AUROC, and its extra time cost is 22× to 33× lower.

  7. Novelty over IntroLM (4vtG) and TSV (2tVd). IntroLM appends tokens before generation to route queries and cannot detect hallucinations. TSV shifts existing hidden states and needs a separate forward pass after every sentence. In contrast, ProbeTok inserts its probe token after each generated sentence to detect whether that sentence is hallucinated in real time.

  8. Presentation (2tVd). We redesign Figure 2 as a grouped bar chart and split the multi-setting appendix tables into single-setting tables.

[1] Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017, July). TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1601–1611).

[2] Srey, P., Wu, X., Nguyen, C. D. T., & Tuan, L. A. (2026, July). Learning Uncertainty from Sequential Internal Dispersion in Large Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 40088–40106).

[3] Binkowski, J., Adamczewski, K., & Kajdanowicz, T. J. (2026). Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306, 8302–8319.

[4] Park, S., Du, X., Yeh, M.-H., Wang, H., & Li, Y. (2025). Steer LLM Latents for Hallucination Detection. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 47971–47990.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.