acceptodds
Under review as a conference paper at ICLR 2027

From Prefill to Decoding: Rethinking Hallucination Detection Throughout LLM Inference

Abstract

Existing hallucination detectors for large language models (LLMs) typically focus on either the prefill or decoding stage. However, hallucination signals across these two stages have largely been studied in isolation. In this work, we reveal that prefill and decoding provide complementary signals for hallucination detection. Each stage captures distinct failure patterns that are often missed by the other. Motivated by these findings, we propose a unified hallucination detection framework that exploits the complementary signals throughout LLM inference. Our framework forms a prefill–decode detection cascade: it first adaptively probes intermediate layers during prefill, and then selectively verifies the remaining cases at critical semantic units as decoding unfolds. This cross-stage design allows the two stages to compensate for each other's detection blind spots, yielding stronger detection within a single decoding pass. It also reduces inference cost by using prefill to filter easy cases before expensive decode-stage verification. Across two open-source LLMs and three QA benchmarks, our method consistently outperforms all probe-based single-stage baselines while incurring substantially lower latency than decode-only detection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Hallucination Detection, Uncertainty Estimation, Inference-Time Methods

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UZg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper starts with a strong motivation explaining two complementing white box methods for hallucination detection, one being to look at the models' internals over the “question” and one over the “Answer”. They put a strong emphasis on trying to reduce cost and time therefore they use a serial method that has an “early stop” mechanism that ends in the first time there is a strong enough signal for an hallucination. The results are promising, demonstrating good generalization beyond the baseline and notable improvements in several cases.

Critical strengths
  1. The paper studies a timely and important problem. The analysis in Figures 4 and 5 shows that each stage (prefill and decoding) might catch hallucinations that the other stage will miss. For example, the question alone can give a signal that the model does not know the answer, while some errors become known only when the model starts to generate the response.
  2. The method is efficient -- it improves detection while also reducing the time needed to detect hallucinations. It reachs the best AUC in 4 out of 6 cases and beats all the probe baselines in all 6 cases. On Qwen3 with Triviaqa it reaches 0.85 AUC, compared to 0.79 for the strongest single generation baseline.
Critical weaknesses

The main weakness in my opinion, are two fold. (1) The proposed method largely combines techniques from existing approaches. As such, the contribution is primarily integrative rather than methodological, with limited innovation beyond the synthesis of prior work. (2) The empirical evaluation is not comprehensive enough. Experiments involving only two LLMs and three datasets fall short of the standards typically expected for empirical papers in this area, making it difficult to assess the generalizability and robustness of the findings.

In short, while the paper presents promising results, it primarily demonstrates which types of errors are detected at each stage, without sufficiently explaining why the internal representations at different stages capture distinct hallucination signals. A deeper analysis of the underlying mechanisms, particularly what makes these representations complementary, would strengthen the paper and enhance its contribution.

Questions for authors
  1. How did you select prefill and decode thresholds? What was the criteria optimized in cross validation for choosing them. Were the thresholds selected separately for each model X dataset combination?
  2. Figure 7 shows about 0.88 AUC for decode with lookahead, but table 2 reports 0.78. Could you explain exactly which samples and scoring way were used in each evaluation?
4 comments

W1: The contribution is integrative rather than methodological.

Response for Weakness 1. We would like to clarify our contribution as not only a synthesis of prior work. It has three parts.

(i) An empirical finding not reported before. Prefill-stage and decode-stage signals are complementary rather than redundant. Each stage systematically misses failure modes that the other catches (Fig. 5), and the split of responsibility between them is controllable (Fig. 4).

(ii) Components derived from this finding, not taken off the shelf. Prior prefill probes read one fixed layer; ours exits adaptively across depth. Prior decode probes either use fixed positions or, like HaMI, select critical tokens only after the full response is generated; ours localizes critical tokens online and confirms span boundaries with a lookahead window.

(iii) The combination outperforms its parts. The cascade improves over both components in isolation (0.64 and 0.78 → 0.85 AUC), and it outperforms single-generation HaMI, which also selects critical tokens, in all six settings while operating in real time.

We also add a mechanistic explanation of the complementarity below, which we will include as a dedicated subsection.

 

W2: Limited models and datasets.

Response for Weakness 2. We thank the reviewer for this suggestion. We have added two datasets, NQ-Open (open-domain QA) and HotpotQA (multi-hop QA), which are the two benchmarks most widely used in recent hallucination-detection work, and one model, Llama3.2-3B-Instruct, which extends the evaluation to the small-model regime relevant to our on-device deployment setting. All baselines from the main table are re-run on every new setting. These results have been added to the revised paper as Appendix E (Table 4). All new content is highlighted in blue.

End-to-end AUC comparison between our cascade method and other methods on more datasets. Our method achieves either the best (bold) or second-best (italic) performance across most settings:

TypeMethod8B NQ-Open8B HotpotQA3B NQ-Open3B TriviaQA3B HotpotQA3B WinoBias
PrefillPrefill Probe (Best Layer).68.73.64.76.78.74
Decode (1 gen)Perplexity.64.65.63.69.60.55
Window Entropy.62.68.62.74.49.55
Logit Entropy (Top-K).65.66.62.70.61.57
LLM-Check (Best Layer).60.53.54.59.54.56
Decode Probe (Last Token).60.70.58.72.56.75
HaMI.72.81.72.70.66.85
Lookback.73.79.72.75.79.79
Decode (5 gens)INSIDE.60.56.63.66.52.52
HaMI.74.80.71.73.77.85
Semantic Entropy.59.63.55.62.50.50
Prefill + DecodeOurs.74.77.73.82.80.85

8B = Llama3-8B-Instruct, 3B = Llama3.2-3B-Instruct.

The new settings support two conclusions:

  1. The central claim generalizes. Combining prefill and decode signals outperforms either stage alone in all six new settings. Our method exceeds the prefill probe by 0.02–0.11 AUROC and the last-token decode probe by 0.08–0.25.
  2. The method is competitive with the strongest baselines. It attains the highest AUROC in 5 of 6 settings. On Llama3-8B / NQ-Open, the strongest baseline needs five sampled generations, while our method uses a single generation.

Why do the two stages capture different signals?

The two probes read representations that encode different things. The prefill probe reads the last prompt token, before any answer exists. Its signal grows monotonically with depth (Fig. 3a), indicating that the model's internal processing of the question gradually forms an estimate of whether it has the relevant knowledge. The decode signal, in contrast, is concentrated in position rather than depth: it peaks at critical tokens and drops sharply at the final token (Fig. 3b, c), consistent with truthfulness information residing in exact-answer tokens. The prefill probe thus asks "does the model know?", while the decode probe asks "is this particular answer right?"

This explains each stage's blind spots. A question-level signal cannot foresee errors that depend on the sampled trajectory. When the model only partially knows an answer, whether it errs depends on what it generates. Such cases form the ambiguous band of prefill scores (hallucination rate 33–50%), and decode-only hallucinations cluster there (median prefill score 0.65). Conversely, we hypothesize that once the model fluently commits to an answer, the representation at the commitment is dominated by the committed content, so the knowledge-gap signal is no longer reliably readable there. This is consistent with confident entity confusion being the largest category of prefill-only cases (Fig. 5b). When the answer is empty, there is nothing for the decode probe to read at all. Both cases are caught only at prefill.

Which signal dominates depends on how a task induces errors. In knowledge-recall tasks (TriviaQA, Movies), errors stem mainly from knowledge gaps, so the prefill signal carries much of the load: on TriviaQA, prefill recovers 89% of decode-missed hallucinations at τ = 0.5. In WinoBias, errors arise when the model resolves the coreference during generation, so decode dominates at every threshold (Fig. 4). The cross-domain results point the same way: probes transfer between the two knowledge-recall datasets (AUC 0.73 / 0.72) but fall to chance on WinoBias (0.50 / 0.49), indicating that what the representations encode is specific to the task type.

This analysis has been added to the revised paper as a new Appendix F ("Why the Two Stages Capture Different Signals"), and a summary sentence pointing to it has been added at the end of the "Stage-Exclusive Failure Modes" paragraph in Section 4.2. All new content is highlighted in blue.

Q1: How were the thresholds selected?

Response for Question 1. For the main table, τ_prefill and τ_decode were selected separately for each model × dataset combination from the grid in Table 8, by 3-fold cross-validation on the random sampling parts of training split, choosing the pair that maximizes end-to-end AUROC; test data are never used for selection. The localization threshold (τ_pos = 0.9) and the lookahead window (w = 5) are shared across all settings. This description has been added to Appendix G.5 (System Hyperparameters) of the revised paper, highlighted in blue.

Q2: Why does Fig. 7 report 0.88 while Table 2 reports 0.78?

Response for Question 2. Both numbers use decode-stage scores only; they differ in which samples are included.

  • Fig. 7 isolates the lookahead mechanism. It is computed only on samples in which the localization probe detects a critical unit, and each sample is scored by the hallucination probe at the confirmed position.
  • Table 2 reports the same decode-stage detector on all test samples, including those in which no critical unit is detected.

No critical unit detected. In the decode-only setting, 188 of 200 samples have a detected critical unit, and the remaining 12 do not. All 12 are hallucinations: the model hedges, leaves the answer empty, or never commits to an answer, so the localization probe finds nothing to anchor on. With no answer span to read, the decode probe assigns them low scores and ranks them below correct samples. If generation ends with a candidate that has not yet been confirmed, the system uses that last candidate position.

Effect on AUC. Restricted to the 188 samples with a detected unit, the AUC at w = 5 is 0.88 (Fig. 7). Including the 12 samples without one lowers it to 0.78 (Table 2). The gap therefore reflects a genuine blind spot of the decode stage rather than an inconsistency: hallucinations that never commit to an answer are invisible to the decode probe. These are precisely the boundary-knowledge failures that the prefill stage catches (Fig. 5), which further motivates the cascade.

We have revised the explanation in Section 4.4 ("Experimental Analysis of Decode-Stage Lookahead") of the paper accordingly, highlighted in blue.

Overall, I appreciate the authors' efforts and the additional results. While the rebuttal partially addresses my concerns, my central concern regarding the methodological novelty remains. I therefore maintain my original rating.

Thank you for your valuable time. We appreciate your comments and suggestions, which have helped improve our paper.

Reviewer_umqLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reveals that prefill and decoding stages provide complementary signals for hallucination detection. Therefore, it proposes a prefill–decode detection cascade that exploits the complementary signals throughout LLM inference. The proposed cascade is evaluated on three QA benchmarks with two open-source LLMs.

Critical strengths
  1. The idea of ​​combining signals from the prefill and decoding stages for hallucination detection is reasonable. The paper is clear, the diagrams are of high quality and easy to follow.
  2. The cascade architecture can effectively reduce the cost of hallucination detection, and the experimental results also show the potential of the proposed method in balancing effectiveness and efficiency.
Critical weaknesses
  1. While the cascaded paradigm proposed in this paper is reasonable, its implementation is rather straightforward and relies heavily—in a somewhat heuristic manner—on multiple thresholds (such as the prefill, positive, and decode thresholds) that are fitted to the training set. The paper offers little valuable insight into how the signals from the prefill and decoding stages should be combined.
  2. The scope of the experiments is narrow relative to the paper’s broad claims. Based on the reported results that cover two LLMs (both 8B models: Qwen3-8B and Llama3-8B-Instruct) and three datasets (all of which are QA datasets), the proposed method outperforms the baselines in only four scenarios out of six.
Questions for authors

None

4 comments

W1: Heuristic thresholds and little insight into how prefill and decode signals should be combined.

Response for Weakness 1. We thank the reviewer. The cascade is mechanically simple by design. Our main insight is that each stage has a structural blind spot, and these blind spots determine how the two signals should be combined.

Decode-stage blind spot. The decode probe reads the model's state where it commits to an answer, so it only works if the model exposes uncertainty there. It misses (i) confident entity confusion, where a fluent wrong entity looks like an ordinary confident answer, and (ii) boundary knowledge, where the model hedges or leaves the answer empty and there is no span to probe. On Qwen3–TriviaQA, decode alone recalls 62% of hallucinations, and 20% are caught only at prefill. Among these prefill-only cases, entity confusion accounts for 53% and boundary knowledge for 18%; boundary knowledge never appears among decode-only cases (Fig. 5).

Prefill-stage blind spot. Prefill sees only the question, so it estimates a prior risk and cannot foresee errors that arise during generation. The clearest example is verbose drift: 33% of decode-only cases and none of the prefill-only cases. Decode-only cases (10% of hallucinations) are not ones where prefill was confidently wrong. Their prefill scores sit just below the threshold (median 0.65). More generally, the hallucination rate grows with the prefill score: 16% below 0.6, 33–50% in (0.6, 0.8], and 64% above 0.8. The middle band is ambiguous from the question alone.

How the observations determine the combination. They lead to three design choices:

  • (a) No score ensemble. On hallucinated samples, the two scores are only moderately correlated (Pearson r = 0.46), and the stages fail on different error types. Averaging the scores would dilute whichever stage is informative for a given error.
  • (b) Prefill decides only on its confident tail and defers the ambiguous band to decode. This is a deferral rule: the cheap stage commits only when its evidence is decisive, and passes the uncertain band to the stage that observes the model's actual answer. Decode-only cases fall exactly in this band.
  • (c) Order by cost. Prefill evidence is available before any token is generated, so it naturally comes first.

Empirically, the cascade reaches 0.85 AUC on Qwen3–TriviaQA, compared with 0.65 for prefill-only and 0.68 for decode-only probes, and it beats every probe-based single-stage baseline in all six settings (Table 1). It also outperforms both of its components in isolation: 0.64 for Depth-Adaptive Prefill and 0.78 for Lookahead Decode (ablation).

Thresholds. The threshold values are chosen empirically on the training split, as with any detector's decision threshold; compared with a single-stage probe, the cascade adds only τ_prefill. The cascade structure, however, is not heuristic. It is the standard form of a cost-sensitive deferral rule (Chow, 1970; selective classification): escalating at prefill is preferable when the expected cost of a false escalation is lower than the expected cost of deferring to decode. τ_prefill therefore expresses a deployment cost preference rather than a fitted quantity, and it can be set directly from a target false-escalation rate α instead of by grid search. For example, our default τ_prefill = 0.8 corresponds to an 11% false-escalation rate while intercepting 41% of hallucinations before decoding (App. A). The same reasoning explains why τ_prefill > τ_decode: a prefill flag is raised without seeing the answer and is less precise (0.58 vs. 0.74).

Task-dependent tendency. Fig. 4 shows that the appropriate τ_prefill depends on where a task's errors are decided.

  • Knowledge-recall tasks (TriviaQA). Whether the model knows the answer is largely determined by the question, so prefill can carry much of the load. At τ = 0.5, it recovers 89% of the hallucinations that decode misses. Responsibility then hands over sharply to decode, with the two curves crossing near τ ≈ 0.73, inside the ambiguous band above. Here τ_prefill is a genuine trade-off and should sit near this hand-over region; a lower value gives more aggressive and cheaper screening.
  • Tasks whose answer is constructed during generation (WinoBias). Decode recovers more than prefill at every threshold (DRR 36–52% vs. PSR ≤ 25%), and lowering τ from 0.9 to 0.5 raises PSR by only 25 points. In other words, the hallucinations missed by decode are rarely recoverable from the question alone. A high τ_prefill therefore costs little coverage on such tasks, and prefill mainly serves to save compute. In short, knowledge-recall workloads favor a lower τ_prefill, while generation-dependent tasks favor a higher one.

Changes in the revised paper (highlighted in blue). The design guidance above is added as a new Appendix F, referenced from Section 4.2, and threshold selection is detailed in Appendix G.5.

W2: The scope of the experiments is narrow relative to the paper's broad claims, and the method outperforms the baselines in only four of six scenarios.

Response for Weakness 2. We thank the reviewer for this comment. We address the two settings where our method is not the best, and then the scope of the evaluation.

The two exceptions involve baselines that either cost more or are unstable across settings.

  • Qwen3–WinoBias (ours .78 vs. HaMI .85). The only method above ours is HaMI with five sampled generations. It requires five full decoding passes and can only produce a score after all of them finish. Among single-generation methods, ours is the best in this setting, ahead of single-generation HaMI (.76). Multi-sample aggregation is also orthogonal to our cascade and can be applied on top of it.
  • Qwen3–Movies (ours .73 vs. Window Entropy .79 and Semantic Entropy .77). The baselines that win here do not hold up in other settings. Window Entropy scores .46–.55 on the other five settings, close to chance. Averaged over the six settings of the main table, our method is the most accurate and the most stable:
MethodGenerationsMean AUCWorst-case AUC
Window Entropy1.56.46
Semantic Entropy5.68.53
HaMI5.70.57
Ours1.79.73

Each of the strongest competitors also drops sharply somewhere: five-generation HaMI falls to .57 on Qwen3–Movies and Llama3–WinoBias, and Semantic Entropy to .53 on Qwen3–WinoBias. Our method never goes below .73.

We have broadened the evaluation. The original three datasets already target different error sources: open-domain factual recall (TriviaQA), long-tail domain knowledge (Movies), and coreference resolution under gender bias (WinoBias). We have now added two datasets, NQ-Open (open-domain QA) and HotpotQA (multi-hop QA), which are the two benchmarks most widely used in recent hallucination-detection work, and one model, Llama3.2-3B-Instruct, which extends the evaluation beyond 8B models to the small-model regime relevant to our on-device deployment setting. All baselines from the main table are re-run on every new setting.

TypeMethod8B NQ-Open8B HotpotQA3B NQ-Open3B TriviaQA3B HotpotQA3B WinoBias
PrefillPrefill Probe (Best Layer).68.73.64.76.78.74
Decode (1 gen)Perplexity.64.65.63.69.60.55
Window Entropy.62.68.62.74.49.55
Logit Entropy (Top-K).65.66.62.70.61.57
LLM-Check (Best Layer).60.53.54.59.54.56
Decode Probe (Last Token).60.70.58.72.56.75
HaMI.72.81.72.70.66.85
Lookback.73.79.72.75.79.79
Decode (5 gens)INSIDE.60.56.63.66.52.52
HaMI.74.80.71.73.77.85
Semantic Entropy.59.63.55.62.50.50
Prefill + DecodeOurs.74.77.73.82.80.85

8B = Llama3-8B-Instruct, 3B = Llama3.2-3B-Instruct.

Our method attains the highest AUROC in 5 of the 6 new settings, so across all 12 settings it is the best in 9. Combining prefill and decode signals outperforms either stage alone in every new setting, exceeding the prefill probe by 0.02–0.11 AUROC and the last-token decode probe by 0.07–0.24. Window Entropy, the strongest baseline on Qwen3–Movies, is again well below our method in all six new settings (.49–.74).

These results have been added to the revised paper as Appendix E (Table 4). All new content is highlighted in blue.

Dear Reviewer,

Thank you for your time and comments. We have revised the manuscript to address the main concerns, including the new analysis of how prefill and decode signals should be combined in Appendix F, clarified threshold selection in Appendix G.5, and the expanded evaluation in Appendix E (Table 4).

As the discussion period nears its end, we would like to follow up on our author response. We would be grateful for any further feedback and are happy to clarify any remaining concerns.

Best regards,

The authors

Thanks for your response and additional experimental results. However, my first concern has not been addressed. I therefore maintain my rating.

Reviewer_CHjLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a two-stage cascade for hallucination detection in short-form QA within a single inference pass. In the prefill stage, a logistic-regression probe is attached to the MLP output of every layer at the final prompt token; layers are evaluated in order, and the query is flagged at the first layer whose score exceeds τprefill=0.8\tau_{\mathrm{prefill}} = 0.8. Queries that pass proceed to decoding, where a localization probe (PCA to 256 dimensions followed by logistic regression on the last-layer residual stream) marks critical tokens online. After a lookahead window of w=5w = 5 tokens confirms the end of a critical unit, a hallucination probe scores its final token. Each sample is scored by whichever stage terminates it. Across Qwen3-8B and Llama3-8B-Instruct on TriviaQA, Movies and WinoBias, the cascade achieves the best AUC in four of six settings (e.g., 0.85 on Qwen3–TriviaQA versus 0.68 for each single-stage probe) and reports up to 72% lower detection latency than decode-only detection.

Critical strengths
  1. Practical framing. Escalating a query at prefill or mid-decode is directly actionable in edge–cloud or small–large model serving.
  2. Simple, lightweight and online probes. The choice of logistic regression over MLPs is motivated by generalization.
  3. Informative analyses. The PSR/DRR responsibility curves (Fig. 4), the taxonomy of stage-exclusive failures (Fig. 5), the exit distributions (App. A), and the lookahead sweep with an oracle upper bound (Fig. 7) help explain the behavior of the system.
Critical weaknesses

W1. The cascade AUC mixes scores from two different classifiers.

Each sample receives the prefill score if it exceeds the threshold (so every prefill exit scores above 0.8) and the decode score otherwise. The evaluated detector is therefore the rule s=sp⋅1[sp>τ]+sd⋅1[sp≤τ]s = s_p \cdot \mathbf{1}[s_p > \tau] + s_d \cdot \mathbf{1}[s_p \le \tau]. This makes the reported AUC dependent on τprefill\tau_{\mathrm{prefill}} rather than threshold-free, and it ranks scores from two separately trained probes as if they were on a common calibrated scale. The paper does not discuss or compare principled ways of combining the two signals, such as taking the maximum, averaging, or a logistic regression stacked on both scores. In terms of AUC, the improvement may simply reflect ensembling two moderately correlated detectors rather than the cascade structure itself.

W2. The key baseline is missing: fusion without a cascade.

The two components reach 0.64 (prefill) and 0.78 (decode with lookahead) on their own, and the cascade reaches 0.85 (Table 2). The central question is whether a simple maximum, average or stacked combination of the best prefill probe and the lookahead (or exact-answer) decode probe achieves the same result. Without this comparison, the gain cannot be attributed to the cascade design.

W3. Latency is averaged only over hallucinated samples.

Latency is defined as the "time until a hallucinated sample is flagged, averaged over hallucination cases" (Section 4.3). This excludes three costs. First, correct queries, which form the majority of test queries in several settings, must still be generated in full, with the localization probe running at every token. Second, at the default threshold of 0.8, 11% of correct answers are escalated (App. A), and each escalation incurs a call to the stronger model. Third, decode-only baselines are charged full-generation latency, although that generation is needed anyway to answer the user. The reported 72% latency reduction therefore compares time-to-flag rather than system cost; a per-query cost over all queries, including escalations, is needed.

W4. Part of the prefill-only coverage is an artifact of the decode-stage design.

Boundary-knowledge cases (blank or hedged answers) account for 18% of the hallucinations caught only at prefill. The decode probe misses them simply because it has no answer span to anchor on, and a trivial rule ("no answer span detected, flag the response") would catch them. Some labels also appear questionable: the verbose-drift example in App. C (Wali/Khedive) contains the gold answer. The paper does not report who assigned the failure-mode categories, for how many samples, or with what agreement.

W5. The lookahead analysis is computed on a selected subset.

The 0.88 AUC at w=5w = 5 in Fig. 7 is computed only on samples with a confirmed critical unit, whereas the system-level value is 0.78 (Table 2). The paper should report how many samples have no detected critical unit and how these samples are scored.

W6. Narrow scope.

The evaluation covers only short-form QA with two 8B models. WinoBias is a coreference-bias dataset rather than a natural hallucination benchmark. There is no long-form evaluation, even though verbose drift and multiple critical units are central to the framing. Cross-domain transfer is also weak, with TriviaQA-to-WinoBias transfer reaching only 0.50 AUC (App. D).

Questions for authors
  1. Fusion baselines (W1, W2). How do the maximum, the average, and a stacked logistic regression of the best prefill probe and the lookahead (or exact-answer) decode probe compare with the cascade at equal compute?
  2. Threshold dependence (W1). Is the cascade AUC computed at a fixed τprefill\tau_{\mathrm{prefill}}? If so, how does it vary across thresholds, and why is it treated as a threshold-free metric?
  3. System cost (W3). What is the per-query cost over all test queries, including correct ones, counting the localization probe at every decoding step and the cost of escalating false positives to the stronger model?
  4. Trivial no-span rule (W4). How many prefill-only catches remain once blank or hedged answers are handled by a simple rule that flags responses without a detected answer span? How were the failure-mode categories in Fig. 5b assigned?
  5. Unscored samples (W5). How are samples scored when no critical unit is detected, or when several units are detected and none is flagged? What fraction of samples falls into each case, and what is the Fig. 7 AUC when these samples are included?
  6. Long-form generation (W6). How does the cascade perform on long-form generation, where a response contains several critical units, for example on a long-form entity-level hallucination benchmark?
  7. Minor. The value of max_new_tokens is missing from Table 5; the y-axis of Fig. 3a reads "Eval Acc" while the caption refers to AUC; and Figs. 3b and 3c use WinoBias and Movies, whereas Fig. 7 uses TriviaQA.
4 comments

We sincerely thank the reviewer for the detailed and constructive review. The questions about how the two stages are combined, how latency is measured, and the evaluation protocol are very helpful. We respond to each point below and will add the corresponding clarifications to the revision.

W1 / W2 / Q1–Q2: Cascade vs. score fusion, and threshold dependence

Response for W1 / W2 / Q1–Q2. We thank the reviewer for these thoughtful questions about how the cascade is evaluated and how it relates to score fusion.

Threshold dependence (W1, Q2). The cascade AUC is computed at the deployed operating point τ = 0.8, the same τ used throughout the paper. This is a deliberate choice. Our system is an early-exit cascade deployed at a fixed threshold, so it is natural to evaluate it under its routing rule rather than as an abstract ranking of scores. As in any multi-stage pipeline, the overall performance reflects both stages and the rule that connects them, and this is part of the system design.How the AUC varies with the threshold is shown in Fig. 6, where each point on the "Ours" curve corresponds to one prefill threshold τ ∈ {0.5, …, 1.0}. As τ increases, more samples are deferred to the decode stage, and AUC rises smoothly from 0.77 (261 ms) to 0.84 (1.6 s). Across this entire range, the cascade stays above every decode-stage baseline at any latency budget, including those that use the full generation (Decode Probe ≤ 0.71, HaMI ≤ 0.69, Lookback ≤ 0.65). The advantage therefore does not depend on a particular choice of τ; τ only selects a point on the accuracy–latency trade-off.

Fusion vs. cascade (W2, Q1). The max, average, and stacked combinations suggested by the reviewer are post-hoc fusions. They need both scores for every sample, so decoding must run to completion on every query. In Fig. 6, such a detector would sit at the full-generation latency (≈2.9 s) and would offer no operating point below it. The cascade covers the whole range below this point and already reaches 0.84 AUC at about 1.6 s, roughly half the full-generation latency. A fusion over fully decoded outputs is therefore not a comparison at equal compute.

The cascade can save this compute because the two stages are complementary rather than redundant. The decode probe misses errors that leave no uncertain answer span to read, such as confident entity confusions and hedged or empty answers. The prefill probe cannot anticipate errors that arise during generation, such as verbose drift. On hallucinated samples, the two scores are only moderately correlated (Pearson r = 0.46). The cascade exploits this by letting the prefill stage commit only on its confident tail and deferring the ambiguous band to the decode stage.

W3 / Q3: Latency definition and system cost

Response for W3 / Q3. We thank the reviewer for the careful reading of our latency definition. Time-to-flag measures how quickly the system can intervene once a hallucination is under way, which is what an early-exit detector is designed to improve. To address the per-query cost directly, in a flag-and-escalate deployment the cost of a query decomposes into local computation plus the escalation rate times the cost of one call to the stronger model.

Local computation over all queries. Counting every test query, including correct ones, and charging unflagged queries for full generation, the cascade generates on average about 36 tokens locally per query on Qwen3–TriviaQA, compared with 100 for decode-only detectors, which must finish generation before flagging. This is a 64% reduction in local decoding per query. The localization probe operates on a hidden state that the forward pass already computes: standardization of a 4096-dimensional vector, a projection to 256 PCA dimensions, and a logistic regression. This costs about 2.1 MFLOPs per token, compared with about 16 GFLOPs for one forward pass of an 8B model (≈0.013%).

Escalation. Any flag-and-escalate detector escalates the queries it flags, including its false positives, so escalation cost is determined by the detector's operating point rather than by the cascade structure. At equal recall, a detector with higher AUC escalates fewer correct answers, and the cascade achieves the highest AUC among single-pass detectors (Table 1). The prefill-stage trade-off is reported in App. A, and both thresholds can be adjusted to match the cost of the fallback model.

Full generation for decode-only baselines. For hallucinated responses, this generation is not needed to answer the user: the output is discarded once the response is flagged and escalated. Decode-only methods can only flag after the answer is complete, whereas the cascade can flag at prefill, before any token is generated.

W4 / Q4: Prefill-only catches and failure-mode labels

Response for W4 / Q4. Cases of the form “flag the response if no answer span is detected” cannot be identified by a rule, because the model does not produce an empty output; rather, it does not output an answer. In addition, the largest group among prefill-only hallucinations is confident entity confusion (53%), which likewise cannot be identified by a rule.

Regarding the verbose-drift example in App. C, to avoid ambiguity we will replace it in the revision with a clearer verbose-drift case from the same analysis set: asked who played Danny Ocean in the 1960 Ocean's Eleven, the model first lists the cast, then fabricates background information, and finally asserts “none of them played Danny Ocean” (prefill score 0.67, below the threshold; decode score 0.87, flagged).

W5 / Q5: Fig. 7 vs. Table 2.

Response for W5 / Q5. Both numbers use decode-stage scores only; they differ in which samples are included.

  • Fig. 7 isolates the lookahead mechanism. It is computed only on samples in which the localization probe detects a critical unit, and each sample is scored by the hallucination probe at the confirmed position.
  • Table 2 reports the same decode-stage detector on all test samples, including those in which no critical unit is detected.

No critical unit detected. In the decode-only setting, 188 of 200 samples have a detected critical unit, and the remaining 12 do not. All 12 are hallucinations: the model hedges, leaves the answer empty, or never commits to an answer, so the localization probe finds nothing to anchor on. With no answer span to read, the decode probe assigns them low scores and ranks them below correct samples. If generation ends with a candidate that has not yet been confirmed, the system uses that last candidate position.

Effect on AUC. Restricted to the 188 samples with a detected unit, the AUC at w = 5 is 0.88 (Fig. 7). Including the 12 samples without one lowers it to 0.78 (Table 2). The gap therefore reflects a genuine blind spot of the decode stage rather than an inconsistency: hallucinations that never commit to an answer are invisible to the decode probe.

We have revised the explanation in Section 4.4 ("Experimental Analysis of Decode-Stage Lookahead") of the paper accordingly, highlighted in blue.

W6 / Q6: Scope.

Response for W6 / Q6. We thank the reviewer for this suggestion and address the points in turn.

Models and datasets. We have added two datasets, NQ-Open (open-domain QA) and HotpotQA (multi-hop QA), and one model, Llama3.2-3B-Instruct, which extends the evaluation to the small-model regime relevant to our on-device deployment setting. Unlike WinoBias, both new datasets are natural hallucination benchmarks, so our conclusions no longer rest on WinoBias. All baselines from the main table are re-run on every new setting.

End-to-end AUC comparison between our cascade method and other methods on more datasets. Our method achieves either the best (bold) or second-best (italic) performance across most settings:

TypeMethod8B NQ-Open8B HotpotQA3B NQ-Open3B TriviaQA3B HotpotQA3B WinoBias
PrefillPrefill Probe (Best Layer).68.73.64.76.78.74
Decode (1 gen)Perplexity.64.65.63.69.60.55
Window Entropy.62.68.62.74.49.55
Logit Entropy (Top-K).65.66.62.70.61.57
LLM-Check (Best Layer).60.53.54.59.54.56
Decode Probe (Last Token).60.70.58.72.56.75
HaMI.72.81.72.70.66.85
Lookback.73.79.72.75.79.79
Decode (5 gens)INSIDE.60.56.63.66.52.52
HaMI.74.80.71.73.77.85
Semantic Entropy.59.63.55.62.50.50
Prefill + DecodeOurs.74.77.73.82.80.85

8B = Llama3-8B-Instruct, 3B = Llama3.2-3B-Instruct.

The new settings support two conclusions:

  1. The central claim generalizes. Combining prefill and decode signals outperforms either stage alone in all six new settings. Our method exceeds the prefill probe by 0.02–0.11 AUROC and the last-token decode probe by 0.08–0.25.
  2. The method is competitive with the strongest baselines. It attains the highest AUROC in 5 of 6 settings. On Llama3-8B / NQ-Open, the strongest baseline needs five sampled generations, while our method uses a single generation.

These results have been added to the revised paper as Appendix E (Table 4). All new content is highlighted in blue.

Long-form evaluation. Although our benchmarks are short-form QA, the evaluated responses are not single-entity outputs: on TriviaQA, each response mentions 6–8 entities on average. The localization probe therefore already has to select the critical unit among multiple candidates during decoding. A full long-form evaluation with entity-level labels is the next step, and we will discuss it in the Limitations section.

Cross-domain transfer. TriviaQA → WinoBias spans two very different task types (factual recall vs. coreference resolution). Weak transfer between such tasks is consistent with the prior finding that truthfulness probes are largely task-specific (Orgad et al., 2025), while transfer between fact-intensive tasks remains robust (e.g., 0.73 for TriviaQA → Movies and 0.72 for Movies → TriviaQA, App. D). We report all transfer results for completeness.

Minor

Response for Minor. We thank the reviewer for spotting these.We will add max_new_tokens 100 to Table 5,correct the y-axis label of Fig.3a to AUC,and state the dataset used for each panel explicitly in the captions of Figs.3 and 7.

Dear Reviewer,

Thank you for your detailed review and constructive comments. We have revised the manuscript to address the main concerns, including the expanded evaluation in Appendix E (Table 4), the analysis of prefill–decode combination and threshold selection in Appendices F and G.5, and clarifications on latency/system cost and the Fig. 7 vs. Table 2 discrepancy in Section 4.4. All changes are highlighted in blue.

As the discussion period nears its end, we would like to follow up on our author response. We would be grateful for any further feedback and are happy to clarify any remaining concerns.

Best regards,

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We would like to thank all reviewers for their constructive feedback. We are encouraged that the reviewers recognize our paper:

  • Well-motivated insight: has "practical framing" that is "directly actionable in edge–cloud or small–large model serving" (Reviewer CHjL), studying a "timely and important problem" (Reviewer UZg4).
  • Simple and lightweight design: proposes "simple, lightweight and online probes" with the choice of logistic regression motivated by generalization (Reviewer CHjL), offering a reasonable idea for combining signals with "diagrams of high quality and easy to follow" (Reviewer umqL).
  • Strong detection accuracy: provides "informative analyses" — the PSR/DRR responsibility curves (Fig. 4), the taxonomy of stage-exclusive failures (Fig. 5), the exit distributions (App. A), and the lookahead sweep with an oracle upper bound (Fig. 7) — that explain the system's behavior (Reviewer CHjL); and shows that "each stage might catch hallucinations that the other stage will miss" (Reviewer UZg4).
  • Efficiency gains: demonstrates that "the cascade architecture can effectively reduce the cost of hallucination detection", improving detection while also reducing the time needed to detect hallucinations (Reviewers umqL and UZg4).

We have been working diligently on improving the paper in several aspects, addressing your concerns and problems. Below, we summarise the main improvements, all of which have already been incorporated in the revised version, with all new content highlighted in blue.

1. Broader evaluation: two new datasets and one new model

We added NQ-Open (open-domain QA) and HotpotQA (multi-hop QA), the two benchmarks most widely used in recent hallucination-detection work, and Llama3.2-3B-Instruct, extending the evaluation beyond 8B models to the small-model regime relevant to our on-device deployment setting. All baselines from the main table are re-run on every new setting.

  • Highest accuracy in 5 of 6 new settings. Our method exceeds the prefill probe by 0.02–0.11 AUROC and the last-token decode probe by 0.07–0.25 in every new setting. Averaged over all 12 settings, it has the highest mean AUC (.79), exceeding even 5-generation HaMI (.70) with only a single generation.
  • Most stable across settings. Our method never falls below .73, while each of the strongest competitors drops sharply in at least one setting (Window Entropy to .49, Semantic Entropy to .50, single-generation HaMI to .57).
TypeMethod8B NQ-Open8B HotpotQA3B NQ-Open3B TriviaQA3B HotpotQA3B WinoBias
PrefillPrefill Probe (Best Layer).68.73.64.76.78.74
Decode (1 gen)Perplexity.64.65.63.69.60.55
Window Entropy.62.68.62.74.49.55
Logit Entropy (Top-K).65.66.62.70.61.57
LLM-Check (Best Layer).60.53.54.59.54.56
Decode Probe (Last Token).60.70.58.72.56.75
HaMI.72.81.72.70.66.85
Lookback.73.79.72.75.79.79
Decode (5 gens)INSIDE.60.56.63.66.52.52
HaMI.74.80.71.73.77.85
Semantic Entropy.59.63.55.62.50.50
Prefill + DecodeOurs.74.77.73.82.80.85

8B = Llama3-8B-Instruct, 3B = Llama3.2-3B-Instruct. Results added as Appendix E (Table 4).

2. Analysis of why the two stages capture different signals

  • Two probes read different representations. The prefill probe reads the last prompt token before any answer exists; its signal grows monotonically with depth (Fig. 3a), reflecting the model's estimate of whether it has the relevant knowledge. The decode signal is concentrated in position: it peaks at critical tokens (Fig. 3b, c). The prefill probe asks "is the model confident before answering"; the decode probe reads "the model's state as it commits to an answer."
  • Each stage has a structural blind spot. Boundary-knowledge failures are caught only at prefill; verbose drift is caught only at decode. On hallucinated samples, the two scores are only moderately correlated (Pearson r = 0.46).
  • Why the cascade, not a fusion. Averaging the scores would dilute the informative signal for each error type; prefill should commit only on its confident tail and defer the ambiguous band to decode; and prefill evidence is available before any token is generated, so it naturally comes first.

The analysis is added as Appendix F, referenced from Section 4.2.

3. Full system cost accounting

Time-to-flag measures how quickly the system can intervene once a hallucination begins. We now also report per-query cost over all queries.

  • Local decoding. Counting every test query, including correct ones, the cascade generates on average about 36 tokens locally per query on Qwen3–TriviaQA, compared with 100 for decode-only detectors that must finish generation before flagging — a 64% reduction in local decoding per query.
  • Probe overhead. The localization probe costs about 2.1 MFLOPs per token (standardization, PCA to 256 dimensions, logistic regression), compared with about 16 GFLOPs for one 8B-model forward pass (~0.013%).
  • Escalation cost. At equal recall, a higher-AUC detector escalates fewer correct answers, and the cascade achieves the highest AUC among single-pass detectors (Table 1). Both thresholds can be adjusted to match the cost of the fallback model.

This analysis is added to Section 4.3 and Appendix G.5.

4. Threshold selection and cascade design guidance

τ_prefill and τ_decode are selected separately for each model × dataset combination by 3-fold cross-validation on the training split, maximising end-to-end AUROC; test data are never used. The localization threshold (τ_pos = 0.9) and lookahead window (w = 5) are shared across all settings.

  • τ_prefill can be set from a target false-escalation rate. Our default τ_prefill = 0.8 corresponds to an 11% false-escalation rate while intercepting 41% of hallucinations before decoding (App. A), and can be set directly from a target α rather than by grid search.
  • The right τ_prefill is task-dependent. Knowledge-recall tasks favour a lower value (at τ = 0.5, prefill recovers 89% of decode-missed hallucinations on TriviaQA); generation-dependent tasks such as WinoBias favour a higher one, since the hallucinations missed by decode are rarely recoverable from the question alone.

This description is added to Appendix G.5.

5. Clarification of Fig. 7 vs. Table 2

Both numbers use decode-stage scores only; they differ in which samples are included.

  • Fig. 7 is computed only on samples in which the localization probe detects a critical unit (188 of 200), giving AUC 0.88 at w = 5.
  • Table 2 includes all 200 samples; the 12 without a detected unit are all hallucinations where the model hedges or leaves the answer empty, and the decode probe assigns them low scores, lowering the AUC to 0.78.

This gap reflects a genuine blind spot of the decode stage — precisely the boundary-knowledge failures the prefill stage catches — and further motivates the cascade. The explanation has been revised in Section 4.4.

6. Presentation and reproducibility

  • Replaced the verbose-drift example in App. C with a clearer case: asked who played Danny Ocean in the 1960 Ocean's Eleven, the model lists the cast, fabricates background information, and concludes "none of them played Danny Ocean" (prefill score 0.67, below threshold; decode score 0.87, flagged).
  • Added max_new_tokens = 100 to Table 5.
  • Corrected the y-axis label of Fig. 3a to "AUC".
  • Stated the dataset used for each panel explicitly in the captions of Figs. 3 and 7.

We thank the reviewers again for their insightful feedback, which has substantially improved the rigor, completeness, and clarity of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.