acceptodds
Under review as a conference paper at ICLR 2027

SeqGuard: Length-Independent Stopping for Streaming Hallucination Detection

Abstract

Streaming detectors can score a language model's output for hallucinations while it is being written. The rule for when to interrupt is still a tuned score threshold, and a threshold has no error control that holds at every length: set to on RAGTruth, it reaches on long outputs, and calibrated on AggreFact or TofuEval and deployed on RAGTruth it runs over budget. We introduce SeqGuard , a stopping rule that turns any streaming detector's scores into an anytime-valid sequential test. Each score is ranked against scores from known-faithful outputs, the rank becomes a bet, and the output is interrupted once the accumulated winnings reach . A martingale argument bounds the probability of ever interrupting a faithful output by , for any detector and any length. The proof covers a conditionally calibrated variant that keeps of the deployed rule's power; the simpler marginal variant we deploy is checked empirically. With a prefix-entailment scorer and a hidden-state probe, the operating configuration's realized false-alarm rate stays at or within its confidence interval of the budget in-domain (its one point estimate above budget is Data2txt at , of episodes) and under shift among RAGTruth, AggreFact, and TofuEval, helped by slack (in-domain FAR is of ). A far-out-of-distribution HaluEval test breaks both rules: SeqGuard overshoots on summarization and on dialogue, against and for the tuned threshold. Against a memoryless rule that sees the same two detectors, the power gain is concentrated on Data2txt (– on three of four annotation-defined classes at matched FAR) and at on QA and Data2txt; it does not win on Summary. The cost is delay ( tokens at ), though – of flags fire before the output ends.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
hallucination detection, streaming detection, sequential testing, anytime-valid inference, e-values, test supermartingales, conformal prediction, false-alarm control, retrieval-augmented generation, faithfulness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z4ikOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a stopping rule for streaming hallucination detection that aims to control the probability of incorrectly interrupting a faithful generation, independently of its length. The method converts scores from existing hallucination detectors into p-values, transforms them into e-values, and accumulates evidence across generation checkpoints using a sequential framework. The authors provide theoretical guarantees under conditional calibration assumptions and evaluate the method across several hallucination detection benchmarks

Critical strengths
  • The paper addresses a practically relevant problem: controlling false alarms across arbitrarily long generations is important for deploying streaming hallucination detectors.
  • The proposed framework is principled and detector-agnostic, building on established techniques from sequential hypothesis testing.
  • The experimental analysis is detailed and covers multiple benchmarks configurations. I also appreciate that the authors explicitly discuss cases where their method underperforms simpler baselines.
Critical weaknesses
  • The theortical guarantee provided in the papaer relies on conditional calibration assumptions that are not ensured by the marginally calibrated variant used in the main experiments. The authors introduce a certified conditional variant, which comes at a noticeable cost in detection power. This means there is a gap between the theoretical contribution and the empirical results.

  • My main problem with the paper is that the empirical improvements over the baselines are very limited. SeqGuard does not consistently outperform calibrated threshold-based methods in terms of detection power. Overall, it is unclear whether the additional complexity is justified by the empirical gains.

  • The evaluation does not sufficiently establish the effectiveness of the method on long generations. The main experiments use relatively short responses, with a median of approximately five sentence-level checkpoints. This is particularly important given that length independence is a central motivation of the paper.

  • Detection delays are relatively large (32–73 tokens) and most successful detections occur after the hallucinated span has already been generated. So unclear what the practical benefits of the approach are.

Questions for authors
  1. Can the authors evaluate the method on substantially longer generations?
3 comments

Reviewer Z4ik: can you say whether you are convinced by the rebuttal, particularly the question of how close the experiments are to the theory adn whether that is enough?

I'd like to thank the authors for the detailed rebuttal and extra experiments. The new long-form evaluation addresses my main concern, and I appreciate the clearer distinction between the marginal and certified variants.

I still have two reservations:

  1. Limited practical gains: The new experiments show substantially lower detection power for SeqGuard (14.2% vs. 39.3% for the threshold), although at different false-alarm rates. It remains unclear whether a more conservatively calibrated threshold could achieve similar false-alarm control with better power.
  2. Practical utility. Detection delays remain substantial (~155 tokens), which means most hallucinations are already generated before detection. This limits the practical benefits of the approach.

Overall, the rebuttal strengthens the paper, particularly the evidence for length-independent false-alarm control. While some concerns remain, I am inclined to raise my score from 2 to 3.

Thank you for re-reading the paper and for raising your score. On your two remaining points:

  1. Pooled over the long-form set, a threshold tuned at half the budget does match SeqGuard: realized FAR .043 and power .222 at nominal α = .05, against .046 and .195 for SeqGuard at α = .10. The difficulty is knowing the right correction in advance. The threshold’s realized FAR runs from 0 to 1.9 times its budget depending on α and output length, so it can only be set with labelled long outputs. SeqGuard needs no adjustment. We will add this comparison to the paper.
  2. We agree the delay is substantial. The threshold itself takes 120 to 130 tokens on these outputs, so most of the delay comes from the detectors, and about 25 to 35 tokens from accumulation. The rule is a stop signal that saves the rest of a long output, not a preventer, and the paper says so.

Thank you again; the long-form experiment exists because you asked for it, and it is now the paper's main result.

Reviewer_Gv5cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SeqGuard, a sequential stopping method for streaming hallucination detection. Instead of interrupting generation based on a fixed confidence threshold, SeqGuard converts detector scores into conformal p-values and accumulates evidence using e-values. The authors provide theoretical results on length-independent false-alarm control and detection delay. Experiments on RAGTruth, AggreFact, TofuEval, and HaluEval compare SeqGuard with fixed and calibrated thresholds, CUSUM, and Bonferroni correction. The results show improved false-alarm control across output lengths, although detection power varies across tasks.

Critical strengths

The paper addresses a well-defined problem in streaming hallucination detection. Existing threshold-based methods can accumulate false alarms as generation length increases, and the proposed sequential testing approach provides a principled way to address this issue.

The evaluation is relatively thorough. The authors consider multiple detectors, datasets, calibration settings, and distribution shifts. They also report detection delay and computational trade-offs rather than focusing only on false-alarm rates.

Critical weaknesses

First of all, I think the length-independent false-alarm guarantee relies on conditional super-uniformity of the p-values, but the main experiments use marginal calibration. Since detector scores are correlated across checkpoints, this assumption does not necessarily hold. The authors provide a certified conditional variant, but its detection power is considerably lower. The strongest empirical results therefore come from a configuration that does not have the claimed formal guarantee.

In addition, the proposed approach mainly combines existing statistical tools, including conformal p-values, e-values, test supermartingales, and Ville's inequality. The theoretical results largely follow from established properties of these methods. While applying them to streaming hallucination detection is useful, the paper does not introduce a substantially new sequential testing method. The contribution is primarily an adaptation of existing techniques to a particular application.

Moreover, SeqGuard does not consistently outperform calibrated threshold methods. On Summary and QA, the threshold-based approaches often achieve higher detection power, while the gains are concentrated on Data2txt. Some improvements also appear to come from combining detectors rather than sequential evidence accumulation itself. The method introduces additional detection delay, and only 22–51% of flags occur before the annotated hallucination span ends. These results limit its practical advantage for real-time intervention.

Questions for authors

1.Can the authors report the main comparisons using the certified conditional calibration variant rather than only the marginal version? How much detection power and coverage are retained under the formal guarantee?

2.Can the authors clarify whether any length-independent guarantee applies to the deployed marginal configuration under the observed temporal dependence? The current empirical tests do not establish such a guarantee.

3.How much of SeqGuard's improvement comes specifically from sequential evidence accumulation rather than detector fusion? The Data2txt results suggest that the probe alone accounts for a substantial part of the gain.

4.How does SeqGuard compare with calibrated threshold methods when both are evaluated at matched realized false-alarm rates and detection delays? The current results suggest different trade-offs across domains.

1 comment

Reviewer Gv5c: if possible, can you reply to the response, saying whether you find it convincing, particularly in relation to the question of which experiments are close to the theory and which aren't, and whether the overall experimental evidence is strong.

Reviewer_m4rdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SeqGuard, a framework for detecting hallucinations during text generation. It converts streaming detector scores into conformal p-values, transforms them into e-values, and accumulates evidence until a stopping threshold is reached.

The method is evaluated using entailment-based and hidden-state detectors across RAGTruth, AggreFact, TofuEval, and HaluEval. The experiments report improved false-alarm control over conventional thresholds, although detection performance varies across tasks and comes at the cost of additional delay.

Critical strengths

SeqGuard can operate on scores from different hallucination detectors without retraining them. The ability to control false interruptions is valuable for real-time language generation.

The paper evaluates multiple datasets, detector combinations, calibration settings, and distribution shifts. The inclusion of negative results, detection delays, and false-alarm analysis provides a more balanced picture of the method's effectiveness.

Critical weaknesses

The main idea of converting streaming verifier scores into sequential statistical tests with controlled false alarms has already been explored in E-valuator (Sadhuka et al., 2025). Online Safety Monitoring for LLMs (Schirmer et al., 2026) also studies calibrated stopping decisions and compares threshold-based monitoring against sequential testing approaches.

The conceptual contribution appears incremental. The absence of these closely related studies, particularly E-valuator, makes it difficult to establish the paper's novelty.

The main experiments use marginal calibration, for which the paper provides empirical evidence and not any guarantees or clarification in that regard. The proposed certified variant conditions on coarse score-history information, but it is unclear whether this is sufficient to satisfy the full-history conditional validity required by Theorem 1.

Furthermore, the reported improvements in detection power are concentrated in particular settings, and the method introduces substantial detection delays. This limits the practical significance of the claimed advantages.

Questions for authors

Novelty: How does SeqGuard differ from E-valuator (2025) and Online Safety Monitoring for LLMs (2026)? Can the authors provide comparisons under matched detector scores and false-alarm budgets?

Theoretical validity: How does conditioning on the previous-score bucket guarantee the full-history conditional validity required by Theorem 1?

Practical performance: Since the deployed marginal method lacks the full guarantee, can the authors clarify its advantages over calibrated thresholds at matched false-alarm rates, particularly considering detection delays?

1 comment

Reviewer m4rd: if possible, can you say whether your concerns (particularly about novelty) have been addressed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.