acceptodds
Under review as a conference paper at ICLR 2027

Early Warning for First Failure in Multi-Round LLM Generation via Survival Analysis

Abstract

Large language models increasingly operate through sequential generation, including multi-turn interactions with users and multi-step chain-of-thought reasoning. To model response-level reliability, existing uncertainty-quantification methods provide useful measures. However, once an early error enters the context, subsequent outputs may remain internally consistent with the corrupted trajectory, causing consistency-based uncertainty estimates to appear deceptively low and obscuring where the trajectory first breaks. This motivates the central problem of anticipating when the first failure will occur. To tackle this problem, we formulate first-failure monitoring as a discrete-time survival problem, scoring each step before its output is released. Beyond the process formulation, we incorporate propagated historical instability to capture pre-failure dynamics, and apply a conformal calibration layer that bounds, with finite-sample validity, the probability that a first failure occurs without a warning. We further show that access to propagated historical instability cannot worsen the oracle warning-burden frontier by score-class nesting, and derive conditions under which this additional history yields a strict burden reduction. Empirically, we evaluate the framework across four HalluHard domains with two open LLMs and on PRM800K reasoning chains, comparing against raw uncertainty scores, alternative feature sets, and two survival models. Within our hazard model, adding propagated historical instability improves both AUROC and AUPR in all 12 settings, and the calibrated propagation-aware rule achieves the lowest warning burden among admissible constructions across the evaluated operating points.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Sequential LLM Generation, First-Failure Prediction, Survival Analysis, Conformal Risk Control, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ffG8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for predicting the first failure in multi-turn and multi-step LLM conversations before it occurs. It formulates the problem using discrete-time survival analysis, incorporating current semantic uncertainty and historical instability to estimate first-failure risk. This work further applies conformal risk control to calibrate warning thresholds with a finite-sample guarantee on missed failures. Experiments on HalluHard and PRM800K show that incorporating historical information improves predictive performance and reduces warning burden compared with the evaluated baselines under a common missed-warning constraint.

Critical strengths
  1. This paper studies an important question and distinguishes between detecting errors in a conversation and anticipating the first failure before the actual output is released, which is important for increasingly popular long-horizon LLM interactions.

  2. The statistical formulation is coherent: survival analysis is a natural fit for this first-failure timing problem, and the paper carefully identifies and handles the risk set, censoring, and pre-response information. The conformal calibration is useful for translating a ranking score into an operational warning rule with a valid finite-sample guarantee on missed failures.

Critical weaknesses
  1. The scope of the empirical evidence is limited. The HalluHard experiments use only two relatively small 3B target models and synthetic six-turn conversations. The first-failure labels are produced by larger same-family LLM judges. The judges are not validated against human factuality annotations and were checked only by having the same judge re-label 30 claims per setting (binary κ as low as 0.46). Also, the HalluHard analysis is conditioned on trajectories surviving until turn 3, which leaves small train/calibration/test splits in some settings (e.g., 24/8/8 split in Llama) and large variations between splits. The paper would benefit from experiments on more and stronger models, more natural interactions, and at least a meaningful human-validated subset of first-failure labels.

  2. The evidence that the history feature H helps is limited. Figure 3 reports only means over 20 splits, without reporting error bars or significance tests, and the claimed gain is described as a ‘point-estimate’ improvement. Appendix A4 shows these gains are much smaller than the split-to-split variation. Also, in ablation results (Figure 4), the benefit of H is inconsistent, as column H falls below random AUROC for some settings (e.g., Qwen coding). As the authors noted, the contribution across survival models is “heterogeneous”. More discussion of this history feature H and clarification of its limitations is needed.

Questions for authors
  1. Could the authors compare their geometrically weighted historical instability feature against a simple moving average of past (q) scores or directly including lagged (q) scores? This could help establish whether the geometric propagation construction itself matters rather than simply the availability of historical information.

  2. The method requires K = 5 diagnostic generations before each monitored output. How does performance change with K (e.g., 3, 10, 20), and what is the extra inference cost and latency per turn?

  3. The weakness I mentioned above.

4 comments

W1 (scope). We agree that the empirical scope is limited. Our main goals are (1) a new formulation of early-warning monitoring for multi-turn dialogue and chain-of-thought reasoning, and (2) a first solution to this problem with a finite-sample guarantee. Accordingly, we used 3B target models to keep the cost of resampling diagnostic responses at every turn manageable, and the six-turn conversations follow the HalluHard protocol. Regarding labels, the PRM800K labels are human annotations, whereas the HalluHard labels come from LLM judges and were checked only for self-consistency. The small landmark splits make per-setting tests underpowered, which we address through tests across settings and the larger PRM800K evaluation (Table R3 in our response to Reviewer MY3o). Larger models, more natural interactions, and human-validated labels are important directions for future work.

W2 (evidence for HH). Thank you for this comment. Overall, HH provides a consistent but modest gain whose direction and size depend on the setting, and we will state this in the Discussion as its main limitation. We address each point below.

  • Significance. Table R3 (response to Reviewer MY3o) reports Nadeau–Bengio corrected tests. The gain from HH is positive in all settings and significant when the settings are considered together and on PRM800K, but not in any individual HalluHard setting.
  • Split-to-split variation. The comparison is paired, and much of the variation across splits reflects split difficulty shared by both models. In the larger settings, the standard deviation of the paired differences is well below that of the individual AUROCs (about 0.04 versus 0.09 in Qwen Medical).
  • Raw HH below random. Table R1 (response to Reviewer 3yaj) shows that the direction of HH is stable within each setting but differs across settings, and raw HH falls below 0.5 exactly in the settings where its fitted coefficient is negative. Fixed-weight combinations therefore perform poorly in Figure 4, while the learned hazard layer recovers the direction from data.
  • Heterogeneity across survival models. We agree that the benefit of HH is heterogeneous. It improves both metrics in every setting in our hazard model, but decreases AUROC for Weibull AFT and Cox PH in both Research settings.

Q1 (geometric kernel). Thank you for this suggestion. We have added the comparison. Holding the hazard model, splits, and memory length fixed, we replaced Hi,tH_{i,t} with (1) the single lag q^i,t−1\hat q_{i,t-1}, (2) the lags q^i,t−1,…,q^i,t−5\hat q_{i,t-1},\dots,\hat q_{i,t-5} as separate covariates, with missing lags at early turns set to 0 and a missing-lag indicator for each lag (indicators collinear with the turn basis are dropped), and (3) a uniform moving average of past q^\hat q. The results separate the two explanations. Availability of history alone is not sufficient. The separate lags contain every input to HH, but they perform no better than the model without history (mean AUROC 0.548 versus 0.549). The likely reason is that estimating a separate coefficient for each lag from small training sets adds variance, whereas HH summarizes them with a single coefficient and one decay parameter. The specific geometric shape also matters little. The geometric kernel and the moving average perform comparably (0.588 versus 0.582), and both outperform the single lag (0.564). What matters is aggregating past instability into a single normalized summary. The geometric family covers both extremes (ρ→0\rho \to 0 recovers the single lag and ρ→1\rho \to 1 the moving average), so the memory length is selected from data.

Table R5. Comparison of history summaries under the same hazard model. Each cell reports AUROC / AUPR on the landmark population (t≥3t \ge 3), averaged over 20 splits.

SettingNo historySingle lagSeparate lagsMoving averageGeometric (ours)
Qwen Medical0.607 / 0.1990.603 / 0.2100.558 / 0.1770.608 / 0.2110.609 / 0.213
Qwen Legal0.609 / 0.2550.621 / 0.2700.595 / 0.2740.603 / 0.2620.621 / 0.277
Qwen Coding0.477 / 0.2070.501 / 0.2420.536 / 0.2410.553 / 0.2670.548 / 0.260
Qwen Research0.494 / 0.1020.521 / 0.1230.504 / 0.1120.488 / 0.1100.513 / 0.124
Llama Medical0.604 / 0.3140.592 / 0.3390.610 / 0.3300.631 / 0.3960.629 / 0.394
Llama Legal0.482 / 0.4410.488 / 0.4750.474 / 0.4370.548 / 0.4820.570 / 0.503
Llama Coding0.539 / 0.5720.601 / 0.6020.584 / 0.6210.648 / 0.6830.628 / 0.663
Llama Research0.581 / 0.3140.582 / 0.3150.523 / 0.2710.580 / 0.3190.584 / 0.322
Mean0.549 / 0.3000.564 / 0.3220.548 / 0.3080.582 / 0.3410.588 / 0.344

Q2 (KK and cost). Due to cost and time constraints, we did not run K=10K = 10 or 2020. However, we analyze how KK affects the cost. Each monitored turn requires KK extra responses from the same prefix, which can be generated in one batch with the released response, plus K(K−1)/2K(K-1)/2 judge calls for semantic grouping (10 at K=5K = 5), which can run in parallel. Generation cost therefore grows linearly and grouping cost quadratically in KK. Larger KK would reduce the saturation of the uncertainty feature documented in Table A1, and we leave a full comparison to future work. The PRM800K setting requires no additional generation, since it uses existing sibling chains.

Q3. Please see our responses to W1 and W2 above.

Thank you for the detailed response and the additional analyses. My Q1 has been addressed. However, my concern about W2 is not fully addressed. The gain from H is not significant in any individual HalluHard setting, the pooled sign test shows a consistent direction rather than a meaningful effect size, and the coefficient on H is negative in many settings, which seems inconsistent with the paper’s motivation. I appreciate the authors acknowledging the limited scope in W1, but the concern remains unaddressed without additional evidence such as larger models or human-validated labels. I will therefore keep my score.

Thank you for the follow-up. We would like to clarify one point regarding the sign of the coefficient on H. Our claim is not that H is positively associated with first-failure risk, but that it carries risk information complementary to current-turn uncertainty. Our theoretical results are stated accordingly. In particular, Prop. B.2 holds regardless of sign, and Remark 1 in App. A.4 requires only a nonzero residual shift in H, in either direction. Section 5 (L417–424) also notes that the association of H with first failure is not uniformly positive. We also believe a negative coefficient is plausible, and the following conceptual example illustrates why.

Example Consider two patients monitored for heart disease whose current BMI is equally close to the alert threshold. The first is an athlete whose BMI has always been high, and the second has historically had a low BMI. If their current readings of BMI are identical, the second patient is more concerning, because the reading departs sharply from their own baseline. Likewise, given similar current uncertainty, a conversation that has been stable and then becomes uncertain can carry higher risk than one that has always been semantically diverse, which is what a negative coefficient on H captures.

Although we understand that the current manuscript may not have changed your assessment, we sincerely thank you for your time and constructive feedback.

Reviewer_MY3oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses multi-turn dialogue and COT as a discrete-time survival problem. They design a logistic hazard model. It’s features are a turn baseline f(t), current semantic entropy U from K=5 diagnostic samples, and a new history feature H, which is a normalized, geometrically weighted average of past of nonmodal semantic mass. Their experiments are done with 3B models on 4 halluHard domains. They use 7B judges. They report that adding H raises both the AUROC and AUPR in all 12 settings.

Critical strengths
  • This is an important problem to address, and their formulation is careful about what is observable at test time, which is critical for evaluating the work. For example, the HalluHard calibration uses one loss per conversation rather than a per turn one.

  • Their reporting is transparent in that they include both positive and negative results in the appendix.

Critical weaknesses
  • The warning rule does not appear to beat a random warning rule in 4 of the 6 landmark settings. By the paper's definitions, WR = P(V > η) and MWR = P(Δ = 1, V ≤ η). A rule that flags each conversation independently with probability w therefore has WR = w and MWR = π(1 − w), where π is the fraction of test conversations that fail (Table A3: events divided by test conversations). Matching the model's realized MWR at α = 0.20 gives w = 1 − MWR/π. Using the "Our Model" rows of Tables A19 (research), A14 (legal) and A16 (coding), the model's WR vs. the random rule's is below.

    • Research-Qwen: 0.477 vs 0.391
    • Research-Llama: 0.776 vs 0.749
    • Legal-Qwen: 0.650 vs 0.640
    • Coding-Qwen: 0.703 vs 0.682
  • The paper's practical claim is a lower warning burden at a controlled miss rate. So a random rule matching it means the claim is not well supported as stated. Please report this random baseline for every setting and show whether the model beats it.

  • There needs to be statistical significance testing. The gain from H seems to be quite small. For example in the main (t ≥ 3) setting the AUROCs are only 0.51 to 0.63, and adding H improves AUROC by as little as +0.002 (Medical-Qwen, A4). And on the full risk set the gain is at most +0.008 (A20-A27). The two largest gains are about +0.09, but they come from settings with only 8 and 10 test conversations.

Questions for authors
  • For PRM800K, is T_{p,k} the annotated length or the generated length? Phase-2 annotation stops at the first error, so if it is the annotated length, sibling pool sizes encode when siblings failed, which would leak label information.

  • Were any failure labels checked by humans?

  • Li et al. (2026), which you do cite, already does treat the first inconsistent response in multi-turn dialogue as a time to event outcome and show that a lightweight AFT model can serve as a turn level risk monitor. How does your monitor differ in practice? Can you compare against theirs?

3 comments

Random-warning baseline. We thank the reviewer for this careful analysis; the observation is correct. We believe the reason is that the rule warns when the maximum score over a conversation exceeds the threshold, and conversations that never fail stay at risk longer, which gives them more chances to trigger an unnecessary warning. This can be addressed by changing only the score compared with the threshold. Instead of the one-step hazard, we use the remaining risk Ri,tremR^{\text{rem}}_{i,t}, which uses the survival structure of our hazard model and is defined as:

1−∏s=tTˉ[1−λ^(s;U^i,t,Hi,t,Bi,t)],1 - \prod_{s=t}^{\bar T}\left[1 - \hat\lambda(s; \hat U_{i,t}, H_{i,t}, B_{i,t})\right],

that is, the probability of a first failure between the current turn and the end of the monitoring horizon Tˉ\bar T (Tˉ=6\bar T = 6 in HalluHard), computed from the same fitted hazard model with the covariates held at their current values. This accounts for the number of turns still to come and offsets the extra exposure. The risk set, features, hazard model, and calibration are unchanged, so the conformal guarantee still holds. The prediction results are also unaffected, since AUROC and AUPR are still computed on the hazard. The finite-sample guarantee holds for any pre-response score, and the burden analysis extends with the same arguments.

For brevity, we report only the worst case, Qwen Research, in Table R2. The new score changes the rule from clearly worse than the random rule to on par with it. The remaining difference is small and not significant, consistent with the weak turn-level signal in this setting (AUROC 0.51). We will report all conformal results with the new score, together with the matched random baseline for every setting.

Table R2. Warning rate versus the matched random rule on Qwen Research (t≥3t \ge 3).

α\alphaWarning scoreOur WRRandom WR (matched MWR)Δ (ours − random)
0.10one-step hazard0.7870.716+0.071
0.10remaining risk0.7800.800−0.020
0.20one-step hazard0.4770.391+0.086
0.20remaining risk0.3500.358−0.008

For each score, the random rule warns each conversation independently with the probability that matches that score's realized MWR. Both scores satisfy the missed-warning constraint at each level.

Statistical significance. We thank the reviewer and have added significance tests (Table R3). For each setting, we compare the models with and without HH on the same 20 conversation-level splits. Because the splits resample the same conversations, a standard paired tt-test over splits understates the variance, so we report the Nadeau–Bengio corrected test.

Overall, the gain from HH is significant when the eight HalluHard settings are considered together and on PRM800K, but not in any individual HalluHard setting. We therefore describe its effect as consistent but modest and setting-dependent in size, and we agree that the absolute AUROCs show that anticipating first failures from pre-response signals remains difficult.

Adding HH improves both AUROC and AUPR in all eight landmark settings, which is significant when the settings are considered together (sign test, p=0.0078p = 0.0078). Excluding the two smallest settings that the reviewer points out (8 and 10 test conversations), HH still improves both metrics in all six remaining settings (sign test, p=0.031p = 0.031). On PRM800K, although the gain is small (AUROC +0.0026), it is significant (p<0.001p < 0.001), since each test fold contains about 1,320 problems. This uses the corrected pipeline (see Q1). On the full risk set, the gains are smaller (at most +0.008 in AUROC) but still positive in all eight settings, as expected, since HH is zero at the first turn and a single lag at the second. Per-setting tests are underpowered given the small test folds; larger multi-turn evaluations are left to future work.

Table R3. Gain from HH and its significance. Per-setting rows are for the landmark population (t≥3t \ge 3).

EvaluationTest conversations per foldΔ AUROCNB ppΔ AUPRNB pp
Qwen Medical35+0.0020.944+0.0140.640
Qwen Legal29+0.0110.474+0.0220.272
Qwen Coding30+0.0700.166+0.0530.104
Qwen Research44+0.0190.592+0.0220.457
Llama Medical21+0.0250.413+0.0800.079
Llama Legal8+0.0880.244+0.0620.375
Llama Coding10+0.0890.215+0.0910.280
Llama Research23+0.0030.912+0.0080.790
All eight, landmark—+0.002 to +0.0890.0078 (sign test)+0.008 to +0.0910.0078 (sign test)
All eight, full risk set—+0.0001 to +0.0080.0078 (sign test)+0.003 to +0.0180.0078 (sign test)
PRM800K (5≤Kp≤205 \le K_p \le 20)≈1,320 problems+0.0026<0.001+0.0023<0.001

Q1 (Tp,kT_{p,k} in PRM800K). We thank the reviewer for this careful observation. In the submitted version, Tp,kT_{p,k} was the annotated length, so, as you note, the size of the sibling pool could encode when siblings failed. We have corrected this by building the reference pool from generated lengths, so no annotation information enters the features. While checking the pipeline, we also corrected the failure labels, which are now taken from the rating of the generated step itself. This raises the chain-level failure rate from 0.19 to 0.85, consistent with Phase 2 of PRM800K preferentially selecting convincing wrong-answer solutions for annotation [1]. Crossing the two pool definitions with the two label definitions shows that the gain from HH in the submitted PRM800K results came from this leakage (about +0.013 in AUROC with the annotated-length pool versus +0.0007 with the generated-length pool). For clarity, we include part of the corrected results in Table R4: the full model reaches AUROC 0.571, compared with 0.569 without HH, and the gain remains significant (Table R3). All PRM800K results in the paper will be rerun with the corrected pipeline.

Table R4. Corrected PRM800K step-level prediction, 5≤Kp≤205 \le K_p \le 20. Mean (SD) over 20 problem-level splits.

Score constructionMin. pool 3: AUROCMin. pool 3: AUPRMin. pool 2: AUROCMin. pool 2: AUPR
R=UR = U0.5577 (0.0036)0.1529 (0.0023)0.5568 (0.0034)0.1526 (0.0022)
R=qR = q0.5520 (0.0033)0.1523 (0.0022)0.5512 (0.0033)0.1522 (0.0022)
R=HR = H0.5617 (0.0033)0.1546 (0.0024)0.5610 (0.0034)0.1545 (0.0024)
R=12(U~+H~)R = \frac{1}{2}(\tilde U + \tilde H)0.5668 (0.0034)0.1580 (0.0026)0.5660 (0.0033)0.1578 (0.0025)
λ^{f(t)}\hat\lambda\{f(t)\}0.5562 (0.0040)0.1498 (0.0018)0.5558 (0.0039)0.1499 (0.0018)
λ^{f(t)+U}\hat\lambda\{f(t) + U\}0.5686 (0.0039)0.1572 (0.0021)0.5679 (0.0038)0.1571 (0.0020)
λ^{f(t)+H}\hat\lambda\{f(t) + H\}0.5666 (0.0038)0.1573 (0.0024)0.5661 (0.0036)0.1573 (0.0023)
Weibull AFT, with HH0.5684 (0.0036)0.1587 (0.0023)0.5658 (0.0036)0.1583 (0.0023)
Cox PH, with HH0.5677 (0.0040)0.1574 (0.0026)0.5655 (0.0040)0.1566 (0.0026)
Our Model0.5711 (0.0039)0.1595 (0.0023)0.5705 (0.0038)0.1595 (0.0022)

Corrected pipeline: the reference pool is built from generated lengths, failure labels are taken from the generated steps, siblings are aligned by content.

Reference

[1] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.

Q2 (human verification). The PRM800K labels are human annotations from the dataset itself. The HalluHard labels were obtained from larger same-family LLM judges. Although these labels were not checked by humans, re-judging 30 claims per cell gives binary agreement of 0.77–0.97 (Table A2).

Q3 (comparison with Li et al., 2026). Thank you for the constructive feedback. Our monitor differs in three ways. First, we predict the first factual or reasoning error, whereas Li et al. (2026) target the first inconsistent response; an inconsistent response need not be wrong, and responses after an early error can stay consistent. Second, our score is computed before the response is released. Third, we calibrate a warning threshold with a finite-sample guarantee on missed first failures. Since Li et al. (2026) build their monitor on an AFT model, we include Weibull AFT as a baseline in our experiments. Using our feature set, our discrete-time hazard achieves higher AUROC in all eight HalluHard landmark settings (Tables A5, A7, A9, A11). Their drift features measure inconsistency under adversarial follow-ups, which our data do not contain. Hence, their features are not designed for our setting, and a direct comparison would be of limited value.

My concerns about the significance of H and the random-warning baseline are noted, though not fully resolved. The PRM800K results are a concern given they need to be rerun. The random-warning baseline is still a concern that is important to the papers claimed contribution to a lower warning burden at a controlled miss rate. I think this work would benefit from another round of revision and review. I will be keeping my rating at 2.

Reviewer_3yajOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies early warning of first failures in multi-turn LLM conversations and multi-step reasoning by formulating first-failure prediction as a discrete-time survival analysis problem. The proposed method combines current semantic uncertainty with a geometrically weighted historical instability feature to estimate the probability of first failure at each step, and uses conformal risk control to calibrate warning thresholds with finite-sample guarantees on missed-warning probability. The paper further provides theoretical analyses of warning efficiency, including conditions under which incorporating historical information can reduce warning burden. Experiments on HalluHard across four domains and two LLMs, as well as on PRM800K reasoning chains, show improvements in first-failure prediction with the historical feature and reduced warning burden under the specified missed-warning constraints.

Critical strengths
  1. This paper defines the problem well. This paper formulates first-failure monitoring as a discrete-time survival problem, explicitly accounting for the at-risk process, right censoring, and the requirement that risk scores be computed before the monitored output is released.
Critical weaknesses
  1. Overall, this paper is hard to follow. For example, the authors do not provide enough motivation for different choices in the paper, such as using a normalized geometric kernel or trying to solve an optimization problem in line 263, instead of solving a more straightforward target like F1. Besides, this optimization problem itself is confusing because the methods in this paper never directly optimize this problem. 2. The definition of warning before the first error does not match its own introduction. This paper also does not provide enough information, such as the information for Bi,jB_{i,j} and the choice of f(t,y)f(t,y).
  2. This paper does not provide any previous work that tries to solve the problem. Even though this paper has cited a lot of previous work in the related work section, there is no single comparison to them. All the comparisons are inside this paper’s own framework. Therefore, it is unclear whether the framework proposed in this paper is useful or not. Besides, this paper claims that uncertainty methods are post-hoc methods, but it is easy to apply uncertainty to this setting, as we can just use the last step’s uncertainty to predict the next step.
Questions for authors
  1. Could the authors provide a fair comparison with other methods that try to solve the same problem?
  2. Could the authors explain why the warning shown before the first error is used? And what is the detailed information for Bi,jB_{i,j} and the choice of f(t,y)f(t,y)?
  3. Could the authors explain whether the logistic regression model is re-trained for each setting or not? If it is re-trained, could the authors provide some similarity analysis of the model?
  4. Could the authors further explain the difference between q and U? What additional information is provided by q in addition to U, intuitively?
  5. Could the authors clarify if the current theoretical results are unique to the information used in the method? Because it seems that all the theoretical results can be applied to any features, and I cannot see why it shows that the current methods or features are unique.
  6. The results show that the difference between f(t) only and the full model is small. Does it mean that f(t) is good enough? Is it possible that the model just remembers which step the LLM will be wrong?
3 comments
AuthorsOpenReview ↗

Response to Weaknesses (1/2)

Modeling choice. Thank you for the constructive feedback. We note that the motivation for the normalized geometric kernel, based on a filtering argument, is given in Section 3 (the paragraph preceding Eq. (2)). To further strengthen this motivation, we compared Hi,tH_{i,t} with three alternatives under the same hazard model and splits: (1) the single lag q^i,t−1\hat q_{i,t-1}, (2) the lags q^i,t−1,…,q^i,t−5\hat q_{i,t-1},\dots,\hat q_{i,t-5} as separate covariates, and (3) a uniform moving average (full tables are included in our response to Reviewer ffG8). Averaged over the eight settings, the normalized geometric summary attains AUROC 0.588 and AUPR 0.344, compared with the moving average (0.582 / 0.341), the single lag (0.564 / 0.322), and separate lags (0.548 / 0.308). The geometric kernel thus performs comparably to the moving average and better than the lag-based alternatives, while covering both short and long memory (ρ→0\rho \to 0 recovers the single lag and ρ→1\rho \to 1 the moving average) without fixing the choice in advance.

Constrained objective (line 263). The problem at line 263 is a population-level oracle objective. Since the warning burden (WSW_S) and the missed-warning risk (RSR_S) depend on the unknown data distribution, it cannot be solved directly. Instead, it serves as the benchmark for our theoretical guarantees on the warning burden under the missed-warning constraint enforced by conformal calibration. More concretely, among rules whose missed-warning rate does not exceed α\alpha, a lower warning rate is better. We use this benchmark to compare score classes with and without HH. Access to HH will not increase the oracle burden (Proposition B.2). The calibrated propagation-aware rule attains this oracle burden up to an approximation gap and estimation error (Theorem 4.2). Under the cascade-separation condition, it achieves a strictly lower burden than the propagation-oblivious oracle, even when the latter is allowed a relaxed missed-warning level α+ε0\alpha + \varepsilon_0 (Corollary 1).

Why not F1. We believe F1 does not fit first-failure monitoring. Missing a first failure is far more costly than an unnecessary warning, and our Neyman–Pearson formulation lets the operator fix the tolerated miss rate, which F1 cannot. F1 also depends on the failure prevalence, which varies about sevenfold across our domains (Table A3), and, unlike the missed-warning rate, it is not an expected per-conversation loss, so conformal risk control cannot give it a finite-sample guarantee.

Specification of Bi,tB_{i,t} and f(t;γ)f(t;\gamma). We apologize for not stating these choices clearly and give them here. The baseline f(t;γ)f(t;\gamma) captures how first-failure risk varies with the turn index independently of the uncertainty features, as is standard in discrete-time survival models (Singer & Willett, 1993). We use a quadratic polynomial in the turn index, that is, linear and quadratic terms, with the intercept kept separately. Bi,tB_{i,t} collects any additional covariates available before the response is released. In our experiments it contains only the context length in tokens, and users may include other pre-response covariates suited to their application. We have added these specifications to Section 3.

Comparison with prior work. Thank you for the constructive feedback. Unfortunately, to the best of our knowledge at the time of submission, no prior method addresses this problem directly, namely warning before the first factual or reasoning failure with a guarantee on missed failures. Nevertheless, our comparisons include methods from prior work applied to this setting. The raw uncertainty score is semantic entropy, and Weibull AFT and Cox PH are the survival model classes used in prior survival analyses of LLM behavior. With the same features, our hazard model attains the highest AUROC in all eight HalluHard landmark settings. The closest work in formulation is Li et al. (2026), which also uses survival analysis but targets a different event, the first inconsistent response under adversarial follow-ups. It does not target the first factual or reasoning failure.

Using the last step's uncertainty. We agree that uncertainty can be applied prospectively, and we have compared against this approach. Using the last step's uncertainty corresponds to the single-lag variant in our kernel comparison (see our response to Reviewer ffG8). It improves the mean AUROC from 0.549 without history to 0.564, and the mean AUPR from 0.300 to 0.322, while our propagated feature reaches 0.588 and 0.344. The last step thus carries part of the signal, and aggregating instability over earlier turns adds more. Conceptually, by post-hoc we refer to methods that score a response after it has been generated. Moreover, after a first failure, later responses can remain consistent with the corrupted context and appear certain, so post-failure uncertainty is not informative about the first failure.

AuthorsOpenReview ↗

Response to Questions (2/2)

Q1 (fair comparison). Please see "Comparison with prior work" above. All constructions, including semantic entropy and the Weibull AFT and Cox PH models, use the same splits, diagnostic samples, threshold grid, and calibration procedure.

Q2 (warning before the first error). We warn before the first error because this is the point at which intervention is most valuable. In particular, once an erroneous response is released, later responses condition on the corrupted context, errors can propagate, and post-failure uncertainty becomes unreliable (Section 1). In our setting, "before the first error" means before the failing response is released. The score for each turn is computed from the history up to the user's current input, which excludes that turn's response, so a warning issued at the failure turn still arrives in time to withhold or regenerate the erroneous response. A warning at any earlier turn also counts as successful. The conformal tables additionally report the mean lead time, that is, how many turns before the failure the warning is issued on average. For Bi,tB_{i,t} and f(t;γ)f(t;\gamma), please see our response above under "Specification of Bi,tB_{i,t} and f(t;γ)f(t;\gamma)."

Table R1. Stability of the 20 split-specific models within each setting (HalluHard, t≥3t \ge 3).

Setting% splits β^U>0\hat\beta_U > 0% splits β^H>0\hat\beta_H > 0Mean Spearman across splits
Qwen Medical90%100%0.854
Qwen Legal100%90%0.847
Qwen Coding90%0%0.713
Qwen Research60%0%0.504
Llama Medical45%0%0.906
Llama Legal65%5%0.560
Llama Coding30%0%0.789
Llama Research95%100%0.644

Each setting has 20 models, one per conversation-level split, each trained only on that split's training conversations. "% splits > 0" is the fraction of splits in which the coefficient is positive. Mean Spearman is the average rank correlation over the 190 model pairs, where each model scores all at-risk turns of its setting.

Q3 (re-training and similarity). Yes. The hazard model is refitted for each setting, since failure rates and feature–risk relations differ across domains and target models. Table R1 summarizes the 20 split-specific models in each setting. The models are stable within a setting: the sign of the history coefficient agrees in at least 90% of splits in every setting, and the average Spearman correlation between the risk rankings of different splits ranges from 0.50 to 0.91. Across settings, however, we observe that the direction differs. The history coefficient is positive in three settings and negative in five. Thus, in those five, among conversations still at risk, earlier instability is associated with lower risk. One explanation is that instability which has not led to failure reflects benign diversity, such as multiple valid answers. The uncertainty coefficient also varies in sign in four settings where its marginal signal is weak. This heterogeneity, also visible in Figure 4, is why the model is fitted separately for each setting and why a learned hazard layer is needed to recover the direction of each feature.

Q4 (q versus U). qq and UU are computed from the same semantic classes but measure different things, and qq is not a function of UU. UU measures overall dispersion across classes. qq is the probability that a fresh sample falls outside the modal answer, so it captures whether a consensus answer exists. For example, with K=5K=5, class counts {3,2} and {3,1,1} both have q=0.4q=0.4, but their UU values are about 0.77 and 1.15. We propagate qq because q∈[0,1)q \in [0,1) regardless of sample size, whereas the upper bound of UU grows as log⁡n+(n−1)/(2n)\log n + (n-1)/(2n). In PRM800K the reference pool shrinks with step depth, so a historical average of UU would mix scales, while the average of qq stays comparable across steps.

Q5 (generality of the theory). We agree that the theoretical results apply to any pre-response feature, and this is intentional. The theory has three parts. First, the finite-sample guarantee on missed warnings holds for any score, with or without HH. Second, at the oracle level, adding HH can never increase the warning burden (Proposition B.2). Third, under a sufficient condition, cascade separation, the fitted rule with HH achieves a strictly lower burden (Corollary 1). The theory therefore does not claim that HH is unique; it characterizes when HH, or any other feature, is useful.

Q6 (the turn baseline). The turn baseline only captures how average failure risk changes with the turn index. It is a quadratic in the turn index and is evaluated on held-out conversations, so it cannot memorize when a particular conversation fails. It is also not sufficient on its own. Averaged over the eight HalluHard landmark settings, its AUROC is 0.525 versus 0.588 for the full model, and the warning rule based on it alone warns on almost every conversation in most settings.

Thank you for the rebuttal. Some concerns remain regarding the empirical and methodological contributions. The improvement from the proposed geometric kernel over simpler historical aggregation methods is relatively small, and the relationship between historical instability and failure risk varies across settings. Furthermore, the theoretical guarantees are general to arbitrary pre-response features rather than specific to the proposed feature design. Overall, the rebuttal improves the clarity of the paper, but the evidence for the distinct benefits of the proposed approach remains somewhat limited. I acknowledge the authors' efforts and would keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions

Summary of Contributions

We thank all reviewers and the AC for their time. To make our paper's main contributions and concepts clear, we provide the following summary.

Why this problem matters. LLMs increasingly work through long conversations and long reasoning chains, where a single early mistake contaminates everything that follows. Existing uncertainty and verification methods judge a response after it is written, and once the first error occurs, later responses can stay consistent with it and appear confident. The moment that matters is before the first wrong response reaches the user.

What this paper contributes. To our knowledge, this is among the first works to formulate and address early warning of the first failure in multi-turn LLM generation. Before each response is released, our monitor asks whether this is the turn where a so-far-correct interaction will go wrong, and it lets the user set how often failures may slip through unwarned. The same framework covers both multi-turn dialogue and chain-of-thought reasoning.

Technical contributions.

  • Formulation. We cast first-failure warning as a discrete-time survival problem, defining the at-risk process, censoring, and a pre-response hazard.
  • Method. Since the trajectories have not yet failed, we design features (i.e., history instability) and calibrate the warning threshold, which gives a finite-sample bound on missed first failures.
  • Theory. At the oracle level, history cannot increase the warning burden, and we characterize when it strictly reduces the burden of the calibrated rule.
  • Evidence. Across four HalluHard domains with two LLMs and on PRM800K, the history feature improves AUROC and AUPR in all eight HalluHard settings (sign test, p=0.0078p = 0.0078) and on PRM800K (p<0.001p < 0.001).

During the rebuttal, we added significance tests, a random-warning baseline, and further ablations, and corrected a PRM800K label leakage identified by Reviewer MY3o; the corrected gain remains significant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.