acceptodds
Under review as a conference paper at ICLR 2027

LLM Verifiers Are Wrong Together: Confident Consensus Errors Set the Floor for Failure Prediction

Abstract

LLM verifiers decide which generated claims reach users, and their confidence decides which verdicts a human rechecks. How much of a verifier's failure that confidence can expose has not been measured. We introduce a detectability decomposition: for any set of failure signals and any review budget, it measures the share of verifier errors that no signal in the set ranks for review. On 22,189 claims from 11 verification datasets, with three further families held out of distribution, the verdict-token margin comes free with the verdict and flags 61.8% of GPT-4o's errors at a 20% review budget. A self-evaluation signal recovers only 37.2% of the errors the margin misses, pooling four signals lowers the missed share from 38.2% to 31.6%, and their union does better only by reviewing 37.0% of claims. The errors that remain are not unstable verdicts. At the same confidence, a verifier from a different vendor is wrong on them 6.0–36.1 times as often as on correct claims, across a nine-verifier panel from five vendors the median cross-vendor error co-occurrence is 7.75 times the independent rate, and flagging every claim on which any panel member dissents recovers 8.8% of the core. A blind re-annotation of 120 core errors shows what this consensus is made of: 59.2% are benchmark-label errors that the verifiers rejected together and correctly, 40.8% are confirmed verifier errors, and the cross-vendor gap survives correction for this label noise (4.3–30.2×). LLM verifiers are right together against bad labels and wrong together on hard claims, and both are invisible to confidence-based review. An intervention on GPT-4o points to one mechanism: once accepted edits that a blind audit found still supported are removed, meaning-violating edits that keep the document's vocabulary are confidently accepted 4.3pp more often than those that change it (CI [2.7, 5.9]). The decomposition prices what can be guaranteed: a 5% residual-risk target is reachable at a 13.9% budget, a finite-sample document-level guarantee costs 52.4% (25.1pp for the loss definition, 13.4pp for finite-sample slack), and thresholds certified before two unannounced replacements of the serving snapshot met their target in 100 and 94 of 100 recalibrations after them.

Primary area:
datasets and benchmarks
Keywords:
evaluation of LLM verifiers, failure prediction, selective prediction, label errors in benchmarks, correlated errors, uncertainty quantification, conformal risk control, model drift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NQJ5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper performs an empirical analysis of various LLM confidence elicitation methods and studies their failure rates when used together on the task of fact verification.

The authors give a detailed breakdown of what % of errors are caught by distributional, self-evaluative, and behavioural approaches to confidence elicitation. They also give a sense of how approaches perform when combined together, and what % of cases are covered by various combinations of approaches.

The authors show that there are often confident consensus decisions among methods, where all elicitation methods collectively fail to flag an erroneous prediction.

Critical strengths
  1. The topics of claim verification and uncertainty estimation are highly relevant.

  2. The paper exposes benchmark errors, and sharing these errors with the community could help increase the quality of shared research resources.

Critical weaknesses
  1. The paper is hard to read, and the statistics are poorly presented.

Many sentences carry several statistics with little contextualization or explanation of what these statistics imply. This issue extends to the abstract as well. Several terms like "a nine-verifier panel from five vendors" are used without proper definition of the key terms.

  1. The paper reports many results, but these do not particularly help inform future decisions on how to think about confidence elicitation or claim verification more broadly:

The signal-level findings largely confirm prior work: the verdict-token margin is a strong free baseline, verbalised confidence is degenerate, and P(True) offers a modest gain at the cost of an extra call. The core is hard to act on. The recommendations in Section 7 are either standard practice or not well supported by the paper's own data. For example, the claim that an agreeing second verifier "adds no evidence of correctness" conflicts with Section F, where error rates are substantially lower on claims Gemini agrees with.

  1. The paper omits study of arguably the most useful confidence elicitation method:

The paper does not evaluate trained probes on hidden states, which prior work identifies as among the most effective failure predictors and which can recover precisely the self-consistent errors that make up the core. The authors acknowledge this gap but attribute it to scope, even though their panel includes open-weight models. Without them, the claim that the core is set by "the information all signals share" is established only for output-level signals. Likewise, the decomposition is computed only for GPT-4o; the other eight verifiers enter only through error co-occurrence, so it is unknown whether the size and composition of the core hold for smaller open models or for fine-tuned verifiers such as MiniCheck.

Questions for authors
  1. Concretely what should a practitioner or verifier designer do differently after reading this paper?
4 comments
AuthorsOpenReview ↗

Response to Reviewer NQJ5

We thank the reviewer for reading §7 against Appendix F and for naming the experiment the paper lacked. The reviewer is right on both counts.

W1 (hard to read; statistics poorly presented; undefined terms). Agreed. Every term is defined before first use (glossary in §3.3; the panel is introduced at the start of §4.2), Algorithm 1 is in §3.3 with a worked example, and the abstract goes from 25 numbers to 9 and §1 from ~45 to 12.

W2 (confirms prior work; hard to act on; §7 vs. Section F).

  • Prior work: agreed for the signal ranking, and §4.1 now opens by saying so. Not in the prior work the reviewer lists: the core's size under volume and label controls (18.4% union, 7.5% confirmed); its composition (59.2% label error, by blind re-annotation); its price (a finite-sample guarantee needs a 52.4% budget: the 13.9% floor, 25.1pp for the loss definition, 13.4pp of slack); and, from the experiments the reviewer asked for, that it is largely the same claims across verifiers.
  • §7 vs. §F: the reviewer is right, and the sentence is corrected. Among accepted claims the error rate is 3.9% when Gemini agrees and 23.5% when it disagrees (2.7% vs 16.9% in the top margin bin; new Table 6b), so agreement is evidence. The old sentence stated a different fact (Gemini agrees with 70.6% of GPT-4o's confident misses) as if it were this one.
  • Standard practice: agreed for escalation and fingerprint logging, now a one-line reminder. §7 leads with the recommendations that are not standard (Table 17).

W3 (no hidden-state probe; floor shown only for output-level signals; only GPT-4o; fine-tuned verifiers). Done on exactly those models (Tables 1b, 1c). The reviewer's expectation holds for errors in general and not for the consensus core.

  • Confirmed. A probe on the hidden states of Qwen3.5-9B or Gemma 4 12B, trained on GPT-4o's errors (Tan et al., 2025), is the strongest in-distribution predictor we measured (AUROC 0.84–0.85; margin 0.78). It recovers 58–63% of the errors the margin misses and lowers the floor by 7.0–8.0pp at matched review volume. As pre-registered, this narrows our claim: the abstract and §1 now say "output-level".
  • Not the core. The probe leaves 59–66% of the core unranked, recovers less of it than a score that knows only each dataset's error rate (34–41% vs 43.8%), ranks it at chance within a dataset (20.0–27.5%; chance 20%), and does not transfer to a held-out benchmark or FELM/FEVER.
  • The control is symmetric. The same within-dataset standardization leaves our own core unchanged (24.0% raw, 23.4% within dataset; difference CI [−3.5, +1.8]). The core is not a dataset effect; the probe's raw recovery of it is.
  • Size and composition on other verifiers. With five panel verifiers and Bespoke-MiniCheck-7B as primary, the 20% core is 16.1–26.8% of errors, 69–81% of it is GPT-4o's core, and 67–80% of its audited items are label errors.
  • A limitation we state (§8). Gemma's probe on its own errors recovers 41.4% of its own core within datasets, concentrated in RAGTruth.

Q (what should a practitioner or verifier designer do differently?). Table 17 gives one row per decision. A deployer: (1) rank by the free margin instead of paying for a second call; (2) budget 13.9% for a 5% residual-risk target and choose the guarantee's loss first, since the loss definition accounts for 25.1pp of the cost; (3) monitor with a ~17× cheaper verifier (AUROC 0.778 vs 0.784); (4) review disagreements, not agreements; (5) supply evidence, which turns silent errors into abstentions. A designer: (6) avoid agreement-weighted pseudo-labels, which inherit a 7.75× consensus error; (7) add vocabulary-preserving meaning violations to training data (+4.3pp confident acceptance); (8) fit any white-box probe on the target benchmark.

Strength 2 (sharing the benchmark errors). The 210 adjudicated labels are released keyed by benchmark IDs, with the per-dataset distribution in Table 18.

If any of these leaves a weakness open, we would be grateful to know which, so that we can address it before the discussion closes.

AuthorsOpenReview ↗

To Reviewer NQJ5

Dear Reviewer NQJ5,

We understand this is a busy period and we are sorry to add to it. We would be very grateful if you could take a moment to look at our response and the revised paper. In short, we corrected the Section 7 sentence you showed to conflict with Appendix F, and we ran the experiments you asked for: hidden-state probes on the open-weight panel verifiers, and the decomposition on five further verifiers and a fine-tuned one (Tables 1b and 1c). If any concern remains, please let us know and we will respond right away. If your concerns have been resolved, we would be very glad to hear that as well. Thank you again for your time and for the careful review.

Reviewer_NQJ5OpenReview ↗

Too much AI-generated content for my liking

Frankly, there is too much Claude-style writing in both the response to my review and the text of the paper for me to actively consider raising my score. It's not that AI writing is by itself bad, but rather it makes the paper one that simply does not communicate an effective argument.

For example, even in the above comment: ", Algorithm 1 is in §3.3 with a worked example, and the abstract goes from 25 numbers to 9 and §1 from ~45 to 12." is classic Claude-Code style output, and it shows a lack of genuine engagement with the spirit of what my review was.

I am pretty sure: "§7 vs. §F: the reviewer is right, and the sentence is corrected. Among accepted claims the error rate is 3.9% when Gemini agrees and 23.5% when it disagrees (2.7% vs 16.9% in the top margin bin; new Table 6b), so agreement is evidence. The old sentence stated a different fact (Gemini agrees with 70.6% of GPT-4o's confident misses) as if it were this one." is copy pasted from a claude console.

I would prefer not to spend more time on this review as a result.

The core issue is, once again, not that AI is in principle bad at writing papers, but rather it has some error modes which this paper falls into as a consequence.

The above sentence: "Among accepted claims the error rate is 3.9% when Gemini agrees and 23.5% when it disagrees (2.7% vs 16.9% in the top margin bin; new Table 6b), so agreement is evidence. " is simply not as coherent to human readers as it is to LLMs.

I prefer to reject papers that are not written with human readers in mind.

AuthorsOpenReview ↗

Response to Reviewer NQJ5

Thank you for clarifying your concerns. We understand your point about the readability of our paper and acknowledge that some of our responses were overly condensed. However, we would like to distinguish the style of our writing from the scientific effort behind it. We took your original review seriously and devoted considerable time to addressing your concerns, including conducting additional analyses and revising the manuscript. We respect your assessment of the writing quality, but we hope that our research and the substance of our responses will also be considered independently of assumptions about how the text was produced.

Reviewer_icG2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies failure prediction for LLM verifiers, focusing on errors that remain undetected by confidence-based signals. The authors introduce a detectability decomposition that quantifies the fraction of verifier errors missed under a given budget. Through experiments across multiple verification datasets and LLMs, they investigate correlated errors among verifiers, benchmark-label errors, and the limitations of confidence-based review. The paper also examines lexical-overlap effects and applies existing risk-control methods to estimate the review budget required for reliability guarantees.

Critical strengths
  • Relevant problem. Understanding when LLM verifiers fail confidently, and whether different verifiers make correlated errors, is important for reliable automated evaluation.
  • Extensive empirical analysis. The paper evaluates multiple confidence signals, verifier models, and datasets, including additional human annotation and intervention experiments.
Critical weaknesses
  • Poor presentation and organization. I found the manuscript extremely difficult to follow. The exposition introduces numerous definitions, experimental settings, and findings without sufficient explanation or a clear narrative connecting them. Important definitions, such as those between error detectability, correlated failures, and attainable risk guarantees, are difficult to follow. It is unclear what the paper contributes beyond these existing approaches.

  • Unclear contributions. The paper covers several topics, including failure prediction, correlated verifier errors, label quality, and risk guarantees, but their relationship and the main technical contribution are difficult to identify. The proposed detectability decomposition appears to be a relatively straightforward accounting of errors missed by a set of confidence signals, while the risk-control analysis largely applies existing methods. It is unclear what the main methodological advance is beyond these existing frameworks.

Questions for authors

Q1. What is the main contribution of the proposed detectability decomposition beyond standard risk–coverage analysis in selective prediction? What additional insights does it provide over existing evaluation measures?

Q2. How do the different analyses (correlated verifier errors, label auditing, lexical-overlap interventions, and risk guarantees) connect to the paper's main contribution? Could the authors clarify which findings are central and which are supporting analyses?

Q3. To what extent is the claimed floor on failure prediction specific to the confidence signals evaluated in this paper? Could alternative signals or trained failure detectors recover errors in the identified core?

4 comments
AuthorsOpenReview ↗

Response to Reviewer icG2

We thank the reviewer for three questions that name the paper's organising problem.

One result of the revision is new and answers Q3 directly. Within a dataset, no signal we tried (panel dissent, a fine-tuned verifier, a second model's hidden states) ranks the consensus core above chance, while a score that knows only each dataset's error rate "recovers" 43.8% of it. What reaches the core is dataset-level prior, not claim-level information (§4.5, Table 1c).

Q1 (contribution beyond risk-coverage analysis). A risk-coverage curve describes one signal; the decomposition describes a set of signals. It returns what no curve does: (i) whether another signal catches what one misses (P(True)P(\mathrm{True}) catches 37.2% of the margin's misses); (ii) whether a multi-signal gain is pooling or review volume (2.9pp vs 16.9pp of the union's 19.8pp); (iii) what the remaining errors are made of (59.2% label error); (iv) the smallest budget at which any threshold on these scores reaches 5% residual risk (13.9%); (v) whether a new signal lowers the floor, and whether what it recovers includes the core (for the probe: yes, and no).

On "methodological advance". We agree that Algorithm 1 is simple accounting and that the risk-control test is Angelopoulos et al.'s. The methodological content is what makes the floor a valid measurement: the union / equal-budget / equal-volume control with paired document-level bootstrap; label-noise correction with tipping points (94.4–99.6% label error would be needed to erase the consensus); the decomposition of a guarantee's price into floor, loss definition and finite-sample slack; and an evaluation audit of three pitfalls that inflate a naive detector from 0.67–0.72 to 0.93 AUROC. That audit is what caught, during the revision, that raw core-recovery figures reflect dataset base rates.

Q2 (central vs. supporting; how the analyses connect). Central: the core as a measured, shared floor (§4.1, §4.5) and its composition as confident consensus (§4.2–4.3). Supporting: the surface-support intervention (§4.4), the guarantee price (§5.1), the drift case study (§6). One chain connects them, now stated at the end of §1 and in the Figure 1 caption: the size of the core is what no output-level signal removes; its composition shows that the remaining errors are shared, so a second verifier does not remove them either; by Eq. (3) that size is the residual risk no calibration can certify below.

On "learned little or nothing new". Not in the cited prior work: (1) that the core is largely the same claims across six further verifiers, and that apparent recovery of it is dataset prior; (2) its size under volume and label controls; (3) that most of it is benchmark-label error, confirmed by blind re-annotation; (4) the price decomposition of a finite-sample guarantee.

Q3 (is the floor specific to these signals? could trained detectors recover the core?). Tested (Tables 1b, 1c).

  • Errors in general: yes. A cross-model hidden-state probe recovers 58–63% of the errors GPT-4o's margin misses and lowers the floor by 7.0–8.0pp in distribution. As pre-registered, the floor is therefore stated for output-level signals (abstract, §1).
  • The core: no. With the probe added, 59–66% of the core is still unranked. Within a dataset the probes reach 20.0–27.5% of it and MiniCheck 15.0% (chance 20%). The same standardization leaves our own core unchanged (24.0% to 23.4%), so the core is not a dataset effect. The probe does not transfer to a held-out benchmark or FELM/FEVER.
  • Other output-level signals were already in the submission (four behavioral features, Table 3; three negative controls, App. C).

Eq. (2) takes any signal set, so the floor can be re-measured whenever a new signal appears, which is what the revision did with the probe.

On soundness. We read the concern as whether the floor is more than an accounting of the chosen signals. Three checks address it: the volume control separates pooling from reviewing more; the blind audit and the tipping points show that the consensus survives label noise; and every core-recovery figure is now also reported within dataset and against the dataset-error-rate oracle. If a specific number, interval or protocol step is in doubt, we will address it before the discussion closes.

AuthorsOpenReview ↗

To Reviewer icG2

Dear Reviewer icG2,

We understand this is a busy period and we are sorry to add to it. We would be very grateful if you could take a moment to look at our response and the revised paper. In short, we answered your three questions directly: what the decomposition returns that a risk-coverage curve does not (Q1), which findings are central and which are supporting (Q2), and whether trained failure detectors recover the core, which we tested with hidden-state probes and a fine-tuned verifier (Q3). If any concern remains, in particular any specific point behind the soundness score, please let us know and we will respond right away. If your concerns have been resolved, we would be very glad to hear that as well. Thank you again for your time and for the careful review.

Thank you for the rebuttal. I still find the paper extremely hard to follow, starting from the very first sentence of the abstract. Could the authors try to explain their ideas in a more human-readable way? What is the motivation? What are the main challenges? What is the actual contribution?

AuthorsOpenReview ↗

Response to Reviewer icG2

Thank you for your interest. Here is a simple explanation of our paper and its main findings.

Motivation

AI models are increasingly used to check facts. Given a document and a claim, they decide whether the document supports that claim. Since people cannot review every answer, a common approach is to check only the answers the model is least confident about. This works only if the model tends to be less confident when it is wrong.

But how often does this assumption hold? And what happens when a model is confidently wrong? These are the questions we explore.


Main Challenges

Finding confidently wrong answers is not straightforward.

First, existing evaluations usually measure how well a confidence score ranks a model's errors. They do not show which mistakes remain invisible to every confidence score.

Second, combining more scores can look helpful simply because more answers are sent for human review. We need to separate the benefit of additional signals from the benefit of checking more answers.

Third, a disagreement with a dataset's answer key does not always mean the model is wrong. The answer key itself may be incorrect.

Finally, when several models make the same mistake, it is unclear whether this is a problem with one model or a blind spot shared across different models and companies.


Contributions

Most previous work asks how well each confidence score identifies a model's mistakes. We ask a different question: which mistakes cannot be detected by any confidence score, and why? We call these mistakes the core.

1. Measuring the core

Previous studies typically evaluate confidence scores one at a time. We combine them and identify the mistakes that none of them flags. To make the comparison fair, we keep the amount of human review the same, so combining scores does not get an advantage simply by checking more answers.

For GPT-4o, reviewing the 20% least confident answers catches 61.8% of its mistakes. Even after combining four confidence scores, 18.4% of its mistakes remain undetected. Most of the apparent improvement comes from reviewing more answers, rather than from the extra scores themselves.

2. What the core is: mistakes models make together

This is our central finding: the core consists of mistakes that AI models share—and make with confidence. When one model gives a wrong answer to a core sentence, models from other companies can give the same wrong answer and be just as confident.

Across nine models from five companies, a typical pair of models from different companies makes the same mistake 7.75 times more often than would be expected if their errors were independent (median). This means that asking a second model to check an answer may not catch the problem: it may share the same blind spot.

Tan et al. (2025) showed that a detector using another model's internal states can identify certain errors in question answering. We find that, in fact-checking, this approach can detect other mistakes but does not catch the core.

3. What the core contains: wrong answer keys and shared model errors

Not every disagreement with an answer key is a genuine model error. To examine this, we asked people to reassess a sample of core sentences without showing them either the answer key or the models' answers.

In 59.2% of these cases, the answer key was wrong, and the models had correctly rejected it. The remaining cases were genuine mistakes shared by the models. In other words, some cases that look like models failing together are actually cases where the models are right and the dataset is wrong.

4. What the core costs

Core mistakes often remain among the answers accepted without human review. This limits how reliable confidence-based checking can be.

To keep the error rate among accepted answers below 5%, 13.9% of all answers must be checked by a person. To provide a strict statistical guarantee, that figure rises to 52.4%.


In One Sentence

AI fact-checkers can be confidently wrong together, so their confidence may not tell us when an answer needs review. We measure these shared blind spots, explain what they consist of, and show how much human review is needed to address them.

We hope this makes our work clearer. Please feel free to reach out if you have any further questions.

Reviewer_KFQvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper deals with a relevant problem relating to the reliability of LLM verifiers. It concentrates on verifiers failures, introducing a detectability decomposition that measures the fraction of verifier errors that no signal in a set of failure signals ranks for further review (given a review budget). The paper conducts a thorought empirical study using a wide set of verifiers from different vendors showing inter-alia that typically LLM verifiers tend to be simultaneously right on incorrect labels but also simultaneously wrong on hard claims yet both are invisible to confidence-based review

Critical strengths

The paper clearly deals with a relevant problem in view of the fact there’s increasingly the need to make sure that LLM pipelines deliver reliable information. The paper investigates failures in LLM based verifiers, it raises an insightful question (where/how does verifier failures originate?), it introduces a new idea to tackle this challenge (detectability decomposition), and it conducts a thorough study using a number of LLMs from different vendors.

Critical weaknesses

My main criticism is that the paper is very dense, convoluted, and hard to read. The introduction is front-loaded with results/details, problem set-up is difficult to parse (in main paper), and the methodology is also not entirely clear (in the main paper). I would have preferred the main paper to cover in greater depth the main methodological elements, including algorithm 1 (detectablity decomposition) and algorithm 2 (escalate then rank with LTT calibration). The insights are also largely empirical so it is difficult to say how much these generalize.

Questions for authors

Please see critical weaknesses above. The authors also remark in section 3.1 that “Three families are out of distribution and never used for fitting:” — i’d like the authors to elaborate how they’re currently fitting.

3 comments
AuthorsOpenReview ↗

Response to Reviewer KFQv

We thank the reviewer for a review that locates the problem in the presentation and says where. All three presentation points are implemented at the locations below, and the fitting question is answered in full.

W (dense and hard to read; §1 front-loaded; set-up and Algorithms 1 and 2 unclear in the main paper; generalization).

Reviewer's pointChange in the revision
§1 front-loaded§1 opens with the question (is the bottleneck the signal, or the information all signals share?), then the approach, then three findings with 12 numbers instead of ~45
Problem set-up hard to parse§3.3 opens with a glossary box and a 100-claim worked example with two signals, before Eq. (1)–(3)
Algorithm 1 in greater depthMoved to §3.3 (p. 5) with a line-by-line reading: escalate, per-signal top-BB, union / equal-budget / equal-volume, conditional core, bootstrap
Algorithm 2 in greater depthFour-step summary in §5.1 (p. 9): route UNKNOWN verdicts to review, walk a threshold grid, test each threshold with a Hoeffding–Bentkus pp-value on the document-mean loss, stop at the first non-rejection. Why the unit is the document is stated in the same paragraph; full pseudo-code stays in App. I
Largely empirical; generalization§8 is re-organised into what generalizes (the core's size and label-error composition on six further verifiers, Table 1b; the consensus across 9 verifiers from 5 vendors, Table 8; margin transfer out of distribution, AUROC 0.707 / 0.835 / 0.740), what holds in distribution only (a hidden-state probe's gain, Table 1c), what is one-verifier (the exact core values, the intervention), and what is a case study (two drift events)

Q ("Three families are out of distribution and never used for fitting": how are you fitting?). Three things are fitted, all on the in-domain LLM-AggreFact pool only. The margin itself is never fitted.

  1. Learned scores. Margin ⊕\oplus P(True)P(\mathrm{True}) and the 17-feature logistic combination are fitted out of fold with document-grouped 5-fold splits over 10 seeds (§3.2, App. C). Each claim's learned score comes from a model that never saw its document.
  2. Guarantee thresholds. λ^\hat\lambda in Algorithm 2 is selected on the calibration half of a document-level split and evaluated on the other half, over 100 splits (§5.1, Table 2). Learned-score guarantees use a 20/40/40 train/calibration/test split (App. I).
  3. Hidden-state probes (added in the revision). Same rule as item 1, with layer and regularization chosen inside the training folds.

FELM, AVeriTeC and FEVER enter in two labelled ways (Table 2). Zero-shot: the unfitted margin and the in-domain thresholds are applied unchanged (rows "calibrated in-domain"). Calibrated on target: rows where λ^\hat\lambda is fitted on that family's own calibration half, reported separately because the threshold is then no longer out of distribution. P(True)P(\mathrm{True}) and verbalized confidence were collected in-domain only, so no learned score containing them is evaluated out of distribution. §3.1 now carries a box "What is fitted, on what" with these items.

If the revised §1–§3 still reads as front-loaded, a pointer to the paragraph would let us fix it before the discussion closes.

AuthorsOpenReview ↗

To Reviewer KFQv

Dear Reviewer KFQv,

We understand this is a busy period and we are sorry to add to it. We would be very grateful if you could take a moment to look at our response and the revised paper. In short, we restructured the introduction, moved Algorithms 1 and 2 into the main text with a glossary and a worked example, and answered your question on how the fitting is done. If any concern remains, please let us know and we will respond right away. If your concerns have been resolved, we would be very glad to hear that as well. Thank you again for your time and for the careful review.

Reviewer_KFQvOpenReview ↗

LLM Verifiers Are Wrong Together: Confident Consensus Errors Set the Floor for Failure Prediction

I've read the authors rebuttal plus the revised article. I continue to think that this paper is difficult to read; i also agree with reviewer NQJ5 view that the paper is difficult to parse in view of the fact that it seems to rely on AI generated content. I will therefore keep my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to all reviewers

We thank the three reviewers. No reviewer reports an incorrect number, interval, protocol step or script; Reviewer NQJ5 reports one sentence that our own Appendix F contradicts, and it is corrected. The revision keeps every result of the submission and changes four things (changes in blue; main text 10 pages).

1. Presentation (all reviewers). Both algorithms are in the main text (Alg. 1 in §3.3, Alg. 2 summarized in §5.1). §3.3 opens with a glossary and a 100-claim worked example. §1 states the question before the results. The abstract goes from 25 numbers to 9 and §1 from ~45 to 12.

2. What the decomposition adds (icG2, NQJ5). The per-signal ranking confirms prior work, and §4.1 now opens by saying so. The contribution is what a risk-coverage curve of one signal cannot return: whether another signal catches what one misses (P(True)P(\mathrm{True}) recovers 37.2% of the margin's misses), whether a multi-signal gain is pooling or review volume (2.9pp vs 16.9pp), what the residual errors are made of (59.2% benchmark-label error), and what any calibration can certify (5% residual risk needs a 13.9% budget; a finite-sample guarantee costs 52.4%).

3. Requested experiments (NQJ5 W3, icG2 Q3). All were run under outcome sentences committed before each run (PREREG_rebuttal.md); failed predictions are reported.

ExperimentResultWhere
Decomposition with 5 further panel verifiers as primary20% core is 23.6–26.8% of errors (GPT-4o, same signals: 26.3%); 71–81% of each core is GPT-4o's core; 67–71% of audited core items are label errors (GPT-4o: 64.0%)Table 1b
Fine-tuned Bespoke-MiniCheck-7B as primaryCore 16.1%; 69.4% shared with GPT-4o's core; 80.0% label errorTable 1b
Hidden-state probe on Qwen3.5-9B and Gemma 4 12B, trained on GPT-4o's errorsRecovers 58–63% of the errors the margin misses; lowers the floor by 7.0–8.0pp at matched review volume, in distributionTable 1c
Same probe, on the consensus coreLeaves 59–66% of the core unranked; a score that knows only each dataset's error rate recovers more of the core (43.8%) than the probe (34–41%); within a dataset the probe ranks the core at chance (20.0–27.5%; chance 20%)Table 1c
Same probe, on new dataNo gain on a held-out benchmark or on FELM/FEVER with raw scoresTable 1c
Same within-dataset control, on our own signalsCore unchanged: 24.0% raw, 23.4% within dataset (difference CI [−3.5, +1.8])App. F.3

The probe sharpens the main claim and does not undo it. A second verifier's hidden states recover errors that output-level signals miss. As pre-registered, the floor is therefore stated for output-level signals (abstract, §1), which is the signal set the submission measured. The consensus core stays: it is the same claims on every verifier we tried, it is mostly label error, and within a dataset no signal we tried ranks it above chance. What reaches the core is dataset-level prior, not claim-level information (§4.5).

4. Correction to §7 (NQJ5 W2). "An agreeing second verifier adds no evidence of correctness" was wrong. Among accepted claims the error rate is 3.9% when Gemini agrees and 23.5% when it disagrees (new Table 6b), yet 63.5% of the missed errors are ones it agrees with. §7 now says that agreement is weak evidence and that disagreement is the event that should trigger review (+6.6pp of the misses at the same review volume, CI [1.9, 10.9]). Escalation and fingerprint logging, which are standard practice, are now a one-line reminder; the decisions the paper changes are listed in new Table 17.

Released. The 210 adjudicated audit labels keyed by benchmark IDs (final_labels_release.csv; distribution in Table 18), and the scripts and derived outputs of every new experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.