acceptodds
Under review as a conference paper at ICLR 2027

Trust but Verify: Prover-Verifier Deliberation for Selective LLM Prediction

Abstract

Reliably knowing when a language model is correct is almost as important as being correct. We introduce prover-verifier deliberation (PVD), an inference-time protocol grounded in interactive proof theory, as a mechanism for selective prediction: the protocol produces both an answer and a structured confidence verdict, allowing a system to report high-confidence answers while abstaining on uncertain cases. In each dialogue, a prover defends a candidate answer through checkable sub-claims while a verifier issues targeted challenges and returns Accept, Challenge, or Reject. Because frozen language models are imperfect provers and verifiers operating over a noisy channel, formal soundness and completeness guarantees do not transfer; instead, we characterize the protocol empirically through its coverage-precision behavior. Our main experiment uses Claude Sonnet 4.6 as prover and Claude Haiku 4.5 as verifier on GPQA Diamond. Questions accepted with no answer revision, which we call Accept + No Change (ANC), are reported as the high-confidence subset; we evaluate this subset by its precision and coverage. PVD achieves HC-Prec 84.2% at HC-Cov 77% with roughly three LLM calls per question. ANC separates reliable from unreliable answers, yielding a 30pp HC-Prec gap over the non-ANC complement. Robustness experiments with GPT and Gemini pairings show that high HC-Prec can transfer across model families, while verifier strictness and domain competence largely determine the size of the selection gap. On Humanity's Last Exam, weaker prover-verifier pairings can collapse or invert the ANC signal, illustrating a practical failure mode when the verifier operates outside its effective region. Comparisons with self-consistency, universal self-consistency, multi-agent debate, and Reflexion suggest that prover-verifier deliberation supplies a distinct argument-defensibility signal for selective prediction rather than merely reproducing sample agreement or debate consensus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
selective prediction, prover-verifier deliberation, interactive proofs, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_W1ndOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies selective prediction using a dialogue between two frozen LLMs. A prover proposes an answer with atomic sub-claims, and a verifier accepts, rejects, or challenges the argument. Answers accepted without revision (ANC) form the high-confidence subset. Experiments on GPQA Diamond and the multiple-choice subset of HLE examine precision and coverage, verifier choice, cost, and failure modes. I view the main contribution as the empirical study of ANC rather than a fundamentally new prover-verifier mechanism.

Critical strengths
  1. The HLE results identify a consequential failure mode: acceptance can become negatively associated with correctness when the verifier is insufficiently capable. This cautions against treating an additional LLM check as automatic evidence of reliability.

  2. The evaluation includes several useful baselines and ablations, and acknowledges competitive or stronger results from verbalized confidence and self-consistency. This makes the empirical picture more informative than a comparison based only on native high-confidence operating points.

Critical weaknesses
  1. Limited methodological novelty. I am not yet convinced that the incremental contribution is sufficient. Multi-round discussion among LLMs is established in multi-agent debate [1], and the answer-defending model plus questioning judge closely resembles the consultancy setting in [2]. The paper acknowledges this lineage, so its distinctive proposal is mainly the ANC selection rule and its empirical analysis. Please clarify what the protocol itself adds beyond these existing interaction structures.

  2. Unclear practical advantage over simpler confidence signals. Table 7 shows that verbalized confidence is already competitive or stronger at several operating points. Showing that ANC selects a different subset does not by itself establish a useful improvement in reliability or cost. What demonstrated setting makes PVD preferable in precision, coverage, and cost? Is the gain from combining ANC with confidence or consistency established on held-out data, rather than only through disagreement-subset analysis?

  3. The benefit of interaction is not fully isolated. The self-verification baseline in A.7 is useful, but changes the checking procedure and verifier relative to the main PVD configuration. What evidence most directly isolates interactive challenges over a one-shot judge using the same verifier on the same candidate answers? Please distinguish improvements in answer generation from improvements in selecting reliable answers.

  4. Limited evidence for the No Change condition. The three-tier analysis supports separating unchanged and revised acceptances descriptively, but the revised-acceptance groups contain only 12-15 questions and the HIGH-versus-MEDIUM difference is not individually significant. Using the existing transcripts, how does ANC compare with accepting all ACCEPT outcomes, including revised answers, at comparable coverage? How strong is the evidence that excluding revised answers is necessary?

  5. Scattered presentation. The short method description is separated from pseudocode and prompts in the appendix, while important empirical analyses appear throughout the discussion and appendix. This makes it harder to assess the precise protocol and connect the main claims to their supporting experiments.

  6. Table 8 is clipped. On page 24, the table extends beyond the page boundary, cutting off the final column and preventing the verifier-competence results from being read in full.

[1] Yilun Du et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate. ICML, 2024. https://arxiv.org/abs/2305.14325

[2] Zachary Kenton et al. On Scalable Oversight with Weak LLMs Judging Strong LLMs. 2024. https://arxiv.org/abs/2407.04622

Questions for authors

See the weaknesses.

2 comments

Thank you for the specific questions. We answered them from our logged runs on GPQA Diamond (n=198 unless noted), plus new runs of two prior interaction-based methods. The revision reports all of these results.

1. Novelty. We agree that the interaction structure is not new. Multi-agent debate (Du et al., 2024) and consultancy (Kenton et al., 2024) came first, and cross-examination (Cohen et al., 2023) is closer still. The contribution is the ANC selection rule and an empirical account of when it carries signal, including where it fails on HLE. The revision states the contribution in those terms.

2. Practical advantage over stated confidence. We have not demonstrated one. At ANC's own coverage, ANC precision minus stated-confidence precision is +1.6 points (95% CI -4.1 to +7.2) for standard PVD, +4.0 (-1.6 to +9.4) for the same-model verifier, and -0.8 (-5.9 to +4.6) for the challenge-first prompt. On held-out data the answer is also no: a logistic combination of ANC and stated confidence, scored out of fold on the 194 questions with a parsed confidence, does not improve AURC (every interval includes zero). Within confidence tiers, ANC still separates correct from incorrect answers in two of the three runs (Cochran-Mantel-Haenszel across four confidence tiers, p=3e-6, p=5e-4, p=0.07), so ANC carries information beyond stated confidence, but at this sample size we cannot claim better selection. The same holds for the closest prior methods, which we have now run on the same questions and answers: cross-examination (Cohen et al., 2023) and Cooperate and Compete (Feng et al., 2024) are not significantly more precise than stated confidence at matched coverage either. Where PVD is preferable is against those methods. At identical coverage (63.1%), PVD with one model in both roles is 7.2 points more precise than Compete (95% CI 1.7 to 13.1), and cross-examination costs about five times as much per question as PVD with a Haiku verifier.

3. Isolating interaction. The closest evidence in our logs compares the verifier's first-turn Accept with the full dialogue, using the same verifier on the same first answer. Coverage and precision move from 14.1% at 92.9% to 54.5% at 93.5% for standard PVD, from 5.1% at 100% to 65.2% at 89.9% for challenge-first, from 24.7% at 100% to 42.9% at 97.6% for GPT-5.4 with GPT-5.4-mini, and from 64.2% at 96.0% to 87.0% at 91.7% for DeepSeek (n=193). Dialogue therefore adds coverage, usually at some cost in precision. This is only a proxy, because a verifier that may challenge will often defer, so we also ran the test you suggest: the same verifier, on the same first proofs, must Accept or Reject at once. With the Haiku verifier, the one-shot judge accepts 50.0% of questions at 88.9% precision, against 54.5% at 93.5% for the dialogue (dialogue minus one-shot: coverage +4.5, 95% CI -2.0 to +11.1; precision +4.6, -1.4 to +11.0). With Sonnet as verifier, the one-shot judge accepts 82.3% at 87.1%, against 63.1% at 94.4%: the dialogue is 7.3 points more precise (2.8 to 12.1) at 19.2 points less coverage. Interaction therefore mainly buys precision; with the Haiku verifier the dialogue is ahead on both, though neither difference is individually significant. On answer generation versus selection: dialogue changes overall answer accuracy by -2.0 to +3.5 points across runs, so the evidence supports PVD as a selector, not as a reliable improvement in answers.

4. The No Change condition. Revised acceptances are less accurate than ANC on GPQA: 66.7% versus 93.5% for standard PVD (12 questions, Fisher exact p=0.013), 86.7% versus 91.1% for min-1 (15, p=0.63), 61.3% versus 94.4% for the same-model verifier (31, p<0.001), and 70.0% versus 89.9% for challenge-first (20, p=0.024). These p-values are uncorrected; after a Bonferroni correction only the same-model run clears 0.05. Compared with accepting every Accept outcome, ANC gains 0.6 to 6.6 points of precision and gives up 6.1 to 15.7 points of coverage. The submission's statement that this step is not individually significant was wrong, and the revision corrects it. The rule does not help on HLE: for Sonnet with Haiku, revised acceptances reach 23.5% against 15.2% for ANC (119 and 158 questions, p=0.088). The rule also matches the formal setting of Zhang et al. (2026), where a verdict is about a fixed statement; once the prover revises its answer, the dialogue is checking a different claim.

5 and 6. Presentation and Table 8. The revision returns Algorithm 1 to the main text beside the protocol description and moves the comparisons that support the main claims next to them. Table 8 now fits the page, with no change to its values.

thanks for the additional exp. in the point 2 of the response, the author showed and acknowledged that PVD shows no advantage compared with model's own stated confidence. also PVD needs more LLM calls than stated confidence (which is one more call). i would keep my original score.

Reviewer_nmAyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a new method for estimating the correctness of outputs by LLMs. The goal is to identify which LLM answers are likely to be correct. The authors borrow terminology from the interactive proofs literature, viewing the LLM that provides the answer as a prover and introducing a verifier that challenges the answer. Specifically, the prover and the verifier engage in a multi-round protocol where the verifier outputs Accept/Reject/Challenge. The verifier is time/resource-bounded while the prover is assumed to be powerful. They propose to mark answers as correct with high confidence if after the interaction, the verifier accepts and the prover doesn’t change the initial answer given (in the setting of multiple choice questions).

The authors provide experimental results by running their protocol on GPQA diamond and humanity’s last exam (only the multiple choice questions), using ground truth answers to measure how accurately the protocol identifies correct answers.

Critical strengths

The paper studies an important and well-motivated question. When an LLM provides an answer, we need to know when to trust it and when not to. A natural approach is to ask another LLM (which is also powerful) to cross examine its reasoning interactively. The authors also give empirical evidence to show that this approach can be effective.

Critical weaknesses

-- The paper’s connection to interactive proofs feels somewhat forced. It may have been clearer if the authors presented their contribution without this IP abstraction. A defining feature of an interactive proof system is that it satisfies completeness and soundness guarantees, where soundness holds against arbitrary malicious provers. The protocol presented in the paper does not provide such guarantees.

-- Some important comparisons with prior work are missing. https://arxiv.org/abs/2305.13281 considers an extremely similar setting with cross examination and interaction. https://arxiv.org/abs/2402.00367 also considers a similar setting. Both papers are missing from the references, so it is difficult to assess how novel the approach is.

Questions for authors

How does the proposed approach compare with prior work that studies similar multi-LLM interactions?

1 comment

Thank you for the review and for pointing us to these two papers.

Interactive-proof framing. Zhang et al. (2026) state the conditions under which this structure does have completeness and soundness guarantees. The revision uses them to say precisely what PVD does and does not guarantee.

Prior work. We should have cited both papers, and the revision discusses them. The designs differ in what the second model sees and does:

  • Cross-examination (Cohen et al., 2023): an examiner sees only the claim, asks free-form questions over several turns, and ends with a verdict of correct or incorrect.
  • Cooperate (Feng et al., 2024): models write feedback on the answer and a judge decides whether to abstain. Compete: another model writes passages for each of the other options, and the model abstains if they change its answer.
  • PVD: the prover commits to an argument of atomic sub-claims, the verifier reads it and accepts, rejects, or challenges one sub-claim, and only acceptances without a change of answer are reported.

We do not claim that multi-turn interaction is new; Cohen et al. is the closest earlier design. Your question: how does PVD compare? We ran both methods on the same 198 GPQA Diamond questions. They examine the same Sonnet 4.6 answers, 76.3% of them correct, so they differ only in which answers they accept; PVD rows come from PVD's own runs on the same questions. Coverage and precision:

  • PVD, Sonnet prover, Haiku verifier: 54.5%, 93.5%
  • PVD, Sonnet in both roles: 63.1%, 94.4%
  • Cross-examination, majority of three, Haiku examiner: 82.8%, 87.8%; Sonnet examiner: 79.3%, 86.6%
  • Compete, Haiku passages: 52.0%, 92.2%; Sonnet passages: 63.1%, 87.2%
  • Cooperate, self: 83.3%, 85.5%; with other models: 80.8%, 82.5% Three results stand out:
  1. At identical coverage, PVD with Sonnet in both roles is 7.2 points more precise than Compete with Sonnet passages (95% CI 1.7 to 13.1).
  2. Cross-examination and Cooperate accept many more answers at lower precision. Cross-examination with a Haiku examiner costs about five times as much per question as PVD with a Haiku verifier.
  3. None of these methods, PVD included, is significantly more precise than the model's own stated confidence at the same coverage. The revision says so plainly.
Reviewer_46JhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper is introducing a framework to apply interactive proofs tools to problem solving setups using LLMs. It proposes a proposer verifier setup, where a verifier should accept/reject/challenge the proper proof for an answer.

Critical strengths
  • The problem is interesting. Having systems that can reason and have certified guaranteed is of importance for a wide range of realworld applications. I also do like the idea of bringing tools from interactive proofs literature to this problem.
Critical weaknesses
  • HC-Prec and HC-Cov is repeatedly used in the abstract and intro for reporting results, but it is not defined (or even explained in words). This makes it hard to understand the high level results based of the first two pages, which is not good.

  • The framework is related to the work of [1]. Particularly, the modeling of the verification problem thorugh the a three-way decision making of accept or reject or escalation (challenge at the authors call it). Furthermore, the idea of having a fast (but potentially unreliable) verification score, and then solving the verificaiton decision task based on that score is also explored in [1]. The author has not discussed the work [1] in their submitted manuscript. It should be properly discussed, and implemented if it fits in a fair experimental setup (or argued why it is not possible to empirically compare). That being said, I understand that the scope of these two works are different.

  • It appears that almost all of the algorithm section is moved to appendix. The reader is left with section 3.2 to figure out what the algorithm is. THIS IS NOT ACCEPTABLE. I understand this is likely due to page limits. However, the current manuscript is misusing the appendix. The point of the page limits is to try to communicate the essence of the contributions, including technical parts and algorithms, in limited pages, so that it incentives conciseness. Moving the whole algorithm to the appendix is not the way to do so!

  • The paper lacks a proper section introducing the framework, notations, and problem formulation. Probabilistic terms have used without a proper definitions and explanation of what are the random variables and what is the probability over. (for instance look at line 185 to 194). Later on, the authors also talk about some arguments that their algorithm does XYZ against an adversary that does ABC. What is the adversary? Are you following an adversarial setup or stochastic? Perhaps most importantly, WHAT IS THE PROBLEM YOU ARE TRYING TO SOLVE? I can only see some high level discussions that you are applying interactive proof ideas to a proper verifier setup.

[1]: When to Trust the Cheap Check: Weak and Strong Verification for Reasoning, ICML 2026

Questions for authors

None.

2 comments

Thank you for the review. We agree with most of these points, and the revision will address them.

Undefined metrics. The abstract and introduction use HC-Prec and HC-Cov before Section 3.1 defines them. The revision will explain both at first use: HC-Cov is the fraction of questions the system reports, and HC-Prec is the accuracy on those questions.

Problem statement. The revision will add a short formulation: given a multiple-choice question, a system returns an answer and a decision to report or abstain, with no access to the label, and we evaluate precision on the reported subset at a given coverage. The probabilities in lines 185 to 194 are empirical rates over a fixed benchmark, which the current text does not say. Section 6.4 is an empirical test against one instructed deceptive prover. It is not an adversarial guarantee, and the paper makes no worst-case claim.

Algorithm. We moved the pseudocode to the appendix to meet the page limit, and that left Section 3.2 too thin. The revision will return Algorithm 1 to the main text, using the additional page allowed at this stage.

Related work. We did not discuss Kiyani et al. (ICML 2026), and we should have. Their policy sets accept, reject and escalate thresholds on a scalar weak-verifier score and controls error rates using strong-verifier labels. Our verifier is a multi-round dialogue that can change the answer under evaluation, so their guarantee does not carry over directly, and 198 questions would be a small sample for it. The revision will discuss the relationship and will compare against a two-threshold policy offline, using our logged runs, where the setups match.

Continuing our comparison with Kiyani et al. (ICML 2026), their policy maps a scalar weak-verifier score to accept, reject, or a query to a strong verifier, and updates two thresholds online so that Type-I and Type-II error relative to the strong verifier stay near target levels. The three-way decision is shared, but the third option differs. In PVD, Challenge does not call a stronger checker: the same verifier asks the prover to justify one sub-claim, and the prover may change its answer. PVD also has no strong verifier at decision time, and the non-ANC questions are withheld. In their framework, those withheld questions are what would go to the strong verifier, so the two approaches compose: PVD's outcome could serve as the weak signal in their policy.

A direct comparison on our benchmarks would not be like-for-like. In their experiments the strong verifier is a ground-truth check, so on GPQA it would consult the answer key for escalated questions, while every method we compare decides without labels. The closest label-free analogue is escalating non-ANC questions to a stronger model, which our appendix evaluates. It does not pay off on GPQA Diamond: PVD (Sonnet/Haiku) deferring non-ANC questions to Opus 4.6 reaches 82.3% accuracy at $0.077 per question, against 84.3% at $0.067 for Opus alone.

We also ran the two closest interaction-based methods, which Reviewer nmAy requested: cross-examination (Cohen et al., 2023) and Cooperate and Compete (Feng et al., 2024). All use the same 198 GPQA Diamond questions and the same Sonnet 4.6 answers. Coverage and precision:

  • PVD, Sonnet/Haiku: 54.5%, 93.5%
  • PVD, Sonnet/Sonnet: 63.1%, 94.4%
  • Cross-examination, Haiku examiner, majority of three: 82.8%, 87.8%
  • Compete, Haiku passages: 52.0%, 92.2%
  • Cooperate, self: 83.3%, 85.5%

At identical coverage, PVD with Sonnet/Sonnet is 7.2 points more precise than Compete with Sonnet passages (95% CI 1.7 to 13.1). Cross-examination covers more questions at lower precision, at about five times PVD's cost per question. None of these methods, PVD included, is significantly more precise than the model's own stated confidence at matched coverage. The revised appendix reports all variants, costs, and our adaptations of both methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.