- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies selective prediction using a dialogue between two frozen LLMs. A prover proposes an answer with atomic sub-claims, and a verifier accepts, rejects, or challenges the argument. Answers accepted without revision (ANC) form the high-confidence subset. Experiments on GPQA Diamond and the multiple-choice subset of HLE examine precision and coverage, verifier choice, cost, and failure modes. I view the main contribution as the empirical study of ANC rather than a fundamentally new prover-verifier mechanism.
Critical strengths
-
The HLE results identify a consequential failure mode: acceptance can become negatively associated with correctness when the verifier is insufficiently capable. This cautions against treating an additional LLM check as automatic evidence of reliability.
-
The evaluation includes several useful baselines and ablations, and acknowledges competitive or stronger results from verbalized confidence and self-consistency. This makes the empirical picture more informative than a comparison based only on native high-confidence operating points.
Critical weaknesses
-
Limited methodological novelty. I am not yet convinced that the incremental contribution is sufficient. Multi-round discussion among LLMs is established in multi-agent debate [1], and the answer-defending model plus questioning judge closely resembles the consultancy setting in [2]. The paper acknowledges this lineage, so its distinctive proposal is mainly the ANC selection rule and its empirical analysis. Please clarify what the protocol itself adds beyond these existing interaction structures.
-
Unclear practical advantage over simpler confidence signals. Table 7 shows that verbalized confidence is already competitive or stronger at several operating points. Showing that ANC selects a different subset does not by itself establish a useful improvement in reliability or cost. What demonstrated setting makes PVD preferable in precision, coverage, and cost? Is the gain from combining ANC with confidence or consistency established on held-out data, rather than only through disagreement-subset analysis?
-
The benefit of interaction is not fully isolated. The self-verification baseline in A.7 is useful, but changes the checking procedure and verifier relative to the main PVD configuration. What evidence most directly isolates interactive challenges over a one-shot judge using the same verifier on the same candidate answers? Please distinguish improvements in answer generation from improvements in selecting reliable answers.
-
Limited evidence for the No Change condition. The three-tier analysis supports separating unchanged and revised acceptances descriptively, but the revised-acceptance groups contain only 12-15 questions and the HIGH-versus-MEDIUM difference is not individually significant. Using the existing transcripts, how does ANC compare with accepting all ACCEPT outcomes, including revised answers, at comparable coverage? How strong is the evidence that excluding revised answers is necessary?
-
Scattered presentation. The short method description is separated from pseudocode and prompts in the appendix, while important empirical analyses appear throughout the discussion and appendix. This makes it harder to assess the precise protocol and connect the main claims to their supporting experiments.
-
Table 8 is clipped. On page 24, the table extends beyond the page boundary, cutting off the final column and preventing the verifier-competence results from being read in full.
[1] Yilun Du et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate. ICML, 2024. https://arxiv.org/abs/2305.14325
[2] Zachary Kenton et al. On Scalable Oversight with Weak LLMs Judging Strong LLMs. 2024. https://arxiv.org/abs/2407.04622
Questions for authors
See the weaknesses.
2 comments
thanks for the additional exp. in the point 2 of the response, the author showed and acknowledged that PVD shows no advantage compared with model's own stated confidence. also PVD needs more LLM calls than stated confidence (which is one more call). i would keep my original score.
Thank you for the specific questions. We answered them from our logged runs on GPQA Diamond (n=198 unless noted), plus new runs of two prior interaction-based methods. The revision reports all of these results.
1. Novelty. We agree that the interaction structure is not new. Multi-agent debate (Du et al., 2024) and consultancy (Kenton et al., 2024) came first, and cross-examination (Cohen et al., 2023) is closer still. The contribution is the ANC selection rule and an empirical account of when it carries signal, including where it fails on HLE. The revision states the contribution in those terms.
2. Practical advantage over stated confidence. We have not demonstrated one. At ANC's own coverage, ANC precision minus stated-confidence precision is +1.6 points (95% CI -4.1 to +7.2) for standard PVD, +4.0 (-1.6 to +9.4) for the same-model verifier, and -0.8 (-5.9 to +4.6) for the challenge-first prompt. On held-out data the answer is also no: a logistic combination of ANC and stated confidence, scored out of fold on the 194 questions with a parsed confidence, does not improve AURC (every interval includes zero). Within confidence tiers, ANC still separates correct from incorrect answers in two of the three runs (Cochran-Mantel-Haenszel across four confidence tiers, p=3e-6, p=5e-4, p=0.07), so ANC carries information beyond stated confidence, but at this sample size we cannot claim better selection. The same holds for the closest prior methods, which we have now run on the same questions and answers: cross-examination (Cohen et al., 2023) and Cooperate and Compete (Feng et al., 2024) are not significantly more precise than stated confidence at matched coverage either. Where PVD is preferable is against those methods. At identical coverage (63.1%), PVD with one model in both roles is 7.2 points more precise than Compete (95% CI 1.7 to 13.1), and cross-examination costs about five times as much per question as PVD with a Haiku verifier.
3. Isolating interaction. The closest evidence in our logs compares the verifier's first-turn Accept with the full dialogue, using the same verifier on the same first answer. Coverage and precision move from 14.1% at 92.9% to 54.5% at 93.5% for standard PVD, from 5.1% at 100% to 65.2% at 89.9% for challenge-first, from 24.7% at 100% to 42.9% at 97.6% for GPT-5.4 with GPT-5.4-mini, and from 64.2% at 96.0% to 87.0% at 91.7% for DeepSeek (n=193). Dialogue therefore adds coverage, usually at some cost in precision. This is only a proxy, because a verifier that may challenge will often defer, so we also ran the test you suggest: the same verifier, on the same first proofs, must Accept or Reject at once. With the Haiku verifier, the one-shot judge accepts 50.0% of questions at 88.9% precision, against 54.5% at 93.5% for the dialogue (dialogue minus one-shot: coverage +4.5, 95% CI -2.0 to +11.1; precision +4.6, -1.4 to +11.0). With Sonnet as verifier, the one-shot judge accepts 82.3% at 87.1%, against 63.1% at 94.4%: the dialogue is 7.3 points more precise (2.8 to 12.1) at 19.2 points less coverage. Interaction therefore mainly buys precision; with the Haiku verifier the dialogue is ahead on both, though neither difference is individually significant. On answer generation versus selection: dialogue changes overall answer accuracy by -2.0 to +3.5 points across runs, so the evidence supports PVD as a selector, not as a reliable improvement in answers.
4. The No Change condition. Revised acceptances are less accurate than ANC on GPQA: 66.7% versus 93.5% for standard PVD (12 questions, Fisher exact p=0.013), 86.7% versus 91.1% for min-1 (15, p=0.63), 61.3% versus 94.4% for the same-model verifier (31, p<0.001), and 70.0% versus 89.9% for challenge-first (20, p=0.024). These p-values are uncorrected; after a Bonferroni correction only the same-model run clears 0.05. Compared with accepting every Accept outcome, ANC gains 0.6 to 6.6 points of precision and gives up 6.1 to 15.7 points of coverage. The submission's statement that this step is not individually significant was wrong, and the revision corrects it. The rule does not help on HLE: for Sonnet with Haiku, revised acceptances reach 23.5% against 15.2% for ANC (119 and 158 questions, p=0.088). The rule also matches the formal setting of Zhang et al. (2026), where a verdict is about a fixed statement; once the prover revises its answer, the dialogue is checking a different claim.
5 and 6. Presentation and Table 8. The revision returns Algorithm 1 to the main text beside the protocol description and moves the comparisons that support the main claims next to them. Table 8 now fits the page, with no change to its values.