acceptodds
Under review as a conference paper at ICLR 2027

Best-of-Evidence: Best-of-N Selection under Partial Verification

Abstract

BoN improves model outputs by sampling several candidates and selecting one with a proxy score, but it assumes that complete candidates can be evaluated reliably. Many vision-language tasks instead provide only partial verification: a finding, span, value, region, or relation may be checkable even when no dependable whole-response verifier exists. Moreover, the same claim may recur across candidates with opposing stances, allowing one observation to support part of the pool and contradict another. We introduce Best-of-Evidence (BoE), an inference-time selection framework that keeps the BoN candidate pool fixed, represents reusable claims with a signed candidate–factor graph, and allocates a limited budget to evidence actions that can change the final choice. BoE formalizes selection under partial verification and provides a practical score-based controller, with the zero-budget case recovering the underlying BoN decision. Theoretically, we show that residual evidence capacity limits any evidence-driven improvement and that shared factor queries can achieve an versus query separation in a factor-code model. Common-ledger experiments on four medical VQA settings show that BoE can improve fixed-pool selection and rescue some BoN failures when evidence is reliable, contrastive, and decision-relevant, while also revealing the channel-quality and candidate-generation limits that prevent universal gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Best-of-N, Medical MLLM, MLLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_R6ZROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Best-of-Evidence (BoE), an inference-time framework for selecting among a fixed pool of candidate answers when reliable whole-response verification is unavailable. BoE represents reusable local claims using a signed candidate–factor graph, allowing a single observation to support some candidates while contradicting others. It allocates a limited verification budget according to the estimated improvement in final selection per unit cost. The framework distinguishes an ideal Bayesian formulation from a practical controller that uses discrimination-weighted score updates and an approximate value-of-information criterion. The theoretical analysis establishes an information-based upper bound on evidence-driven improvement and demonstrates a logarithmic-versus-linear query complexity separation under an idealized factor-code model. Experiments using shared evidence ledgers across four medical visual question answering settings show modest improvements over majority voting. The results indicate that useful verification requires reliable, contrastive, and decision-relevant evidence, while gains remain constrained by candidate-pool coverage and do not consistently establish an advantage over random evidence acquisition.

Critical strengths
  1. The paper addresses the mismatch between selecting complete responses and having access only to partial verification. This setting is particularly relevant to multimodal tasks, where individual findings or visual relationships may be checkable even when the correctness of an entire response is difficult to assess.

  2. The analysis identifies a residual-information ceiling on evidence-driven improvement and demonstrates a logarithmic-versus-linear query complexity separation under an idealized factor-code model. These results help explain when shared verification can be advantageous and why additional checks may offer limited benefits.

  3. Reporting full-set accuracy, rescue rates, harmful selection changes, and candidate-pool coverage provides a detailed picture of where improvements arise and what constrains them.

Critical weaknesses
  1. The proposed method is interesting, but it relies on several conditions that may be difficult to meet in practice. It needs to correctly identify claims shared by different answers and reliably estimate how useful each verification result is for selecting the correct answer. The paper does not sufficiently test what happens when these estimates or identified claims are wrong. In addition, the theoretical advantage is shown in an idealized setting, while the experimental improvements are small. More evidence is needed to show that the method works reliably across different tasks.

  2. BoE improves accuracy over majority voting by only 0.26–0.58 percentage points, and its advantage over random evidence acquisition is statistically significant in only one of four settings. Moreover, the main comparison gives BoE and the random baseline different verification options, while channel parameters are estimated from the same data used for evaluation. These choices make it difficult to assess the method’s actual contribution and performance on new data. The experiments also lack sufficient analysis of errors in claim extraction and changes in verifier reliability. Consequently, it remains unclear whether the theoretical benefits translate into stable practical improvements.

  3. The experiments use manually assigned verification costs without sufficiently reporting actual runtime, token usage, or monetary costs. It therefore remains unclear whether BoE offers an advantage under realistic resource constraints. Furthermore, the four main settings are all medical visual question answering tasks and primarily use a single generator model. Broader evaluation is needed to establish whether the method remains effective across different tasks and models.

Questions for authors
  1. How robust is BoE to errors in claim extraction and changes in verifier reliability? Can you evaluate these effects using channel parameters estimated from a separate calibration set?

  2. When BoE and random evidence acquisition have the same verification options and budget, does BoE consistently outperform the random baseline across all four settings and multiple runs?

  3. What are BoE’s actual runtime and token costs, including claim extraction and verification selection? Does its advantage hold under these costs and extend to other models and non-medical tasks?

1 comment

We thank the reviewer for the thoughtful review and for recognizing the partial-verification setting, the theoretical analysis, and our detailed reporting of rescues, harms and pool coverage!

[Q1] Robustness to claim errors, verifier reliability, and independent calibration

We agree that the original channel indices were fitted on the evaluation data. We re-ran all four datasets with grouped 5-fold cross-fitting: κ is estimated only from the other four folds, and questions from the same image (VQA-Med) or article (PMC-VQA) always stay in one fold. Our PathVQA ledger retains no image ids, so its folds group by question and same-image leakage cannot be excluded. Comparisons use the factor-only menu and C=16, with Random averaged over 20 seeds and 95% paired-bootstrap intervals.

DatasetIn-sample gap (pp)Cross-fit gap (pp)Cross-fit 95% CI
VQA-Med+0.16+0.11[−0.29, +0.52]
PathVQA+0.14+0.19[−0.07, +0.45]
PMC-VQA−0.10−0.06[−0.31, +0.18]
MedXpertQA-MM+0.22+0.24[−0.05, +0.54]

Cross-fitting changes BoE's accuracy by at most 0.04 pp and the fitted indices by at most 0.04, so the in-sample calibration did not inflate the results. The two anti-correlated PathVQA slots (κ≈0.27–0.33) stay below 0.5 in every fold, a stable property of the channel.

To test claim and verifier errors, we perturbed the cached ledger and replayed BoE without re-fitting κ: removing a fraction of candidate–claim edges (omission), reattaching edges to another claim of the same slot (mis-linking), and flipping VLM verdicts (verifier errors). Each setting averages three perturbation seeds; the tables report the change in BoE accuracy (pp) relative to the clean ledger. Omission is measured on all four datasets:

Omission rateVQA-MedPathVQAPMC-VQAMedXpertQA-MM
10%−0.10−0.02−0.09−0.08
20%−0.23−0.14−0.05−0.12

The mis-linking and verifier-flip runs completed on VQA-Med within the rebuttal window; we will extend them to the other datasets in the revision.

RateMis-linkingVerifier flips
10%0.00−0.09
20%−0.33−0.09

With up to 20% of claims omitted, mis-linked or flipped, BoE's accuracy decreases by at most 0.33 pp, and its rescues and harms relative to raw BoN stay within four questions of the clean ledger (26–30 and 31–38, versus 28 and 34). Across all perturbed settings the gap over Random stays between −0.05 and +0.21 pp, inside the clean-ledger intervals above, so BoE degrades gracefully under claim and verifier errors. Mis-linking is a pessimistic test, since it attaches a candidate to another answer's claim. We will add the cross-fitting protocol and these robustness results to the revised manuscript.

[Q2] Matched action spaces and multiple runs

We agree that the original main comparison gave BoE a richer menu. We now compare BoE-factor, which keeps only factor-level actions, with Random under the same menu, calibration and budget on all four datasets, averaging Random over 20 seeds. BoE-factor outperforms Random on three of four datasets (+0.11 to +0.24 pp under cross-fitting, table above), though all intervals include zero. With the original full menu, BoE's gap over the 20-seed Random is +0.67 pp [+0.14, +1.21] on VQA-Med and +0.45 pp [+0.14, +0.76] on PMC-VQA, and this gain mainly comes from whole-response verification. We will report these matched-menu comparisons and replace the original single-seed Random baseline.

[Q3] Actual cost and generalization

Our instrumented full-set VQA-Med run (235B judge, TP=8, vLLM) logged 55,846 model calls and 38.7M input / 103.6M output tokens for the full evidence ledger. Claim extraction is integrated into generation, with claim slots accounting for about 9% of generated text. Based on measured throughput, the evidence BoE selects at C=16 corresponds to about 876 s and 11M judge tokens, or 9.7% of generation time; graph construction takes 8.7 s in total and verification selection 78 ms per question.

Under these costs, the extra compute is better spent on verification than on sampling. With 16 additional candidates per question, majority voting does not improve (62.50 to 62.27 as K grows from 16 to 32), although doubling generation costs about 10× BoE's evidence. A single whole-response verification step of about 684 s improves selection over raw BoN on all four datasets (+0.47 to +0.95 pp; pooled McNemar p<1e-13).

Our partial-verification formulation and shared-claim mechanism are not specific to medical VQA, while the current experiments cover four medical settings with one generator–judge pair. We will state this empirical scope clearly and discuss evaluation on other tasks and model families as future work.

Reviewer_EH9yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Best-of-Evidence (BoE), a framework that extracts shared facts from candidate answers and selectively verifies evidence that may influence the final decision, aiming to improve answer selection under a limited verification budget.

Critical strengths

1.The research problem is meaningful, and using shared evidence to reduce redundant verification is a reasonable idea.

1.The proposed framework is clearly presented and supported by theoretical analysis.

Critical weaknesses

1.Marginal performance improvements. BoE improves accuracy by only 0.26–0.58 percentage points over the original BoN baseline. Moreover, compared with random evidence selection, statistically significant improvements are observed on only one dataset. These results provide limited evidence for the effectiveness of the proposed method.

2.Lack of computational efficiency evaluation. The paper uses manually assigned verification costs without measuring actual inference latency, token consumption, or computational expenses. Since BoE introduces additional steps, including fact extraction, graph construction, and evidence verification, it is unclear whether such small accuracy gains justify the additional overhead.

3.Insufficiently fair comparisons. BoE can select from more types of verification actions than the random baseline. When the action spaces are matched, the improvement on MedXpertQA-MM drops to only 0.15 percentage points, raising concerns about the actual benefit of the proposed evidence allocation strategy.

4.Insufficient ablation studies. Although the paper includes a factor-only comparison and experiments with different budgets, it does not adequately isolate the contributions of EVSI-based selection, the shared factor graph, and discrimination weighting. Therefore, it remains unclear which components are responsible for the observed improvements.

Questions for authors

1.How do the authors justify the practical significance of the marginal accuracy improvements?

2.Can the authors report actual inference latency, token consumption, and computational overhead?

3.Can the authors provide fair comparisons using matched verification action spaces across all datasets?

4.Can the authors provide ablations to isolate the contributions of EVSI-based selection, the shared factor graph, and discrimination weighting?

If the authors can adequately address the above concerns, I would be willing to reconsider my rating.

3 comments

We thank the reviewer for the positive assessment and constructive feedback!

[Q1] Practical significance

We appreciate the reviewer's concern about the modest accuracy gains (0.26–0.58 pp) and will refine our empirical claims accordingly. Beyond these improvements, we believe BoE's main contribution lies in its theoretical framework for selection under partial verification. We establish a residual-information bound characterizing the limits of evidence-driven selection and prove an O(log⁡K)O(\log K) vs. Θ(K)\Theta(K) query-complexity separation between shared-factor and candidate-level verification under a factor-code model. These results provide a principled foundation for understanding when shared evidence can be valuable. Our new ablations further support this motivation, showing consistent benefits from evidence reuse under fixed budgets. We will emphasize these theoretical and empirical contributions while presenting the gains of the current acquisition heuristic more conservatively.

[Q2] Computational Costs

We thank the reviewer for pointing out the missing discussion of computational cost.

Our instrumented full-set VQA-Med re-run (235B judge, TP=8, vLLM) logged 55,846 model calls and 38.7M input / 103.6M output tokens. The measured times cover batched construction of the full evidence ledger: evidence accounts for 2,191 s, or 16.7% of total time and 24% of generation time.

We would also like to clarify how the design costs in Sec. 5.1 and Appendix C.3 relate to actual inference overhead. In our implementation, whole-response verification costs 8 units per question and evaluates each distinct candidate answer (4.6 judge calls on average). At C=16, the full-menu BoE replay selects 2,353 factor-judge calls, 9,668 whole-response calls, and 3,963 MedSAM crops (about 11M judge tokens). Based on measured throughput, this corresponds to an estimated 876 s, or 9.7% of generation time (not online latency). Graph construction takes 8.7 s in total and the controller 78 ms per question. Claim extraction is integrated into generation, with claim slots accounting for about 9% of generated text. Under matched menus, early stopping reduces design-budget usage by about 28% versus Random. We will report these details in the revised manuscript.

StageTime (s)
Candidate generation9,063
Evidence: factor + whole-response judging1,538
Evidence: MedSAM + request preparation653
Open-answer evaluation (not a policy cost)1,878

[Q3] Using matched verification action

We agree that the original comparison used different verification menus, making it difficult to isolate the benefit of evidence allocation. To address this, we introduce BoE-factor, a BoE variant that removes whole-response verification and retains only factor-level actions, matching Random's action space. We compare both strategies with the same verification options, calibration, and budget (C=16) on all four datasets, averaging Random over 20 seeds.

DatasetBoE-factorRandomGap (pp)
VQA-Med63.0762.91 ± 0.23+0.16
PathVQA29.6029.46 ± 0.16+0.14
PMC-VQA36.4436.54 ± 0.09−0.10
MedXpertQA-MM33.5533.33 ± 0.14+0.22

BoE-factor outperforms Random on three of four datasets (+0.14 to +0.22 pp). The matched menus make this a fairer assessment of evidence allocation. We will also update the original VQA-Med Random baseline using the 20-seed results, revise its significance analysis, and include these comparisons in the manuscript.

[Q4] Component ablations

We thank the reviewer for suggesting this ablation. We conducted three ablations using the same settings as Q3; the table reports accuracy changes (pp) relative to BoE-factor.

Removed/replacedVQA-MedPathVQAPMC-VQAMedXpertQA
Cross-candidate sharing−0.51−0.26−0.48−0.05
EVSI acquisition → Random−0.16−0.14+0.10−0.22
Weights → equal magnitude−0.04+0.19+0.03−0.25

Sharing: Removing it reduces accuracy on all four datasets (473 vs. 386 discordant correct selections; pooled exploratory McNemar p=0.003). Replacing shared checks with candidate-local ones expands the action menu by 1.6–5.6×. This supports our central motivation: one verification result can serve multiple candidates, reducing redundant acquisition under a fixed budget. For EVSI, the comparison with Random in Q3 shows modest differences, while BoE's stopping rule reduces design-budget use, as reported in Q2. Thus, the comparison evaluates the overall acquisition policy rather than ranking alone. Discrimination weighting gives mixed results against equal-magnitude updates, and we will present it as a practical heuristic. These ablations clarify the value of shared verification, and we will refine the claims about individual components in the revised paper.

Thank you to the authors for their response, which addressed most of my initial concerns. I will raise my score accordingly.

We are pleased that we were able to address your concerns, and thank you again for your feedback and for your recognition of our work! We will incorporate these suggestions and all details mentioned above into the revised version to ensure that our claims are appropriately scoped. Thank you again for your thoughtful feedback!

Reviewer_BsLBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Best-of-N samples N answers and picks one with a proxy score, which assumes the whole answer can be scored reliably. The paper argues that in medical VQA you usually cannot do that, but you can check individual facts (modality, body part, view, main finding, one attribute), and that the same fact recurs across many candidates, sometimes asserted and sometimes denied. BoE keeps the N = 16 candidates fixed, extracts these facts into a signed candidate–claim graph, and spends a small budget on checks chosen by a plug-in value-of-information score divided by a design cost (1 for a structured lookup, 4 for a VLM judging one claim, 8 for a VLM judging a whole answer). Each check outcome moves the scores of every candidate that asserts or denies the claim, weighted by a per-dataset reliability index κ fit from the data. With zero budget it is plain majority voting.

Three theoretical results come with it: a local formula for when a shared check is worth buying (under an assumed quadratic model), a constructed example where checking shared bits needs log K queries while checking whole candidates needs order K, and an information-theoretic ceiling on how much any budgeted evidence can improve selection.

Experiments use four medical VQA sets, Qwen3-VL-30B as generator and Qwen3-VL-235B as judge. All check outcomes are pre-computed into a ledger, so policies differ only in which entries they reveal. At budget 16, BoE beats majority voting by 0.26 to 0.58 points. Against randomly chosen checks it is significantly better on one set out of four (VQA-Med, +1.1 points); on the other three the difference is within noise. That comparison also lets BoE buy whole-answer judgments the random policy cannot; with matched menus on MedXpertQA-MM the gap is 0.15 points, i.e. three questions.

Critical strengths
  1. The problem framing is good and I would like to see it developed. Treating a claim that candidates disagree on as the thing to verify, and valuing a check by whether it can flip the decision rather than by its accuracy, is the right way to think about verifying VLM outputs when no whole-answer scorer exists. The paper is also unusually candid about the distance between the Bayesian ideal and the heuristic it runs.

  2. The ledger design is the right experimental setup. Caching one candidate pool plus every possible check outcome and replaying different policies on it isolates the allocation rule from everything else, and the integer accounting of corrections versus harms (Table 8) makes the small effects legible.

Critical weaknesses
  1. The method does not demonstrably beat picking checks at random. Gains over majority voting are under 0.6 points. The comparison that matters, BoE versus random checks at the same budget, is significant on one dataset out of four, and in that comparison BoE has a richer menu than random. With matched menus the difference is three questions in two thousand, and the budget sweep shows no trend. On top of that κ is fit on the evaluation data, so even the VQA-Med number is optimistic. The costs are invented (1/4/8) and never measured; at budget 16 BoE can call a 235B judge on two full answers per question, and the paper never asks whether that compute would do more as extra samples. So the central claim, that value-of-information allocation is what helps, is not established.

  2. The theory is decoration. The method is a heuristic, and none of the three results touches it. The query-separation result needs a candidate pool that enumerates every combination of M binary claims, which the authors admit real pools are not, and it amounts to binary search versus linear scan. The information ceiling is a standard sub-Gaussian bound whose key quantity Λ_C is never estimated; the paper's own proxy is explicitly disclaimed as unrelated to it. The local-dominance proposition assumes a quadratic model with a weight matrix that nothing estimates. Each result carries its own disclaimer that it is not the controller objective. Delete all three and the method and the experiments are unchanged. That leaves one empirical contribution, and it is weak.

Questions for authors
  • Menu-matched BoE versus random on all four sets, with paired intervals. How much of the VQA-Med gain comes from whole-answer judgments?
  • What is the measured token or wall-clock cost of a budget-16 run, and how does BoE compare with spending the same compute on more candidates with majority voting, or on one whole-answer judgment by the same judge?
  • How are claims extracted and merged across candidates? Specifically: are the generator and judge the Instruct or the Thinking variants of Qwen3-VL, what is the generation prompt and output format (the five slots plus final answer appear to be a structured template, but it is never shown)?
2 comments

We thank the reviewer for the detailed review and for recognizing our problem framing and ledger design!

[Q1] Matched menus with paired intervals

We agree that the original comparison used different action menus. We compare BoE-factor, which keeps only factor-level actions, with Random using the same menu, calibration and budget (C=16) on all four datasets. Random is averaged over 20 seeds, and the 95% intervals come from a paired bootstrap (10,000 resamples; clustered by image on VQA-Med).

DatasetBoE-factorRandomGap (pp)95% CI
VQA-Med63.0762.91+0.16[−0.24, +0.54]
PathVQA29.6029.46+0.14[−0.11, +0.40]
PMC-VQA36.4436.54−0.10[−0.34, +0.14]
MedXpertQA-MM33.5533.32+0.22[−0.06, +0.52]

BoE-factor outperforms Random on three of four datasets, though all intervals include zero. We also replaced the original single-seed Random baseline with the 20-seed mean; the full-menu gap on VQA-Med becomes +0.67 pp [+0.14, +1.21]. On VQA-Med, whole-response judgments account for the full-menu gain over raw BoN, while factor-only BoE is 0.26 pp below raw BoN. Removing cross-candidate sharing reduces accuracy on all four datasets (pooled McNemar p=0.003), as reported in our response to Reviewer EH9y. We will report these matched-menu intervals and the 20-seed baseline in the revised manuscript.

[Q2] Actual cost, and the same compute spent elsewhere

Our full VQA-Med re-run (235B judge, TP=8, vLLM) measured the cost of BoE's evidence at C=16 at about 876 s and 11M judge tokens, or 9.7% of generation time. Whole-response verification costs 8 units and judges each distinct answer once, so a budget of 16 buys one whole-response action plus factor checks rather than two full-answer judgments. Full cost details are in our response to Reviewer EH9y.

We also drew 16 more candidates per question with the same generator and grader (averages over 200 random subsets of the 32):

KMajority voteOracle@KExtra generation (× BoE's evidence)
1662.5081.2—
2062.4682.12,266 s (2.6×)
2462.4382.74,531 s (5.2×)
3262.2783.59,063 s (10.3×)

Extra samples do not improve majority voting: doubling the pool adds correct candidates (Oracle@K +2.3 pp), but voting cannot select them. We then purchased only the single whole-response action of BoE, at about 684 s:

DatasetRaw BoNWhole-response onceGain (pp)
VQA-Med63.3263.80+0.47
PathVQA29.2829.88+0.60
PMC-VQA36.4137.14+0.73
MedXpertQA-MM33.3534.30+0.95

This one verification step improves selection on all four datasets (pooled McNemar p<1e-13) at a fraction of the compute that gives no gain as extra samples. Full BoE performs similarly to this baseline (−0.15 to −0.55 pp). These results show that more compute does not by itself lead to better selection. The key question is which evidence to acquire and how to apply it, which is the question this paper studies. Under the current channels, the whole-response check is the most informative action, and factor-level checks will need more discriminative verifiers to add further gains. We will add these compute comparisons, replace the per-candidate whole-response baseline, and revise our claims accordingly.

[Q3] Models, prompt, and claim merging

Both models are Thinking variants: Qwen3-VL-30B-A3B-Thinking (generator) and Qwen3-VL-235B-A22B-Thinking (judge). The generator outputs its reasoning followed by a JSON object with the answer and five fixed slots: modality, anatomical region with a box, view, primary finding with a present/absent polarity, and one answer-relevant attribute. Inapplicable slots are set to none.

Within each question, claims are merged by normalized (slot, value), so candidates reporting the same value share one claim node. Opposing stances on a node come from the finding's polarity and occur on 0.3–7.9% of shared nodes, so most disagreement appears as candidates connecting to different nodes. We will add the full prompt, output examples and merging rules to the appendix.

[Q4] Role of the theory

We agree that the three results are not the objective of the implemented controller, and we will present them as analysis rather than as justification for the heuristic. They do explain the empirical findings. The decision-effective-reuse account (Sec. 4.1) predicts that sharing helps in proportion to decision-relevant reuse, consistent with the sharing ablation, which has the smallest effect on the least-shared graph. Together with Oracle@K, the residual-information bound limits the attainable gain; on MedXpertQA-MM, 706 of 1,333 raw-BoN failures contain no correct candidate. We will also clarify that Λ_C is not estimated on these ledgers.

Reviewer_BsLBOpenReview ↗

Thank you

Thank the authors for the reply, which has addressed some of my concerns. In general, theoretical analysis is appreciated when it is relevant and necessary. I would keep my score due to the limited connection between the theory and the method itself in this paper. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.