- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Best-of-Evidence (BoE), an inference-time framework for selecting among a fixed pool of candidate answers when reliable whole-response verification is unavailable. BoE represents reusable local claims using a signed candidate–factor graph, allowing a single observation to support some candidates while contradicting others. It allocates a limited verification budget according to the estimated improvement in final selection per unit cost. The framework distinguishes an ideal Bayesian formulation from a practical controller that uses discrimination-weighted score updates and an approximate value-of-information criterion. The theoretical analysis establishes an information-based upper bound on evidence-driven improvement and demonstrates a logarithmic-versus-linear query complexity separation under an idealized factor-code model. Experiments using shared evidence ledgers across four medical visual question answering settings show modest improvements over majority voting. The results indicate that useful verification requires reliable, contrastive, and decision-relevant evidence, while gains remain constrained by candidate-pool coverage and do not consistently establish an advantage over random evidence acquisition.
Critical strengths
-
The paper addresses the mismatch between selecting complete responses and having access only to partial verification. This setting is particularly relevant to multimodal tasks, where individual findings or visual relationships may be checkable even when the correctness of an entire response is difficult to assess.
-
The analysis identifies a residual-information ceiling on evidence-driven improvement and demonstrates a logarithmic-versus-linear query complexity separation under an idealized factor-code model. These results help explain when shared verification can be advantageous and why additional checks may offer limited benefits.
-
Reporting full-set accuracy, rescue rates, harmful selection changes, and candidate-pool coverage provides a detailed picture of where improvements arise and what constrains them.
Critical weaknesses
-
The proposed method is interesting, but it relies on several conditions that may be difficult to meet in practice. It needs to correctly identify claims shared by different answers and reliably estimate how useful each verification result is for selecting the correct answer. The paper does not sufficiently test what happens when these estimates or identified claims are wrong. In addition, the theoretical advantage is shown in an idealized setting, while the experimental improvements are small. More evidence is needed to show that the method works reliably across different tasks.
-
BoE improves accuracy over majority voting by only 0.26–0.58 percentage points, and its advantage over random evidence acquisition is statistically significant in only one of four settings. Moreover, the main comparison gives BoE and the random baseline different verification options, while channel parameters are estimated from the same data used for evaluation. These choices make it difficult to assess the method’s actual contribution and performance on new data. The experiments also lack sufficient analysis of errors in claim extraction and changes in verifier reliability. Consequently, it remains unclear whether the theoretical benefits translate into stable practical improvements.
-
The experiments use manually assigned verification costs without sufficiently reporting actual runtime, token usage, or monetary costs. It therefore remains unclear whether BoE offers an advantage under realistic resource constraints. Furthermore, the four main settings are all medical visual question answering tasks and primarily use a single generator model. Broader evaluation is needed to establish whether the method remains effective across different tasks and models.
Questions for authors
-
How robust is BoE to errors in claim extraction and changes in verifier reliability? Can you evaluate these effects using channel parameters estimated from a separate calibration set?
-
When BoE and random evidence acquisition have the same verification options and budget, does BoE consistently outperform the random baseline across all four settings and multiple runs?
-
What are BoE’s actual runtime and token costs, including claim extraction and verification selection? Does its advantage hold under these costs and extend to other models and non-medical tasks?
We thank the reviewer for the thoughtful review and for recognizing the partial-verification setting, the theoretical analysis, and our detailed reporting of rescues, harms and pool coverage!
[Q1] Robustness to claim errors, verifier reliability, and independent calibration
We agree that the original channel indices were fitted on the evaluation data. We re-ran all four datasets with grouped 5-fold cross-fitting: κ is estimated only from the other four folds, and questions from the same image (VQA-Med) or article (PMC-VQA) always stay in one fold. Our PathVQA ledger retains no image ids, so its folds group by question and same-image leakage cannot be excluded. Comparisons use the factor-only menu and C=16, with Random averaged over 20 seeds and 95% paired-bootstrap intervals.
Cross-fitting changes BoE's accuracy by at most 0.04 pp and the fitted indices by at most 0.04, so the in-sample calibration did not inflate the results. The two anti-correlated PathVQA slots (κ≈0.27–0.33) stay below 0.5 in every fold, a stable property of the channel.
To test claim and verifier errors, we perturbed the cached ledger and replayed BoE without re-fitting κ: removing a fraction of candidate–claim edges (omission), reattaching edges to another claim of the same slot (mis-linking), and flipping VLM verdicts (verifier errors). Each setting averages three perturbation seeds; the tables report the change in BoE accuracy (pp) relative to the clean ledger. Omission is measured on all four datasets:
The mis-linking and verifier-flip runs completed on VQA-Med within the rebuttal window; we will extend them to the other datasets in the revision.
With up to 20% of claims omitted, mis-linked or flipped, BoE's accuracy decreases by at most 0.33 pp, and its rescues and harms relative to raw BoN stay within four questions of the clean ledger (26–30 and 31–38, versus 28 and 34). Across all perturbed settings the gap over Random stays between −0.05 and +0.21 pp, inside the clean-ledger intervals above, so BoE degrades gracefully under claim and verifier errors. Mis-linking is a pessimistic test, since it attaches a candidate to another answer's claim. We will add the cross-fitting protocol and these robustness results to the revised manuscript.
[Q2] Matched action spaces and multiple runs
We agree that the original main comparison gave BoE a richer menu. We now compare BoE-factor, which keeps only factor-level actions, with Random under the same menu, calibration and budget on all four datasets, averaging Random over 20 seeds. BoE-factor outperforms Random on three of four datasets (+0.11 to +0.24 pp under cross-fitting, table above), though all intervals include zero. With the original full menu, BoE's gap over the 20-seed Random is +0.67 pp [+0.14, +1.21] on VQA-Med and +0.45 pp [+0.14, +0.76] on PMC-VQA, and this gain mainly comes from whole-response verification. We will report these matched-menu comparisons and replace the original single-seed Random baseline.
[Q3] Actual cost and generalization
Our instrumented full-set VQA-Med run (235B judge, TP=8, vLLM) logged 55,846 model calls and 38.7M input / 103.6M output tokens for the full evidence ledger. Claim extraction is integrated into generation, with claim slots accounting for about 9% of generated text. Based on measured throughput, the evidence BoE selects at C=16 corresponds to about 876 s and 11M judge tokens, or 9.7% of generation time; graph construction takes 8.7 s in total and verification selection 78 ms per question.
Under these costs, the extra compute is better spent on verification than on sampling. With 16 additional candidates per question, majority voting does not improve (62.50 to 62.27 as K grows from 16 to 32), although doubling generation costs about 10× BoE's evidence. A single whole-response verification step of about 684 s improves selection over raw BoN on all four datasets (+0.47 to +0.95 pp; pooled McNemar p<1e-13).
Our partial-verification formulation and shared-claim mechanism are not specific to medical VQA, while the current experiments cover four medical settings with one generator–judge pair. We will state this empirical scope clearly and discuss evaluation on other tasks and model families as future work.