acceptodds
Under review as a conference paper at ICLR 2027

Discrimination Is Not Selection: Rethinking Verifier Evaluation for Multimodal Test-Time Scaling

Abstract

A verifier can rank correct responses above incorrect ones on average and still select the wrong answer. This distinction matters for multimodal test-time scaling, where the value of additional sampling depends on recovering the correct answers it produces. We investigate this verification–selection gap and identify a mismatch between how verifiers are evaluated and how their scores are used. Pooled discrimination compares candidates across queries that never compete for selection, while within-query ranking averages over pairs even though the final decision depends on the score maximum. We formalize these distinctions and derive an exact decomposition of selection loss into incorrect candidates that strictly dominate and mixed-quality ties that are resolved incorrectly. Across multimodal reasoning benchmarks, informative verifier scores coexist with substantial failures to recover available correct answers, with different failure patterns across scoring interfaces. Broader candidate sampling and targeted interventions over frozen representations do not consistently eliminate this loss. Our findings motivate treating the recovery of available correct answers as an explicit evaluation target: verifiers for test-time scaling should be assessed through the decisions their scores induce, alongside their ability to discriminate correctness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Scaling, Multimodal Reasoning, Candidate Selection, Verifier Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_69mUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a detailed and complete analysis for test-time answers/responses selection. They analyze what the failure is and how it may occur when a verifier selects from lots of generated responses. Then they propose a better metric or perspective to identify when and how selection failures occur. The extensive results can support what they claim.

Critical strengths
  1. They find an interesting and important problem and analyze the failure mode/pattern. This may help to advance future related research and realistic applications of LLMs/agents.

  2. The empirical results also support their claim in main text. Based on the results and their findings/conclusion, the performance of LLMs can be improved without substantial additional compute, by refining the selection process at test time.

Critical weaknesses
  1. They provides a detailed analysis for selection failure pattern but did not offer a valid solution for this failure, which makes their contribution limited.

  2. Also their experiments are only conducted on four small benchmarks. I know it's test-time. So it won't lead to much time and cost for experiments. They may conduct more extensive experiments to support their claims. Therefore, I am not convinced that the current contribution meets the bar for ICLR.

Questions for authors

N/A

Reviewer_VJQVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the gap between verifier ranking discrimination and downstream selection performance in multimodal test-time scaling. The authors demonstrate that conventional discrimination metrics (pooled and within-query AUROC) do not correlate well with a verifier's ability to pick correct answers from a pool of sampled candidates. To quantify this mismatch, the authors propose dedicated selection-oriented metrics alongside a formal error decomposition that categorizes failures into strictly sub-optimal top predictions versus erroneous tie-breaking among maximal scores. Through extensive evaluations across four benchmarks (MathVista, MathVerse, MMStar, MMMU) and three verifier interfaces, the paper shows that high discrimination performance often masks substantial degradation in actual selection accuracy. Notably, scaling candidate pool sizes does not reliably improve outcomes, and common lightweight interventions (selection-aware losses, context conditioning, pairwise comparison) provide limited remedy. Overall, the work delivers a cautionary analysis of conventional verifier evaluation, urging the community to evaluate multimodal verifiers directly on test-time selection fidelity.

Critical strengths
  • Well-motivated formulation and rigorous theoretical grounding. The paper pinpoints a critical yet often overlooked discrepancy between conventional verifier metrics and realized downstream selection performance in test-time scaling. The authors theoretically elucidate why pooled and within-query AUROC fail to faithfully characterize selection outcomes. Crucially, the exact decomposition of selection loss—differentiating between strictly incorrect maxima and misresolved ties—offers an elegant, interpretable diagnostic framework that provides actionable conceptual value beyond traditional discrimination-oriented evaluation.

  • Methodologically sound evaluation with robust empirical takeaways. The empirical validation across four multimodal reasoning benchmarks and three verifier backbones is commendable for its rigorous experimental control. By utilizing shared, frozen candidate pools, the study cleanly isolates selection errors from generation bottlenecks. The resulting empirical insights are compelling: substantial selection gaps persist even when discrimination scores appear informative. Furthermore, extensive supplementary analyses—spanning scaled candidate budgets, alternative scoring objectives, and pairwise comparison formulations—consistently reinforce the thesis that higher discrimination metrics do not guarantee superior selection efficacy.

Critical weaknesses
  • Questionable fidelity and limited sample size in label verification. The paper's core selection-gap conclusions depend on automated correctness labels that exhibit non-trivial divergence from human judgment (84.5% and 82.5% agreement). Furthermore, the human validation covers merely 200 of the ~51,200 candidates, which is statistically underpowered to reliably validate the dataset. While the authors include synthetic noise sensitivity tests, uniform random label perturbations cannot fully account for structured, systematic misclassifications from rule-based parsers, leaving the robustness of the empirical findings in doubt.

  • Restricted verifier pool and absence of state-of-the-art multimodal baselines. The empirical scope is constrained to two generalist LLMs and a single specialized verifier (VisualPRM). The omission of more capable, state-of-the-art multimodal judges and alternative specialized PRMs limits the external validity of the study. Without testing on stronger verifier architectures, it is difficult to determine whether the reported selection gaps represent inherent verification bottlenecks or are simply consequences of employing moderately capable verifiers.

Questions for authors
  • Reliability and accuracy of correctness labels. Given the relatively low agreement between the rule-based grader and human annotators (84.5% and 82.5%), could the authors improve the accuracy of the automatic correctness labels, for example, by refining the rule-based evaluator or incorporating LLM verification? Since the selection-gap analysis critically depends on label correctness, it would be important to demonstrate whether the main findings remain consistent when evaluated using more reliable labels.

  • Generalizability to stronger and specialized verifiers. The evaluation covers only three verifier interfaces, including one dedicated multimodal verifier. Is there evidence that the observed selection gaps and failure patterns persist when using stronger state-of-the-art multimodal judges or other specialized verification models? To what extent might the reported findings be influenced by the limited capabilities or scoring interfaces of the evaluated models, rather than reflecting a general limitation of verifier-guided selection?

Reviewer_VmVhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines the gap between a multimodal verifier’s ability to discriminate correct responses and its ability to select a correct answer through Best-of-N selection. It defines selection loss relative to sampled oracle availability, introduces a headroom-normalized selection efficiency, and analyzes pooled AUROC, within-query AUROC, and maximum-score outcomes. It also separates selection failures caused by strictly higher-scoring incorrect candidates from those caused by incorrectly resolved mixed-quality ties.

Experiments compare three verifier interfaces on shared candidate pools from four multimodal benchmarks, reporting selection losses of 10.75–29 percentage points. Further experiments consider larger candidate pools and lightweight interventions over frozen representations, but the tested interventions do not establish statistically significant accuracy improvements.

Critical strengths

Using shared, identically ordered candidate pools with question-level resampling and paired comparisons provides a useful framework for separating candidate availability from verifier selection.

Critical weaknesses
  1. The principal measurements depend on reliable correctness labels, but the paper reports human–grader agreement of only 84.5% and 82.5% on a 200-candidate audit, compared with 91.0% inter-annotator agreement. These disagreements do not establish which labels are wrong, but they make independent validation particularly important. Appendix B.5 contains an unresolved “Table ??” reference, while page 17 contains “[Tables unchanged]” and “[Table unchanged]” in place of the audit and sensitivity tables. The benchmark-specific disagreement patterns, directional error rates, and quantitative sensitivity results therefore cannot be inspected. These omissions concern substantive evidence, not merely presentation.

This matters particularly at the pool level. A single false-positive candidate label can create apparent oracle availability, while an incorrectly labeled selected response can create an apparent selection failure. Candidate-level agreement does not establish the reliability of these pool-level measurements. Independent random label perturbations are not a substitute for adjudication, because random noise can itself create oracle–selection gaps and does not capture systematic, correlated grading errors. Complete candidate pools need to be independently adjudicated and the principal metrics recomputed before the reported gap magnitudes and failure decomposition can be considered adequately validated.

  1. The mathematical observations are valid but straightforward. Query-specific score shifts preserve within-query decisions, average pairwise ranking does not determine the top-ranked candidate, and selection failures can be partitioned into strict-maximum and tie cases. Equation 7 is an exact accounting identity once scores and correctness labels are known; it does not explain why the scores fail or predict when a verifier will recover an available correct answer. The paper also acknowledges prior deployment-oriented judge evaluation and adapts its headroom normalization from JETTS. In this context, documenting a nonzero oracle–selection gap does not by itself establish a sufficiently substantive advance.

  2. The intervention experiments could have provided a more informative empirical explanation, but Appendix E does not give the actual loss definitions, scoring-head architectures, training-data construction, train/validation/test separation, or optimization settings. Without these details, it is difficult to assess what the tests establish. The authors appropriately describe the results as inconclusive and limited to lightweight modules over frozen representations, but these under-specified negative results provide little explanatory evidence beyond showing that the tested selectors remain imperfect.

Questions for authors
  1. Can you provide the missing audit and label-sensitivity tables, including benchmark- and stratum-specific confusion counts, directional disagreement rates, and quantitative changes in the reported metrics? Is there an independently adjudicated subset of complete candidate pools on which oracle availability, selection loss, and the strict-maximum/tie decomposition can be recomputed? Such an evaluation would directly affect my assessment of the central empirical results.

  2. What are the exact losses, architectures, representation-extraction procedures, training datasets, optimization settings, and model-selection protocols for SAVR, SCES, and J2S? Are the evaluation questions disjoint from all training and hyperparameter selection? These details are necessary to interpret the negative intervention results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.