acceptodds
Under review as a conference paper at ICLR 2027

Candidate Selection Changes Language-Model Verifier Comparisons

Abstract

Candidate selection changes measured verifier gaps. On a separate set of 2,250 new questions, switching from Qwen-selected answers to uniform-all weighting widens the Qwen–Granite option-label AUROC gap by 0.097 [97.5% CI 0.069, 0.126], replicating a 0.100 change on an earlier 700-question shared candidate bank. An independent three-family MMLU holdout finds uniform-all AUROC advantages of 0.076–0.108 over model-chosen answers; a TruthfulQA extension tests the effect on a second benchmark. First-pass probability controls distinguish these selection effects from the incremental value of verification. Verifier comparisons require shared candidates and explicit policy weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model evaluation, answer verification, candidate selection, selective prediction, evaluation protocols

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_DNCjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper shows that the candidate-selection policy affects verifier evaluation results. The experiments are careful and the results are reproducible. However, the formatting does not comply with the template, the writing is hard to follow, the experiments are confined to small models and multiple-choice questions, and the contribution is thin. I am inclined to reject.

Critical strengths

I think the question is relatively clear and the main result replicates on 2250 new questions. Besides, authors report honestly the unfavorable results.

Critical weaknesses

The weakness is very obvious. I think this paper is not professional seriously. The formatting and writing don't comply with the iclr requirements. Let alone the thin contribution. Its motivation is not real and the experimental scope is narrow.

Questions for authors

Please revise carefully your paper presentation. At lease you need to use the right template, right?

Reviewer_pHSzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how candidate-answer selection affects the evaluation of language-model verifiers. The authors consider three policies: evaluating the initially selected answer, uniformly weighting all answer options, and uniformly weighting only the nonselected options.

In the main experiment, multiple verifiers evaluate the same answer options for the same questions. The authors then change which answers contribute to AUROC and their respective weights. They show that this changes the measured gap between verifiers and replicate the primary result on new questions. Additional analyses examine agreement with the model’s initial choice, the incremental predictive value of verification beyond first-pass information, and the transfer of acceptance thresholds across candidate distributions.

Critical strengths
  1. The experimental design isolates the effect of candidate weighting. All verifiers score all four options for each question, and their scores are saved. The policies are then compared by recomputing metrics from these fixed scores. Thus, changing the policy does not involve rerunning the models, and different verifiers are evaluated on identical candidates. This supports attributing the observed gap change to candidate weighting.
  2. The central observation is tested on a separate sample. The authors repeat the experiment on new questions, and the Qwen–Granite gap change persists for option-label presentation. Confidence intervals account for the dependence among the four candidates belonging to the same question. The evidence therefore extends beyond a single question sample.
  3. Verification is compared against information already available from the first pass. The authors use the candidate’s first-pass probability, agreement with the verifier’s own initial choice, and first-pass entropy. They then test whether adding the verifier score improves correctness prediction. This is a meaningful baseline for assessing whether an additional verification call provides useful information.
Critical weaknesses
  1. The presentation makes the main findings difficult to follow. The text moves rapidly between individual-verifier performance, between-verifier gaps, and changes in those gaps across policies. Numerous study codes and supplementary analyses make it difficult to track which data are used and what each experiment establishes. A worked example connecting initial choices, verifier scores, and policy-specific metric calculations would substantially improve clarity. Furthermore, the abstract and introduction mention first-pass controls without explicitly stating their important result: no statistically significant incremental AUROC improvement from verification is established in the reported comparisons.

  2. The scientific novelty appears bery limited. Changing the composition of correct and incorrect candidates can change AUROC without changing the verifier. The paper usefully quantifies this effect and demonstrates its impact on between-verifier gaps. However, it is less clear what general understanding of model behavior is established beyond this demonstration. The analysis of self-verification also overlaps with the cited work of Phalod (2026), which compares self-verification against first-pass likelihood-based confidence signals. This does not make the papers equivalent, but it calls for a clearer distinction between the new contribution and previously studied questions.

  3. The strongest result holds under a limited combination of conditions. The gap change replicates for Qwen versus Granite when Qwen supplies the initial choices and the proposed answer is presented as an option label. For answer-text presentation, the adjusted confidence intervals include zero. In the exploratory comparison of alternative candidate sources, only the Qwen-source, option-label contrast has an adjusted interval excluding zero. Replication on new questions therefore supports the result within this configuration, but does not establish its robustness to changes in source or presentation. The authors acknowledge this dependence, but it substantially limits the generality of the central finding.

  4. The analysis provides only a partial explanation. The authors establish why correct candidates are more likely to coincide with the model’s initial choice and quantify the contribution of pairs involving such collisions to AUROC. However, the resulting performance also depends on the scores assigned to those candidates. The analysis does not fully explain why gap changes differ across verifiers and between option-label and answer-text presentations. This limits the explanatory contribution, although the paper acknowledges the limitation.

5.The threshold-transfer results are not sufficiently connected to a concrete practical objective. Thresholds developed under uniform-all weighting accept more answers and incur a higher error rate among accepted answers when applied to source-chosen candidates. This establishes a change in the operating point, but does not by itself determine how consequential or undesirable that change is for a particular application. Reporting coverage under a specified risk constraint, or losses under explicit costs for errors and abstentions, would better connect the observation to practical decisions.

Questions for authors
  1. Appendix K already evaluates Phi as a candidate source outside the Qwen-Granite verifier pair, but the adjusted interval for the gap change includes zero. Could the authors extend the new-question replication to this source to better establish whether the central result generalizes beyond Qwen-selected candidates?

  2. What explains the difference between option-label and answer-text results? Does the effect persist when option labels and ordering are permuted while answer content is preserved?

  3. What are the practical consequences of threshold transfer under a specified acceptable risk: what coverage is achieved, and how does it compare with thresholds developed directly on the target distribution?

Comments on the paper

1 comment
Area_Chair_8Cu7OpenReview ↗

Discussion Period - Awaiting Author Response

Dear Authors,

We have not yet received a response to the reviews. Unfortunately one of the reviewers has experienced a health crisis and could not complete their review, but if you respond to the two authors who did submit their reviews on time, we will only ask you to address their concerns,

Please let us know if you intend to continue with the discussion process or would prefer to withdraw the paper from consideration.

Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.