acceptodds
Under review as a conference paper at ICLR 2027

COVERAGE IS NOT CONVERSION: A SAME-POOL AUDIT OF LLM SELECTION

Abstract

An LLM may generate a correct answer yet fail to select it. Consider a gate that switches between a baseline and a proposed selector. Even a perfect gate cannot fix cases where both actions miss a correct candidate. We develop a same-pool audit that compares this missed opportunity with the most a gate could add to the always-deployed proposal. It also separates information loss from errors in using that information. One joint confidence set answers two questions: does the proposal improve the baseline, and does missed opportunity exceed the remaining gate budget? The criteria are sharp for that set. In a secondary analysis of nine fixed LiveAoPS and FEVEROUS comparisons, the budget ordering holds even though no positive gain is certified. A prospective MedMCQA study improves answer accuracy by 1.005 percentage points over direct answering on 20,000 new groups, but loses .470 points to unchanged plurality. A further 4,494-group reasoning study does not confirm improvement over ordinary reasoning. The audit thus identifies limits on gate-only improvement; a proposed intervention still needs validation against a strong control.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
large language models, test-time scaling, candidate selection, statistical auditing, uncertainty quantification, confidence bounds, decision theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_faUhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies selection in test-time scaling: after a model produces a fixed pool of K candidate answers, a decision rule must choose which one to submit. Its central observation is that increased candidate coverage need not convert into better final answers — on MATH-500, appending eight candidates raises pool coverage from 51.6% to 57.2% while majority accuracy moves only from 38.4% to 38.6% [§1]. The contribution is a same-pool audit rather than a new selector. For a fixed baseline B and proposal S acting on the same pool, the paper defines opportunity O = Pr(pool solves, B misses), rescue R, and harm L, with missed capture M = O − R [§2.1]; it proves a three-term decomposition of the unconverted opportunity — missed capture + information loss + gate regret — and shows that information loss equals the rescue–harm overlap, specializing Blackwell's ordering to this paired decision [Prop. 1–2, §2.2]. A single joint confidence set then yields two decisions: whether the proposal certifiably improves the baseline, and whether missed capture exceeds the remaining gate budget [§3.1]. Validation is deliberately local: nine fixed baseline–proposal comparisons across two domains certify the capture-budget ordering in 9/9 cases while certifying positive gain in 0/9 [§4.2]; a prospective MedMCQA study finds a gate that improves on direct answering (+1.005 pp) but fails its unchanged-plurality control (−0.470 pp) [§4.3]; and a separate reasoning study is unconfirmed (+0.067 pp, p = .4652) [§4.4].

Critical strengths

1. The framework turns an uninformative null result into an actionable diagnosis, and it addresses a real gap in how multi-sample performance is analyzed.

The paper's core move is to split the unconverted opportunity into missed capture, information loss, and gate regret, and then to observe that the capture-budget ordering M > L can be certified even when the positive-gain test cannot [§3.1]. This matters because it directly addresses a common practical situation — an ensemble or selection pipeline that "does not significantly improve" — and converts it into a statement about where the loss is and whether a better gate could recover it. Applied through a unified baseline–proposal abstraction, the framing transfers across selectors, and the accompanying exclusion rule gives the diagnosis a concrete stopping criterion. For a literature in which most claimed selection gains are marginal, this is a useful and reusable analytical tool; the distinction between missed capture, where no selector can help, and the gate-recoverable budget is one I had not seen drawn this cleanly, and I expect to reuse it.

2. The study is systematic across datasets and settings, and the paper holds itself to a strong control and reports its own failure.

The audits span four datasets (LiveAoPS, FEVEROUS, MedMCQA, and a reasoning benchmark) and a range of selectors, under a single abstraction rather than as disconnected case studies. The prospective MedMCQA study is further designed so that a plausible improvement over direct answering is checked against the strongest trivial control — unchanged plurality — and the paper reports that the control fails [§4.3, Table 4]. Combined with the repeated separation of "resolved limit" from "observed benefit" from "successful intervention," this discipline establishes the standard against which the paper's own claims should be read.

Critical weaknesses

1. The paper studies test-time scaling but never uses the model's own test-time scaling: every candidate-producing model is a small non-reasoning variant.

All of the added compute is external — more samples from a non-reasoning model, chosen by a selector — and thinking is explicitly disabled wherever settings are reported (Qwen3-32B "without thinking"; Qwen3-4B-Instruct-2507 with "thinking disabled"). No frontier model is tested. The study's one reasoning model, DeepSeek-R1-8B, serves only as a verifier on the historical MATH routes, and the separate reasoning study in §4.4 compares two short prompt styles on the same non-reasoning model rather than a reasoning-mode one. The authors already run Qwen3, so thinking variants in the same family and weight class — Qwen3-4B-Thinking and Qwen3-30B-A3B — were available without any change to the setup, and neither is used.

This matters because the certified result (capture-budget dominance in 9/9 comparisons, positive gain in 0/9) describes one model class. Whether the ordering survives once a model can convert compute through its own reasoning — or whether "collective intelligence" emerges only above some capability threshold — is exactly the open question, and the evidence cannot address it.

The framework is genuinely useful, but its empirical support comes from a capability regime that may not be the one the argument is about. The fix is cheap: repeating the core audit with the model's own reasoning enabled would decide it, and I would revise upward if the ordering survives.

Questions for authors
  1. Model and reasoning mode. The reported generation settings disable thinking (Qwen3-32B "without thinking"; Qwen3-4B-Instruct-2507 with "thinking disabled"), and no thinking-mode model is evaluated. Could the authors repeat the core audit with the model's own reasoning enabled — at minimum Qwen3-4B-Thinking and Qwen3-30B-A3B, which belong to the same family and weight class as models already used — and with at least one frontier model? This is the single change that would most affect my assessment, because it determines whether the capture-budget ordering is a property of selection or of a particular capability regime.

  2. Sensitivity to selector adaptations. Could the authors quantify how much the nine certified orderings depend on the adaptation choices — in particular the Best-of-Majority backfill to four indices and the top-4 extension of Self-Certainty? If a more faithful implementation were used, would any row's capture-budget conclusion change?

Reviewer_QD2tOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asked a question that: in test-time scaling, more produced answers will help to improve the overall accuracy, but how much of that improvements, what is the bar of that improvement etc? They decompose the gap between the ceiling and the baseline by 3 terms, the decomposition is based on 3 factors: missed capture, information loss and gate regret. They carried out experiments to examine those quantities and draw conclusions about the true bottlenecks in current test-time scaling systems.

Critical strengths

The paper targets a bottleneck: test-time scaling allocation and the efficacy of selection architectures. I think the decomposition of unconverted opportunity giving practitioners a way to audit why their test-time scaling is plateauing. They showed empirically that a conditionally gated policy can falsely appear successful against a baseline while losing to the strong control of the always-deployed proposal

Critical weaknesses

It is very difficult to read the paper. It is not only due to the lack of explanation, but also the coherence. For example to understand Figure 1, we need to read definitions of all notations in the figure in latter part of the paper. The paper mentions MedMCQA, but there is no introduction about it. What is the definition of α(G)\alpha(G), when α\alpha is a constant between 0 and 1, line 150? It is not straightforward to show definitions of ΓG\Gamma_G in equation (F) and (F2) are equivalent.

Understanding theoretical properties of the quantities defined in the paper is fine, but what is the deduced empirical rule to help improve overall performance in test-time scaling?

Questions for authors

I would have many questions that require the authors rewrite the paper in a more understanding form for me. I try to understand that there are three terms in Proposition 1, and later in the experiment, the authors use the empirical results to analyses them and deduce some conclusions. But I get lost from Section 4.2. I mean here it is not just the methods i.e. FEVEROUS proposals vs DPP, to. the five selectors DAKPO, verifier-Borda and so on. There is no introduction for these kind of stuffs. As I understand some quantities are similar to pass@1 and pass@k. If the paper was explained in familiar terminologies, then it would be easier to follow. My score is partly an intuition guess.

Reviewer_mvxsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares two rules that each select four candidates from the same pool of 64 FEVEROUS answers. It counts when the proposed rule fixes the baseline, breaks a correct selection, or joins the baseline in missing a correct candidate. The paper develops confidence bounds for this distinction and establishes that ordering in nine fixed comparisons.

Critical strengths

The paper provides a useful formal separation of three sources of selection failure: correct candidates missed by both selectors, rescues and harms that cannot be distinguished using the available information, and errors in how the gate uses that information.

Critical weaknesses
  • The paper doesn't give the FEVEROUS output template or parser. This process is vague; if the verdict and evidence from the answers (rollouts) are extracted by a reward model (this is my guess), the experiments are not fair. Why not use the reward model to get the correct answers from the get-go?

  • The experiments lack detail.

  • The paper is hard to follow. Formalization is overused. E.g., what is "public state G" in L-112?

  • The main diagnosis depends on which selectors are compared. On LiveAoPS, the observed loss ordering changes when the full selector family is included, and the family-level ordering remains statistically unresolved. It is unclear how the pairwise diagnosis helps choose a better selector in practice.

Questions for authors
  • Please explain how FEVEROUS works. How is supporting evidence extracted?
  • Provide the experiment details. E.g., which sampling method did you use? What were the sampling parameters? If you used the same setting as for MedMCQA, are 96 tokens enough?
  • How should this diagnosis be used to decide what to improve?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.