acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Amplification of Weak Reasoning Models

Abstract

How much of the capability of a reasoning model exposed by repeated sampling can an imperfect selector recover? We study this problem as \em inference-time amplification, separating \em coverage—whether a useful candidate is generated—from \em identifiability—whether it can be recognized among the alternatives. We show that better coverage alone does not guarantee better selection, while even imperfect local selection signals can be amplified through repetition. We also characterize proposal-side blind spots that additional sampling cannot overcome. On SWE-bench Verified, a single GPT-5.4 nano trajectory solves 67.0% of tasks, while eight proposals contain a correct patch on 79.0%; critic–comparator selection reaches 76.4%, recovering 78.3% of the oracle-exposed gain. The same pattern appears across proposer families and in mathematical reasoning, competitive programming, and formal proof. On MATH L4–5 and GSM-Plus, our selector recovers 85% and 77% of the oracle gap and substantially outperforms majority vote. Holding proposal coverage fixed, providing the selector with additional repository information also improves recovery. Conversely, when the available selection signal is weak, redundant, or misaligned, amplification can diminish or fail. These results identify selection quality as the key factor governing how much capability exposed by repeated sampling can actually be recovered.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent systems, agents, llm, swe bench, pass@k

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3i1QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper separates two requirements for inference-time amplification: generating a useful candidate and recognizing it among alternatives. It gives sufficient conditions under which repeated criticism and pairwise comparison reduce selection error, and connects local error bounds to a multi-step search process. Experiments compare practical selectors against oracle coverage on software repair, mathematics, and other tasks. Fixed-pool interventions and negative cases examine how the information available to the selector changes the recovered gain.

Critical strengths
  1. Holding the proposal pool fixed while changing repository information is a useful intervention.
  2. The negative controls add substance to the paper.
Critical weaknesses
  1. The empirical evidence does not closely test the condition that makes repeated weak judgment amplifiable. Assumption 2 and Theorem 2 require a positive comparator edge for each sound–unsound pair, together with conditionally independent repeated judgments.
  2. The reported oracle-gap recovery uses two different single-proposal references. The headline 78.3% uses the separate 67.0% single-trajectory run.
Questions for authors
  1. Can the cached judgments provide uncertainty-aware diagnostics of persistent misranking on difficult correct–incorrect pairs, and show whether repeated voting reduces errors on those pairs?
  2. Can all fixed-pool recovery results be reported against the same random-proposal reference, with the separate single-run comparison labeled separately?
2 comments
AuthorsOpenReview ↗

Thank you for the careful and thoughtful review. Part (1/2)

We thank the reviewer for the careful and focused review, and for highlighting the fixed-pool intervention and negative controls. Let us know whether below we addres them all conclusively for you! We are very willing to further work with you to clear up any further question! We have added a heterogeneous-edge theoretical bound, conducted a held-out analysis of cached SWE-bench comparisons, and corrected the oracle-gap recovery measurements throughout the paper (revisions in red). Since our initial response, we have also conducted new experiments directly examining repeated judgments on fixed candidate pairs.

Reviewer: Assumption 2 and Theorem 2 require a positive comparator edge for every sound–unsound pair and conditionally independent repeated judgments. Can the cached judgments provide uncertainty-aware diagnostics of persistent misranking on difficult correct–incorrect pairs, and show whether repeated voting reduces errors on those pairs?

Response: We agree that a uniform positive edge and conditional independence are sufficient conditions that need not hold for every pair in practice. We address this concern theoretically and through both held-out diagnostics and new experiments.

Theory (Proposition 5, Appendix A.2.6). We extend the analysis to heterogeneous comparison edges, allowing each sound–unsound pair to have its own edge σab\sigma_{ab}, including negative values. The bound introduces ρs(σ)\rho_s(\sigma), the probability that the proposal pool contains a pair with edge below . This identifies where the uniform-edge guarantee no longer applies. On a fixed pair with a negative edge, repeated independent majority voting increasingly favors the incorrect candidate.

New direct comparator experiment (Appendix D.10). To further investigate the reviewer's concern, we collected 1,440 judgments across 80 fixed correct–incorrect SWE-bench pairs. Mean comparator accuracy is 61.28% [53.16%, 69.13%], demonstrating an average advantage over chance, but 13 of the 80 pairs remain incorrectly ranked even when all 18 judgments are aggregated.

In the entirely fresh evaluation block, increasing the voting budget from one to nine judgments changes correct selection from 61.88% to 63.75%, a gain of only +1.88 percentage points [−6.25, 10.00]. The interval includes zero, so this experiment does not establish an improvement from additional voting. Together with the persistent misranking, it illustrates why positive average comparator accuracy need not translate into reliable amplification on every pair.

Held-out empirical analysis (new Appendix D.9 and Table 9). We also analyzed 1,000 correct–incorrect candidate pairs across 104 SWE-bench tasks, each with matched records in three saved five-vote batches, without making additional model calls. To avoid evaluating difficult pairs on the votes used to identify them, we classified each pair using only the first batch. A Jeffreys Beta(1/2,1/2) posterior with a 95% probability threshold on the sign of the comparison edge, conditional on a decisive vote, identified 601 positively ranked, 319 uncertain, and 80 negatively ranked pairs. We evaluated these groups exclusively on the two remaining batches.

This analysis establishes two findings:

  1. Misranking persists on held-out judgments. Of the 80 initially negatively ranked pairs (22 tasks), 71 continue to favor the incorrect patch in both held-out batches (88.75%; task-cluster bootstrap 95% CI [75.51%, 96.45%]). Across decisive held-out judgments on these pairs, 95.89% favor the incorrect candidate (CI [91.75%, 98.33%]). The finding is robust to alternative correctness labels (73/82), although a stricter requirement of an incorrect absolute majority across all five slots yields 52/80. These results establish persistent incorrect preferences in the saved data, not the prevalence of permanently negative underlying edges.

  2. Repeated voting improves resolution but does not uniformly correct errors. Increasing votes from r=1r=1 to r=5r=5 raises correct decisions on initially positive pairs from 89.70% to 99.08%. On initially negative pairs, incorrect decisions increase from 72.88% to 93.13% (+20.25 points; 95% CI [13.06, 28.87]). This increase primarily reflects fewer unresolved comparisons (24.00% to 4.38%), rather than a statistically established decline in accuracy conditional on a decision. Thus, more votes can make decisions more decisive without correcting systematic misranking.

AuthorsOpenReview ↗

Part (2/2)

New multi-domain amplification experiments (Appendix D.10, Table 10). We additionally tested whether repeated pointwise judgments improve selection while holding correct–incorrect candidate pairs fixed. Averaging three pointwise judgments improves expected correct selection across four reasoning domains:

BenchmarkSingle scoreMean of threeGain
SWE-bench53.96%60.63%+6.67 pp
GSM-Plus78.17%84.50%+6.33 pp
MATH L4–580.17%89.50%+9.33 pp
Lean/miniF2F61.36%64.77%+3.41 pp

All four gains have positive 95% task-bootstrap confidence intervals. Because the candidate pairs are held fixed, these improvements arise from additional selection evidence rather than increased proposal coverage. They are conditional pair-selection improvements, not full benchmark solve-rate gains. These pointwise experiments provide complementary evidence of amplification, while the direct comparator experiments examine the limitations relevant to Theorem 2.

Conditional independence remains an explicit assumption. The saved comparison batches contain different outcomes, but their metadata does not establish independence of the underlying model calls. Our dependence diagnostics found no statistically detectable departures under the tested comparisons, but this does not verify conditional independence. Neither the held-out analysis nor the new experiments establish the theorem's independence assumption or quantitatively validate its bound.

At the aggregate SWE-bench level, comparator-only performance increases from 75.48% with one vote per pair to 76.16% with three, with little further improvement (76.19% at five votes and 75.89% at fifteen; Table 6).

The central implication is that positive average comparator accuracy does not guarantee a positive edge on every pair. Repeated selection evidence can improve decisions when local signals are informative, as the multi-domain results demonstrate, but additional judgments need not overcome persistent incorrect preferences. Our heterogeneous-edge extension formalizes this distinction, and the direct comparator and held-out experiments demonstrate its practical relevance.

Reviewer: The headline recovery uses the 67.0% separate single-trajectory result rather than the 68.4% pool mean. Can all fixed-pool recovery results use the same reference?

Response: Yes. We have corrected all fixed-pool oracle-gap recovery calculations to use the mean pass@1 of a uniformly selected proposal from the same pool (Appendix C.3). The separate 67.0% single-run result remains only as a labeled reference.

  • Headline nano pool: The 76.4% committee result now recovers 75.5% of the gain exposed by the 79.0% oracle, rather than the previously reported 78.3% (Section 7.1, Table 1).

  • Table 1: All recovery values now use the 68.4% pool mean; for example, ungated Copeland at 75.8% recovers 69.8%.

  • Fixed-coverage intervention: On the Qwen3.6-27B pool (pool mean 66.4%, oracle 79.2%), recovery rises from 5% with patch-only comparison to 25% with repository context, rather than the previously reported 24% to 40%. The underlying solve rates (67.0% and 69.6%) remain unchanged. Thus, repository information improves recovery without changing the proposals (Figure 3(c), Table 5). Tables 2, 3, and 6 already used pool-mean references and are unchanged.

We thank the reviewer again. The new experiments directly address the relationship between repeated judgments, heterogeneous comparison quality, and selection accuracy. Together with the theoretical extension, held-out analysis, and corrected recovery measurements, they strengthen our account of when weak selection signals can be amplified, why amplification may fail, and how much capability exposed through sampling can reliably be recovered.

Please, let us know if we addressed your concerns and questions or we could anything else to further strenghten the manuscript. Let us know also whether this is enough to raise your score, otherwise, what your remaining dubts are! We're very willing to work with you towards strengthening the paper!

Best,

The authors

Reviewer_XSD5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper asks when an imperfect selector can recover the capability that repeated sampling exposes in a reasoning model. It separates coverage, whether the candidate pool contains a useful candidate, from identifiability, whether the available evidence distinguishes that candidate from the alternatives. On the theory side, it constructs task families with arbitrarily high coverage but no positive critic or comparator edge, shows that any uniformly positive local edge is amplified by repeated critic and comparator calls and composes along bounded-progress trajectories, and characterizes proposal-side blind spots that cap oracle best-of-k below 100%. Empirically, a committee of binary critics followed by position-swapped pairwise comparisons with Copeland aggregation is evaluated on fixed proposal pools, using oracle-gap recovery as the main measure. The evaluation covers SWE-bench Verified with GPT-5.4 nano, Qwen3.6 and Gemma proposers, as well as MATH, GSM-Plus, LiveCodeBench, miniF2F and several additional tasks with Qwen models. On SWE-bench, the committee raises GPT-5.4 nano from 67.0% to 76.4% against a 79.0% oracle; recovery varies widely across pools and tasks, and the authors also report cases where a direct verifier makes learned selection unnecessary or where comparison hurts.

Critical strengths
  1. A clear decomposition with a matching measurement design. Separating coverage from identifiability, and reporting pool-mean pass@1, oracle best-of-k and the implemented selector on the same cached pools (Table 2), lets the reader tell a proposal failure from a selection failure, which a single accuracy number hides. The paper also reports cases where its own selector is unnecessary (Blocksworld, where a public plan checker matches the committee) or harmful (CommonGen, where comparison lowers success from 80.5% to 73.0%; Table 8), so the claim is presented as conditional rather than universal.

  2. A controlled test of the identifiability claim. Holding the Qwen3.6-27B proposal pool and its 79.2% oracle fixed, giving the comparator repository context raises the measured comparator edge from 0.04 to 0.09 and the resolve rate from 67.0% to 69.6% (Figure 3c, Table 5). This is the one experiment in which only the selector's information changes, so it supports the central claim that identifiability governs recovery more directly than the cross-task comparisons, which change proposers, judges and tasks at once.

Critical weaknesses
  1. The headline SWE-bench result is the best entry of a test-set sweep, and its configuration is not fully specified. Appendix D.1 describes the 76.4% as a "separate full critic–comparator run". Its critic threshold is not stated, and its vote budget is stated inconsistently: Table 1 lists five critics and five comparator calls, whereas Table 7 lists m = 5, r = 3, the setting whose Table 6 entry (76.36%) is the highest of the twelve budget settings. Under either reading, 76.4% is the best entry of a sweep scored with hidden tests on SWE-bench Verified itself (Tables 1 and 6), and the paper describes no separate split on which the threshold, vote budget or aggregation rule were chosen. Finally, the 78.3% recovery uses the separate 67.0% single-run baseline, whereas Appendix C.3 defines p1 for fixed pools as the pool mean, 68.4%; under that definition the recovery is (76.4 − 68.4)/(79.0 − 68.4) = 75.5%.

    The supplementary code adds a further uncertainty. The README's comparator step runs the standalone comparator.py with five position-swapped votes per pair, matching Table 1, and its prompt, like that of direct_binary_dynamic_hints.py, includes the FAIL_TO_PASS test names and the test_patch source, that is, the benchmark's held-out tests. The prompts in orchestra/ match Appendix C.4 and do not. If any reported number came from the standalone scripts, the selector saw the evaluation tests, which do not exist at deployment.

    This matters because the paper places 76.4% beside the 76.4% frontier references (Figures 1 and 4a). The gap to zero-cost rules (at most 71.2%, Table 4 and Figure 2b) does not depend on these choices; the headline recovery figure and that placement do.

  2. The comparison with existing selection methods is incomplete. The committee is compared with zero-cost rules (Table 4), majority vote (Table 3) and critic-only scoring, but not, on the same fixed pools, with the selectors the paper itself cites: trained verifiers or reward models (Cobbe et al., 2021; Lightman et al., 2024), pairwise rankers (Jiang et al., 2023), verifier ensembles (Saad-Falcon et al., 2025a) or Universal Self-Consistency (Chen et al., 2023). The last needs only a prompt and one call over all candidates, whereas the all-pairs tournament queries 28 pairs per task at k = 8, each several times. Nor is it compared with the test-based selection of SWE-bench systems such as Agentless (Xia et al., 2025), which filters candidates with regression tests and generated reproduction tests before majority voting. No existing selector is compared at matched cost, although on MATH and GSM-Plus judging uses about 49× and 60× as many tokens as proposal generation (Table 7). On GSM-Plus, critic-only scoring (62.8%) falls below majority vote (66.9%) because 92.6% of tasks tie at the top critic score (Table 3); breaking those ties by majority answer needs no further calls and is not tested, so the margin of pairwise selection over simple practice is not established.

    The paper frames itself as asking when a selection signal suffices rather than proposing another scoring rule (Section 2), but its empirical claims, such as substantially outperforming majority vote and operating in a cost range similar to frontier models, are claims about a selector. Without an existing selector on the same pools, a recovery of 78.3% does not tell me whether the committee improves on current practice.

Questions for authors
  1. Which critic threshold, vote budget (m, r) and aggregation rule were fixed before hidden-test outcomes were examined, and on what data were they chosen? Which (m, r) and threshold produced the headline 76.4%, given that Table 1 states five critics and five comparator calls while Table 7 lists m = 5, r = 3? What does the committee score under the τ = 0.5 gate stated in Appendix C.2?
  2. Why is the critic stage kept rather than using pairwise comparison alone, given that it adds 3 tasks in Table 1 and does not help at r = 5 in Table 6? Is this difference distinguishable from run-to-run variation? How many replays does each Table 6 entry average, and what is the standard deviation across them?
  3. Which code path produced each reported SWE-bench number: orchestra/ or the standalone comparator.py and direct_binary_dynamic_hints.py? Was --allow-cached-eval-traces enabled in any reported run?
  4. Why does the headline recovery use the 67.0% single-run baseline rather than the pool-mean p1 defined in Appendix C.3?
  5. How does the committee compare with an existing selector, for example a trained verifier or reward model, a single selection call over all candidates, critic scores with ties broken by majority answer, or Agentless-style test-based selection on SWE-bench, on the same fixed pools and at matched judging cost? How are the visible-test filter and the shared-test plurality rule in Figure 2b defined?
5 comments
AuthorsOpenReview ↗

Thank you for the careful and thoughtful feedback. Part 1/3

We thank the reviewer for the unusually careful reading, including of the supplementary code, and for the constructive feedback. We agree that several aspects of the protocol and experimental reporting needed clarification. We have revised the paper accordingly (changes in red), corrected the recovery calculations, and narrowed claims not supported by matched comparisons. We have also conducted an additional experiment directly addressing the reviewer's proposed majority-answer tie-breaking baseline.

Reviewer: Which critic threshold, vote budget (m,r)(m,r), and aggregation rule were fixed before hidden-test outcomes were examined, and on what data were they chosen? Which settings produced the headline 76.4%? What does the committee score under the previously stated τ=0.5\tau=0.5 gate?

Response: We agree that the headline configuration was insufficiently documented and that its selection was not independently validated. We have made the following corrections:

Exact configuration. The 76.4% headline result uses m=5m=5 critic votes per patch, r=3r=3 comparator voting rounds per pair, the permissive critic gate τ≥1\tau \geq 1 (retaining patches with at least one positive vote), and Copeland aggregation. Table 1 incorrectly described the threshold rows as using five comparator calls; we have corrected this to r=3r=3. The separate tournament-rule ablations use five calls per matchup. Table 6 reports means over 15 replay seeds, explaining the small difference between its 76.36% mean and the 382/500 = 76.4% headline selection.

The reviewer's threshold. A yes-rate threshold of 0.5 with five critic votes requires at least three positive votes, corresponding to τ≥3\tau \geq 3 in Table 1. This configuration resolves 76.2% of tasks (Table 1, Appendix C.5).

Configuration selection. No independent validation split was used to select the reported configuration. We now explicitly identify 76.4% as the best observed result on the evaluation set, rather than a held-out estimate. Importantly, the main amplification effect does not depend on selecting this configuration: even comparator-only selection with one vote per pair reaches 75.48%, compared with 68.4% for a randomly selected proposal. All tested all-pairs judging budgets achieve 75.48–76.36% (Table 6), and all critic thresholds achieve 75.8–76.4% (Table 1).

The Abstract and Introduction now report the performance range rather than presenting the best setting alone. We also identify frontier-model comparisons as unmatched contextual references and have removed the claim of comparable deployment costs.

Reviewer: Why retain critics when they add only three tasks in Table 1 and do not help at r=5r=5 in Table 6? Is this distinguishable from run-to-run variation? How many replays were averaged, and what are their standard deviations?

Response: We agree that the critic's incremental benefit on SWE-bench is small. In Table 1, permissive gating improves performance from 75.8% to 76.4%, or three additional tasks out of 500. In the separate budget sweep, critics change accuracy by only +0.20 percentage points at r=3r=3 and −0.06 points at r=5r=5.

Each Table 6 entry is reported as a mean over 15 replay seeds. Unfortunately, the individual seed-level outputs were not retained, so the original replay standard deviations cannot be recovered. We have made this limitation explicit and no longer claim statistical significance for the critic increment.

As an additional check on the recoverable five-vote cache, adding the critic gate produces three gains and zero losses across the 500 tasks (exact paired McNemar test, p=0.25p=0.25). This does not establish a statistically significant improvement, and it is separate from the missing budget-sweep replay statistics.

We therefore attribute the main SWE-bench gain to pairwise comparison, not critic gating. Critics remain an optional component because their effectiveness varies across domains: critic-only selection is strongest on IFEval (91.4%), and on CommonGen critics achieve 80.7%, compared with 72.7% for pairwise selection (Table 8). Our framework is intended to characterize these differences, not prescribe one universally optimal selection rule.

Reviewer: Which code path generated the reported SWE-bench results? Were the standalone test-aware scripts used? Was --allow-cached-eval-traces enabled?

Response: We thank the reviewer for identifying this ambiguity in the supplementary material. We confirm that all reported SWE-bench judgments were obtained under blinded evaluation using the orchestra/ pipeline. The reported critics and comparators used the issue text and candidate patches, with additional repository context only in the explicitly labeled intervention. They did not receive FAIL_TO_PASS or PASS_TO_PASS test identifiers, test_patch source, cached evaluation traces, or hidden-test outcomes. The --allow-cached-eval-traces flag was not enabled in any reported run.

The standalone comparator.py and direct_binary_dynamic_hints.py scripts were exploratory, test-aware variants and did not generate any reported result. The README step invoking the standalone comparator did not correspond to the reported evaluation protocol. We apologize for including these scripts without clearly distinguishing them from the blinded experimental pipeline and have clarified this distinction in the manuscript.

Importantly, offline aggregation of the archived judgments reproduces the reported headline result exactly: 382/500 tasks (76.4%), with comparator-only selection at 379/500 (75.8%). Thus, the reported accuracy is reproducible from the archived votes. The surviving records do not contain the original rendered request payloads, so this offline reproduction does not independently authenticate the historical prompts; our statement about blinding describes the experimental protocol used.

Reviewer: Why was the separate 67.0% single-trajectory result used instead of the 68.4% proposal-pool mean for oracle-gap recovery?

Response: We agree that this was inconsistent with our definition and have corrected it throughout. All fixed-pool recoveries now use the mean success rate of a uniformly selected proposal from the same pool. The headline recovery therefore changes from 78.3% to 75.5%: Rec=(76.4−68.4)/(79.0−68.4)=75.5Rec = (76.4 - 68.4)/(79.0-68.4)=75.5%. The 67.0% single-trajectory result remains only as a labeled reference.

Reviewer: How does the committee compare with trained verifiers, reward models, Universal Self-Consistency, Agentless-style selection, majority tie-breaking, and other methods at matched judging cost? How are the visible-test filter and shared-test plurality controls defined?

Response: We agree that oracle-gap recovery alone does not establish superiority over existing selection methods. Our central contribution is to characterize when sampled capability is recoverable, not to propose a state-of-the-art selector. We have narrowed our comparative claims accordingly and conducted an additional experiment addressing the reviewer's proposed majority-answer tie-breaking baseline.

Comparisons supported by our fixed pools. On MATH L4–5, pairwise selection reaches 71.2%, versus 58.6% for majority vote (+12.6 percentage points) (Table 2). On GSM-Plus, it reaches 75.5%, versus 66.9% for majority vote (+8.6 points). On SWE-bench, the common comparator-only replay reaches 75.7%, compared with at most 70.0% for the reported judge-free heuristics (random selection, first proposal, shortest patch, fewest files, and similarity consensus) (Table 4).

New experiment: Majority-answer tie-breaking. Following the reviewer's suggestion, we tested majority-answer tie-breaking among top-scoring critic candidates on GSM-Plus, where 92.56% of tasks have ties at the highest critic score. This improves critic-only selection from 828/1318 (62.8%) to 935/1318 (70.9%), an improvement of +8.12 percentage points (95% CI [6.37, 9.86]), without additional model calls.

Pairwise selection still achieves 75.5% versus 70.9% for this improved critic-only strategy, retaining an approximately 4.6-point advantage on the same saved proposal pool (Appendix D.4). Thus, the advantage over critic-only selection is not solely an artifact of unresolved top-score ties. This is an observed fixed-pool comparison, not evidence of superiority over specialized verifiers or compute-matched alternatives.

Comparisons not performed. We have not established matched-pool results for trained verifiers, reward models, pairwise rankers, verifier ensembles, Universal Self-Consistency, Agentless-style generated-test selection, or token-matched additional sampling. The new tie-breaking experiment addresses one requested baseline, but does not resolve these broader comparisons. We explicitly acknowledge these limitations and make no superiority claims relative to methods we have not compared.

Test-based controls. The previous SWE-bench figure included visible-test filtering and shared-test plurality, but their precise executable rules and test provenance could not be recovered from the retained artifacts. Rather than continue reporting comparisons that cannot be sufficiently specified, we have removed these controls from the revised figure and text. Neither should be interpreted as a reproduction of Agentless-style generated-test selection.

Resource accounting. On SWE-bench, one comparator vote per pair achieves 75.48% at approximately $0.07 in judging cost per task, around 7% of the reported total (Table 6). On MATH and GSM-Plus, judging uses approximately 49 and 60 times as many tokens as proposal generation, respectively (Table 7). We therefore do not claim matched-compute superiority over additional sampling or other selection strategies.

Compute and identifiability are distinct questions. The new tie-breaking experiment strengthens the fixed-pool evidence that more informative selection can recover capability missed by simpler aggregation. However, whether additional judging is the most compute-efficient way to obtain this improvement remains an open comparison.

Separately, our fixed-pool intervention directly tests the role of selection information: holding the Qwen3.6-27B SWE-bench proposal pool and its 79.2% oracle fixed, providing repository context increases the comparator edge from 0.04 to 0.09 and oracle-gap recovery from 5% to 25%. This demonstrates that stronger selection information can improve recoverability without additional proposals.

The GSM-Plus scaling results identify loss of pointwise score resolution (Table 3), while our negative controls (Table 8) and held-out per-pair diagnostics (Appendix D.9, Table 9) characterize settings in which additional judging is ineffective or harmful. Together, these findings distinguish improvements arising from the information available to the selector from those arising merely from additional proposal coverage.

We thank the reviewer again. The revised analysis, including the new majority-answer tie-breaking experiment, strengthens the evidence that selection quality governs recoverability, while clarifying the limits of our comparisons. Our contribution is to characterize when and why sampled capability can—or cannot—be recovered through additional inference.

Reviewer_HW2zOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a setting of repeated sampling and imperfect verifiers (inference time sampling of weak reasoning models). This paper separates proposal coverage from the ability to identify correct solutions, and provide theoretical results on when weak verification signals can be amplified. They show improvements on swe-bench, math, coding and formal reasoning benchmarks.

Critical strengths

One main strength is that understanding how weaker models can be amplified at inference time is an important problem that this paper formalizes well. The empirical results are interesting across several benchmarks (but somewhat known in the literature by now). I appreciate the attempt to theoretically characterize when imperfect verification can improve performance, and the analysis of proposal coverage and selection failures.

Critical weaknesses

It is unclear to me how novel this work is empirically, given the extensive research on test-time scaling and pairwise verification. For example, V1 (https://arxiv.org/abs/2603.04304) already shows that pairwise verification can outperform pointwise verification across coding and math tasks, and proposes an efficient tournament-based selection strategy. LLM-as-a-Verifier (https://llm-as-a-verifier.notion.site/) studies repeated verification and efficient candidate selection, including on SWE-bench Verified. I do not see these works discussed in the paper. As a result, it is unclear from the empirical results whether this paper brings new learning to the community.

Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Thank you for the careful and thoughtful feedback

We thank the reviewer for the careful reading, positive assessment, and for bringing two relevant papers to our attention. We have added both to Related Work (Section 2, "Selection, verifiers, and judges") and clarified how our contribution differs.

Reviewer: It is unclear how novel this work is empirically, given the extensive research on test-time scaling and pairwise verification. V1 already shows that pairwise verification can outperform pointwise verification and proposes an efficient tournament-based selection strategy. LLM-as-a-Verifier studies repeated verification and efficient candidate selection, including on SWE-bench Verified. These works are not discussed.

Response: We agree that the benefits of pairwise and repeated verification have already been demonstrated. Both cited works advance selection methods, whereas our goal is to understand when imperfect selection can recover the capability exposed by repeated sampling, and what limits that recovery.

  • V1 (Singh et al., 2026) develops pairwise self-verification with an uncertainty-guided Swiss-style tournament (V1-Infer) and joint generator–verifier training (V1-PairRL). It reports improvements over pointwise verification, including 33.3% versus 28.3% on SWE-bench Lite at (N=8N=8).

  • LLM-as-a-Verifier (Kwok et al., 2026) improves selection using continuous logit-based scores, repeated evaluation, and criteria decomposition, reaching 78.2% on SWE-bench Verified.

We do not claim that our Copeland-based selector outperforms these approaches, which use different models, proposal pools, and computational budgets. Our contribution is instead a theoretical framework supported by several distinct empirical findings:

  1. Separating proposal failure from selection failure. We measure pass@1, oracle best-of-(k), and recovered oracle-gap improvement on fixed proposal pools. On our headline SWE-bench experiment, 89% of the remaining failures contain no correct proposal (Section 7.1). Thus, further judging has limited potential at this proposal budget; improving proposal coverage is the more important remaining opportunity.

  2. Characterizing the limitations of discrete pointwise scores as pools grow. On GSM-Plus, the pointwise critic's top-score tie rate increases from 56% to 93% as kk grows from 2 to 8, and its oracle-gap recovery falls from approximately 55% to 40%. Pairwise comparison, in contrast, consistently recovers 77–80% of the exposed gain (Table 3). This identifies loss of score resolution as a concrete limitation of discrete pointwise judgments.

  3. Isolating the effect of identifiability. Across domains, larger measured comparator edges are associated with greater recovery: MATH and GSM-Plus have σ^=0.40\hat{\sigma}=0.40 and 0.380.38, with 77–85% recovery; LiveCodeBench has σ^=0.18\hat{\sigma}=0.18, with 46% recovery; and a SWE-bench pool with σ^=0.04\hat{\sigma}=0.04 recovers only 5% (Table 5). More importantly, we conduct a controlled intervention: holding the SWE-bench proposal pool and its 79.2% oracle fixed, providing repository context increases the comparator edge from 0.04 to 0.09 and recovery from 5% to 25% (Figure 3c). This directly demonstrates that better information can improve recovery without changing proposal coverage.

  4. Identifying when selection fails or becomes unnecessary. Direct verification can make learned judging redundant (Blocksworld), simple heuristics can outperform learned selection (Gemma), and misaligned comparisons can reduce accuracy (CommonGen: 80.5% to 73.0%). We also add a held-out per-pair analysis (Appendix D.9): among 80 SWE-bench pairs initially classified as negatively ranked, the incorrect-decision rate increases from 72.9% to 93.1% as the voting budget grows from r=1r=1 to r=5r=5, primarily because previously unresolved comparisons become decisive. This illustrates why additional judgments cannot be assumed to correct systematic misranking.

Our theory provides a common explanation for these findings. We prove that coverage alone does not imply identifiability (Proposition 1), that a uniformly positive comparison edge can be amplified with pairwise error bounded by e−2rσ2e^{-2r\sigma^2} (Theorem 2), and that local guarantees compose along bounded-progress trajectories (Section 5). We also characterize proposal-side blind spots (Section 6) and extend the analysis to heterogeneous comparator edges (Appendix A.2.6).

The main contribution is therefore not another demonstration that pairwise verification works, but an account of how much sampled capability imperfect selection can recover, why recovery varies, and what fundamentally limits further amplification. Our fixed-pool intervention, recovery measurements, and negative controls provide empirical evidence for this distinction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.