- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper separates two requirements for inference-time amplification: generating a useful candidate and recognizing it among alternatives. It gives sufficient conditions under which repeated criticism and pairwise comparison reduce selection error, and connects local error bounds to a multi-step search process. Experiments compare practical selectors against oracle coverage on software repair, mathematics, and other tasks. Fixed-pool interventions and negative cases examine how the information available to the selector changes the recovered gain.
Critical strengths
- Holding the proposal pool fixed while changing repository information is a useful intervention.
- The negative controls add substance to the paper.
Critical weaknesses
- The empirical evidence does not closely test the condition that makes repeated weak judgment amplifiable. Assumption 2 and Theorem 2 require a positive comparator edge for each sound–unsound pair, together with conditionally independent repeated judgments.
- The reported oracle-gap recovery uses two different single-proposal references. The headline 78.3% uses the separate 67.0% single-trajectory run.
Questions for authors
- Can the cached judgments provide uncertainty-aware diagnostics of persistent misranking on difficult correct–incorrect pairs, and show whether repeated voting reduces errors on those pairs?
- Can all fixed-pool recovery results be reported against the same random-proposal reference, with the separate single-run comparison labeled separately?
2 comments
Part (2/2)
New multi-domain amplification experiments (Appendix D.10, Table 10). We additionally tested whether repeated pointwise judgments improve selection while holding correct–incorrect candidate pairs fixed. Averaging three pointwise judgments improves expected correct selection across four reasoning domains:
| Benchmark | Single score | Mean of three | Gain |
|---|---|---|---|
| SWE-bench | 53.96% | 60.63% | +6.67 pp |
| GSM-Plus | 78.17% | 84.50% | +6.33 pp |
| MATH L4–5 | 80.17% | 89.50% | +9.33 pp |
| Lean/miniF2F | 61.36% | 64.77% | +3.41 pp |
All four gains have positive 95% task-bootstrap confidence intervals. Because the candidate pairs are held fixed, these improvements arise from additional selection evidence rather than increased proposal coverage. They are conditional pair-selection improvements, not full benchmark solve-rate gains. These pointwise experiments provide complementary evidence of amplification, while the direct comparator experiments examine the limitations relevant to Theorem 2.
Conditional independence remains an explicit assumption. The saved comparison batches contain different outcomes, but their metadata does not establish independence of the underlying model calls. Our dependence diagnostics found no statistically detectable departures under the tested comparisons, but this does not verify conditional independence. Neither the held-out analysis nor the new experiments establish the theorem's independence assumption or quantitatively validate its bound.
At the aggregate SWE-bench level, comparator-only performance increases from 75.48% with one vote per pair to 76.16% with three, with little further improvement (76.19% at five votes and 75.89% at fifteen; Table 6).
The central implication is that positive average comparator accuracy does not guarantee a positive edge on every pair. Repeated selection evidence can improve decisions when local signals are informative, as the multi-domain results demonstrate, but additional judgments need not overcome persistent incorrect preferences. Our heterogeneous-edge extension formalizes this distinction, and the direct comparator and held-out experiments demonstrate its practical relevance.
Reviewer: The headline recovery uses the 67.0% separate single-trajectory result rather than the 68.4% pool mean. Can all fixed-pool recovery results use the same reference?
Response: Yes. We have corrected all fixed-pool oracle-gap recovery calculations to use the mean pass@1 of a uniformly selected proposal from the same pool (Appendix C.3). The separate 67.0% single-run result remains only as a labeled reference.
-
Headline nano pool: The 76.4% committee result now recovers 75.5% of the gain exposed by the 79.0% oracle, rather than the previously reported 78.3% (Section 7.1, Table 1).
-
Table 1: All recovery values now use the 68.4% pool mean; for example, ungated Copeland at 75.8% recovers 69.8%.
-
Fixed-coverage intervention: On the Qwen3.6-27B pool (pool mean 66.4%, oracle 79.2%), recovery rises from 5% with patch-only comparison to 25% with repository context, rather than the previously reported 24% to 40%. The underlying solve rates (67.0% and 69.6%) remain unchanged. Thus, repository information improves recovery without changing the proposals (Figure 3(c), Table 5). Tables 2, 3, and 6 already used pool-mean references and are unchanged.
We thank the reviewer again. The new experiments directly address the relationship between repeated judgments, heterogeneous comparison quality, and selection accuracy. Together with the theoretical extension, held-out analysis, and corrected recovery measurements, they strengthen our account of when weak selection signals can be amplified, why amplification may fail, and how much capability exposed through sampling can reliably be recovered.
Please, let us know if we addressed your concerns and questions or we could anything else to further strenghten the manuscript. Let us know also whether this is enough to raise your score, otherwise, what your remaining dubts are! We're very willing to work with you towards strengthening the paper!
Best,
The authors
Thank you for the careful and thoughtful review. Part (1/2)
We thank the reviewer for the careful and focused review, and for highlighting the fixed-pool intervention and negative controls. Let us know whether below we addres them all conclusively for you! We are very willing to further work with you to clear up any further question! We have added a heterogeneous-edge theoretical bound, conducted a held-out analysis of cached SWE-bench comparisons, and corrected the oracle-gap recovery measurements throughout the paper (revisions in red). Since our initial response, we have also conducted new experiments directly examining repeated judgments on fixed candidate pairs.
Reviewer: Assumption 2 and Theorem 2 require a positive comparator edge for every sound–unsound pair and conditionally independent repeated judgments. Can the cached judgments provide uncertainty-aware diagnostics of persistent misranking on difficult correct–incorrect pairs, and show whether repeated voting reduces errors on those pairs?
Response: We agree that a uniform positive edge and conditional independence are sufficient conditions that need not hold for every pair in practice. We address this concern theoretically and through both held-out diagnostics and new experiments.
Theory (Proposition 5, Appendix A.2.6). We extend the analysis to heterogeneous comparison edges, allowing each sound–unsound pair to have its own edge σab, including negative values. The bound introduces ρs(σ), the probability that the proposal pool contains a pair with edge below . This identifies where the uniform-edge guarantee no longer applies. On a fixed pair with a negative edge, repeated independent majority voting increasingly favors the incorrect candidate.
New direct comparator experiment (Appendix D.10). To further investigate the reviewer's concern, we collected 1,440 judgments across 80 fixed correct–incorrect SWE-bench pairs. Mean comparator accuracy is 61.28% [53.16%, 69.13%], demonstrating an average advantage over chance, but 13 of the 80 pairs remain incorrectly ranked even when all 18 judgments are aggregated.
In the entirely fresh evaluation block, increasing the voting budget from one to nine judgments changes correct selection from 61.88% to 63.75%, a gain of only +1.88 percentage points [−6.25, 10.00]. The interval includes zero, so this experiment does not establish an improvement from additional voting. Together with the persistent misranking, it illustrates why positive average comparator accuracy need not translate into reliable amplification on every pair.
Held-out empirical analysis (new Appendix D.9 and Table 9). We also analyzed 1,000 correct–incorrect candidate pairs across 104 SWE-bench tasks, each with matched records in three saved five-vote batches, without making additional model calls. To avoid evaluating difficult pairs on the votes used to identify them, we classified each pair using only the first batch. A Jeffreys Beta(1/2,1/2) posterior with a 95% probability threshold on the sign of the comparison edge, conditional on a decisive vote, identified 601 positively ranked, 319 uncertain, and 80 negatively ranked pairs. We evaluated these groups exclusively on the two remaining batches.
This analysis establishes two findings:
Misranking persists on held-out judgments. Of the 80 initially negatively ranked pairs (22 tasks), 71 continue to favor the incorrect patch in both held-out batches (88.75%; task-cluster bootstrap 95% CI [75.51%, 96.45%]). Across decisive held-out judgments on these pairs, 95.89% favor the incorrect candidate (CI [91.75%, 98.33%]). The finding is robust to alternative correctness labels (73/82), although a stricter requirement of an incorrect absolute majority across all five slots yields 52/80. These results establish persistent incorrect preferences in the saved data, not the prevalence of permanently negative underlying edges.
Repeated voting improves resolution but does not uniformly correct errors. Increasing votes from r=1 to r=5 raises correct decisions on initially positive pairs from 89.70% to 99.08%. On initially negative pairs, incorrect decisions increase from 72.88% to 93.13% (+20.25 points; 95% CI [13.06, 28.87]). This increase primarily reflects fewer unresolved comparisons (24.00% to 4.38%), rather than a statistically established decline in accuracy conditional on a decision. Thus, more votes can make decisions more decisive without correcting systematic misranking.