acceptodds
Under review as a conference paper at ICLR 2027

ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation

Abstract

Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods treat all tests equally or use ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a circular dependency. Our key insight is that we need not determine test correctness at all: test votes should rank, not merely count. What matters is not how many codes pass a test, but whether it can distinguish correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC (LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose ACES (AUC ConsistEncy Scoring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@k on multiple code generation benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code generation, large language models, test-time scaling, test case generation, AUC optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NLRKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates post-hoc selection from code candidates produced by large language models, using tests also generated by large language models. A key setting here is that neither the candidate code nor these generated tests are guaranteed to be correct. The core re-framing is that, for ranking purposes, what matters for a test is not whether the test itself is correct, but its discriminative power. A test that is technically incorrect but still constructive carries useful ranking signal, while filters built around checking test correctness throw away these useful tests alongside truly misleading ones. To estimate discriminative power without ground-truth correctness labels, the authors introduce a leave-one-out AUC metric. For each target test, that test is held aside. The code candidates are ranked based on weighted pass counts from all the remaining tests, and the AUC of this ranking is computed using pass results from the held-out test as labels. The analysis assumes a class-conditional Bernoulli model, where rows are independent and test outcomes for a single code candidate are conditionally independent. From this setup, three main theoretical results are derived. First, a lower bound for Pass at k, which depends on a signal-to-noise ratio term; this bound is maximized by ideal weights proportional to the positive part of each test’s discriminative power. Second, a population-level identity linking the leave-one-out AUC quantity and discriminative power. Third, an exact expectation for a pass-rate-adjusted empirical score. Under the stated assumption that average discriminative power is positive, the sign of this expected score matches the sign of the underlying discriminative power. Two practical algorithms are built from this framework. The first, ACES-C, requires no hyperparameters and uses closed-form weights derived directly from the empirical scores, with a theoretical guarantee that its signal-to-noise ratio stays within a constant fraction of the optimal oracle signal-to-noise ratio. The second, ACES-O, uses gradient ascent over a logistic surrogate objective. It uses softmax parameterization and an initial filtering step via top-K majority voting. Both algorithms take only the binary pass-fail matrix as input.

Critical strengths

S1. Separating test correctness from discriminative power is a genuine conceptual contribution, and the authors formalize this idea clearly. It is insightful that a test with a wrong expected output can still carry useful discriminative signal, while a correct but trivial test carries almost none. This shifts the problem of selecting good tests from verification to ranking. Theorem 3 and especially Theorem 6 make this idea empirically verifiable. The adjusted empirical score, built from leave-one-out AUC and a pass-rate correction term, has an expectation directly proportional to discriminative power; this correction is derived theoretically instead of tuned manually. The proof of Theorem 6 in Appendix A.4 is elegant. The key step cancels out paired counts from the AUC calculation against the pass-rate term, leaving a sum over ordered pairs where terms within the same class cancel out due to exchangeability. This is a rigorous and appropriate argument. This point weighs heavily in my evaluation. Much prior work in this area relies on ad hoc rules to discount unreliable tests. Here, that heuristic is replaced by a quantity with a derived, testable expectation. Figure 3 further validates this sign property for informative tests, with recall above 94.8 percent across the three main benchmarks. S2. ACES-C requires no hyperparameters, runs very fast at roughly 9 milliseconds per task, and can be combined with other independent signals. Algorithm 1 has no tunable settings, does not rely on pre-filtering, and only needs the binary pass-fail matrix as input. Its overhead is only around 5 milliseconds compared with majority voting, as shown in Figure 14. It improves Pass at 1 over majority voting across all three main benchmarks and all three additional code-generation models tested. The combined ACES-C plus DS3 variant achieves the strongest Pass at 1 results reported in the paper. A lightweight, plug-and-play alternative to majority voting for execution-based reranking is practically valuable, regardless of the supporting theory. Appendix C.5 includes a carefully designed ablation study showing that more than 70 percent of ACES-C’s performance gain comes from continuous weight values, rather than simple binary filtering. This is a non-trivial and practically useful observation.

Critical weaknesses

The theoretical analysis does not cover the algorithms actually evaluated, and the only provided guarantee becomes nearly meaningless in the setting where ACES is supposed to deliver benefits. The paper lists a theoretical foundation as its first contribution, so this issue carries substantial weight in my assessment. Theorem 2 does not apply to ACES. Both Section 2.2 and Appendix A.2 require the weight vector to be fixed and independent of the observed pass matrix. ACES-C and ACES-O both compute weights directly from this matrix. As a result, the core reasoning used in the paper, which links higher signal-to-noise ratio to better Pass at k performance, cannot be applied to the learned weights of either algorithm. Appendix A.1 acknowledges a separate issue related to candidate pre-filtering but never addresses this weight dependence, which is the more critical limitation. Corollary 8 provides a bound for an estimator that is not implemented in the experiments. It describes population-level weights defined from the expected value of the empirical score. The actual algorithm uses sample estimates of this score. No concentration bound is provided to connect these population quantities to the empirical weights used at test time. Each empirical score is a U-statistic computed over pairs of passing and failing candidates, and a Hoeffding-style bound would appear compatible with the independence assumptions in the paper. This gap could potentially be closed, but in the current manuscript, the advertised oracle approximation bound applies neither to the weights the algorithm uses nor to the resulting Pass at k metric. The bound becomes meaningful only in the easy regime where strong performance is already achievable. With the experimental setting of around 500 tests, the constant from the bound is roughly 0.22 near the boundary of the core theoretical assumption and becomes invalid below that boundary. The constant approaches one only when the average discriminative power is fairly large. However, Section 4.3 reports that performance gains are concentrated in the middle-difficulty region. In the easy regime with high average discriminative power, all methods already reach pass rates above 97 percent. The theoretical guarantee therefore converges toward the optimal oracle only in the regime where simple majority voting already works well, and weakens or vanishes in the middle and hard tasks that the paper identifies as practically important. While the constant does converge to one as the number of tests grows for a fixed positive average discriminative power, this asymptotic behavior does not characterize the harder tasks with low discriminative signal. There are no guarantees for ACES-O, and its objective function is misaligned with the theoretically preferred weighting scheme. The population objective weights each test by a coefficient that is not proportional to discriminative power. One term in this coefficient amplifies tests with very high or very low pass rates, so maximizing this objective optimizes for a different weighting rule than the oracle weighting derived in Theorem 2. There is no result showing that maximizing this objective, or its logistic surrogate, improves the signal-to-noise ratio or Pass at k. Empirically, ACES-O relies on several tuned settings: separate learning rates for each benchmark, a limited stopping step selected from a longer optimization run, and a fixed top-K filter size. No cross-validation or holdout protocol for these hyperparameters is described anywhere in the paper. HumanEval and HumanEval+ share the same problem set, candidate pool, and generated tests, with the same range of discriminative values, differing only in the ground-truth labels used for final evaluation. Using a benchmark-specific learning rate in this setup suggests the hyperparameter was chosen using the held-out evaluation labels.

Questions for authors
  1. Does any guarantee apply to the weights you actually use? Theorem 2 and Appendix A.1 require w ⫫ B, but ACES computes w from B. Can you either (a) prove a concentration bound for q_j around E[q_j] (it is a U-statistic; your independence model should support a Hoeffding-type argument) and propagate it to R(w) and Pass@k, or (b) evaluate a sample-split variant — estimate weights on half the candidate rows, rank the other half — and report how much Pass@1 it costs? Answer (b) alone would tell me whether the reported gains survive honest independence.

  2. Redundancy. Please report (i) the distribution of pairwise column agreement / number of distinct columns in B per task, and (ii) ACES performance after exact-duplicate column deduplication, and (iii) a synthetic check in which a single misleading test (Δ_j < 0) is replicated 2/5/20 times — does its ACES-C weight grow with the number of copies? My reading of the LOO-AUC definition says yes, which would mean the clip in Eq. 8 can admit misleading tests by construction rather than by estimation error.

4 comments
AuthorsOpenReview ↗

[Response 1/4] Guarantees for Fitted Weights and Sample-Split Evaluation

Q1/W1.1: Do the guarantees cover weights fitted on the pass matrix, and how does Pass@1 change when fitting and ranking use disjoint candidate rows?

For Theorem 2, fixing ww in advance makes the qualifier “independent of BB” redundant. Under the model in §2.2, fix any nonnegative ww with w⊤δ>0w^\top\delta > 0. The proof uses this same ww on every draw of BB to lower-bound the probability that the top kk contains a correct program. We have clarified the meaning of “fixed” in Appendix A.1 and removed the redundant qualifier from the restatement in Appendix A.2.

Theorem 2 gives a separate lower bound for each admissible choice of ww. We can therefore ask which choice has the largest bound. This comparison depends on ww only through R(w)=(w⊤δ)2/∣w∣22R(w)=(w^\top\delta)^2/\\|w\\|_2^2, and uses no observed entries of BB. Here δj\delta_j is the unknown difference between the population pass probabilities of correct and incorrect programs on test jj. For any given population vector δ\delta, Cauchy–Schwarz gives the maximizing direction w⋆∝[δ]+w^\star\propto[\delta]_+. Thus w⋆w^\star is itself one of the fixed choices already covered by the theorem.

Since δj\delta_j is unknown, ACES-C computes a proxy qjq_j from the observed BB using pass-rate-corrected leave-one-out AUC. Under Assumption 4, Theorem 6 shows that E[qj]>0\mathbb E[q_j] > 0 exactly when the oracle assigns test jj positive weight. This motivates the rule in Eq. (8), wj=max⁡(qj,0)w_j=\max(q_j,0). A Pass@kk bound for these empirical weights must also account for using the same BB to fit the weights and rank the programs, as we do below.

Fitting ACES-C on the opposite half changed observed Pass@1 by +0.11 to +0.67 percentage points across the three benchmarks. We randomly halved each task’s candidates ten times and evaluated both directions. We ranked the same evaluation half using weights fitted on that half or on the opposite half. ACES-C used exact Eq. (8) counts and the submitted all-zero fallback.

BenchmarkMV on evaluation half (%)ACES-C fitted on same half (%)ACES-C fitted on opposite half (%)Opposite − same, pp [95% CI]
HE80.8581.9582.56+0.61 [0.12, 1.28]
HE+70.0970.6471.31+0.67 [0.03, 1.49]
MBPP69.1270.9071.01+0.11 [−0.27, 0.52]

We averaged splits within each task before computing paired bootstrap intervals. ACES-C fitted on the opposite half exceeded MV on all three benchmarks. The gain over MV on MBPP was 1.89 percentage points (95% CI 0.21–3.63). The intervals for the gain over MV on HE and HE+ included zero.

For disjoint ACES-C, Theorem 2 applies conditional on the fitted weights, provided their mean signal is positive and the evaluation half contains both classes under §2.2.

(a) Concentration of the ACES-C score qjq_j. Consider Eq. (8) on the full candidate pool under §2.2, with fixed tests and fixed labels from both correctness classes. In the pair-sum expression for qjq_j (Appendix A.4), the two orderings of each candidate pair together contribute at most 1/21/2 in absolute value to n2qjn^2q_j. Replacing one of the nn independent candidate rows changes only the n−1n-1 pairs containing that row, so qjq_j changes by at most (n−1)/n2(n-1)/n^2. By McDiarmid’s inequality and a union bound over the mm tests, with probability at least 1−η1-\eta,

max⁡j∣qj−E[qj]∣≤ε,ε=n−1nlog⁡(2m/η)2n,0<η<1.\max_j|q_j-\mathbb E[q_j]|\le\varepsilon, \qquad \varepsilon=\frac{n-1}{n}\sqrt{\frac{\log(2m/\eta)}{2n}}, \quad 0 < \eta < 1.

Effect on R(w)R(w). Taking positive parts gives the same error bound for the empirical weights w=[q]+w=[q]_+ relative to wˉ=[Eq]+\bar w=[\mathbb E q]_+. On this event, w⊤δ≥D:=wˉ⊤δ−mεw^\top\delta\ge D:=\bar w^\top\delta-m\varepsilon, since ∣δj∣≤1|\delta_j|\le1, and ∣w∣2≤∣wˉ∣2+m,ε\\|w\\|_2\le\\|\bar w\\|_2+\sqrt m\\,\varepsilon. Thus, if D>0D > 0,

R(w)≥D2(∣wˉ∣2+m,ε)2.R(w)\ge\frac{D^2}{(\\|\bar w\\|_2+\sqrt m\\,\varepsilon)^2}.

Effect on Pass@kk using the same candidates. On the same event, replacing wˉ\bar w by ww changes any two programs’ score difference by at most mεm\varepsilon. Fix a correct program. A top-kk miss under ww then requires at least kk incorrect programs to score at or above that program’s score minus mεm\varepsilon under wˉ\bar w. For each fixed incorrect program, §2.2 gives a Hoeffding bound on this margin under the fixed weights wˉ\bar w. Applying Markov’s inequality to the count over all n−n^- incorrect programs and adding η\eta for failure of the concentration event gives, for D>0D > 0,

Pass@k(w(B))≥1−η−n−kexp⁡!(−D22∣wˉ∣22).\mathrm{Pass}@k(w(B))\ge 1-\eta-\frac{n^-}{k} \exp\\!\left(-\frac{D^2}{2\\|\bar w\\|_2^2}\right).

At the nominal dimensions of Table 2, the 95% error allowance gives D≤0D\le 0, so this bound cannot certify Pass@kk there. The matched split above measures the change in Pass@1 at that scale.

AuthorsOpenReview ↗

[Response 2/4] Test Redundancy, Deduplication, and Replication of a Misleading Test

Q2: How redundant are the test columns, what changes after deduplication, and can copying one misleading test give it positive ACES-C weight?

(i) Test-column redundancy. For each task with mm tests, pairwise agreement is the fraction of the m(m−1)m(m-1) ordered pairs of distinct test indices whose columns in BB match on every candidate program. Deduplication keeps one test per distinct column.

BenchmarkPairwise agreement (%), median [25th, 75th percentiles]Mean number of tests per task after deduplication
HumanEval and HumanEval+41.1 [19.8, 62.1]22.63
MBPP48.4 [28.4, 66.7]17.78

(ii) Pass@1 after exact deduplication. On the full LeetCodeDataset test split, we kept one test per distinct pass/fail column within each problem. For each generator in Appendix C.2, we reranked its original candidates. The table reports Pass@1 (%) before and after deduplication. Parentheses give the percentage-point difference from MV on the same test pool, computed before rounding.

Test poolMethodQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
Original (Appendix C.2)MV14.0423.2517.11
Original (Appendix C.2)ACES-C17.54 (+3.51)28.07 (+4.82)25.00 (+7.89)
Original (Appendix C.2)ACES-O15.35 (+1.32)21.49 (−1.75)19.74 (+2.63)
Deduplicated (new)MV15.3524.1221.05
Deduplicated (new)ACES-C21.49 (+6.14)27.63 (+3.51)25.44 (+4.39)
Deduplicated (new)ACES-O16.67 (+1.32)25.44 (+1.32)20.61 (−0.44)

After deduplication, ACES-C exceeded MV by 3.51–6.14 percentage points across the three LeetCodeDataset generators. ACES-O exceeded MV on two of the three.

(iii) Copying a misleading test can give it positive ACES-C weight. In the Easy case (Appendix C.1, Table 3), t9t_9 and t10t_{10} have the same pass/fail vector. Every incorrect program passes both tests and every correct program fails both (δ=−1\delta=-1). We varied the total number of copies rr from one to twenty, keeping the other eight tests fixed.

To obtain a control that initially meets the threshold in Assumption 4, we added eight informative tests to Table 3. Each differs from the perfect test t1t_1 on a different candidate program and matches it on all others. When ACES-C holds out one copy, the remaining r−1r-1 copies still vote for the same incorrect programs. The table reports the Eq. (8) weight per copy.

Test poolWeight at r=2r=2Weight at r=5r=5Weight at r=20r=20First rr below the Assumption 4 thresholdFirst rr with positive weight
Table 300.031250.1171915
Table 3 with eight additional informative tests000.11719311

In the control, ACES-C assigned zero weight to the misleading test at both one and two copies, when the Assumption 4 threshold held. Even below this sufficient threshold, ACES-C continued to filter out the misleading test through four copies in Table 3 and through ten in the control.

AuthorsOpenReview ↗

[Response 3/4] Corollary 8 and the ACES-O Population Objective

W1.2: At about 500 tests, is Corollary 8 informative only when majority voting already performs well?

The purpose of Corollary 8 is to prove convergence of the ACES-C weighting rule in Eq. (8) as the number of tests grows. It introduces no new method. The proof in Appendix A.6 uses conservative inequalities, which explain why the guaranteed coefficient is small near the Assumption 4 threshold (about 0.22 at m=500m=500). A tighter proof could give a larger coefficient for the same weights and test budget. Finding the sharpest finite-test coefficient was outside the scope of this analysis. For convergence, it suffices to show that the lower bound on the fraction of oracle SNR retained by the population ACES-C weights tends to one, as it does at fixed positive average test discrimination.

Appendix C.6 directly examines whether adding tests leaves little room to improve on MV. Figure 9 compares the methods on the same candidate programs as the number of generated tests increases. MV’s mean Pass@1 largely plateaus after 50–100 tests. On HumanEval, HumanEval+, and MBPP, ACES-C with 100 tests already achieves higher mean Pass@1 than MV with 500 tests.

W1.3: Given its pass-rate factor, how does the ACES-O objective relate to the oracle weights in Theorem 2, and what does optimizing it guarantee about program ranking?

The factor 1/[pr(1−pr)]1/[p_r(1-p_r)] in Eq. (10) can amplify the contribution of a misleading test as well as an informative one. The condition below accounts for this factor on both sides. It compares the agreement of one informative pair with an upper bound on the agreement of every misleading test under any weighting of its peers.

Assumption (Separation of informative and misleading tests). Let π∈(0,1)\pi\in(0,1) denote the fraction of correct programs among the candidates used to fit the weights. For test rr, let δr=αr−βr\delta_r=\alpha_r-\beta_r be the difference in pass rates between correct and incorrect programs and pr=παr+(1−π)βrp_r=\pi\alpha_r+(1-\pi)\beta_r its overall pass rate. The pool contains at least one misleading test and two distinct informative tests j,kj,k satisfying

Ljk:=δjδkpj(1−pj)>U−:=max⁡ℓ:,δℓ<0∣δℓ∣pℓ(1−pℓ).L_{jk}:=\frac{\delta_j\delta_k}{p_j(1-p_j)} \quad > \quad U_-:=\max_{\ell:\\,\delta_\ell < 0}\frac{|\delta_\ell|}{p_\ell(1-p_\ell)}.

Proposition. Under the model of §2.2 on the fitting pool, if Ljk>U−L_{jk} > U_-, every sequence of weights approaching the supremum of the population objective concentrates on informative tests. Let J⋆=sup⁡w∈ΔmJpop(w)J^\star=\sup_{w\in\Delta^m}J_{\mathrm{pop}}(w). For any sequence of weight vectors w(t)∈Δmw^{(t)}\in\Delta^m,

Jpop(w(t))→J⋆⟹∑r:,δr>0wr(t)→1.J_{\mathrm{pop}}(w^{(t)})\to J^\star \quad\Longrightarrow\quad \sum_{r:\\,\delta_r > 0}w_r^{(t)}\to 1 .

If w⋆w^\star attains J⋆J^\star, it assigns all weight to informative tests and its true program AUC satisfies A(w⋆)>1/2A(w^\star) > 1/2. Hence A(w⋆)>A(wunif)A(w^\star) > A(w_{\mathrm{unif}}) whenever A(wunif)≤1/2A(w_{\mathrm{unif}})\le1/2.

Proof sketch. Write ar(v)=Ar(v)−1/2a_r(v)=\mathcal A_r(v)-1/2. For any test rr and any reference weighting v∈Δmv\in\Delta^m, we have J⋆≥ar(v)J^\star\ge a_r(v). To see this, take uε=(1−ε)er+εvu_\varepsilon=(1-\varepsilon)e_r+\varepsilon v, where ere_r assigns unit weight to rr and 0<ε<10 < \varepsilon < 1. After holding out rr, all scores under vv are scaled by ε\varepsilon, so ar(uε)=ar(v)a_r(u_\varepsilon)=a_r(v). As ε↓0\varepsilon\downarrow0, the weight on rr tends to one, and the weighted average Jpop(uε)J_{\mathrm{pop}}(u_\varepsilon) tends to ar(v)a_r(v).

Apply this bound with r=jr=j and v=ekv=e_k. Ranking by kk alone gives true AUC (1+δk)/2(1+\delta_k)/2, so Theorem 3 gives J⋆≥aj(ek)=cj(ek)δj=π(1−π)2LjkJ^\star\ge a_j(e_k)=c_j(e_k)\delta_j=\tfrac{\pi(1-\pi)}{2}L_{jk}. The pair j,kj,k supplies a lower bound on the supremum over all mm tests. For any w∈Δmw\in\Delta^m, Theorem 3 also gives ar(w)≤π(1−π)2U−a_r(w)\le \tfrac{\pi(1-\pi)}{2}U_- whenever δr≤0\delta_r\le0. Each term in J⋆−Jpop(w)=∑rwr(J⋆−ar(w))J^\star-J_{\mathrm{pop}}(w)=\sum_r w_r(J^\star-a_r(w)) is nonnegative. Summing over the tests with δr≤0\delta_r\le0 and using Ljk>U−L_{jk} > U_- gives

0≤∑r:,δr≤0wr≤2[J⋆−Jpop(w)]π(1−π)(Ljk−U−).0\le \sum_{r:\\,\delta_r\le0}w_r \le \frac{2\bigl[J^\star-J_{\mathrm{pop}}(w)\bigr]}{\pi(1-\pi)(L_{jk}-U_-)} .

Increasing Jpop(w)J_{\mathrm{pop}}(w) lowers this upper bound. As Jpop(w)J_{\mathrm{pop}}(w) approaches J⋆J^\star, the bound tends to zero, so the total weight on misleading and neutral tests must also tend to zero.

Under the separation condition, any attained population maximum assigns weight only to informative tests, giving a true program-ranking AUC above 1/21/2. Maximizing SNR in Theorem 2 further requires the relative weights wr∝[δr]+w_r\propto[\delta_r]_+. The proposition does not imply that optimizing Eq. (10) or its logistic surrogate always improves SNR or ranked Pass@kk. In Table 2, the fitted ACES-O solver outperformed MV at each reported cutoff (k=1,2,5k=1,2,5) on HumanEval, HumanEval+, and MBPP.

AuthorsOpenReview ↗

[Response 4/4] Learning Rates and Hyperparameter Selection

W1.4: How were ACES-O’s learning rate, stopping step and shortlist size K selected and validated? Why did HumanEval and HumanEval+ use different learning rates despite sharing the same pass matrix, and did held-out evaluation labels influence that choice?

We mistakenly used a learning rate of 0.05 for HumanEval+ when rerunning ACES-O for Table 2. Five-fold validation had selected 0.01 for both HumanEval and HumanEval+. At the selected learning rate, with all other settings fixed, HumanEval+ Pass@1, @2 and @5 are 75.00%, 76.83% and 80.49%, respectively. Figure 13 shows the corresponding learning-rate sweep. Appendix B.3 and Table 2 now report the corrected setting and results.

For the standalone ACES-O results, we reserved the benchmarks’ hidden tests for final evaluation. Drawing on MPSC’s use of generated data to set its consistency weight (Huang et al., 2024, Appendix E), we used a separate model, DeepSeek-V3, to generate validation tests for each problem. We selected γ\gamma, the learning rate, the iteration count, and the shortlist size KK by five-fold cross-validation over problems within each benchmark. In each round, we compared settings on four folds using each top-ranked program’s performance on the generated validation tests and then fixed the settings to measure Pass@kk on the held-out fold.

Reviewer_aHi1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ACES ranks candidate programs using only a binary code-test execution matrix. It estimates test weights through leave-one-out AUC consistency, using either a closed-form rule (ACES-C) or iterative optimization (ACES-O). Under a class-conditional independence model, the paper links consistency to test discriminative power. Its population-level oracle-approximation bound for ACES-C additionally assumes sufficiently positive average test quality. Experiments across several benchmarks and generation models demonstrate gains over uniform voting with low reranking overhead.

Critical strengths
  1. The paper gives a principled account of why a test's discriminative value matters beyond its correctness. The population identity and empirical-score expectation formula motivate a simple, label-free weighting method, providing a useful theoretical contribution to execution-based code selection.

  2. The empirical study includes controlled comparisons on shared candidate pools, additional models and LeetCodeDataset, and extensive ablations and sensitivity analyses. ACES-O improves main-table Pass@1 over majority voting by 3.66 to 4.88 percentage points, while ACES-C also shows useful gains on the harder candidate pools.

Critical weaknesses
  1. The practical effect of correlated tests is insufficiently assessed. The theory assumes conditional independence of test outcomes (Section 2.2), whereas generated tests may share inputs, logic, or incorrect expected outputs. Leaving one test out can retain correlated copies of its signal. The average-quality analysis in Appendix C.4 does not address this dependence, limiting the evidence that consistency reliably reflects discriminative power in such cases.

  2. The empirical evidence for transfer to more recent models and harder tasks is limited. The main comparison uses GPT-3.5-Turbo-generated candidates and tests on HumanEval, HumanEval+, and MBPP. Appendix C.2 adds Qwen2.5-Coder, DeepSeek-Coder-V2, and LeetCodeDataset, but the additional model families date to 2024 and the reranking comparisons there include only majority voting and ACES. Evaluation with a recent strong code or reasoning model and competitive reranking baselines on harder, temporally separated tasks would better establish whether the gains generalize beyond the original candidate distribution.

Questions for authors
  1. Do the gains and test-quality estimates remain stable after deduplicating tests or introducing controlled groups of correlated tests with shared incorrect outputs?

  2. Do the gains persist with a recent strong code or reasoning model on harder, temporally separated tasks, when compared with competitive reranking baselines using the same candidate and test pools?

  3. How are ACES-C scores normalized for Eq. (13), what values of alpha and beta are used, and what validation protocol is used to select these coefficients and the benchmark-specific ACES-O settings?

2 comments
AuthorsOpenReview ↗

[Response 1/2] Ranking and Test Quality after Deduplication

Q1/W1: Do ranking gains and test-quality estimates hold up after deduplication or when different tests share an incorrect output?

Removing duplicate tests. ACES-C continued to distinguish informative from misleading tests after deduplication. On LeetCodeDataset it also retained a Pass@1 advantage over majority voting (MV) for all three generators. We kept one test per distinct pass/fail column and reran both methods on the same candidates. The two tables report Pass@1 (%). Parentheses give the percentage-point gain over MV on the same pool, computed before rounding.

GPT-3.5-Turbo benchmarks from Table 2.

Test poolMethodHumanEvalHumanEval+MBPP
Original (Table 2)MV80.4969.5168.62
Original (Table 2)ACES-C82.93 (+2.44)71.34 (+1.83)71.19 (+2.58)
Deduplicated (new)MV75.0066.4669.79
Deduplicated (new)ACES-C75.00 (0.00)67.07 (+0.61)71.43 (+1.64)

After deduplication, ACES-C tied MV on HumanEval and retained smaller gains on HumanEval+ and MBPP. All three paired 95% intervals for these differences included zero.

Full LeetCodeDataset test split, with all three generators from Appendix C.2.

Test poolMethodQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
Original (Appendix C.2)MV14.0423.2517.11
Original (Appendix C.2)ACES-C17.54 (+3.51)28.07 (+4.82)25.00 (+7.89)
Deduplicated (new)MV15.3524.1221.05
Deduplicated (new)ACES-C21.49 (+6.14)27.63 (+3.51)25.44 (+4.39)

For Qwen2.5-Coder-7B, the median reduction in tests per problem was 72.5%. After deduplication, ACES-C led MV by 6.14 percentage points (paired 95% CI 2.63 to 10.09).

Test-quality estimates. We evaluated whether the raw ACES-C quality score qjq_j correctly identified informative tests (δj=αj−βj>0\delta_j=\alpha_j-\beta_j > 0). A positive prediction means qj>0q_j > 0; zero scores are nonpositive predictions. TP/TN denote correctly identified informative/misleading tests, and FP/FN denote the two errors. Both rows evaluate the same distinct, nonconstant pass/fail patterns, counting each once per problem. Only the pool used to compute the scores changes. Entries are percentages pooled over these test representatives. TP+TN and FP+FN give the correct and incorrect classification rates, calculated before rounding.

BenchmarkPool used to compute scoresTPTNFPFNTP+TNFP+FN
HumanEvalOriginal50.624.320.54.674.925.1
HumanEvalDeduplicated46.823.021.88.369.930.1
HumanEval+Original50.425.020.44.275.424.6
HumanEval+Deduplicated46.423.721.78.270.229.8
MBPPOriginal42.829.220.57.572.028.0
MBPPDeduplicated39.530.219.510.869.730.3

After deduplication, ACES-C continues to identify most informative test patterns. The main change is that some informative patterns no longer receive positive scores, reflected in lower TP and higher FN. Precision and false-positive shares change less, indicating that the effect is primarily on informative-test recall.

AuthorsOpenReview ↗

[Response 2/2] Temporal Evaluation, Score Normalization, and Hyperparameters

Q2/W2: Do gains persist with a recent model on harder, temporally separated tasks when competitive rerankers use the same programs and tests?

Codeforces with two stronger model. On contests held after Gemini 2.0 Flash’s August 2024 knowledge cutoff, ACES-C outperformed pass/fail SRank at Pass@1, Pass@2, and Pass@5. On the same problems with DeepSeek-V3.2, ACES-C tied SRank at Pass@1 and led by 1.96 percentage points at both Pass@2 and Pass@5, matching MV at all three cutoffs.

The contests ran from September 2024 to February 2025. Each ranker used the same candidates and generated tests within each column. For Codeforces, a program had to pass all archived official tests for its problem. SRank was designed to select a single solution and was evaluated primarily on Pass@1 (To et al., 2024). We adapted its output-agreement score to use the same pass/fail matrix as ACES. Both tables report Pass@1 / Pass@2 / Pass@5 (%).

RankerHumanEval · GPT-3.5-TurboHumanEval+ · GPT-3.5-TurboCodeforces · Gemini 2.0 FlashCodeforces · DeepSeek-V3.2
MV80.49 / 82.93 / 83.5469.51 / 73.17 / 76.8337.25 / 39.22 / 41.1847.06 / 49.02 / 50.98
SRank (pass/fail)77.44 / 77.44 / 77.4465.85 / 68.29 / 69.5137.25 / 37.25 / 37.2547.06 / 47.06 / 49.02
ACES-C82.93 / 82.93 / 82.9371.34 / 71.95 / 74.3939.22 / 39.22 / 41.1847.06 / 49.02 / 50.98
ACES-O84.15 / 85.98 / 86.5975.00 / 76.83 / 80.4937.25 / 37.25 / 43.1447.06 / 49.02 / 50.98

With Gemini, ACES-O reached 43.14% at Pass@5, compared with 41.18% for MV and 37.25% for SRank. On problems rated 2100 or higher, all four rankers tied at each cutoff for both generators.

LeetCodeDataset after model release. We used the Appendix C.2 rankings for MV and ACES and kept problems whose estimated publication date followed each model family’s release. The dataset estimates these dates from weekly contests and problem IDs. ACES-C improved Pass@1 over MV for every generator. On Qwen2.5-Coder-14B, its lead over SRank grew from 4.62 percentage points at Pass@1 to 6.15 at Pass@2 and 8.46 at Pass@5.

GeneratorQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
MV12.97 / 15.14 / 18.9221.54 / 24.62 / 29.2317.11 / 20.18 / 24.12
SRank (pass/fail)18.92 / 18.92 / 18.9222.31 / 23.08 / 23.0824.56 / 25.44 / 25.44
ACES-C15.14 / 16.76 / 17.8426.92 / 29.23 / 31.5425.00 / 25.88 / 26.75
ACES-O15.68 / 17.30 / 20.0020.77 / 23.08 / 26.9219.74 / 21.05 / 24.56

Across both tables, Qwen2.5-Coder-7B was the only setting where pass/fail SRank led ACES-C at Pass@1. In every other setting, ACES-C matched or exceeded SRank at Pass@1 and exceeded it at both Pass@2 and Pass@5. In a separate comparison using CodeT’s released group-wise evaluation, CodeT reached 48.14% Pass@5 on Gemini and had the highest Pass@5 on all three LeetCodeDataset splits.

Q3: How are ACES-C scores normalized in Eq. (13), which fusion coefficients were used, and how were those coefficients and the ACES-O settings selected?

In Eq. (13), the ACES-C score is the weighted pass rate si=∑jwjBij∈[0,1]s_i=\sum_j w_j B_{ij}\in[0,1], with test weights normalized to sum to one. For ACES-C + DS³, the fusion coefficients (α,β)(\alpha,\beta) were (1,0.1)(1,0.1) on HumanEval/HumanEval+, and (0.5,0.2)(0.5,0.2) on MBPP.

For hyper-parameter selection, we reserved the benchmarks’ hidden tests for final evaluation. Drawing on MPSC’s use of generated data to set its consistency weight (Huang et al., 2024, Appendix E), we used a separate model, DeepSeek-V3, to generate validation tests for each problem. We selected the fusion coefficients (α,β)(\alpha,\beta) and the ACES-O hyperparameters (γ\gamma, the learning rate, the iteration count, and the shortlist size KK) by five-fold cross-validation over problems within each benchmark. In each round, we compared settings on four folds using each top-ranked program’s performance on the generated validation tests. We then fixed the settings and measured Pass@kk on the held-out fold.

Reviewer_Tyy7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates generated-test-based code selection as a weighted voting problem over a binary code-by-test pass matrix. It introduces leave-one-out AUC (LOO-AUC) as an unsupervised test-consistency signal, proves identities connecting population LOO-AUC and a pass-rate-corrected empirical statistic to each test’s latent discriminative power, and proposes a closed-form method, ACES-C, plus an iterative differentiable optimization method, ACES-O. Experiments on HumanEval, HumanEval$^+$, and MBPP report improvements over majority voting and several code reranking methods.

Critical strengths

The central reframing is useful and well articulated. Rather than equating test reliability with semantic correctness, the paper distinguishes whether a test improves ranking of correct versus incorrect candidates. Figure 1 communicates this distinction effectively: a test can be semantically imperfect, αj<1\alpha_j<1, yet still be constructive when δj=αj−βj>0\delta_j=\alpha_j-\beta_j>0. This is a more appropriate criterion for the stated code-selection objective than simply retaining tests believed to be correct.

The paper does not merely show an aggregate benchmark number. Figure 2(a) gives a useful task-level characterization of where the method helps: on MBPP, gains concentrate in the intermediate-δˉ\bar\delta regime, where majority voting is neither already nearly perfect nor completely devoid of signal. Figure 2(b) also supports the mechanism claim more directly than a headline Pass@1 number alone, showing that ACES-O is less sensitive than majority voting to highly negative-δj\delta_j test groups and places more value on positive-δj\delta_j tests.

Critical weaknesses
  1. The paper’s central “breaking the circular dependency” claim is only conditionally true, and ACES-O has no mechanism that resolves the fundamental orientation ambiguity.

    The pass matrix alone is invariant to a global reversal of the latent interpretation of its consensus: a coherent cluster of tests that systematically rewards incorrect programs can have high mutual LOO-AUC, just as a coherent cluster of informative tests can. Theoretical identification of the desirable orientation enters through Assumption 4 and Proposition 5, namely that the average discriminative power is sufficiently positive so that the leave-one-out aggregate already has A(−j)(wunif)>1/2A^{(-j)}(w_{\rm unif})>1/2. This is not a minor technical condition, it is the anchor that distinguishes “consensus about correctness” from “consensus about an incorrect behavior.”

    This issue is sharper for ACES-O. Eq. 9 maximizes weighted consistency among tests, but the paper provides no theorem that this objective identifies positive-δj\delta_j tests, avoids a coherent misleading-test mode, or even improves true AUC when Assumption 4 fails. The wording that ACES-O “drops this assumption” should be qualified: it drops the assumption for running the algorithm, not for any established correctness or recovery guarantee. A synthetic stress test with two internally coherent but oppositely oriented test clusters would be valuable, as would a formal impossibility statement or explicit sufficient conditions under which the desirable fixed point is identifiable.

  2. The conditional-independence model is very strong for LLM-generated tests, and the paper does not quantify how violations affect the conclusions.

    The proofs of Theorem 2, Theorem 3, and Theorem 6 depend on test outcomes being conditionally independent across tests given the candidate correctness label. Generated tests often share templates, boundary-case choices, oracle mistakes, or a common misunderstanding of the specification. Such correlations are particularly consequential here because LOO-AUC measures agreement with the other tests. Correlated redundant tests can collectively receive high consistency scores even when they add little independent ranking signal.

    The paper notes correlation as future work in Section 6, but this is more central than a future extension because the theoretical interpretation of LOO-AUC depends on it. The empirical analysis should measure test redundancy or cluster structure, for example using pairwise test-column correlations or duplicate behavioral partitions, and test ACES under deliberately duplicated, correlated, or collectively misleading test subsets. This would clarify whether ACES is primarily detecting discriminative tests or merely amplifying a majority cluster.

  3. The empirical comparison does not yet fully support broad state-of-the-art language.

    Table 2 is compelling for the specific GPT-3.5-Turbo candidate and generated-test pools used, but it does not include several relevant ranking approaches discussed in Section 5, including SRank and ALGO. More importantly, a directly relevant code-ranking approach that optimizes Pass@k should be compared under the same candidate pools and evaluation protocol. The paper’s claim is not merely that ACES improves majority voting, but that it advances code selection, so a strong Pass@k-oriented learned ranking baseline is important.

    In addition, Table 2 mixes two different meanings of Pass@k: direct inference uses the random-kk-subset estimator, whereas post-hoc methods are evaluated according to the top-kk positions of a deterministic reranking. The latter is appropriate for reranking, but the two values should not be visually or rhetorically treated as directly comparable measures of the same selection protocol. I recommend separating direct-generation reference numbers from the ranked-list comparison and focusing claims of improvement on matched candidate pools and matched ranking metrics.

  4. ACES-O’s reported gains are confounded with majority-vote pre-filtering, and key model-selection details are unclear.

    Appendix B.3 states that ACES-O first pre-filters to the top-KK candidates by majority voting, with K=24K=24 for HumanEval and K=32K=32 for MBPP, whereas ACES-C is run directly on all candidates. This means the standalone ACES-O pipeline is not solely an alternative test-weighting rule on the original pass matrix. It benefits from an additional candidate-selection procedure, and its performance can depend materially on whether the majority-vote shortlist retains a correct candidate.

    The manuscript should state precisely how candidates outside this shortlist are handled in the final ranking, whether all codes are subsequently rescored using optimized weights, and whether Pass@k is computed over all original candidates or only the shortlist. It should also report a matched ablation: ACES-O without pre-filtering, ACES-C with the same pre-filtering, and multiple KK values under the exact protocol used for Table 2. Finally, the manuscript gives hyperparameters for γ\gamma, learning rate, number of steps, and KK, but does not state how these were selected without using the benchmark correctness labels. This needs a validation protocol or a genuinely fixed, pre-specified setting.

  5. The main experimental evidence is limited to one candidate/test-generation source and lacks uncertainty estimates.

    The main-paper setup in Section 4.1 uses candidate solutions and tests generated by GPT-3.5-Turbo from a prior protocol. This is a reasonable initial setup, but it leaves open whether the gains arise from properties of this particular generator, prompt design, test generation process, or candidate quality distribution. The appendix provides broader experiments, but the main claims should be stated more narrowly unless robustness across independently generated candidate and test pools is established in the core evaluation.

    Further, Table 2 reports point estimates over 164 or 427 tasks with no confidence intervals, bootstrap intervals, paired tests, or per-task improvement distribution. Several reported differences are small, especially for some combinations with DS3\mathcal{DS}^3. Since all methods are evaluated on the same tasks, paired bootstrap confidence intervals or a paired randomization test would be inexpensive and substantially strengthen the empirical case.

Questions for authors
  1. Can the authors formally characterize the orientation ambiguity of using only the pass matrix? In particular, under what assumptions, if any weaker than Assumption 4, can ACES-O distinguish a mutually consistent informative-test cluster from a mutually consistent misleading-test cluster?
  2. Please clarify the exact ACES-O evaluation protocol. Is optimization done only on the top-KK majority-vote candidates? Are learned weights then used to score all original candidates, and if so, can candidates outside the shortlist enter the final top-kk? Please include matched ablations controlling for pre-filtering.
  3. How were γ\gamma, learning rate, iteration count, and the dataset-specific KK selected? Was there a held-out validation split of tasks, or were these choices informed by benchmark Pass@1? Reporting a validation protocol and fixed settings would improve confidence in the reported gains.
  4. Can the authors add paired uncertainty estimates for Table 2, along with a per-task win/loss analysis against majority voting and the strongest execution-only baseline? This is particularly important for modest differences and for the combined DS3\mathcal{DS}^3 variants.
  5. Can the authors evaluate robustness to correlated tests, duplicated tests, and a deliberately coherent misleading subset? Such experiments would directly test the conditional-independence assumption underlying Theorems 2, 3, and 6.
  6. Can the authors compare against a Pass@k-focused code-ranking baseline and clarify why SRank and ALGO, discussed in Section 5, are absent from Table 2?
6 comments
AuthorsOpenReview ↗

[Response 1/5] Ranking Direction and the ACES-O Separation Condition

Q1: Can the pass matrix alone identify the correct ranking direction? Can ACES-O distinguish informative from misleading test clusters under a weaker condition than Assumption 4?

Any method that infers program correctness solely from unverified generated tests needs an assumption linking passes to correctness. Common baselines such as majority voting (MV) and CodeT also rely on assumptions, even when they do not state them formally. MV treats a higher pass count as evidence of correctness. CodeT gives additional weight to pass patterns shared by more programs and explicitly assumes that two incorrect programs are unlikely to pass exactly the same tests (CodeT, §2.2). In your W1 example, where the link between test outcomes and program correctness is entirely unrestricted and wild, the reversal corresponds to replacing each program’s hidden correctness label yiy_i with 1−yi1-y_i. The informative and misleading clusters switch roles. With BB unchanged, the program ranking also stays the same, so a previously correct first choice is now incorrect under the reversed labels. The reversal can break the assumptions behind MV and CodeT. If ACES-C’s Assumption 4 held before the reversal, it fails afterward.

ACES gives a sufficient condition for rank consistency (LOO-AUC consistency) to identify informative tests. For each held-out test jj, the condition is A(−j)(wunif)>1/2A^{(-j)}(w_{\mathrm{unif}}) > 1/2, which follows from Assumption 4 under the model in §2.2 (Proposition 5). Imagine repeatedly drawing one correct and one incorrect program at random. Among pairs with different pass rates on the remaining tests, the correct program has the higher rate more than half the time. Given this pairwise advantage, Theorem 3 shows that test jj’s population LOO-AUC exceeds 1/21/2 exactly when correct programs pass it more often than incorrect programs do. This pairwise criterion requires less consensus in two respects. An incorrect program can still receive the most MV votes, and a group of incorrect programs can still earn the highest CodeT score by sharing the same pass pattern. On problems meeting the empirical Assumption 4 threshold, ACES-C gained 2.0–4.1 percentage points in ranked Pass@1 over MV across the three benchmarks. For the full benchmarks, Table 2 reports ranked Pass@1 for ACES-C and CodeT, respectively, of 82.93% and 78.05% on HumanEval, 71.34% and 67.87% on HumanEval+, and 71.19% and 71.90% on MBPP.

When can ACES-O distinguish the two test clusters? The population objective in Eq. (10) can favor informative tests even when mean test discrimination is negative. It optimizes over all mm tests without a prescribed partition into clusters. The condition below compares the agreement of one informative pair with an upper bound on the agreement of every misleading test under any weighting of its peers.

Assumption (Separation of informative and misleading tests). Let π∈(0,1)\pi\in(0,1) denote the fraction of correct programs among the candidates used to fit the weights. For test rr, let δr=αr−βr\delta_r=\alpha_r-\beta_r be the difference in pass rates between correct and incorrect programs and pr=παr+(1−π)βrp_r=\pi\alpha_r+(1-\pi)\beta_r its overall pass rate. The pool contains at least one misleading test and two distinct informative tests j,kj,k satisfying

Ljk:=δjδkpj(1−pj)>U−:=max⁡ℓ:,δℓ<0∣δℓ∣pℓ(1−pℓ).L_{jk}:=\frac{\delta_j\delta_k}{p_j(1-p_j)} \quad > \quad U_-:=\max_{\ell:\\,\delta_\ell < 0}\frac{|\delta_\ell|}{p_\ell(1-p_\ell)}.

Proposition. Under the model of §2.2 on the fitting pool, if Ljk>U−L_{jk} > U_-, every sequence of weights approaching the supremum of the population objective concentrates on informative tests. Let J⋆=sup⁡w∈ΔmJpop(w)J^\star=\sup_{w\in\Delta^m}J_{\mathrm{pop}}(w). For any sequence of weight vectors w(t)∈Δmw^{(t)}\in\Delta^m,

Jpop(w(t))→J⋆⟹∑r:,δr>0wr(t)→1.J_{\mathrm{pop}}(w^{(t)})\to J^\star \quad\Longrightarrow\quad \sum_{r:\\,\delta_r > 0}w_r^{(t)}\to 1 .

If w⋆w^\star attains J⋆J^\star, it assigns all weight to informative tests and its true program AUC satisfies A(w⋆)>1/2A(w^\star) > 1/2. Hence A(w⋆)>A(wunif)A(w^\star) > A(w_{\mathrm{unif}}) whenever A(wunif)≤1/2A(w_{\mathrm{unif}})\le1/2.

AuthorsOpenReview ↗

[Response 2/5] Separation Proof, Candidate Shortlisting, and Hyperparameter Selection

Proof sketch. Write ar(v)=Ar(v)−1/2a_r(v)=\mathcal A_r(v)-1/2. For any test rr and any reference weighting v∈Δmv\in\Delta^m, we have J⋆≥ar(v)J^\star\ge a_r(v). To see this, take uε=(1−ε)er+εvu_\varepsilon=(1-\varepsilon)e_r+\varepsilon v, where ere_r assigns unit weight to rr and 0<ε<10 < \varepsilon < 1. After holding out rr, all scores under vv are scaled by ε\varepsilon, so ar(uε)=ar(v)a_r(u_\varepsilon)=a_r(v). As ε↓0\varepsilon\downarrow0, the weight on rr tends to one, and the weighted average Jpop(uε)J_{\mathrm{pop}}(u_\varepsilon) tends to ar(v)a_r(v).

Apply this bound with r=jr=j and v=ekv=e_k. Ranking by kk alone gives true AUC (1+δk)/2(1+\delta_k)/2, so Theorem 3 gives J⋆≥aj(ek)=cj(ek)δj=π(1−π)2LjkJ^\star\ge a_j(e_k)=c_j(e_k)\delta_j=\tfrac{\pi(1-\pi)}{2}L_{jk}. The pair j,kj,k supplies a lower bound on the supremum over all mm tests. For any w∈Δmw\in\Delta^m, Theorem 3 also gives ar(w)≤π(1−π)2U−a_r(w)\le \tfrac{\pi(1-\pi)}{2}U_- whenever δr≤0\delta_r\le0. Each term in J⋆−Jpop(w)=∑rwr(J⋆−ar(w))J^\star-J_{\mathrm{pop}}(w)=\sum_r w_r(J^\star-a_r(w)) is nonnegative. Summing over the tests with δr≤0\delta_r\le0 and using Ljk>U−L_{jk} > U_- gives

0≤∑r:,δr≤0wr≤2[J⋆−Jpop(w)]π(1−π)(Ljk−U−).0\le \sum_{r:\\,\delta_r\le0}w_r \le \frac{2\bigl[J^\star-J_{\mathrm{pop}}(w)\bigr]}{\pi(1-\pi)(L_{jk}-U_-)} .

Increasing Jpop(w)J_{\mathrm{pop}}(w) lowers this upper bound. As Jpop(w)J_{\mathrm{pop}}(w) approaches J⋆J^\star, the bound tends to zero, so the total weight on misleading and neutral tests must also tend to zero.

Relation to Assumption 4. Our condition relaxes Assumption 4’s requirement on mean discrimination. In the example in the appendix to this response (π=1/2\pi=1/2), two tests pass exactly the correct programs, and the remaining q>4q > 4 tests have (α,β)=(1/4,3/4)(\alpha,\beta)=(1/4,3/4). The mean discrimination is negative, δˉ=(2−q/2)/(q+2)<0\bar\delta=(2-q/2)/(q+2) < 0, so Assumption 4 fails. Yet Ljk=4>2=U−L_{jk}=4 > 2=U_-, and equal weights on the informative pair attain the population maximum. By the proposition, every maximizer assigns weight only to informative tests and thus ranks every correct program above every incorrect one in this example.

Q2: Where does ACES-O fit its weights, which programs can enter the final top k, and what does MV pre-filtering contribute?

ACES-O fits test weights on the MV top-KK shortlist and only shortlisted candidates are ranked and can enter the final top kk.

Entries give Pass@1 / Pass@2 / Pass@5 (%). For the two full-pool ablations, we use Figure 11 for Pass@1 wherever it reports the corresponding cutoff. Pass@2 and Pass@5 come from the supplementary ablation runs.

MethodWeight-fitting poolEligible for final top kkHumanEvalHumanEval+MBPP
ACES-O (main setting)MV top KKMV top KK84.15 / 85.98 / 86.5975.00 / 76.83 / 80.4972.37 / 73.54 / 73.77
MV (pre-filtering only)—MV top KK80.49 / 82.93 / 83.5469.51 / 73.17 / 76.8368.62 / 70.49 / 72.83
ACES-CFull poolFull pool82.93 / 82.93 / 82.9371.34 / 71.95 / 74.3971.19 / 71.43 / 72.37
ACES-O (scored full pool)MV top KKFull pool78.66 / 81.71 / 84.1569.51 / 73.17 / 77.4472.60 / 72.83 / 73.77
ACES-O (no pre-filtering)Full poolFull pool77.44 / 81.10 / 83.5468.29 / 71.34 / 77.4462.30 / 65.11 / 68.62

For k≤Kk\le K, pre-filtering alone leaves MV's top-kk unchanged. Reranking those same candidates with ACES-O improves on MV at all three cutoffs on every benchmark. The main setting also outperforms ACES-O without pre-filtering at every cutoff.

Q3: How were the four ACES-O settings chosen, and did selection use held-out tasks or target-benchmark Pass@1?

Drawing on MPSC’s use of generated data for parameter selection (Huang et al., 2024, Appendix E), we used a separate LLM, DeepSeek-V3, to generate candidate programs and tests for validation. We selected γ\gamma, the learning rate, the iteration count, and the shortlist size KK by five-fold cross-validation within each benchmark, keeping all candidates for a problem in the same fold. In each round, we compared settings on four folds by running each problem’s top-ranked program against its generated validation tests. We then fixed the settings and measured Pass@kk on the held-out fold using its hidden tests.

AuthorsOpenReview ↗

[Response 3/5] Paired Comparisons and Correlated Tests

Q4: What do paired intervals and per-task wins and losses show against MV and the strongest execution-only baseline, including for the DS³ variants?

Both ACES-C and ACES-O selected a correct program more often than majority voting (MV) on all three benchmarks. The paired 95% confidence intervals for the Pass@1 gain lie above zero for both methods on MBPP and for ACES-O on HumanEval+. On MBPP, the gains over MV reached 7.49 percentage points for ACES-C + DS³ and 7.96 for ACES-O + DS³.

We use majority voting (MV), the execution-only baseline with the highest average Pass@1 across the three benchmarks in Table 2. A win means only the listed method’s top-ranked program passed. A loss means only MV’s top-ranked program passed.

BenchmarkMethod compared with MVWins vs MV (%)Losses vs MV (%)Δ Pass@1 (pp)95% CI (pp)
HumanEvalACES-C4.882.44+2.44[−1.83, 6.71]
ACES-O4.881.22+3.66[0.00, 7.32]
ACES-C + DS³7.322.44+4.88[0.00, 9.76]
ACES-O + DS³7.324.27+3.05[−1.83, 8.54]
HumanEval+ACES-C4.883.05+1.83[−2.44, 6.10]
ACES-O6.100.61+5.49[1.83, 9.76]
ACES-C + DS³12.804.88+7.93[1.83, 14.02]
ACES-O + DS³12.206.71+5.49[−1.22, 12.20]
MBPPACES-C3.981.41+2.58[0.47, 4.68]
ACES-O4.921.17+3.75[1.41, 6.09]
ACES-C + DS³10.302.81+7.49[4.22, 11.01]
ACES-O + DS³10.302.34+7.96[4.68, 11.24]

Q5: How do correlated tests, exact duplicates and a deliberately misleading group affect ACES under violations of conditional independence?

Repeated and correlated tests. On the full LeetCodeDataset test set from Appendix C.2, ACES-C outperformed majority voting (MV) for all three generators with the original tests. We then kept one test per distinct pass/fail pattern on the same candidate programs. Each pattern appears once in the resulting pool, regardless of its original frequency. Across generators, the median fraction of tests removed per nonempty pool ranged from 72.5% to 85.9%.

The table reports Pass@1 (%), with percentage-point differences from MV on the same pool in parentheses.

Test poolMethodQwen2.5-Coder-7BQwen2.5-Coder-14BDeepSeek-Coder-V2-16B
Original (Appendix C.2)MV14.0423.2517.11
Original (Appendix C.2)ACES-C17.54 (+3.51)28.07 (+4.82)25.00 (+7.89)
Original (Appendix C.2)ACES-O15.35 (+1.32)21.49 (−1.75)19.74 (+2.63)
Deduplicated (new)MV15.3524.1221.05
Deduplicated (new)ACES-C21.49 (+6.14)27.63 (+3.51)25.44 (+4.39)
Deduplicated (new)ACES-O16.67 (+1.32)25.44 (+1.32)20.61 (−0.44)

After deduplication, ACES-C retained a 3.51–6.14 percentage-point advantage over MV across all three generators. ACES-O exceeded MV on two of the three generators in each condition.

On HumanEval, HumanEval+, and MBPP (Table 2), both ACES variants outperformed MV on every original pool despite correlations among distinct tests on incorrect programs. Averaged equally across these benchmarks, Pass@1 was higher on the original pools than on the deduplicated pools by 3.99 points for ACES-C, 7.75 for ACES-O, and 2.46 for MV.

A deliberately coherent misleading subset. In the Hard case in Appendix C.1 (Table 4, Figure 4), the four misleading tests have distinct pass/fail patterns. All four accept the same incorrect programs c4,…,c7c_4,\ldots,c_7, while each correct program passes at most one of them. Under MV, the incorrect program c4c_4 ties the best correct program c1c_1 for first place at six passes each.

Under uniform weighting, all four misleading tests have leave-one-out AUC below 0.5. ACES-C therefore assigns them zero weight and raises program-ranking AUC from 0.60 (MV) to 0.77. ACES-O places most of its weight on three informative tests and ranks all three correct programs above all five incorrect ones (AUC 1.00).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.