- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates post-hoc selection from code candidates produced by large language models, using tests also generated by large language models. A key setting here is that neither the candidate code nor these generated tests are guaranteed to be correct. The core re-framing is that, for ranking purposes, what matters for a test is not whether the test itself is correct, but its discriminative power. A test that is technically incorrect but still constructive carries useful ranking signal, while filters built around checking test correctness throw away these useful tests alongside truly misleading ones. To estimate discriminative power without ground-truth correctness labels, the authors introduce a leave-one-out AUC metric. For each target test, that test is held aside. The code candidates are ranked based on weighted pass counts from all the remaining tests, and the AUC of this ranking is computed using pass results from the held-out test as labels. The analysis assumes a class-conditional Bernoulli model, where rows are independent and test outcomes for a single code candidate are conditionally independent. From this setup, three main theoretical results are derived. First, a lower bound for Pass at k, which depends on a signal-to-noise ratio term; this bound is maximized by ideal weights proportional to the positive part of each test’s discriminative power. Second, a population-level identity linking the leave-one-out AUC quantity and discriminative power. Third, an exact expectation for a pass-rate-adjusted empirical score. Under the stated assumption that average discriminative power is positive, the sign of this expected score matches the sign of the underlying discriminative power. Two practical algorithms are built from this framework. The first, ACES-C, requires no hyperparameters and uses closed-form weights derived directly from the empirical scores, with a theoretical guarantee that its signal-to-noise ratio stays within a constant fraction of the optimal oracle signal-to-noise ratio. The second, ACES-O, uses gradient ascent over a logistic surrogate objective. It uses softmax parameterization and an initial filtering step via top-K majority voting. Both algorithms take only the binary pass-fail matrix as input.
Critical strengths
S1. Separating test correctness from discriminative power is a genuine conceptual contribution, and the authors formalize this idea clearly. It is insightful that a test with a wrong expected output can still carry useful discriminative signal, while a correct but trivial test carries almost none. This shifts the problem of selecting good tests from verification to ranking. Theorem 3 and especially Theorem 6 make this idea empirically verifiable. The adjusted empirical score, built from leave-one-out AUC and a pass-rate correction term, has an expectation directly proportional to discriminative power; this correction is derived theoretically instead of tuned manually. The proof of Theorem 6 in Appendix A.4 is elegant. The key step cancels out paired counts from the AUC calculation against the pass-rate term, leaving a sum over ordered pairs where terms within the same class cancel out due to exchangeability. This is a rigorous and appropriate argument. This point weighs heavily in my evaluation. Much prior work in this area relies on ad hoc rules to discount unreliable tests. Here, that heuristic is replaced by a quantity with a derived, testable expectation. Figure 3 further validates this sign property for informative tests, with recall above 94.8 percent across the three main benchmarks. S2. ACES-C requires no hyperparameters, runs very fast at roughly 9 milliseconds per task, and can be combined with other independent signals. Algorithm 1 has no tunable settings, does not rely on pre-filtering, and only needs the binary pass-fail matrix as input. Its overhead is only around 5 milliseconds compared with majority voting, as shown in Figure 14. It improves Pass at 1 over majority voting across all three main benchmarks and all three additional code-generation models tested. The combined ACES-C plus DS3 variant achieves the strongest Pass at 1 results reported in the paper. A lightweight, plug-and-play alternative to majority voting for execution-based reranking is practically valuable, regardless of the supporting theory. Appendix C.5 includes a carefully designed ablation study showing that more than 70 percent of ACES-C’s performance gain comes from continuous weight values, rather than simple binary filtering. This is a non-trivial and practically useful observation.
Critical weaknesses
The theoretical analysis does not cover the algorithms actually evaluated, and the only provided guarantee becomes nearly meaningless in the setting where ACES is supposed to deliver benefits. The paper lists a theoretical foundation as its first contribution, so this issue carries substantial weight in my assessment. Theorem 2 does not apply to ACES. Both Section 2.2 and Appendix A.2 require the weight vector to be fixed and independent of the observed pass matrix. ACES-C and ACES-O both compute weights directly from this matrix. As a result, the core reasoning used in the paper, which links higher signal-to-noise ratio to better Pass at k performance, cannot be applied to the learned weights of either algorithm. Appendix A.1 acknowledges a separate issue related to candidate pre-filtering but never addresses this weight dependence, which is the more critical limitation. Corollary 8 provides a bound for an estimator that is not implemented in the experiments. It describes population-level weights defined from the expected value of the empirical score. The actual algorithm uses sample estimates of this score. No concentration bound is provided to connect these population quantities to the empirical weights used at test time. Each empirical score is a U-statistic computed over pairs of passing and failing candidates, and a Hoeffding-style bound would appear compatible with the independence assumptions in the paper. This gap could potentially be closed, but in the current manuscript, the advertised oracle approximation bound applies neither to the weights the algorithm uses nor to the resulting Pass at k metric. The bound becomes meaningful only in the easy regime where strong performance is already achievable. With the experimental setting of around 500 tests, the constant from the bound is roughly 0.22 near the boundary of the core theoretical assumption and becomes invalid below that boundary. The constant approaches one only when the average discriminative power is fairly large. However, Section 4.3 reports that performance gains are concentrated in the middle-difficulty region. In the easy regime with high average discriminative power, all methods already reach pass rates above 97 percent. The theoretical guarantee therefore converges toward the optimal oracle only in the regime where simple majority voting already works well, and weakens or vanishes in the middle and hard tasks that the paper identifies as practically important. While the constant does converge to one as the number of tests grows for a fixed positive average discriminative power, this asymptotic behavior does not characterize the harder tasks with low discriminative signal. There are no guarantees for ACES-O, and its objective function is misaligned with the theoretically preferred weighting scheme. The population objective weights each test by a coefficient that is not proportional to discriminative power. One term in this coefficient amplifies tests with very high or very low pass rates, so maximizing this objective optimizes for a different weighting rule than the oracle weighting derived in Theorem 2. There is no result showing that maximizing this objective, or its logistic surrogate, improves the signal-to-noise ratio or Pass at k. Empirically, ACES-O relies on several tuned settings: separate learning rates for each benchmark, a limited stopping step selected from a longer optimization run, and a fixed top-K filter size. No cross-validation or holdout protocol for these hyperparameters is described anywhere in the paper. HumanEval and HumanEval+ share the same problem set, candidate pool, and generated tests, with the same range of discriminative values, differing only in the ground-truth labels used for final evaluation. Using a benchmark-specific learning rate in this setup suggests the hyperparameter was chosen using the held-out evaluation labels.
Questions for authors
-
Does any guarantee apply to the weights you actually use? Theorem 2 and Appendix A.1 require w ⫫ B, but ACES computes w from B. Can you either (a) prove a concentration bound for q_j around E[q_j] (it is a U-statistic; your independence model should support a Hoeffding-type argument) and propagate it to R(w) and Pass@k, or (b) evaluate a sample-split variant — estimate weights on half the candidate rows, rank the other half — and report how much Pass@1 it costs? Answer (b) alone would tell me whether the reported gains survive honest independence.
-
Redundancy. Please report (i) the distribution of pairwise column agreement / number of distinct columns in B per task, and (ii) ACES performance after exact-duplicate column deduplication, and (iii) a synthetic check in which a single misleading test (Δ_j < 0) is replicated 2/5/20 times — does its ACES-C weight grow with the number of copies? My reading of the LOO-AUC definition says yes, which would mean the clip in Eq. 8 can admit misleading tests by construction rather than by estimation error.
4 comments
[Response 2/4] Test Redundancy, Deduplication, and Replication of a Misleading Test
Q2: How redundant are the test columns, what changes after deduplication, and can copying one misleading test give it positive ACES-C weight?
(i) Test-column redundancy. For each task with tests, pairwise agreement is the fraction of the ordered pairs of distinct test indices whose columns in match on every candidate program. Deduplication keeps one test per distinct column.
| Benchmark | Pairwise agreement (%), median [25th, 75th percentiles] | Mean number of tests per task after deduplication |
|---|---|---|
| HumanEval and HumanEval+ | 41.1 [19.8, 62.1] | 22.63 |
| MBPP | 48.4 [28.4, 66.7] | 17.78 |
(ii) Pass@1 after exact deduplication. On the full LeetCodeDataset test split, we kept one test per distinct pass/fail column within each problem. For each generator in Appendix C.2, we reranked its original candidates. The table reports Pass@1 (%) before and after deduplication. Parentheses give the percentage-point difference from MV on the same test pool, computed before rounding.
| Test pool | Method | Qwen2.5-Coder-7B | Qwen2.5-Coder-14B | DeepSeek-Coder-V2-16B |
|---|---|---|---|---|
| Original (Appendix C.2) | MV | 14.04 | 23.25 | 17.11 |
| Original (Appendix C.2) | ACES-C | 17.54 (+3.51) | 28.07 (+4.82) | 25.00 (+7.89) |
| Original (Appendix C.2) | ACES-O | 15.35 (+1.32) | 21.49 (−1.75) | 19.74 (+2.63) |
| Deduplicated (new) | MV | 15.35 | 24.12 | 21.05 |
| Deduplicated (new) | ACES-C | 21.49 (+6.14) | 27.63 (+3.51) | 25.44 (+4.39) |
| Deduplicated (new) | ACES-O | 16.67 (+1.32) | 25.44 (+1.32) | 20.61 (−0.44) |
After deduplication, ACES-C exceeded MV by 3.51–6.14 percentage points across the three LeetCodeDataset generators. ACES-O exceeded MV on two of the three.
(iii) Copying a misleading test can give it positive ACES-C weight. In the Easy case (Appendix C.1, Table 3), and have the same pass/fail vector. Every incorrect program passes both tests and every correct program fails both (). We varied the total number of copies from one to twenty, keeping the other eight tests fixed.
To obtain a control that initially meets the threshold in Assumption 4, we added eight informative tests to Table 3. Each differs from the perfect test on a different candidate program and matches it on all others. When ACES-C holds out one copy, the remaining copies still vote for the same incorrect programs. The table reports the Eq. (8) weight per copy.
| Test pool | Weight at | Weight at | Weight at | First below the Assumption 4 threshold | First with positive weight |
|---|---|---|---|---|---|
| Table 3 | 0 | 0.03125 | 0.11719 | 1 | 5 |
| Table 3 with eight additional informative tests | 0 | 0 | 0.11719 | 3 | 11 |
In the control, ACES-C assigned zero weight to the misleading test at both one and two copies, when the Assumption 4 threshold held. Even below this sufficient threshold, ACES-C continued to filter out the misleading test through four copies in Table 3 and through ten in the control.
[Response 3/4] Corollary 8 and the ACES-O Population Objective
W1.2: At about 500 tests, is Corollary 8 informative only when majority voting already performs well?
The purpose of Corollary 8 is to prove convergence of the ACES-C weighting rule in Eq. (8) as the number of tests grows. It introduces no new method. The proof in Appendix A.6 uses conservative inequalities, which explain why the guaranteed coefficient is small near the Assumption 4 threshold (about 0.22 at ). A tighter proof could give a larger coefficient for the same weights and test budget. Finding the sharpest finite-test coefficient was outside the scope of this analysis. For convergence, it suffices to show that the lower bound on the fraction of oracle SNR retained by the population ACES-C weights tends to one, as it does at fixed positive average test discrimination.
Appendix C.6 directly examines whether adding tests leaves little room to improve on MV. Figure 9 compares the methods on the same candidate programs as the number of generated tests increases. MV’s mean Pass@1 largely plateaus after 50–100 tests. On HumanEval, HumanEval+, and MBPP, ACES-C with 100 tests already achieves higher mean Pass@1 than MV with 500 tests.
W1.3: Given its pass-rate factor, how does the ACES-O objective relate to the oracle weights in Theorem 2, and what does optimizing it guarantee about program ranking?
The factor in Eq. (10) can amplify the contribution of a misleading test as well as an informative one. The condition below accounts for this factor on both sides. It compares the agreement of one informative pair with an upper bound on the agreement of every misleading test under any weighting of its peers.
Assumption (Separation of informative and misleading tests). Let denote the fraction of correct programs among the candidates used to fit the weights. For test , let be the difference in pass rates between correct and incorrect programs and its overall pass rate. The pool contains at least one misleading test and two distinct informative tests satisfying
Proposition. Under the model of §2.2 on the fitting pool, if , every sequence of weights approaching the supremum of the population objective concentrates on informative tests. Let . For any sequence of weight vectors ,
If attains , it assigns all weight to informative tests and its true program AUC satisfies . Hence whenever .
Proof sketch. Write . For any test and any reference weighting , we have . To see this, take , where assigns unit weight to and . After holding out , all scores under are scaled by , so . As , the weight on tends to one, and the weighted average tends to .
Apply this bound with and . Ranking by alone gives true AUC , so Theorem 3 gives . The pair supplies a lower bound on the supremum over all tests. For any , Theorem 3 also gives whenever . Each term in is nonnegative. Summing over the tests with and using gives
Increasing lowers this upper bound. As approaches , the bound tends to zero, so the total weight on misleading and neutral tests must also tend to zero.
Under the separation condition, any attained population maximum assigns weight only to informative tests, giving a true program-ranking AUC above . Maximizing SNR in Theorem 2 further requires the relative weights . The proposition does not imply that optimizing Eq. (10) or its logistic surrogate always improves SNR or ranked Pass@. In Table 2, the fitted ACES-O solver outperformed MV at each reported cutoff () on HumanEval, HumanEval+, and MBPP.
[Response 4/4] Learning Rates and Hyperparameter Selection
W1.4: How were ACES-O’s learning rate, stopping step and shortlist size K selected and validated? Why did HumanEval and HumanEval+ use different learning rates despite sharing the same pass matrix, and did held-out evaluation labels influence that choice?
We mistakenly used a learning rate of 0.05 for HumanEval+ when rerunning ACES-O for Table 2. Five-fold validation had selected 0.01 for both HumanEval and HumanEval+. At the selected learning rate, with all other settings fixed, HumanEval+ Pass@1, @2 and @5 are 75.00%, 76.83% and 80.49%, respectively. Figure 13 shows the corresponding learning-rate sweep. Appendix B.3 and Table 2 now report the corrected setting and results.
For the standalone ACES-O results, we reserved the benchmarks’ hidden tests for final evaluation. Drawing on MPSC’s use of generated data to set its consistency weight (Huang et al., 2024, Appendix E), we used a separate model, DeepSeek-V3, to generate validation tests for each problem. We selected , the learning rate, the iteration count, and the shortlist size by five-fold cross-validation over problems within each benchmark. In each round, we compared settings on four folds using each top-ranked program’s performance on the generated validation tests and then fixed the settings to measure Pass@ on the held-out fold.
[Response 1/4] Guarantees for Fitted Weights and Sample-Split Evaluation
Q1/W1.1: Do the guarantees cover weights fitted on the pass matrix, and how does Pass@1 change when fitting and ranking use disjoint candidate rows?
For Theorem 2, fixing w in advance makes the qualifier “independent of B” redundant. Under the model in §2.2, fix any nonnegative w with w⊤δ>0. The proof uses this same w on every draw of B to lower-bound the probability that the top k contains a correct program. We have clarified the meaning of “fixed” in Appendix A.1 and removed the redundant qualifier from the restatement in Appendix A.2.
Theorem 2 gives a separate lower bound for each admissible choice of w. We can therefore ask which choice has the largest bound. This comparison depends on w only through R(w)=(w⊤δ)2/∣w∣22, and uses no observed entries of B. Here δj is the unknown difference between the population pass probabilities of correct and incorrect programs on test j. For any given population vector δ, Cauchy–Schwarz gives the maximizing direction w⋆∝[δ]+. Thus w⋆ is itself one of the fixed choices already covered by the theorem.
Since δj is unknown, ACES-C computes a proxy qj from the observed B using pass-rate-corrected leave-one-out AUC. Under Assumption 4, Theorem 6 shows that E[qj]>0 exactly when the oracle assigns test j positive weight. This motivates the rule in Eq. (8), wj=max(qj,0). A Pass@k bound for these empirical weights must also account for using the same B to fit the weights and rank the programs, as we do below.
Fitting ACES-C on the opposite half changed observed Pass@1 by +0.11 to +0.67 percentage points across the three benchmarks. We randomly halved each task’s candidates ten times and evaluated both directions. We ranked the same evaluation half using weights fitted on that half or on the opposite half. ACES-C used exact Eq. (8) counts and the submitted all-zero fallback.
We averaged splits within each task before computing paired bootstrap intervals. ACES-C fitted on the opposite half exceeded MV on all three benchmarks. The gain over MV on MBPP was 1.89 percentage points (95% CI 0.21–3.63). The intervals for the gain over MV on HE and HE+ included zero.
For disjoint ACES-C, Theorem 2 applies conditional on the fitted weights, provided their mean signal is positive and the evaluation half contains both classes under §2.2.
(a) Concentration of the ACES-C score qj. Consider Eq. (8) on the full candidate pool under §2.2, with fixed tests and fixed labels from both correctness classes. In the pair-sum expression for qj (Appendix A.4), the two orderings of each candidate pair together contribute at most 1/2 in absolute value to n2qj. Replacing one of the n independent candidate rows changes only the n−1 pairs containing that row, so qj changes by at most (n−1)/n2. By McDiarmid’s inequality and a union bound over the m tests, with probability at least 1−η,
maxj∣qj−E[qj]∣≤ε,ε=nn−12nlog(2m/η),0<η<1.
Effect on R(w). Taking positive parts gives the same error bound for the empirical weights w=[q]+ relative to wˉ=[Eq]+. On this event, w⊤δ≥D:=wˉ⊤δ−mε, since ∣δj∣≤1, and ∣w∣2≤∣wˉ∣2+m,ε. Thus, if D>0,
R(w)≥(∣wˉ∣2+m,ε)2D2.
Effect on Pass@k using the same candidates. On the same event, replacing wˉ by w changes any two programs’ score difference by at most mε. Fix a correct program. A top-k miss under w then requires at least k incorrect programs to score at or above that program’s score minus mε under wˉ. For each fixed incorrect program, §2.2 gives a Hoeffding bound on this margin under the fixed weights wˉ. Applying Markov’s inequality to the count over all n− incorrect programs and adding η for failure of the concentration event gives, for D>0,
Pass@k(w(B))≥1−η−kn−exp!(−2∣wˉ∣22D2).
At the nominal dimensions of Table 2, the 95% error allowance gives D≤0, so this bound cannot certify Pass@k there. The matched split above measures the change in Pass@1 at that scale.