acceptodds
Under review as a conference paper at ICLR 2027

Witness Before You Judge: Where Test-Time Program Selection Fails and How to Fix It

Abstract

Test-time scaling for code generation samples many candidate programs and spends LLM calls to pick one. When the pick is wrong, did the judge misread the evidence, or was the evidence never found? We study selection through witnesses, inputs on which two candidates behave differently. On a development pool of LiveCodeBench tasks, a pairwise LLM judge picks the correct program in all 60 comparisons in which it is shown a witness, yet 41 comparisons end without one. Time-outs account for 18 of them, and for 14 of these the first failing hidden test is longer than the inputs a judge is shown. We therefore separate evidence construction from judgement. WitnessFirst asks the LLM to propose objective witnesses, constraint-scale inputs from model-written generators, checked by a model-written validator, on which execution alone rejects programs that crash or time out; a judge then decides the disagreements that remain, reading the code when no witness is found. These eliminations are sound whenever the validator is and execution is reproducible, so each disagreement with a benchmark label is an invalid input, an unreproducible outcome, or a gap in the benchmark's tests. Across eight held-out candidate pools from LiveCodeBench and CodeContests+ and four judges, placing the objective stage in front of each of four selectors improves it with open-weight judges, by 3.2 to 5.7 points under the benchmarks' labels and 5.2 to 9.3 under audited labels; behind it, a six-call vote does at least as well as a 36-call one, and with Qwen3-32B the full WitnessFirst is the most accurate of the main selectors. In our pools, 1.9% of the benchmark-correct LiveCodeBench programs fail on legal inputs, spanning at least 13 of the 342 problems. With a frontier judge, reading the code alone is the most accurate selector, and objective witnesses still help the selectors that do not read code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Code Generation, Program Selection, Test-Time Scaling, LLM-as-a-Judge, Test Generation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ZEssOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether failures to select a correct candidate program arise from incorrect judgment or missing discriminative evidence. It distinguishes objective witnesses settled by crashes/timeouts from subjective witnesses requiring a model to judge outputs. WITNESS FIRST generates inputs near constraint limits and validators, eliminates candidates through execution, and then uses small-input comparisons with a code-judgment fallback. Evaluation covers LiveCodeBench, CodeContests+, eight candidate pools, and four judges. The objective stage improves multiple selectors, but the full method is not best for every judge. The audit also finds that some benchmark-passing programs fail under inputs accepted by the authors' validators and their resource settings.

Critical strengths
  1. The central distinction is explanatory and reusable. Separating large inputs settled by execution from small inputs requiring reading directly targets scale-related errors. The objective filter composes with several selectors, providing broader practical value than a single monolithic pipeline.
  2. Reporting and statistics are transparent. Both original and audited labels are reported, with problem-clustered intervals and paired analyses. The paper discloses the full method's disadvantage with the strongest judge, favorable results after removing the validator, and protocol changes, and describes plans to release traces and prompts.
Critical weaknesses
  1. The diagnosis and incremental value of the full method require narrower claims. GPT-5.5's 60/60 correct witnessed verdicts on development data do not establish that all judges merely lack witnesses. Held-out Qwen3-32B results include 10 incorrect witnessed judgments, and other models produce many ties. This limits empirical applicability of the lemma's premise without refuting the lemma itself. The full method's gain over objective-stage-plus-code-judgment has an interval spanning zero while requiring 42 versus 17 calls. Of 57 extra successes over knockout, 39 come from code fallback. Current evidence more strongly supports a composable objective filter than a generally superior complete pipeline. Gains from large-scale inputs also need to be separated from gains from adding execution filtering.
  2. Audited labels and efficiency claims rely on insufficiently validated conditions. Reference validators are mainly checked for accepting valid inputs, without independent evaluation of false acceptance on invalid inputs. Audited labels are related to similar witness construction, so shared bias may remain despite leave-one-judge-out labels. The audit also changes some mixed-task sets, preventing strict same-population comparison of gains across label sets. The relationship between the 6-second/512 MiB environment and original contest limits is unclear, while call counts omit large-input generation and repeated execution costs. Independent validator checks, fixed comparison sets, and explicit resources and costs are needed to support the benchmark-gap and efficiency claims. These addressable evidence gaps lead me to lean toward rejection.
Questions for authors
  1. Diagnostic scope: Please separate correct, incorrect, and tied witnessed verdicts by judge. Which filters produced the development-set 60/60 result, and will the claim about missing evidence rather than wrong judgment be restricted to supported conditions?
  2. Validators: Is there an independently human-labeled set of valid and invalid inputs for evaluation in both directions? Inputs labeled invalid by the same reference validator cannot independently establish its false-accept rate; please explain the actual audit ground truth.
  3. Audit comparison: Please report official resource limits for the 13 LiveCodeBench problems and compare both label sets on the same task set, distinguishing test-coverage gaps from different resource standards.
  4. Incremental value and cost: Using existing logs, please report tokens, execution counts, and runtime for the complete method, objective-filter-plus-code-judgment, and objective-filter-plus-vote. Which differences have paired statistical support?
  5. Mechanism: Are small-input-only objective-filter results or program-aware ablations with a fixed generator taxonomy already available? If not, please narrow attribution to input scale and the complete pipeline and describe existing controls for contamination on public problems.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZEss (1/2)

Thank you for the precise questions.

Q1 (diagnostic scope). Witnessed discriminative comparisons by judge (correct / wrong / tie): Qwen3-4B 0 / 1 / 31, Qwen3-8B 13 / 3 / 15, Qwen3-32B 24 / 10 / 13, GPT-5.5 187 / 5 / 0. The development-set 60/60 needed no extra filter: GPT-5.5 decided every witnessed discriminative comparison with both orders agreeing, and there were no witnessed ties. We now restrict "missing evidence rather than wrong judgement" to that setting. A new held-out attribution table shows that missing witnesses still cause most knockout errors for every judge (42/55, 46/52, 39/51, 10/14), with wrong or inconsistent witnessed verdicts behind the rest.

Q2 (validators in both directions). No human-labelled set of valid and invalid inputs existed for these problems, so the ground truth we added comes from the statements rather than from the validator. We mutate valid public and hidden inputs so that exactly one stated constraint is violated, which gives 1,237 mutants over 107 constraint families, structural ones included; the reference validators reject all of them. Separately, a second checker that shares no code with them measures each constrained quantity on the 58 valid eliminating inputs and agrees with the reference validators on all 58. The validators, and the scripts and results of the mutation test and of the second checker, are in the supplementary material.

Q3 (resource limits and a fixed task set). All 12 AtCoder problems in the audit have official limits of 2 s / 1024 MiB; Codeforces 1582G has 2 s / 512 MB; LeetCode 3612 publishes none. Our sandbox is more lenient in time (6 s, matching LiveCodeBench's harness) and stricter in memory (a 512 MiB address space, where LiveCodeBench sets no cap). We re-ran all 88 relabelled programs on their eliminating inputs under 512 MiB, 1024 MiB and 16 GiB. For 84, some eliminating input makes every run fail under every cap, so the gaps do not come from our memory limit. The four programs whose failures did not recur in every run are removed (now 82 programs on 12 problems). On the fixed official set of mixed units, the audited gains of the filter are +7.5 (vote), +6.0 (CodeT), +5.2 (code-only) and +5.5 (knockout), all with intervals that exclude zero.

AuthorsOpenReview ↗

Response to Reviewer ZEss (2/2)

Q4 (incremental value and cost). Per mixed unit with the Qwen3 judges, WitnessFirst uses 40.0k tokens and 55 candidate executions, filter plus code-only judging 16.4k and 44, and filter plus vote 9.8k and 178. Their candidate runs take 10.4, 9.9 and 17.7 s in total. The paired differences are WitnessFirst over filter plus code-only judging +0.9 [-1.5, 3.3], and WitnessFirst against filter plus vote -2.3 [-8.5, 4.3]; neither is supported. At matched calls, with the code-only judge taking three votes per order (45 against 42 calls), WitnessFirst is +0.9 [-1.8, 3.5] above filter plus code-only judging (Table 30). Against tokens, the Qwen3 frontier consists of the first passer, the vote and CodeT, each alone and behind the filter, while with GPT-5.5 WitnessFirst solves 18.1 points [11.1, 25.4] more than the filter plus vote (Table 25). The differences with paired support are the filter's gains over all four selectors under the audited labels and the full pipeline's gain over the knockout (+9.5 [3.3, 15.6], and +8.8 [3.1, 14.7] pooled over three runs, Table 33). The filter's gain over the unfiltered vote under the benchmarks' labels also holds over three runs when counted on the 54 units where the filter removes a program (+3.6 [0.3, 7.5] pooled, Table 33); on the other units the two selectors differ only by their suites. The revision recommends the filter with a vote or a code-only judge.

Q5 (mechanism and contamination). Two new controls address the mechanism. A prompt-matched run with the Qwen3 judges that withholds the programs changes no selector by more than 1.1 points (Tables 22 and 23), so reading code is not what drives the gain. Replaying the objective stage with only inputs of at most 300 bytes keeps a significant gain (+1.7 [0.3, 4.0] with the Qwen3 judges, +6.0 [1.7, 11.6] with GPT-5.5), and larger inputs add to it (Table 27). Prompting for inputs of at most 300 bytes directly agrees: +3.7 [0.9, 6.8] behind the vote, with the unrestricted prompt adding +3.7 to +4.3 under the audited labels (Table 32). We also ran a fixed-taxonomy baseline without any LLM: maximum-size, random maximum-size and boundary inputs from specifications transcribed from the statements. On both benchmarks it filters at least as well as the model-written generators (on LiveCodeBench +7.0 [1.0, 13.9] against +3.7 [0.0, 7.8] over the first passer with the Qwen3 judges, Table 31; on CodeContests+ 18.8 points [0.0, 41.7] ahead, Appendix F), so we narrow the attribution as you suggest, to executing inputs at the scale of the constraints. A program-agnostic variant with that fixed taxonomy, where the model writes only an input constructor, removed more flawed programs than our generators on the development pool but fewer on the held-out units (73 against 117), and we report it as it came out (Table 34). On contamination, every held-out problem predates the Qwen3 release (29 April 2025; the latest LiveCodeBench problem is from 6 April 2025), and we found no public source of later contest problems with official hidden tests: the latest LiveCodeBench release ends in April 2025, and AtCoder has suspended publishing its test cases. So we describe the controls we have (Appendix E). The objective stage removes a program only when it crashes or times out on an input the validator accepts, which no judge decides; in front of the first passer, which consults no judge either, it gains +4.0 [0.6, 7.8] with the Qwen3 judges (Table 24). A model that remembers a problem might write better inputs for it, but inputs from a generator without any LLM gain +7.0 [1.0, 13.9] over the first passer (Table 31). Contamination can still favour the judges' verdicts, and with them code-only judging. A split by contest date (before and after 2025-01-01) gives the vote's filter gain +5.9 and +4.8, but first-passer accuracy drops as well (58.8% to 51.0%, code-only 62.1% to 57.8%), so it cannot separate contamination from difficulty. Section 7 states this as a limitation.

Thank you again; your questions led directly to the corrected audit.

AuthorsOpenReview ↗

Follow-up for Reviewer ZEss

Thank you again for the review. Since our response we have posted further updates and revised the PDF; for your questions:

  • Diagnostic scope (Q1): verdicts split by judge into correct, incorrect and tied, and the claim restricted to the supported conditions (Tables 4 and 16).
  • Labels (Q3): both label sets on the same fixed units (Table 3) and their relation to the official limits (Table 18).
  • Full method against the filter and the code-only judge at matched calls (Q4): Table 30 with the Qwen3 judges and Table 40 with GPT-5.5, where all selectors that use the judge are near the top.
  • Contamination (Q5): judges whose weights predate every held-out LiveCodeBench problem (Qwen2.5-7B, 14B and 32B) show the gains of the code-only judge and of WitnessFirst over the knockout (8.3 points [1.4, 14.7]) as well, and smaller gains of the objective stage (Table 41), together with the generator without any LLM (Table 31).

If anything remains, we would be glad to address it before the PDF deadline on November 19. If these changes resolve some of your concerns, we would be grateful if you would take them into account in your assessment.

AuthorsOpenReview ↗

Are any concerns left? (Reviewer ZEss)

Your review closes with "These addressable evidence gaps lead me to lean toward rejection." We believe each gap is now closed and would like to know whether anything still keeps your rating at weak rejection.

  • Narrower claims (W1, Q1): the diagnosis is restricted to the conditions that support it, with correct, incorrect and tied witnessed verdicts per judge (Tables 4 and 16), and the paper now presents the composable filter, not the full pipeline, as the method.
  • Large inputs versus execution filtering (W1, Q5): inputs of at most 300 bytes and a prompt for small inputs, with both judge families (Tables 27, 32 and 40), and a fixed-taxonomy variant (Table 34).
  • Independent validator checks (W2, Q2): the mutation test takes its ground truth from the statement, not from our validators: each of the 1,237 mutants violates exactly one stated constraint, and all are rejected. A second checker that shares no code with the validators agrees on all 58 valid eliminating inputs (Table 18). We have not built a human-labelled set; these two tests are our answer to the false-accept question.
  • Fixed comparison sets (W2, Q3): both label sets on the same units in Table 3, and the official limits for the relabelled problems in Table 18.
  • Resources and costs (W2, Q4): Tables 17, 25 and 28.
  • Contamination (Q5): judges whose weights predate every held-out LiveCodeBench problem keep the gains of WitnessFirst over the knockout, 8.3 points [1.4, 14.7] (Table 41).

If these close the gaps you named, we would be grateful if you would consider raising your rating. If something remains, please tell us and we will address it before the PDF deadline on November 19.

Reviewer_tEh1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper diagnoses witness-search and judging failures in program selection. WitnessFirst uses generated and validated large inputs to filter crashing or repeatedly timing-out candidates, then performs witness-based comparisons with a code-only fallback. It also audits benchmark labels using discovered counterexamples.

Critical strengths

1/ Witness-level diagnosis. This paper distinguishes missing witnesses from incorrect verdicts and attributes knockout failures in its development setting to missing witnesses through a lemma and empirical analysis.

2/ Reference-free filtering at test time. This paper uses generator-validator to construct large inputs and eliminates crashing or timing-out candidates. It reports a 5.7 points improvement when adding the filter to voting across Qwen judges on mixed units under the original labels.

Critical weaknesses

1/ Novelty. Generator–validator testing and distinguishing-input selection have precedents in CodeContests+ and S*. The lemma also follows directly from the knockout procedure under correct witnessed judgments. The substantive conceptual advance of the reference-free application and diagnostic framework should be clarified.

2/ Validation. Independent expert checks of relabelling inputs and tests of invalid-input rejection are needed to substantiate the audit. Controlled input-size experiments should also validate the scale-mismatch diagnosis. The full pipeline adds only 0.9 points over filtering + code-only judging, at about 42 vs. 17 calls; comparisons under the same compute budget should establish its additional benefit.

Questions for authors

Please see Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tEh1

Thank you for the review. We take your points in order; table numbers refer to the revised PDF.

Novelty. The difference from CodeContests+ and S* is what is available at selection time and who makes the call. CodeContests+ builds tests offline, with a reference solution in hand. S* constructs inputs that separate two programs and then asks an LLM which output is right. At test time we have no reference solution, and the objective stage never asks anyone for a verdict: a program is removed only because it crashes or times out on an input the validator accepts. What we claim is the diagnosis (most knockout errors come from a missing witness), that this filter can be placed in front of any selector, and the audit, in which the same inputs expose gaps in benchmark tests. You are right that the lemma follows directly from the knockout procedure. The revision calls it an organising statement, not a technical result.

Validating the audit. We added two checks on the validators and one on execution. To test rejection, we took valid inputs and broke one stated constraint at a time; the reference validators rejected all 1,237 such mutants, which cover 107 constraint families. A second checker that shares no code with the validators accepts all 58 valid eliminating inputs. Finally, we re-ran every relabelled program under the official AtCoder memory limit (1024 MiB) and with 16 GiB; our 6 s time limit is already looser than the official 2 s. For 84 of the 88 programs some eliminating input fails in every run, and we dropped the other four. That leaves 12 LiveCodeBench problems, each found by at least two judges. The inputs, the programs and a standalone checker are in the supplementary material.

Input size. We replayed the objective stage on the same generated inputs, keeping only accepted inputs of at most 300 bytes, which a judge can read (Table 27). With the Qwen3 judges these small inputs alone give a significant gain over the first passer, +1.7 [0.3, 4.0]. The larger inputs add +2.3 [-0.6, 5.4] under the benchmarks' labels and +3.7 [1.4, 6.6] under the audited labels. With GPT-5.5 the same three numbers are +6.0 [1.7, 11.6], +3.4 [-0.9, 8.5] and +6.0 [1.9, 10.6]. So executing readable inputs already helps, and inputs too large to read help more. We also asked the model for small inputs directly, with the same prompt except that every input must be at most 300 bytes (Table 32). The models comply poorly (0.29 accepted inputs per unit, against 1.33), yet the small-input filter still gains +3.7 [0.9, 6.8] behind the vote. The original prompt adds +2.0 to +2.3 under the benchmarks' labels (intervals include zero) and +3.7 to +4.3 under the audited labels (intervals exclude zero). On the scale side, maximum-size and boundary inputs from a generator without any LLM gain +7.0 [1.0, 13.9] over the first passer on LiveCodeBench (Table 31). A prompt-matched control also shows that seeing the code is not what drives the gain (Tables 22 and 23).

42 versus 17 calls. Agreed. The full pipeline beats filter plus code-only judging by only +0.9 points [-1.5, 3.3]. We also compared them at the same budget: with three sampled votes per order, the code-only judge uses 45 calls and 39.8k tokens per unit against 42 calls and 40.0k tokens for the full pipeline, and the full pipeline is still only +0.9 [-1.8, 3.5] ahead (Table 30). The revision therefore recommends the cheaper combination and treats the full pipeline as one combination among several.

AuthorsOpenReview ↗

Follow-up for Reviewer tEh1

Thank you again for the review. Since our response we have posted further updates and revised the PDF. For the two critical weaknesses:

  • Novelty (W1): the revision states the relation to generator-validator test construction directly; at test time there is no reference solution, so the stage only removes programs that fail on validated inputs and never asks for a verdict.
  • Input size and the audit (W2): a prompt restricted to inputs of at most 300 bytes (Table 32), inputs from a generator without any LLM (Table 31), and the full method against the filter followed by the code-only judge at matched calls (Table 30). The eliminating inputs and validators are in the supplementary material so that they can be checked independently. With GPT-5.5, which follows the small-input prompt, the small-input stage gains 6.0 points [1.9, 10.8] over the first passer and the unrestricted prompt adds 4.3 [0.0, 9.2] under the audited labels (Table 40).

If anything remains, we would be glad to address it before the PDF deadline on November 19. If these changes resolve some of your concerns, we would be grateful if you would take them into account in your assessment.

AuthorsOpenReview ↗

Are any concerns left? (Reviewer tEh1)

Your review rated soundness 2, "minor or fixable concerns", and named two critical weaknesses. We believe both are now addressed and would like to know whether anything still keeps your rating at weak rejection.

  • Novelty (W1): S* compares candidates on distinguishing inputs judged by an LLM, and CodeContests+ builds generators and validators offline with reference solutions. Our filter needs neither a reference solution nor a verdict: it removes a program only when it crashes or times out on an input the validator accepts. Beyond the filter, the contributions are the diagnosis that most held-out knockout errors lack a witness, for every judge (Table 4), and the audit of the benchmarks' labels.
  • Validation (W2): invalid-input rejection is tested with 1,237 single-constraint mutants, all rejected, and an independently written second checker agrees on all 58 valid eliminating inputs (Table 18); every eliminating input, validator and relabelled program is in the supplementary material for independent inspection. Input size is controlled with inputs of at most 300 bytes and with a prompt for small inputs (Tables 27, 32 and 40). At the same number of calls, WitnessFirst is 0.9 points [-1.8, 3.5] above the filter followed by the code-only judge (Table 30), so we no longer claim the full pipeline is better and present the filter as the method.

If these resolve your concerns, we would be grateful if you would consider raising your rating. If something remains, please tell us and we will address it before the PDF deadline on November 19.

Thank the author for their efforts. I carefully read the content of the rebuttal. After comprehensive consideration, I think this paper does not meet the standards of ICLR. Therefore, I will maintain the original score.

Reviewer_7Yk5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to select the correct program from multiple LLM-generated candidate programs. It points out that pairwise judges often fail because they cannot find inputs that expose behavioral differences between programs, particularly crashes and timeouts that occur only on constraint-scale inputs. WITNESSFIRST uses an LLM to generate inputs that are checked by a validator, first eliminating such programs based on execution results and then performing pairwise judgment. Across eight held-out candidate pools, this stage improves the average performance of several existing selectors. The authors also find that at least 1.9% of the LiveCodeBench programs in their candidate pools that pass the hidden tests fail on valid inputs, revealing insufficient benchmark test coverage.

Critical strengths
  1. The evidence-centered problem formulation is insightful. Distinguishing missing witnesses from incorrect judgments explains selection failures more effectively than reporting overall accuracy alone. 
  2. The method is simple, modular, and incorporates executable checks. The objective stage requires neither ground-truth answers for the generated inputs nor reference solutions, and it can be combined with various selectors. 
  3. The experimental comparisons are comprehensive, and the statistical analysis is generally careful. The experiments include eight candidate pools, two benchmarks, ablation studies, and paired uncertainty estimates. Both original and audited labels are reported, providing thorough experimental comparisons, and negative results and unfavorable factors are honestly included.
Critical weaknesses
  1. The number of LLM calls does not adequately represent the actual cost. The claim of “low cost” needs to be supported by input/output token counts or inference time. In general, we should not use the number of LLM calls to represent cost, since it is otherwise clearly possible to perform many operations within a single call, allowing the model to reason for a long time and produce lengthy outputs. 
  2. The robustness and scope of the experimental improvements remain unclear, and the contributions of failure attribution and program-aware construction have not been fully isolated. In the held-out experiments, the open-weight judges make incorrect judgments despite being shown witnesses and produce a large number of ties. Meanwhile, the program-agnostic generator differs from the full method by only 0.9 percentage points, and Appendix B states that the differences between the two prompts are not limited to whether candidate code is provided. Therefore, the current ablation cannot independently demonstrate the value of program-aware witness construction. 
  3. The audited labels are not entirely independent of the methods being evaluated. Leave-one-judge-out labeling avoids evaluating a judge using its own witnesses, but the other judges still run highly correlated objective stages on the same candidate pools, so shared generator biases and common misunderstandings of the problem statements may still exist. In addition, the fact that the reference validator accepts all public and hidden inputs only verifies whether it incorrectly rejects valid inputs; it cannot verify whether it incorrectly accepts invalid inputs.
Questions for authors

Could you report the input/output token counts, number of candidate-program executions, total size in bytes of the generated inputs, CPU time, actual wall-clock time, and specific execution hardware? How do the methods perform under budgets that are roughly aligned in terms of tokens, time, or computational cost?  Excluding hybrid problems, what are the end-to-end selection accuracies on all solvable problems and on all problems, respectively? Are the results stable across multiple random seeds for test generation and selection and, ideally, on independently sampled candidate pools?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7Yk5 (1/2)

Thank you for the careful review.

Cost. We agree that calls are a weak proxy. From the logs (means per mixed unit, Qwen3 judges): vote 5.4k tokens, filter plus vote 9.8k, the 36-call vote 46.2k, code-only 12.8k, filter plus code-only 16.4k, knockout 23.9k, WitnessFirst 40.0k. Candidate executions per unit range from 36 (code-only) to 438 (the 36-call vote), taking 1.5 to 27 s in total (Table 17, Appendix E). The objective stage accepts on average 0.8 / 1.3 / 1.9 / 3.2 inputs per unit (Qwen3-4B / 8B / 32B / GPT-5.5), 0.8 to 1.3 MB in total.

The open judges were served with vLLM, Qwen3-4B and -8B sharing one A100-80GB and Qwen3-32B on another, and candidates ran in a bubblewrap sandbox on 32-core CPU workers. Wall-clock times were measured under concurrent load on shared servers, so we only report orders of magnitude: about 30 s per unit for the code-only judge and about 10 minutes for the 36-call baselines. We also measured CPU time directly on 64 sampled units: the objective stage takes 9.4 s of CPU per unit on average (median 2.7 s), 87% of it in candidate executions (Table 28).

In tokens the conclusion holds and gets stronger: the filter plus a 6-call vote matches the 36-call vote with about a fifth of the tokens. For the Qwen3 judges, the token frontier consists of the first passer, the vote and CodeT, each alone and behind the filter (Table 25). With GPT-5.5, the code-only judge alone (21.7k tokens) is the most accurate selector under the benchmarks' labels, and WitnessFirst under the audited labels. Aligned in calls instead, the full pipeline and filter plus a code-only judge with three votes per order (42 and 45 calls per unit) are within 0.9 points [-1.8, 3.5] (Table 30).

End-to-end accuracy. Over all 514 solvable (pool, task) units, the first passer reaches 89.5%; with Qwen3-8B, the vote reaches 90.7% and filter plus vote 93.0%; with Qwen3-32B, WitnessFirst reaches 93.2%; with GPT-5.5, code-only reaches 98.4%. Over all 2,092 (pool, task) units every method is between 22% and 24%, because most problems have no correct candidate in the pools. The revision reports both (Table 19), together with the mixed-unit numbers that Proposition 1 motivates.

AuthorsOpenReview ↗

Response to Reviewer 7Yk5 (2/2)

Robustness and attribution. You are right that the original agnostic ablation does not isolate code reading. A new prompt-matched control with the Qwen3 judges, identical to our prompt except that the programs are withheld, changes no selector by more than 1.1 points (Tables 22 and 23), so we withdraw that attribution. On witnessed judgements: with a witness in hand, the open judges often tie because their two orders disagree (31 of 32 witnessed comparisons for Qwen3-4B, 15 of 31 for 8B, 13 of 47 for 32B). The revision states this as a second failure mode next to missing witnesses. Per-benchmark intervals are now in the main text (Table 3, Section 5.1) and per-pool intervals in Appendix E (Table 21).

On independently sampled pools, we drew the four LiveCodeBench pools again with the same generators and settings but another seed (a resampled program repeats an original one no more often than two programs of one pool repeat each other), labelled them with the same protocol, and reran every selector with the Qwen3 judges on the 270 new mixed units. One task of the 32B pool was left out because two of its programs exceed the scorer's time budget. Behind the five selectors the filter gains 1.9 to 5.2 points under the benchmarks' labels (2.3 to 5.3 on the original pools), and 3.0 to 6.7 under audited labels drawn from the other Qwen3 judges (3.7 to 6.7 originally); under the audited labels the intervals exclude zero for four of the five. WitnessFirst gains 9.6 points [3.7, 16.1] over the first passer (7.7 originally). Table 29 (Appendix F) has the details. On seeds: the objective stage and the judges decode greedily, and we reran the sampled steps (pairwise inputs and shared suites) twice more with seeded requests (Table 33). WitnessFirst over the knockout stays at +8.0 to +9.5 in all three runs. The filter's gain behind the vote moves from +0.6 to +5.7 under the benchmarks' labels, but the movement is the vote's own randomness on the 294 units where the filter removes nothing; on the 54 units where it removes a program, the gain is +3.2 to +4.0 in every run, +3.6 [0.3, 7.5] pooled (+5.6 [2.6, 9.1] under the audited labels). We did not resample the CodeContests+ pools.

Independence of audited labels. Leave-one-judge-out labels remove a judge's own witnesses, but, as you say, the other judges share the pools and the construction procedure, so a shared bias cannot be excluded. What we added addresses the validator side: statement-derived invalid mutants (1,237, all rejected), an independent second checker (58/58 agreement), and re-execution under official and relaxed limits (84 of 88 failures reproduce; the four that do not are removed). On the judge side, we relabelled the Qwen3 units only with inputs found by GPT-5.5, and the GPT-5.5 units only with inputs found by the Qwen3 judges. Every filter gain keeps an interval that excludes zero (Qwen3: vote +7.2 [3.1, 11.6], CodeT +5.7, code-only +4.9, knockout +5.2; GPT-5.5: vote and CodeT +10.3; Table 26). A generator that uses no LLM at all (Table 31) also removes 30 of the relabelled programs, each failing in every re-execution, so these labels do not rest only on inputs that the judges' models wrote. We also recomputed the audited gains on the fixed official set of mixed units, so that both label sets describe the same population, and all four filter gains exclude zero there (vote +7.5 [3.5, 11.9], CodeT +6.0, code-only +5.2, knockout +5.5; Table 3).

AuthorsOpenReview ↗

Follow-up for Reviewer 7Yk5

Thank you again for the review. Since our response we have posted further updates and revised the PDF; in brief, for your questions:

  • Costs (W1): tokens, calls, executions, input bytes, CPU time and wall-clock time per unit, with the hardware (Tables 17, 25 and 28).
  • Budget-aligned comparisons: the filter with a 6-call vote against a 36-call vote at about a fifth of the tokens (Table 25), and all components at matched calls (Table 30 with the Qwen3 judges, Table 40 with GPT-5.5).
  • End-to-end accuracy on all solvable units (Table 19).
  • Independently resampled pools and repeated runs, now with both judge families (Tables 29, 33, 37, 38 and 39). On LiveCodeBench the gains reappear on the resampled pools; on CodeContests+ they do not with the Qwen3 judges, which the Limitations now state.
  • The prompt-matched control, now also with GPT-5.5 (Tables 35 and 36).

If anything remains unclear, we would be glad to address it before the PDF deadline on November 19. If these changes resolve some of your concerns, we would be grateful if you would take them into account in your assessment.

AuthorsOpenReview ↗

Are any concerns left? (Reviewer 7Yk5)

Your review rated soundness 2, "minor or fixable concerns", for three reasons, and asked five questions. We believe all of them are now answered and would like to know whether anything still keeps your rating at weak rejection.

  • Cost (W1, Q): tokens, executions, input bytes, CPU and wall-clock time with the hardware (Tables 17 and 28); under aligned token budgets the filter with a 6-call vote matches the 36-call vote at about a fifth of the tokens (Table 25), and all components are compared at matched calls (Tables 30 and 40).
  • Robustness (W2, Q): seeded repeated runs with both judge families (Tables 33 and 39), with WitnessFirst 8.8 points [3.1, 14.7] above the knockout pooled over three Qwen3 runs; independently resampled pools (Tables 29, 37 and 38), where on LiveCodeBench WitnessFirst gains 9.6 points [3.7, 16.1] over the first passer. The CodeContests+ gains do not reproduce on resampled pools, and the Limitations say so.
  • Program-aware construction (W2): the prompt-matched control shows that code access does not drive the accuracy gain, and we removed that claim.
  • Independence of the audit (W3): labels drawn only from the other model family keep the gains (Table 26); the non-LLM inputs remove 33 benchmark-correct programs, 30 of which the audit already relabels; and the mutation test and the second checker test false acceptance (Table 18).
  • End-to-end accuracy on all solvable units (Q): Table 19.

If these resolve your concerns, we would be grateful if you would consider raising your rating. If something remains, please tell us and we will address it before the PDF deadline on November 19.

Reviewer_VcDtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies program selection among multiple LLM-generated code candidates through the lens of witnesses, namely valid inputs on which candidates behave differently. It argues that many selection failures arise because a discriminating input is never generated, especially for performance bugs that appear only at constraint scale, and proposes WITNESSFIRST: generate and validate large objective stress inputs to eliminate programs that crash or time out, then use small subjective witnesses and a code-only fallback for remaining comparisons.

The evaluation considers held-out candidate pools from LiveCodeBench v5/v6 and CodeContests+, compares several selectors under four judge models, and additionally audits programs that pass benchmark hidden tests but fail on generated valid stress inputs.

Critical strengths

The paper offers a useful decomposition of selection failures into evidence construction versus judgement. Lemma 1 makes the dependence of incumbent-preserving knockout selection on witnessed discriminative comparisons explicit, and Proposition 2 cleanly states the conditions under which objective crash/time-out eliminations are sound. The latter is deliberately conditional, but the condition is stated clearly enough to distinguish an execution-backed elimination from an LLM verdict.

The central motivation is communicated particularly well in Figure 1. The top panel illustrates why small, judge-readable inputs can fail to expose asymptotic complexity errors, while the lower panel gives a concrete constraint-scale example where execution eliminates six public-passing but inefficient candidates. This directly supports the paper's core design choice: witnesses intended for execution need not be compressed to a size that an LLM judge can reason over.

The experimental study is unusually transparent about selector behavior rather than reporting only aggregate accuracy. For example, Table 1 distinguishes wrong-answer, time-out, and crash failures on the development pool, and demonstrates that the GPT-5.5 knockout's remaining errors are concentrated in failures for which no witness was found. This is a much more informative analysis than simply stating that one selector outperforms another.

Critical weaknesses

1. The formal soundness statement is correct but rests on exactly the component that is empirically weakest: input validity.

Proposition 2 proves only that no semantically correct program is removed if the generated validator accepts only valid inputs and execution is reproducible. This is a valid conditional result, but it should not be read as evidence that the observed eliminations are generally sound. The paper itself shows that the condition can fail severely: on Page 8, the reference-validator audit accepts only 9 of 64 benchmark-correct programs removed by Qwen3-4B as valid, while 55 were invalid. Moreover, the objective stage removes every benchmark-correct passer on between 1 and 4 of the 116 mixed units depending on the judge.

The validation procedure described on Page 6 checks that the authors' reference validators accept all public and hidden inputs. This establishes an important recall-style property, but does not establish that they reject malformed inputs, invalid semantic structures, or inputs violating nonlocal constraints. Since the central safety guarantee depends on exactly this latter property, the empirical validation is one-sided. The authors should substantially strengthen this audit, for example through independently implemented validators, systematic mutation tests that create invalid inputs, public release of validators and witness cases for manual inspection, and ideally benchmark-maintainer review of the claimed benchmark gaps.

2. The evidence does not isolate the value of program-aware LLM stress construction from generic “generate maximum-size tests” prompting.

A major motivation on Page 5 is that the model reads surviving programs, identifies risks such as an O(n2)O(n^2) loop, and targets them with suitable stress inputs. Yet Table 2 reports that WITNESSFIRST with program-agnostic generators is within 0.8 to 0.9 points of the program-aware version for the Qwen3 judges. The paper acknowledges that program-aware and agnostic prompts differ in more than merely whether code is visible, so this comparison does not identify whether reading candidate code is responsible for the result.

This matters because constraint-scale max/min/structured-edge tests are a strong non-LLM baseline for performance-bug detection. The work would be considerably stronger with controlled comparisons to: fixed hand-written max-size and boundary generators, randomized valid stress generators, property-based testing, and a program-agnostic LLM prompt matched token-for-token to the program-aware prompt except for removal of candidate code. Without these controls, the evidence supports the usefulness of execution on generated stress inputs, but less strongly supports the claimed benefit of candidate-aware witness construction.

3. The broadest empirical claims should be qualified by the restricted evaluation population and uncertain benchmark-label effects.

The main comparison contains 116 mixed (pool,task)(\text{pool},\text{task}) units, of which 100 come from LiveCodeBench and only 16 from CodeContests+ (Page 6 and Table 2). Conditioning on solvable mixed tasks is well motivated by Proposition 1 because these are the cases where selection can alter the answer. However, it means the reported selector gains do not establish the end-to-end benefit on arbitrary tasks, including tasks with no correct candidate or only one public passer.

The paper is careful in Section 5.1 to note that, under the benchmark labels, only the vote improvement has an interval excluding zero when pooling the Qwen3 judges. Nonetheless, the abstract and Figure 2 emphasize that the objective stage improves every selector, which is accurate as a point-estimate statement but stronger than the benchmark-label uncertainty supports for all compositions. The results should report per-benchmark and per-pool uncertainty prominently in the main text, repeat stochastic input-generation runs, and distinguish “consistent positive point estimates” from statistically established gains.

4. The full WITNESSFIRST result conflates the objective filter with the code-only fallback.

Algorithm 1 does not simply place objective witnesses before knockout selection. On Page 5, when no small separating input is found, it replaces the incumbent-preserving tie with a code-only LLM comparison. The paper's own decomposition on Page 8 reports that, across Qwen3 judges, 39 of WITNESSFIRST's 57 wins over knockout come from the code-only fallback, versus 18 from objective eliminations. This is important because the fallback is subjective and can be wrong, whereas the paper's central conceptual contribution emphasizes objective, checkable evidence.

The relevant ablations are partially present, but the presentation should make the distinction much sharper. In particular, Figure 3 plots an objective-stage knockout without the fallback alongside full WITNESSFIRST, while Table 2 compares the latter at 42 calls with a plain knockout at 25 calls. A clearer evaluation would separately report: objective stage alone; objective stage plus incumbent-preserving knockout; code-only fallback alone under the same comparison budget; and the exact incremental contribution of each component under matched call budgets.

5. The diagnosis “selection fails for want of witnesses rather than judgement” is compelling for the development setup but should not be generalized beyond it.

The development diagnosis uses GPT-5.5 on 272 LiveCodeBench tasks and finds 60 witnessed discriminative comparisons, all decided correctly, versus 41 comparisons lacking a witness. This is informative evidence for that exact model, candidate pool, small-input budget, and knockout protocol. But the held-out analysis is more mixed: Section 5.3 reports that Qwen3 judges frequently contradict themselves under program-order swaps, and the paper's conclusion explicitly notes that held-out traces contain incorrect witnessed verdicts.

Thus, the strongest defensible claim is that missing witnesses dominate in the analyzed development condition, while both witness search and judgement reliability matter elsewhere. The paper should report a held-out table analogous to Table 1, broken down by failure type and by whether the final error arose from missing witnesses, erroneous witnessed judgement, or code-only fallback. That would better establish how broadly the motivating diagnosis transfers.

Questions for authors
  1. For the programs removed by the objective stage, can the authors provide a stricter validation study of the validators' specificity, not only their acceptance of public and hidden inputs? For example, how often do independently constructed invalid mutations get rejected, especially for structural and semantic constraints such as graph connectivity guarantees?

  2. Can the authors rerun the program-aware versus program-agnostic comparison with identical generator prompts, decoding settings, output budgets, and requested input categories, differing only in whether candidate code is supplied? This would clarify whether code reading materially improves witness construction.

  3. What is the performance of simple non-LLM stress-testing baselines, such as hand-written maximum-size generators, random valid maximum-scale generators, and property-based boundary generators? This comparison seems necessary to establish the marginal value of LLM-generated stress witnesses.

  4. Could the authors provide a fully matched component ablation of WITNESSFIRST: objective filter only, code-only fallback only, objective filter plus standard knockout, and the full method, each at comparable call budgets? In particular, how much of the Qwen3-32B gain remains when the fallback is removed?

  5. For each held-out judge, what fraction of final errors is attributable to: no separating witness found; erroneous witnessed judgement; invalid-input elimination; time-out reproducibility failure; and erroneous code-only fallback? A held-out analogue of Table 1 would make the paper's causal narrative more convincing.

  6. Can the authors report results over repeated stochastic witness-generation runs and provide per-benchmark rather than primarily pooled results? The CodeContests+ control contains only 16 mixed units, so it is difficult to assess robustness of the cross-benchmark claim from the presented aggregate alone.

7 comments
AuthorsOpenReview ↗

Response to Reviewer VcDt (1/2)

Thank you for a very detailed review. We agree with most of it and have changed the paper accordingly.

W1 / Q1 (validity). Agreed: Proposition 2 is conditional and says nothing about whether the observed eliminations are sound. The revision says this next to the proposition and points to Qwen3-4B, where 55 of 64 removed benchmark-correct programs were removed on invalid inputs. For the audit we added the three checks in the general response. On specificity, 1,237 single-constraint mutants of valid inputs, covering every stated constraint including the structural ones you mention (trees, reachability-style structure, at least one empty cell), were all rejected. On graph guarantees in particular, the reference validator for abc395_e checks that vertex N is reachable, which is how the audit classifies those eliminations as false (Section 5.2 case study). An independent second checker agrees on all 58 valid eliminating inputs, and re-execution under the official 1024 MiB limit and with 16 GiB confirms the failures for 84 of 88 programs. We corrected the audit for the four that did not reproduce. We have not asked the benchmark maintainers to review the gaps while the paper is anonymous; the released cases are meant to make that review possible.

W2 / Q2 / Q3 (program-aware construction). We agree that the original agnostic comparison does not isolate code reading, so we ran the control you describe with the three Qwen3 judges: our program-aware prompt word for word, minus the programs (only their number is stated), with the same output format, parser, decoding, validator and fallback (prompt in Appendix G). On all 348 mixed units, behind every selector the two prompts are within 1.1 points (WitnessFirst +1.1 [-2.2, 4.4] for the code-aware prompt, filter plus vote +0.6 [-3.1, 4.1]), and no interval excludes zero (Tables 22 and 23). Seeing the code does make the eliminations somewhat more precise: 117 versus 109 flawed programs removed, and 10 versus 19 benchmark-correct programs removed on inputs the reference validators reject. So we withdraw the attribution to reading code, and claim only that executing inputs at the scale of the constraints filters out programs that a judge-based search misses. On input size, replaying the same generated inputs but keeping only those of at most 300 bytes still gives a significant filter gain (+1.7 [0.3, 4.0] with the Qwen3 judges, +6.0 [1.7, 11.6] with GPT-5.5), and larger inputs add to it (Table 27); asking the model for small inputs directly gives the same picture (Table 32, details in our reply to Reviewer tEh1). The non-LLM baselines of Q3 are in part 2.

W3 / Q6 (population and uncertainty). The revision reports per-benchmark intervals and the fixed-population comparison in the main text (Table 3, Section 5.1), and per-pool intervals in Appendix E (Table 21). The caption of Figure 2 now says only what the figure shows, that every selector solves more units behind the filter, and points to these intervals. Pooled over the Qwen3 judges on LiveCodeBench, the vote's gain is +5.3 [0.7, 10.4], and the other three intervals include zero. On CodeContests+ all four point estimates are positive (+4.2 to +8.3), with wide intervals over 48 units. Across the 32 pool-by-selector cells the point estimate is positive in 21, zero in 4 and negative in 7, and the negative ones are mostly in the two Qwen2.5-Coder-7B pools. End to end over all 514 solvable (pool, task) units (Table 19), the gains shrink as Proposition 1 implies (for example with Qwen3-8B: vote 90.7%, filter plus vote 93.0%, first passer 89.5%). We also repeated the stochastic parts. The objective stage and the judges decode greedily, so the randomness is in the sampled pairwise inputs and the shared suites; we reran the vote, filter plus vote, the knockout and WitnessFirst twice more with every sampled request seeded (Table 33). WitnessFirst's gain over the knockout is stable (+8.0 to +9.5, each interval excluding zero; +8.8 [3.1, 14.7] pooled). The filter's gain behind the vote, measured as in Table 2, moves more: +5.7, +3.7 and +0.6 under the benchmarks' labels. The movement comes from the vote, not the filter. The unfiltered vote alone ranges from 211 to 224 units, and the filter removes a program on only 54 units; on the other 294 the two selectors run the same procedure, so their difference there (+6, +2, -11 units) is pure suite noise. On the 54 units the filtered vote wins by 14, 11 and 13 units, i.e. +4.0, +3.2 and +3.7 points, +3.6 [0.3, 7.5] pooled under the benchmarks' labels and +5.6 [2.6, 9.1] under the audited labels. Section 5.1 and Section 7 report this. We did resample the four LiveCodeBench pools and rerun every selector with the Qwen3 judges, and the filter gains are similar to those on the original pools (Table 29; details in our reply to Reviewer 7Yk5).

AuthorsOpenReview ↗

Response to Reviewer VcDt (2/2)

Q3 (non-LLM baselines). We ran the three kinds you list on LiveCodeBench (Table 31). For each of the 70 problems behind the Qwen3 units, we wrote an input specification from the statement (ranges, lengths, and structure such as permutations, trees and simple graphs), transcribed with model assistance; a generator that uses no LLM and never sees a candidate then emits maximum-size inputs with fixed value patterns, random maximum-size inputs with fixed seeds, and boundary inputs at the bounds of the size parameters, 12 per problem on average. The specifications accept 2,805 of the 2,806 public and hidden inputs (the one exception violates its own statement), and every generated input passes them. Followed by the first passer, the non-LLM inputs gain +7.0 [1.0, 13.9] with the Qwen3 judges, against +3.7 [0.0, 7.8] for the model-written inputs, and they are ahead under the audited labels by 4.7 [1.9, 8.2]. Their first two inputs alone match the model-written stage (+0.3 [-3.0, 3.5]). With GPT-5.5 both solve the same 62 units, and adding the model-written inputs on top changes the units solved by at most one. This goes against the model-written part of our method, and we now say so: the gain comes from executing inputs at the scale of the constraints, and what the model adds is that it needs no specification (Sections 1, 5.2 and 7). On the 12 CodeContests+ problems the same procedure removes 72 flawed and no benchmark-correct programs and is 18.8 points [0.0, 41.7] ahead of the model-written inputs; pooled over both benchmarks the non-LLM inputs gain +9.5 [3.2, 16.7]. We also tried letting the model write only the specification part, an input constructor that the same engine expands into the same configurations; tuned on the development pool and then run once on the held-out units, it did worse than our original generators (73 against 117 flawed programs removed; Table 34), mostly because the models often fail to return a usable constructor.

W4 / Q4 (filter versus fallback). Agreed, and the revision makes the split explicit (Table 24). The filter alone, followed by the first surviving public passer, gains +4.0 [0.6, 7.8] over the first passer with the Qwen3 judges and +9.5 [3.2, 16.8] with GPT-5.5, at 4.7k and 5.9k tokens per unit. With Qwen3-32B the knockout solves 65 of 116 units, the filter in front of it 70, and adding the code-only fallback 82; the code-only judge alone solves 78, and 81 behind the filter. As you note, most of the full method's gain over the knockout comes from the fallback (39 of 57 wins). Behind the filter, code-only judging at 17 calls matches the full method at 42 calls (+0.9 [-1.5, 3.3]). We therefore recommend the filter with a vote or a code-only judge, and no longer present the full pipeline as the method of choice. We also ran the two missing cells with the Qwen3 judges (Table 30). A knockout with the fallback but without the filter gains +6.3 [1.1, 11.6] over the knockout, and putting the filter in front of it, which gives the full method, adds +3.2 [0.0, 6.5] under the benchmarks' labels and +5.2 [2.3, 8.2] under the audited labels. At matched calls, with the code-only judge taking three votes per order (45 calls against 42), the full method is +0.9 [-1.8, 3.5] above filter plus code-only judging. For the fallback itself, we let the knockout behind the filter spend the fallback's budget on a longer witness search instead (47 calls per unit, more than the full method's 42). The extra search finds few more witnesses (6.4% of comparisons against 5.3%) and adds only +0.9 [-1.1, 2.7], while the full method stays +5.2 [0.0, 10.1] above it, solving 35 units it misses against 17 (Table 30). So the fallback's contribution is not a matter of budget.

W5 / Q5 (held-out attribution). The revision adds this table (Table 4 in Section 5.3; Table 16 in Appendix E covers every selector). For each judge and selector, every wrong selection gets one cause: no witness found, a wrong witnessed verdict, an order-inconsistent witnessed tie, the objective stage removing every correct passer, or a code-only verdict or tie. For the knockout, missing witnesses dominate for every judge (42/55, 46/52, 39/51, 10/14), and wrong or inconsistent witnessed verdicts make up the rest. For WitnessFirst with the small judges, most remaining errors are code-only fallback ties (40 of 51 with Qwen3-4B). The objective stage removes every correct passer on 1 to 4 units per judge (10 units in total). Each of the 56 programs removed there fails on a valid input in every re-execution, so these are gaps in the benchmarks' tests rather than false eliminations (Appendix F).

We hope the narrower claims, the new checks on the audit and the attribution table address your main concerns.

AuthorsOpenReview ↗

Follow-up for Reviewer VcDt

Thank you again for the detailed review. Since our response we have posted further updates and revised the PDF, and we list what bears on your questions in case it helps before the PDF deadline on November 19.

  • Validator specificity (W1, Q1): all 1,237 statement-derived invalid mutants are rejected, an independently written second checker agrees on 58 of 58 eliminating inputs (Table 18), and the validators and eliminating inputs are in the supplementary material.
  • Access to the programs (W2, Q2): the prompt-matched control now covers both judge families; reading the programs changes which flawed programs are removed but not the accuracy gain (Tables 22, 23, 35 and 36).
  • Non-LLM stress inputs (W2, Q3): a generator without any LLM filters at least as well on both benchmarks (Table 31).
  • Uncertainty and stability (W3, Q6): repeated runs with both judge families (Tables 33 and 39) and independently resampled pools (Tables 29, 37 and 38), including the CodeContests+ result that did not reproduce with the Qwen3 judges, which the Limitations now state.
  • Components at matched calls (W4, Q4): Table 30 with the Qwen3 judges and Table 40 with GPT-5.5. With GPT-5.5 the selectors that use the judge are all near the top, and WitnessFirst and the knockout with the fallback but no objective stage differ by -0.9 points [-6.7, 4.7]; we report this as it came out.

If any concern remains, we would be glad to address it while the PDF can still be revised. If these changes resolve some of your concerns, we would be grateful if you would take them into account in your assessment.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Summary of revisions

We have uploaded a revised PDF. Substantive changes are in blue; a few other passages were shortened without changing what they say. The main text now uses the tenth page.

Claims. The abstract, introduction and contributions now present the judge-free objective filter as the main method. They keep positive point estimates apart from intervals that exclude zero, as does the caption of Figure 2, and Section 5.1 now reports repeated runs of the sampled steps (Table 33). They also restrict "missing evidence rather than wrong judgement" to the frontier judge on the development pool (Sections 1 and 2). Proposition 2 is stated as conditional, next to the Qwen3-4B counterexample (Section 4). The title stays as submitted, since ICLR does not allow title changes during the discussion. If the paper is accepted, we would change the subtitle to Execution Evidence for Test-Time Program Selection, which matches the narrower claims.

Audit (Section 5, Appendices C and E). The reference validators reject all 1,237 statement-derived invalid mutants, an independent second checker accepts all 58 valid eliminating inputs, and every relabelled program was re-run under 512 MiB, 1,024 MiB (the official AtCoder limit) and 16 GiB. A program is now relabelled only if its failure recurs in every run. Four programs (one in abc374_c, three in abc378_d) do not meet this and no longer count, which leaves 82 of 4,645 LiveCodeBench programs (1.8%) on 12 problems, each found by at least two judges (Table 18). The audited gains move slightly (5.5 to 9.0 points for the filter; WitnessFirst over the knockout +13.9). Results under the benchmarks' labels do not change.

New analyses.

  • Table 3 (Section 5.1): filter gains per benchmark, and on a fixed set of units under both label sets.
  • Table 4 (Section 5.3): how witnessed comparisons were judged, and the main cause of every held-out knockout error. Table 16 covers all selectors, and Appendix E splits the errors by failure type.
  • Where WitnessFirst's gain comes from, and why we now recommend the filter followed by a vote or a code-only judge (Sections 5.1 and 5.3).
  • Tokens, executions and wall-clock time (Section 5.3, Table 17); end-to-end accuracy on all solvable units (Table 19); per-pool gains (Table 21); a split by contest date and the controls for contamination (Appendix E).
  • A new Appendix F: a prompt-matched control for code access, newly run with the three Qwen3 judges (Tables 22 and 23); the filter alone and the other components (Table 24); accuracy against tokens (Table 25); audited labels taken only from the other model family (Table 26); the objective stage limited to inputs of at most 300 bytes (Table 27); measured CPU time (Table 28); the units on which the filter removes every correct passer; and every selector on independently resampled LiveCodeBench pools, newly sampled and run with the three Qwen3 judges (Table 29).
  • Also in Appendix F, newly run: the components compared at matched calls with the three Qwen3 judges (Table 30); stress inputs from a generator that uses no LLM on both benchmarks, for the Qwen3 judges and GPT-5.5 (Table 31 and Appendix F); the objective stage with a prompt that asks for small inputs (Table 32); two further runs of the sampled steps (Table 33); and a fixed-taxonomy objective stage in which the model only writes an input constructor, developed on the development pool and run once on the held-out units (Table 34).

Correction. On one all-correct unit (abc393_e), one run's public filter let through a program close to the time limit that the gate labels flawed. Section 5 now says so.

Limitations (Section 7) now say that the results support executing inputs at the scale of the constraints rather than model-written inputs as such, that the vote's own suites make single-run comparisons between selectors noisy, and that the prompt-matched control and the repeated runs use the Qwen3 judges only. They also cover a bias shared by all judges that we cannot rule out, and the inconclusive date split.

The audit materials are in the supplementary material: the reference validators, the scripts and results of the mutation test and of the second checker, the eliminating inputs with the generators that wrote them, the relabelled programs, a standalone checker, and the non-LLM generator with its specifications.

AuthorsOpenReview ↗

General response to all reviewers (1/2)

We thank the reviewers for careful and constructive reviews. The reviews agree on four points, and we agree with the substance of each, so the revised PDF narrows its claims. The new numbers come from the existing run logs, CPU-only re-execution, validator tests, stress inputs from a generator that uses no LLM (Table 31), and new LLM runs with the Qwen3 judges: the prompt-matched control below, independently resampled candidate pools (Table 29), the components at matched calls (Table 30), a prompt that asks for small inputs (Table 32), two further runs of the sampled steps (Table 33), and a fixed-taxonomy objective stage (Table 34). All of them are regenerated by script.

1. The filter is the robust contribution, not the full pipeline. The paper now presents three things: a witness-level diagnosis of knockout selection, a reference-free, judge-free objective filter that composes with existing selectors, and an audit of benchmark labels. The full WitnessFirst pipeline is one combination among several. Its gain over filter plus code-only judging is +0.9 points [-1.5, 3.3] at 42 versus 17 calls, and +0.9 [-1.8, 3.5] when the code-only judge takes three votes per order and so uses about as many calls (45). We no longer claim it is the better choice. The abstract now keeps positive point estimates apart from intervals that exclude zero. Under the benchmarks' labels only the vote's gain has such an interval. Two further runs of the sampled steps give it +3.7 and +0.6, but that movement is the vote's own randomness: the filter removes a program on only 54 of the 348 units, and on the other 294 the two selectors are the same procedure with independently drawn suites (their difference there is +6, +2 and -11 units). Counted on the 54 units where it can act, the filter's gain is +4.0, +3.2 and +3.7 in the three runs, +3.6 [0.3, 7.5] pooled (+5.6 [2.6, 9.1] under the audited labels; Table 33); re-executing both selectors on one shared suite there gives the same differences. Counted the same way in the reported run, CodeT's gain also excludes zero under the benchmarks' labels (+3.4 [0.3, 7.2]), and under the audited labels all four selectors' gains do (Appendix F). The title stays as submitted, since ICLR does not allow title changes during the discussion; if the paper is accepted we would change the subtitle to Execution Evidence for Test-Time Program Selection.

2. Validator specificity, re-execution, and an independent check of the audit. Every reviewer pointed out that our reference validators were only tested for accepting valid inputs. We added three checks.

For rejection, we took valid public and hidden inputs of the 13 LiveCodeBench problems behind the audit and broke exactly one constraint at a time. The mutants cover every constraint in the statements: bounds, lengths and counts, character sets, format, and structural ones such as a tree, at least one empty cell, or M <= N. Whether a mutant is invalid follows from the statement, not from the validator. The validators rejected all 1,237 mutants in 107 families.

A second, per-problem checker that shares no code with the reference validators measures every constrained quantity of the 58 valid eliminating inputs, and accepts all 58.

We also re-ran each of the 88 relabelled programs on its eliminating inputs, twice under each of 512 MiB, 1024 MiB and 16 GiB. For context, AtCoder's limits for 12 of the problems are 2 s / 1024 MiB; our sandbox uses 6 s (already more lenient in time) and 512 MiB, and LiveCodeBench's own harness sets no memory cap. For 84 programs, some eliminating input makes every run fail under every cap (5 change from MemoryError to time-out). Four do not: one in abc374_c, and three in abc378_d that finish in 4.0 to 5.4 s with more memory, above the official 2 s but within our 6 s. We now require reproduction and no longer count those four. The corrected numbers are 82 of 4,645 benchmark-correct LiveCodeBench programs (1.8%), on 12 of 342 problems, each found by at least two judges. The audited range of filter gains moves from 5.2 to 9.3 points to 5.5 to 9.0; WitnessFirst over the knockout stays at +13.9; numbers under the benchmarks' labels do not change. Table 18 (Appendix E) has the per-problem details.

The validators, the scripts and results of the mutation test and of the second checker, the regenerated inputs and their generators, the failing programs and a standalone checker are in the supplementary material, with a sheet that lists the measured constraints of each eliminating input for comparison with its statement.

AuthorsOpenReview ↗

General response to all reviewers (2/2)

3. The diagnosis transfers, but judgement errors are real. A new held-out table in the main text (Table 4, Section 5.3; all selectors in Table 16) gives every wrong knockout selection one cause. Missing witnesses are still the main cause for every judge (Qwen3-4B 42 of 55, 8B 46 of 52, 32B 39 of 51, GPT-5.5 10 of 14), but wrong or order-inconsistent witnessed verdicts account for the rest (13, 6, 12 and 4). The revision limits "witnesses, not verdicts" to the development setting with GPT-5.5 and reports both causes elsewhere (Sections 1, 2 and 5.3).

4. Cost beyond call counts. From the logs, per mixed unit with the Qwen3 judges: vote 5.4k tokens, filter plus vote 9.8k, the 36-call vote 46.2k, code-only 12.8k, filter plus code-only 16.4k, knockout 23.9k, WitnessFirst 40.0k. The filter plus a 6-call vote matches the 36-call vote at about a fifth of its tokens. Candidate runs take 1.5 to 27 s per unit in total, and the objective stage writes 0.8 to 1.3 MB of inputs on average (Table 17, Appendix E).

Code access, with matched prompts (new runs). We ran the control several reviewers asked for, with the three Qwen3 judges on all 348 mixed units: our prompt with the programs withheld, everything else unchanged. Seeing the code makes the eliminations somewhat more precise, but accuracy does not move: behind every selector the two prompts are within 1.1 points, and no interval excludes zero (Tables 22 and 23). The gain comes from executing the inputs, not from reading the candidates.

Non-LLM stress inputs (new). For the 70 LiveCodeBench problems behind the Qwen3 units, we wrote input specifications from the statements and generated stress inputs with no LLM and without looking at any candidate: maximum-size inputs with fixed value patterns, random maximum-size inputs, and boundary inputs, 12 per problem on average (Table 31). Followed by the first passer, they gain +7.0 [1.0, 13.9] with the Qwen3 judges, against +3.7 [0.0, 7.8] for the model-written inputs; with GPT-5.5 both solve the same 62 units, and adding the model-written inputs on top changes the units solved by at most one. So the gain comes from executing inputs at the scale of the constraints, not from a model writing them. What the model contributes is that it needs no specification. The revision says so in Sections 1, 5.2 and 7. These inputs also remove 33 benchmark-correct programs, each failing in every re-execution, and 30 of them are programs our audit already relabels, which gives an LLM-free check on the audit. We did the same for the 12 CodeContests+ problems: there the non-LLM inputs remove 72 flawed programs and no benchmark-correct one and are 18.8 points [0.0, 41.7] ahead of the model-written inputs, and pooled over both benchmarks they gain +9.5 [3.2, 16.7] over the first passer (Appendix F).

What we did not do. The prompt-matched control, the matched-call runs, the small-input prompt and the repeated runs use only the Qwen3 judges, not GPT-5.5, and we did not resample the CodeContests+ pools.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.