- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether failures to select a correct candidate program arise from incorrect judgment or missing discriminative evidence. It distinguishes objective witnesses settled by crashes/timeouts from subjective witnesses requiring a model to judge outputs. WITNESS FIRST generates inputs near constraint limits and validators, eliminates candidates through execution, and then uses small-input comparisons with a code-judgment fallback. Evaluation covers LiveCodeBench, CodeContests+, eight candidate pools, and four judges. The objective stage improves multiple selectors, but the full method is not best for every judge. The audit also finds that some benchmark-passing programs fail under inputs accepted by the authors' validators and their resource settings.
Critical strengths
- The central distinction is explanatory and reusable. Separating large inputs settled by execution from small inputs requiring reading directly targets scale-related errors. The objective filter composes with several selectors, providing broader practical value than a single monolithic pipeline.
- Reporting and statistics are transparent. Both original and audited labels are reported, with problem-clustered intervals and paired analyses. The paper discloses the full method's disadvantage with the strongest judge, favorable results after removing the validator, and protocol changes, and describes plans to release traces and prompts.
Critical weaknesses
- The diagnosis and incremental value of the full method require narrower claims. GPT-5.5's 60/60 correct witnessed verdicts on development data do not establish that all judges merely lack witnesses. Held-out Qwen3-32B results include 10 incorrect witnessed judgments, and other models produce many ties. This limits empirical applicability of the lemma's premise without refuting the lemma itself. The full method's gain over objective-stage-plus-code-judgment has an interval spanning zero while requiring 42 versus 17 calls. Of 57 extra successes over knockout, 39 come from code fallback. Current evidence more strongly supports a composable objective filter than a generally superior complete pipeline. Gains from large-scale inputs also need to be separated from gains from adding execution filtering.
- Audited labels and efficiency claims rely on insufficiently validated conditions. Reference validators are mainly checked for accepting valid inputs, without independent evaluation of false acceptance on invalid inputs. Audited labels are related to similar witness construction, so shared bias may remain despite leave-one-judge-out labels. The audit also changes some mixed-task sets, preventing strict same-population comparison of gains across label sets. The relationship between the 6-second/512 MiB environment and original contest limits is unclear, while call counts omit large-input generation and repeated execution costs. Independent validator checks, fixed comparison sets, and explicit resources and costs are needed to support the benchmark-gap and efficiency claims. These addressable evidence gaps lead me to lean toward rejection.
Questions for authors
- Diagnostic scope: Please separate correct, incorrect, and tied witnessed verdicts by judge. Which filters produced the development-set 60/60 result, and will the claim about missing evidence rather than wrong judgment be restricted to supported conditions?
- Validators: Is there an independently human-labeled set of valid and invalid inputs for evaluation in both directions? Inputs labeled invalid by the same reference validator cannot independently establish its false-accept rate; please explain the actual audit ground truth.
- Audit comparison: Please report official resource limits for the 13 LiveCodeBench problems and compare both label sets on the same task set, distinguishing test-coverage gaps from different resource standards.
- Incremental value and cost: Using existing logs, please report tokens, execution counts, and runtime for the complete method, objective-filter-plus-code-judgment, and objective-filter-plus-vote. Which differences have paired statistical support?
- Mechanism: Are small-input-only objective-filter results or program-aware ablations with a fixed generator taxonomy already available? If not, please narrow attribution to input scale and the complete pipeline and describe existing controls for contamination on public problems.
4 comments
Response to Reviewer ZEss (2/2)
Q4 (incremental value and cost). Per mixed unit with the Qwen3 judges, WitnessFirst uses 40.0k tokens and 55 candidate executions, filter plus code-only judging 16.4k and 44, and filter plus vote 9.8k and 178. Their candidate runs take 10.4, 9.9 and 17.7 s in total. The paired differences are WitnessFirst over filter plus code-only judging +0.9 [-1.5, 3.3], and WitnessFirst against filter plus vote -2.3 [-8.5, 4.3]; neither is supported. At matched calls, with the code-only judge taking three votes per order (45 against 42 calls), WitnessFirst is +0.9 [-1.8, 3.5] above filter plus code-only judging (Table 30). Against tokens, the Qwen3 frontier consists of the first passer, the vote and CodeT, each alone and behind the filter, while with GPT-5.5 WitnessFirst solves 18.1 points [11.1, 25.4] more than the filter plus vote (Table 25). The differences with paired support are the filter's gains over all four selectors under the audited labels and the full pipeline's gain over the knockout (+9.5 [3.3, 15.6], and +8.8 [3.1, 14.7] pooled over three runs, Table 33). The filter's gain over the unfiltered vote under the benchmarks' labels also holds over three runs when counted on the 54 units where the filter removes a program (+3.6 [0.3, 7.5] pooled, Table 33); on the other units the two selectors differ only by their suites. The revision recommends the filter with a vote or a code-only judge.
Q5 (mechanism and contamination). Two new controls address the mechanism. A prompt-matched run with the Qwen3 judges that withholds the programs changes no selector by more than 1.1 points (Tables 22 and 23), so reading code is not what drives the gain. Replaying the objective stage with only inputs of at most 300 bytes keeps a significant gain (+1.7 [0.3, 4.0] with the Qwen3 judges, +6.0 [1.7, 11.6] with GPT-5.5), and larger inputs add to it (Table 27). Prompting for inputs of at most 300 bytes directly agrees: +3.7 [0.9, 6.8] behind the vote, with the unrestricted prompt adding +3.7 to +4.3 under the audited labels (Table 32). We also ran a fixed-taxonomy baseline without any LLM: maximum-size, random maximum-size and boundary inputs from specifications transcribed from the statements. On both benchmarks it filters at least as well as the model-written generators (on LiveCodeBench +7.0 [1.0, 13.9] against +3.7 [0.0, 7.8] over the first passer with the Qwen3 judges, Table 31; on CodeContests+ 18.8 points [0.0, 41.7] ahead, Appendix F), so we narrow the attribution as you suggest, to executing inputs at the scale of the constraints. A program-agnostic variant with that fixed taxonomy, where the model writes only an input constructor, removed more flawed programs than our generators on the development pool but fewer on the held-out units (73 against 117), and we report it as it came out (Table 34). On contamination, every held-out problem predates the Qwen3 release (29 April 2025; the latest LiveCodeBench problem is from 6 April 2025), and we found no public source of later contest problems with official hidden tests: the latest LiveCodeBench release ends in April 2025, and AtCoder has suspended publishing its test cases. So we describe the controls we have (Appendix E). The objective stage removes a program only when it crashes or times out on an input the validator accepts, which no judge decides; in front of the first passer, which consults no judge either, it gains +4.0 [0.6, 7.8] with the Qwen3 judges (Table 24). A model that remembers a problem might write better inputs for it, but inputs from a generator without any LLM gain +7.0 [1.0, 13.9] over the first passer (Table 31). Contamination can still favour the judges' verdicts, and with them code-only judging. A split by contest date (before and after 2025-01-01) gives the vote's filter gain +5.9 and +4.8, but first-passer accuracy drops as well (58.8% to 51.0%, code-only 62.1% to 57.8%), so it cannot separate contamination from difficulty. Section 7 states this as a limitation.
Thank you again; your questions led directly to the corrected audit.
Follow-up for Reviewer ZEss
Thank you again for the review. Since our response we have posted further updates and revised the PDF; for your questions:
- Diagnostic scope (Q1): verdicts split by judge into correct, incorrect and tied, and the claim restricted to the supported conditions (Tables 4 and 16).
- Labels (Q3): both label sets on the same fixed units (Table 3) and their relation to the official limits (Table 18).
- Full method against the filter and the code-only judge at matched calls (Q4): Table 30 with the Qwen3 judges and Table 40 with GPT-5.5, where all selectors that use the judge are near the top.
- Contamination (Q5): judges whose weights predate every held-out LiveCodeBench problem (Qwen2.5-7B, 14B and 32B) show the gains of the code-only judge and of WitnessFirst over the knockout (8.3 points [1.4, 14.7]) as well, and smaller gains of the objective stage (Table 41), together with the generator without any LLM (Table 31).
If anything remains, we would be glad to address it before the PDF deadline on November 19. If these changes resolve some of your concerns, we would be grateful if you would take them into account in your assessment.
Are any concerns left? (Reviewer ZEss)
Your review closes with "These addressable evidence gaps lead me to lean toward rejection." We believe each gap is now closed and would like to know whether anything still keeps your rating at weak rejection.
- Narrower claims (W1, Q1): the diagnosis is restricted to the conditions that support it, with correct, incorrect and tied witnessed verdicts per judge (Tables 4 and 16), and the paper now presents the composable filter, not the full pipeline, as the method.
- Large inputs versus execution filtering (W1, Q5): inputs of at most 300 bytes and a prompt for small inputs, with both judge families (Tables 27, 32 and 40), and a fixed-taxonomy variant (Table 34).
- Independent validator checks (W2, Q2): the mutation test takes its ground truth from the statement, not from our validators: each of the 1,237 mutants violates exactly one stated constraint, and all are rejected. A second checker that shares no code with the validators agrees on all 58 valid eliminating inputs (Table 18). We have not built a human-labelled set; these two tests are our answer to the false-accept question.
- Fixed comparison sets (W2, Q3): both label sets on the same units in Table 3, and the official limits for the relabelled problems in Table 18.
- Resources and costs (W2, Q4): Tables 17, 25 and 28.
- Contamination (Q5): judges whose weights predate every held-out LiveCodeBench problem keep the gains of WitnessFirst over the knockout, 8.3 points [1.4, 14.7] (Table 41).
If these close the gaps you named, we would be grateful if you would consider raising your rating. If something remains, please tell us and we will address it before the PDF deadline on November 19.
Response to Reviewer ZEss (1/2)
Thank you for the precise questions.
Q1 (diagnostic scope). Witnessed discriminative comparisons by judge (correct / wrong / tie): Qwen3-4B 0 / 1 / 31, Qwen3-8B 13 / 3 / 15, Qwen3-32B 24 / 10 / 13, GPT-5.5 187 / 5 / 0. The development-set 60/60 needed no extra filter: GPT-5.5 decided every witnessed discriminative comparison with both orders agreeing, and there were no witnessed ties. We now restrict "missing evidence rather than wrong judgement" to that setting. A new held-out attribution table shows that missing witnesses still cause most knockout errors for every judge (42/55, 46/52, 39/51, 10/14), with wrong or inconsistent witnessed verdicts behind the rest.
Q2 (validators in both directions). No human-labelled set of valid and invalid inputs existed for these problems, so the ground truth we added comes from the statements rather than from the validator. We mutate valid public and hidden inputs so that exactly one stated constraint is violated, which gives 1,237 mutants over 107 constraint families, structural ones included; the reference validators reject all of them. Separately, a second checker that shares no code with them measures each constrained quantity on the 58 valid eliminating inputs and agrees with the reference validators on all 58. The validators, and the scripts and results of the mutation test and of the second checker, are in the supplementary material.
Q3 (resource limits and a fixed task set). All 12 AtCoder problems in the audit have official limits of 2 s / 1024 MiB; Codeforces 1582G has 2 s / 512 MB; LeetCode 3612 publishes none. Our sandbox is more lenient in time (6 s, matching LiveCodeBench's harness) and stricter in memory (a 512 MiB address space, where LiveCodeBench sets no cap). We re-ran all 88 relabelled programs on their eliminating inputs under 512 MiB, 1024 MiB and 16 GiB. For 84, some eliminating input makes every run fail under every cap, so the gaps do not come from our memory limit. The four programs whose failures did not recur in every run are removed (now 82 programs on 12 problems). On the fixed official set of mixed units, the audited gains of the filter are +7.5 (vote), +6.0 (CodeT), +5.2 (code-only) and +5.5 (knockout), all with intervals that exclude zero.