acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

Abstract

Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B’s expected score above the gold-medal threshold under the official submission limit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Competitive Programming, Code Verification, Verifiable Feedback, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_snoHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies single-candidate code verification: given a programming problem and a candidate solution, a verifier predicts acceptance and provides a diagnosis and a counterexample generator.

Critical strengths

A useful evaluation setup for code verification. USACO-JUDGE includes both accepted and unaccepted programs, enabling evaluation of acceptance decisions alongside counterexample discovery.

Substantial improvement over the same backbone with the agentic scaffold. The reported increase in USACO-JUDGE verdict accuracy from 71.41% to 87.41% supports the practical effectiveness of the training recipe.

Critical weaknesses
  1. Limited novelty. [1] already uses multi-turn code execution and GRPO to generate test inputs that make two programs produce different outputs (Sections 3.1–3.4, pp. 4–6). [2] also trains a test generator by rewarding correct tests that expose bugs (Section 4.1, pp. 3–4). The main steps of the proposed method therefore closely follow existing work.

    The paper combines a verdict, diagnosis, and counterexample in �0��0�, but does not introduce a clear new training method for this combined task. For example, the diagnosis receives no reward for being correct (Section 4.2, p. 5, lines 216–221). The experiments compare different feedback fields, but do not compare the method with an existing counterexample generator that also produces an explanation (Section 6.2 and Table 2, pp. 8–9). Although the benchmark and results are useful, the method mainly combines known ideas. It offers limited new insight into how to build better code judges.

  2. The reward values are not well justified. The paper gives 0.05 for a correct rejection, 0.05 for running the generator, 0.10 for a valid input, 0.75 for exposing a program failure, and 0.05 for matching the failure type (Section 4.2 and Figure 3, pp. 4–5). The authors explain why finding a real failure should receive more reward than the earlier steps. However, they do not explain how they chose these specific values, or show whether other reasonable values would work equally well.

    The existing experiments do not answer this question. The verdict-only experiment shows that a model can learn to guess the majority class when 85% of training programs are incorrect (Appendix G.1, p. 26, lines 1384–1395). This supports rewarding actual evidence of failure, but does not explain why the reward should be split in the proposed way. The experiment in Appendix G.2 and Table 18 (pp. 26–27) changes a separate penalty for incorrectly accepting a program; it does not test the weights above. Since the reward is a key part of the method, the paper should explain how these values were chosen and test whether the results hold when they change. It should also compare against a simpler reward based mainly on whether a valid counterexample exposes a failure.

[1] Zeyao Ma et al. Scaling Agentic Verifier for Competitive Coding. 2026. arXiv:2602.04254.

[2] Qingyao Li et al. ATGen: Adversarial Reinforcement Learning for Test Case Generation. 2025. arXiv:2510.14635.

Questions for authors
  1. Does the order of ARBITER's output (v,d,g)(v,d,g) matters? UTGen [1] already uses generated tests to guide debugging, while Repair-R1 [2] trains test generation and bug repair together. Can you compare ARBITER’s joint training of verdict prediction, diagnosis, and counterexample generation (Sections 3–4, pp. 3–5) with a model trained only to generate counterexamples, then prompted to provide a verdict and explanation? With the same base model and similar training and inference budgets, does ARBITER give more accurate judgments or better repair results (Section 6.2, pp. 8–9)?
  2. How much does the reward design help? The verdict-only experiment uses training data with 85% unaccepted programs (Appendix G.1, p. 26). Does verdict-only training still fail if the two classes are balanced through sampling or reward weights? Under the same training settings, how much better is the full reward (Section 4.2 and Figure 3, pp. 4–5) than a simpler reward for producing a valid counterexample that exposes a failure?
  3. How was the length-penalty schedule chosen? Appendix C (p. 17, lines 864–866) says the penalty schedule was chosen manually from the reward curve. What rule and which data were used to decide when to turn it on? Since episodes that exceed the time limit are excluded from training updates (Section 4.2, p. 4, lines 187–191), how much does final verdict accuracy change with different penalty schedules (Section 4.3 and Figure 5, p. 5; Appendix G.3, p. 27)?

[1] Archiki Prasad et al. Learning to Generate Unit Tests for Automated Debugging (UTGen/UTDebug). 2025. arXiv:2502.01619.

[2] Haichuan Hu, Xiaochen Xie, and Quanjun Zhang. Repair-R1: Better Test Before Repair. 2025. arXiv:2507.22853.

4 comments

We would like to thank the reviewer for acknowledging (i) the usefulness of USACO-JUDGE for making the acceptance decision and finding counterexamples, and (ii) the considerable improvement from training with the same backbone and agentic scaffold.

W1: Task and methodological contribution

We would like to note that the methodological contribution of this paper is in designing training for a verifier that makes the acceptance decision regarding the candidate program and provides the rationale and an executable counterexample in case of a suspicion of any failure other than a compilation one. GRPO optimization and execution capabilities are not claimed to be novel.

Distinction from previous test generation methods. Ma et al. [1] generate test inputs that distinguish between the behaviors of pairs of candidate programs and use them for execution-based voting. ATGen [2] employs adversarial training to improve test-output correctness and attack success. Our input is a problem statement and a single candidate program of unknown correctness. The verifier is supposed to make an acceptance decision, give a rationale for it and provide an executable counterexample in case of a rejection other than because of a compilation error. The latter task additionally requires making the decision and validating the evidence of rejecting the program. The differences are discussed in Section 2.

Reward for the whole judging task. The correct program does not have any valid counterexample; therefore, the correct acceptance should be rewarded as well. In case of rejections that require a counterexample, the main reward is in exposing the failure with the executable evidence. Successful generator execution and input validity serve as the checkable signs of progress for unsuccessful attempts. Section 4.2 incorporates the attempts into the single reward scheme for the verifier.

USACO-JUDGE consists of both correct and incorrect candidates, allowing evaluating the acceptance decision as well as the ability of finding counterexamples. With the framework of the same backbone and agentic scaffold, the training raises USACO-JUDGE Verdict Accuracy from 71.41% to 87.41% and overall Hack Success from 40.61% to 62.67%. The additional verdict-only and binary outcome-reward ablations of the training objective are conducted to investigate it further; the evidence will be discussed in our response to W3, W4, and Q2.

W2: Training of the feedback interface and evaluation of diagnosis quality

We would like to note that (V, D, X) defines the feedback interface of the verifier, and the training is based on agentic GRPO. The contribution of the paper is in training this capability with verifiable verdict and counterexample rewards. The three output fields are not presented as a new optimization algorithm.

Why D does not get a correctness reward. This is the deliberate design choice. It is possible to check if a valid input causes a failure of the program through execution. Evaluating the diagnosis to be correct, complete and helpful would require the capabilities of a reasoning-capable judge, which is costly for training. Therefore, the D is requested via the prompt, and the entire trajectory is optimized via reinforcement learning without a separate scorer of diagnosis correctness. The 0.05 penalty for D exceeding 1,000 tokens in Section 4.2 is applied only for length control. The quality and helpfulness of the diagnosis are evaluated via separate audits and feedback ablations.

Section 6.1, Figure 8 and Appendix F.6 (Table 21) compare the agentic base and ARBITER on the same 400 USACO-JUDGE candidates:

Diagnosis metricAgentic Qwen3.6-35B-A3B baseARBITER
Correctness across all candidates, counting partially correct diagnoses69.2%83.2%
Correctness when partially correct diagnoses are not counted as correct57.8%73.5%
Correct causal explanation on officially unaccepted programs48.3%65.3%

Three competitive programmers evaluated 1,200 diagnoses generated for these candidates. When partially correct diagnoses are not counted as correct, the denominator is all 400 candidates. The base is already agentic and prompted to provide an explanation, thus this comparison evaluates the gain in diagnosis quality from the training directly, not only in prompting the model to give an explanation.

Does the diagnosis help repair? Table 2 adds the V-only condition that maintains the same verifier, solver, starting programs, and selection protocol, but passes V only to the solver. Adding D improves all six combinations of solver and difficulty at @8 and five combinations at @1, with one tie. For example, GPT-OSS-120B improves from 79.2 to 89.2 on Medium at @8, and from 33.1 to 39.1 on Hard. This supports the additional benefit from D for repairing over the selection and stopping based on verdict alone.

These results show that training of a verifier on verifiable verdict and counterexample rewards can improve the quality of its diagnosis and usefulness of the feedback.

Q1: Output order, alternative training and related methods

We would like to clarify that ARBITER is trained as an independent verifier that provides V, D, and X within a single agentic review. The review itself can consist of multiple rounds of reasoning and code execution.

Output order and scope. V, D, and X are three named fields in the final submit call. Before the submission, the model can analyze the code, execute tests and modify its verdict. The output order does not specify the reasoning order for the entire review. We compare the base model and ARBITER using the same submission interface in order to measure the gains from the training. The optimization of the output order is beyond the scope of this study.

Settings of UTGen [3] and Repair-R1 [4]. UTGen trains a model to generate the test inputs and expected outputs for candidate programs, and uses them for debugging and selection of candidates [3]. The applications of UTGen are relevant to Sections 6.2 and 6.3. However, we were not able to find publicly released trained UTGen weights, which prevented a fair comparison with the trained model.

Repair-R1 jointly trains test generation and program repair on code that is known to contain errors [4]. We evaluate an independent verifier on the candidates of unknown correctness. Section 6.2 fixes the Qwen3.6-35B-A3B and GPT-OSS-120B solvers to evaluate the feedback quality, while Section 6.3 selects candidates from existing ones. The direct adoption of the native workflow of Repair-R1 would change both the repair model and its training objective, thus preventing us from isolating the contribution of the verifier feedback. The joint training of the repair model is beyond the scope of this study.

Training only for counterexamples, and then prompting for verdicts and explanations. We would like to clarify the reason for the verdict supervision in the training objective. Our aim is to make the acceptance decision and search for the failure within a single agentic review for both the correct and incorrect candidates. The correct programs do not have valid counterexamples, and searching for failures within the incorrect programs can also fail. Therefore, the absence of the counterexample cannot distinguish the decision about the correctness from the suspicion of a bug, for which the evidence is not found. V, D, and X define the final feedback interface; D is requested via the prompt without a separate correctness reward. The training only for the counterexamples and obtaining V/D through the subsequent prompting is another approach, which is beyond the scope of this study.

Judgment accuracy and repair results under comparable backbones and budgets. Our experiments support two related findings. Under the same backbone and agentic scaffold, the training raises USACO-JUDGE Verdict Accuracy from 71.41% to 87.41%. With the verifier, solver, starting programs, and submission protocol fixed, V+D improves repair scores over V-only in all six combinations at @8 (Table 2). The additional evidence on diagnosis quality and repair benefits is provided in our response to W2.

W3+W4+Q2: Reward design, binary outcomes and class-balanced verdict training

We would like to clarify the reasons behind the reward allocation and additional ablations from Section 4.2. Appendix G.1 investigates the verdict-only training on the original and class-balanced data distributions; G.2 compares the full reward and binary outcome reward. The false positive penalty experiment is reported in G.3 and is not used as the evidence for the process rewards.

Reason for reward allocation. For the branches where a counterexample is required, correct rejection, successful generator execution, and input validity get 0.05, 0.05, and 0.10, respectively, for a total of 0.20. Discovery of the actual program failure earns the main reward of 0.75, and matching the failure type – 0.05. Thus, the reward distinguishes the unsuccessful attempts according to their checkable progress, rewarding most of all for the successful counterexamples. The correct acceptance gets 1.00.

Direct comparison with the simple binary reward. The new control uses only 0 or 1 as the task reward. Agreement on Accepted receives 1. For the branches where a counterexample is required, the reward is 1 if the coarse verdict classes agree, and the valid input exposes the actual failure, and 0 otherwise. The failure type predicted does not have to match the verdict returned by the hack, and there are no separate rewards for rejection, generator execution, or input validity.

With the same backbone, training data, optimization settings and 80 training steps, the results are as follows (Appendix G.2, Table 25):

Evaluation metric after 80 training stepsFull rewardBinary reward
Verdict Accuracy78.00%77.50%
Hack Success59.77%56.02%
Mean output tokens per candidate33.36K35.31K
Successful counterexamples per million output tokens11.9110.55

Using the unrounded records, the full reward improves Hack Success by 3.76 percentage points, reduces the mean output tokens by 5.52%, and creates 12.94% more successful counterexamples per million output tokens. Table 24 also shows that, over the first 80 training steps, the fraction of the groups where all 16 samples have identical raw task rewards decreases from 24.12% under the binary reward to 15.00% under the full reward, which is a relative decrease of 37.8%. The fraction is lower in every consecutive 20-step window. These results support the role of the process rewards in distinguishing the progress of unsuccessful attempts, as well as in increasing the discovery and output efficiency of the counterexamples within the training budget.

Verdict-only training after class balancing. We retain the Accepted samples and sample an equal number of unaccepted programs. Both the original-distribution and balanced variants receive a binary reward depending only on whether the coarse verdict is correct; the quality of the diagnosis and counterexample does not affect the reward. Both runs conduct 40 training steps, and are evaluated at 10, 20, 30 and 40 steps. Their results on the common evaluation set after 40 steps are provided below (Figure 4; Appendix G.1, Table 23):

Evaluation metric after 40 training stepsVerdict-only, original distributionVerdict-only, balanced distributionFull reward
Verdict Accuracy62.00%65.75%66.50%
Hack Success25.94%34.96%50.00%

Class balancing mitigates the drop in the tool usage observed under the original distribution and increases Hack Success. However, while the balanced variant approaches the Verdict Accuracy of the full reward, its Hack Success still lags behind by 15.04 percentage points. For the task of making the acceptance decision and providing the failure evidence, optimizing only verdicts does not provide the same capability of counterexample discovery.

Q3: Length-penalty scheduling and sensitivity

We would like to clarify the criteria of scheduling, and the additional comparisons provided in Appendix C, Appendix G.4 and Figure 14. We address the criteria of scheduling, evaluation results and choice of ρ below.

1. What criterion and data determine whether to enable or disable the penalty? We monitor the raw task reward, response length, abort rate and interaction depth during training. When the level of the task reward plateaus while the response length continues to increase, we resume from a nearby checkpoint at a ten-step boundary with the penalty enabled. When the continuation of length reduction is accompanied by the decreasing interaction depth and task performance, we resume with the penalty disabled. The penalty is enabled during [90, 110), [160, 180), and [230, 250); otherwise, it is disabled. This schedule is chosen manually based on these observations.

Why disable the penalty after checkpoint 109 rather than keep it enabled? Appendix G.4 reports the evaluation results of the continuous ρ=75 branch: Verdict Accuracy drops from 79.75% at checkpoint 109 to 75.75% at checkpoint 119. This supports resuming with the penalty disabled from checkpoint 109.

2. How does enabling the penalty affect the final accuracy and abort rates? Starting from the same checkpoint 89, with the identical data and all other training parameters, we compare no length penalty with the phased ρ=75 schedule at checkpoint 179 (Table 27):

Evaluation metricNo length penaltyρ=75
Verdict Accuracy78.00%82.25%
Hack Success61.28%64.29%
Aborted trajectories25.50%10.25%

The phased schedule improves Verdict Accuracy by 4.25 percentage points and Hack Success by 3.01 percentage points, and reduces the abort rate by 15.25 percentage points. The accuracy here includes all 400 candidates, with aborted evaluation trajectories being assigned the Accepted prediction, consistent with Table 27 and Appendix G.4.

Appendix D.1 also reports an alternative scoring rule that counts every incomplete trajectory as incorrect. Under this rule, the accuracy without the penalty decreases from 74.8% at checkpoint 79 to 62.5% at checkpoint 179; the phased schedule achieves 74.5% at checkpoint 179. Both scoring rules demonstrate the benefit from the length control, comparing under their definitions. The trajectories that exceed the wall-clock budget are excluded from the policy update in all branches. The experiments above vary the length-penalty setting while keeping the rule common.

3. Why ρ=75 rather than ρ=50, and why not keep the penalty enabled throughout? The threshold is a percentile of the response length among the trajectories with the highest reward in each group and is recomputed at every step. With ρ=50, a stronger compression is enforced: approximately 47% of the successful trajectories get a penalty in the compared training windows, while it is approximately 25% with ρ=75. The continuous-penalty runs are reported in Table 28 below. The metrics are aggregated over the consecutive five-step windows; response length and interaction turns are averaged over per-step medians of each five-step window.

ρTraining stepsResponse length (tokens)Interaction turnsAbort rateHack SuccessVerdict Accuracy
50[105, 110)20,0178.67.58%71.24%84.76%
50[115, 120)15,2777.02.36%69.98%84.65%
75[105, 110)21,0829.28.60%71.36%84.65%
75[115, 120)17,2748.45.43%72.26%85.11%
75[120, 125)14,6917.83.34%72.49%85.21%
75[125, 130)12,7367.22.10%70.57%85.34%
75[130, 135)10,8136.80.99%67.61%83.92%

Across the common [105, 110)→[115, 120) windows, ρ=50 decreases the response length by 23.7%, interaction turns from 8.6 to 7.0, while lowering Hack Success by 1.26 percentage points. When ρ=75 is chosen, there is an 18.1% reduction in the response length, better preservation of interaction turns, and 0.90 percentage points increase in the Hack Success metric. It supports our decision to use the milder ρ=75 configuration.

However, when ρ=75 is always used, Hack Success begins to decline in [125, 130), followed by Verdict Accuracy in [130, 135), and response length becomes shorter, with the abort rate continuing to decrease. As a result, we alternate enabling and disabling the penalty to control the abort rate and preserve the ability to discover counterexamples.

Once again, thank you very much for your detailed comments and suggestions for the necessary comparisons in order to measure the contribution of our work. We will really appreciate your reconsideration of your assessment and score based on the above responses.

References

[1] Zeyao Ma et al. Scaling Agentic Verifier for Competitive Coding. 2026.

[2] Qingyao Li et al. ATGen: Adversarial Reinforcement Learning for Test Case Generation. 2025.

[3] Archiki Prasad, Elias Stengel-Eskin, Justin Chih-Yao Chen, Zaid Khan, and Mohit Bansal. Learning to Generate Unit Tests for Automated Debugging. 2025.

[4] Haichuan Hu, Xiaochen Xie, and Quanjun Zhang. Repair-R1: Better Test Before Repair. 2025.

Reviewer_1Fk4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies single-candidate verification for competitive programming. Given a problem statement and candidate program, ARBITER uses an execution sandbox to predict a verdict, explain suspected errors, and generate counterexamples. The authors train a 35B MoE model through reinforcement learning, emphasizing rewards for valid counterexamples that expose failures. They also introduce USACO-JUDGE, containing 900 accepted and unaccepted programs across 39 problems. Evaluation covers verification, code selection, and refinement, including an IOI 2026 case study.

Critical strengths
  • S1: The benchmark evaluates both failure discovery and recognition of accepted programs. USACO-JUDGE includes human- and LLM-written programs, with official-test labels and executable validation of counterexamples. This addresses an important limitation of hacking-only benchmarks, which cannot assess whether a verifier incorrectly rejects accepted programs.

  • S2: The improvements over the tool-using base model are substantial. Training increases overall counterexample success from 40.61% to 62.67% and USACO-JUDGE verdict accuracy from 71.41% to 87.41%. These within-backbone comparisons provide meaningful evidence that training improves verification beyond simply providing execution tools.

Critical weaknesses
  • W1: Insufficiently controlled downstream comparisons. Code-selection results use regenerated candidate pools while baseline scores are taken from prior work, leaving selection difficulty potentially unmatched. Refinement experiments control official submission limits but not internal search or inference compute. Although appropriate for studying submission-limited workflows, these comparisons do not clearly separate the benefits of better feedback from additional computation.

  • W2: The contribution of diagnostic feedback is not sufficiently isolated. Although the paper provides a human audit and compares different feedback combinations, it lacks a verdict-only refinement baseline and a direct comparison of diagnosis quality against the base model. Consequently, the evidence does not clearly establish how much training improves diagnostic quality or how much diagnosis contributes beyond verdict-based selection and stopping. Improved counterexample success alone cannot establish that the accompanying explanations are more accurate or useful.

Questions for authors
  • Q1: Can the strongest selection baseline be evaluated on exactly the same candidate pools as ARBITER? Matching uniform-selection accuracy alone does not establish comparable selection difficulty.

  • Q2: What are the total solver-generation and inference budgets for each condition? Does the refinement advantage persist when these budgets are matched?

  • Q3: Can you provide a verdict-only refinement condition using the same review and stopping protocol? This would isolate the incremental benefit of the diagnosis.

2 comments

We appreciate the recognition of the reviewer regarding (i) the evaluation of both accepted and unaccepted programs by USACO-JUDGE, (ii) the human- and LLM-written candidates with official-test labels and executable counterexample validation, and (iii) the substantial gains from training over the same tool-using base model.

W1+Q1: Selection baselines on identical candidate pools

We have rerun six baselines on exactly the same candidate programs used by ARBITER: Grading RM, MBR-Exec (hard and soft), CodeT, CodeRM, and Random Generator. Following the method descriptions and using the open models specified by Ma et al. (2026), we evaluated two solvers (Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking), two benchmarks (LiveCodeBench v6 and OJBench), and two sizes of candidate pools (8 and 64). The candidate programs are reused to match the candidate pools. The relevant sections (6.3), Table 3, and Appendix E were updated accordingly.

For candidate pools of size 64, Table 3 compares ARBITER with the best baselines rerun on the same pools:

SolverBenchmarkStrongest rerun baselineBaseline scoreARBITER, 1 callGain
Qwen3-30B-A3B-ThinkingLiveCodeBench v6CodeRM78.782.6+3.9
Qwen3-30B-A3B-ThinkingOJBenchCodeT33.143.6+10.5
Qwen3-235B-A22B-ThinkingLiveCodeBench v6Random Generator81.387.7+6.4
Qwen3-235B-A22B-ThinkingOJBenchRandom Generator44.553.3+8.8

The resource check did not find any official code or weights for Agentic Verifier. We keep the published results for it in Ma et al. (2026) and refer to it with a footnote. The above gains are with respect to the six baselines rerun on our candidate pools. If we consider the published Agentic Verifier result, then the gain in the case of the 30B solver on OJBench with a pool size of 64 is 43.6 - 34.9 = 8.7 points, as expected in Section 6.3.

W1+Q2: Refinement budgets and output-token efficiency

It is important to emphasize that all conditions use the same initial pools of code and solver configurations with 128K-token budget per call. In the case of @1, Previous code and ARBITER are limited to eight internal solver revisions, whereas Online Judge can make at most one revision. The budget of additional official submissions is one for all conditions.

We compute cumulative additional output-token costs using call counts up to each trajectory's stopping point and mean output lengths measured from refinement trajectories, accounting for both solver generations and ARBITER reviews. The @1 comparison with the Previous code baseline is given below. The scores and additional output token numbers are means per problem.

SolverARBITER (V+D) scorePrevious code scoreARBITER (V+D) output tokens (K)Previous code output tokens (K)
Qwen3.6-35B-A3B53.529.8282.6288.0
GPT-OSS-120B66.657.9284.4376.0

ARBITER achieves higher scores than Previous code while using fewer cumulative output tokens for both solvers, including the cost of its reviews. Figure 9 provides full score-output token curves for ARBITER (V+D), Previous code, and Online Judge.

W2: Direct evidence for diagnosis quality and usefulness

We have added direct diagnosis audits and refinements where the feedback is controlled, analyzing diagnosis correctness and usefulness for repair.

Comparison of the quality of diagnosis before and after training. The direct comparison is presented in Section 6.1, Figure 8, and Appendix F.6 (Table 21). We compare the agentic Qwen3.6-35B-A3B base model and ARBITER on the same 400 USACO-JUDGE candidates with DeepSeek-V4-Flash-Preview as the additional reference. The results for the base and ARBITER are:

Diagnosis metricAgentic Qwen3.6-35B-A3B baseARBITER
Correctness across all 400 candidates, counting partially correct diagnoses69.2%83.2%
Correctness when partially correct diagnoses are not counted as correct57.8%73.5%
Correct causal explanation on the 265 officially unaccepted programs48.3%65.3%

Three competitive programmers evaluated 1,200 diagnoses. In the case of partially correct diagnoses not being counted, the denominator is all 400 candidates. The criterion for the correct causal explanation is strict, finding the root cause. Even when restricted to candidates for which both models provide a diagnosis, correctness rises from 74.5% to 83.3%, so the improvement is not solely due to fewer missing diagnoses.

Evaluation of the counterexamples and explanation correctness. Table 22 and the examples in Appendix F.6 show some successful counterexamples with incorrect explanations as well as the correct diagnoses without successful counterexamples. We evaluate D through independent diagnosis annotations and do not treat hack success as sufficient evidence of explanation correctness.

Additionally, there is the separate diagnosis audit of 400 ARBITER diagnoses generated from GPT-OSS-120B refinement trajectories. Among the 163 correct bug diagnoses, 82.2% find the cause and 54.0% suggest a fix (Appendix F.5; Figure 8d). These trajectories are independent of the USACO-JUDGE candidates audit above.

To assess the contribution of D beyond verdict-based selection and stopping, we conducted an additional V-only control following the same protocol. The settings and the results are given in our answer to Q3.

Q3: A verdict-only refinement control

We conducted the additional experiment and added ARBITER: V to Table 2.

In the case of ARBITER: V, we keep the same ARBITER model, review prompt, and the tool budget. We conduct the full review process. The only difference is in the feedback that the solver receives. Namely, the solver receives the problem statement, previous code, and verdict V, but not the natural language diagnosis D, counterexample generator X, review tool traces, or the online-judge score.

The experiment uses the same initial codes and the same two solvers as before with the same 8-round limit and selection protocol: @1 – the first version predicted as Accepted by ARBITER, or round 8 otherwise; @8 – the highest score among at most eight additional submissions by the same rules. See Appendix F.1 for the protocol description. Thus, V-only already includes the verdict-based selection and stopping, allowing to separate the gains from V+D.

The direct comparison in Table 2 is given below; the values are the means of scores:

SolverDifficultyV @1V+D @1V @8V+D @8
Qwen3.6-35B-A3BEasy93.095.693.6100.0
Qwen3.6-35B-A3BMedium61.966.468.569.0
Qwen3.6-35B-A3BHard11.512.319.720.1
GPT-OSS-120BEasy93.393.395.095.4
GPT-OSS-120BMedium78.085.079.289.2
GPT-OSS-120BHard29.532.833.139.1

In the case of @8, the addition of D improves scores in all three difficulty tiers for both solvers: by 6.4, 0.5, and 0.4 points for Qwen3.6-35B-A3B, and by 0.4, 10.0, and 6.0 points for GPT-OSS-120B. At @1, five settings improve and GPT-OSS-120B Easy remains the same. These results support the contribution of diagnosis text to the code repair process beyond verdict-based selection and stopping.

Thank you again for your thorough review and constructive suggestions. We would appreciate if you could reconsider your assessment and score in the light of the provided answers and evidence.

Reviewer_ugTXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARBITER, an agentic code verifier trained with RL to judge whether a candidate program is correct, diagnose errors, and generate verifiable counterexamples. The authors build a new benchmark called USACO-JUDGE and design a reward mechanism based on the validity and effectiveness of generated counterexamples. Experiments show that ARBITER improves both verdict accuracy and counterexample generation success rate, and achieves promising results in code selection, code refinement, and an IOI 2026 case study.

Critical strengths

S1: The paper proposes an agentic verification framework that combines code correctness prediction, error diagnosis, and counterexample generation. Compared with traditional verifiers that only predict whether a program is correct, ARBITER provides executable and verifiable counterexamples, making its feedback more informative and useful.

S2: The paper designs a RL reward mechanism based on the effectiveness of generated counterexamples. This encourages the model to actively find test cases that expose bugs rather than simply guessing the verdict. The experiments show clear improvements in both counterexample generation success rate and verdict accuracy.

Critical weaknesses

W1: The paper reports significant improvements over existing methods, but the inference compute budgets are not matched across different methods. Since ARBITER can perform multiple rounds of reasoning and tool calls, it is difficult to determine how much of the improvement comes from better verification ability rather than additional test-time computation.

W2: The paper mainly uses counterexample generation success rate as a proxy for the quality of natural language diagnoses. However, finding a valid counterexample does not necessarily mean that the model correctly identifies the cause of the error. Although the authors conduct a human evaluation, the relationship between counterexample generation and diagnosis accuracy still needs further analysis.

W3: USACO-JUDGE contains only 900 C++17 candidate programs from 39 USACO 2026 problems, and the ratio of accepted to unaccepted programs is manually controlled. Therefore, it is unclear how well this benchmark represents real-world code verification scenarios and whether ARBITER can generalize to other programming languages, problem distributions, and software engineering tasks.

Questions for authors

Q1: The paper mainly reports accuracy for binary verdict prediction. Although precision and recall are reported in some experiments, could the authors provide precision, recall, and F1-score for both Accepted and Unaccepted classes across the main models and benchmarks? This would give a more complete picture of the models' verification performance.

Q2: What are the main advantages of generating test cases specifically designed to break a given candidate program, compared with generating general test cases based only on the problem statement? Could the model be misled by the incorrect logic in the candidate program, which might affect the quality of generated counterexamples?

Q3: Could the authors categorize the reasons why the base model and the trained ARBITER fail to generate valid counterexamples? It would be helpful to show pie charts of the failure reason distributions for both models and compare the failure rates across categories. This could help explain which aspects of counterexample generation are improved by reinforcement learning.

Q4: Could the authors provide curves showing how verification performance changes with different inference compute budgets and compare ARBITER with other agentic verification methods under matched compute budgets? This would help evaluate the computational efficiency of ARBITER and better separate the gains from reinforcement learning and additional test-time computation.

3 comments

We thank the reviewer for acknowledging (i) the usefulness of verdicts, diagnoses and executable counterexamples; and (ii) the effectiveness of our RL reward scheme in improving verdict accuracy and counterexample success.

W1+Q4: Inference budget and verification efficiency

We want to clarify that the multi-turn verification done by ARBITER has a limit of 128K-token context window. To assess the effect of training, our primary comparison is made with the same Qwen3.6-35B-A3B backbone and agentic scaffold, enabling both the baseline model and ARBITER to perform multiple-turn reasoning and executing code.

Figure 7 checks whether allowing the baseline model to take more chances can replicate the gain of ARBITER. On USACO-JUDGE, one ARBITER sample per candidate yields 84.13% hack success, surpassing the eight samples drawn from the same baseline model with the same agentic scaffold (80.67%). The eight baseline samples consume 480.3K output tokens on average per candidate, compared to 43.3K by ARBITER, 11.1 times as much (Section 6.1). Thus, the advantage of ARBITER in this comparison stands out regardless of the lower number of samples and output tokens. The Hack@k curves with k=1–8 show the effect of inference budget on both benchmarks.

Table 6 in Appendix D.2 records the following output token metrics. The output lengths are averages per candidate in K tokens. Overall hack success covers four benchmark splits.

MethodUOJ EasyUOJ HardUSACO EasyUSACO HardOverall hack success
Agentic Qwen3.6-35B-A3B base54.164.648.070.040.61%
Agentic DeepSeek-V4-Flash-Preview43.248.927.846.357.76%
ARBITER42.046.935.046.962.67%

Training increases overall hack success by 22.06 percentage points and reduces the output tokens by 22% to 33% across the four splits over the agentic backbone. ARBITER also outperforms the agentic DeepSeek's overall hack success at the comparable scale of output tokens.

Under a budget of one additional official submission (@1), Figure 9 clearly demonstrates the advantage in token efficiency over the Previous code baseline:

SolverARBITER (V+D) scorePrevious code scoreARBITER (V+D) output tokens (K)Previous code output tokens (K)
Qwen3.6-35B-A3B53.529.8282.6288.0
GPT-OSS-120B66.657.9284.4376.0

The scores and additional output tokens are averages per problem over the 30 non-Bronze problems. The token costs include the outputs of both the solver and the verifier and account for early stopping. Thus, ARBITER gets substantially higher scores using fewer output tokens, even after considering the review cost. Appendix F.2 (Figures 12–13) provides refinement score–output-token curves for all six methods.

W2: Diagnosis correctness and its relationship to counterexample success

We would like to clarify that the success of the counterexample provides evidence of program failure, while the diagnosis accuracy is evaluated separately. We extend our analysis in Section 6.1 and Figure 8.

On the same 400 USACO-JUDGE candidates, three competitive programmers assess diagnoses from three agentic verifiers, giving 1,200 diagnoses in total. ARBITER achieves 83.2% diagnosis accuracy versus 69.2% from its Qwen3.6-35B-A3B base under a rubric that includes partially correct diagnoses. Under the stricter rubric of correctly identifying the cause of failure, the rates are 65.3% versus 48.3% on the 265 officially unaccepted programs (Appendix F.6, Table 21).

On these 265 programs, ARBITER's diagnosis accuracy under the broad rubric is 95.6% with a successful counterexample and 25.6% without a successful counterexample. When limiting to the cases where there is both a diagnosis and a counterexample generator, the numbers become 95.6% versus 47.6% (Table 22 and the analysis therein). We also record the successes of the counterexamples with wrong diagnoses. Counterexample success is associated with diagnosis correctness, but does not guarantee it.

Separately, among the 163 correct bug diagnoses in the audit of 400 diagnoses from GPT-OSS-120B refinement trajectories, 82.2% locate the cause and 54.0% suggest a fix (Appendix F.5; Figure 8d).

W3: Benchmark representativeness and generalization

We would like to clarify that USACO-JUDGE is designed to evaluate both the acceptance judgment and the discovery of failure. We use a 1:2 ratio of Accepted to Unaccepted to emphasize the harder problem of counterexample generation. The candidates come from five LLMs and five human competitive programmers, varying in the difficulty level and severity of errors (Section 5.2).

Our evaluation also includes 1,525 real submissions on UOJ-Bench (Table 1), code selection on Python candidates from LiveCodeBench v6 and C++ candidates from OJBench (Table 3; Appendix E), and refinement on IOI 2026 problems (Section 6.2). These results provide the evidence of transferability to other programming languages and problem sources in competitive programming.

The repository-level software engineering is not in the scope of our current work, but is an important direction for future research.

Q1: Precision, recall, and F1 for both verdict classes

We add the precision, recall, and F1-scores for both Accepted and Unaccepted in Appendix D.4 (Table 9). The table covers all 12 model/inference settings in Table 1 and reports the Easy and Hard splits of USACO-JUDGE separately.

For the convenience, we restate Table 9 here. All values are percentages. AC = Accepted, UA = Unaccepted, P = precision, and R = recall.

Model / inference settingEasy AC PEasy AC REasy AC F1Easy UA PEasy UA REasy UA F1Hard AC PHard AC RHard AC F1Hard UA PHard UA RHard UA F1
Qwen3.6-35B-A3B (Non-agentic)50.2550.3750.3074.9974.4474.7246.0227.7834.5570.0682.9475.94
GPT-OSS-120B (Non-agentic)44.6473.7055.6080.4854.2664.8150.3950.3250.3575.1675.1675.16
DeepSeek-V4-Flash-Preview (Non-agentic)58.8783.2668.9789.4070.7878.9453.3450.9952.1475.9777.5276.73
Qwen3.5-397B-A17B (Non-agentic)75.9881.8578.7990.5787.0488.7663.8753.0257.9378.4984.8481.54
GLM-5.2 (Non-agentic)88.5494.4491.4098.2693.8996.0281.7770.4875.7089.7885.7187.70
Kimi-K2.5 (Non-agentic)68.5096.6780.1897.8776.6785.9858.0668.5762.8882.8975.0078.75
DeepSeek-V4-Pro-Preview (Non-agentic)83.2190.7486.7895.1390.7492.8773.6175.7174.6587.6886.4387.05
Claude-Opus-4.8 (Non-agentic)91.4094.4492.9098.8495.0096.8892.2167.6278.0293.3289.7691.50
GPT-5.5 (Non-agentic)87.8896.6792.0698.2593.3395.7391.3580.4885.5790.7795.9593.29
Qwen3.6-35B-A3B (Agentic)82.9375.5679.0789.5690.5690.0654.1958.5756.2979.5668.7473.75
DeepSeek-V4-Flash-Preview (Agentic)90.00100.0094.74100.0094.4497.1463.0786.1972.8491.9473.3381.59
ARBITER (Agentic)97.3085.1090.7993.0798.7595.8283.3767.3774.5285.1993.0788.95

On Hard, training improves the Accepted and Unaccepted F1 scores over the agentic backbone by 18.23 and 15.20 percentage points, respectively. Model ranking depends on the class and difficulty: the agentic DeepSeek-V4-Flash-Preview achieves the highest Easy F1-scores, while the GPT-5.5 reaches the highest Hard F1-scores.

UOJ-Bench (Hacking) consists of only Unaccepted programs. The verdict accuracy reported in Table 1 is equivalent to the Unaccepted recall on this benchmark.

Q2: Candidate-specific tests and misleading candidate logic

We would like to clarify that the candidate program provides hypotheses to test, while the problem statement still remains the specification. Inspecting the implementation allows ARBITER to target boundary errors, overflow, or worst-case complexity that problem-only tests may miss. A successful counterexample also offers an executable case for the subsequent repair.

UOJ-Bench[1] provides the evidence for this motivation. Its Hard split includes submissions that passed the standard tests but were later broken by human-generated counterexamples. In another zero-day experiment, GPT-OSS-120B found bugs in 5.32% (269/5,060) of the sampled full-score submissions at Pass@10 (Appendix C.11 [1]). These results prove that existing official tests can miss bugs. ARBITER targets this coverage gap by generating candidate-specific counterexamples.

Table 3 compares ARBITER with the baselines for problem-only test generation, namely MBR-Exec, CodeT, CodeRM, and Random Generator, which are rerun on the same candidate pools with 512 tests per problem (Appendix E). With the 30B solver and pools of 64, one ARBITER review per candidate achieves 82.6 versus 78.7 for CodeRM on LiveCodeBench and 43.6 versus 33.1 for CodeT on OJBench, the best such baselines in these settings. The comparison evaluates the complete verification pipelines.

Incorrect candidate logic may mislead the model during its search. This is an issue tackled by our training objective: the candidate is considered as the object to test, while the problem statement determines the correctness. The main reward is obtained only by the valid input exposing an actual program failure under independent execution-based validation (Section 3).

Q3: Failure categories and improvements from reinforcement learning

We add a failure analysis in Appendix D.6, with the category counts and rates in Table 11 and the corresponding pie charts in Figure 10. We compare one agentic run of the Qwen3.6-35B-A3B base and ARBITER on the same 600 officially unaccepted USACO-JUDGE programs. We also include the agentic DeepSeek-V4-Flash-Preview as an additional baseline.

There are six failure categories mutually exclusive to each other. Percentages below use all 600 programs as the denominator:

Failure categoryQwen baseARBITER
Incorrectly predicted Accepted87 (14.5%)32 (5.3%)
Incorrectly predicted Compile Error10 (1.7%)3 (0.5%)
Missing submission or missing/malformed generator code33 (5.5%)3 (0.5%)
Generator failure, timeout, or empty output14 (2.3%)1 (0.2%)
Invalid input33 (5.5%)2 (0.3%)
Valid input that exposes no failure96 (16.0%)51 (8.5%)
All failures273 (45.5%)92 (15.3%)

It shows that training reduces the failure rate from 45.5% to 15.3%. The number of false Accepted predictions falls from 87 to 32, generator failures and invalid inputs together fall from 47 to 3, and valid inputs that expose no failure fall from 96 to 51. The improvements span verdict prediction, reliable test generation, and failure discovery.

Valid inputs that expose no failure remain ARBITER's largest failure category. While their rate over all candidates goes down from 16.0% to 8.5%, their share of all failures goes up from 35.2% to 55.4% in the pie charts due to shrinking other categories faster. It indicates that discovering counterexamples that expose the errors of the program remains an important direction for future research.

Thank you again for your thorough review and constructive suggestions. If our responses address your concerns, we would appreciate it if you could reconsider your assessment and score.

Reference

[1] Tingqiang Xu, Hangrui Zhou, Tianle Cai, Alex Gu, and Kaifeng Lyu. Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming. 2026. Section 3.2, Section 5, and Appendix C.11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.