acceptodds
Under review as a conference paper at ICLR 2027

DisJudge: Buying Reliable Adjudication Where Candidates Disagree

Abstract

Large language models often write several candidate programs for one task, and a common way to select one of them is to count the generated tests that each candidate passes. The expected outputs of these tests come from an oracle, which is usually a model and often the one that wrote the candidates. We show that the count compares two candidates only on the inputs where they disagree, so only the accuracy of the oracle on this disagreement set matters, and its overall accuracy can hide a strong bias there. On six code benchmarks, a same-model oracle is right on of all inputs, yet on the disagreement set it returns the output of the correct candidate of the time and that of a wrong candidate of the time. We prove that for a pair of candidates the count selects the wrong one with probability tending to one whenever the oracle favors it on this set, whereas a calibrated likelihood score stays consistent at a rate set by the Hellinger distance between the answers of the oracle under the two hypotheses. Based on this analysis we propose DisJudge, which spends a small budget of reliable answers on the disagreement set and ranks the candidates by the calibrated likelihood of all answers. With reliable calls per disagreement input, DisJudge raises the mean selection accuracy over benchmark and generator settings from to , while the best of baseline rules reaches . The bias, and with it the gain of DisJudge, shrinks as the oracle shares less with the generator.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
disagreement set

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uHmgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies test-based selection among generated programs when expected test outputs are themselves model-generated. It argues that ranking depends on oracle behavior on disagreement inputs, where shared errors can favor incorrect candidates despite high overall oracle accuracy. DisJudge queries a reliable source on informative disagreement inputs and combines answers through calibrated likelihood scores. The paper provides theoretical analysis, code-generation experiments, and controlled simulations.

Critical strengths
  1. The distinction between overall oracle accuracy and its behavior on disagreement inputs is useful. The reported tendency to favor incorrect candidates in the same-model setting is the most interesting finding of the paper.
  2. The paper includes comparisons across several benchmarks and generators, along with ablations separating query selection from answer aggregation.
Critical weaknesses
  1. The central empirical finding is not documented sufficiently for reproduction. The main diagnosis uses sampled candidate pairs, but their selection and aggregation are unclear. The exact oracle models and prompts in the similarity comparison are also insufficiently specified. Since the disagreement-set bias is the strongest contribution, its measurement protocol and uncertainty need clearer treatment.
  2. The connection between the theory and the implemented method is incomplete. The consistency result assumes a correctly specified answer distribution, while the practical definition of query types, estimation of confusion matrices for diverse outputs, and separation of calibration and evaluation tasks remain unclear. The construction of the bounded-exhaustive reference is also underspecified. These omissions make it difficult to assess whether the reported gains rely on calibration and reference information that would be available in practice.
Questions for authors
  1. How were the candidate pairs in Figure 1 sampled and weighted? Please specify the oracle checkpoints and prompts, and report uncertainty for the disagreement-bias estimates with task reuse taken into account.
  2. How are query types and confusion matrices implemented for real program outputs? Are calibration and evaluation tasks disjoint, and how robust are the gains when calibration is transferred to new tasks?
1 comment

Thank you for addressing my questions. Your response resolved some of my concerns, and I am raising my score to 3.

Reviewer_R9ZsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies an important bias in execution-based code selection when the oracles are generated by LLMs. It shows that only inputs where candidate solutions disagree affect pairwise rankings, and that even an oracle with high overall accuracy can favor wrong candidate solutions on these disagreement cases. It shows that simply counting passed tests can be asymptotically inconsistent when the oracle is biased on disagreement inputs. They propose DisJudge, which uses generalized binary search to prioritize informative disagreement inputs for reliable evaluation, and combines these evaluations with oracle outputs through a calibrated likelihood model. Experiments across six code benchmarks and multiple generators show consistent improvements over 21 baselines, especially under limited query budgets.

Critical strengths
  • The paper identifies an important bias in execution-based code selection. It shows that only disagreement inputs affect candidate rankings. On six code benchmarks, a same-model oracle with 87.3% overall accuracy actually favors incorrect candidates on these inputs (51.1% vs. 37.2%). The theoretical analysis further explains why adding more tests can increase this bias rather than resolve it.

  • DisJudge provides a simple and well-motivated way to address this problem by focusing reliable queries on disagreement inputs, selecting informative queries through GBS, and combining answers using calibrated likelihoods. The evaluation is reasonably broad, covering six benchmarks, three generators, and 21 baselines. Improving average selection accuracy from 54.8% to 60.5%, compared with 57.6% for the strongest baseline, is a meaningful result.

Critical weaknesses
  • The theory assumes independent oracle errors and accurate class-conditional calibration. Since the measured bias varies considerably across candidate pairs (Fig. 1), it is unclear whether a shared confusion matrix per query type is sufficient.
  • The method depends on reliable answers and labeled data for calibration. The experiments use a bounded-exhaustive executable reference covering about 83% of disagreement inputs, but obtaining such references may be difficult for real-world tasks without known solutions.
Questions for authors
  • How sensitive is DisJudge to variations in oracle bias across candidate pairs, given that the calibration is shared within query types?
  • Can you provide more details on how the bounded-exhaustive references are constructed for different benchmarks, and whether the costs of constructing these references and obtaining calibration labels are considered? I am concerned that such reliable references may be difficult to obtain in practice, potentially limiting the real-world applicability of DisJudge.
Reviewer_k7J8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies selection among generated programs when test expected outputs come from a model. It expresses pairwise pass-count differences using only inputs where candidates disagree, measures cases where an accurate oracle nevertheless favors a wrong program on those inputs, and analyzes pass counting and calibrated likelihood ranking under a matched class-conditional response model. DisJudge selects informative disagreement inputs for a bounded-exhaustive reference and combines the reference and model answers through calibrated likelihoods. Across six benchmarks and three generators, it reports mean selection accuracy of 60.5%, versus 54.8% for pass counting. Its contribution is primarily a disagreement-conditioned diagnosis and an application of established query-selection and noise-aggregation ideas to code selection.

Critical strengths

S1. The disagreement-conditioned measurements provide a useful diagnostic.

The gap between overall oracle accuracy and its preference on candidate-disagreement inputs makes a concrete failure mode of generated-test selection visible. CodeT already studies toxic tests [R1], and broader evaluator self-preference is known [R7]; the useful increment here is the conditional margin analysis and its measurement across generator-oracle similarity levels. These results help explain why overall test accuracy can conceal selection errors (Section 3; Figure 1 and Table 1).

S2. The experimental design attempts to separate the sources of improvement.

Shared pools, matched reliable-call budgets, query-location/order ablations, and vote/hard/soft updates help distinguish design choices. The paper also reports pools without a correct candidate, reference abstention, and errors beyond the reference bound. These disclosures and the separation of controlled simulations from real-pool results are valuable, although implementation and uncertainty details below limit the conclusions (Tables 2-4; Appendices E-F).

R1. Bei Chen et al. CodeT: Code Generation with Generated Tests. https://arxiv.org/pdf/2207.10397

Critical weaknesses

W1. The incremental algorithmic and theoretical contribution over established methods is limited.

Querying inputs that distinguish candidate programs is central to oracle-guided synthesis [R2]. More importantly, Noisy GBS already combines weighted disagreement queries with Bayesian soft updates and maximum-weight hypothesis selection [R3]; confusion-model aggregation is established in Dawid-Skene and learning from crowds [R4-R5]. The paper acknowledges these origins in Section 2. Under Assumption 1, Theorem 1 reduces to L_T = log(p_star/p_wrong) M_T, with consistency and the Hellinger-form bound obtained by a likelihood-ratio/Chernoff argument. This is a useful specialization, but the manuscript does not establish a substantial theoretical advance beyond these principles. Proposition 3's hard-elimination survival probability is likewise the direct product of independent per-call survival probabilities.

The remaining increment therefore rests mainly on the new code-specific diagnosis and empirical integration. CodeT's toxic-test analysis already establishes the underlying risk of tests rewarding generated candidates while rejecting a correct reference [R1], and ALGO already uses exhaustive reference programs for code verification [R6]. The latter is missing from the supplied manuscript's references and main comparison. The conditional bias measurements remain distinct; however, a clearer account of what additional knowledge and capability the full method establishes is needed. This concern is about the size of the remaining contribution, not an allegation that the authors claim to have invented the cited components.

W2. The evidence for the practical increment depends on underspecified calibration, reference construction, and baseline implementations.

Equation (3) needs a deployable P(y|z,q) for arbitrary program outputs. Appendix C.3 reports errors from 200 labeled calibration inputs but does not sufficiently define query types, output classes, unseen-output handling, or a task-disjoint calibration split. It is unclear how the pairwise parameters become a consistent pool-level score without knowing which evaluated candidate is correct. The bounded-exhaustive reference also needs concrete per-task specifications, bounds, and construction costs; exhaustive execution alone does not establish a reference program's semantic correctness.

Table 2 includes Noisy GBS, but Appendix C.7 does not establish whether that baseline retains its Bayesian soft update or mainly uses its query rule. This distinction is central to demonstrating improvement over the closest existing framework. Exact learned-baseline checkpoints/adaptations are also missing. Finally, the 4,096 instances reuse tasks, and the bootstrap description does not establish task-clustered uncertainty. Without these details, the reported gains do not yet convincingly establish a reproducible improvement over the relevant prior methods.

  1. Chen et al. CodeT: Code Generation with Generated Tests. https://arxiv.org/abs/2207.10397

  2. Jha et al. Oracle-Guided Component-Based Program Synthesis. https://www.csl.sri.com/users/tiwari/papers/icse2010.pdf

  3. Nowak. Noisy Generalized Binary Search. https://papers.neurips.cc/paper_files/paper/2009/file/556f391937dfd4398cbac35e050a2177-Paper.pdf

  4. Dawid et al. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. https://crowdsourcing-class.org/readings/downloads/ml/EM.pdf

  5. Raykar et al. Learning From Crowds. https://www.jmlr.org/beta/papers/v11/raykar10a.html

  6. Zhang et al. ALGO: Synthesizing Algorithmic Programs with Generated Oracle Verifiers. https://www.cs.cmu.edu/~leili/pubs/zhang2023algo.pdf

  7. Panickssery et al. LLM Evaluators Recognize and Favor Their Own Generations. https://arxiv.org/abs/2404.13076

Questions for authors

What is the precise algorithmic or theoretical increment over Noisy GBS with Bayesian updating [R3] and calibrated noise aggregation [R4-R5]? Please specify the Table 2 Noisy GBS implementation, including its update, calibration, source information, and budget. How does the conditional diagnosis extend CodeT's toxic-test analysis [R1]?

Please provide the complete real-program query/output parameterization, handling of unseen outputs, and calibration/evaluation task splits. Can every candidate score be computed without evaluation correctness labels? A zero-reliable-call calibrated result on the existing pools would help isolate calibration from the value of reliable queries.

Please specify the executable reference sources, enumeration bounds, abstention rules, and construction costs across the six benchmarks, and explain the relationship to ALGO [R6]. Which evidence establishes that the reference is exact where it answers, rather than simply exhaustive but potentially based on an incorrect specification?

Please list unique tasks, provide paired task-clustered uncertainty against the closest baselines, and specify learned-baseline checkpoints/adaptations. Does the reported 138.6 ms include candidate generation, test generation, model answers, and calibration? These requests concern validation and specification of the existing study.

Comments on the paper

2 comments

Dear Reviewers,

A friendly reminder to check the authors’ responses and the other reviewers’ comments. It would be helpful to hear your thoughts on the differing assessments and whether they affect your evaluation.

Thank you for your time and effort!

AC

AuthorsOpenReview ↗

General Response 2: What the paper finds and what is new, in plain terms

Dear AC and reviewers,

Thank you for the reviews and the discussion. As the discussion closes, we restate in plain terms what the paper finds and what is new.

The problem. Code models now write both the candidate programs and the tests that choose among them, expected outputs included. When one model does both, its misunderstandings show up on both sides. A test can only change the ranking where the candidates disagree: where all candidates return the same output, the test rewards all of them or none. On these few inputs, the model's expected outputs tend to side with the wrong programs that share its misunderstanding. It is like a grader who shares a student's mistake: checking the grader on questions that every student gets right says nothing about it. In the same way, the overall accuracy of the tests cannot reveal this bias, because most test inputs are ones where all candidates agree.

What we find. On six code benchmarks, tests written by the same model are right on 87.3% of all inputs, yet where the candidates disagree they side with a wrong program more often than with the correct one, 51.1% against 37.2%. More tests do not help: from 10 to 200 tests, the usual pass count gains less than half a point. The bias shrinks as the test writer shares less with the program writer, and so does the gain of our fix.

What is new: three ideas.

  1. Look only where the candidates disagree. Inputs where they agree cannot change a ranking (Proposition 1), so both the diagnosis and every reliable check belong on the disagreement inputs.
  2. Calibrate the biased tests there; neither trust nor discard them. On the disagreement inputs, a test's agreement with a candidate can be evidence against that candidate. Calibrated the usual way, on all inputs, the classical tools learn the opposite sign and do worse than giving every candidate the same starting score; calibrated on the disagreement inputs, they work (Table R1).
  3. Spend a few reliable checks where they decide the most. DisJudge asks a reliable source about a small number of the most informative disagreement inputs, and weighs each answer by how reliable it is.

What it buys. Over 18 benchmark and generator settings, DisJudge raises selection accuracy from 54.8% to 60.5%, while the best of 21 baseline rules reaches 57.6%. It needs no human-written reference and no labels on a new benchmark: a reasoning model or a model-written brute-force script can serve as the reliable source. Placed after existing selectors such as CodeT, it improves them too.

What is classical, and what is new. How DisJudge picks a query and how it weighs answers come from classical work, generalized binary search and Dawid-Skene, and Section 2 credits it. What is new is the knowledge that makes these tools work here: where generated tests carry information, why their agreement can count against a program, and that overall accuracy hides this. In the terms of the ICLR reviewer guidelines, this is new knowledge, empirical and theoretical, with a direct use for practitioners. All three reviews single out the central finding: Reviewer uHmg calls it "the most interesting finding of the paper", Reviewer k7J8 says it "makes a concrete failure mode of generated-test selection visible", and Reviewer R9Zs finds DisJudge "a simple and well-motivated way to address this problem".

We thank the AC and the three reviewers again for their time and careful reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.