- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies test-based selection among generated programs when expected test outputs are themselves model-generated. It argues that ranking depends on oracle behavior on disagreement inputs, where shared errors can favor incorrect candidates despite high overall oracle accuracy. DisJudge queries a reliable source on informative disagreement inputs and combines answers through calibrated likelihood scores. The paper provides theoretical analysis, code-generation experiments, and controlled simulations.
Critical strengths
- The distinction between overall oracle accuracy and its behavior on disagreement inputs is useful. The reported tendency to favor incorrect candidates in the same-model setting is the most interesting finding of the paper.
- The paper includes comparisons across several benchmarks and generators, along with ablations separating query selection from answer aggregation.
Critical weaknesses
- The central empirical finding is not documented sufficiently for reproduction. The main diagnosis uses sampled candidate pairs, but their selection and aggregation are unclear. The exact oracle models and prompts in the similarity comparison are also insufficiently specified. Since the disagreement-set bias is the strongest contribution, its measurement protocol and uncertainty need clearer treatment.
- The connection between the theory and the implemented method is incomplete. The consistency result assumes a correctly specified answer distribution, while the practical definition of query types, estimation of confusion matrices for diverse outputs, and separation of calibration and evaluation tasks remain unclear. The construction of the bounded-exhaustive reference is also underspecified. These omissions make it difficult to assess whether the reported gains rely on calibration and reference information that would be available in practice.
Questions for authors
- How were the candidate pairs in Figure 1 sampled and weighted? Please specify the oracle checkpoints and prompts, and report uncertainty for the disagreement-bias estimates with task reuse taken into account.
- How are query types and confusion matrices implemented for real program outputs? Are calibration and evaluation tasks disjoint, and how robust are the gains when calibration is transferred to new tasks?
Thank you for addressing my questions. Your response resolved some of my concerns, and I am raising my score to 3.