acceptodds
Under review as a conference paper at ICLR 2027

From Solving to Judging: SpecialJudgeBench for Verdict Prediction and Executable Judge Repository Synthesis

Abstract

Competitive programming benchmarks typically evaluate programs generated by large language models (LLMs) using existing judges. Yet it remains unclear whether LLMs can perform the judging itself, especially for constructive, interactive, and communication problems whose correctness depends on constraints, protocols, and information isolation rather than output matching. To study this gap, we introduce SpecialJudgeBench, comprising 270 problems and 4,540 candidate programs, 903 of which have numerical reference scores. The benchmark assesses two complementary capabilities: predicting reference verdicts and scores for candidate programs, and synthesizing complete executable judge repositories from public problem specifications. We evaluate generated repositories on hidden candidate programs through binary verdict agreement, alongside build, protocol, and runtime checks. Across seven models, even the best verdict accuracy stays below 90% for each problem type; all models have higher false-rejection than false-acceptance rates, and their valid score predictions underestimate reference scores on average. Across eight model–scaffold configurations, execution completion reaches 90.9%–98.8%, but only 42.2%–55.9% of repositories correctly judge every hidden candidate in their evaluation pools. Whether generative models directly judge candidate programs or are tasked with building reliable, complex judging systems, achieving trustworthy evaluation remains a substantial challenge.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Competitive Programming, Automated Program Evaluation, Agentic Coding, Program Verification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v4GyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SpecialJudgeBench, a benchmark of 270 constructive, interactive, and communication problems with 4,540 candidate programs. It evaluates two capabilities: predicting candidate verdicts and scores, and synthesizing complete executable judge repositories. Experiments reveal persistent errors in direct judgments and a substantial gap between generated repositories’ execution completeness and their ability to correctly judge all hidden candidates

Critical strengths
  1. Interesting evaluation setting. The paper extends code evaluation from solution generation to candidate judgment and complete judge-repository synthesis. Covering constructive, interactive, and communication problems provides a useful setting for studying specification reasoning and protocol implementation beyond simple output matching.

  2. Informative empirical observations. The results distinguish execution completeness from judging correctness, showing that repositories returning valid verdicts can still make substantial judgment errors. Analyses of false acceptance, false rejection, and score underestimation provide additional insights beyond aggregate accuracy

Critical weaknesses

The benchmark’s incremental value needs stronger justification. Evaluating generated verifiers is a relevant goal, but the paper does not clearly establish what additional capabilities full-repository synthesis reveals beyond checker/interactor generation within a trusted runtime. A controlled comparison would help distinguish special-judging challenges from general system-integration difficulty. The practical motivation for predicting verdicts and scores without execution could also be better developed.

Questions for authors

See the weaknesses

Reviewer_ppGpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpecialJudgeBench, a benchmark for evaluating LLMs' ability to judge competitive programming solutions rather than merely generate them. It contains 270 problems and 4,540 candidate programs across constructive, interactive, and communication tasks.

The benchmark evaluates two capabilities: predicting program verdicts and numerical scores, and generating complete executable judge repositories. Experiments across seven prediction models and eight model–scaffold configurations show that direct judgment remains imperfect and that generated repositories often fail to consistently reproduce reference verdicts.

Critical strengths
  1. Meaningful benchmark contribution. The paper addresses an important and relatively underexplored capability: constructing and applying program judges for tasks beyond exact-output matching. Covering constructive, interactive, and communication problems adds practical value.

  2. Comprehensive evaluation dimensions. The benchmark distinguishes verdict accuracy, scoring errors, execution completeness, and repository-level correctness. This helps identify limitations that conventional code-generation benchmarks may overlook.

Critical weaknesses
  1. Insufficient validation of judge correctness. Repository correctness is measured against a finite pool of 4,540 candidate programs across 270 problems, averaging fewer than 17 candidates per problem. This coverage may be inadequate for detecting subtle implementation errors, particularly in interactive protocols and communication constraints. Without systematic adversarial or mutation-based testing, agreement with the reference verdicts provides limited evidence of general judging reliability.

  2. Limited experimental comparisons and diagnostic analysis. Repository synthesis evaluates only two models with four agent scaffolds and a single run per configuration. Table 5 reports aggregate results without separating constructive, interactive, and communication tasks or analyzing specific failure categories. Furthermore, no direct comparison with existing judge-generation approaches is provided. These omissions make it difficult to identify the main technical bottlenecks or assess the benchmark's discriminative value.

Questions for authors
  1. Can the authors evaluate generated repositories using additional adversarial or mutation-based candidates, particularly those targeting protocol violations, boundary cases, and communication isolation? How sensitive are repository exact-match scores to candidate-pool size and composition?

  2. Can the authors provide per-category results, a breakdown of semantic versus runtime/protocol failures, and comparisons with existing judge-generation methods? Repeated runs would also help establish whether the reported differences are robust.

Reviewer_TnYxOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpecialJudgeBench to evaluate two capabilities on constructive, interactive, and communication problems: directly predicting candidate verdicts and scores, and synthesizing complete executable judge repositories from public statements and interfaces. The benchmark contains 270 problems and 4,540 candidate programs, including 903 with numerical reference scores. Experiments cover seven direct-prediction model configurations and eight repository-generation configurations combining two models with four agent scaffolds. The results show substantial errors in direct prediction and a gap between successful execution and correct judgment in generated repositories.

Critical strengths

The paper introduces a practically relevant benchmark that brings program judgment and judge-repository synthesis across constructive, interactive, and communication settings into a unified evaluation. The substantial data collection effort and experiments across multiple models and agent scaffolds provide a useful resource and initial results for studying automated judging systems.

Critical weaknesses
  1. The additional diagnostic value of complete repository evaluation lacks empirical support. AutoCode already generates and validates checkers and interactors (Sections 3.3–3.4), ScaleBox evaluates synthesized special judges on correct and incorrect submissions (Section 5), and InteractBench validates generated interactive evaluation components against submissions with reference verdicts during benchmark construction (Section 3.3). This paper makes complete repository synthesis an evaluation task, includes communication settings, and restricts the reference information available during generation. These are meaningful distinctions. However, the experiments do not sufficiently establish which additional difficulties arise when models assume responsibility for the complete system, or how these differ from existing test- and judging-component generation problems. Analysis of actual generated repositories or a limited controlled comparison would help substantiate the additional diagnostic value of this extension.

  2. The experimental analysis does not adequately explain the main performance gaps. The paper analyzes false rejection, false acceptance, and score bias, but the repository-synthesis experiments primarily report aggregate EC, PA, RE, and resource usage. Without breakdowns by problem type and failure cause, it is difficult to determine whether errors arise from inadequate test coverage, judging logic, interaction protocols, communication constraints, or general runtime implementation. The results therefore establish that the tasks remain challenging, but offer limited insight into specific model limitations or the additional diagnostic value of this benchmark.

Questions for authors
  1. Please report repository-synthesis EC, PA, and RE separately for constructive, interactive, and communication problems. Together with candidate-pool sizes and representative failure cases, what do these results reveal about which types of judging programs models handle better and their principal difficulties?

  2. Relative to the generation and validation of judging components in work such as AutoCode and ScaleBox, what additional failure modes does complete repository evaluation expose? Could the authors provide cases from actual generated repositories and their frequencies, distinguishing errors in test generation or judging logic within individual components from errors in component coordination, execution management, and information isolation? Where feasible, a trusted-runtime control on a representative subset could further clarify the impact of these additional system responsibilities.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.