- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SpecialJudgeBench, a benchmark of 270 constructive, interactive, and communication problems with 4,540 candidate programs. It evaluates two capabilities: predicting candidate verdicts and scores, and synthesizing complete executable judge repositories. Experiments reveal persistent errors in direct judgments and a substantial gap between generated repositories’ execution completeness and their ability to correctly judge all hidden candidates
Critical strengths
-
Interesting evaluation setting. The paper extends code evaluation from solution generation to candidate judgment and complete judge-repository synthesis. Covering constructive, interactive, and communication problems provides a useful setting for studying specification reasoning and protocol implementation beyond simple output matching.
-
Informative empirical observations. The results distinguish execution completeness from judging correctness, showing that repositories returning valid verdicts can still make substantial judgment errors. Analyses of false acceptance, false rejection, and score underestimation provide additional insights beyond aggregate accuracy
Critical weaknesses
The benchmark’s incremental value needs stronger justification. Evaluating generated verifiers is a relevant goal, but the paper does not clearly establish what additional capabilities full-repository synthesis reveals beyond checker/interactor generation within a trusted runtime. A controlled comparison would help distinguish special-judging challenges from general system-integration difficulty. The practical motivation for predicting verdicts and scores without execution could also be better developed.
Questions for authors
See the weaknesses
Dear Reviewers,
The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.
The discussion period ends on November 18.
Thanks,
Area Chair