- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SmallJudge-Bench, a benchmark of 2,800 individually reviewed original–edited pairs across four datasets, to evaluate whether evidence-grounded judges retain correct decisions under meaning-preserving changes in presentation. It combines accuracy, paired correctness, prediction changes, and conditional error analysis. The experiments find that Sonnet achieves higher overall paired correctness, while a task-trained ModernBERT encoder makes fewer errors after edits on examples both models initially classify correctly. The paper also explores training augmentation and confidence-based routing as potential applications of the evaluation.
Critical strengths
The evaluation distinguishes prediction stability from correctness. Reporting paired correctness alongside label flips, and comparing models on examples they both initially answer correctly, helps avoid interpreting consistently incorrect predictions as robustness and provides a more informative view of model behavior.
Critical weaknesses
- The paper does not sufficiently justify the need for a dedicated benchmark to evaluate a capability that substantially overlaps with paraphrase robustness. Why focus on this narrower capability rather than evaluate paraphrase robustness more broadly? Are there fundamental differences that warrant a separate evaluation? The paper should more clearly establish the significance of this capability and the rationale for evaluating it separately.
- The practical implications of the experimental findings also require further discussion. The paper should explain more concretely how the conclusions drawn from this benchmark can inform real-world use and what practical value they provide.
Questions for authors
- What substantive differences between correct invariance in evidence-grounded judgments and general paraphrase robustness justify a dedicated benchmark? Which limitations of existing robustness evaluations does this benchmark specifically address?
- In which concrete application settings would the benchmark’s findings be useful? How should practitioners translate these findings into decisions about judge selection, training, or deployment, and what evidence supports the expected practical benefits?