acceptodds
Withdrawn from ICLR 2027

SmallJudge-Bench: Evaluating Correct Invariance in Evidence-Grounded Judges

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An evidence-grounded judge should retain a correct verdict when a claim is rephrased without changing its factual content or relation to the supplied evidence. Prediction stability alone misses this requirement because a judge can be consistently wrong. We introduce SmallJudge-Bench, with 2,800 individually reviewed original–edited pairs across four datasets. Claude Sonnet 4.6 exceeds a clean-trained ModernBERT-large encoder in paired correctness by 4.80 percentage points (95% confidence interval ). Yet, on originals both models answer correctly, Sonnet's edited error rate is 2.64 points higher (). Matched subsets from two editing generators show the same conditional ordering, although the interaction between generator and model remains unresolved. Label-preserving training augmentation improves paired correctness and reduces prediction changes relative to repeated-clean training. These findings distinguish overall correctness from retaining correct decisions under edits, within the evaluated configurations. The protocol reports both, with separate controls for evidence-free stability and responses to changed content.

Primary area:
datasets and benchmarks
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_B7cyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SmallJudge-Bench, a benchmark of 2,800 individually reviewed original–edited pairs across four datasets, to evaluate whether evidence-grounded judges retain correct decisions under meaning-preserving changes in presentation. It combines accuracy, paired correctness, prediction changes, and conditional error analysis. The experiments find that Sonnet achieves higher overall paired correctness, while a task-trained ModernBERT encoder makes fewer errors after edits on examples both models initially classify correctly. The paper also explores training augmentation and confidence-based routing as potential applications of the evaluation.

Critical strengths

The evaluation distinguishes prediction stability from correctness. Reporting paired correctness alongside label flips, and comparing models on examples they both initially answer correctly, helps avoid interpreting consistently incorrect predictions as robustness and provides a more informative view of model behavior.

Critical weaknesses
  • The paper does not sufficiently justify the need for a dedicated benchmark to evaluate a capability that substantially overlaps with paraphrase robustness. Why focus on this narrower capability rather than evaluate paraphrase robustness more broadly? Are there fundamental differences that warrant a separate evaluation? The paper should more clearly establish the significance of this capability and the rationale for evaluating it separately.
  • The practical implications of the experimental findings also require further discussion. The paper should explain more concretely how the conclusions drawn from this benchmark can inform real-world use and what practical value they provide.
Questions for authors
  • What substantive differences between correct invariance in evidence-grounded judgments and general paraphrase robustness justify a dedicated benchmark? Which limitations of existing robustness evaluations does this benchmark specifically address?
  • In which concrete application settings would the benchmark’s findings be useful? How should practitioners translate these findings into decisions about judge selection, training, or deployment, and what evidence supports the expected practical benefits?
Reviewer_z2CdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces SmallJudge-Bench, a set of 2,800 human-reviewed original–edited claim pairs built from 1,400 originals across SciFact, FEVER, VitaminC, and AggreFact/TRUE. The edits are verbosity and style rewrites generated by Qwen3-32B, plus a FEVER-only model-identity tag, and the evidence is never changed. The paper argues that judges should be evaluated on paired correctness (RC: correct on both the original and the edit) rather than stability alone. It compares one fine-tuned ModernBERT-large encoder with zero-shot GPT-4.1, Claude Sonnet 4.6, and Qwen3-32B. Sonnet has higher RC (+4.80 pp), but on originals both models answer correctly, it makes more errors after editing (+2.64 pp). Secondary studies add a second edit generator, evidence-free probes, a content-changing control, training augmentation, and an Encoder-to-Sonnet confidence router

Critical strengths
  1. Every edit is reviewed by two annotators with adjudication, and a separate team re-audits a sample blind to labels and model outputs. The data are small but carefully validated.

  1. The controls are well designed: evidence-free probes show that stability without grounding is possible, and the content-changing control requires the new label to be correct. These address the obvious failure modes of invariance metrics, though not the confounds in the headline comparison.

  1. Confidence intervals resample whole originals, model comparisons are paired, and the null primary result is reported honestly. This keeps the statistics sound within the paper's limited scale.
Critical weaknesses

The paper is clearly AI generated, with significant number of artefacts supporting it. Please check the last point.


The benchmark is too small for its claims: 1,400 originals, two rewrite operators, and secondary analyses on only 200--400 originals. The key finding (+2.64+2.64 pp) amounts to about 60 pairs, fewer than the $\sim$70 invalid pairs implied by the paper's own 2.5% audit failure rate.


The headline result is unstable. The RC gap falls from +4.80+4.80 pp to +1.40+1.40 pp (CI includes zero) on the matched subset with the same edits, and the pre-specified primary comparison is null. API non-determinism, unreported invalid outputs, and Encoder evidence truncation are never controlled.


The baselines are weak: one small model trained once, no MiniCheck, AlignScore, or NLI baselines on the main benchmark, and LLMs run without reasoning. The comparison is in-domain fine-tuned versus zero-shot, so it mostly measures supervision, not robustness.


The paper reads as if it was largely AI-generated from result summaries, and the authors do not seem to have direct access to their own experiments. Several basic details are reported as unknown, such as the router's confidence function (Appendix C, L726--728) and the generation seed for the second editing batch (Appendix K, L955--958). These are facts anyone who ran the experiments would know. The AI Use Statement (L510--517) also says AI was used to transcribe ``supplied result summaries'' and to help interpret the results. The writing itself is repetitive, with the same caveats restated in nearly every section. Since only a minimal code demo is released and no checkpoint, there is no way to check whether the reported numbers are correct.

Questions for authors

Check weakness section.

Reviewer_pWLUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a benchmark for LLMs as judges to evaluate the ability to retain the correct judgement across different lexical patterns of the query while the evidence is kept constant. To this end, the authors create query-pairs (original-perturbed) from five, mainly NLI, datasets. A quite comprehensive framework of metrics is also proposed that covers correctness and flip rate of the labels across original and perturbed propositions. They evaluate six models on this benchmark.

Critical strengths
  1. The evaluation framework, including the metrics, is quite neat and useful for evaluating judges.
  2. The evaluation itself is quite thorough and has provided/verified some interesting points.
Critical weaknesses
  1. The scope of the paper––correctness invariance to perturbed claims, given some evidence, of LLM judges––seems to me too narrow and a well-explored area with a substantial body of work. "Interpreting the Robustness of Neural NLP Models to Textual Perturbations." "Same Meaning, Different Wording: Evaluating Paraphrase Robustness in Financial Natural Language Inference"
  2. The evaluation does not span the smaller models and the evaluated models are a bit outdated now. The smaller/cheaper models, closed or open, are important as those are often used as cheap verifiers/evaluators. This is ironically a mismatch with the title "SmallJudge-Bench".
  3. The source datasets are too simple to be soundly evaluated in open-domain LLM-as-judge setups.
Questions for authors
  1. Perturbation of the evidence could also be an important factor. Is there any reason why it was not considered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.