acceptodds
Under review as a conference paper at ICLR 2027

And a Good Judge Too?: Methodological Lessons for Auditing LLM Judges in Rubric-Based Evaluation

Abstract

LLM judges have become core infrastructure in automated AI evaluation, yet their reliability remains underexamined: in our rapid review of 37 recent ACM benchmarks using LLM judge scoring, over 40% report little or no judge analysis. We audit and extend L2-Bench, a rubric-based benchmark for second language learning design. This is a "hard'" domain marked by expert disagreement, where ground truth is contested and LLM judges are most likely to be deployed. Guided by a recent judge-failure taxonomy, we show that L2-Bench's production judge (Claude Sonnet 4.6) is statistically stable, but most importantly we demonstrate that stability alone cannot establish validity because of rater label skew—a problem common to evaluation in “hard” domains. To anchor judge scores to human expertise, we import rater-severity modeling from educational measurement, with a GLMM attributing 10.1% of latent-scale variance to rater severity, which we propagate to provide strong evidence of convergent validity ( vs. human ceiling, overlapping 95% CIs). We argue that valid judge claims in hard domains require both kinds of evidence, and propose the Minimum Viable Judge Assessment (MVJA), a tiered reporting template for standardizing judge analysis across the field.

Primary area:
datasets and benchmarks
Keywords:
LLM judges, evaluation design, AIED, rubric-based evaluation, judge stability, judge bias, convergent validity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hRxjOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This article conducts a methodological audit of the LLM judge of the second language education evaluation benchmark L2-Bench, focusing on distinguishing between judgment stability, human consistency and measurement validity. The author compared 32 judge configurations and conducted ten repeated scoring experiments on 44 configurations; then analyzed how label skew, consensus label filtering, and rater differences affect the consistency index. The paper uses 17,093 standard-level ratings provided by 142 practitioners, fits a logistic GLMM containing random intercepts for raters, tasks, and rating units, and then corrects rater severity through a latent residual re-thresholding rule. On the 169 units to which judge was comparable, the human-corrected leave-one-out κ was 0.788, producing a judge-corrected expert panel κ of 0.861. The authors interpreted this as evidence compatible with convergent validity and proposed the Minimum Viable Judge Assessment (MVJA), which recommends the disclosure of information such as label distribution, content-blind baseline, human referents, stability, and rater-level data.

Critical strengths
  1. Content-blind baseline and rater-level analyses reveal issues easily overlooked in real-world evaluations. As shown in Table 1, the "Naive baseline"—which achieves 96.3% accuracy and a kappa of 0.842 without even reading the responses—outperforms all model configurations. The authors further note that 76 of the 85 "gold Fail" cases involved penalty criteria that were not triggered, while rater disagreement persisted in only 10 of the 650 "gold" units. These specific results suggest that high agreement may primarily reflect the rubric's structure and consensus-based filtering rather than an ability to discriminate based on response content. This finding has direct implications for the design and auditing of rubric-based evaluations.
  2. The audit process is comprehensive and transparent, translating results into actionable recommendations. The study integrates multiple models, prompts, repeated scoring, rater variance analysis, and qualitative commentary, going beyond the mere reporting of judge-human correlations. The appendix explicitly acknowledges limitations such as the use of a single response-generation model, a non-randomly expanded sample, confidence intervals that fail to account for clustering, and alternative correction rules that may over-eliminate variance. The recommendations within the MVJA framework regarding content-blind baselines, configuration disclosure, and rater-level data hold practical value, independent of the study's ultimate conclusions on validity.
Critical weaknesses
  1. The core conclusion remains unsubstantiated. The procedure fails to demonstrate an ability to distinguish between "genuine judgment" and "taking shortcuts": the naive baseline serves as the strongest counter-example yet was omitted from the comparison in Table 16 (Frame B), effectively missing the most critical control group. Furthermore, incremental gains relative to the baseline were not fully reported regarding cases of genuine disagreement, failures of positive criteria, or triggered penalty items; thus, it remains unclear whether the results represent "genuine improvement" or merely the exploitation of heuristics. A κ of 0.861 may still stem partly from the predictable structure of the rubric, so one cannot conclude that the original validation issue has been resolved; the assertion in Section 3.5 that future improvements must rely solely on data and rubrics significantly overreaches the available evidence.
  2. The human reference and consensus benchmarks are problematic. Agreement between a single human and the consensus serves as a meaningful reference point rather than a theoretical upper bound—a model that consistently predicts the majority opinion can align more closely with the consensus than a random individual without necessarily being more accurate. A distinction must be made between the goals of "simulating the distribution of typical experts" and "predicting the consensus of an expert panel"; overlapping confidence intervals cannot substitute for paired difference tests or non-inferiority/equivalence tests. Additionally, there is a mismatch in evidence: the human reference in Table 2/Figure 2 derives from Frame A while the "judge" comes from Frame B; the appendix characterizes this as merely an illustrative comparison, yet the main text emphasizes that all six categories met the standards.
  3. The severity adjustment is flawed. First, the directionality is inconsistent: the text includes both positive criteria and negative penalty items. If the GLMM utilizes raw judgments without normalizing directionality, a higher propensity for "true" ratings cannot simultaneously represent leniency across both types of criteria; the claim that "10.1% of the variance stems from severity" requires re-examination. Second, even assuming correct directionality, rater effects are not synonymous with threshold noise that should be eliminated: a single intercept might conflate differences arising from professional domains, standard categories, and stances, yet the appendix fails to present data on distribution overlaps or model diagnostics. Third, deriving scores by inverting conditional latent residual means constitutes a counterfactual reconstruction dependent on model assumptions rather than a direct observation. Fourth, the CI does not specify whether uncertainty is propagated, nor does it set aside data for validation; the claim that "approximately one-third of the advantage stems from inconsistent human thresholds" has not yet been independently verified.
Questions for authors
  1. Did the naive baseline pass final validation? Please report the raw and corrected κ, accuracy, and paired difference for the naive baseline using the exact same Frame B units and sub-panels as the judge. Also, report results for positive criteria, penalty terms, and units with rating disagreements. If this baseline also meets or exceeds human benchmarks, how does MVJA avoid incorrectly classifying it as possessing valid judgment capabilities?
  2. How does the severity model handle label directionality and heterogeneity? Have the binary labels in Equation (1) been standardized—via positive/negative weighting—to align with "favoring answer quality"? If not, please explain how a single rater intercept can still be interpreted as severity. Are the variance proportions, number of flips, and final κ values ​​stable when using standardized directions, separate modeling, or rater-by-criterion-type interactions?
  3. How is corrected uncertainty estimated? Does bootstrapping involve refitting the GLMM and re-performing the correction, or are the estimated intercepts and corrected labels treated as fixed? How are dependencies arising from multiple criteria within a task and a single rater scoring across units handled? Does the "approximately one-third" conclusion hold when estimating rater effects via held-out tasks or using an analysis that fully propagates uncertainty?
  4. What type of "human parity" does the paper aim to validate? Is the goal to simulate a random expert or to predict expert consensus? Please provide paired judge–human differences and associated uncertainties within the same Frame B; if claiming the system can substitute for experts, please provide appropriate statistical criteria. Category-specific analyses should also use matched samples rather than directly comparing κ values ​​between Frame A and Frame B.
  5. Is the audit configuration consistent with the actual production configuration? Table 7 notes that the Bedrock experiment disabled "extended thinking," whereas the production system utilizes a 1,024-token thinking budget. Please clarify which results correspond to the actual deployment configuration and which to variants; have stability and corrected human alignment under the production configuration been verified? This affects the scope of the paper's claim regarding "auditing the production judge."
Reviewer_h43MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper evaluates LLM judges for rubric-based evaluations by extending the L2 benchmark with a taxonomy of judge failures. It is found that Claude-sonnet 4.6 is a stable judge but doesn't guarantee reliability. Furthermore, the authors use a statistical model to show that roughly one-tenth of a score's variation is driven purely by how strict or lenient the judge is, rather than the actual quality of what is being evaluated. Hence, by removing the noise caused by judge severity, the LLM judge performs as well as a human expert. By having this evaluation, the paper proposes a Minimum Viable Judge Assessment (MVJA that considers both pieces of evidence for LLM judges in hard domains.

Critical strengths

The paper's targeted problem, auditing LLM judges for rubric-based English as a Second Language task evaluation, is important in AIED and has been under consideration since the advent of automatic evaluations. The study extended L2-Bench, which was previously developed by the same authors and tested on 32 model configurations, which is commendable and covers the vast majority of the open- and closed-source models. The statistical methods used to test judge severity are appropriate and help justify the need for the proposed method to improve convergence in LLM-based judgment. Last but not least, MVJA's proposal to standardize judge analysis can serve as a stepping stone toward an improved LLM-based evaluation system in L2 education.

Critical weaknesses

The motivation for the target problem is not well established in the literature.

The writing is hard to follow and consists of fragmented sentences with little narrative flow, making the content difficult to follow. The use of terms such as "full battery of tests", "purportedly", and "painstakingly" in the introduction is highly likely to be AI-generated and should be discouraged in academic writing.

Despite reading the introduction twice, I can't establish where the term "hard" context comes from. It would be helpful if the authors referenced the literature that coined and established the term, as it's not a known term.

The introduction uses the term "assess the tractability," but it is nowhere assessed in the paper; it focuses on stability and later reliability. These are not the same terms, so they can't be used interchangeably.

The literature review seems like coompressed, more citations than the actual content and doesn’t even establish the status Quoe.

It would be better to understand the audit methodology if presented in a form of a framework; currently it looks more like a experrimental setup than a research framework.

The core argument (stability != validity; look where humans disagree; model rater severity) is sound and well motivated, but the evidence in Section 3 is thinner than the writing suggests, and there are several internal inconsistencies a careful reviewer will catch.

Stage B’s preference is 50.3% in subsection 3.1 but 51.3% in subsection 3.2; subsection 3.1 says nine models from "five families" while the prompt-sensitivity paragraph says "eight model families" and the stability test says "ten judge models" (Table 1 alone shows at least six vendors); and 9 models × 3 prompts = 27, so how 32 configurations arise is never stated. T

The headline convergent-validity claim (k = 0.861 vs 0.788) rests on Frame B, which is only 169 units, all Sonnet 4.6 responses. With per-category n as low as 1-7 and two categories at k = 1.0 on n = 5, this is a validity result for one model's outputs, not for the leaderboard use case the judge actually serves. Then how does Section 4 generalize from it?

The jump from k = 0.651 (vs 650 gold units) to k = 0.826 (Frame B LOO) is large and comes from changing the frame, not the judge, yet it is not explained.

Most of the 650 gold units appear to rest on just two agreeing raters (only 169 have >=3), so "gold" is weak, and the leaderboard-stability check (dropping 10 units, \rho = 0.9989) is close to trivial, as you partly admit.

The GLMM has rater, unit, and task random effects but no rater x criterion term, so the sentence "raters differ in strictness at the level of individual criteria, not tasks" is not something the model tested.

The judge's "offset of -0.183" is reported without saying how the judge was entered into a model fit on human votes, and the severity correction is applied to the human panel but the gold labels used elsewhere are never re-derived after flipping 236 votes.

Section 4 is reasonable but reads as a long restatement; R2 ("ceiling as target, not floor") sits a bit awkwardly next to the paper celebrating a judge that exceeds the ceiling.

If MVJA is the key contribution of the paper then it should be propley discussed in the main paper rather than pushed in the appendices.

Questions for authors

Why only L2-bench for evaluation? Understandably, no exact dataset is available, but related datasets can be modified for comparative evaluation, such as UniversalCEFR (Imperial et al., 2025) and Ace-CEFR (David et al., 2025).

What is GLMM? The abbreviation is used without definition in the abstract. I assumed it to be a Generalized Linear Mixed Model.

Is the Stage B reference preference 50.3% or 51.3%? Which N (142 or 221) does each refer to?

How many model families are there really: five, six, or eight? And how do you get 32 configurations from nine models and three prompts?

Why does the judge's k jump from 0.651 to 0.826 when you move to Frame B? Is Frame B just an easier subset?

How many of the 650 gold units are based on only two raters? Should those count as "gold"?

How did you estimate the judge's severity offset (−0.183) if the GLMM was fit on the 17,093 human votes?

After flipping 236 votes, did any gold labels change? If so, were the leaderboard and Table 1 recomputed?

All Frame B units are Sonnet 4.6 responses. Can you say anything about whether the judge is valid when scoring other models' outputs?

Can the GLMM support the claim about strictness "at the level of individual criteria" without a rater × criterion effect?

With n = 5 in two categories, should Figure 2 show them at all, or flag them as not estimable like the others?

Reviewer_7HPMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits the LLM judge of L2-Bench, a rubric-based benchmark for second-language education, and shows that standard checks such as agreement and kappa can mislead when gold labels are mostly Pass and expert disagreement has been removed. It proposes correcting each rater's severity to build a leave-one-out human ceiling to compare judges against, and adds a minimum reporting template for judge audits.

Critical strengths

The paper brings rater-severity modeling from educational measurement into LLM judge evaluation, comparing the judge with experts on equal terms after removing each expert's personal strictness, which is fairer than comparing against the majority vote.

Critical weaknesses
  1. The choice of second-language education is not motivated, and since all evidence comes from L2-Bench alone, key phenomena such as label skew may simply reflect how its rubrics were written, leaving the general claims unsupported.
  2. The rater-level votes behind Phases 2 and 3 are not released, so these results cannot be reproduced, which contradicts the paper's own call to release such data.
  3. The related work is thin and omits several directly relevant lines of work, which undermines the claim that rater severity has been overlooked in LLM evaluation.
  4. No new method is proposed, since the tools are borrowed from prior work and the one new piece, the severity-corrected leave-one-out ceiling, is never validated against simulation or known ground truth, so the paper reads more like an audit report.
  5. Most findings, such as the kappa paradox and disagreement as signal, are known phenomena shown again on one benchmark, and the one new positive claim, that the production judge has convergent validity, cannot distinguish a good judge from a rule that never reads the response.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.