acceptodds
Withdrawn from ICLR 2027

From LLM Judge Scores to Evaluation Design: A Generalizability-Theory Framework

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM judges provide task-specific feedback for selecting responses, comparing models, and training policies. Their scores combine stable differences between responses with variation introduced by the judging procedure. We present a generalizability-based framework that profiles each fixed judge, projects grading precision from a pilot, and checks whether a protocol improves decisions against an external criterion. We instantiate it with twelve open-weight judges grading the same responses on three sources with correctness, rubric-quality, or human-preference references. The crossed design varies two close prompt paraphrases and three generations. We find a small pilot per source predicts generation-noise rankings on separate queries (correlations 0.93–0.95) and approximate grading disagreement. A second grading under unchanged wording improves aggregate ordering point estimates by 0.89–2.95 percentage points. An independent-call diagnostic shows that averaging can strengthen both reference-consistent and reference-opposed preferences. The framework yields judge-specific precision and validity profiles under declared conditions and separates protocol planning from judge selection.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, generalizability theory, evaluation reliability, measurement error, variance decomposition, decision study, prompt sensitivity, automatic evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_AsW6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper applies Generalizability Theory to LLM judges. For each judge, it separates score variance into stable item differences, template variance, and generation noise. It then uses a small pilot and D-studies to project the precision of alternative grading allocations. Validity is assessed separately against benchmark references. Experiments on twelve open-weight judges across JudgeBench, BiGGenBench, and HelpSteer3 show that pilot estimates predict held-out noise rankings well. Repeated grading gives only modest ordering gains and can reinforce reference-opposed preferences.

Critical strengths
  1. The statistical execution is careful and well-scoped. Calibration and test queries are disjoint, and the paper does not conflate consistency with correctness. This is the main reason I consider the work technically sound.

  2. The pilot-based variance estimates are validated on held-out queries, not only on the fitting data. This gives some practical support to the proposed budgeting framework.

Critical weaknesses
  1. The framework is largely a direct application of standard Generalizability Theory, and closely related work such as Messing already studies crossed variance decomposition, decision studies, and external validation for judge pipelines. The contribution therefore reads more as a careful systematization than a substantial methodological advance.

  2. The main empirical findings are fairly expected, and the scope is narrow. The strata are defined by the judge's own estimated direction, so it is almost expected by construction that averaging reinforces reference-opposed preferences rather than correcting them. Much of the gain from repetition is concentrated on first-call ties, which suggests that part of the benefit simply compensates for the coarse 0–5 score discretization. The prompt study uses only two closely related paraphrases, and the evaluation is limited to two model families and pointwise scoring. These limitations make it unclear whether the framework materially improves evaluation decisions beyond this setting.

Questions for authors
  1. What does this work establish beyond the decomposition and decision study already reported by Messing?

  2. Under the same call budget, does the D-study allocation outperform a simple rule that repeats only tied judgments, or a probability-weighted single-call readout?

  3. Do the variance profiles and allocation recommendations hold with more diverse prompts or pairwise judging?

Reviewer_6eoLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper applies generalizability theory to LLM-judge evaluation. For each fixed judge, it decomposes score variation into stable response differences, prompt-template sensitivity, and generation noise. A pilot G-study estimates these components, while a D-study predicts the precision of alternative grading allocations. External correctness, rubric, and preference labels are then used to test whether greater precision also produces better candidate ordering.

The experiment evaluates 12 open weight judges from the Gemma and Qwen families on JudgeBench, BiGGenBench, and HelpSteer3. Each response is graded under two prompt paraphrases with three generations per template. The authors use 16 calibration queries and 100 held-out test queries per source.

The main findings are sensible and well supported within this experimental setting. Pilot estimates rank judges’ generation noise similarly on held-out queries, with correlations of 0.93–0.95. Repeating a grading reduces score SEM substantially but improves ordering accuracy by only 0.89–2.95 percentage points. Repetition can also make a systematically wrong preference more stable. Finally, probability-weighted readout improves accuracy mainly by resolving ties created by the integer score scale.

While the paper is technically careful and addresses a real weakness in LLM-judge research: reliability and validity are often conflated, my main reservations concern the incremental novelty over prior generalizability-theory studies and the narrow empirical support for calling the framework general.

Critical strengths
  1. The main contribution is the separation of precision from correctness. A judge can be internally consistent while disagreeing with the intended criterion, and repeated calls can strengthen either a correct or an incorrect preference. The paper formalizes this distinction clearly and demonstrates it empirically.
Critical weaknesses
  1. Simply put, two prompt paraphrases do not establish prompt robustness. The template universe consists of exactly two closely related paraphrases, and the universe score is defined as their equal-weight average. Under this definition, evaluating both templates necessarily removes deviation from that two-template mean. This is a correct finite-universe result, but it does not show robustness to unseen prompt wording.
Questions for authors
  1. What is technically new relative to the closest previous generalizability-theory analysis of LLM judges?

  2. How stable are judge rankings, variance estimates, and allocation choices across random pilot splits and different pilot sizes?

  3. Please fix the citation issues, various places with ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.