- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper applies Generalizability Theory to LLM judges. For each judge, it separates score variance into stable item differences, template variance, and generation noise. It then uses a small pilot and D-studies to project the precision of alternative grading allocations. Validity is assessed separately against benchmark references. Experiments on twelve open-weight judges across JudgeBench, BiGGenBench, and HelpSteer3 show that pilot estimates predict held-out noise rankings well. Repeated grading gives only modest ordering gains and can reinforce reference-opposed preferences.
Critical strengths
-
The statistical execution is careful and well-scoped. Calibration and test queries are disjoint, and the paper does not conflate consistency with correctness. This is the main reason I consider the work technically sound.
-
The pilot-based variance estimates are validated on held-out queries, not only on the fitting data. This gives some practical support to the proposed budgeting framework.
Critical weaknesses
-
The framework is largely a direct application of standard Generalizability Theory, and closely related work such as Messing already studies crossed variance decomposition, decision studies, and external validation for judge pipelines. The contribution therefore reads more as a careful systematization than a substantial methodological advance.
-
The main empirical findings are fairly expected, and the scope is narrow. The strata are defined by the judge's own estimated direction, so it is almost expected by construction that averaging reinforces reference-opposed preferences rather than correcting them. Much of the gain from repetition is concentrated on first-call ties, which suggests that part of the benefit simply compensates for the coarse 0–5 score discretization. The prompt study uses only two closely related paraphrases, and the evaluation is limited to two model families and pointwise scoring. These limitations make it unclear whether the framework materially improves evaluation decisions beyond this setting.
Questions for authors
-
What does this work establish beyond the decomposition and decision study already reported by Messing?
-
Under the same call budget, does the D-study allocation outperform a simple rule that repeats only tied judgments, or a probability-weighted single-call readout?
-
Do the variance profiles and allocation recommendations hold with more diverse prompts or pairwise judging?