- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to estimate the reliability of LLM judges under limited human annotation budgets. It proposes Metric Match, which selects subsets by matching auxiliary statistics derived from inter-model agreement to their full-dataset values. Human annotations on these subsets are then used to estimate judge reliability. Experiments across multiple datasets and reliability metrics compare the method with random and stratified sampling, reporting improvements in average estimation accuracy and potential annotation efficiency.
Critical strengths
-
The paper addresses a relevant practical problem: assessing LLM-judge reliability when human annotations are costly. Using inter-model agreement exploits information available before human annotation and provides an intuitive basis for subset selection.
-
The method is straightforward, and its evaluation covers multiple datasets and reliability metrics. The reported average improvements suggest potential value as a simple strategy for allocating annotation budgets.
Critical weaknesses
-
The theoretical justification is not sufficiently established. The relationship between inter-model and human-model estimation errors requires more explicit conditions. In particular, the derivation of Lemma 1 appears to rely on conditional independence that may not hold after subset selection. Since this relationship supports the selection criterion, unresolved assumptions weaken the method’s central justification.
-
The empirical evidence does not establish consistently robust gains over well-controlled baselines. The stratified baseline is described differently in the main text and appendix, performance deteriorates under auxiliary-judge mismatch, and Metric Match underperforms random sampling across all four metrics on MSLR Outcome. The substantial gap between macro and micro win rates also warrants uncertainty analysis. These issues limit confidence in the generality of the claimed annotation-efficiency benefits.
Questions for authors
-
What assumptions establish the relationship between inter-model and human-model estimation errors? In Lemma 1, why does the required conditional independence remain valid after subset selection?
-
Which stratification implementation was actually used: discrete score categories or four empirical-quantile strata? How were ties and weighting handled, and what distinguishes Metric Match from established balanced or model-assisted sampling methods?
-
What properties must the auxiliary ensemble satisfy for Metric Match to improve estimation? Can the existing results identify conditions under which it is likely to perform worse than random sampling?
-
Can you report uncertainty estimates that account for repeated trials and the fact that the 75 evaluation settings come from only four corpora? How should the substantial macro–micro win-rate gap affect the interpretation of the average gains?
-
Are the annotation savings calculated separately for each reliability metric? Can the selected subsets be reused across metrics, and what is the total annotation cost when several metrics are estimated jointly?
6 comments
Response to Reviewer P2
- Q3: What properties must the auxiliary ensemble satisfy for Metric Match to improve estimation? Can the existing results identify conditions under which it is likely to perform worse than random sampling? Metric Match underperforms random sampling across all four metrics on MSLR Outcome. This is true, and it can be detected a priori from the model label distribution alone: if model outputs collapse toward uniform predictions, practitioners should be cautious, as variance-dependent methods lose signal. We provide further analysis in Appendix J.4 showing that Krippendorff’s is particularly sensitive because it penalizes mismatches in mean and variance between raters in a way that rank- and correlation-based metrics do not. Accordingly, practitioners evaluating with should inspect both the mean shift between models and the variance of the model scores: when mean shift is low (label collapse) or variance is excessively high, random sampling should be preferred over our method for . For ICC, , and , the only dataset where our method underperforms the baselines is MSLR — Outcome, which is likewise detectable a priori via label collapse (). For all metrics, practitioners should screen for model label collapse as an a priori warning sign that our method may underperform baselines. For Krippendorff’s , the threshold for what constitutes collapse is higher, and practitioners should also enforce an upper ceiling on model variance due to the metric's sensitivity to scale and mean alignment. Once these outliers are accounted for via their label distributions—MSLR — Outcome across all metrics, alongside SummEval — Fluency, Hanna — Coherence, and Hanna — Surprise for -only failures—our method achieves consistent improvements across the remaining evaluations: out of 60 total comparisons (15 dataset–axis pairs across 4 metrics), all 7 underperforming cases are accounted for by label collapse alone (4 cases in MSLR — Outcome), label collapse combined with sensitivity (2 cases: SummEval — Fluency and Hanna — Coherence), and excessive model variance combined with sensitivity (1 case: Hanna — Surprise).
- Q4: Can you report uncertainty estimates that account for repeated trials and the fact that the 75 evaluation settings come from only four corpora? Table 12 and13 show win rate by dataset can show overall confidence interval + mean+-CI per dataset Table 12 and 13 shows per corpora results, which highlights the relative independence of method performance across subsets of the corpora, as MSLR outcome is a canonical example of where our method underperforms while MSLR fluency is an example of where our method consistently beats baselines. Nonetheless, we present Table 12 and 13 here with confidence intervals that are bootstrapped over trials to reflect the variance introduced by the repeated trials.
Table 12 (micro-average win-rate) with bootstrapped confidence intervals (From Part 2)
| Rand. α | Rand. ICC | Rand. ρ | Rand. τ | Rand. BC α | Rand. BC ICC | Rand. BC ρ | Rand. BC τ | Stratif. α | Stratif. ICC | Stratif. ρ | Stratif. τ | Clust. α | Clust. ICC | Clust. ρ | Clust. τ | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hanna | Coherence | [17.60, 21.00] | [46.30, 50.50] | [56.26, 60.57] | [52.36, 56.72] | [9.25, 11.95] | [43.90, 48.10] | [56.93, 61.29] | [57.39, 61.60] | [17.15, 20.50] | [46.40, 50.45] | [58.50, 62.74] | [56.81, 61.16] | [26.40, 30.35] | [54.90, 59.00] | [60.16, 64.42] | [56.84, 61.21] |
| Hanna | Complexity | [56.90, 61.10] | [58.20, 62.30] | [54.74, 59.11] | [53.69, 58.00] | [62.75, 66.85] | [54.95, 59.25] | [55.75, 60.01] | [56.60, 60.86] | [59.00, 63.15] | [59.55, 63.75] | [58.50, 62.70] | [55.65, 59.95] | [74.30, 77.85] | [69.00, 72.95] | [56.43, 60.79] | [54.12, 58.38] |
| Hanna | Empathy | [57.95, 62.15] | [56.15, 60.35] | [51.31, 55.57] | [47.06, 51.38] | [58.25, 62.50] | [52.40, 56.75] | [54.32, 58.64] | [51.48, 55.75] | [55.80, 60.05] | [51.60, 55.85] | [52.58, 56.83] | [49.60, 53.85] | [62.65, 66.80] | [59.70, 63.70] | [55.32, 59.64] | [52.94, 57.20] |
| Hanna | Engagement | [50.20, 54.40] | [53.20, 57.40] | [54.81, 59.17] | [52.58, 56.94] | [46.85, 51.05] | [52.25, 56.55] | [57.35, 61.60] | [55.07, 59.32] | [53.85, 58.10] | [58.55, 62.75] | [58.93, 63.28] | [55.34, 59.64] | [63.30, 67.25] | [59.90, 63.90] | [59.89, 64.12] | [57.24, 61.52] |
| Hanna | Relevance | [48.05, 52.25] | [54.70, 58.90] | [55.31, 59.52] | [55.06, 59.32] | [46.85, 51.10] | [47.60, 51.85] | [54.91, 59.17] | [53.46, 57.77] | [48.05, 52.20] | [58.70, 62.90] | [56.30, 60.60] | [54.95, 59.20] | [50.20, 54.25] | [59.95, 64.15] | [56.23, 60.55] | [56.63, 60.80] |
| Hanna | Surprise | [40.20, 44.45] | [46.85, 51.15] | [47.70, 52.00] | [48.15, 52.45] | [48.85, 53.15] | [48.20, 52.50] | [52.65, 56.95] | [51.70, 56.05] | [48.40, 52.70] | [50.95, 55.20] | [50.15, 54.45] | [49.65, 53.95] | [47.50, 51.85] | [52.65, 56.95] | [53.48, 57.78] | [54.88, 59.13] |
| MedVal | Risk | [54.30, 58.60] | [58.35, 62.50] | [53.55, 57.86] | [54.03, 58.29] | [53.75, 58.10] | [54.30, 58.60] | [54.05, 58.36] | [54.08, 58.39] | [56.10, 60.35] | [58.55, 62.80] | [54.53, 58.79] | [53.50, 57.76] | [52.95, 57.20] | [53.00, 57.20] | [50.43, 54.73] | [52.28, 56.53] |
| MSLR | Fluency | [56.53, 60.84] | [58.74, 63.05] | [58.06, 62.43] | [57.69, 62.01] | [60.12, 64.34] | [55.50, 59.82] | [57.47, 61.83] | [56.40, 60.87] | [58.25, 62.30] | [52.60, 56.80] | [56.87, 61.05] | [56.17, 60.41] | [70.16, 74.14] | [74.90, 78.61] | [70.92, 75.09] | [71.58, 75.70] |
| MSLR | Intervention | [52.95, 57.20] | [56.40, 60.60] | [54.47, 58.68] | [54.27, 58.43] | [61.75, 65.85] | [57.25, 61.60] | [58.94, 63.15] | [55.37, 59.69] | [54.05, 58.30] | [56.55, 60.75] | [55.37, 59.63] | [54.64, 58.95] | [63.55, 67.50] | [65.00, 69.00] | [64.71, 68.72] | [64.36, 68.27] |
| MSLR | Outcome | [35.80, 40.00] | [27.99, 31.80] | [37.79, 42.00] | [38.20, 42.45] | [41.95, 46.30] | [30.95, 34.90] | [46.61, 51.01] | [46.10, 50.41] | [37.35, 41.45] | [34.35, 38.36] | [42.94, 47.20] | [42.39, 46.65] | [31.85, 35.85] | [33.06, 37.07] | [41.70, 45.90] | [41.14, 45.39] |
| MSLR | Population | [55.40, 59.70] | [56.65, 60.90] | [53.85, 58.12] | [54.09, 58.41] | [63.35, 67.50] | [60.20, 64.40] | [64.25, 68.33] | [60.22, 64.44] | [51.85, 56.15] | [53.70, 58.00] | [50.88, 55.19] | [51.18, 55.59] | [51.75, 56.05] | [50.83, 55.13] | [48.84, 53.12] | [47.71, 52.04] |
| SummEval (SE) | Coherence | [58.15, 62.35] | [57.65, 61.80] | [55.19, 59.50] | [52.38, 56.69] | [60.25, 64.45] | [56.25, 60.55] | [57.19, 61.45] | [54.64, 58.95] | [58.00, 62.25] | [57.35, 61.55] | [58.95, 63.15] | [55.65, 60.00] | [62.70, 66.80] | [63.15, 67.20] | [60.85, 65.05] | [61.25, 65.35] |
| SEl | Consistency | [55.97, 60.24] | [57.82, 62.04] | [54.45, 58.95] | [52.36, 56.92] | [59.41, 63.63] | [62.40, 66.56] | [59.33, 63.72] | [59.17, 63.56] | [51.70, 56.00] | [51.90, 56.15] | [54.78, 59.02] | [52.31, 56.63] | [58.01, 62.19] | [58.62, 62.91] | [57.67, 61.94] | [55.77, 60.11] |
| SE | Fluency | [34.65, 38.80] | [52.00, 56.25] | [52.16, 56.37] | [50.26, 54.62] | [46.25, 50.55] | [62.55, 66.60] | [61.60, 65.81] | [58.87, 63.18] | [45.50, 49.55] | [63.10, 67.00] | [61.94, 66.01] | [59.13, 63.19] | [42.20, 46.30] | [56.38, 60.63] | [54.79, 59.16] | [54.44, 58.75] |
| SE | Relevance | [56.55, 60.85] | [57.05, 61.30] | [57.19, 61.49] | [53.78, 58.04] | [67.80, 71.70] | [64.80, 68.85] | [67.40, 71.36] | [62.14, 66.25] | [62.75, 66.70] | [56.75, 60.95] | [68.45, 72.35] | [63.70, 67.80] | [60.20, 64.35] | [59.45, 63.55] | [59.16, 63.41] | [57.61, 61.71] |
| Mean | [50.30, 51.39] | [54.76, 55.85] | [54.71, 55.83] | [53.33, 54.44] | [54.01, 55.08] | [55.14, 56.24] | [58.84, 59.93] | [57.09, 58.22] | [52.06, 53.13] | [55.58, 56.68] | [57.56, 58.64] | [55.64, 56.74] | [56.02, 57.09] | [59.53, 60.62] | [58.29, 59.39] | [57.48, 58.57] |
Response for Reviewer P1
Q1: What assumptions establish the relationship between inter-model and human-model estimation errors? The derivation of Lemma 1 appears to rely on conditional independence that may not hold after subset selection. Why does the required conditional independence remain valid after subset selection?
We note a few ways to interpret the independence issue raised. We are not sure which is meant, but we will elucidate the cases here, one of which does point to a minor error in the proof that we have now corrected:
Q2: Which stratification implementation was actually used: discrete score categories or four empirical-quantile strata? How were ties and weighting handled, and what distinguishes Metric Match from established balanced or model-assisted sampling methods?
We split over four empirical-quantile strata so as not to over-index on minority scores.