acceptodds
Withdrawn from ICLR 2027

Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation. However, the reliability of these judges depends critically on their alignment with human raters — a property that itself depends on costly human annotations. In this work, we develop a method (Metric Match) for estimating correlation-based reliability metrics of LLM judges from limited annotations. Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric with respect to acquired synthetic labels. We empirically show that Metric Match achieves a win-rate of at least 0.794 with respect to four natural baselines across four different correlation metrics and 15 datasets. Furthermore, Metric Match exhibits a 16.2% decrease in average estimation error and reduces annotation needs by 28.1%. We provide a cost model and highlight a medical case study where our method saves $1,083.33 on expert annotations compared to baselines. Finally, for the downstream task of classifying whether a judge is sufficiently reliable, Metric Match also outperforms all baselines in terms of accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Judge, reliability, synthetic labels, sample efficiency, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SH2ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to estimate the reliability of LLM judges under limited human annotation budgets. It proposes Metric Match, which selects subsets by matching auxiliary statistics derived from inter-model agreement to their full-dataset values. Human annotations on these subsets are then used to estimate judge reliability. Experiments across multiple datasets and reliability metrics compare the method with random and stratified sampling, reporting improvements in average estimation accuracy and potential annotation efficiency.

Critical strengths
  1. The paper addresses a relevant practical problem: assessing LLM-judge reliability when human annotations are costly. Using inter-model agreement exploits information available before human annotation and provides an intuitive basis for subset selection.

  2. The method is straightforward, and its evaluation covers multiple datasets and reliability metrics. The reported average improvements suggest potential value as a simple strategy for allocating annotation budgets.

Critical weaknesses
  1. The theoretical justification is not sufficiently established. The relationship between inter-model and human-model estimation errors requires more explicit conditions. In particular, the derivation of Lemma 1 appears to rely on conditional independence that may not hold after subset selection. Since this relationship supports the selection criterion, unresolved assumptions weaken the method’s central justification.

  2. The empirical evidence does not establish consistently robust gains over well-controlled baselines. The stratified baseline is described differently in the main text and appendix, performance deteriorates under auxiliary-judge mismatch, and Metric Match underperforms random sampling across all four metrics on MSLR Outcome. The substantial gap between macro and micro win rates also warrants uncertainty analysis. These issues limit confidence in the generality of the claimed annotation-efficiency benefits.

Questions for authors
  1. What assumptions establish the relationship between inter-model and human-model estimation errors? In Lemma 1, why does the required conditional independence remain valid after subset selection?

  2. Which stratification implementation was actually used: discrete score categories or four empirical-quantile strata? How were ties and weighting handled, and what distinguishes Metric Match from established balanced or model-assisted sampling methods?

  3. What properties must the auxiliary ensemble satisfy for Metric Match to improve estimation? Can the existing results identify conditions under which it is likely to perform worse than random sampling?

  4. Can you report uncertainty estimates that account for repeated trials and the fact that the 75 evaluation settings come from only four corpora? How should the substantial macro–micro win-rate gap affect the interpretation of the average gains?

  5. Are the annotation savings calculated separately for each reliability metric? Can the selected subsets be reused across metrics, and what is the total annotation cost when several metrics are estimated jointly?

6 comments
AuthorsOpenReview ↗

Response for Reviewer P1

  • Q1: What assumptions establish the relationship between inter-model and human-model estimation errors? The derivation of Lemma 1 appears to rely on conditional independence that may not hold after subset selection. Why does the required conditional independence remain valid after subset selection?

    We note a few ways to interpret the independence issue raised. We are not sure which is meant, but we will elucidate the cases here, one of which does point to a minor error in the proof that we have now corrected:

    • There may be overlap in the subsets S1,…,SCS_1, \dots, S_C, SrS_r that would impact the joint distribution over the metric ρ^S1,…,ρ^SC\hat\rho_{S_1}, \dots, \hat\rho_{S_C}, ρ^Sr\hat\rho_{S_r} or the estimation errors ϵ1,…,ϵC\epsilon_1, \dots, \epsilon_C, ϵrand\epsilon^{\text{rand}}. However, due to exchangeability, the ranks and relative errors are independent for any randomly selected i,j∈{1,…,C}i, j \in \{1, \dots, C\} and SrandS^{\text{rand}}.
    • Note that Sj⋆S_{j^\star} was chosen based on the ranks from the inter-model estimation errors on the observed CC subsets S1,…,SCS_1, \dots, S_C but not with access to the C+1C+1th subset (“random” subset), SrS_r, that is drawn completely independently and not used for selection of Sj⋆S_{j^\star}. The ranks we care about are the human-model estimation errors. Once j⋆j^\star is selected from the subsets j=1,…,Cj = 1, \dots, C, the distribution of rank on human-model estimation error of SjS_j for some j∈{1,…,C}j \in \{1, \dots, C\} indeed will not be independent of the rank of Sj⋆S_{j^\star} because the inter-model estimation errors and human-model estimation errors may be correlated and j⋆j^\star was selected by reasoning on information from SjS_j. But r∉{1,…,C}r \notin \{1, \dots, C\}. If Sj⋆S_{j^\star} had been chosen on the inter-model estimation error ranks such that the rank included SrS_r, then indeed this independence would not hold. However, since j⋆j^\star is selected independent of SrS_r, and only based on S1,…,SCS_1, \dots, S_C, the rank distribution of ϵrand\epsilon^{\text{rand}} remains uniform over the human-model estimation error ranks even given R[ϵj⋆]R[\epsilon_{j^\star}]. We can improve the notation here by indicating that r∉{1,…,C}r \notin \{1, \dots, C\} or using a different indicator (for instance SrandS^{\text{rand}}), if that seems more appropriate to the reviewer. Additionally, we can introduce notation such that the set of subsets involved in the rank calculation is more clear. For instance, we can denote RCR^C to refer to a rank variable over the original CC subsets, and RC+randR^{C+\text{rand}} to refer to a rank variable over the CC subsets plus the random auxiliary subset.
    • Nonetheless indeed, it is correct that the probability is impacted by the setting of R[ϵj⋆]=jR[\epsilon_{j^\star}] = j because the fact that there already exists a subset with rank jj means only CC ranks are left, so the divisor is no longer C+1C+1 but CC. Thus the only component which was not handled was the fact that the jjth rank is already taken, in which case the probability shifts from C+1−jC+1\frac{C+1 - j}{C+1} to C+1−jC\frac{C+1 - j}{C} as the jjth rank is taken. Therefore, we end up with Pr⁡[ϵj⋆<ϵrand]=C+1C−E[R[ϵj⋆]]C\Pr[\epsilon_{j^\star} \lt \epsilon^{\text{rand}}] = \frac{C+1}{C} - \frac{\mathbb{E}[R[\epsilon_{j^\star}]]}{C} and the associated condition E[R[ϵj⋆]]<C+22\mathbb{E}[R[\epsilon_{j^\star}]] \lt \frac{C+2}{2} where the rank includes the random subset. Shifting the rank variables back to over only the original CC subsets becomes E[R[ϵj⋆]]<C+12\mathbb{E}[R[\epsilon_{j^\star}]] \lt \frac{C+1}{2}, as in the paper. However, we will make this correction to the original probability calculation and we appreciate the reviewer’s thorough review.
  • Q2: Which stratification implementation was actually used: discrete score categories or four empirical-quantile strata? How were ties and weighting handled, and what distinguishes Metric Match from established balanced or model-assisted sampling methods?

    We split over four empirical-quantile strata so as not to over-index on minority scores.

AuthorsOpenReview ↗

Response to Reviewer P2

  • Q3: What properties must the auxiliary ensemble satisfy for Metric Match to improve estimation? Can the existing results identify conditions under which it is likely to perform worse than random sampling? Metric Match underperforms random sampling across all four metrics on MSLR Outcome. This is true, and it can be detected a priori from the model label distribution alone: if model outputs collapse toward uniform predictions, practitioners should be cautious, as variance-dependent methods lose signal. We provide further analysis in Appendix J.4 showing that Krippendorff’s α\alpha is particularly sensitive because it penalizes mismatches in mean and variance between raters in a way that rank- and correlation-based metrics do not. Accordingly, practitioners evaluating with α\alpha should inspect both the mean shift between models and the variance of the model scores: when mean shift is low (label collapse) or variance is excessively high, random sampling should be preferred over our method for . For ICC, τ\tau, and ρ\rho, the only dataset where our method underperforms the baselines is MSLR — Outcome, which is likewise detectable a priori via label collapse ((μX−μY)2=0.02(\mu_X - \mu_Y)^2 = 0.02). For all metrics, practitioners should screen for model label collapse as an a priori warning sign that our method may underperform baselines. For Krippendorff’s α\alpha, the threshold for what constitutes collapse is higher, and practitioners should also enforce an upper ceiling on model variance due to the metric's sensitivity to scale and mean alignment. Once these outliers are accounted for via their label distributions—MSLR — Outcome across all metrics, alongside SummEval — Fluency, Hanna — Coherence, and Hanna — Surprise for α\alpha-only failures—our method achieves consistent improvements across the remaining evaluations: out of 60 total comparisons (15 dataset–axis pairs across 4 metrics), all 7 underperforming cases are accounted for by label collapse alone (4 cases in MSLR — Outcome), label collapse combined with α\alpha sensitivity (2 cases: SummEval — Fluency and Hanna — Coherence), and excessive model variance combined with α\alpha sensitivity (1 case: Hanna — Surprise).
  • Q4: Can you report uncertainty estimates that account for repeated trials and the fact that the 75 evaluation settings come from only four corpora? Table 12 and13 show win rate by dataset can show overall confidence interval + mean+-CI per dataset Table 12 and 13 shows per corpora results, which highlights the relative independence of method performance across subsets of the corpora, as MSLR outcome is a canonical example of where our method underperforms while MSLR fluency is an example of where our method consistently beats baselines. Nonetheless, we present Table 12 and 13 here with confidence intervals that are bootstrapped over trials to reflect the variance introduced by the repeated trials.
AuthorsOpenReview ↗

Table 12 (micro-average win-rate) with bootstrapped confidence intervals (From Part 2)

Rand. αRand. ICCRand. ρRand. τRand. BC αRand. BC ICCRand. BC ρRand. BC τStratif. αStratif. ICCStratif. ρStratif. τClust. αClust. ICCClust. ρClust. τ
HannaCoherence[17.60, 21.00][46.30, 50.50][56.26, 60.57][52.36, 56.72][9.25, 11.95][43.90, 48.10][56.93, 61.29][57.39, 61.60][17.15, 20.50][46.40, 50.45][58.50, 62.74][56.81, 61.16][26.40, 30.35][54.90, 59.00][60.16, 64.42][56.84, 61.21]
HannaComplexity[56.90, 61.10][58.20, 62.30][54.74, 59.11][53.69, 58.00][62.75, 66.85][54.95, 59.25][55.75, 60.01][56.60, 60.86][59.00, 63.15][59.55, 63.75][58.50, 62.70][55.65, 59.95][74.30, 77.85][69.00, 72.95][56.43, 60.79][54.12, 58.38]
HannaEmpathy[57.95, 62.15][56.15, 60.35][51.31, 55.57][47.06, 51.38][58.25, 62.50][52.40, 56.75][54.32, 58.64][51.48, 55.75][55.80, 60.05][51.60, 55.85][52.58, 56.83][49.60, 53.85][62.65, 66.80][59.70, 63.70][55.32, 59.64][52.94, 57.20]
HannaEngagement[50.20, 54.40][53.20, 57.40][54.81, 59.17][52.58, 56.94][46.85, 51.05][52.25, 56.55][57.35, 61.60][55.07, 59.32][53.85, 58.10][58.55, 62.75][58.93, 63.28][55.34, 59.64][63.30, 67.25][59.90, 63.90][59.89, 64.12][57.24, 61.52]
HannaRelevance[48.05, 52.25][54.70, 58.90][55.31, 59.52][55.06, 59.32][46.85, 51.10][47.60, 51.85][54.91, 59.17][53.46, 57.77][48.05, 52.20][58.70, 62.90][56.30, 60.60][54.95, 59.20][50.20, 54.25][59.95, 64.15][56.23, 60.55][56.63, 60.80]
HannaSurprise[40.20, 44.45][46.85, 51.15][47.70, 52.00][48.15, 52.45][48.85, 53.15][48.20, 52.50][52.65, 56.95][51.70, 56.05][48.40, 52.70][50.95, 55.20][50.15, 54.45][49.65, 53.95][47.50, 51.85][52.65, 56.95][53.48, 57.78][54.88, 59.13]
MedValRisk[54.30, 58.60][58.35, 62.50][53.55, 57.86][54.03, 58.29][53.75, 58.10][54.30, 58.60][54.05, 58.36][54.08, 58.39][56.10, 60.35][58.55, 62.80][54.53, 58.79][53.50, 57.76][52.95, 57.20][53.00, 57.20][50.43, 54.73][52.28, 56.53]
MSLRFluency[56.53, 60.84][58.74, 63.05][58.06, 62.43][57.69, 62.01][60.12, 64.34][55.50, 59.82][57.47, 61.83][56.40, 60.87][58.25, 62.30][52.60, 56.80][56.87, 61.05][56.17, 60.41][70.16, 74.14][74.90, 78.61][70.92, 75.09][71.58, 75.70]
MSLRIntervention[52.95, 57.20][56.40, 60.60][54.47, 58.68][54.27, 58.43][61.75, 65.85][57.25, 61.60][58.94, 63.15][55.37, 59.69][54.05, 58.30][56.55, 60.75][55.37, 59.63][54.64, 58.95][63.55, 67.50][65.00, 69.00][64.71, 68.72][64.36, 68.27]
MSLROutcome[35.80, 40.00][27.99, 31.80][37.79, 42.00][38.20, 42.45][41.95, 46.30][30.95, 34.90][46.61, 51.01][46.10, 50.41][37.35, 41.45][34.35, 38.36][42.94, 47.20][42.39, 46.65][31.85, 35.85][33.06, 37.07][41.70, 45.90][41.14, 45.39]
MSLRPopulation[55.40, 59.70][56.65, 60.90][53.85, 58.12][54.09, 58.41][63.35, 67.50][60.20, 64.40][64.25, 68.33][60.22, 64.44][51.85, 56.15][53.70, 58.00][50.88, 55.19][51.18, 55.59][51.75, 56.05][50.83, 55.13][48.84, 53.12][47.71, 52.04]
SummEval (SE)Coherence[58.15, 62.35][57.65, 61.80][55.19, 59.50][52.38, 56.69][60.25, 64.45][56.25, 60.55][57.19, 61.45][54.64, 58.95][58.00, 62.25][57.35, 61.55][58.95, 63.15][55.65, 60.00][62.70, 66.80][63.15, 67.20][60.85, 65.05][61.25, 65.35]
SElConsistency[55.97, 60.24][57.82, 62.04][54.45, 58.95][52.36, 56.92][59.41, 63.63][62.40, 66.56][59.33, 63.72][59.17, 63.56][51.70, 56.00][51.90, 56.15][54.78, 59.02][52.31, 56.63][58.01, 62.19][58.62, 62.91][57.67, 61.94][55.77, 60.11]
SEFluency[34.65, 38.80][52.00, 56.25][52.16, 56.37][50.26, 54.62][46.25, 50.55][62.55, 66.60][61.60, 65.81][58.87, 63.18][45.50, 49.55][63.10, 67.00][61.94, 66.01][59.13, 63.19][42.20, 46.30][56.38, 60.63][54.79, 59.16][54.44, 58.75]
SERelevance[56.55, 60.85][57.05, 61.30][57.19, 61.49][53.78, 58.04][67.80, 71.70][64.80, 68.85][67.40, 71.36][62.14, 66.25][62.75, 66.70][56.75, 60.95][68.45, 72.35][63.70, 67.80][60.20, 64.35][59.45, 63.55][59.16, 63.41][57.61, 61.71]
Mean[50.30, 51.39][54.76, 55.85][54.71, 55.83][53.33, 54.44][54.01, 55.08][55.14, 56.24][58.84, 59.93][57.09, 58.22][52.06, 53.13][55.58, 56.68][57.56, 58.64][55.64, 56.74][56.02, 57.09][59.53, 60.62][58.29, 59.39][57.48, 58.57]
Reviewer_kUMjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a metric, Metric Match, to estimate the correlation-type reliability metrics between an LLM judge and human raters. Besides, this metric has been tested on 15 datasets. According to this paper, it has been tested and outperformed other baselines to a large extend.

Critical strengths
  1. This metric explores an important question to estimate the correlation between LLM judges and human annotation, such as the subset matches.
Critical weaknesses
  1. the method is lack of novelty, for example selecting the C random draws from a cheap statistic matches ints population value has been mentioned in the related work. The use of metric at the balancing target should be framed against these methods.

  2. many important arguement lacks of support: for example, the "win-rate of at least 0.794 against four baselines"? I cannot find any details or experiments mentioned in Section 4.2 and 4.3. Where does it comes from?

  3. Overall format are quite wierd, why Metric Match is always highlighted and in bold?

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer

  • Q1: “The use of the metric as the balancing target should be framed against these methods.” We are unclear exactly what part the reviewer is referring to, but we want to clarify here: (1) selecting bb i.i.d. datapoints randomly and computing the statistic on the chosen subset comprising these datapoints is standard, and we do compare against this method ("random selection"). (2) It's correct that the generic idea of drawing random subsets and selecting the best subset is a frequently used heuristic in random search and Monte Carlo sampling. However, these usually select the best subset with respect to estimation of the true metric. In our case, this would require estimation on the statistic between models and humans for each candidate subset, to which we do not have access, since this would require collecting C⋅bC\cdot b human annotations, well beyond the budget bb. Otherwise, we believe we differentiate and explain explicitly why other related works in subset selection and estimation do not fit inter-rater reliability metrics or our problem setup and thus are not appropriate as baselines in Section 2.3.
  • Q2: “Where does the "win-rate of at least 0.794 against four baselines” The 0.794 comes from taking the average Macro Win Rate across the four metrics (Alpha, ICC, Rho, Tau) for the Stratified baseline. We will make this more clear, as well as other instances of aggregation, in the text.
  • Q3: “Why is Metric Match always highlighted and in bold?” We bold our method in order to highlight its usage throughout the paper. We can improve this formatting in the camera ready.
  • This is not an AI paper. All writing and experimental design was done by us, the authors, and AI was used primarily for code execution and copy editing of the final draft of the paper, as stated in our AI use statement.
Reviewer_oX19OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on the human annotation bottleneck when evaluating LLM-as-a-judge reliability. To this end, it proposes a method (MetricMatch) for selecting a subset of samples for human labeling by matching their inter-model reliability (AI-AI agreement score) with the inter-model reliability of the entire dataset. Using 4 reliability metrics (ICC, Krippendorff's, Spearman's, Kendall's) authors evaluate 5 LLM judge models across 15 dataset-axis settings and show that MetricMatch reduces average estimation error and human annotation costs compared to stratified selection.

Critical strengths
  • Compared to the strongest baseline (stratified selection), MetricMatch achieves substantial decrease in estimation error and human annotation costs. In expert-critical domains like clinical text evaluation (MedVAL), authors show that this efficiency translates to actual savings.
Critical weaknesses
  1. The proposed method requires a synthetic ensemble of judges to be aligned with the target judge model in terms of the task capability. This means if/when the ensemble if weaker than the target judge, MetricMatch based subset could potentially result in estimation error worse than simple random sampling.
    • This is also related to another limitation mentioned by the authors about how MetricMatch fails when all of synthetic judges are mis-calibrated (e.g., skewed towards a a single label while human annotators do not).
Questions for authors

Questions:

  1. How can someone using MetricMatch detect shared model collapse or annotator scale shifts before proceeding to collect human annotations for the selected subset?
  2. From Appendix, I understand that the subsets are selected by evaluating only C=20 random candidates. How is this number selected?

Suggestions:

  • As underlined in recent LLM-as-a-judge research, rank correlations (Spearman, Kendall) ignore absolute scoring bias, and IAA metrics (Krippendorff's, ICC) suffer from severe statistical deflation under skewed score distributions (Kappa Paradox). However, MetricMatch explicitly relies on these. Authors could discuss how their method fares if other LLM-judge evaluation metrics are considered.
1 comment
AuthorsOpenReview ↗

Response to Reviewer

  • W1: “The proposed method requires a synthetic ensemble of judges to be aligned with the target judge model in terms of task capability.” We highlight the results of our method across different models throughout the paper, and across different model strengths in Appendix G.2. We show that our method performs best when the ensemble and the target are both from “strong” closed-source models. We note that this is an empirical result and thus not an inherent defect of the method. Further work can be done to identify optimal model ensemble composition.
  • Q1: “MetricMatch fails when all of the synthetic judges are mis-calibrated (e.g., skewed towards a single label while human annotators are not). How can someone using MetricMatch detect shared model collapse or annotator scale shifts before proceeding to collect human annotations for the selected subset?” This is true, and it can be detected a priori from the model label distribution alone: if model outputs collapse toward uniform predictions, practitioners should be cautious, as variance-dependent methods lose signal. We provide further analysis in Appendix J.4 showing that Krippendorff’s α\alpha is particularly sensitive because it penalizes mismatches in mean and variance between raters in a way that rank- and correlation-based metrics do not. Accordingly, practitioners evaluating with α\alpha should inspect both the mean shift between models and the variance of the model scores: when mean shift is low (label collapse) or variance is excessively high, random sampling should be preferred over our method for α\alpha. For ICC, τ\tau, and ρ\rho, the only dataset where our method underperforms the baselines is MSLR — Outcome, which is likewise detectable a priori via label collapse ((μX−μY)2=0.02(\mu_X - \mu_Y)^2 = 0.02). For all metrics, practitioners should screen for model label collapse as an a priori warning sign that our method may underperform baselines. For Krippendorff’s α\alpha, the threshold for what constitutes collapse is higher, and practitioners should also enforce an upper ceiling on model variance due to the metric's sensitivity to scale and mean alignment. Once these outliers are accounted for via their label distributions—MSLR — Outcome across all metrics, alongside SummEval — Fluency, Hanna — Coherence, and Hanna — Surprise for -only failures—our method achieves consistent improvements across the remaining evaluations: out of 60 total comparisons (15 dataset–axis pairs across 4 metrics), all 7 underperforming cases are accounted for by label collapse alone (4 cases in MSLR — Outcome), label collapse combined with α\alpha sensitivity (2 cases: SummEval — Fluency and Hanna — Coherence), and excessive model variance combined with sensitivity (1 case: Hanna — Surprise).
  • Q2: “From the Appendix, I understand that the subsets are selected by evaluating only C=20 random candidates. How is this number selected?” We ablate the value of C in appendix E.2, highlighting that this hyperparameter does not materially affect outcomes beyond C=20. As such, we choose 20 for computational efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.