acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Seek Human Guidance for Best LLM Judge Identification

Abstract

LLM judges make it possible to evaluate model outputs at a scale that would be costly to achieve through human review alone. Before deploying a judge at scale, however, practitioners need to determine whether its judgments agree with human evaluations on the task of interest. Once a reliable judge is identified, it can support large-scale evaluation with substantially less human supervision. The main bottleneck is therefore the human labeling required to distinguish among candidate judges. This problem differs from classical best-arm identification because each human label simultaneously reveals the correctness of all candidate judges on the same item. As a result, the value of labeling an item depends on how much it helps distinguish the judges that are most difficult to separate. Overall disagreement alone does not capture this information, since disagreement among clearly inferior judges has little relevance to identifying the best judge. This shared-information structure motivates Weighted Pairwise Discrepancies (WPD), which assigns higher sampling priority to items that are informative for the most consequential pairwise comparisons. Because such selective labeling changes the distribution of reviewed items, we also develop valid inference for judge accuracies and their differences under the learned allocation. Across simulations and five real-data evaluation panels, WPD improves the probability of correct selection, reduces the number of human labels required to identify the best judge, and maintains valid uncertainty quantification.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM-as-a-judge, large deviations, adaptive sampling, statistical inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qh2ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the problem of identifying the LLM judge whose judgments agree most closely with human judgments, given a limited budget for human annotations. The authors propose a two-stage algorithm. They first randomly sample a small number of human labels to estimate certain quantities, and then optimize the sampling weights to maximize the probability of correct selection (PCS). The proposed method is supported by theoretical guarantees.

Critical strengths

Strong motivation. The problem is well motivated and relevant to practical LLM evaluation, where human annotations are expensive and selecting a reliable LLM judge is important.

Theoretical guarantees. The proposed method comes with theoretical guarantees, which is an important strength of the paper.

Critical weaknesses

The method can identify the best LLM among the candidates, but it does not tell us whether the selected LLM is actually useful. Even the best LLM may perform poorly.

I wonder whether the paper is essentially applying standard techniques, such as active learning, to a new application. It is not clear to me what the methodological novelty is beyond the application itself.

I could not follow Section 3.2. In particular, it is unclear to me how the results are derived.

Questions for authors

Since the loss is formulated using a minimax criterion, what happens if some very poor judges are included? What happens if all judges have similar performance? I think it is important to understand how the distribution of judge quality affects the method. Are there any interesting theoretical or empirical phenomena under different configurations?

Can the proposed method be extended to a sequential setting? Would a sequential approach lead to better performance or sample efficiency?

2 comments

Thanks for the response. My concerns have been solved.

Thank you for the thoughtful review. We address your comments below.

1. Practical usefulness of the selected judge (W1)

We agree that even the best judge may not be sufficiently accurate for deployment. An advantage of our framework is that, beyond identifying the best judge, we can also estimate each judge's accuracy and quantify the estimation uncertainty. Theorem 6.3 establishes joint asymptotic normality and consistent covariance estimation, allowing us to construct confidence bounds for judge accuracies that remain asymptotically valid for the selected judge. In practice, one possible approach is to specify a minimum acceptable accuracy and use the selected judge's lower confidence bound to assess whether it exceeds this threshold. If the threshold cannot be certified, one can consider collecting more human labels or improving the candidate judges. Thus, our framework can support assessments of practical reliability in addition to best-judge identification.

2. Methodological novelty (W2)

WPD differs from existing best-arm identification and active learning methods in information structure and labeling objective. Unlike best-arm identification, we select items for human labeling rather than individual judges: one human label reveals the correctness of every candidate judge on that item, creating shared information. Unlike standard active learning, which typically collects labels to train a predictive model, we maximize the probability of correct selection (PCS). We derive WPD from a large-deviation approximation to PCS under this shared-information structure, yielding a novel labeling rule based on weighted pairwise discrepancies. Together, these establish a novel human-labeling and inference framework for best-judge identification, beyond standard best-arm identification and active learning.

3. Derivation in Section 3.2 (W3)

The WPD allocation rule follows from optimizing our approximation to the probability of correct selection (PCS). For each challenger to the best judge, the misranking probability depends on its accuracy gap and the variance of the corresponding pairwise comparison. Since selecting the best judge requires winning all pairwise comparisons, we optimize the hardest one. The equivalent convex formulation and KKT conditions then yield the square-root labeling rule based on weighted pairwise discrepancies. Thus, WPD prioritizes items that are most informative for identifying the best judge. The complete derivation of the WPD allocation rule is provided in Appendix A.1 (Proof of Lemma 6.1), which we will explicitly reference in the revision of Section 3.2.

4. Poor and similarly performing judges (Q1)

WPD maintains its advantage over Uniform when inferior judges are added or when judge accuracies are similar. When inferior judges are added, their large accuracy gaps generally prevent them from becoming the hardest comparisons, so WPD continues to prioritize the competitive judges. When judge accuracies are similar, WPD uses weighted pairwise discrepancies to prioritize items that distinguish closely matched judges. We tested both settings in synthetic simulations:

Judge-quality configurationWPD PCSUniform PCS
Original 10 judges87.9%79.7%
+5 inferior judges88.1%81.7%
+10 inferior judges87.9%82.7%
+20 inferior judges88.6%82.7%
Similar judges, 2 pp range37.4%35.1%
Similar judges, 5 pp range56.9%53.5%
Similar judges, 10 pp range75.9%70.6%

WPD maintains about 88% PCS even after adding 20 inferior judges. For similarly accurate judges, WPD outperforms Uniform at each tested accuracy range, and its PCS rises from 37.4% to 75.9% as the range widens from 2 to 10 percentage points. These results demonstrate WPD's effectiveness across different judge-quality configurations.

5. Sequential extension (Q2)

WPD naturally extends to sequential labeling by re-estimating the best judge and adjusting labeling probabilities as new human labels become available. We evaluated this extension across five datasets and 15 dataset–budget configurations, comparing two-stage WPD with four-round sequential WPD at the same initial labeling batch size and total labeling budget. Compared with two-stage WPD, four-round sequential WPD increased average PCS by 2.35 and 1.03 percentage points for initial labeling batches of 25 and 50 human labels, respectively, while decreasing it by 0.38 percentage points for 100 labels. These results show that additional labeling rounds can improve PCS, with benefits depending on hyperparameter choices. We will discuss this sequential extension in more detail in the revision.

Reviewer_hXuMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WPD (Weighted Pairwise Discrepancies); an adaptive human labeling framework to automatically select which data points are most efficient to label for selecting best possible judge that can be deployed at scale. The core idea is to use item labels where judges disagree as that can help distinguish the quality of the judges.

The main method has 2 stages - a pilot sampling to get the best possible provisional judge at that stage along with gaps between other judges by using a small human-labeled budget; followed by optimized sampling stage which uses these gaps along with remaining human-labeling budget to sample remaining items with some probability for human-labeling. All labeled items are then combined appropriately to estimate the best judge that can be deployed for large scale inference. The authors have performed both simulated and real experiments on 5 datasets showing better performance compared to previous methods.

Critical strengths
  • The problem is motivating and important enough which has several industry applications when viewed from different lenses assuming LLMs are used for both data annotations and evaluations.
    • Eg: Crowdsourcing platforms such as Amazon's Mechnanical Turk use techniques from active learning, threshold based auto labeling systems, so any improvement in choosing the labels means saving both time and cost!! Whether the method is for choosing best LLM-Judge for evaluation deployment or gathering datasets for training; I felt the underlying domain changes but the motivation stays mostly the same i.e human labeling and time is bottleneck even with LLMs.
  • The paper is written neatly, easy to follow-through at most places (see Questions, On formatting)
  • Experiments that were designed are well executed and clearly presented.
Critical weaknesses
  • (Discussion on pre-available LLM-Judge costs) The judge predictions are assumed to be free while it takes N x M LLM-calls in real-world. This cost needs to be discussed further. If uniform doesn't require judge predictions (L365-366) it creates asymmetry and needs to be discussed to understand the trade-offs.
  • (Experiment on Ablations) Ablations to understand each component's/stage and hyperparameter importance for the method. For eg:
    • How often does the judge chosen during pilot stage be the final best judge setting - distinguishes the contribution of pilot stage to stage 2 and whether a bad judge chosen at pilot stage can be recovered at optimized sampling stage - something like understanding the top-k provisioned judges for final best judge.
    • Feel-free to choose the ones that are possible to run and important to understand the importance/contribution of each stage and hyperparameter; not specifically the above ones. For eg: B, B1 hyperparameters are discussed in detal; so something like that for others to strengthen the paper further.
  • (Discussion, Potential toy experiments) Previous research [1, 2] has shown that LLMs have and share biases and that impacts their overall quality when we use these as judges. I am wondering, in this setup because we pick the items to label from those (shared biased) judge labels (i.e judge labels are available from datasets prior), I would like to hear more from authors on how this will impact the quality of the final judge selection, how the items to be selected for human-labeling might vary based on choice of LLMs, whether modeling and removing these shared bias terms will have any impact etc;
  • (Discussion, Optional potential experiment on new dataset) I believe the method works for numerical and ordinal labels as well (I've read Sec 5 on extension to non-binary labels). While LLMJudge dataset is 4-level (L421-422), it uses exact-agreement (Appendix C.1) thus I am not sure if it's technically ordinal. Just interested to understand if there needs to be any extensions made to the method to accommodate these types of other labeling and any potential datasets to consider to strengthen the paper.
  1. CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (ICML 2026): https://openreview.net/forum?id=3WPDFjZ1UT
  2. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025): https://openreview.net/forum?id=3GTtZFiajM
Questions for authors

On formatting

  • Sec 1.1: I felt 2nd para needs some revision as it is slightly difficult to follow the motivation clearly.
  • L84: Expand PCS before it's first usage.
  • What values of ϵ\epsilon and κ\kappa were used, and how sensitive is PCS to κ\kappa?
    • Because there are lot of notations in the paper; it would be beneficial to have an Notations table in Appendix with default values for easier cross referencing.
  • Fig 1 has "Pilot Sampling", "Optimized Sampling" but Sec 4 uses "Pilot labeling" and "Optimized labeling". Also, there's Step 1, 2, 3 in Fig 1 and Stage 1, 2 in Sec 4.

Expanding related literature: Maybe in Appendix if the authors find some of these useful as adjacent literature: on Active learning/active testing [1, 2], Optimal Computing Budget Allocation; OCBA [3], PPI-style evaluation with LLM judges [4, 5], and auto-labeling domain [6, 7] among many others.

  1. Active Testing: Sample-Efficient Model Evaluation (ICML 2021): https://arxiv.org/abs/2103.05331
  2. Active statistical inference (ICML 2024): https://arxiv.org/abs/2403.03208
  3. Simulation Budget Allocation for Further Enhancing the Efficiency of Ordinal Optimization, (Discrete Event Dynamic Systems): https://link.springer.com/article/10.1023/A:1008349927281
  4. Prediction-Powered Inference (Science): https://arxiv.org/abs/2301.09633
  5. AutoEval Done Right: Using Synthetic Data for Model Evaluation (ICML 2025): https://arxiv.org/abs/2403.07008
  6. MCAL: Minimum Cost Human-Machine Active Labeling (ICLR 2023): https://arxiv.org/abs/2006.13999
  7. Pearls from Pebbles: Improved Confidence Functions for Auto-labeling (NeurIPS 2024): https://arxiv.org/abs/2404.16188
2 comments

Thank you for the thoughtful review. We address your comments below.

1. Cost of judge predictions (W1)

In our setting, the complete NM candidate LLM judge prediction matrix is available prior to human labeling, as is common in offline evaluation. Our goal is hence to maximize the chance of correctly selecting the best judge under a limited human-labeling budget. Although LLM inference is not free, both WPD and Uniform labeling use the same complete judge prediction matrix in our experiments (L365–366), so their inference costs are the same. The distinction lies in the human-label allocation rule: Uniform selects items randomly, whereas WPD uses the available LLM judge predictions to identify the items most informative for distinguishing the leading judges. Our comparisons therefore use the same prediction information and human-label budget, and the PCS improvements directly demonstrate the benefit of WPD's optimized allocation. We will clarify this in the revision.

2. Pilot and hyperparameters (W2; hyperparameter question)

a. Incorrect pilot. The pilot estimates a provisional best judge and accuracy gaps, but the optimized second stage is not restricted to the pilot selection. We conducted additional experiments to examine how often the second stage recovers the correct best judge when the pilot selects an incorrect leader. The results are summarized below:

DatasetPilot correctWPD recoveryUniform recovery
RewardBench32.8%98.8%64.4%
LLMJudge12.2%88.5%70.8%
Defect detection31.4%78.7%61.8%
Arena1740.0%75.8%68.7%

These results demonstrate that WPD can reliably recover from incorrect pilot selections. In particular, the optimized second stage substantially improves the final selection even when the pilot identifies an incorrect leading judge.

b. Hyperparameter configurations and sensitivity. We set the gap-stabilization parameter κ=0.02\kappa=0.02. For fixed-size sampling, the sampling-probability lower bound is ϵ=0.05(B−B1)/(N−B1)\epsilon = 0.05(B-B_1)/(N-B_1), where (B−B1)/(N−B1)(B-B_1)/(N-B_1) is the second-stage labeling rate. We also examined how sensitive WPD's PCS is to κ\kappa:

κ\kappaRewardBench PCSLLMJudge PCS
0.00597.33%88.1%
0.01098.00%88.1%
0.020 (default)98.97%87.9%
0.04099.70%88.0%
0.08096.63%84.7%

WPD's PCS remains stable as varies from 0.01 to 0.04, with slightly lower performance at =0.08. Additional random exploration did not improve recovery, supporting our targeted WPD labeling.

3. Shared judge biases (W3)

We agree that LLM judges can have and share biases in their judgments. One advantage of WPD is that it does not assume independent LLM judge errors; instead, it uses their disagreement structure to save human labeling. Consider the case where two LLM judges make the same prediction: even if both are biased, that item cannot distinguish their relative accuracy; WPD instead prioritizes disagreements that matter for identifying the best judge. The allocation rule adapts naturally to the choice of candidate judges, since different candidate sets yield different pairwise disagreement patterns. Final accuracy estimates use human labels rather than assuming shared judge predictions are correct. In our correlated binary simulation at budget B=1,000, WPD achieved 96.8% PCS, compared with 89.7% for Uniform, 90.3% for Hara, and 91.9% for CODA. Five-class experiments also include shared errors and a shared-wrong-label bloc. Thus, WPD retains its advantage under the shared-error structures tested. Explicit bias correction could change the candidate predictions and hence the allocation, but is not needed to identify the most accurate judge among the given candidates.

4. Ordinal and numerical labels (W4)

We agree that it is valuable to evaluate ordinal labels beyond the exact-agreement score used in our original experiments. Section 5 supports general bounded scores ℓ(g,y)∈[0,1]\ell(g,y)\in[0,1], including ordinal and numerical scores. Since our original LLMJudge experiment used exact agreement, we conducted an additional experiment on the same four-level relevance dataset using the ordinal-aware score ℓ(g,y)=1−∣g−y∣/3\ell(g,y)=1-|g-y|/3. The results are summarized below:

MethodB=500B=1,100B=2,000
WPD conservative49%73%97%
WPD imputation42%72%96%
Uniform42%62%88%

Both WPD variants outperform Uniform at moderate and large budgets, with WPD conservative performing slightly better than imputation empirically. These results demonstrate that our framework accommodates ordinal scoring without requiring a new allocation method.

We thank the reviewer for the helpful suggestions on presentation, notation, and related work. We will incorporate them in the revision.

Reviewer_hXuMOpenReview ↗

Reply to rebuttal

Hi authors,

Thanks a lot for the rebuttal. I believe all my questions are addressed to a good extent and would like to see the final PDF version for any suggestions. In ICLR, we can update the paper, so if you get a chance, please update the PDF so it'll be easy to check the details on all the points.

All the very best with your submission.

Reviewer_4cr6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies identifying the most accurate judge on a fixed item pool with limited reference labels. WPD allocates labels using gap-weighted pairwise discrepancies, then uses inverse-probability estimation and a joint covariance estimator for inference. Simulations and five complete evaluation panels show improved correct-selection probabilities and label efficiency.

Critical strengths

S1. The acquisition criterion is well aligned with best-judge identification. It prioritizes comparisons that can determine the winner, and Figure 5 connects this motivation to observed allocation behavior.

S2. The paper addresses estimation as well as acquisition. The two-stage estimator and joint covariance analysis explicitly account for shared labels and the learned allocation, with clearly stated asymptotic conditions.

S3. The empirical comparison includes relevant active model-selection baselines and several error structures and task types. Label savings include the pilot cost.

Critical weaknesses

W1. Finite-pilot robustness in the near-tie regime needs more direct evidence. Assumption 1 requires accurate pilot gap estimation, while Section 7 uses a main pilot of 100 and some real panels have very small full-pool gaps. The pilot-size sensitivity is useful, but PCS conditional on whether the pilot identifies the correct leader would better expose the failure regime.

W2. The scope of uncertainty quantification should be more explicit. Section 4 and Theorem 6.3 establish marginal and prespecified pairwise Wald inference. These results do not by themselves provide simultaneous coverage for claims that the adaptively selected winner outperforms every competitor. This distinction matters for the deployment motivation.

W3. The budget is an expected budget. The Bernoulli second-stage design in Section 4 can exceed a hard annotation limit in an individual run, whereas Figure 4 summarizes efficiency through expected labels. Reporting realized-budget variability or a justified hard-budget implementation would strengthen the practical claim.

Questions for authors

Q1. How much of the final selection error comes from runs in which the pilot identifies the wrong leading judge, and can additional exploration recover from these runs?

Q2. Are the reported intervals intended only for prespecified contrasts, or also as certificates for the selected winner? Can the latter use be evaluated directly?

Q3. What are the realized annotation-count quantiles at the reported expected budgets, and how would the estimator change under a strict cap?

2 comments

Thank you for the thoughtful review. We address your questions below.

1. Finite-pilot robustness and recovery (W1/Q1)

a. Incorrect pilot. Thanks for raising this concern! Assumption 1 is introduced to establish the asymptotic convergence of the learned WPD allocation to the oracle allocation. It does not require accurate gap estimates or correct identification of the best judge in the finite pilot.

In our experiments, with a pilot of 100 expert human labels, the pilot often selects the wrong judge as the leader when the true gap between the best and second-best judges is small. We conducted additional experiments and found that our WPD method still has a much higher recovery rate in the second round than the Uniform baseline, conditional on the pilot selecting the wrong best judge. We summarize the results in the following table:

DatasetPilot correctWPD recoveryUniform recoveryWPD−Uniform (95% MC interval, pp)
RewardBench32.8%98.8%64.4%[32.3, 36.5]
LLMJudge12.2%88.5%70.8%[14.6, 20.7]
Defect detection31.4%78.7%61.8%[13.1, 20.7]
Arena1740.0%75.8%68.7%[3.5, 10.8]

These results demonstrate that WPD can reliably recover from an incorrect pilot selection.

b. Contribution of incorrect pilots. We further examined the contribution of incorrect pilots to final selection errors. Across the four datasets, incorrect-pilot runs account for 77.4%, 83.5%, 74.9%, and 62.5% of the remaining selection errors, respectively. Importantly, these errors represent only a small fraction of the initially incorrect pilot decisions in several settings.

Thus, while pilot errors account for a substantial share of the residual selection errors, WPD corrects the vast majority of incorrect pilot decisions in these settings. Additional random exploration did not improve pilot-error recovery in our experiments. The original WPD allocation already incorporates 5% exploration and achieves strong recovery rates when the pilot identifies an incorrect leader.

c. Near-tie situation. We also added a near-tie simulation regime where the accuracy gap between the best and second-best judges is only 0.25%. In this setting, WPD achieved 68.0% PCS at a total budget of 1,000, versus 61.9% for Uniform, 60.9% for Hara, and 61.8% for CODA. Thus, WPD retains an advantage even under the near-tie regime.

2. Selected-winner inference (W2/Q2)

Our reported 95% intervals are for prespecified judges and pairwise contrasts. For the adaptively selected winner, Theorem 6.3's joint asymptotic normality and consistent covariance estimator also support simultaneous inference.

We can construct Bonferroni-adjusted 95% confidence intervals for all M(M-1)/2 pairwise accuracy differences and certify the selected judge only when the lower confidence bound against each competitor is positive. This controls the asymptotic familywise risk of falsely certifying a winner, including after selection.

We further verified this procedure in an additional simulation, observing 97.5% simultaneous coverage, close to the nominal 95% level. These results demonstrate that our joint inference framework supports statistically valid selected-winner certification, extending beyond the marginal and pairwise intervals illustrated in the original paper.

3. Realized budgets and strict caps (W3/Q3)

We measured the actual number of human labels used (including the pilot) on RewardBench across Monte Carlo repetitions:

Expected label budgetMean (SD) labels95th / 99th percentilesP(realized budget>1.05expected budget)
200200.01 (8.96)215 / 22212.0%
1,2001199.83 (16.52)1227 / 1238≈0%
2,2002200.04 (9.72)2215 / 2222≈0%

The results show that the realized number of labels is concentrated around the expected budget, with no substantial budget overruns observed at budgets of 1,200 and 2,200 labels. For a strict budget cap, we can use fixed-size stratified sampling without replacement to ensure that exactly B human labels are collected.

Under this design, the adjusted accuracy estimator uses population-weighted stratum sample means instead of Bernoulli inverse-probability weighting, with a corresponding sampling-without-replacement variance estimator (Horvitz and Thompson, 1952).

Reference

Horvitz, D. G., & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685.

Thank you for the detailed response and additional experiments. My overall assessment remains positive.

Moreover, I encourage expanding the related-work discussion on active statistical inference [1], robust sampling [2], prediction-powered ranking [3], and calibration via human ranking [4]. I would appreciate seeing these additions incorporated into the final version.

[1] Active Statistical Inference. ICML'24.

[2] Robust Sampling for Active Statistical Inference. NeurIPS'25.

[3] Prediction-Powered Ranking of Large Language Models, NeurIPS'24.

[4] AtC: Aggregate-then-Calibrate for Human-centered Assessment. ICLR'26.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.