- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of identifying the LLM judge whose judgments agree most closely with human judgments, given a limited budget for human annotations. The authors propose a two-stage algorithm. They first randomly sample a small number of human labels to estimate certain quantities, and then optimize the sampling weights to maximize the probability of correct selection (PCS). The proposed method is supported by theoretical guarantees.
Critical strengths
Strong motivation. The problem is well motivated and relevant to practical LLM evaluation, where human annotations are expensive and selecting a reliable LLM judge is important.
Theoretical guarantees. The proposed method comes with theoretical guarantees, which is an important strength of the paper.
Critical weaknesses
The method can identify the best LLM among the candidates, but it does not tell us whether the selected LLM is actually useful. Even the best LLM may perform poorly.
I wonder whether the paper is essentially applying standard techniques, such as active learning, to a new application. It is not clear to me what the methodological novelty is beyond the application itself.
I could not follow Section 3.2. In particular, it is unclear to me how the results are derived.
Questions for authors
Since the loss is formulated using a minimax criterion, what happens if some very poor judges are included? What happens if all judges have similar performance? I think it is important to understand how the distribution of judge quality affects the method. Are there any interesting theoretical or empirical phenomena under different configurations?
Can the proposed method be extended to a sequential setting? Would a sequential approach lead to better performance or sample efficiency?
2 comments
Thank you for the thoughtful review. We address your comments below.
1. Practical usefulness of the selected judge (W1)
We agree that even the best judge may not be sufficiently accurate for deployment. An advantage of our framework is that, beyond identifying the best judge, we can also estimate each judge's accuracy and quantify the estimation uncertainty. Theorem 6.3 establishes joint asymptotic normality and consistent covariance estimation, allowing us to construct confidence bounds for judge accuracies that remain asymptotically valid for the selected judge. In practice, one possible approach is to specify a minimum acceptable accuracy and use the selected judge's lower confidence bound to assess whether it exceeds this threshold. If the threshold cannot be certified, one can consider collecting more human labels or improving the candidate judges. Thus, our framework can support assessments of practical reliability in addition to best-judge identification.
2. Methodological novelty (W2)
WPD differs from existing best-arm identification and active learning methods in information structure and labeling objective. Unlike best-arm identification, we select items for human labeling rather than individual judges: one human label reveals the correctness of every candidate judge on that item, creating shared information. Unlike standard active learning, which typically collects labels to train a predictive model, we maximize the probability of correct selection (PCS). We derive WPD from a large-deviation approximation to PCS under this shared-information structure, yielding a novel labeling rule based on weighted pairwise discrepancies. Together, these establish a novel human-labeling and inference framework for best-judge identification, beyond standard best-arm identification and active learning.
3. Derivation in Section 3.2 (W3)
The WPD allocation rule follows from optimizing our approximation to the probability of correct selection (PCS). For each challenger to the best judge, the misranking probability depends on its accuracy gap and the variance of the corresponding pairwise comparison. Since selecting the best judge requires winning all pairwise comparisons, we optimize the hardest one. The equivalent convex formulation and KKT conditions then yield the square-root labeling rule based on weighted pairwise discrepancies. Thus, WPD prioritizes items that are most informative for identifying the best judge. The complete derivation of the WPD allocation rule is provided in Appendix A.1 (Proof of Lemma 6.1), which we will explicitly reference in the revision of Section 3.2.
4. Poor and similarly performing judges (Q1)
WPD maintains its advantage over Uniform when inferior judges are added or when judge accuracies are similar. When inferior judges are added, their large accuracy gaps generally prevent them from becoming the hardest comparisons, so WPD continues to prioritize the competitive judges. When judge accuracies are similar, WPD uses weighted pairwise discrepancies to prioritize items that distinguish closely matched judges. We tested both settings in synthetic simulations:
| Judge-quality configuration | WPD PCS | Uniform PCS |
|---|---|---|
| Original 10 judges | 87.9% | 79.7% |
| +5 inferior judges | 88.1% | 81.7% |
| +10 inferior judges | 87.9% | 82.7% |
| +20 inferior judges | 88.6% | 82.7% |
| Similar judges, 2 pp range | 37.4% | 35.1% |
| Similar judges, 5 pp range | 56.9% | 53.5% |
| Similar judges, 10 pp range | 75.9% | 70.6% |
WPD maintains about 88% PCS even after adding 20 inferior judges. For similarly accurate judges, WPD outperforms Uniform at each tested accuracy range, and its PCS rises from 37.4% to 75.9% as the range widens from 2 to 10 percentage points. These results demonstrate WPD's effectiveness across different judge-quality configurations.
5. Sequential extension (Q2)
WPD naturally extends to sequential labeling by re-estimating the best judge and adjusting labeling probabilities as new human labels become available. We evaluated this extension across five datasets and 15 dataset–budget configurations, comparing two-stage WPD with four-round sequential WPD at the same initial labeling batch size and total labeling budget. Compared with two-stage WPD, four-round sequential WPD increased average PCS by 2.35 and 1.03 percentage points for initial labeling batches of 25 and 50 human labels, respectively, while decreasing it by 0.38 percentage points for 100 labels. These results show that additional labeling rounds can improve PCS, with benefits depending on hyperparameter choices. We will discuss this sequential extension in more detail in the revision.
Thanks for the response. My concerns have been solved.