acceptodds
Under review as a conference paper at ICLR 2027

ABC: Certified Selective Evaluation with a Panel of LLM Judges

Abstract

LLM-as-a-judge has reshaped evaluation by reducing the human effort required to assess response quality, yet LLM judgments can disagree with human ratings. More capable models may narrow this gap, but they can also make the evaluation substantially more expensive. In practice, users may therefore rely on a panel of affordable judges and, for each response, decide whether to accept the panel's judgment or escalate it to a stronger model or a human. This raises two questions: how should the judges' scores be combined, and which resulting judgments can be accepted automatically at a controlled error rate. In this paper, we propose the ABC panel: Admit, Blend, and Certify. ABC constructs a statistical test based on cross-fitted residual association, admitting judges whose scores remain associated with human ratings beyond observed surface features, such as response length and style. It then blends the admitted judges using penalized logistic regression fitted to human ratings. Finally, it selects and certifies an acceptance rule such that, with a high probability, the population error rate among admitted judgments is below a threshold. Experiments on extensive human-rated benchmarks show that the ABC panel automates a larger share of evaluation than single-judge and ensemble baselines while controlling the error rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, selective evaluation, risk control, judge aggregation, double machine learning, human evaluation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_BwDUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the core problem in Large Language Model Evaluation (LLM-as-a-judge): "How to combine multiple inexpensive evaluation models and provide statistically guaranteed selective automated scoring within a limited manual annotation budget?" It proposes an end-to-end ABC panel (Admit, Blend, Certify) framework. Specifically, the authors divide manually annotated data into training and certification sets, and construct a selective evaluation system through three steps:

  • Step A (Admit): Based on Robinson residual construction and K-fold cross-fitting using dual machine learning (DML), a conditional correlation coefficient ρj\rho_j is defined after deducting 76-dimensional surface features (such as length, format, readability, etc.). Bonferroni-corrected confidence interval tests are used to eliminate "surface judges" that rely solely on surface features.

  • Step B (Blend): A two-stage logistic regression is employed. The first stage uses ridge-penalized logistic regression to obtain the out-of-division predicted score h^(−k(i))(Xi)\widehat{h}^{(-k(i))}(X_i) for surface features. The second stage uses ℓ1\ell_1 penalized logistic regression to fuse the standardized scores of the admission judges with the surface feature predictions into a single panel score η(ZA,X)\eta(Z_{\mathcal{A}}, X).

  • Step C (Certify): On an independent certification set, fixed-sequence testing is used to solve for the minimum rejection threshold τ^\widehat{\tau} that satisfies the risk control condition (ε,δ)(\varepsilon, \delta) (i.e., the probability that the error rate of the accepted score exceeds ε\varepsilon is at most δ\delta). The starting level l0l_0 of the fixed-sequence testing is adaptively selected using bootstrap on the training set to maximize automated coverage.

Clustering Dependency Extension and Benchmark Contribution: Addressing the intra-cluster dependency problem (Kmax⁡>1K_{\max} > 1) caused by multiple responses to the same prompt, the authors elevate sample partitioning, cross-fit standard error calculation, and hypothesis testing in step C to the prompt level. They introduce a game-theoretic pp value (Betting pp-value) based on non-negative martingale and Ville inequality to replace the exact binomial test, ensuring statistical validity with limited samples. Experiments on the BiGGen Bench, HelpSteer2, and Chatbot Arena datasets validate that the ABC panel has significantly higher coverage than the best single judge, majority voting, and judge cascade under the same error rate constraint. The authors also publicly release a newly collected multi-model judging score benchmark.

Critical strengths

Statistical safety and robustness of the certification step (Step C): Theorems 1 and 2 prove that the (ε,δ)(\varepsilon, \delta)-risk control of Step C is completely independent of the accuracy of the admission test in Step A and the correctness of the regression model specification in Step B. Furthermore, using the training set bootstrap to select the starting index l0l_0 for the fixed-sequence test is a clever engineering and statistical innovation: it avoids the inability to reject the null hypothesis due to an excessively small sample size mlm_l at extremely high thresholds, while strictly adhering to the principle of not prying into the certification set data, thus significantly improving the test power (coverage) without sacrificing validity.

Critical weaknesses
  • The baseline selection fails to cover the most advanced adaptive multi-expert/multi-judge selection schemes:

    • The paper only compared the best single judge, majority voting, and simple cascading (Cascade, Jung et al., 2025) in its experiments. However, as mentioned in its references, a large number of advanced schemes have emerged recently (related conferences in 2026) such as multi-expert conformal risk control (e.g., Cheng et al., 2026) or selective evaluation with uncertainty guards (SCOPE, Badshah et al., 2026b). The lack of direct comparison with these state-of-the-art multi-model collaboration, dynamic routing, or risk control frameworks makes it difficult for readers to assess the absolute technological advantage of the ABC panel.
  • Lack of Ablation Studies for Each Component of Step A (Admit) and Step B (Blend):

    • Section 3.2 of the text states that the main purpose of Step A is to eliminate surface judges before they are randomly selected in Step B. However, the second stage of Step B itself explicitly introduces the surface feature prediction h^(−k(i))(Xi)\widehat{h}^{(-k(i))}(X_i) and applies a ℓ1\ell_1 sparse penalty (Lasso) to the judge weights. From a statistical learning perspective, given that h(X)h(X) is controlled, the ℓ1\ell_1 penalty itself has the ability to eliminate conditionally irrelevant variables.

    • The paper currently only compares the baselines of the aggregation rules for replacing step B (single judge, majority voting, cascading), but does not provide ablation variant experiments: (1) removing step A and directly inputting all JJ judges into step B (i.e., the "BC panel"); (2) removing the surface feature term θhh^(X)\theta_h \widehat{h}(X) in step B; (3) not using Bootstrap selection l0l_0 in step C (e.g., fixing l0=1l_0=1 or using Bonferroni correction). The lack of these ablation experiments makes it difficult for readers to quantify the specific contributions of step A, surface feature calibration, and Bootstrap l0l_0 to the final coverage.

Questions for authors
  • Regarding the comparison with recent multi-expert methods: "Recent multi-expert conformal risk control (e.g., Cheng et al., 2026) and selective pairwise evaluation (e.g., Badshah et al., 2026b) have also achieved risk control for multiple evaluators. Under the same data partitioning and the same ε\varepsilon constraint, what are the specific advantages and disadvantages of ABC panels compared to these methods in terms of coverage and computational complexity? Why weren't they included as baselines in Table 1 of the experiments?"
  • Ablation contribution of each module: If step A is skipped (all judges are sent directly to step B with ℓ1\ell_1 penalty), or if the surface feature scores θhh^(X)\theta_h \widehat{h}(X) are removed in step B, how would the certified coverage and test error rate on the three benchmarks in Table 1 change? How many judges does the ℓ1\ell_1 regularization in step B ultimately assign non-zero weights (especially those models that were on the admission edge in step A, such as Qwen2.5-0.5B on Chatbot Arena)?
Reviewer_yfoUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ABC, a framework for building reliable panels of LLM judges. ABC filters out judges that provide little information beyond superficial response features. ABC combines useful judges and selectively accepts only high-confidence evaluations. The proposed method can provide statistical guarantees on the error rate while substantially increasing automated evaluation coverage.

Critical strengths
  1. To my knowledge, the problem studied in this paper is practical and novel

  2. The proposed Admit–Blend–Certify framework is clean, intuitive, and easy to understand.

  3. This paper provides a meaningful finite-sample guarantee for selective multi-judge evaluation.

  4. The experiments consistently show substantial coverage gains over single-judge and ensemble baselines.

Critical weaknesses
  1. The necessity of Step A beyond the sparsity-inducing Blend step is not sufficiently established.

  2. The definition of a “surface judge” depends strongly on the predefined surface-feature set.

  3. The guarantee relies on an in-distribution certification sample and does not cover distribution shift or subgroup risk.

  4. The method requires substantial human-labeled training and certification data, but the overall human-cost tradeoff is not thoroughly studied.

Questions for authors

Please see the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.