- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the core problem in Large Language Model Evaluation (LLM-as-a-judge): "How to combine multiple inexpensive evaluation models and provide statistically guaranteed selective automated scoring within a limited manual annotation budget?" It proposes an end-to-end ABC panel (Admit, Blend, Certify) framework. Specifically, the authors divide manually annotated data into training and certification sets, and construct a selective evaluation system through three steps:
-
Step A (Admit): Based on Robinson residual construction and K-fold cross-fitting using dual machine learning (DML), a conditional correlation coefficient is defined after deducting 76-dimensional surface features (such as length, format, readability, etc.). Bonferroni-corrected confidence interval tests are used to eliminate "surface judges" that rely solely on surface features.
-
Step B (Blend): A two-stage logistic regression is employed. The first stage uses ridge-penalized logistic regression to obtain the out-of-division predicted score for surface features. The second stage uses penalized logistic regression to fuse the standardized scores of the admission judges with the surface feature predictions into a single panel score .
-
Step C (Certify): On an independent certification set, fixed-sequence testing is used to solve for the minimum rejection threshold that satisfies the risk control condition (i.e., the probability that the error rate of the accepted score exceeds is at most ). The starting level of the fixed-sequence testing is adaptively selected using bootstrap on the training set to maximize automated coverage.
Clustering Dependency Extension and Benchmark Contribution: Addressing the intra-cluster dependency problem () caused by multiple responses to the same prompt, the authors elevate sample partitioning, cross-fit standard error calculation, and hypothesis testing in step C to the prompt level. They introduce a game-theoretic value (Betting -value) based on non-negative martingale and Ville inequality to replace the exact binomial test, ensuring statistical validity with limited samples. Experiments on the BiGGen Bench, HelpSteer2, and Chatbot Arena datasets validate that the ABC panel has significantly higher coverage than the best single judge, majority voting, and judge cascade under the same error rate constraint. The authors also publicly release a newly collected multi-model judging score benchmark.
Critical strengths
Statistical safety and robustness of the certification step (Step C): Theorems 1 and 2 prove that the -risk control of Step C is completely independent of the accuracy of the admission test in Step A and the correctness of the regression model specification in Step B. Furthermore, using the training set bootstrap to select the starting index for the fixed-sequence test is a clever engineering and statistical innovation: it avoids the inability to reject the null hypothesis due to an excessively small sample size at extremely high thresholds, while strictly adhering to the principle of not prying into the certification set data, thus significantly improving the test power (coverage) without sacrificing validity.
Critical weaknesses
-
The baseline selection fails to cover the most advanced adaptive multi-expert/multi-judge selection schemes:
- The paper only compared the best single judge, majority voting, and simple cascading (Cascade, Jung et al., 2025) in its experiments. However, as mentioned in its references, a large number of advanced schemes have emerged recently (related conferences in 2026) such as multi-expert conformal risk control (e.g., Cheng et al., 2026) or selective evaluation with uncertainty guards (SCOPE, Badshah et al., 2026b). The lack of direct comparison with these state-of-the-art multi-model collaboration, dynamic routing, or risk control frameworks makes it difficult for readers to assess the absolute technological advantage of the ABC panel.
-
Lack of Ablation Studies for Each Component of Step A (Admit) and Step B (Blend):
-
Section 3.2 of the text states that the main purpose of Step A is to eliminate surface judges before they are randomly selected in Step B. However, the second stage of Step B itself explicitly introduces the surface feature prediction and applies a sparse penalty (Lasso) to the judge weights. From a statistical learning perspective, given that is controlled, the penalty itself has the ability to eliminate conditionally irrelevant variables.
-
The paper currently only compares the baselines of the aggregation rules for replacing step B (single judge, majority voting, cascading), but does not provide ablation variant experiments: (1) removing step A and directly inputting all judges into step B (i.e., the "BC panel"); (2) removing the surface feature term in step B; (3) not using Bootstrap selection in step C (e.g., fixing or using Bonferroni correction). The lack of these ablation experiments makes it difficult for readers to quantify the specific contributions of step A, surface feature calibration, and Bootstrap to the final coverage.
-
Questions for authors
- Regarding the comparison with recent multi-expert methods: "Recent multi-expert conformal risk control (e.g., Cheng et al., 2026) and selective pairwise evaluation (e.g., Badshah et al., 2026b) have also achieved risk control for multiple evaluators. Under the same data partitioning and the same constraint, what are the specific advantages and disadvantages of ABC panels compared to these methods in terms of coverage and computational complexity? Why weren't they included as baselines in Table 1 of the experiments?"
- Ablation contribution of each module: If step A is skipped (all judges are sent directly to step B with penalty), or if the surface feature scores are removed in step B, how would the certified coverage and test error rate on the three benchmarks in Table 1 change? How many judges does the regularization in step B ultimately assign non-zero weights (especially those models that were on the admission edge in step A, such as Qwen2.5-0.5B on Chatbot Arena)?