acceptodds
Under review as a conference paper at ICLR 2027

When Do LLM Judge Panels Outperform a Single Large Judge? The Diversity-Bias Tradeoff

Abstract

LLM evaluation pipelines increasingly face a practical choice: use one strong judge or aggregate a panel of diverse judges. The usual argument for panels is variance reduction through less-correlated errors, but this view ignores an opposing effect: provider diversity can also increase bias heterogeneity. We formalize this diversity-bias tradeoff with a two-term decomposition of panel gain and derive a phase boundary characterizing when aggregation should outperform a single judge. Across four independently composed cross-provider panels spanning seven AI providers and four NLG benchmarks (21 dimensions, 2,635 items), provider diversity reduces inter-judge error correlation on every dimension tested (rho = 0.493-0.726 versus 0.925 for a same-provider baseline), yet lower correlation does not translate monotonically into higher evaluation accuracy because bias heterogeneity can offset the variance benefit. The decomposition predicts the direction of panel gain on 94% of dimensions satisfying its same-sign-bias assumption (81% overall). An equal-budget comparison finds a diverse panel approximately equivalent to a comparably priced single judge (10 versus 11 wins across 21 dimensions), showing that apparent panel advantages can reflect judge quality rather than diversity alone. Item-level analysis provides additional directional support for the variance-reduction mechanism (Spearman r = -0.10, p < 0.0001, N = 4,200). Finally, we translate the theory into a validation-based deployment rule that selects between panel and single-judge evaluation at the dimension level. These results explain when judge diversity helps, when heterogeneous bias cancels its benefit, and why panel deployment should be selective rather than automatic.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-judge, judge panels, provider diversity, error correlation, bias-spread tradeoff

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hc1SOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when should we use a panel of judges from different providers instead of a single large judge. The authors introduce a Diversity-Bias Tradeoff: cross-provider diversity can reduce inter-judge error correlation and improve variance reduction, but it may also increase bias heterogeneity and offset these benefits. To study this tradeoff, the authors propose a two-term decomposition and a corresponding phase boundary, and conduct experiments on four NLG benchmarks. The results show that lower error correlation does not always lead to better evaluation accuracy, and that a diverse panel performs similarly to a single judge under an equal-budget setting. Finally, the authors propose a validation-based deployment rule to choose between a panel and a single judge for each evaluation dimension.

Critical strengths
  1. I find the question of choosing between a panel of multiple judges and a single large judge interesting. The proposed Diversity-Bias Tradeoff provides an intuitive way to understand the potential benefits and costs of judge diversity. I also think this perspective has practical value for designing LLM evaluation pipelines.

  2. The experiments are fairly comprehensive and include several useful comparisons and analyses. I especially appreciate the cost-neutral comparison that takes model capability and API costs into account and helps provide a fairer comparison between panels and single judges.

Critical weaknesses

The diversity-bias decomposition might rely on an incorrect MAE approximation. In Appendix A, the authors use a first-order Taylor approximation under residual symmetry (A2) to obtain f(b)=E∣b+ξ∣≈σeff+∣b∣/2f(b)=\mathbb E|b+\xi|\approx\sigma_{\mathrm{eff}}+|b|/2. However, for a continuous and symmetric residual distribution, we have f′(0)=0f'(0)=0, so the first-order expansion around b=0b=0 should be f(b)=f(0)+o(∣b∣)f(b)=f(0)+o(|b|), without the linear term ∣b∣/2|b|/2. This also raises questions about the following derived bias-spread cost (sk+sk′)/6(s_k+s'_k)/6. In addition, converting the variance reduction of the averaged residuals into a reduction in expected absolute error seems to require stronger distributional assumptions.

Questions for authors

I have two small questions, mainly out of curiosity.

  1. The paper mainly uses simple averaging to combine multiple LLM judges. However, there are many other ways to aggregate judgments, such as weighted averaging, median-based methods, and rank aggregation methods like Borda count. Simple averaging is perhaps the most intuitive choice, but it may not be optimal when judges have different levels of reliability or systematic biases. I wonder how much of the observed Diversity-Bias Tradeoff depends on the choice of aggregation method. Could more sophisticated aggregation methods help reduce the bias cost while preserving the benefits of judge diversity?

  2. The results of this paper also suggest that choosing between a judge panel and a single strong judge is not straightforward, as neither consistently outperforms the other across different settings. Another interesting possibility is to combine the strengths of judge panels and a single strong judge. E.g. a recent paper AtC: Aggregate-then-Calibrate for Human-Centered Assessment (ICLR 2026) has achieved this kind of combination. I'm thinking that similar idea might work for LLM judge panels, potentially reducing bias heterogeneity while making use of the complementary information from different judges. I would be curious to hear the authors' thoughts on such a strategy and whether it might change the Diversity-Bias Tradeoff discussed in this paper.

Reviewer_DKCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when combining diverse LLM judges is more effective than using a single strong judge. The authors propose a framework that explains the tradeoff between reducing error correlation and introducing systematic biases. They show that lower error correlation does not always lead to better evaluation performance and propose a rule for deciding when to use a diverse panel or a single judge.

Critical strengths
  1. The motivation and research topic are interesting and important.
  2. The paper provides extensive experimental results to support its findings.
  3. I can see that the authors have carefully considered various factors in designing their experiments and conducting their analyses.
Critical weaknesses
  1. I have concerns about one mathemtical derivation, and I think this is the most critical. Please see Question 1 below.
  2. I think it would have been better if the paper would have shown other aggregation methods such as weighted averaging or voting, given theat the paper focuses only on average.For example, if some judges have larger biases than others, weighted averaging might reduce their influence. That could change the observed tradeoff between diversity and bias.
  3. The authors acknowledge the limited practical improvements in Section 5.8, noting that the absolute gains are small and that the estimates can be noisy. But given that the authors acknowledge this, I believe this is what the authors leave for future work.
  4. I have several additional questions, and depending on the authors' responses, some of these concerns may also be considered weaknesses
Questions for authors

1.(Critical) In Appendix A, the authors approximate MAE as σeff+∣b∣2\sigma_{\mathrm{eff}}+\frac{|b|}{2} using a first-order Taylor expansion. However, under the symmetry assumption, f′(0)≈0f'(0)\approx 0, so I think the first-order Taylor expansion shoild maybe be f(b)≈f(0)f(b)\approx f(0), without the additional linear ∣b∣2\frac{|b|}{2} term. Could the authors clarify how the linear ∣b∣2\frac{|b|}{2} term is derived? I wonder if this is methematical flaw. If clarified well, I am willing to raise my score since this is my main concern for giving a score in the accept range.

2.Could different distributions of pairwise error correlations lead to different MAE gains, even when their average correlation is the same?

3.Could the authors also compare the panel against its best individual judge (e.g.,the one with the lowest MAE)? I ask because federation gain is defined relative to this baseline.

4.The authors explain that increased bias spread can reduce the benefits of diversity. However, I wonder whether other factors, such as individual judge quality and noise magnitude, could also explain the results.Could the authors provide a clearer theoretical explanation/logi or additional experiments to better isolate the effect of bias spread?

5.Section 5.6 shows that lower correlation is associated with more panel wins, but it does not directly test the theoretical phase boundary. Could the authors show how accurately the predicted threshold identifies when a panel wins or loses?

6.The theory assumes same-sign biases (A3) and approximately symmetric residual noise (A2), but these assumptions hold for only 81% and 57% of dimensions, respectively. So I am not sure if this assumption holds consistently

Reviewer_m9wAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

LLM-based evaluation has increasingly shifted toward using a panel of judges to make collective judgments. This paper studies whether a panel of LLM judges from different providers outperforms a single judge. It adapts the classic ensemble bias–variance decomposition to MAE and splits the panel's gain over its best member into two parts: a variance-reduction gain that grows as the judges' errors become less correlated, and a bias-spread cost that grows as their systematic biases diverge. Using the derived correlation threshold and a per-dimension rule estimated on a validation split, practitioners can decide whether to use a panel or a single judge. Empirical results show that provider diversity lowers error correlation on every dimension, but this lower correlation does not translate into more wins. The decomposition correctly predicts the sign of the panel's gain on most dimensions. At equal cost, the panel and a single Claude Haiku judge are roughly tied.

Critical strengths

I enjoyed the problem statement and its motivation. Automated evaluation increasingly relies on a group of models to make collective decisions, and the correlation and bias among them can undermine decision quality, since most of them share similar priors from training on overlapping web-crawled datasets. Rather than naively averaging their judgments, this paper studies the diversity and bias within the panel and proposes a pipeline to decide whether to trust the collective system or fall back to a single judge model. I think the problem is interesting and important to the field, and the empirical studies are diverse enough to support the paper's claims.

Critical weaknesses
  1. The deployment rule needs ground truth, which is usually missing in deployment. Every coefficient in the rule is estimated from residuals against human scores on an 80% validation split. In practice, a judge panel is used precisely because human labels are expensive or unavailable. The paper does not discuss how many labels are needed, or whether the estimates carry over to new tasks without them.
  2. Missing prior work on correlation-aware aggregation. There are two papers address the same core problem, correlated judge errors and shared biases, by improving the aggregation step rather than choosing between a panel and one judge: https://arxiv.org/abs/2601.22336, https://arxiv.org/abs/2603.00039, which needs no ground-truth labels.
  3. The baselines are outdated. The single-judge baseline, GPT-4o-mini, is old, less capable, compared with the panel members. I think a better setup is to use a strong single model and comparing to a panel of weak models, where the total inference costs are matched.
  4. Choosing between "panel" and "single judge" somehow sounds an odd framing. If a panel underperforms, the natural response is to curate a better panel, not to fall back to a single judge.
Questions for authors
  1. How many labeled items per dimension does the deployment rule need before its decision becomes stable? Can the coefficients be estimated without human labels, for example by reusing estimates from a related task?
  2. When a panel underperforms, why fall back to a single judge instead of curating a better panel?
  3. How does this study relate to correlation-aware aggregation methods, such as the Ising-model approach (arXiv:2601.22336) and CARE (arXiv:2603.00039)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.