- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when should we use a panel of judges from different providers instead of a single large judge. The authors introduce a Diversity-Bias Tradeoff: cross-provider diversity can reduce inter-judge error correlation and improve variance reduction, but it may also increase bias heterogeneity and offset these benefits. To study this tradeoff, the authors propose a two-term decomposition and a corresponding phase boundary, and conduct experiments on four NLG benchmarks. The results show that lower error correlation does not always lead to better evaluation accuracy, and that a diverse panel performs similarly to a single judge under an equal-budget setting. Finally, the authors propose a validation-based deployment rule to choose between a panel and a single judge for each evaluation dimension.
Critical strengths
-
I find the question of choosing between a panel of multiple judges and a single large judge interesting. The proposed Diversity-Bias Tradeoff provides an intuitive way to understand the potential benefits and costs of judge diversity. I also think this perspective has practical value for designing LLM evaluation pipelines.
-
The experiments are fairly comprehensive and include several useful comparisons and analyses. I especially appreciate the cost-neutral comparison that takes model capability and API costs into account and helps provide a fairer comparison between panels and single judges.
Critical weaknesses
The diversity-bias decomposition might rely on an incorrect MAE approximation. In Appendix A, the authors use a first-order Taylor approximation under residual symmetry (A2) to obtain . However, for a continuous and symmetric residual distribution, we have , so the first-order expansion around should be , without the linear term . This also raises questions about the following derived bias-spread cost . In addition, converting the variance reduction of the averaged residuals into a reduction in expected absolute error seems to require stronger distributional assumptions.
Questions for authors
I have two small questions, mainly out of curiosity.
-
The paper mainly uses simple averaging to combine multiple LLM judges. However, there are many other ways to aggregate judgments, such as weighted averaging, median-based methods, and rank aggregation methods like Borda count. Simple averaging is perhaps the most intuitive choice, but it may not be optimal when judges have different levels of reliability or systematic biases. I wonder how much of the observed Diversity-Bias Tradeoff depends on the choice of aggregation method. Could more sophisticated aggregation methods help reduce the bias cost while preserving the benefits of judge diversity?
-
The results of this paper also suggest that choosing between a judge panel and a single strong judge is not straightforward, as neither consistently outperforms the other across different settings. Another interesting possibility is to combine the strengths of judge panels and a single strong judge. E.g. a recent paper AtC: Aggregate-then-Calibrate for Human-Centered Assessment (ICLR 2026) has achieved this kind of combination. I'm thinking that similar idea might work for LLM judge panels, potentially reducing bias heterogeneity while making use of the complementary information from different judges. I would be curious to hear the authors' thoughts on such a strategy and whether it might change the Diversity-Bias Tradeoff discussed in this paper.