- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper reveals the error dependence among LLM judges, showing that judge agreement does not necessarily provide independent evidence. The authors quantify how correlated errors reduce the effective size and reliability of multi-judge ensembles. The authors also demonstrate that shared errors remain strong even across frontier models and different model providers. The authors also propose several correlation-aware aggregation methods to improve the reliability of LLM-based evaluation.
Critical strengths
-
This paper addresses an important and timely problem in LLM-as-a-Judge evaluation.
-
This paper provides clear empirical evidence that shared judge errors can substantially overstate confidence in consensus.
-
The analysis of frontier models and cross-provider judges makes the findings more important and convincing.
-
The authors propose several correlation-aware aggregation methods to handle this issue.
Critical weaknesses
-
This paper provides limited insight into why LLM judges exhibit correlated errors.
-
The global co-failure and vulnerable-subgroup settings are somewhat stylized and may not capture complex real-world dependence patterns.
-
The proposed mitigation methods, CorrFilter and Bias-Cluster, seem somewhat ad hoc.
-
The reliance on a trusted calibration set may limit the practicality and scalability of the proposed approach.
Questions for authors
Please refer to the weakness part.