acceptodds
Under review as a conference paper at ICLR 2027

Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

Abstract

Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In practice, LLM judges are often trained and evaluated in similar ways, so they can make the same mistakes. We study how this dependency affects the reliability of consensus. We find substantial error correlation across both open-weight and frontier LLM judges. In our main bank of ten judges, the average pairwise correlation between judge errors is 0.21. As a result, the ten judges only provide roughly as much statistical information as 3.5 independent judges. The dependency is even stronger among the high-accuracy frontier judges we evaluate, including judges from different providers. In up to 28% of our comparisons, ignoring shared errors leads to the conclusion that one system is significantly better, while accounting for them does not. We also find that the pattern of errors matters. Errors shared by most judges and errors concentrated among a smaller group affect consensus differently and favor different voting methods. Measuring the overall amount of correlation alone is therefore insufficient. Our results suggest a simple approach: use a small set of trusted examples to estimate judge accuracy and identify shared mistakes. These shared errors should then be considered when analyzing the results, and the voting method should be chosen using trusted examples before it is applied to new data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM judges, consensus reliability, correlated errors, judge dependence, ensemble voting, evaluation reliability, shared mistakes, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tDnVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reveals the error dependence among LLM judges, showing that judge agreement does not necessarily provide independent evidence. The authors quantify how correlated errors reduce the effective size and reliability of multi-judge ensembles. The authors also demonstrate that shared errors remain strong even across frontier models and different model providers. The authors also propose several correlation-aware aggregation methods to improve the reliability of LLM-based evaluation.

Critical strengths
  1. This paper addresses an important and timely problem in LLM-as-a-Judge evaluation.

  2. This paper provides clear empirical evidence that shared judge errors can substantially overstate confidence in consensus.

  3. The analysis of frontier models and cross-provider judges makes the findings more important and convincing.

  4. The authors propose several correlation-aware aggregation methods to handle this issue.

Critical weaknesses
  1. This paper provides limited insight into why LLM judges exhibit correlated errors.

  2. The global co-failure and vulnerable-subgroup settings are somewhat stylized and may not capture complex real-world dependence patterns.

  3. The proposed mitigation methods, CorrFilter and Bias-Cluster, seem somewhat ad hoc.

  4. The reliance on a trusted calibration set may limit the practicality and scalability of the proposed approach.

Questions for authors

Please refer to the weakness part.

Reviewer_kmbxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how shared errors among LLM Judges affect their reliability of consensus. Testing 10 judges on RewardBench v2, they obtained a mean error correlation of 0.21, which means the amount of consensus information provided by the 10 judges is equivalent to that of 3.5 actual independent judges. They also showed that frontier models like GPT/Claude/Grok is more accurate (0.92) but more correlated (0.56). Finally, they identified two error mode: 1. most judges failing together, and 2. small group of judges failing; And they proposed CorrFilter and Bias-Cluster for each mode for routed judge selection.

Critical strengths

The error correlation estimate itself is quite robust and reliable, which stays between 0.13 and 0.28 across all 68 banks and doesn't change much when 10% of the labels are randomly flipped. The paper also reports its failures openly, like the fact that its router was missing a preregistered target in 22 of 24 comparisons.

Critical weaknesses

In summary, the paper has four contributions:

  • C1: effective number of judges
  • C2: more accurate judges are more error-correlated
  • C3: different filters should be used for different error modes
  • C4: decide which filter to use by calibration on a small trusted set

The first two are not new results.

  • C1: Conneting the average error correlation to the effective number of judges is the textbook design effect. The number is nice, but the practice is not new.
  • C2: That more accurate models share more errors is already known from Kim et al. (ICML 2025) and Goel et al. (ICML 2025), and the former is not cited.

The remaining two, which are the actual new contributions, are weak.

  • C3: Of the two error modes the paper mentioned, Sec. 4.4 "The Error Pattern Determines Which Filter Helps" only shows results for error mode 2 (vulnerable-subgroup failure). The first error mode (Global co-failure) is omitted.
    • Moreover, even when the error mode is known, the error-matched filter can still lose to a single fixed filter.
    • The paper even states this plainly in Sec 4.5 as "Recognizing the error pattern does not identify the best rule".
  • C4: The last contribution is to choose filter by calibration on a small trusted set, but it has no reported results under its exact setting, as is presented in Appendix M.1 itself, which says "the strict selector result remains uncertain".
    • The only reported test with trusted labels is from eight external datasets, and in this setting the selector "chooses plain majority in every case", which is trivial.

Ref:

  • Kim et al. (ICML 2025) "Correlated Errors in Large Language Models."
  • Goel et al. (ICML 2025) "Great Models Think Alike and this Undermines AI Oversight."
Questions for authors

None

Reviewer_WNsqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how correlated errors among LLM judges can cause consensus to overstate statistical evidence. It measures error dependence across judge panels and reports that a panel of ten logical judges has a variance-based effective size of approximately 3.51. It also shows that treating correlated votes as independent observations can change significance conclusions, and investigates filtering strategies under different error structures and transfer settings.

Critical strengths
  1. An important problem with concrete statistical consequences. Figure 2 demonstrates how the same votes can yield different confidence intervals and significance conclusions depending on the unit of analysis. This is directly relevant to practitioners using multi-judge evaluation.

  2. Broad empirical coverage and transparent reporting of negative results. The paper examines multiple model combinations and data sources, and reports settings in which the proposed filtering strategies do not improve performance. This helps clarify their practical scope.

Critical weaknesses
  1. The contribution relative to closely related work is insufficiently established. [1] already studies correlated judge errors, effective vote counts, and RewardBench. [2] also documents correlated errors among strong models from different providers. The current manuscript does not adequately discuss these directly relevant studies, making it difficult to identify which central findings constitute substantive advances rather than replications or extensions. [1] Guneet Kohli. Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels. arXiv preprint arXiv:2605.29800, 2026. https://arxiv.org/abs/2605.29800 [2] Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated Errors in Large Language Models.(ICML) 2.Robust gains from the proposed strategies have not been established. Some improvements in Table 4 are only 0.24 or 0.02 percentage points, while transfer to RewardBench yields a negative gain. Moreover, bootstrapping potentially overlapping test batches may underestimate uncertainty. Given that a central message of the paper is that dependent observations should not be treated as independent evidence, the method comparisons should follow the same principle. These issues weaken the evidence for reliable benefits beyond the diagnostic findings.
Questions for authors
  1. Which central conclusions or mechanistic findings go beyond the closely related studies identified above? Please distinguish replication of established phenomena from the paper’s substantive new contributions.
  2. If the confidence intervals in Table 4 are recomputed using the underlying independent items or an appropriate hierarchical resampling scheme, which gains remain supported? How would this affect the conclusions about effectiveness and transfer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.