acceptodds
Under review as a conference paper at ICLR 2027

When All LLM Judges Agree: Silent Consensus Makes Deployed Risk Unmeasurable

Abstract

LLM judge panels now gate model outputs: unanimous approvals ship automatically, and humans see only disagreements. This design manufactures its own evidence of reliability—when all judges share a blind spot, the bad output ships and no label is ever collected, a regime we call silent consensus. We first prove the error rate of never-audited released outputs is unidentifiable: the same logs arise whether it is 0%, 100%, or anything in between. We then measure how often whole panels fail together, with frozen open-weight judges on three tasks with independent labels (policy compliance, logical reasoning, medical advice). Joint failures are common where judgment is subjective: a panel of three model sizes from one family unanimously approved 42.9% of medical responses violating physician-written criteria, and three samples of one model erred almost identically, jointly approving 39% of corrupted reasoning claims—a three-judge panel in name only. Finally, we compare auditing strategies in simulated deployments where the case mix shifts partway through (e.g., unseen expert criteria). Two natural strategies fail. Auditing only where judges disagree never checks a shared blind spot, by definition. Auditing the least-confident cases barely helps: judges are confident precisely where they are blind, and 68.5% of unsafe outputs released after the shift could never be audited. Auditing a small random fraction of released outputs, with logged probabilities, makes the error rate estimable, with a confidence bound valid under continuous monitoring. Randomized audits do not make a system safe, but without them deployed risk cannot be measured at all.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, selective labels, correlated errors, deployment monitoring, anytime-valid inference, scalable oversight

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YiQDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the issue of selective labeling in an automatic review system based on multiple LLM Judges. In the deployed scenario studied, the model-generated content will only be automatically released if all the Judges unanimously agree; otherwise, it will be submitted for manual review. The authors point out that if multiple Judges reach a consensus on incorrect outputs, these errors will not only be released but also, due to the lack of independent human review, their true error rate cannot be reliably estimated from the existing logs. This phenomenon is referred to as "Silent Consensus".

To this end, the paper first theoretically proves that when a certain automatic release group always has a zero audit probability, without additional assumptions or external label sources, its true error rate cannot be identified. Subsequently, the authors measured the joint false-accept rate (Joint False-Accept Rate) of different combinations of LLM Judges on three types of tasks (PolicyDelta, ReasonBench, and HealthBench), analyzed the joint errors and their correlations of multiple models.

Finally, the paper proposes a set of random auditing protocols. By retaining a non-zero audit probability for the automatic release outputs and combining inverse probability weighting estimation and anytime-valid confidence bounds, it achieves statistical estimation of the cumulative error quantity. The authors further compared different auditing strategies through simulation deployment replays that include distribution changes.

Experiments show that in some tasks, multiple LLM Judges may indeed jointly accept incorrect outputs. Random auditing can provide independent labels for the previously unobserved group. However, the proposed complete protocol does not show a significantly better effect than simple stratified random auditing.

Critical strengths
  1. The paper addresses a practically relevant deployment problem. This paper points out a potential risk in the LLM automatic assessment system: Consistent judgments by multiple Judges do not guarantee the correctness of the result, and only conducting manual review for discrepant cases will lead to some common errors remaining without independent verification for a long time. This issue has certain practical significance for LLM applications that rely on automated verification.
  2. The structure of the paper is clear, and the theoretical analysis and experimental design are relatively complete. The paper covers problem definition, theoretical analysis, empirical measurement, and audit protocol. The overall logic is relatively coherent. The author conducted measurements on multiple tasks and different Judge combinations, and attempted to verify the audit protocol through simulated deployment. At the same time, The authors acknowledge the limitations of some experimental results, including that the method comparison did not show significant advantages and the distribution change detection failed.
Critical weaknesses
  1. The main contribution has limited novelty and significance. The core issue is essentially a known selective-label / missing-data problem in a new LLM-judge setting, and the proposed remedy—random audits with inverse-propensity estimation and anytime-valid bounds—mainly combines existing statistical tools. As a result, I am not convinced the paper delivers a sufficiently new scientific insight or methodological contribution.

  2. The experiments do not clearly establish the unique value of the proposed framework. They mostly confirm expected phenomena, and the full protocol does not show a clear advantage over simpler randomized auditing baselines. In particular, the replay results show no clear advantage in error interception, provide insufficient comparisons of estimation quality, and fail to demonstrate effective shift detection.

Questions for authors
  1. Regarding the differences between the core innovation points and existing research. The main theoretical conclusions of this paper can be derived from the traditional selective-label and positivity theories, and the random auditing, inverse probability weighting, and anytime-valid confidence bounds adopted in this paper also have mature research foundations. Could the author further explain what new scientific insights or methodological contributions this paper has made, apart from applying these existing theories and methods to the unanimous LLM judging scenario? In particular, compared to the existing works such as Ao et al. (2026) and Berrada et al. (2025) that involve LLM evaluation and selective auditing, what are the substantive research increments of this paper?

  2. Regarding the practical value of the complete auditing protocol. Since the main goal of this paper is to make deployment risks estimable, compared to simple uniform random auditing or stratified random auditing, can the complete protocol provide more accurate risk estimations, tighter confidence upper bounds, or higher auditing efficiency under the same auditing budget? The current experiments mainly report the number of error interceptions and auditing support conditions, but do not adequately compare the quality of risk estimations of different methods. Could the author provide the relationship between estimation error, confidence-bound width, and audit cost under the same budget? Furthermore, how can one understand the difference between the approximately 28% actual auditing rate in the experiment and the 2% minimum random auditing probability emphasized in the paper, and their impact on the actual deployment cost?

  3. Regarding the universality of the multi-model joint error experiment. Table 2 and Appendix A indicate that some models have an extremely low acceptance rate for correct cases, and some panels with low joint error rates also have a very high manual review ratio. Could the author further distinguish whether the decrease in joint error rate is due to effective error complementation among different Judges, or due to individual Judges generally refusing to output or the model format not meeting requirements? If different panels are compared under similar automatic acceptance rates for correct cases, does the main empirical conclusion of this paper still hold?

Reviewer_GXSHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the utility of LLM judge panels being increasingly deployed in LLM automation pipelines to measure / control risk. The paper is primarily concerned with the fact that judge panels can share blind spots that can compromise reliability of simple gating protocols where automation proceeds where judges agree or delegation to a human occurs where judges disagree.

The paper attempts to answer two overarching questions: 1/ The first relates to whether or not it is possible to correct for silent consensus (a form of bias arising in judge panels) 2/ The second relates to how to develop mechanisms to make risks measurable again. The paper offers a mix of theoretical and empirical results addressing these two issues; it also proposes, implements, and stress-tests a randomized audit protocol to improve the reliability/utility of judge panels.

Critical strengths

The paper clearly addresses a very relevant / consequential problem that is becoming increasingly prevalent in automation pipelines; it studies the utility of judge panels, it reveals issues associated with judge panels; and it also proposes an approach based on randomized audit protocols to address the limitations associated with judge panels, making risk measurable again.

The paper complements theoretical results with empirical results; it also offers a variety of experimental results illustrating the limitations of judge panels plus the potential of coupling a judge panel with randomized audit protocols. A variety of judge panels composed of identical and different LLMs have also been studied.

Critical weaknesses

The main limitation is that the result / approach is relatively trivial ; it is clear that where judges do not act independently (and in the limit where a family of similar judges behaves identically) then it is always possible full agreement to let erroneous generations through [independently of the number of judges in the panel]; it is also clear that a simple randomized audit protocol can partly mitigate this issues. The authors themselves acknowledge that the paper relies on well established statistical machinery, with the novelty lying primarily on the application.

I personally think it would be more interesting to try to tackle adjacent questions such as: (1) What is the (mathematical) tradeoff between the number/fraction of erroneous generations that a panel of judges approves and the number/fraction of correct generations that the panel escalates given a (probablisitic) model for the joint behaviour of a panels of judges (trying to capture how the judges agree / diasgree perhaps strata dependent) [i recognize the authors have tried to do this empirically] (2) How could one learn such a model given calibration data? and (3) How could such a pipeline be deployed in practice in a way that one minimizes approvals and erroneous generations and escalations of correct generations. I think this could come across more technically novel than the current angle.

Questions for authors

Please see critical weaknesses above. I might be willing to raise my score depending on how the authors address the critical weaknesses i highlighted earlier.

Reviewer_HhcWOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles a timely and relevant issue of usefulness of LLM judges and construction of judge panels. More precisely, they identify silent consensus phenomenon, that says that models, especially from the same lineage share blind spots which causes panels that look big on surface to be effectively equivalent to as little as a single judge. The paper mathematically formalizes this problem, as well as evaluates it across a number of datasets and LLM models as well as panel composition. In a process the paper comes to an explicit trade-off that same-family panels have high JFAR (FN rate), but low Safe escalation (FP rate), while cross-family panels reduce FN at the cost of FP. Furthermore, through the lens of effective panel count they clearly estimate that the size of same family panels is approximately 1, while cross-family panels are closer to their real judge count yet still some way far from it.

Critical strengths
  • The paper looks at the problem through multiple lenses including math behind it and empirical evaluation giving something for everyone.
  • The experimental suite is extensive in terms of judge panels and datasets. I particularly like Figure 2 showing a trade-off between JFAR and Safe escalation.
  • Table 4 offers an ablation like study looking beyond disagreement only policy.
  • I really like "Implications for practice" paragraph in Section 8 and would definitely love to see this more across applicable works in the general ML community.
Critical weaknesses
  • The manuscript is quite challenging to understand. I surrender that part of it comes to my lack of knowledge of the field and prerequisites, but even parts that I do I feel could be better written. As the cause I mainly attribute high density of technical terms and jargon, but to a great part lack of explicit language contributes to this. Below I point some of issues contributing to this:
  1. It is unclear throughout the paper what the outputs are reviewed for. Is it safety, correctness, fairness, this could be better specified. Also, unclear why labeling is necessary and what human annotations are used for: further alignment, just blank auditing, controlling showing outputs to the end user of the system? Moreover, on line 52 the authors use term "missed errors" as a blank term rather than being explicit what errors exactly is within the context. These might clear to the community, but for anyone that would use this paper as an introduction these may increase ambiguity.
  2. In formula for DeffD_\text{eff} there appear q^J\hat{q}_J and q^j\hat{q}_j (small jj) that are not introduced anywhere in the paper. Together with this in Eq (1) there appears a number of conditional variables that are never explained. On the other hand, there are redundancies such as explicitly defining set VtV_t at line 181. Overall, notations could have been handled better.
  3. In table 2 it is unclear what models panels consist of, nor what number next to cross-family means, is it size of the panel, and if so what's the fifth judge as there are four cross-family judges mentioned in 5.1? Also which 3 would be part of cross-family 3 then? How is executor's vote incorporated in the panel decision? These terms should have been explicitly defined.
  4. In Table 4 it is unclear what panels consist of and which datasets are used.
  • I get an impression that the novelty is somewhat limited: the math (that I understood) seems interesting but not overly novel. Experiments show nice results but the methodology isn't new. Section 6 raises a question of its purpose in the paper.

My score reflects my lack of understanding of the text as well as the big part of math. I feel reluctant to give positive evaluation for work I do not understand at an acceptable level, however I find the underlying issue relevant, presented results interesting, and guidelines presented in the closing of the manuscript useful for practitioners. Despite the positives, I believe a valuable work should be accompanied by proper presentation which in my eyes the paper is currently lacking.

Questions for authors
  • I struggle to understand the purpose of Section 6, could authors better clarify how it fits in the paper and what it aims to achieve? Could you write a clear step by step algorithm of the underlying approach?
  • I find some metrics very interesting, namely JFAR and effective judge count, however I believe they are not a contribution of the paper. Could authors give an explicit list of their methodology/metrics contributions, and cite existing metrics accordingly?
  • I have a concern of dataset selection as across PolicyDelta and ReasonBench many of these panels are effectively giving JFAR 0% safe escalation 100% meaning they are for all practically purposes on par with always rejecting the sample which is not a smart classifier at all. I would like to better understand the author's justification for choosing these datasets and outlook of value they add to the paper.

Comments on the paper

1 comment

Thank you to the authors for the detailed response and the additional experiments. I appreciate the effort to address my questions and the transparent reporting of the results.

Regarding Q1, my main concern about the novelty and significance of the contribution remains. I agree with Reviewer GXSH that further investigating the joint behavior of LLM judges, learning such models from data, and using them to optimize auditing and release decisions could be a promising research direction. However, these directions remain outside the scope of the current work.

For Q2 and Q3, I appreciate the additional analyses and the authors' candid discussion of the findings. While these experiments help clarify the limitations of the proposed protocol and the original panel comparisons, they do not substantially alleviate my concerns about the paper's incremental scientific contribution and practical significance.

Therefore, although I appreciate the authors' efforts and the improvements proposed for the revision, I am inclined to maintain my original assessment and rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.