- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the issue of selective labeling in an automatic review system based on multiple LLM Judges. In the deployed scenario studied, the model-generated content will only be automatically released if all the Judges unanimously agree; otherwise, it will be submitted for manual review. The authors point out that if multiple Judges reach a consensus on incorrect outputs, these errors will not only be released but also, due to the lack of independent human review, their true error rate cannot be reliably estimated from the existing logs. This phenomenon is referred to as "Silent Consensus".
To this end, the paper first theoretically proves that when a certain automatic release group always has a zero audit probability, without additional assumptions or external label sources, its true error rate cannot be identified. Subsequently, the authors measured the joint false-accept rate (Joint False-Accept Rate) of different combinations of LLM Judges on three types of tasks (PolicyDelta, ReasonBench, and HealthBench), analyzed the joint errors and their correlations of multiple models.
Finally, the paper proposes a set of random auditing protocols. By retaining a non-zero audit probability for the automatic release outputs and combining inverse probability weighting estimation and anytime-valid confidence bounds, it achieves statistical estimation of the cumulative error quantity. The authors further compared different auditing strategies through simulation deployment replays that include distribution changes.
Experiments show that in some tasks, multiple LLM Judges may indeed jointly accept incorrect outputs. Random auditing can provide independent labels for the previously unobserved group. However, the proposed complete protocol does not show a significantly better effect than simple stratified random auditing.
Critical strengths
- The paper addresses a practically relevant deployment problem. This paper points out a potential risk in the LLM automatic assessment system: Consistent judgments by multiple Judges do not guarantee the correctness of the result, and only conducting manual review for discrepant cases will lead to some common errors remaining without independent verification for a long time. This issue has certain practical significance for LLM applications that rely on automated verification.
- The structure of the paper is clear, and the theoretical analysis and experimental design are relatively complete. The paper covers problem definition, theoretical analysis, empirical measurement, and audit protocol. The overall logic is relatively coherent. The author conducted measurements on multiple tasks and different Judge combinations, and attempted to verify the audit protocol through simulated deployment. At the same time, The authors acknowledge the limitations of some experimental results, including that the method comparison did not show significant advantages and the distribution change detection failed.
Critical weaknesses
-
The main contribution has limited novelty and significance. The core issue is essentially a known selective-label / missing-data problem in a new LLM-judge setting, and the proposed remedy—random audits with inverse-propensity estimation and anytime-valid bounds—mainly combines existing statistical tools. As a result, I am not convinced the paper delivers a sufficiently new scientific insight or methodological contribution.
-
The experiments do not clearly establish the unique value of the proposed framework. They mostly confirm expected phenomena, and the full protocol does not show a clear advantage over simpler randomized auditing baselines. In particular, the replay results show no clear advantage in error interception, provide insufficient comparisons of estimation quality, and fail to demonstrate effective shift detection.
Questions for authors
-
Regarding the differences between the core innovation points and existing research. The main theoretical conclusions of this paper can be derived from the traditional selective-label and positivity theories, and the random auditing, inverse probability weighting, and anytime-valid confidence bounds adopted in this paper also have mature research foundations. Could the author further explain what new scientific insights or methodological contributions this paper has made, apart from applying these existing theories and methods to the unanimous LLM judging scenario? In particular, compared to the existing works such as Ao et al. (2026) and Berrada et al. (2025) that involve LLM evaluation and selective auditing, what are the substantive research increments of this paper?
-
Regarding the practical value of the complete auditing protocol. Since the main goal of this paper is to make deployment risks estimable, compared to simple uniform random auditing or stratified random auditing, can the complete protocol provide more accurate risk estimations, tighter confidence upper bounds, or higher auditing efficiency under the same auditing budget? The current experiments mainly report the number of error interceptions and auditing support conditions, but do not adequately compare the quality of risk estimations of different methods. Could the author provide the relationship between estimation error, confidence-bound width, and audit cost under the same budget? Furthermore, how can one understand the difference between the approximately 28% actual auditing rate in the experiment and the 2% minimum random auditing probability emphasized in the paper, and their impact on the actual deployment cost?
-
Regarding the universality of the multi-model joint error experiment. Table 2 and Appendix A indicate that some models have an extremely low acceptance rate for correct cases, and some panels with low joint error rates also have a very high manual review ratio. Could the author further distinguish whether the decrease in joint error rate is due to effective error complementation among different Judges, or due to individual Judges generally refusing to output or the model format not meeting requirements? If different panels are compared under similar automatic acceptance rates for correct cases, does the main empirical conclusion of this paper still hold?
Thank you to the authors for the detailed response and the additional experiments. I appreciate the effort to address my questions and the transparent reporting of the results.
Regarding Q1, my main concern about the novelty and significance of the contribution remains. I agree with Reviewer GXSH that further investigating the joint behavior of LLM judges, learning such models from data, and using them to optimize auditing and release decisions could be a promising research direction. However, these directions remain outside the scope of the current work.
For Q2 and Q3, I appreciate the additional analyses and the authors' candid discussion of the findings. While these experiments help clarify the limitations of the proposed protocol and the original panel comparisons, they do not substantially alleviate my concerns about the paper's incremental scientific contribution and practical significance.
Therefore, although I appreciate the authors' efforts and the improvements proposed for the revision, I am inclined to maintain my original assessment and rating.