acceptodds
Under review as a conference paper at ICLR 2027

When a Reliability Coefficient Orders References Backwards: Identifiability, a Calibrated Test, and Design Rules for the Human References of LLM Judges

Abstract

LLM judges are validated by checking that they agree with a panel of human raters, and the panel is trusted when its reliability coefficient is high. We show that this can go wrong in two ways. First, a judge can pass validation by fitting one panel. When two independent panels rate the same items, their covariance measures what they agree on: the common signal, which is not necessarily ground truth. Under an additive model, and for a judge whose errors are independent of the reference's, this sets a validity ceiling on how well any judge can agree with a panel through that signal. On SummEval, G-Eval's published correlations with the expert panel (0.455–0.582) exceed the upper confidence bound of this ceiling (at most 0.30). Recomputed from its released outputs, G-Eval correlates 0.50–0.57 with the experts and only 0.02–0.06 with the crowd. Nine recent judges show the same split in 33 of 36 judge–dimension pairs, one of them borderline. Second, the reliability coefficient can rank references backwards. A component shared only within one panel raises its reliability and lowers its fidelity at the same time. In 19 of 28 D3code region pairs, the panel that looks more reliable is the less faithful one. We give a per-panel test for this component, with 4.8% false positives at a nominal 5% and 93.4% power at \tau^2=0.3. Without additional information, no one- or two-way variance-components model of a single panel can separate the component from signal, and the designs that maximise reliability or fidelity cannot be checked. A checkable design costs only O(1/b) in fidelity. Only five of the fourteen released references we survey allow the check. Identifying the common signal from two panels is classical, and normalized cross-replication reliability already reads it as a disattenuated agreement. What is new is the per-panel test and what it reveals about how LLM judges are validated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, evaluation, human evaluation, reliability, identifiability, variance components, measurement error, benchmark reliability, experimental design

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dJxaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper discusses how the agreement between LLM-human expert might not translate to LLM-human crowd (basically a different panel/pool) which might reduce the trust on using LLMs as efficient judges. This is because the reliability statistic using a single rater is ρ1\rho_1 which is used in reports but benchmarks usually use the mean of kk independent raters, whose reliability factor ρk\rho_{k} is different (and more accurate to use) compared with ρ1\rho_1 -- this is the first problem the authors bring on unit . But even ρk\rho_k is not completely reliable as it has 2 components - the signal a panel should track; fidelity ρp∗\rho_p^{*} which is supposed to capture the panel-independent quality i.e something a different panel from a different pool can/ideally should also come up with and the covariance/common-factor among panel raters which is τp2\tau_p^{2} i.e a panel-specific shared-component that can't be removed -- this is the second point on validity. And understanding how the judge score fitting to panels is important to trust LLM-Judge scores because it's possible to have a high ρk\rho_k but a low ρp∗\rho_p^{*} making the LLM-judge less trustworthy.

Critical strengths

The problem the paper tries to discuss is very important given the scale at which LLM-Judges are being used - modeling expert human-LLM agreement, crowd-LLM agreement and LLM biases (LLM-LLM agreement) are important to study and design pipelines for data collection, evaluations is critical. The paper tries to understand human-human (different panels) and human-LLM (panel-fitting) settings. So, understanding what metrics to rely on is very important where the authors provide different tools, algorithms to understand for eg: by having >= 2 raters and >= 2 panels, the references can be checkable (L265-269).

Critical weaknesses
  • Contributions (L58-59, 75-77, 121-127): While I appreciate the authors genuine intentions on specifying their particular contributions, in it's current form this is largely diagnostic analysis using classical tools for LLM-Judge x human panels. On L61, the 3 contributions listed are 3 findings and while they are interesting, I feel a lot of it is already well-studied (already cited xRR paper, Reliability of aggregated human annotations in related works) and the authors can expand the study meaningfully to strengthen the paper. For eg, understanding what exactly the judges actually fit in expert vs crowd panels is a good extension:
    • Possible experiment, understanding what LLM-Judge and panels share: Currently the paper says that LLM-Judge agrees to one human-panel (experts) better than another panel (crowd) and there's a common signal and some panel-specific noise. Which means, the judge and the experts share something beyond true quality (Fig. 3). Now, there's research on aggregating multiple LLM judges on same problem but from the judge side: judges agree with each other because of shared biases such as verbosity or style, not quality [1, 2, 3]. Discussing this connection, or ideally testing it (e.g., treating judges as panels in the proposed audit), would help explain what the judge and the experts actually share and strengthen the paper's 1st point.
  • On presentation and sentence formations: I feel the paper can be presented a lot better. I strongly recommend the authors to consider polishing the writing and work on the presentation as the current form is very hard to understand at various sections. Few examples from introduction section below:
    • "We start with what it does to a decision. It can put two references in the wrong order, and a judge can agree with one of them far beyond anything a second panel confirms. The designs that maximise the number (one large panel) or the reference’s fidelity (many single-rater panels) are exactly the ones that cannot be checked"
    • "The two share a name, and they can rank panels, and the judges tested against them, in opposite orders. That can change which system, or which LLM judge, a study declares better."
  1. CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (ICML 2026): https://openreview.net/forum?id=3WPDFjZ1UT
  2. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025): https://openreview.net/forum?id=3GTtZFiajM
  3. Large Language Models are not Fair Evaluators (ACL 2024): https://aclanthology.org/2024.acl-long.511/
Questions for authors

On formatting: The paper might be benefitted from notations table in Appendix

Reviewer_D9j6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to identify the effect of human panel selection when evaluating llm judges. The paper suggests two problems in the current judge evaluations: the current unit of evaluation cannot successfully rule out the noise induced by panel selections, hence the validity of such measurement is questionable. The authors mathematically expands their reasoning and concludes that 1) a judge can pass validation by fitting one panel without fitting a common signal; 2) the reliability coefficient can rank references backwards; and 3) the judge can maximize reliability/fidelity which cannot be checked. They showed their result both in mathematical reasoning and empirical/synthesized results.

Critical strengths

This paper has the following strengths:

  • Starting from the traditional work on identification of common signal from multiple measures, the paper proves the possible pitfalls in the current evaluation.
  • Furthermore, the paper claims their findings based on empirical datasets which are commonly used in the field.
Critical weaknesses

This paper has the following weakness:

  • The diversity of panels could be due to lack of agreement for evaluating the criteria. Specifically, experts could have a clearer view for the criteria, while crowds could not. Such differences might break the assumption about the idiosyncratic noises.
Questions for authors

See the weakness above.

Reviewer_FaeUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits the human references used to validate LLM judges. Under an additive multi-panel model, it estimates panel-specific components, derives common-signal fidelity and a conditional judge-correlation ceiling, and studies auditable reference designs. It reports panel-specific judge agreement and reliability–fidelity reversals across several datasets.

Critical strengths

S1. Auditing the reference itself addresses an important gap in judge validation. The contrast between SummEval and DICES demonstrates that similar internal reliability can coexist with very different cross-panel support.

S2. The relationship to classical measurement theory and cross-replication reliability is unusually explicit. Section 2 appropriately limits novelty to the per-panel procedure, its use for machine evaluation, and the associated empirical and design implications.

S3. The theory and simulations explain why independent panels and within-panel replication serve different purposes. The fixed-budget analysis provides a concrete reason to preserve both.

Critical weaknesses

W1. The empirical interpretation of positive panel-specific estimates needs stronger robustness to scale differences. The model in Section 3.2 fixes the common-signal loading to one in every panel. Appendix G already acknowledges that a smaller loading can produce a negative estimate. The same mechanism can produce a positive estimate for the other panel even without a panel-specific component. This is an out-of-model sensitivity issue rather than a contradiction of the propositions, but it directly affects interpretation of Tables 1 and 2. Loading or scale diagnostics and a sensitivity analysis are needed.

W2. Calibration should be evaluated under the dependence structure of the empirical data. The test in Section 3.2 resamples items, while SummEval summaries are nested within documents; the implementation notes use item resampling for some component tests and document resampling for some ceiling analyses. Independent-item size simulations do not establish nominal calibration for all these empirical tests. Cluster-aware comparisons would clarify robustness.

W3. The design rule in Sections 3.3 and 3.4 needs a more operational definition of independent panels. Disjoint annotator identities do not by themselves guarantee independent panel-specific effects. The paper acknowledges population and construct differences, but should explain how a new benchmark can assess the assumptions and interpret legitimate differences without treating a group as defective.

Questions for authors

Q1. How do the per-panel flags and reliability–fidelity inversions change under plausible unequal common-signal loadings or explicit scale diagnostics?

Q2. How stable are the SummEval component-test intervals under document-level resampling, and what dependence across repeated raters is represented?

Q3. What practical recruitment and diagnostic protocol would justify treating two newly collected panels as independent measurements of the intended common target?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.