- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper discusses how the agreement between LLM-human expert might not translate to LLM-human crowd (basically a different panel/pool) which might reduce the trust on using LLMs as efficient judges. This is because the reliability statistic using a single rater is which is used in reports but benchmarks usually use the mean of independent raters, whose reliability factor is different (and more accurate to use) compared with -- this is the first problem the authors bring on unit . But even is not completely reliable as it has 2 components - the signal a panel should track; fidelity which is supposed to capture the panel-independent quality i.e something a different panel from a different pool can/ideally should also come up with and the covariance/common-factor among panel raters which is i.e a panel-specific shared-component that can't be removed -- this is the second point on validity. And understanding how the judge score fitting to panels is important to trust LLM-Judge scores because it's possible to have a high but a low making the LLM-judge less trustworthy.
Critical strengths
The problem the paper tries to discuss is very important given the scale at which LLM-Judges are being used - modeling expert human-LLM agreement, crowd-LLM agreement and LLM biases (LLM-LLM agreement) are important to study and design pipelines for data collection, evaluations is critical. The paper tries to understand human-human (different panels) and human-LLM (panel-fitting) settings. So, understanding what metrics to rely on is very important where the authors provide different tools, algorithms to understand for eg: by having >= 2 raters and >= 2 panels, the references can be checkable (L265-269).
Critical weaknesses
- Contributions (L58-59, 75-77, 121-127): While I appreciate the authors genuine intentions on specifying their particular contributions, in it's current form this is largely diagnostic analysis using classical tools for LLM-Judge x human panels. On L61, the 3 contributions listed are 3 findings and while they are interesting, I feel a lot of it is already well-studied (already cited xRR paper, Reliability of aggregated human annotations in related works) and the authors can expand the study meaningfully to strengthen the paper. For eg, understanding what exactly the judges actually fit in expert vs crowd panels is a good extension:
- Possible experiment, understanding what LLM-Judge and panels share: Currently the paper says that LLM-Judge agrees to one human-panel (experts) better than another panel (crowd) and there's a common signal and some panel-specific noise. Which means, the judge and the experts share something beyond true quality (Fig. 3). Now, there's research on aggregating multiple LLM judges on same problem but from the judge side: judges agree with each other because of shared biases such as verbosity or style, not quality [1, 2, 3]. Discussing this connection, or ideally testing it (e.g., treating judges as panels in the proposed audit), would help explain what the judge and the experts actually share and strengthen the paper's 1st point.
- On presentation and sentence formations: I feel the paper can be presented a lot better. I strongly recommend the authors to consider polishing the writing and work on the presentation as the current form is very hard to understand at various sections. Few examples from introduction section below:
- "We start with what it does to a decision. It can put two references in the wrong order, and a judge can agree with one of them far beyond anything a second panel confirms. The designs that maximise the number (one large panel) or the reference’s fidelity (many single-rater panels) are exactly the ones that cannot be checked"
- "The two share a name, and they can rank panels, and the judges tested against them, in opposite orders. That can change which system, or which LLM judge, a study declares better."
- CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (ICML 2026): https://openreview.net/forum?id=3WPDFjZ1UT
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025): https://openreview.net/forum?id=3GTtZFiajM
- Large Language Models are not Fair Evaluators (ACL 2024): https://aclanthology.org/2024.acl-long.511/
Questions for authors
On formatting: The paper might be benefitted from notations table in Appendix