- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper challenges the common practice of selecting LLM judge panels by individual accuracy ("curation"). The authors argue that curation's apparent success is an artifact of aggregator misspecification: conditionally independent (CI) models like Dawid-Skene double-count correlated judges, making a small curated subset look artificially better than the full panel. The paper proves this mechanism analytically in a two-block Gaussian model, isolates it empirically via a matched covariance intervention on four benchmarks with 38–144 judges, and proposes risk-based panel selection under dependence-aware aggregation as a practical alternative. CI aggregators favor Top-5 by 0.09–0.13 NLL; dependence-aware aggregators shrink this gap to ≤0.015 or reverse it. Risk-based selection stays within 0.017 NLL of the best accuracy-ranked rule without requiring manual panel size tuning.
Critical strengths
- S1: It is really appreciated that this work studies a practical problem, extremely relevant in the industrial setting.
- S2: It is appreciated that you tried to understand why in JudgeBench the results are not following the pattern and giving an explanation.
- S3: The Experimental Setup is clearly described with all the necessary details.
Critical weaknesses
- W1: Code is available. But running it is impossible. There is no
requirements.txtfile and there are lots of files. Navigating all of them to understand how code works is practically impossible. - W2: I think that the Related Work Section should be expanded a bit. This is a well-known topic in the literature and lots of works try to address this problem. Half of a page maybe is not enough.
- W3: It is not clear to me why you are not comparing your results to FCPS, which is doing something very similar and is the most related work.
- W4: With , , candidates, and selection slices of 35–299 items, the bound is 1.6–4.7 nats. This means it is two to three orders of magnitude larger than any NLL difference reported (0.001–0.022).
Questions for authors
- M1: All theory and experiments assume .
- M2: Proposition 3 requires knowing which judges are "redundant" and which are "diverse." No procedure is given for estimating this from votes.
Also the text in the Weaknesses Section of the form can be considered as additional questions.