acceptodds
Under review as a conference paper at ICLR 2027

THE JUDGE LOTTERY: DIAGNOSING WHETHER COMPARISONS BETWEEN STOCHASTIC LLM JUDGES ARE ADEQUATELY SUPPORTED BY RESAMPLING EVIDENCE

Abstract

LLM judges gate safety decisions and supply RLHF/DPO training signal, yet a single verdict is stochastic – shaped by decoding temperature/seed, presentation order, and which ensemble member is queried. Prior work shows judges are biased; it does not tell a practitioner whether a specific small-sample comparison is adequately supported by evidence or is instead an artifact of a favorable resampling draw. We call this the Judge Lottery and formalize it with a disagreement rate D(K), a matched-pair order-flip rate, and a scoped, single-example planning estimate K min , together with a five-step protocol (calibration, resampling, a degenerate-verdict screen, regime classification, and Resample-Consensus reporting) for diagnosing, not certifying, whether a comparative judge claim rests on adequate resampling evidence. In a direct pilot on two open-weight judges (Qwen2.5-1.5B/3B-Instruct, 30 shared MT-Bench examples, 1,920 calls, valid rate 1.000), matched AB/BA verdicts disagree at 0.744 (1.5B) and 0.662 (3B), a large effect with no detectable temperature variation that persists under deterministic decoding and that further resampling at fixed order does not correct; a cluster-respecting re-analysis finds the two judges’ rates statistically indistinguishable despite each being individually large. Synthetic content-blind controls show low verdict variance can coexist with a stable, non-evaluative heuristic. We support the protocol with a mechanism-level analogue from a concurrently submitted differential-privacy fairness study and map it onto representative LLM-judge failure modes. A verified post-submission extension to a 7B same-family judge and a second benchmark finds order bias shrinking, but not disappearing, with scale, alongside improving agreement with real human preference labels (Cohen’s κ: −0.122 at 1.5B, +0.368 at 7B). All direct findings are scoped to same-family judges, two benchmarks, and 30 examples; the degenerate-verdict screen and RCS’s order-sampling policy remain diagnostics under refinement rather than finished gates.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM-as-a-Judge, LLM Evaluation, Judge Reliability, Evaluation Robustness, Resampling, Preference Evaluation, Safety Evaluation, MT-Bench, Reinforcement Learning from Human Feedback, Direct Preference Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qvMtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses LLM judge reliability, providing a framework and running a couple experiments on Qwen judges.

Critical strengths

none

Critical weaknesses
  • This paper is very hard to follow - I would recommend a thorough rewrite starting from the abstract and intro to really focus your ideas. It is highly disorganized throughout, very challenging for a reader to understand what the contribution is beyond something to do with judge reliability. I don't want to overclaim exactly in what way AI was used to write this paper but I will say that it reads as AI slop: sentences such as "We present this contrast as a substantive, worked demonstration of pseudo-replication in LLM evaluation, not as a record of what changed between drafts of this manuscript” suggest this is an intermediate artifact outputted in a research project by an agent rather than a worthwhile submission
  • experiments seem quite narrow: even beyond the disorganized communication, it seems that only a single small dataset was used (authors say themselves that these results cannot be generalized, " All findings hold only for these two Qwen2.5 judges and 30 MTBench examples"
Questions for authors

n/a

Reviewer_AuAfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how much confidence we should place in comparisons made using stochastic LLM judges. It proposes the “Judge Lottery” framework, combining a disagreement rate, a matched-pair order-flip rate, a limited sample-size planning estimate, and a five-step diagnostic protocol. The goal is to distinguish random variation across judge calls from systematic bias caused by presentation order, and to encourage reporting judgments based on repeated evaluations rather than a single run.

The main experiments use Qwen2.5-1.5B and 3B judges on 30 MT-Bench examples. The authors find high order-flip rates (0.744 and 0.662), including under deterministic decoding, and show that treating repeated evaluations of the same examples as independent can produce misleading statistical significance. An additional same-family 7B evaluation and a second benchmark suggest that order bias decreases with model scale, although it does not disappear.

Critical strengths

1- The paper raises an important practical question: whether a comparison involving LLM judges is actually supported by enough repeated evaluations. The distinction between decoding randomness and systematic presentation-order bias is useful, especially for applications where judge outputs become training signals or safety decisions. The observation that order effects persist under deterministic decoding helps illustrate why simply increasing the number of samples is not enough.

2- The statistical re-analysis is a meaningful part of the paper. In particular, the authors show that an apparently significant difference between the two judges disappears when the 30 underlying examples, rather than the 450 repeated matched pairs, are treated as the experimental units. I also appreciate that the paper explicitly acknowledges several limitations and provides details of the experimental setup and reproducibility gaps.

Critical weaknesses

1- Most of the proposed components rely on established techniques, including bootstrap resampling, majority voting, and positional-bias analysis. The paper combines these ideas into a diagnostic framework, but the methodological contribution is limited. More importantly, several central components remain incomplete. The degenerate-verdict screen lacks a well-defined statistical test and threshold, and the Resample-Consensus procedure has unresolved sampling and tie-handling rules. There is also an inconsistency in how Flip P. is defined and interpreted. These concerns make it difficult to assess the reliability of the proposed framework.

2- The experiments are limited to a small number of examples and models from the same family. Although the additional 7B experiments are useful, broader validation and comparisons against established debiasing methods are still missing. Some parts of the analysis are also not fully reproducible. Furthermore, the presentation needs substantial improvement. Table 1 extends beyond the normal page margins, making it difficult to read. Many paragraphs are unnecessarily long, dense, and difficult to understand, particularly in the methodology, experiments, and discussion sections. The paper would benefit from shorter paragraphs, clearer explanations, and better organization. These issues significantly weaken the clarity and supporting evidence of the work.

Questions for authors

1- Section 3 says that Flip P. pools all 32 configurations, including both AB and BA orderings, but describes the statistic as decoding-only. How is order-induced variation excluded? Can the authors report separate decoding-only and order-sensitivity measures and show how this affects the regime assignments?

2- How would the degenerate-verdict screen and STABLE/UNSTABLE classification work in an actual deployment when the screen has no finalized test or threshold and the RCS order-sampling policy is unresolved? Which parts of the five-step procedure are currently executable without additional design choices?

3- Can the authors compare the proposed reporting protocol directly against standard AB/BA aggregation, swap-tie evaluation, or other order-debiasing approaches under the same call budget? Results on at least one additional judge family or a dedicated reward/judge model would also help establish generality.

4- Can the authors clarify the discrepancy between Section 6, which says cluster-bootstrap intervals for each judge remain uncomputed, and Section 8, which reports such intervals for a tie-excluding variant?

Reviewer_pidLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper examines the reliability of LLM-as-a-judge evaluations, focusing on how decoding randomness and response presentation order affect comparative judgments. It proposes diagnostic workflow combines repeated sampling, order-swapped evaluations, and statistical analyses to assess the strength of evidence supporting judgments.

Critical strengths
  1. Extensive statistical analyses strengthen the credibility of the findings. The paper conducts multiple statistical analyses, including bootstrap confidence intervals, paired permutation tests, paired t-tests, Wilcoxon tests, and cluster-aware repeated-measures analysis. Importantly, correcting the pseudo-replication error changes the judge-to-judge conclusion, making the interpretation more statistically defensible.

  2. The paper addresses a practically important evaluation problem. As LLM judges increasingly inform benchmarking, preference training, and safety decisions, understanding whether their comparative judgments are supported by sufficient evidence is valuable. The paper appropriately shifts attention from merely identifying judge bias to assessing the evidential basis of specific comparative claims.

Critical weaknesses
  1. The methodological novelty is limited. Most components, including repeated sampling, matched order swapping, bootstrap uncertainty estimation, and majority-vote aggregation, are established techniques. The main contribution lies in organizing them into a diagnostic workflow rather than introducing a substantially new estimator, algorithm, or theoretical result.

  2. The diversity of evaluated models is very limited. The main experiments consider only two small Qwen2.5 models. Although the supplementary extension adds a 7B model, all successfully evaluated judges belong to the same family. Consequently, the findings provide limited evidence about other model families, frontier models, or purpose-built evaluators.

  3. The number of independent evaluation examples is too small. Despite the 1,920 calls in the main experiment, the study contains only 30 distinct examples. Repeated calls improve within-example characterization but do not replace broader sample coverage. This small sample limits statistical power and the generalizability of the conclusions.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.