acceptodds
Under review as a conference paper at ICLR 2027

When the Judge Changes the Verdict: Rank Instability and an Auditing Protocol for LLM-as-a-Judge

Abstract

LLM-as-a-judge reliability is typically treated as a model property, but we show it requires auditing along two orthogonal axes. Rank stability across sessions is not guaranteed: even with identical inputs, greedy decoding, and the same weights, a served judge can realize distinct deterministic scoring functions across sessions. This failure—operating as a score re-permutation across documents—is invisible to within-session retests and distributional checks, yet reverses downstream difference-in-differences estimates on up to dimensions. Calibration to a shared reference is pervasively weaker: across – open-weight judges, within-session agreement is near-perfect (), yet alignment to a production scale correlates at only – (median ). We propose two mechanism-agnostic protocols: a 20-document reference anchor detecting session deviations, and multi-session consensus recovering majority-mode estimates when the anomalous mode is minority. On 188k legal documents, the anchor classifies all 16 campaigns perfectly; consensus eliminates worst-case sign-flips when and reduces them three-fold at . The framework generalizes to new tasks or stacks by substituting the anchor; external validation of the reference mode against human annotations remains outstanding for this domain. Reliability is a property of a measurement campaign, not a model alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-as-a-Judge, Rank Stability, Measurement Reliability, Mode Switching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rQLXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper stuties the reliability of LLM-as-judge methods. The author focus on two orthogonal axes: cross-session rank stability and cross-model scale calibration.

The paper did experiment on a 188k Chinese criminal court judgments dataset scored across six ordinal dimensions using Qwen2.5-14B-Instruct and 10 other open-weight models, and discovered that judge can switch between two nearly orthogonal scoring modes across sessions. The paper then proposed a 20 document reference anchoring and multi-session voting method to detect and mitigate the issue.

Critical strengths
  1. the paper demonstrates why within-session retesting and marginal score calibration are insufficient, the cross model analysis across 11 open-weight model judges shows that even when model is bit2bit deterministic, their document ranking still has a low correlation, this provides solid evidence for the claim.

  2. The 20-document reference anchor protocal is simple, mechanism-agnostic and computationally effecient.

Critical weaknesses
  1. Mode B appeared in 4/16 campaigns, but didn't appear in 8 controlled conditions and 63 new sessions (M.7, M.12), and its root cause is unknown. The paper's main claim of mode B is weakened given it's only relying on those 4 campaigns.

  2. "5/6 DID sign reversal" is overstated given sampling uncertainty and permutation null expectations. B-mode DID estimates are all close to zero, which is what random re-permutation predicts.

  3. In axis 2, the calibration is measured by comparing judges against Qwen2.5-14B's scores, not against human labels (the human labeled data is only 8 documents and a single annotator). This only shows that other LLMs disagree with Qwen2.5-14B so this does not establish a new reliability axis.

Questions for authors
  1. Since mode B wasn't reproduced in later sessions, coud you provide more comment on data side explainations considered to rule out the data-misalignment error?

  2. For the DID hazard, could you report the B-mode estimates against a permutation null (A scores randomly re-permuted)?

  3. The abstract says the anchor classifies "all 16 campaigns," but Table M.4 lists 7.

Reviewer_jd6bOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work examines the reliability of LLM-as-a-Judge across sessions and finds that ranking stability is not guaranteed, even under identical settings, and that such instability may go undetected by conventional tests. The authors further analyze its impact on downstream tasks and propose strategies to mitigate the issue.

Critical strengths
  1. The paper addresses an important topic, as the reliability of LLM judges is critical for downstream tasks and analyses.

  2. The paper reveals previously overlooked issues with evaluating the reliability of LLM judges.

Critical weaknesses
  1. The novelty is limited. The paper primarily analyzes existing issues, while the proposed voting strategy is based on established statistical methods.

  2. The underlying cause of the transition from Mode A to Mode B remains unclear.

  3. Mode A is treated as the reference for scoring, but its validity has not been verified against human annotations. Therefore, the proposed method may improve alignment with Mode A rather than with human judgments.

Questions for authors

See Weaknesses above.

Reviewer_Rmn6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper claims to have identified problems with the current measurement standards for LLM-as-a-judge. It points out two types of problems. The first is that, with fixed model weights and prompts under greedy decoding, the model can still randomly adopt two modes of scoring, one of which is anomalous. The second is that most open-source judge models, though stable in their rankings, align poorly with the benchmark. Accordingly, it proposes two orthogonal dimensions for measuring the scoring capability of LLMs: cross-session ranking stability and calibration to a common reference standard. The paper further proposes two mitigation methods: the Reference-anchor protocol and Multi-session consensus. The experiments were validated on open-source models led by Qwen2.5-14B-Instruct and the CAIL2018 dataset. On the main results, the reference anchor classified all 16 measurement campaigns perfectly ; multi-session consensus eliminated the worst-case 5/6 sign flips when the anomaly rate p_B = 0.1, and reduced sign flips roughly three-fold at the historical arrival rate p_B ≈ 0.25, while when the two modes were equally frequent no number of sessions could recover the reference estimates.

Critical strengths

The paper identifies cross-session scoring inconsistencies in Qwen2.5-14B-Instruct despite fixed model weights, prompts, and greedy decoding, and distinguishes cross-session ranking stability from calibration to a common reference standard. This finding and the resulting evaluation perspective constitute the paper’s strongest contribution: they show that stable rankings do not necessarily imply reliable judgments and provide a useful basis for scrutinizing existing evaluation results.

Critical weaknesses

1.The experiments primarily use the CAIL legal dataset, and the two-regime scoring phenomenon is observed only in Qwen2.5-14B-Instruct. It therefore remains unclear whether the findings extend to other domains and judge models. The possibility of three or more scoring regimes is also insufficiently examined. These limitations directly affect the applicability of the proposed metrics and mitigation methods, leaving the broader claims about LLM-as-a-judge inadequately supported.

2.Calibration to a reference standard is already well studied, while the paper does not sufficiently establish why cross-session instability under greedy decoding has enough practical impact to warrant a substantial standalone contribution. Multi-session consensus also changes the single-session evaluation setting used in the main experiments, yet its mechanisms, assumptions, and sources of improvement are not clearly distinguished from self-consistency based on repeated stochastic sampling from the same model. This weakens my assessment of the methods’ novelty and distinct value.

Questions for authors

1.The experiments primarily use the CAIL legal dataset. What evidence supports extending the paper’s conclusions and auditing framework to other LLM-as-a-judge tasks and domains?

2.What experiments or analyses can demonstrate that the observed two-regime anomaly is a general issue rather than a model-specific phenomenon? Have you considered the possibility of more than two scoring regimes?

3.Given the extensive prior work on reference-based calibration, what is the paper’s specific incremental contribution over existing calibration research? Why does cross-session stability under greedy decoding warrant dedicated investigation?

4.Please clearly define the proposed multi-session consensus method. How does it differ from conventional self-consistency based on repeated higher-temperature sampling within the same session? Why is aggregation across independent sessions an appropriate approach to addressing the observed scoring-regime anomalies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.