acceptodds
Under review as a conference paper at ICLR 2027

Second-Order Response Laws for LLM Judges: From Prompt Stability to Decision Quality

Abstract

When an LLM judge changes its verdict after a prompt is reworded, the change may come from the wording or the randomness of another call. Confusing these sources can mislead model comparisons and response selection, yet separating them with few repeated calls is difficult. Plug-in prompt-dispersion scores combine both sources. We develop a framework that connects measurements of judge variability to decision quality. Second-order response laws describe how verdict distributions vary across prompts; crossing prompts with candidate orders separates prompt, position, interaction, and call variation. Agreement between distinct calls removes finite-call bias. We derive the prompt estimator's exact variance and establish matching upper and lower bounds for detecting weak prompt effects under a fixed, balanced design. Identical-input controls and an independent reference with more repeated calls validate the correction. On a matched panel, correction reverses the Qwen–Gemma-3 ranking by prompt sensitivity. We analyze over one million recorded categorical judgments across six judge models and three benchmarks. Cooling makes Qwen's repeated calls more consistent while its corrected prompt estimate rises nearly fivefold, from .011 to .054; the plug-in estimate rounds to .05456 at both temperatures. To examine what greater repeatability means for decisions, we evaluate independent groups of votes balanced across candidate orders. In Qwen, cooling produces more ties; changing the tie rule on the same votes shifts the estimated accuracy effect of cooling by 9.84 percentage points. An analytical model of position dependence explains how balanced votes can concentrate near a tie boundary and lower accuracy when ties lead to abstention, even when the judge favors the correct answer on average. The value of consistency therefore depends on alignment with correct answers, how votes are combined, and the cost of abstention.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM-as-a-judge, LLM evaluation, prompt sensitivity, position bias, variance decomposition, finite-sample inference, uncertainty quantification, evaluation reliability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XTubOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the sources of variation in "LLM-as-a-judge" evaluations—specifically whether they stem from prompts, the order of candidate answers, the interaction between the two, or the inherent randomness of repeated calls. Focusing on the distribution of categorical responses for each input condition, the authors establish a cross-variance decomposition and utilize consistency across calls to construct an unbiased estimator, thereby correcting the overestimation of prompt sensitivity caused by a limited number of repeated calls. The theoretical analysis derives the exact variance of the estimator and establishes matching upper and lower bounds for detecting weak prompt effects under fixed, balanced designs. Furthermore, the paper examines the relationships among response distributions, alignment with ground-truth answers, voting rules, and the cost of abstention; it also derives the risk difference and conditions for unbiased estimation regarding deployment designs with fixed budgets. The experiments cover six judge models and three benchmarks, incorporating controls for identical inputs, high-repetition independent baselines, and simulations based on known distributions, alongside interventions involving temperature, prompting strategies, and reasoning modes. The results demonstrate that bias correction can alter the ranking of models based on prompt sensitivity; lowering the temperature can simultaneously increase estimates of both repeatability and prompt sensitivity; and applying different tie-breaking rules to identical voting outcomes can change the assessment of intervention effects.

Critical strengths
  1. This article rigorously handles limited call bias and supports statistical verification: directly using empirical distribution to measure cue dispersion can easily confuse call noise and cue effects. This bias is affected by the number of repetitions, models and decoding conditions, and even changes the model ranking. The author clarifies that this estimation method is equivalent to trace mean square correction on a sample-by-sample basis. The core is to derive the upper and lower bounds of precise variance and matching detection under a fixed cross design, and verify the practicability of the correction method through input control, independent reference evaluation and simulation.
  2. The article also combines independent voting experiments to analyze that stability is not equal to decision quality: not only does it generally point out that small variance does not necessarily mean accuracy, Proposition 2 constructs a scenario to illustrate that when vector risk and Brier loss decrease, if a tie is recorded as an abstention, the accuracy will decrease instead, reflecting the role of position dependence and voting boundaries. Experiments show that voting rules will significantly change the results. Qwen's accuracy fluctuates by up to 9.84 percentage points under the cool down setting. The inclusion of abstention costs also improves the interpretability of conclusions.
Critical weaknesses
  1. The statistical certainty regarding conclusions about temperature parameters is insufficient. For Qwen, the point estimate for the prompting effect is +0.04262; however, the confidence interval at the invocation level includes zero, as does the confidence interval for Gemma. At the question-set level, the positive intervals are derived solely from existing invocation data. In Table 2, the confidence intervals for both results concerning accuracy changes include zero. Current evidence merely indicates that evaluation outcomes are influenced by voting rules; it does not definitively establish whether lower temperatures increase or decrease accuracy—the reported "nearly fivefold increase" is merely a point estimate.
  2. The scope of the experiments is relatively limited. The temperature experiments utilized only 40 questions, and the reasoning experiments involved just 32. Although the total number of model invocations is substantial, there has been no large-scale, independent verification at the task level.
  3. There is a lack of robust empirical support for the incremental value of these findings in informing deployment decisions. While the identity relating Decision/Evaluation risk differentials is highly interpretable, the evidence for using it to select superior deployment strategies is weak. This is reflected in the confidence intervals across several supplementary tables, equivalence tests, predictive comparisons, and the sign reversals observed across different datasets in JudgeBench. While these results do not invalidate the identity itself, they highlight a gap between achieving unbiased measurement and selecting reliable deployment strategies under a constrained budget. Furthermore, the risk metrics derived from this formula cannot serve as a substitute for accuracy or objective functions that account for the costs of abstention; therefore, conclusions regarding deployment should be approached with greater caution.
Questions for authors
  1. Regarding inferences on temperature effects: Is there support—such as from independent replications or more rigorous validated inference methods—for the observed direction of the temperature effect? ​​If not, can a clear distinction be made between temperature-related conclusions based solely on point estimates and those backed by guarantees at the query level (accounting for finite samples)? Uncertainty at the query level should not be conflated with intervals derived from the set of prompts itself.
  2. Regarding the practical value of deployment choices: Once the selection rule is fixed during development, what actual gains are realized during testing compared to baselines such as always selecting D, always selecting E, or a zero-risk-difference prediction? Please report the common-mean risk and final decision loss separately, and explain whether the measurement of query costs is justified.
  3. Regarding cross-model rankings: Regarding the rank reversal between Qwen and Gemma-3, can you provide the direct difference between the models and the corresponding query-level intervals? Given that the experiment compares full configurations—including decoding, stopping, and parsing settings—please clarify whether this rank reversal supports a conclusion specific to the configuration or a generalizable model property.
  4. Regarding the interpretation of the prompt set: What specific standard variations were excluded during the automated equivalence check of the 48 semantic prompts? Subsets consisting of the same sentences in different orders serve as valuable controls; could you clarify whether the main conclusions and their associated uncertainties hold when using only this stricter subset?
6 comments
AuthorsOpenReview ↗

Response to Reviewer XTub (1/4): W1 and Q1, temperature inference

Thank you for the careful reading and the specific questions. We have resolved the cooling contrast on the original fixed items and prompts with additional calls and an independent replication, extended the cooling study to 320 new items from three benchmarks and 24 unseen prompts, replicated the reasoning result on 80 new RewardBench 2 items, separated the decision effects by tie rule, and reported held-out deployment comparisons with risk and decision loss kept apart. In total we ran nine pre-specified studies with 775,680 new judge calls. For each study, we fixed the analysis plan and success criterion before data collection and ran the specified analysis unchanged.

Experiments: E1–E7, E9 and E10 collected new data; E8 compares aggregation methods using E7 calls.

W1 and Q1. Which temperature conclusions carry call-level guarantees

Independent replication and calibrated call-level inference both support the direction of the effect. Every row below targets the original 40 items and 48 prompts (the census) under fresh calls, the call-level question you pose; intervals over items or over the prompt set answer different questions, and we keep them separate.

T = 0 − T = 1 prompt contrast, original items and promptsQwenGemma
Table 1 (worst-case bound, R = 4, per-temperature bounds differenced)[−.0570, .1259][−.0799, .0987]
Worst-case bound on the contrast itself, all runs pooled+.0440 [.008, .080] at R = 8+.0133 [.0026, .0240] at R = 40
The 32 new repeats per cell alone (E6)[.0010, .0257]
Exponential distribution-free bound at Table 1's four-contrast error level, all runs pooled (post hoc)[.0173, .0701][.0046, .0219]
Independent replication with fresh seeds (E1, R = 4), EV interval+.0454 [.0398, .0510]+.0119 [.0100, .0138]
Calibrated intervals on the original calls, EV and IP[.0363, .0489] and [.0212, .0641][.0120, .0164] and [.0012, .0272]

The second and third rows apply the paper's Cantelli bound directly to the contrast under the Theorem S1 envelope, with no approximation; their error allocation covers the prompt and call contrasts between T = 0 and T = 1 jointly within each judge. Gemma's pooled interval was the pre-specified primary endpoint of E6, and Qwen's was computed post hoc. At the error level of Table 1's four contrasts, the exponential bound of the revised paper (Proposition S11, same assumptions) gives a distribution-free interval that excludes zero for both judges on the pooled runs (post hoc), and this still holds at an error level one thousand times smaller. The calibrated intervals on the original calls, Bonferroni-adjusted over the same four contrasts, also exclude zero for both judges, and across ten known-law settings their worst joint coverage was 99.8% and 95.4%. The replication met its pre-specified criterion (EV interval above zero) for both judges. The exact-variance (EV) interval uses the exact finite-sample variance of Theorem S1 with a normal reference. The item-paired (IP) variance estimate is conservative in expectation for the same fixed items and prompts.

Call variation falls under cooling in every study, as Table 1 established with finite-sample intervals. The revised paper reports the "nearly fivefold" rise with its call-level interval, 4.8 (3.3 to 8.5); the replication gives 5.5, and both runs pooled give 5.2 (4.2 to 6.8).

Accuracy under cooling depends on the tie rule, and the decision effects replicate.

Qwen, T = 0 − T = 1, ppOriginal censusFresh-call replication200 new items (E2, E3)120 new items (E9)
Tie-rule interaction+9.84 [1.68, 18.01]+8.44 [3.17, 13.71]+3.94 [2.02, 5.85]+4.43 [1.76, 7.09]
Wrong decisions, abstention at ties−12.81 [−23.07, −2.56]−10.62 [−16.53, −4.72]−4.69 [−6.71, −2.66]−5.94 [−9.13, −2.74]
Accuracy, fair tie-breaking+7.97 [2.83, 13.11]+4.69 [−.20, 9.58]+2.19 [.14, 4.23]+2.14 [−.62, 4.89]
Accuracy, abstention at ties−1.88 [−10.86, 7.11]−3.75 [−10.77, 3.27]−1.75 [−5.01, 1.51]−2.29 [−6.31, 1.72]

For the original census, the interaction and wrong-decision intervals are the simultaneous ones of Table 2, whose Qwen accuracy intervals, [−.87, 16.81] and [−14.43, 10.68], include zero. All other intervals are pointwise item-paired 95% intervals. The accuracy rows of the first two columns and the pooled E2–E3 column were computed after the frozen analyses, while the E9 column reports pre-specified secondary analyses. Under fair tie-breaking, the estimated accuracy gain is positive in every column, and pooled post hoc over 240 distinct items it is +2.88 pp [1.02, 4.73]; under abstention at ties the change is not significant. The tie-rule interaction is positive, with an interval excluding zero, in all four columns. For Gemma, the estimates in every row and column are within 1 pp of zero. The revised paper reports the decision results rule by rule (Section 5 and Table S51).

AuthorsOpenReview ↗

Response to Reviewer XTub (2/4): W2 and Q4, scope and the prompt set

W2. New items, a third benchmark, unseen prompts and the reasoning study

The cooling increase now holds on 320 items never used before, from three benchmarks, and under 24 unseen prompts. E2 and E3 met their pre-specified criterion for both judges, and in E9 the same increase was a pre-specified secondary expectation that held for both (T = 0 − T = 1 prompt contrast, item-paired 95%):

New itemsQwenGemma
E2: 120 items (60 LLMBar, 60 JudgeBench)+.0421 [.0305, .0536]+.0106 [.0051, .0162]
E3: 80 RewardBench 2 items (20 each from Factuality, Focus, Math and Precise IF)+.0262 [.0149, .0375]+.0109 [.0033, .0185]
E9: 120 items (24 LLMBar, 24 JudgeBench, 72 RewardBench 2)+.0305 [.0207, .0403]+.0067 [.0030, .0104]

The point estimate is positive for both judges in both E2 datasets and all four RewardBench 2 domains. Pooled post hoc over the census and the first 200 new items, the increase is +.0369 [.0303, .0435] for Qwen and +.0113 [.0077, .0149] for Gemma. Cooling raised the corrected prompt dispersion on 177 of these items and lowered it on 55 for Qwen, and on 64 versus 12 for Gemma (exact sign tests, p < 10⁻⁸; post hoc). Call variation falls in every study. Under 24 new prompts from two families absent from the bank (layout and voice), the increase is +.0387 [.0306, .0467] for Qwen and +.0114 [.0087, .0140] for Gemma (EV). We added items before prompts because items carry most of the uncertainty (for Qwen, an item standard error of .0084 against .0027 for prompts).

The reasoning result was replicated on the same 80 new RewardBench 2 items used in E3 (E5, pre-specified; 7,680 judgments). Thinking raises call variation from .0234 to .0766, +.0531 [.0222, .0840], meeting the pre-specified criterion.

Q4. The automated equivalence check and the clause-order subset

No prompt was excluded. All 48 candidates were accepted, each with 4 of 4 equivalence judgments from two reviewer models in two input orientations, while both identical-text positive controls passed and all four meaning-changing negative controls were flagged (216 checks). The check required the same task, criteria, scope and weights, exclusions, tie rule, label meanings and output contract, no added procedure or examples, and entailment in both directions. An off-the-shelf NLI model (RoBERTa-large-MNLI), independent of how the prompts were built and of both judges, also finds bidirectional entailment for all 48 prompts (lowest probability .974) and for the 24 unseen prompts.

The clause-order subset uses exactly the same five sentences, including the output-format line, in 12 orders, so the wording is held fixed. The rise in prompt dispersion and the fall in call variation hold there for both judges, with intervals that exclude zero:

Clause-order subset (K = 12), T = 0 − T = 1QwenGemma
Corrected prompt dispersion, EV+.0383 [.0262, .0505]+.0127 [.0092, .0163]
Corrected prompt dispersion, IP[.0150, .0617][.0014, .0241]
Call variation, IP−.3530 [−.4156, −.2903]−.0398 [−.0737, −.0058]

Across each family alone (K = 12) and each set of three families (K = 36), the prompt increase ranges from +.0346 to +.0436 for Qwen and from +.0127 to +.0148 for Gemma. EV excludes zero in all 16 analyses, and IP in all eight for Qwen and seven of eight for Gemma, the exception being the lexical family alone, [−.0011, .0274].

In the revised paper, Section 4 summarizes these analyses, Section 6 reports the reasoning replication, and Section M.3 gives the full results (Tables S49 and S50).

AuthorsOpenReview ↗

Response to Reviewer XTub (3/4): W3 and Q2, deployment value

W3 and Q2. Held-out gains, risk and decision loss, and query cost

We report the held-out comparison against always D, always E and a zero-difference prediction, with common-mean risk and decision accuracy kept apart. Gains from choosing between D and E are not significant at the studied budgets, consistent with the small cross-order covariance that determines their risk contrast. Choosing between one call and voting produced a prospective accuracy gain: in the pre-specified E9 study, choices fixed in development gained 5.89 pp [4.48, 7.30] in accuracy over the alternative in the five pre-specified settings, on 120 unseen items.

In ten settings (three judges on two benchmarks, and four judging policies), we froze the development choice between D and E and compared it with always using D and always using E on independent test deployments. For each measure, the table gives the range of selected-minus-fixed estimates across settings and the estimate with the largest absolute magnitude, together with its 95% interval; in every setting and for every measure, the interval includes zero.

Selected minus fixed design, 10 settingsAlways DAlways E
Squared-error risk around the common mean (lower is better)−.0025 to +.0014; largest −.0025 [−.0098, .0049]−.0015 to +.0044; largest +.0044 [−.0044, .0132]
Accuracy with abstention at ties, pp−1.25 to +.42; largest −1.25 [−3.68, 1.18]−1.07 to +1.77; largest +1.77 [−1.14, 4.68]
Accuracy with fair tie-breaking, pp−.73 to +.03; largest −.73 [−2.66, 1.20]−.71 to +.89; largest +.89 [−1.02, 2.80]

Across the ten settings and any abstention cost c between 0 and 1, the largest absolute estimated difference in decision loss, P(wrong) + c·P(abstain), is 1.25 pp against always D and 1.77 pp against always E.

The risk contrast has the exact form (Eq. 12):

RE−RD=K4(K−1)(Uint−Uprompt(q))R_E-R_D=\frac{K}{4(K-1)}\left(U_{\mathrm{int}}-U_{\mathrm{prompt}}^{(q)}\right)

The term in parentheses is minus the covariance between a prompt's effects in the two orders. Thus, E has lower risk when these effects carry over between orders, and D when they reverse. That covariance is small and its sign varies (median correlation .04 over 22 configurations, where the largest estimated ∣RE−RD∣|R_E − R_D| is .0026), which is why the sign of the observed difference can change between datasets.

A zero-difference prediction has lower expected squared error than an unbiased estimate when ∣Δ∣\lvert\Delta\rvert is below the estimator's standard error, which is the regime of the predictive comparisons in Table S39. Across 170 known-law settings, zero is better in all 62 settings with a signal-to-noise ratio below one and in none of the other 108.

The new deployments extend the calibration check of Table S32 to decision outcomes: across 120 comparisons in eight panels, the median forecast–observation gap is .4 pp and the largest 3.5 pp. Table S32's own intervals were too wide to establish equivalence within its ±.005 margin, but every interval contained zero.

Risk around the common mean and decision loss can disagree, and the choice between one call and voting shows it. For the same prompt–order sampling distribution, averaging D's four verdicts cannot increase expected squared-error loss, by convexity, yet with abstention at ties D is less accurate than a single call for both judges on the E9 items. Counting errors, abstentions and calls separately on E9 (post hoc):

D minus single call (pp, item-paired 95%)Qwen T = 1Qwen T = 0Gemma T = 1Gemma T = 0
Abstention at ties: wrong−12.8 [−15.1, −10.6]−15.3 [−18.8, −11.7]−5.0 [−7.4, −2.6]−5.2 [−7.6, −2.7]
Abstention at ties: abstain+16.2 [12.6, 19.8]+26.8 [19.9, 33.6]+10.4 [5.6, 15.2]+10.7 [5.8, 15.5]
Fair tie-breaking: wrong−4.1 [−6.0, −2.2]−2.1 [−3.2, −1.0]−.2 [−1.1, .8]−.2 [−1.1, .7]
Fair tie-breaking: abstain−1.3 [−2.0, −.6]+.4 [−.4, 1.2]+.7 [−.3, 1.7]+.8 [−.3, 1.8]

With the cost of a wrong decision set to one, an abstention cost c and a cost λ per call, D beats a single call when ΔP(wrong)+c,ΔP(abstain)+3λ<0\Delta P(\mathrm{wrong})+c\\,\Delta P(\mathrm{abstain})+3\lambda < 0 Before the three extra calls are charged, the estimated decision loss favors D for Qwen under fair tie-breaking; under abstention at ties, the estimated break-even abstention cost is .57 to .79 of an error for Qwen and about .48 for Gemma. The measurement calls estimate all of these rates before deployment, which turns the comparison into an explicit selection rule for given costs. The next comment shows that, with abstention at ties, this rule gives lower decision loss on unseen items than either fixed strategy, on average over abstention costs from 0 to 1.

Reviewer_WrrJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how prompt wording, candidate order, and randomness across repeated calls affect LLM judgments. The authors develop a variance decomposition result and correct the bias caused by a limited number of repeated calls. They derive variance and detection bounds. They also examine how temperature, prompting, and reasoning affect these components and how the changes relate to decision quality.

Critical strengths

The paper addresses an interesting practical problem. Distinguishing prompt dependence from repeated-call randomness is useful for interpreting judge comparisons. I find that the exact variance analysis and matching detection bounds strengthen the contribution. The identical-input controls and independent reference with more repeats also provide useful validation. The use of separate calls for measurement and deployment evaluation is also interesting.

Critical weaknesses
  1. The novelty of the proposed method needs further clarification. In particular, the manuscript shows that the corrected estimator equals the classical trace-mean-square correction under the balanced design. The decomposition and the bias correction appear to have limited methodological novelty. The new contribution beyond the existing methods should be further clarified.

  2. Some empirical conclusions require more careful explanations. While the cooling results are interesting, the reported call-level intervals do not fully support the claimed increase. In particular, in Table 1, the call-level intervals for the prompt-component changes include zero for both models. The composition intervals exclude zero, but they do not provide the same inference for the fixed item and prompt set under fresh repeated calls.

  3. The practical benefit for deployment selection is less clear. The exact D/E risk comparison is useful, but the empirical evidence for choosing a better deployment design is less convincing. All eight risk-contrast intervals in Table S34 include zero and the point estimates are all very small. Moreover, in Table S39, the raw predictor improves on zero in only two of eight conditions, as the authors also note. While these results do not contradict the theoretical identities, they provide limited evidence that the framework could reliably select a better deployment design at the studied budgets.

Questions for authors
  1. It would be helpful to further clarify the theoretical advances beyond classical variance-component analysis.
  2. Can a more precise call-level analysis or independent replication clarify the cooling effect on prompt dispersion?
  3. Does the development-selected design improve held-out risk over always using D or E? Please quantify uncertainty and distinguish squared-error risk from decision accuracy.
4 comments
AuthorsOpenReview ↗

Response to Reviewer WrrJ (1/3): W1 and Q1, theoretical advances

Thank you for the positive assessment and the precise questions. We have resolved the cooling contrast on the original fixed items and prompts with additional calls and an independent replication, answered the held-out design question with intervals for both squared-error risk and accuracy, and validated the choice of evaluation strategy prospectively on unseen items. In total we ran nine pre-specified studies with 775,680 new judge calls. For each study, we fixed the analysis plan and success criterion before data collection and ran the specified analysis unchanged. We will release the plans, code and other materials with the paper.

Experiments: E1–E7, E9 and E10 collected new data; E8 compares aggregation methods using E7 calls.

W1 and Q1. Theoretical advances beyond classical variance-component analysis

Our contribution is the finite-sample theory for the crossed design with categorical verdicts: how precise the estimate is, how small an effect a given budget can detect, and which comparisons need repeated calls at all. It includes the exact variance and matching detection bounds you highlight. The point estimator itself is the classical trace-mean-square correction, a U-statistic over pairs of distinct calls, as Section 2 and Appendix A.4 state. The table pairs each result with its classical foundation. It is now Table S47 of the revised paper, cited in Section 2, with two new rows.

QuestionOur result for the categorical crossed designClassical foundation
Variance of the estimateExact finite-sample variance for fixed, heterogeneous categorical cells, including TIE and invalid outputs (Thm S1)Hoeffding's variance formula for U-statistics of i.i.d. samples
Smallest detectable effectA test valid for every fixed set of cell laws (Thm S2) and a matching lower bound (Thm S3), both of order 1/(RNK)1/(R\sqrt{NK}) (Thm 1)F tests for variance components under normality
Covariance constantThe sharp bound tr⁡(ΣaΣb)≤1/4\operatorname{tr}(\Sigma_a\Sigma_b)\le 1/4, which is attained; in the paper's certification experiment (N = 152, R = 8) it raises the share of datasets certifying θ < .01 from 79.1% to 97.1%The trace-refined bound (1 − 1/C)/2
Which design comparisons need repeated callsAn exact condition for deployment designs, equal collision moments (Prop. S6–S7); comparing D with E needs one call per cellDegree of unbiased estimability
Distribution-free tails (added)An exponential bound from a self-bounding property (Prop. S11)Concentration of self-bounding functions
Precision (added)Minimum-variance unbiased estimates and forecasts; an unbiased estimate of the exact variance for R ≥ 4, and none exists for R < 4 (Prop. S12–S13)Complete sufficient statistics
Link to decisionsProp. 2, the tie-rule identity, and unbiased forecasts for any vote rule on D's four distinct cells, checked on independent deployments (Q3)

The last row has no classical entry because variance components describe how much verdicts vary, while the accuracy of a vote also depends on how often the calls split evenly and on whether they lean toward the correct answer. Prop. 2 gives a simple case in which squared-error risk and Brier loss both improve while strict-vote accuracy (abstention at ties) falls. Because the four calls in design D come from four different prompt–order cells and are drawn independently, the chance of each vote pattern is a product of four cell probabilities. The measurement calls therefore give unbiased forecasts for any vote rule and let us choose the evaluation strategy by its forecast accuracy.

The theory also shaped the new evidence. At the effect size estimated from the first eight repeats per cell, the worst-case bound projected that about 32 to 40 repeats in total would resolve Gemma's smaller increase, as E6's frozen plan states. E6 therefore collected 32 additional repeats, giving 40 in all, and the resulting distribution-free interval excluded zero (Q2). The one-call condition answers part of Q3 before any deployment: D and E differ only through the carry-over of prompt effects between orders, so their risk difference can be estimated without repeated calls and is at most K/(4(K−1)) times the within-order prompt dispersion.

We also compared our intervals with classical ones. In simulations with known cell laws at nominal 95% (Table S60; post hoc), the classical per-category fixed-effects interval covers 85–87%, against 94.3–94.9% for the interval built on our unbiased variance estimate. For the call contrast, the classical chi-square interval covers 96.0–98.4% for Qwen but 51.8–53.5% for Gemma, while the paper's Hoeffding interval covers in every dataset.

The correction itself is closed-form and needs only logged labels. In summary, these results give a finite-sample inference framework for categorical judge responses that goes beyond classical variance-component analysis.

AuthorsOpenReview ↗

Response to Reviewer WrrJ (2/3): W2 and Q2, call-level evidence for the cooling effect

W2 and Q2. Call-level inference for the cooling effect

Yes. We collected both additional repeated calls and an independent replication, and they resolve the T = 0 − T = 1 prompt contrast for both judges on the original fixed set of 40 items and 48 prompts. Every row below targets that fixed set (the census) under fresh calls, which is the inference you ask for.

T = 0 − T = 1 prompt contrast, original items and promptsQwenGemma
Table 1 (worst-case bound, R = 4, per-temperature bounds differenced)[−.0570, .1259][−.0799, .0987]
Worst-case bound on the contrast itself, all runs pooled+.0440 [.008, .080] at R = 8+.0133 [.0026, .0240] at R = 40
The 32 new repeats per cell alone (E6)[.0010, .0257]
Exponential distribution-free bound at Table 1's four-contrast error level, all runs pooled (post hoc)[.0173, .0701][.0046, .0219]
Independent replication with fresh seeds (E1, R = 4), EV interval+.0454 [.0398, .0510]+.0119 [.0100, .0138]
Calibrated intervals on the original calls, EV and IP[.0363, .0489] and [.0212, .0641][.0120, .0164] and [.0012, .0272]

The exact-variance (EV) interval is the precise call-level analysis: it uses the exact finite-sample variance of Theorem S1 for the fixed items and prompts, with unbiased estimates of the cell covariances and a normal reference. The item-paired (IP) variance estimate is conservative in expectation for the same fixed-census target, because its expectation equals the call-level variance plus a nonnegative between-item term. The calibrated intervals on the original calls use Table 1's full family of four contrasts with a Bonferroni adjustment. Across ten known-law settings, including ones where all call noise sits in one or two items, their worst joint coverage was 99.8% (EV) and 95.4% (IP). With an unbiased estimate of the variance itself (Proposition S13; post hoc), the exact-variance intervals narrow to [.0384, .0469] for Qwen and [.0124, .0160] for Gemma, and the resulting interval covers 94.3–94.9% at nominal 95% in simulations with known cell laws (Table S60).

The replication is a fresh set of calls on the same items and prompts. It met its pre-specified criterion (EV interval above zero) for both judges. Its estimates differ from the original estimates by +.0028 and −.0023; both intervals for these differences include zero.

The second and third rows apply the paper's Cantelli bound directly to the contrast under the Theorem S1 envelope, with no approximation. Their error allocation jointly covers, within each judge, the two contrasts the new runs measured: prompt and call between T = 0 and T = 1. Gemma's pooled interval was the pre-specified primary endpoint of E6, and its 32 newly collected repeats resolve the contrast on their own, with a distribution-free interval of [.0010, .0257]. Qwen's pooled interval was computed post hoc.

At the error level of Table 1's four contrasts, the exponential bound of the revised paper (Proposition S11, same assumptions) gives a distribution-free interval for this contrast that excludes zero for both judges on the pooled runs: [.0173, .0701] for Qwen and [.0046, .0219] for Gemma, whose 32 new repeats alone give [.0035, .0230] (post hoc). Here the exponential bound is narrower than Cantelli's once the error is split over two or more contrasts, and the pooled intervals still exclude zero at an error level one thousand times smaller. In simulations with known laws, the interval covered the true contrast in every dataset and never excluded zero when the two temperatures shared one set of laws. The calibrated intervals on the original calls, which use the same four contrasts, also exclude zero for both judges.

The increase also holds beyond the original census. Pre-specified extensions find it on 120 new items (Qwen +.0421, Gemma +.0106), on 80 RewardBench 2 items (+.0262, +.0109) and on a further 120 items (+.0305, +.0067), and under 24 unseen prompts (+.0387, +.0114). In the revised paper, the lower panel of Table 1 carries the calibrated, pooled and replication rows, Section 4 summarizes the extensions, and Sections M.2, M.3 and M.9 give the methods and the full tables (Tables S48, S49 and S58).

AuthorsOpenReview ↗

Response to Reviewer WrrJ (3/3): W3 and Q3, design selection on held-out data

W3 and Q3. Held-out value of the development-selected design

We report the held-out comparison against both fixed designs, separately for squared-error risk and decision accuracy. Gains from choosing between D and E are not significant at the studied budgets, consistent with the small cross-order covariance that determines their squared-error-risk contrast. In the pre-specified E9 study, however, choosing between one call and four-call voting (design D) yields an accuracy gain of 5.89 pp [4.48, 7.30] across the five pre-specified settings on 120 unseen items.

In ten settings (three judges on two benchmarks in the semantic study, and four judging policies), we froze the development choice between D and E and compared it with always using D and always using E on independent test deployments. For each measure, the table gives the range of selected-minus-fixed estimates across settings and the estimate with the largest absolute magnitude, together with its 95% interval; in every setting and for every measure, the interval includes zero.

Selected minus fixed design, 10 settingsAlways DAlways E
Squared-error risk around the common mean (lower is better)−.0025 to +.0014; largest −.0025 [−.0098, .0049]−.0015 to +.0044; largest +.0044 [−.0044, .0132]
Accuracy with abstention at ties, pp−1.25 to +.42; largest −1.25 [−3.68, 1.18]−1.07 to +1.77; largest +1.77 [−1.14, 4.68]
Accuracy with fair tie-breaking, pp−.73 to +.03; largest −.73 [−2.66, 1.20]−.71 to +.89; largest +.89 [−1.02, 2.80]

The squared-error-risk contrast has the exact form (Eq. 12):

RE−RD=K4(K−1)(Uint−Uprompt(q))R_E-R_D=\frac{K}{4(K-1)}\left(U_{\mathrm{int}}-U_{\mathrm{prompt}}^{(q)}\right)

The term in parentheses is minus the covariance between a prompt's effects in the two orders. Prompt effects carry over little between orders (median correlation .04 over 22 configurations), so the estimated ∣RE−RD∣|R_E − R_D| is at most .0026 across those configurations, the development estimates were at most .0034, and the estimated differences in Table S34 are at most .0022. On the test items, the design selected on the development set has the lower observed test risk in six of the ten settings.

With differences this small, predicting zero beats the raw estimate in six of the eight conditions of Table S39: an unbiased predictor has mean squared error Var⁡(Δ^)\operatorname{Var}(\widehat{\Delta}), while predicting zero has mean squared error Δ2\Delta^2, so zero has lower expected squared error when ∣Δ∣\lvert\Delta\rvert is below the estimator's standard error. The practical gains come from choosing between one call and voting, as the prospective E9 results below show.

The comparison between a single call and D illustrates why lower squared-error risk need not imply higher decision accuracy. For the same prompt–order sampling distribution, averaging D's four one-hot verdicts cannot increase expected squared-error loss around the common mean, by convexity. Yet with abstention at ties D is less accurate than a single call for both judges, by 3.38 to 11.49 pp on the 120 E9 items, while with fair tie-breaking D is more accurate for Qwen.

Forecasts from the measurement calls let us select the strategy for the target decision metric. In E9, strategies chosen on the original 40 items and their gold labels, and fixed before any E9 call, gained 5.89 pp [4.48, 7.30] in accuracy over the alternative in the five pre-specified settings. In post hoc comparisons over all eight settings, they had the higher observed accuracy in each and beat always using one call by .88 pp [.54, 1.22] and always voting by 3.36 pp [2.03, 4.69]. With only 8 calls per development item and temperature, the forecast's choice is on average at most .18 pp less accurate than the better strategy on unseen items, while the two fixed strategies are .88 to 3.36 pp less accurate (Table S61; post hoc).

Accuracy is the endpoint c = 1 of the loss P(wrong) + c·P(abstain). With abstention at ties, when the forecast chooses the strategy at each abstention cost c from 0 to 1, the mean loss is 3.02 pp [2.39, 3.65] lower than always using one call and 1.45 pp [.77, 2.14] lower than always voting on the E9 items. In the ten held-out settings above, with the strategy chosen from each setting's development calls, the differences are 3.69 pp [3.18, 4.20] and 1.20 pp [.88, 1.52] (both post hoc; Section M.9).

Section 6 of the revised paper bounds the design gain with Eq. 12 and adds E9 (Table 3), and Section M.7 reports squared-error risk and accuracy separately (Table S54, where abstention at ties is the "original rule" and fair tie-breaking is "fair 2:2").

The new results resolve the cooling contrast on the original fixed items and prompts, answer the held-out design question with intervals for both criteria, and show prospective gains from choosing the evaluation strategy. We would appreciate your reassessment in light of these additions.

Thank you for your response and clarifications. I will maintain my original score.

Reviewer_toTfOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the variability of LLM judges across prompts, candidate orders, and repeated calls. It proposes an unbiased variance decomposition with finite-sample guarantees and investigates how judge consistency affects downstream decision quality, voting outcomes, and deployment strategies.

Critical strengths
  1. The paper addresses a challenge in LLM-as-a-Judge evaluation: distinguishing systematic prompt sensitivity from stochastic generation variability. This distinction is valuable for understanding judge reliability.
  2. The proposed framework provides a principled decomposition of judge variability into prompt, order, interaction, and call components. The exact variance analysis and finite-sample detection bounds are technically interesting.
Critical weaknesses
  1. The paper lacks comparisons with several relevant baselines, such as [1] and [2]. Moreover, many other closely related studies on LLM-as-a-Judge consistency and reliability are not adequately discussed, making it difficult to assess the novelty and advantages of the proposed approach over existing methods.
  2. The variance decomposition and unbiased correction are closely related to classical variance-component estimation and U-statistics. The authors also acknowledge that the proposed estimator is algebraically equivalent to the classical trace-mean-square correction under the balanced design (Appendix A.4). While the finite-sample theoretical analysis is valuable, the paper should more clearly distinguish its novel contributions from established statistical techniques.
  3. One of the main conclusions is that cooling increases corrected prompt dependence while reducing call variation. However, the finite-sample confidence intervals for the prompt-dispersion contrasts in Table 1 include zero for both Qwen and Gemma. Although the item-composition intervals suggest consistent increases, they quantify a different source of uncertainty. More repeated calls or independent replications would strengthen the evidence supporting this conclusion.
  4. Although the paper establishes theoretical connections between response variability and deployment risk, it does not convincingly demonstrate that these measurements lead to better practical evaluation strategies. For example, the prediction experiments in Section K.2 show that the raw estimator outperforms the zero baseline in only two of eight conditions. Furthermore, the deployment-design comparisons in Table S32 do not establish the proposed equivalence criterion. Stronger comparisons against simple aggregation strategies, particularly in terms of downstream accuracy and evaluation cost, would help demonstrate the practical value of the framework.
  5. The theoretical analysis assumes fixed prompt sets and independent calls, which may not fully reflect practical LLM evaluation scenarios. Additionally, despite the large number of collected judgments, some key experiments involve relatively few distinct evaluation items (e.g., 40 items in the temperature experiments). The observed relationship between position dependence and tie frequency is also primarily supported by correlations and a stylized analytical model. Additional experiments on more diverse benchmarks, unseen prompts, and independently validated paraphrases would strengthen the conclusions.

[1] Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluators.

[2] TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them.

Questions for authors

See weakness

10 comments
AuthorsOpenReview ↗

Response to Reviewer toTf W1

Thank you for the detailed review. We have added direct comparisons with TrustJudge and PGED, resolved the fixed-census cooling contrasts with additional calls and an independent replication, and validated evaluation-strategy choices prospectively on unseen items. The cooling result also holds on 320 new items and 24 unseen prompts. In total we ran eight pre-specified studies with 752,640 new judge calls. For each study, we fixed the analysis plan and success criterion before data collection and ran the specified analysis unchanged. We will release the plans, code and other materials with the paper.

Experiments: E1–E7 and E9 collected new data; E8 compares aggregation methods using E7 calls.

W1. Comparison with [1] and [2], and related work

We compared both methods on the same stored calls (E8: 120 new items, 12 prompts, both candidate orders, T = 1, with top-20 token probabilities). TrustJudge's bidirectional likelihood-aware aggregation averages each judge's label probabilities over the two orders and takes the argmax; the frozen E8 analysis also included an equal-weight ensemble of the two judges' TrustJudge probabilities. Post hoc, we applied PGED's graph ensemble and denoising to this two-candidate setting, with the two judges as evaluators. For each item and prompt, each judge's four sampled verdicts give win counts (ties and invalid outputs ignored), the two judges' counts are summed into one graph, and cycle removal keeps the heavier of two opposite arcs. For equal-weight opposing arcs (5.4% of item–prompt pairs) we report both abstention and a fair coin, and the disagreement rate keeps these cases as TIE.

Single callTrustJudgePGEDProbability ensemble
Judges, calls per verdictone judge, 1one judge, 4both judges, 8both judges, 8
Accuracy67.5% (Qwen), 78.8% (Gemma)82.6%, 82.3%83.5% with ties as abstentions, 86.3% with ties split83.4%
Order conflict42%, 17%000
Prompt pairs on which the verdict differs32.5%, 6.7%2.6%, 4.3%11.0% [7.4, 14.5]3.4%

All three raise accuracy and remove order conflict, as intended. The verdict is the first generated token in 99.7% of Qwen's calls and 100% of Gemma's, so TrustJudge reads the probabilities at the decision itself. In PGED the merged graph had arcs in both directions for 61.8% of item–prompt pairs, so cycle removal decided most verdicts; with two candidates, longer cycles do not arise.

Our decomposition, which needs only verdict labels, finds the source of the remaining instability. We formed two independent aggregated verdicts per item and prompt from disjoint calls: two calls per verdict for TrustJudge and four for PGED. TrustJudge's systematic component is 2.37 pp [.75, 3.98] for Qwen and 4.18 pp [1.80, 6.56] for Gemma, out of a prompt-pair disagreement of 2.65% and 4.32%. PGED's disagreement of 18.3% consists of 17.2 pp of call noise and 1.1 pp [−.2, 2.5] of systematic dependence. Going from four to eight calls lowers PGED's disagreement to 11.0%, while going from two to four calls leaves TrustJudge's nearly unchanged, at 2.59% and 4.32%. The call-budget comparison demonstrates the practical value of the decomposition: PGED's noise-dominated disagreement decreases substantially with additional calls, while TrustJudge's predominantly systematic prompt disagreement remains nearly unchanged. This provides a concrete basis for allocating evaluation calls.

The same separation changes conclusions drawn from agreement rates. Over 240 items, cooling lowers the single-call disagreement between Qwen and Gemma from .332 to .249, while their systematic disagreement rises from .160 to .248 (+.088 [.071, .105]), so the apparent gain in agreement comes entirely from less call noise. For Qwen, on the original items, 120 new items and 80 RewardBench 2 items, plug-in prompt dispersion, a ProSA-style correctness gap, the raw paraphrase flip rate and the order-swap flip rate all show lower sensitivity or no resolved change under cooling, whereas corrected prompt dispersion increases in every panel.

We will add this comparison and decomposition to Section 5, with details in the appendix, and organize the related work by what each line of work measures. Position-bias studies (Wang et al., 2024; Li et al., 2024; Shi et al., 2025) measure order effects, which our order and interaction components estimate net of call noise. Prompt-sensitivity studies (Zhuo et al., 2024; Bellibatlu et al., 2026) study how verdicts change with the prompt. Agreement and reliability audits (Thakur et al., 2025; Norman et al., 2026) report human alignment, repeatability and bias metrics; our decomposition places call randomness, prompt dependence, order dependence and their interaction in one categorical framework. We will also cover bias benchmarks (Ye et al., 2025) and methods for improving judgment inference and aggregation ([1], [2]; Wang et al., 2025).

AuthorsOpenReview ↗

Response to Reviewer toTf W2–W3

W2. Relation to variance components and U-statistics

Our contribution is the finite-sample theory for the crossed design with categorical verdicts: how precise the estimate is, how small an effect a given budget can detect, and which comparisons need repeated calls at all. The point estimator itself is the classical trace-mean-square correction, a U-statistic over pairs of distinct calls, as Section 2 and Appendix A.4 state. The table pairs each result with its classical foundation, and we will add it to Section 2.

QuestionOur result for the categorical crossed designClassical foundation
Variance of the estimateExact finite-sample variance for fixed, heterogeneous categorical cells, including TIE and invalid outputs (Thm S1)Hoeffding's variance formula for U-statistics of i.i.d. samples
Smallest detectable effectA test valid for every fixed set of cell laws (Thm S2) and a matching lower bound (Thm S3), both of order 1/(RNK)1/(R\sqrt{NK}) (Thm 1)F tests for variance components under normality
Covariance constantThe sharp bound tr⁡(ΣaΣb)≤1/4\operatorname{tr}(\Sigma_a\Sigma_b)\le 1/4, which is attained; in the paper's certification experiment (N = 152, R = 8) it raises the share of datasets certifying θ < .01 from 79.1% to 97.1%The generic bound (1 − 1/C)/2
Which design comparisons need repeated callsAn exact condition for deployment designs, equal collision moments (Prop. S6–S7); comparing D with E needs one call per cellDegree of unbiased estimability
Link to decisionsProp. 2, the tie-rule identity, and unbiased forecasts for any vote rule on D's four distinct cells, checked on independent deployments (W4)None, because variance components do not determine voting accuracy (Prop. 2)

The finite-sample theory also guided the design of E6 before its calls were collected. At the effect size estimated from the first eight repeats per cell, the worst-case bound projected that about 32 to 40 repeats in total would resolve Gemma's smaller increase, as E6's frozen plan states. E6 therefore collected 32 additional repeats, giving 40 in all, and the resulting distribution-free interval excluded zero (W3).

JudgeSense v3 (Bellibatlu et al., September 2026) already uses a repeat-adjusted agreement contrast: it subtracts the agreement between two runs of the same template from the agreement across paraphrases. For two prompts kk and ℓ\ell in one order oo, with the repeat baseline averaged over both prompts, this contrast estimates −12∥Pko−Pℓo∥2-\tfrac{1}{2}\lVert P_{ko} - P_{\ell o}\rVert^2 without bias. Our framework separates prompt and prompt–order interaction effects across K prompts in a crossed design, and supplies exact variance, finite-sample detection guarantees and decision forecasts.

W3. Intervals in Table 1 that include zero

We collected the two things you suggested, more repeated calls and an independent replication, and they now resolve the T = 0 − T = 1 prompt contrast for both judges on the original fixed set of 40 items and 48 prompts.

T = 0 − T = 1 prompt contrast, original items and promptsQwenGemma
Table 1 (worst-case bound, R = 4, per-temperature bounds differenced)[−.0570, .1259][−.0799, .0987]
Worst-case bound on the contrast itself, all runs pooled+.0440 [.008, .080] at R = 8+.0133 [.0026, .0240] at R = 40
The 32 new repeats per cell alone (E6)[.0010, .0257]
Independent replication with fresh seeds (E1, R = 4), EV interval+.0454 [.0398, .0510]+.0119 [.0100, .0138]
Calibrated intervals on the original calls, EV and IP[.0363, .0489] and [.0212, .0641][.0120, .0164] and [.0012, .0272]

Gemma's 32 newly collected repeats resolve the contrast on their own, with a distribution-free interval of [.0010, .0257]. The pooled and E6-only rows apply the paper's Cantelli bound directly to the contrast under the Theorem S1 envelope, without approximation. Their error allocation covers the two contrasts the new runs collected, prompt and call between T = 0 and T = 1, jointly within each judge, and Gemma's pooled interval was the pre-specified primary endpoint of E6. Qwen's pooled interval was computed post hoc. Over Table 1's four contrasts, the pooled intervals widen to [−.0145, .1026] for Qwen and [−.0032, .0298] for Gemma. The calibrated intervals on the original calls, Bonferroni-adjusted over the same four contrasts, exclude zero for both judges. EV uses the exact variance of Theorem S1. IP is item-paired, and its variance estimate is conservative in expectation for the fixed-census target. Across ten known-law settings, their worst joint coverage was 99.8% and 95.4%. The replication met its pre-specified criterion (EV interval above zero) for both judges. Its estimates differ from the original estimates by +.0028 and −.0023; both intervals for these differences include zero. We will add these rows to Table 1 and the methods to Section 4.

AuthorsOpenReview ↗

Response to Reviewer toTf W4

W4. Practical value, Section K.2 and Table S32

Using the measurement calls to choose an evaluation strategy produces prospective gains. Strategies chosen on the original 40 items and their gold labels, and fixed before any E9 call, gained 5.89 pp [4.48, 7.30] in accuracy over the alternative strategy in the five pre-specified settings, on 120 unseen items. At the budgets we study, the practical decision is whether to use one call or a four-call majority vote (design D: two prompts in both orders), and the right answer depends on the judge, the temperature and the tie rule. Before any E9 call, we fixed the strategy in each of the eight settings and the primary endpoint, the average gain over the five settings in which an earlier test on 200 items had found a clear difference. We ran both judges on 120 previously unused items drawn from three benchmarks (53,760 calls per judge).

Comparison on the 120 E9 itemsAccuracy gain, pp (item-paired 95%)
Chosen strategy minus the other one, five pre-specified settings+5.89 [4.48, 7.30]
Chosen strategy minus always using one call, all eight settings (post hoc)+.88 [.54, 1.22]
Chosen strategy minus always using D, all eight settings (post hoc)+3.36 [2.03, 4.69]
Largest single gain (Qwen, T = 0, abstention at ties, one call instead of D)+11.49 [7.81, 15.17]

The chosen strategy had the higher observed accuracy in all eight settings. Each fixed rule falls short somewhere: always voting loses up to 11.49 pp, and always using one call loses 5.37 pp for Qwen at T = 1 with fair tie-breaking, where voting is better.

A natural alternative is to try both strategies on the development calls themselves. With the same 4k calls per item (k prompts, both orders, two repeats), our forecast averages D's accuracy over every way of grouping these calls into votes, so it equals the expected result of such a pilot given the calls and removes the noise of any one grouping at no extra cost. In a post hoc comparison on 200 new items, the mean regret (accuracy lost against the better strategy on the new items, in pp) is:

Calls per item and temperature81624324896192
Forecast.19.10.05.06.05.02.00
Pilot on the same calls.28.12.13.08.09.03.02

We also counted errors, abstentions and calls separately on E9 (post hoc). With fair tie-breaking at T = 1, Qwen with D makes 4.1 pp fewer wrong decisions [2.2, 6.0] and 1.3 pp fewer abstentions [.6, 2.0] than with a single call, at the price of three extra calls. With abstention at ties, D trades fewer errors for more abstentions; before accounting for the extra call cost, D is preferable when an abstention costs less than .57 to .79 of an error for Qwen, or .48 for Gemma. The measurement calls forecast all of these rates before deployment. Finally, we compared one call with four-call strategies (one prompt in one order, two calls per order, D and E) by resampling stored calls in eight settings. With abstention at 2:2 ties, a single call is the most accurate in all eight. With fair tie-breaking, D or E is the most accurate in all six Qwen settings and beats a single call by 3.9 to 4.8 points in five of them, while for Gemma all strategies are within 1.8 points of each other.

Section K.2 predicts the risk difference between designs D and E, whose estimated size is at most .0022 in absolute value in all eight conditions of Table S34. An unbiased predictor has mean squared error Var⁡(Δ^)\operatorname{Var}(\widehat{\Delta}), while predicting zero has mean squared error Δ2\Delta^2, so zero can outperform the raw predictor when the signal is small. Across the 170 known-law settings, zero is better in all 62 settings with a signal-to-noise ratio below one and in none of the other 108.

Table S32 tested calibration within a ±.005 margin. All six errors were small (at most .0052) and every interval contained zero, but the intervals were too wide to establish equivalence within the ±.005 margin.

The new deployments extend this validation to decision-level outcomes. Across 120 comparisons of decision metrics in eight panels, the gap between forecast and observation has a median of .4 pp and a maximum of 3.5 pp, the forecast–observation risk differences have 95% intervals containing zero in 23 of 24 comparisons, and all 16 pre-specified forecast checks in E9 include zero, with a largest gap of 1.5 pp. Across 22 configurations, the largest absolute estimated D/E risk difference is .0026, and none of the eight differences in Table S34 is distinguishable from zero. We report these contrasts separately from the prospective accuracy gains of choosing between one call and D. Together, these results show a practical use of the framework: deciding when four-call voting improves decisions and when a single call is more accurate at a quarter of the deployment calls. We will add E9 and these comparisons to Section 6.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reviews. The new evidence resolves the fixed-census cooling contrast, extends the findings to unseen items and prompts, and shows prospective gains from evaluation-strategy selection. We also clarify the theoretical advances beyond classical variance-component analysis and compare with TrustJudge and PGED.

Scale and protocol. We ran eight pre-specified studies with 752,640 new judge calls (E1–E7 and E9; E8 compares aggregation methods using E7 calls). Each plan and success criterion was fixed before data collection, and the specified analysis was run unchanged. We will release the plans, code and other materials.

Theory. The paper gives the exact finite-sample variance of the corrected estimator (Thm S1), matching upper and lower detection bounds at rate 1/(RNK)1/(R\sqrt{NK}) (Thm 1, S2, S3), a sharp covariance bound that raises certification of θ<.01\theta < .01 from 79.1% under the trace-refined bound to 97.1% in the paper's experiment, and the condition under which contrasts between deployment designs can be estimated with one call per cell (Prop. S6–S7). It also shows that lower squared-error risk does not guarantee a more accurate majority vote (Prop. 2). The correction is closed-form, uses only logged labels and needs no model fitting; computing it for 30,720 judgments averaged 2.6 ms on a laptop CPU.

The cooling result at the call level. On the original items and prompts, an independent replication with fresh seeds (E1) met its criterion for both judges: cooling raises corrected prompt dispersion by +.0454 [.0398, .0510] for Qwen and +.0119 [.0100, .0138] for Gemma (EV intervals). The paper's distribution-free bound on the contrast itself needs no approximation. With all runs pooled it gives +.0440 [.008, .080] for Qwen at R=8R = 8 (post hoc) and +.0133 [.0026, .0240] for Gemma at R=40R = 40, after E6 added 32 repeats per cell, a budget set in advance from the worst-case bound. Both intervals are simultaneous over the prompt and call contrasts within each judge.

Beyond the census. The increase holds on 320 new items from three benchmarks and on unseen prompts:

T=0T = 0 − T=1T = 1 prompt contrastQwenGemma
120 new LLMBar and JudgeBench items (E2)+.0421 [.0305, .0536]+.0106 [.0051, .0162]
80 new RewardBench 2 items (E3)+.0262 [.0149, .0375]+.0109 [.0033, .0185]
120 further new items, three benchmarks (E9)+.0305 [.0207, .0403]+.0067 [.0030, .0104]
24 prompts from two new families (E4, EV)+.0387 [.0306, .0467]+.0114 [.0087, .0140]

Item rows report item-paired 95% intervals, adjusted jointly for the prompt and call contrasts within each study and judge. For decisions, the tie rule matters: for Qwen, the estimated accuracy gain from cooling under fair tie-breaking is positive in every data collection, +2.88 pp [1.02, 4.73] pooled post hoc over 240 items, while under abstention at ties the change is not resolved.

Practical value. Before any E9 call, we fixed the choice between one call and a four-call majority vote (design D: two prompts in both orders) in each of eight settings, using the original 40 items and their gold labels. On 120 unseen items, the selected strategy gained 5.89 pp [4.48, 7.30] in accuracy over the alternative, averaged over the five pre-specified primary settings; post hoc over all eight settings, it beat always using one call by .88 pp [.54, 1.22] and always using D by 3.36 pp [2.03, 4.69]. The measurement calls also estimate error and abstention rates before deployment, so the choice can be made for given costs.

Comparison with existing methods. In our two-candidate setting (E8; PGED analysis post hoc), TrustJudge at four calls per verdict and PGED at eight calls both raise accuracy and remove order conflict, with residual prompt-pair disagreement of 2.6–4.3% and 11.0%, respectively. Our decomposition explains their different responses to additional calls. For PGED at four calls per verdict, the disagreement of 18.3% is almost all call noise (17.2 pp) and falls to 11.0% at eight calls. For TrustJudge at two calls per verdict, it is mostly systematic (2.37 pp [.75, 3.98] for Qwen, 4.18 pp [1.80, 6.56] for Gemma) and stays nearly unchanged at four calls. Common metrics also miss the cooling result: for Qwen, plug-in dispersion and raw flip rates show lower sensitivity or no resolved change in all three panels, while the corrected estimate increases in each.

Planned revisions. Sections 4 and 5 will add the call-level rows of Table 1, the new studies and the comparison with TrustJudge and PGED; Section 6 will use Eq. 12 to bound the design gain in squared-error risk before deployment, add E9, and report risk and decision loss separately. We will report "nearly fivefold" with its call-level interval, the decision results rule by rule alongside Table 2, and the Qwen–Gemma-3 ranking change as a configuration-specific reversal of the estimated ranking.

AuthorsOpenReview ↗

Revised paper uploaded

We have uploaded a revised PDF. New and rewritten text is in blue, the main text is ten pages, and the numbering of all earlier sections, tables, and equations is unchanged.

The main text now reports the call-level evidence for the cooling contrast in the lower panel of Table 1, together with the replication and the results on 320 unseen items and 24 unseen prompts (Section 4). The contributions state which part of the estimator is classical and what we add, with a point-by-point comparison in Table S47. Section 3 gives the Qwen–Gemma-3 comparison as a configuration-specific reversal of the estimated ranking, with its call-level intervals. Section 5 adds the replications of the tie-rule interaction and the comparison with TrustJudge and PGED. Section 6 uses Eq. 12 to bound the design gain in squared-error risk before deployment and reports the prospective strategy test in the new Table 3. Related work is organized by what each line of work measures. We also tightened the abstract and the openings of Sections 2 and 4 to 7 for readability, and redrew Figures 1 and 2 with the same content. No result changed.

Appendix M is new and reports the pre-specified replication and extension studies in full. The reference list adds five entries (Li et al., 2024; Thakur et al., 2025; TrustJudge; PGED; Halmos, 1946) and cites JudgeSense in its current version.

AuthorsOpenReview ↗

Revised paper updated: Section M.9 and a new pre-specified study (E10)

We have uploaded an updated revision (75 pages). The main text is still ten pages, new text is in blue, and no earlier section, table, equation, or proposition number has changed. The update adds Section M.9 at the end of the appendix (Propositions S11–S13 and Tables S58–S63) and states its main results in Table 1 and Sections 4 to 6. It includes a new pre-specified study, E10, so the paper now reports nine pre-specified studies with 775,680 new judge calls (the General Response, posted before E10, lists the earlier eight, with 752,640 calls). The reference list adds three entries (Maurer, 2006; Maurer & Pontil, 2009; Lehmann & Casella, 1998).

The new results are:

  • A distribution-free interval at the four-contrast error level of Table 1. A new exponential bound (Proposition S11), which holds for independent calls with arbitrary cell laws, gives an interval for the prompt contrast between T = 0 and T = 1 that excludes zero for both judges: [.0173, .0701] for Qwen and [.0046, .0219] for Gemma on the pooled runs (post hoc). It still excludes zero at an error level one thousand times smaller.
  • An unbiased variance estimate, and a comparison with classical intervals. An unbiased estimate of the exact variance (Proposition S13, for R ≥ 4) narrows the exact-variance intervals of Table 1 to [.0384, .0469] for Qwen and [.0124, .0160] for Gemma (post hoc). In simulations with known cell laws, its interval covers 94.3–94.9% at nominal 95%, while the classical per-category fixed-effects interval covers 85–87% (Table S60; post hoc).
  • Simpler diagnostics at matched budgets. An unadjusted flip rate falls under cooling, from .374 to .114 for Qwen, while prompt sensitivity rises. A repeat-adjusted flip rate is, up to a constant factor, the paper's estimator within one presentation order, and one order cannot measure the order component. Under the more favorable of two assumptions about the unseen order, a one-order forecast of the order-balanced vote misses the rate of pure 2:2 ties by 10.3 pp on average, against 1.4 to 2.8 pp for the crossed forecast (Table S59 and Section M.9; post hoc).
  • Choosing the evaluation strategy. With only 8 development calls per item and temperature, the forecast's choice between one call and voting is on average at most .18 pp less accurate than the better strategy on unseen items, while the two fixed strategies are .88 to 3.36 pp less accurate (Table S61). When the forecast chooses the strategy at each abstention cost from 0 to 1, with ties counted as abstentions, the mean decision loss is 3.02 pp [2.39, 3.65] lower than always using one call and 1.45 pp [.77, 2.14] lower than always voting on the E9 items. In ten held-out settings the differences are 3.69 pp [3.18, 4.20] and 1.20 pp [.88, 1.52] (all post hoc). Added on Nov 16: Table S61 also includes a forecast from the same number of calls in one presentation order. In the condition with the most pure ties (Qwen at T = 0, fair tie-breaking), it chooses the better strategy in 9% of draws at 8 calls, 3% at 32 and none from 96 calls on, against 67%, 98% and every draw for the crossed forecast. Averaged over all eight settings, from 32 calls on it stays .30 and .21 pp less accurate than the better strategy on the two test sets, while the crossed forecast reaches zero (post hoc).
  • A pre-specified test of the link between position preference and ties (E10). E10 met its primary criterion. A clause favoring the second-shown candidate, tested on 120 items never judged with such a clause, lowered Qwen's position effect (the share of calls for the first-shown candidate minus the share for the second-shown one) by .178 [.137, .220]. It also moved Qwen's ties toward that position: ties in which all four calls choose the first-shown candidate fell from 11.0% to 4.2% of votes, ties in which all four choose the second-shown one rose from 1.0% to 3.1%, and their difference, the tie imbalance, changed by −8.9 pp [−11.7, −6.1] (Table S63). The composition of the ties points the same way: at T = 0, all four calls choose the first-shown candidate in 905 of Qwen's 929 pure ties (Table S62; post hoc).

Our replies to Reviewers WrrJ and XTub now cite the results that bear on their questions, with the revised paper's sections and tables, and our latest replies to Reviewer toTf discuss all five results point by point.

In the camera-ready version, the main text will include a condensed summary of the pre-specified studies (now Table S46 in Section M), every analysis reported in our responses will appear in the paper, and we will further simplify the presentation to make the paper clearer and easier to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.