- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the sources of variation in "LLM-as-a-judge" evaluations—specifically whether they stem from prompts, the order of candidate answers, the interaction between the two, or the inherent randomness of repeated calls. Focusing on the distribution of categorical responses for each input condition, the authors establish a cross-variance decomposition and utilize consistency across calls to construct an unbiased estimator, thereby correcting the overestimation of prompt sensitivity caused by a limited number of repeated calls. The theoretical analysis derives the exact variance of the estimator and establishes matching upper and lower bounds for detecting weak prompt effects under fixed, balanced designs. Furthermore, the paper examines the relationships among response distributions, alignment with ground-truth answers, voting rules, and the cost of abstention; it also derives the risk difference and conditions for unbiased estimation regarding deployment designs with fixed budgets. The experiments cover six judge models and three benchmarks, incorporating controls for identical inputs, high-repetition independent baselines, and simulations based on known distributions, alongside interventions involving temperature, prompting strategies, and reasoning modes. The results demonstrate that bias correction can alter the ranking of models based on prompt sensitivity; lowering the temperature can simultaneously increase estimates of both repeatability and prompt sensitivity; and applying different tie-breaking rules to identical voting outcomes can change the assessment of intervention effects.
Critical strengths
- This article rigorously handles limited call bias and supports statistical verification: directly using empirical distribution to measure cue dispersion can easily confuse call noise and cue effects. This bias is affected by the number of repetitions, models and decoding conditions, and even changes the model ranking. The author clarifies that this estimation method is equivalent to trace mean square correction on a sample-by-sample basis. The core is to derive the upper and lower bounds of precise variance and matching detection under a fixed cross design, and verify the practicability of the correction method through input control, independent reference evaluation and simulation.
- The article also combines independent voting experiments to analyze that stability is not equal to decision quality: not only does it generally point out that small variance does not necessarily mean accuracy, Proposition 2 constructs a scenario to illustrate that when vector risk and Brier loss decrease, if a tie is recorded as an abstention, the accuracy will decrease instead, reflecting the role of position dependence and voting boundaries. Experiments show that voting rules will significantly change the results. Qwen's accuracy fluctuates by up to 9.84 percentage points under the cool down setting. The inclusion of abstention costs also improves the interpretability of conclusions.
Critical weaknesses
- The statistical certainty regarding conclusions about temperature parameters is insufficient. For Qwen, the point estimate for the prompting effect is +0.04262; however, the confidence interval at the invocation level includes zero, as does the confidence interval for Gemma. At the question-set level, the positive intervals are derived solely from existing invocation data. In Table 2, the confidence intervals for both results concerning accuracy changes include zero. Current evidence merely indicates that evaluation outcomes are influenced by voting rules; it does not definitively establish whether lower temperatures increase or decrease accuracy—the reported "nearly fivefold increase" is merely a point estimate.
- The scope of the experiments is relatively limited. The temperature experiments utilized only 40 questions, and the reasoning experiments involved just 32. Although the total number of model invocations is substantial, there has been no large-scale, independent verification at the task level.
- There is a lack of robust empirical support for the incremental value of these findings in informing deployment decisions. While the identity relating Decision/Evaluation risk differentials is highly interpretable, the evidence for using it to select superior deployment strategies is weak. This is reflected in the confidence intervals across several supplementary tables, equivalence tests, predictive comparisons, and the sign reversals observed across different datasets in JudgeBench. While these results do not invalidate the identity itself, they highlight a gap between achieving unbiased measurement and selecting reliable deployment strategies under a constrained budget. Furthermore, the risk metrics derived from this formula cannot serve as a substitute for accuracy or objective functions that account for the costs of abstention; therefore, conclusions regarding deployment should be approached with greater caution.
Questions for authors
- Regarding inferences on temperature effects: Is there support—such as from independent replications or more rigorous validated inference methods—for the observed direction of the temperature effect? If not, can a clear distinction be made between temperature-related conclusions based solely on point estimates and those backed by guarantees at the query level (accounting for finite samples)? Uncertainty at the query level should not be conflated with intervals derived from the set of prompts itself.
- Regarding the practical value of deployment choices: Once the selection rule is fixed during development, what actual gains are realized during testing compared to baselines such as always selecting D, always selecting E, or a zero-risk-difference prediction? Please report the common-mean risk and final decision loss separately, and explain whether the measurement of query costs is justified.
- Regarding cross-model rankings: Regarding the rank reversal between Qwen and Gemma-3, can you provide the direct difference between the models and the corresponding query-level intervals? Given that the experiment compares full configurations—including decoding, stopping, and parsing settings—please clarify whether this rank reversal supports a conclusion specific to the configuration or a generalizable model property.
- Regarding the interpretation of the prompt set: What specific standard variations were excluded during the automated equivalence check of the 48 semantic prompts? Subsets consisting of the same sentences in different orders serve as valuable controls; could you clarify whether the main conclusions and their associated uncertainties hold when using only this stricter subset?
6 comments
Response to Reviewer XTub (2/4): W2 and Q4, scope and the prompt set
W2. New items, a third benchmark, unseen prompts and the reasoning study
The cooling increase now holds on 320 items never used before, from three benchmarks, and under 24 unseen prompts. E2 and E3 met their pre-specified criterion for both judges, and in E9 the same increase was a pre-specified secondary expectation that held for both (T = 0 − T = 1 prompt contrast, item-paired 95%):
| New items | Qwen | Gemma |
|---|---|---|
| E2: 120 items (60 LLMBar, 60 JudgeBench) | +.0421 [.0305, .0536] | +.0106 [.0051, .0162] |
| E3: 80 RewardBench 2 items (20 each from Factuality, Focus, Math and Precise IF) | +.0262 [.0149, .0375] | +.0109 [.0033, .0185] |
| E9: 120 items (24 LLMBar, 24 JudgeBench, 72 RewardBench 2) | +.0305 [.0207, .0403] | +.0067 [.0030, .0104] |
The point estimate is positive for both judges in both E2 datasets and all four RewardBench 2 domains. Pooled post hoc over the census and the first 200 new items, the increase is +.0369 [.0303, .0435] for Qwen and +.0113 [.0077, .0149] for Gemma. Cooling raised the corrected prompt dispersion on 177 of these items and lowered it on 55 for Qwen, and on 64 versus 12 for Gemma (exact sign tests, p < 10⁻⁸; post hoc). Call variation falls in every study. Under 24 new prompts from two families absent from the bank (layout and voice), the increase is +.0387 [.0306, .0467] for Qwen and +.0114 [.0087, .0140] for Gemma (EV). We added items before prompts because items carry most of the uncertainty (for Qwen, an item standard error of .0084 against .0027 for prompts).
The reasoning result was replicated on the same 80 new RewardBench 2 items used in E3 (E5, pre-specified; 7,680 judgments). Thinking raises call variation from .0234 to .0766, +.0531 [.0222, .0840], meeting the pre-specified criterion.
Q4. The automated equivalence check and the clause-order subset
No prompt was excluded. All 48 candidates were accepted, each with 4 of 4 equivalence judgments from two reviewer models in two input orientations, while both identical-text positive controls passed and all four meaning-changing negative controls were flagged (216 checks). The check required the same task, criteria, scope and weights, exclusions, tie rule, label meanings and output contract, no added procedure or examples, and entailment in both directions. An off-the-shelf NLI model (RoBERTa-large-MNLI), independent of how the prompts were built and of both judges, also finds bidirectional entailment for all 48 prompts (lowest probability .974) and for the 24 unseen prompts.
The clause-order subset uses exactly the same five sentences, including the output-format line, in 12 orders, so the wording is held fixed. The rise in prompt dispersion and the fall in call variation hold there for both judges, with intervals that exclude zero:
| Clause-order subset (K = 12), T = 0 − T = 1 | Qwen | Gemma |
|---|---|---|
| Corrected prompt dispersion, EV | +.0383 [.0262, .0505] | +.0127 [.0092, .0163] |
| Corrected prompt dispersion, IP | [.0150, .0617] | [.0014, .0241] |
| Call variation, IP | −.3530 [−.4156, −.2903] | −.0398 [−.0737, −.0058] |
Across each family alone (K = 12) and each set of three families (K = 36), the prompt increase ranges from +.0346 to +.0436 for Qwen and from +.0127 to +.0148 for Gemma. EV excludes zero in all 16 analyses, and IP in all eight for Qwen and seven of eight for Gemma, the exception being the lexical family alone, [−.0011, .0274].
In the revised paper, Section 4 summarizes these analyses, Section 6 reports the reasoning replication, and Section M.3 gives the full results (Tables S49 and S50).
Response to Reviewer XTub (3/4): W3 and Q2, deployment value
W3 and Q2. Held-out gains, risk and decision loss, and query cost
We report the held-out comparison against always D, always E and a zero-difference prediction, with common-mean risk and decision accuracy kept apart. Gains from choosing between D and E are not significant at the studied budgets, consistent with the small cross-order covariance that determines their risk contrast. Choosing between one call and voting produced a prospective accuracy gain: in the pre-specified E9 study, choices fixed in development gained 5.89 pp [4.48, 7.30] in accuracy over the alternative in the five pre-specified settings, on 120 unseen items.
In ten settings (three judges on two benchmarks, and four judging policies), we froze the development choice between D and E and compared it with always using D and always using E on independent test deployments. For each measure, the table gives the range of selected-minus-fixed estimates across settings and the estimate with the largest absolute magnitude, together with its 95% interval; in every setting and for every measure, the interval includes zero.
| Selected minus fixed design, 10 settings | Always D | Always E |
|---|---|---|
| Squared-error risk around the common mean (lower is better) | −.0025 to +.0014; largest −.0025 [−.0098, .0049] | −.0015 to +.0044; largest +.0044 [−.0044, .0132] |
| Accuracy with abstention at ties, pp | −1.25 to +.42; largest −1.25 [−3.68, 1.18] | −1.07 to +1.77; largest +1.77 [−1.14, 4.68] |
| Accuracy with fair tie-breaking, pp | −.73 to +.03; largest −.73 [−2.66, 1.20] | −.71 to +.89; largest +.89 [−1.02, 2.80] |
Across the ten settings and any abstention cost c between 0 and 1, the largest absolute estimated difference in decision loss, P(wrong) + c·P(abstain), is 1.25 pp against always D and 1.77 pp against always E.
The risk contrast has the exact form (Eq. 12):
The term in parentheses is minus the covariance between a prompt's effects in the two orders. Thus, E has lower risk when these effects carry over between orders, and D when they reverse. That covariance is small and its sign varies (median correlation .04 over 22 configurations, where the largest estimated is .0026), which is why the sign of the observed difference can change between datasets.
A zero-difference prediction has lower expected squared error than an unbiased estimate when is below the estimator's standard error, which is the regime of the predictive comparisons in Table S39. Across 170 known-law settings, zero is better in all 62 settings with a signal-to-noise ratio below one and in none of the other 108.
The new deployments extend the calibration check of Table S32 to decision outcomes: across 120 comparisons in eight panels, the median forecast–observation gap is .4 pp and the largest 3.5 pp. Table S32's own intervals were too wide to establish equivalence within its ±.005 margin, but every interval contained zero.
Risk around the common mean and decision loss can disagree, and the choice between one call and voting shows it. For the same prompt–order sampling distribution, averaging D's four verdicts cannot increase expected squared-error loss, by convexity, yet with abstention at ties D is less accurate than a single call for both judges on the E9 items. Counting errors, abstentions and calls separately on E9 (post hoc):
| D minus single call (pp, item-paired 95%) | Qwen T = 1 | Qwen T = 0 | Gemma T = 1 | Gemma T = 0 |
|---|---|---|---|---|
| Abstention at ties: wrong | −12.8 [−15.1, −10.6] | −15.3 [−18.8, −11.7] | −5.0 [−7.4, −2.6] | −5.2 [−7.6, −2.7] |
| Abstention at ties: abstain | +16.2 [12.6, 19.8] | +26.8 [19.9, 33.6] | +10.4 [5.6, 15.2] | +10.7 [5.8, 15.5] |
| Fair tie-breaking: wrong | −4.1 [−6.0, −2.2] | −2.1 [−3.2, −1.0] | −.2 [−1.1, .8] | −.2 [−1.1, .7] |
| Fair tie-breaking: abstain | −1.3 [−2.0, −.6] | +.4 [−.4, 1.2] | +.7 [−.3, 1.7] | +.8 [−.3, 1.8] |
With the cost of a wrong decision set to one, an abstention cost c and a cost λ per call, D beats a single call when Before the three extra calls are charged, the estimated decision loss favors D for Qwen under fair tie-breaking; under abstention at ties, the estimated break-even abstention cost is .57 to .79 of an error for Qwen and about .48 for Gemma. The measurement calls estimate all of these rates before deployment, which turns the comparison into an explicit selection rule for given costs. The next comment shows that, with abstention at ties, this rule gives lower decision loss on unseen items than either fixed strategy, on average over abstention costs from 0 to 1.
Response to Reviewer XTub (1/4): W1 and Q1, temperature inference
Thank you for the careful reading and the specific questions. We have resolved the cooling contrast on the original fixed items and prompts with additional calls and an independent replication, extended the cooling study to 320 new items from three benchmarks and 24 unseen prompts, replicated the reasoning result on 80 new RewardBench 2 items, separated the decision effects by tie rule, and reported held-out deployment comparisons with risk and decision loss kept apart. In total we ran nine pre-specified studies with 775,680 new judge calls. For each study, we fixed the analysis plan and success criterion before data collection and ran the specified analysis unchanged.
Experiments: E1–E7, E9 and E10 collected new data; E8 compares aggregation methods using E7 calls.
W1 and Q1. Which temperature conclusions carry call-level guarantees
Independent replication and calibrated call-level inference both support the direction of the effect. Every row below targets the original 40 items and 48 prompts (the census) under fresh calls, the call-level question you pose; intervals over items or over the prompt set answer different questions, and we keep them separate.
The second and third rows apply the paper's Cantelli bound directly to the contrast under the Theorem S1 envelope, with no approximation; their error allocation covers the prompt and call contrasts between T = 0 and T = 1 jointly within each judge. Gemma's pooled interval was the pre-specified primary endpoint of E6, and Qwen's was computed post hoc. At the error level of Table 1's four contrasts, the exponential bound of the revised paper (Proposition S11, same assumptions) gives a distribution-free interval that excludes zero for both judges on the pooled runs (post hoc), and this still holds at an error level one thousand times smaller. The calibrated intervals on the original calls, Bonferroni-adjusted over the same four contrasts, also exclude zero for both judges, and across ten known-law settings their worst joint coverage was 99.8% and 95.4%. The replication met its pre-specified criterion (EV interval above zero) for both judges. The exact-variance (EV) interval uses the exact finite-sample variance of Theorem S1 with a normal reference. The item-paired (IP) variance estimate is conservative in expectation for the same fixed items and prompts.
Call variation falls under cooling in every study, as Table 1 established with finite-sample intervals. The revised paper reports the "nearly fivefold" rise with its call-level interval, 4.8 (3.3 to 8.5); the replication gives 5.5, and both runs pooled give 5.2 (4.2 to 6.8).
Accuracy under cooling depends on the tie rule, and the decision effects replicate.
For the original census, the interaction and wrong-decision intervals are the simultaneous ones of Table 2, whose Qwen accuracy intervals, [−.87, 16.81] and [−14.43, 10.68], include zero. All other intervals are pointwise item-paired 95% intervals. The accuracy rows of the first two columns and the pooled E2–E3 column were computed after the frozen analyses, while the E9 column reports pre-specified secondary analyses. Under fair tie-breaking, the estimated accuracy gain is positive in every column, and pooled post hoc over 240 distinct items it is +2.88 pp [1.02, 4.73]; under abstention at ties the change is not significant. The tie-rule interaction is positive, with an interval excluding zero, in all four columns. For Gemma, the estimates in every row and column are within 1 pp of zero. The revised paper reports the decision results rule by rule (Section 5 and Table S51).