- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of LLM-as-a-Judge when the number of candidate responses varies across inputs. The authors show that confidence scores obtained with different candidate counts can have substantially different distributions, making a shared confidence threshold unreliable. They propose a kernel-smoothed CDF transformation that normalizes confidence scores separately for each candidate count. The method is integrated into an existing selective prediction framework to control disagreement with human preferences. Experiments on six datasets and eleven judge models show improvements in confidence monotonicity, guarantee success rate, and coverage. The authors also evaluate the method in multi-model cascades.
Critical strengths
The paper addresses a practical problem in LLM-based evaluation. Varying numbers of candidate responses are common in real applications, and the authors identify a limitation of applying the same confidence threshold across these settings.
The experimental evaluation is relatively extensive, covering multiple datasets, model families, and cascade configurations. The reported improvements are consistent across these settings, and the proposed transformation adds little computational overhead.
Critical weaknesses
As described in Appendix E, the original datasets contain pairwise human preference annotations. To construct instances with more than two candidates, the authors retain the preferred response from the original pair and add randomly sampled alternatives, sometimes from different queries. The original preferred response is then treated as the ground-truth best response among all candidates.
However, a human preference for response A over response B does not establish that A is preferred over all newly introduced alternatives. The resulting multi-candidate labels are not directly supported by human annotations. Responses sampled from unrelated queries may also be easier to reject simply because they do not address the question.
This is particularly concerning because the paper's main objective is to guarantee agreement with human preferences in multi-candidate settings. The experiments instead measure agreement with labels constructed from pairwise preferences. Without validating these labels, the reported guarantee success rates are difficult to interpret.
Moreover, the proposed transformation applies a conditional CDF to confidence scores for each candidate count. Theorem 3.1 establishes that the resulting scores are approximately uniformly distributed, which follows from the standard probability integral transform and CDF estimation bounds. However, having comparable score distributions does not imply that the same normalized score corresponds to the same probability of a correct judgment across candidate counts. The paper does not establish this stronger relationship theoretically.
Theorem 3.2 largely follows the existing fixed-sequence testing framework of Jung et al. (2025). It is also unclear why restoring monotonicity should be necessary for the statistical validity of the risk guarantee, rather than primarily improving the selection efficiency of the confidence score. The paper does not clearly separate these two issues.
The experimental baselines are also limited. A straightforward alternative would be to calibrate separate acceptance thresholds for different candidate counts, without transforming the confidence scores. This would directly address the same problem. The current experiments mainly compare the proposed method with uncalibrated scores under a shared threshold, leaving its advantage over group-specific calibration unclear.
Questions for authors
1.How is the ground-truth preferred response validated for multi-candidate instances? The original pairwise annotations do not establish a preference ordering over the additional candidates. Have any of these instances been evaluated by human annotators?
2.What proportion of the additional candidates are sampled from unrelated queries? How does performance change when all candidates are valid responses to the same query?
3.Does the proposed method outperform separate risk-calibrated thresholds for each candidate count? This appears to be a natural baseline for the variable-candidate setting.
4.Does mapping the confidence distributions to Uniform(0,1) establish any relationship between normalized confidence and disagreement risk across different candidate counts? Theorem 3.1 appears to establish distributional comparability rather than risk comparability.
5 comments
Response to Reviewer 1B66 (2/3)
W (baselines) & Q3: Per-candidate-count thresholds.
We agree that this is the natural baseline. At the same calibration budget and pooled guarantee, u(x) has higher coverage, but per-m thresholds give a stronger per-m guarantee.
We compared u(x) with one threshold per m, using the same fixed-sequence procedure on c(x) and no CDF. Each of the four per-m tests runs at δ/4 = 0.025, so by a union bound the pooled risk stays below ε with probability at least 1 − δ, matching u(x).
Evidence. 18 dataset-judge pairs; m ∈ {2, 5, 10, 20}; ε = 0.20, δ = 0.10; 1,000 splits; GSR on the pooled accepted set; no certified threshold counts as 0 coverage.
| Calibration labels (per m) | Per-m thresholds, Cov. / GSR | u(x), Cov. / GSR |
|---|---|---|
| 6,000 (≈1,500) | 41.4% / 95.6% | 50.2% / 93.1% |
| 2,000 (≈500) | 33.2% / 95.9% | 44.7% / 92.8% |
| 800 (≈200) | 21.5% / 96.4% | 36.8% / 92.3% |
Both methods meet the 90% target at every budget. On average, u(x) covers 9–15 points more. Each per-m test uses a quarter of the labels at the stricter level δ/4, whereas the single test for u(x) uses all labels at level δ.
The two guarantees differ: per-m thresholds control risk within each m, while u(x) controls only the pooled risk. The gain is therefore in label efficiency when a pooled guarantee suffices, and per-m thresholds remain the right choice when a per-m guarantee is required.
In the revised PDF, we have added this baseline and stated this distinction in Appendix P (Table 15).
W (Theorem 3.1) & Q4: Distributional versus risk comparability.
We agree. Theorem 3.1 is a standard consequence of the probability integral transform and establishes distributional comparability, not risk comparability. The same u(x) can correspond to different probabilities of a correct judgment at different m. The guarantee does not require risk comparability, because Theorem 3.2 controls the pooled risk of the accepted instances.
Uniformity does establish one relationship, for the pooled risk. Let q be the accepted fraction, πₘ the share of candidate count m, and Rₘ(q) the risk among the top-q fraction of that count. If u(x) is uniform within each m, every m contributes the same fraction q, so the pooled risk is R(q) = Σₘ πₘ Rₘ(q), a fixed-weight average of the per-m risks. It is therefore monotone whenever each Rₘ is. Theorem 3.1 gives uniformity approximately, so this holds approximately. A shared raw threshold accepts different fractions of each m, so this argument does not apply to raw scores.
Example. Take m = 2 and m = 10 with equal shares, and suppose every raw score at m = 2 exceeds every raw score at m = 10 (a simplified Figure 1(b)). Within each m, risk rises with the accepted fraction:
| Accepted top fraction of each m | 25% | 50% | 75% | 100% |
|---|---|---|---|---|
| R₂ | 0.22 | 0.24 | 0.26 | 0.30 |
| R₁₀ | 0.04 | 0.14 | 0.30 | 0.46 |
| Pooled, u(x) | 0.13 | 0.19 | 0.28 | 0.38 |
At the same u(x), the two counts carry different risks (0.22 vs. 0.04 for the top 25%), so u(x) is not risk-comparable. Still, both counts contribute the same fraction, so the pooled risk is the average of the two rows and rises monotonically. A shared raw threshold instead accepts only m = 2 first, so the pooled risk rises from 0.22 to 0.30 (all of m = 2) and then falls to about 0.25 once the safest m = 10 instances enter.
The required condition, monotone risk within each m, is empirical and we do not prove it (Figures 1(a), 6(a), 7(a)).
Evidence. L_rank over all 66 dataset-judge pairs (Tables 1 and 8); lower is more monotone. u(x) applies the CDF to c(x).
| Score | max S | Vote | c(x) | c_KL | u(x) |
|---|---|---|---|---|---|
| Range | 0.23–0.42 | 0.21–0.41 | 0.10–0.28 | 0.10–0.23 | 0.01–0.05 |
| Mean | 0.33 | 0.30 | 0.19 | 0.16 | 0.03 |
We have stated this result with its condition as Proposition 3.3 (proof in Appendix C), noted after Theorem 3.1 that the comparability is distributional, and replaced the claims that CDF normalization restores monotonicity or risk comparability.
Response to Reviewer 1B66 (3/3)
W (Theorem 3.2): Validity versus efficiency.
We agree on both points. Theorem 3.2 is the guarantee of Jung et al. (2025) applied to u(x). Our contribution is the score u(x), which lets one shared threshold accept more instances under the same guarantee, and the pooled-risk result in Q4 that explains why.
Monotonicity matters for efficiency (power), not validity. We withdraw the claim in the Remark after Theorem 3.2 that monotonicity is an assumption of the theorem. Theorem 3.2 never uses monotonicity: validity needs i.i.d. data, a score and threshold order fixed before testing, a valid upper bound at each threshold, and stopping at the first non-rejection. Non-monotone pooled risk can stop the test sequence early and lower coverage, which is what CDF normalization improves (see Q4 in our previous comment).
Evidence. 18 dataset-judge pairs; ε = 0.20, δ = 0.10; 1,000 splits; no certified threshold counts as 0 coverage.
| Score / rule | Guarantee | Mean Cov. |
|---|---|---|
| c(x), Eq. (6) | valid | 12% |
| c(x), first threshold kept even if it fails (Table 2) | uncertified; GSR 31.9% | 38.7% |
| u(x), Eq. (6) | GSR 93.1% | 50.2% |
Table 2's c(x) rows use the second row's rule. Under Eq. (6), raw c(x) stays valid but certifies little, while u(x) keeps GSR above 90% and certifies far more. Likewise, for u(x), changing the base score, the number of simulated annotators, or the calibration labels per m (25 to 1,500) keeps GSR at 90.1–93.5% but moves coverage from 13.5% to 51.7% (18-pair averages; Table 14 in Appendix O).
In the revised PDF, we have separated validity (Theorem 3.2) from efficiency (Theorem 3.1 and Proposition 3.3, the pooled-risk result in Q4) in the Remark after Theorem 3.2, corrected the claims that non-monotone risk invalidates the guarantee and that CDF normalization restores validity, and reported the table above in Appendix O (Table 13).
References
- Jung et al. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. ICLR, 2025.
Response to Reviewer 1B66 (1/3)
Thank you for the careful review and for noting that varying numbers of candidate responses are "common in real applications."
W (labels) & Q1: Validating the multi-candidate labels.
We agree that preferring A over B does not establish that A beats the added candidates. For m > 2 the labels are therefore a proxy, and GSR on them measures agreement with that proxy. Within a query, the label is the candidate most often preferred by humans across all available pairwise comparisons, but not every pair was compared. We therefore checked two settings with stronger human evidence.
Evidence. Eq. (6); ε = 0.20, δ = 0.10; 1,000 splits; mean over MiMo-7B, Gemma-3-27B and GPT-OSS-120B; no certified threshold counts as 0 coverage.
On both, u(x) meets the 90% GSR target against these stronger labels, and the coverage gain holds. It is smaller on OASST1, as expected: CDF normalization only aligns scores across m, and m varies less there. At m = 10 and 20, almost no instance has a label directly compared with every other candidate, so the strict check is limited to m ≤ 5.
New human annotations. To answer the last part of Q1 directly, three annotators who saw neither the labels nor the judges' choices picked the best response in 150 constructed TL;DR instances (m = 5 and 10). The constructed label matched their majority choice in 73% of instances, more often than two annotators agreed with each other (61%). Among instances accepted by u(x) at the thresholds certified in our main runs, the judges' choices matched the majority in 83% on average, consistent with the 80% target.
In the revised PDF, we have stated this rule in Appendix E, described the constructed labels as a proxy (Section 4.1 and Appendix E), and added the strict TL;DR subset and OASST1 results (Appendix Q, Table 17).
W (other-query candidates) & Q2: Their share and same-query results.
Candidates come from the same query whenever possible, and from other queries only when that query's pool is too small. The share of other-query candidates among the candidates other than the label is:
To measure their effect, we removed every instance with an other-query candidate, refit the CDFs and re-ran Eq. (6).
Evidence. 18 dataset-judge pairs; ε = 0.20, δ = 0.10; 1,000 splits; no certified threshold counts as 0 coverage.
With same-query candidates only, GSR stays above the 90% target and u(x) coverage barely changes (48.6% vs. 50.2%), so the results do not rest on other-query candidates. The gap to c(x) narrows, as expected: the removed instances are mostly at large m, so m varies less in the remaining set, and CDF normalization only aligns scores across m. The strict TL;DR subset, OASST1 and the newly annotated instances in our response to Q1 also contain only same-query candidates.
We have reported these shares and this comparison in Appendix E.
References