acceptodds
Under review as a conference paper at ICLR 2027

When Random Number Choices Break Confidence: Restoring Reliability in LLM Judgment Guarantee

Abstract

Large language models (LLMs) are increasingly used as automated evaluators to assess output quality and preference alignment. However, providing reliable guarantees that LLM judgments agree with human preferences remains challenging. Recent confidence-thresholding approaches offer such guarantees for pairwise comparisons, where the judge LLM selects one response from two candidates, relying on the assumption that higher estimated confidence implies lower disagreement risk with human judgments. In practice, however, the number of candidate responses often varies across inputs, requiring the judge to select one response from a variable number of candidates. Because confidence scores are derived from normalized judge model output preference probabilities, their scales are generally not comparable across different candidate counts. This mismatch can violate the monotonicity assumption underlying confidence-thresholding guarantees and thereby invalidate them. To address this issue, we introduce a kernel-based Gaussian cumulative distribution function (CDF) transformation that calibrates confidence scores across heterogeneous candidate sizes. We theoretically show that the proposed transformation approximately maps raw confidence scores onto a common scale, restoring comparability across candidate settings. Experiments across multiple datasets and judge LLMs demonstrate that our method re-builds monotonicity, improves guarantee success rates and coverage compared with existing baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, selective evaluation, human-AI agreement guarantee, distribution-free guarantees

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1B66OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliability of LLM-as-a-Judge when the number of candidate responses varies across inputs. The authors show that confidence scores obtained with different candidate counts can have substantially different distributions, making a shared confidence threshold unreliable. They propose a kernel-smoothed CDF transformation that normalizes confidence scores separately for each candidate count. The method is integrated into an existing selective prediction framework to control disagreement with human preferences. Experiments on six datasets and eleven judge models show improvements in confidence monotonicity, guarantee success rate, and coverage. The authors also evaluate the method in multi-model cascades.

Critical strengths

The paper addresses a practical problem in LLM-based evaluation. Varying numbers of candidate responses are common in real applications, and the authors identify a limitation of applying the same confidence threshold across these settings.

The experimental evaluation is relatively extensive, covering multiple datasets, model families, and cascade configurations. The reported improvements are consistent across these settings, and the proposed transformation adds little computational overhead.

Critical weaknesses

As described in Appendix E, the original datasets contain pairwise human preference annotations. To construct instances with more than two candidates, the authors retain the preferred response from the original pair and add randomly sampled alternatives, sometimes from different queries. The original preferred response is then treated as the ground-truth best response among all candidates.

However, a human preference for response A over response B does not establish that A is preferred over all newly introduced alternatives. The resulting multi-candidate labels are not directly supported by human annotations. Responses sampled from unrelated queries may also be easier to reject simply because they do not address the question.

This is particularly concerning because the paper's main objective is to guarantee agreement with human preferences in multi-candidate settings. The experiments instead measure agreement with labels constructed from pairwise preferences. Without validating these labels, the reported guarantee success rates are difficult to interpret.

Moreover, the proposed transformation applies a conditional CDF to confidence scores for each candidate count. Theorem 3.1 establishes that the resulting scores are approximately uniformly distributed, which follows from the standard probability integral transform and CDF estimation bounds. However, having comparable score distributions does not imply that the same normalized score corresponds to the same probability of a correct judgment across candidate counts. The paper does not establish this stronger relationship theoretically.

Theorem 3.2 largely follows the existing fixed-sequence testing framework of Jung et al. (2025). It is also unclear why restoring monotonicity should be necessary for the statistical validity of the risk guarantee, rather than primarily improving the selection efficiency of the confidence score. The paper does not clearly separate these two issues.

The experimental baselines are also limited. A straightforward alternative would be to calibrate separate acceptance thresholds for different candidate counts, without transforming the confidence scores. This would directly address the same problem. The current experiments mainly compare the proposed method with uncalibrated scores under a shared threshold, leaving its advantage over group-specific calibration unclear.

Questions for authors

1.How is the ground-truth preferred response validated for multi-candidate instances? The original pairwise annotations do not establish a preference ordering over the additional candidates. Have any of these instances been evaluated by human annotators?

2.What proportion of the additional candidates are sampled from unrelated queries? How does performance change when all candidates are valid responses to the same query?

3.Does the proposed method outperform separate risk-calibrated thresholds for each candidate count? This appears to be a natural baseline for the variable-candidate setting.

4.Does mapping the confidence distributions to Uniform(0,1) establish any relationship between normalized confidence and disagreement risk across different candidate counts? Theorem 3.1 appears to establish distributional comparability rather than risk comparability.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1B66 (1/3)

Thank you for the careful review and for noting that varying numbers of candidate responses are "common in real applications."

W (labels) & Q1: Validating the multi-candidate labels.

We agree that preferring A over B does not establish that A beats the added candidates. For m > 2 the labels are therefore a proxy, and GSR on them measures agreement with that proxy. Within a query, the label is the candidate most often preferred by humans across all available pairwise comparisons, but not every pair was compared. We therefore checked two settings with stronger human evidence.

  • Strict TL;DR subset. At m = 5, we keep the 768 instances with all candidates from the same query and a label that beat each of the other four candidates in direct pairwise human comparisons. At m = 2, we use 768 randomly sampled original pairs.
  • OASST1 (Köpf et al., 2023). Each prompt has its own human-written replies, ranked by humans, so m varies naturally. We use English prompts with all replies ranked and m ∈ {2, 3, 4}, with about 500 prompts per m so that every m has enough calibration data. The top-ranked reply is the label.

Evidence. Eq. (6); ε = 0.20, δ = 0.10; 1,000 splits; mean over MiMo-7B, Gemma-3-27B and GPT-OSS-120B; no certified threshold counts as 0 coverage.

DatamCov., c(x)Cov., u(x)GSR, u(x)
TL;DR strict subset{2, 5}15%41.3%92.9%
OASST1{2, 3, 4}21%33%92.7%

On both, u(x) meets the 90% GSR target against these stronger labels, and the coverage gain holds. It is smaller on OASST1, as expected: CDF normalization only aligns scores across m, and m varies less there. At m = 10 and 20, almost no instance has a label directly compared with every other candidate, so the strict check is limited to m ≤ 5.

New human annotations. To answer the last part of Q1 directly, three annotators who saw neither the labels nor the judges' choices picked the best response in 150 constructed TL;DR instances (m = 5 and 10). The constructed label matched their majority choice in 73% of instances, more often than two annotators agreed with each other (61%). Among instances accepted by u(x) at the thresholds certified in our main runs, the judges' choices matched the majority in 83% on average, consistent with the 80% target.

In the revised PDF, we have stated this rule in Appendix E, described the constructed labels as a proxy (Section 4.1 and Appendix E), and added the strict TL;DR subset and OASST1 results (Appendix Q, Table 17).

W (other-query candidates) & Q2: Their share and same-query results.

Candidates come from the same query whenever possible, and from other queries only when that query's pool is too small. The share of other-query candidates among the candidates other than the label is:

Datasetm = 5m = 10m = 20
TL;DR0%0%19%
Chatbot Arena0%7%26%
HH-RLHF25%39%55%
AlpacaEval0%0%22%
HelpSteer335%41%58%
Comm. Align.0%11%27%

To measure their effect, we removed every instance with an other-query candidate, refit the CDFs and re-ran Eq. (6).

Evidence. 18 dataset-judge pairs; ε = 0.20, δ = 0.10; 1,000 splits; no certified threshold counts as 0 coverage.

InstancesCov., c(x)Cov., u(x)GSR, u(x)
All instances12%50.2%93.1%
Same-query candidates only21%48.6%92.4%

With same-query candidates only, GSR stays above the 90% target and u(x) coverage barely changes (48.6% vs. 50.2%), so the results do not rest on other-query candidates. The gap to c(x) narrows, as expected: the removed instances are mostly at large m, so m varies less in the remaining set, and CDF normalization only aligns scores across m. The strict TL;DR subset, OASST1 and the newly annotated instances in our response to Q1 also contain only same-query candidates.

We have reported these shares and this comparison in Appendix E.

References

  • Köpf et al. OpenAssistant Conversations – Democratizing Large Language Model Alignment. NeurIPS, 2023.
AuthorsOpenReview ↗

Response to Reviewer 1B66 (2/3)

W (baselines) & Q3: Per-candidate-count thresholds.

We agree that this is the natural baseline. At the same calibration budget and pooled guarantee, u(x) has higher coverage, but per-m thresholds give a stronger per-m guarantee.

We compared u(x) with one threshold per m, using the same fixed-sequence procedure on c(x) and no CDF. Each of the four per-m tests runs at δ/4 = 0.025, so by a union bound the pooled risk stays below ε with probability at least 1 − δ, matching u(x).

Evidence. 18 dataset-judge pairs; m ∈ {2, 5, 10, 20}; ε = 0.20, δ = 0.10; 1,000 splits; GSR on the pooled accepted set; no certified threshold counts as 0 coverage.

Calibration labels (per m)Per-m thresholds, Cov. / GSRu(x), Cov. / GSR
6,000 (≈1,500)41.4% / 95.6%50.2% / 93.1%
2,000 (≈500)33.2% / 95.9%44.7% / 92.8%
800 (≈200)21.5% / 96.4%36.8% / 92.3%

Both methods meet the 90% target at every budget. On average, u(x) covers 9–15 points more. Each per-m test uses a quarter of the labels at the stricter level δ/4, whereas the single test for u(x) uses all labels at level δ.

The two guarantees differ: per-m thresholds control risk within each m, while u(x) controls only the pooled risk. The gain is therefore in label efficiency when a pooled guarantee suffices, and per-m thresholds remain the right choice when a per-m guarantee is required.

In the revised PDF, we have added this baseline and stated this distinction in Appendix P (Table 15).

W (Theorem 3.1) & Q4: Distributional versus risk comparability.

We agree. Theorem 3.1 is a standard consequence of the probability integral transform and establishes distributional comparability, not risk comparability. The same u(x) can correspond to different probabilities of a correct judgment at different m. The guarantee does not require risk comparability, because Theorem 3.2 controls the pooled risk of the accepted instances.

Uniformity does establish one relationship, for the pooled risk. Let q be the accepted fraction, πₘ the share of candidate count m, and Rₘ(q) the risk among the top-q fraction of that count. If u(x) is uniform within each m, every m contributes the same fraction q, so the pooled risk is R(q) = Σₘ πₘ Rₘ(q), a fixed-weight average of the per-m risks. It is therefore monotone whenever each Rₘ is. Theorem 3.1 gives uniformity approximately, so this holds approximately. A shared raw threshold accepts different fractions of each m, so this argument does not apply to raw scores.

Example. Take m = 2 and m = 10 with equal shares, and suppose every raw score at m = 2 exceeds every raw score at m = 10 (a simplified Figure 1(b)). Within each m, risk rises with the accepted fraction:

Accepted top fraction of each m25%50%75%100%
R₂0.220.240.260.30
R₁₀0.040.140.300.46
Pooled, u(x)0.130.190.280.38

At the same u(x), the two counts carry different risks (0.22 vs. 0.04 for the top 25%), so u(x) is not risk-comparable. Still, both counts contribute the same fraction, so the pooled risk is the average of the two rows and rises monotonically. A shared raw threshold instead accepts only m = 2 first, so the pooled risk rises from 0.22 to 0.30 (all of m = 2) and then falls to about 0.25 once the safest m = 10 instances enter.

The required condition, monotone risk within each m, is empirical and we do not prove it (Figures 1(a), 6(a), 7(a)).

Evidence. L_rank over all 66 dataset-judge pairs (Tables 1 and 8); lower is more monotone. u(x) applies the CDF to c(x).

Scoremax SVotec(x)c_KLu(x)
Range0.23–0.420.21–0.410.10–0.280.10–0.230.01–0.05
Mean0.330.300.190.160.03

We have stated this result with its condition as Proposition 3.3 (proof in Appendix C), noted after Theorem 3.1 that the comparability is distributional, and replaced the claims that CDF normalization restores monotonicity or risk comparability.

AuthorsOpenReview ↗

Response to Reviewer 1B66 (3/3)

W (Theorem 3.2): Validity versus efficiency.

We agree on both points. Theorem 3.2 is the guarantee of Jung et al. (2025) applied to u(x). Our contribution is the score u(x), which lets one shared threshold accept more instances under the same guarantee, and the pooled-risk result in Q4 that explains why.

Monotonicity matters for efficiency (power), not validity. We withdraw the claim in the Remark after Theorem 3.2 that monotonicity is an assumption of the theorem. Theorem 3.2 never uses monotonicity: validity needs i.i.d. data, a score and threshold order fixed before testing, a valid upper bound at each threshold, and stopping at the first non-rejection. Non-monotone pooled risk can stop the test sequence early and lower coverage, which is what CDF normalization improves (see Q4 in our previous comment).

Evidence. 18 dataset-judge pairs; ε = 0.20, δ = 0.10; 1,000 splits; no certified threshold counts as 0 coverage.

Score / ruleGuaranteeMean Cov.
c(x), Eq. (6)valid12%
c(x), first threshold kept even if it fails (Table 2)uncertified; GSR 31.9%38.7%
u(x), Eq. (6)GSR 93.1%50.2%

Table 2's c(x) rows use the second row's rule. Under Eq. (6), raw c(x) stays valid but certifies little, while u(x) keeps GSR above 90% and certifies far more. Likewise, for u(x), changing the base score, the number of simulated annotators, or the calibration labels per m (25 to 1,500) keeps GSR at 90.1–93.5% but moves coverage from 13.5% to 51.7% (18-pair averages; Table 14 in Appendix O).

In the revised PDF, we have separated validity (Theorem 3.2) from efficiency (Theorem 3.1 and Proposition 3.3, the pooled-risk result in Q4) in the Remark after Theorem 3.2, corrected the claims that non-monotone risk invalidates the guarantee and that CDF normalization restores validity, and reported the table above in Appendix O (Table 13).

References

  • Jung et al. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. ICLR, 2025.
Reviewer_dBKoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks at how to decide when to trust an LLM judge when the number of candidate answers changes. The main idea is to turn confidence scores into percentiles within each candidate-count group, making them more comparable. The authors then use these scores to accept confident judgments and abstain on uncertain ones. They provide theoretical results on score normalization and disagreement-risk control. Across six datasets and eleven judge models, they report better confidence–agreement relationships and higher coverage while meeting the target agreement level more consistently. These experiments use synthetically constructed candidate sets.

Critical strengths
  • Simple method with a clear statistical basis. The CDF transformation is straightforward to apply to an existing judge without retraining. The approximate uniformity result gives a clear justification for how it aligns confidence distributions across candidate counts.

  • Extensive eval across several datasets and judge models. The reported improvements appear across six datasets and eleven models, covering both confidence agreement monotonicity and practical measures such as coverage. This suggests the gains are not limited to one model or benchmark, although the synthetic candidate construction limits generalizability.

Critical weaknesses

They proposed a method for calibrating LLM-judge confidence scores across varying numbers of candidate responses to support human–LLM agreement guarantees. However, I’m not entirely sure how important this problem is in practical settings. The first and most important point that comes to mind is that the judge is supposed to choose the best response, which it can already do when the number of candidates varies. While the paper addresses the reliability of accepting those judgments rather than the selection itself, I remain unconvinced of the practical significance of this additional calibration. Despite the soundness of the proposed methodology, I therefore see the paper’s practical scope as somewhat limited.

Another weakness is the highly synthetic experimental setting, in which additional candidates beyond the original pair are sampled from a response pool that may include answers to other queries, while the original preferred response is retained as the ground truth label. Although the reported results are consistent with the claimed guarantee that the disagreement risk exceeds ε with probability at most δ over calibration draws, it remains unclear whether these findings generalize to practical settings with naturally occurring candidate sets and human preferences assessed over the full set.

Questions for authors
  1. Re the 2nd weakness mentioned above: how reliable are the expanded candidate set labels? The experiments retain the original human-preferred response as the ground-truth winner after adding sampled alternatives. Can you evaluate on naturally occurring multi-candidate sets with human/proxy annotations?

  2. How much does the CDF transformation add over other simpler alternatives? I'd recommend comparing against separate risk-calibrated thresholds for each candidate count and/or an unsmoothed empirical CDF? These comparisons would help isolate the benefits of distribution normalization and kernel smoothing.

4 comments
AuthorsOpenReview ↗

Response to Reviewer dBKo (1/2)

Thank you for the constructive comments. We appreciate your view that the method is simple and has "a clear statistical basis," and that the evaluation is extensive. Our responses follow.

W1: Practical significance.

We agree that the judge can pick a response at any m. However, its picks often disagree with the benchmark labels (32–65% agreement when all are accepted, against an 80% target; Table 2), so one must decide which picks a human reviews. In the settings below, m can vary across inputs.

  • Post-training labels. LLM judges now label preference data in place of humans, as GPT-4o does in Tülu 3 (Lambert et al., 2024). A judge taking over existing human ranking would face varying m: InstructGPT labelers ranked 4 to 9 responses per prompt (Ouyang et al., 2022), and among English OASST1 prompts with two or more replies, all ranked, 67.3% have three or more (our count; Köpf et al., 2023).
  • Clinical settings. Physician review of an AI summary of real patient records costs an estimated USD 50, versus USD 0.05 for one LLM-judge evaluation (Croxford et al., 2025). Diagnosis lists also vary in length: on 302 challenging cases, unassisted clinicians listed a median of 6 diagnoses (IQR 5–9; McDuff et al., 2025), so a judge picking among the listed diagnoses would face varying m.
  • Coding agents. Trae Agent's LLM selector chooses among the deduplicated patches that pass regression tests, so its candidate count varies by issue (Trae Research Team et al., 2025).

What the calibration adds. In each setting, a team can set one agreement target for the accepted picks as a whole (pooled over m) and send the rest to annotators, physicians or developers. Raw confidence c(x) drops as m grows (Figures 1(b) and 3(a)). On our benchmarks (18 dataset-judge pairs; ε = 0.20, δ = 0.10), one raw threshold under Eq. (6) covers only 12% of picks on average (splits with no certified threshold count as 0). Table 2's c(x) rows instead keep the first threshold even if it fails the test, covering 38.7% but meeting the target in only 31.9% of splits. Under Eq. (6), u(x) covers 50.2% and meets the target in 93.1% of splits (GSR), so at the same guarantee, human review falls from 88% to about 50% of picks. On OASST1, u(x) covers 33% versus 21% for c(x) (see W2 & Q1). Per-m thresholds give a per-m guarantee and are the right choice when one is required, but split the calibration labels (see Q2 in our next comment).

In the revised PDF, we have added these scenarios to the introduction; Appendix O (Table 13) reports the 18-pair numbers above.

W2 & Q1: Label reliability and naturally occurring candidate sets.

We agree: for m > 2, the labels are a proxy, not a human preference over the full set. Within a query, the label is the candidate most often preferred by humans across all available pairwise comparisons, not the winner of a single pair, but not every pair was compared. The other candidates come from the same query whenever possible, and from other queries only when that pool is too small. We therefore checked two settings with stronger labels.

  • Strict TL;DR subset (label reliability). At m = 5, we keep the 768 instances with all candidates from the same query and a label that beat each of the other four candidates in direct pairwise human comparisons. At m = 2, we use 768 randomly sampled original pairs.
  • OASST1 (naturally occurring sets; Köpf et al., 2023). Each prompt has its own human-written replies, ranked by humans, so m varies naturally. We use English prompts with all replies ranked and m ∈ {2, 3, 4}, with about 500 prompts per m so that every m has enough calibration data. The top-ranked reply is the label.

Evidence. Eq. (6); ε = 0.20, δ = 0.10; 1,000 splits; mean over MiMo-7B, Gemma-3-27B and GPT-OSS-120B; no certified threshold counts as 0 coverage.

DatamCov., c(x)Cov., u(x)GSR, u(x)
TL;DR strict subset{2, 5}15%41.3%92.9%
OASST1{2, 3, 4}21%33%92.7%

The gain holds on OASST1, although it is smaller than on the constructed data, as expected: CDF normalization only aligns scores across m, and m varies less there. At m = 10 and 20, almost no instance has a label directly compared with every other candidate, so the strict check is limited to m ≤ 5.

We have stated this rule in Appendix E, described the constructed labels as a proxy (Section 4.1 and Appendix E), and added these results (Appendix Q, Table 17).

AuthorsOpenReview ↗

Response to Reviewer dBKo (2/2)

Q2: Per-m thresholds and an unsmoothed empirical CDF.

We agree that these two baselines separate the two parts of the transformation: per-m thresholds isolate distribution normalization, and the unsmoothed empirical CDF (ECDF) isolates kernel smoothing. In short, normalization adds 8–14 coverage points at the same pooled guarantee, and smoothing adds another 0.6–1.3.

  • Per-m thresholds. One threshold per m, using the same fixed-sequence procedure on c(x) and no CDF. Each of the four per-m tests runs at δ/4 = 0.025, so by a union bound the pooled risk stays below ε with probability at least 1 − δ, matching u(x).
  • ECDF. u(x) with the per-m empirical CDF in place of the kernel CDF, i.e., the hₘ → 0 limit of Eq. (3); everything else is unchanged.

Evidence. 18 dataset-judge pairs; m ∈ {2, 5, 10, 20}; ε = 0.20, δ = 0.10; 1,000 splits, the same for all three methods; GSR on the pooled accepted set; no certified threshold counts as 0 coverage.

Calibration labels (per m)Per-m thresholds, Cov. / GSRECDF, Cov. / GSRu(x), Cov. / GSR
6,000 (≈1,500)41.4% / 95.6%49.6% / 92.7%50.2% / 93.1%
2,000 (≈500)33.2% / 95.9%43.8% / 92.4%44.7% / 92.8%
800 (≈200)21.5% / 96.4%35.5% / 91.8%36.8% / 92.3%

All three methods meet the 90% target at every budget. On average, u(x) covers 9–15 points more than per-m thresholds. Each per-m test uses a quarter of the labels at the stricter level δ/4, whereas the single test for u(x) uses all labels at level δ. The two guarantees differ: per-m thresholds control risk within each m, while u(x) controls only the pooled risk, so per-m thresholds remain the right choice when a per-m guarantee is required.

The ECDF covers 0.6–1.3 points less than u(x), with the largest gap at the smallest budget, as expected: smoothing reduces discretization, which matters most when nₘ is small. Over 90% of u(x)'s gain over per-m thresholds therefore comes from per-m normalization, the percentile mapping in your summary.

We have added both baselines to Appendix P (Table 15) and reported there how much of the gain comes from normalization and how much from smoothing.

References (cited in our previous comment)

  • Croxford et al. Evaluating Clinical AI Summaries with Large Language Models as Judges. npj Digital Medicine, 2025.
  • Köpf et al. OpenAssistant Conversations – Democratizing Large Language Model Alignment. NeurIPS, 2023.
  • Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124, 2024.
  • McDuff et al. Towards Accurate Differential Diagnosis with Large Language Models. Nature, 2025.
  • Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS, 2022.
  • Trae Research Team et al. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling. arXiv:2507.23370, 2025.

Thanks for the helpful response and for the new experiments!! Your clarifications cleared up my concerns, and I’ll update my rating accordingly.

Thank you very much for your positive feedback and for raising your rating to weak acceptance. We are glad that our clarifications and the new experiments have addressed your concerns, and we sincerely appreciate your constructive suggestions, which have helped us strengthen the motivation and empirical evaluation of our work.

Thank you again for your time, careful evaluation, and valuable suggestions.

Reviewer_7J2hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies confidence-based selection when an LLM judge faces different numbers of candidate responses. It normalizes confidence scores using a candidate-count-specific CDF and applies fixed-sequence testing to decide which judgments to accept. Experiments report improved coverage and agreement across several datasets and judge models.

Critical strengths

The paper identifies a practical issue: confidence scores from different candidate-set sizes may not be directly comparable. The proposed correction is lightweight, and the reported coverage gains suggest that it could be useful in selective evaluation.

Critical weaknesses

The main guarantee is not adequately justified. The paper argues that non-monotonic risk invalidates fixed-sequence testing and that CDF normalization restores its validity. However, fixed-sequence testing controls error with valid individual tests, a pre-specified order, and stopping at the first non-rejection; risk monotonicity mainly affects power and coverage. This distinction is explicit in Learn then Test [1]. Also, making confidence distributions approximately uniform does not by itself establish a monotonic relationship with correctness. The current argument therefore does not establish the claimed restoration of guarantees. [1] Angelopoulos et al. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. arXiv:2110.01052.

The experiments do not yet establish the value of the correction in realistic multi-candidate judging. Appendix E keeps the original pairwise winner as the best response after adding alternatives, including responses from other queries. This does not establish a valid human preference label for the full candidate set. The comparison also misses closely related approaches: Localize-Then-Decide addresses candidate-count-induced confidence distortion through shortlisting [2], while Margin-Adaptive Confidence Ranking learns a confidence ordering [3]. These methods differ from grouped CDF normalization, but their relationship and performance should be examined.

[2] Li et al. Localize-Then-Decide Guarantees for LLM Judgments. arXiv:2608.25824, 2026. [3] Jin et al. Margin-Adaptive Confidence Ranking for Reliable LLM Judgement. arXiv:2605.15416, 2026.

Questions for authors
  1. Can you distinguish improvements in testing power from validity of the risk guarantee, and clarify the assumptions needed for each?
  2. Do the gains hold on same-query candidate sets with human labels covering the full set?
  3. How does the method compare with per-candidate-count threshold calibration and an adapted Localize-Then-Decide baseline under the same calibration budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 7J2h (1/3)

We thank the reviewer for the careful review and for recognizing that the paper identifies "a practical issue" with a lightweight correction. We respond point by point.

W1 (first part) & Q1: Validity versus power.

The reviewer is right, and we modify the claim that non-monotone risk invalidates fixed-sequence testing. Theorem 3.2 never uses monotonicity, so the guarantee holds for any score fixed before testing. What varying m reduces is usable coverage (power), and that is what CDF normalization improves.

Assumptions (Q1).

  • Validity needs i.i.d. data, a score and threshold order fixed before testing, a valid upper bound at each threshold, and stopping at the first non-rejection. It needs no monotone risk or comparable scores.
  • Power (coverage) depends on how well the score orders instances within and across m (cf. Jung et al., 2025, Sec. 2.2), and non-monotone risk can stop the test sequence early. Per-m CDF normalization yields an approximately monotone path when risk is monotone within each m (see W1, second part).

Evidence. MiMo-7B, TL;DR; ε = 0.20, δ = 0.10; 1,000 splits.

Score / ruleGuaranteeMedian Cov.
c(x), Eq. (6)valid, but nothing certified0%
c(x), first threshold kept even if it fails (Table 2)uncertified; GSR 18.4%27%
u(x), Eq. (6)passes in 96.8% of splits; GSR 93.1%36%
  • All 18 pairs under Eq. (6). Mean coverage is 12% for raw c(x), versus 50.2% for u(x) (splits with no certified threshold count as 0).
  • Score quality and labels. For u(x), changing the base score, the number of simulated annotators, or the labels per m (25 to 1,500) keeps GSR at 90.1–93.5% but moves coverage from 13.5% to 51.7% (18-pair averages; Table 14 in Appendix O). These factors change power, not validity.

In the revised PDF, we have corrected these overstated claims (invalidation and restored validity), listed the validity conditions in the Remark after Theorem 3.2, and cited Learn then Test in Section 3.2. Section 4.1 now states Table 2's threshold rule, and Appendix O (Table 13) reports the results above.

W1 (second part): Uniformity versus monotonicity.

The reviewer is right: uniformity alone does not imply a monotone relationship with correctness. CDF normalization also cannot create monotonicity within an m, because it preserves the ordering of scores there.

What it changes is the mix of m among accepted points. Let q be the accepted fraction, πₘ the share of candidate count m, and Rₘ(q) the risk among the top-q fraction of that count. If u(x) is uniform within each m, every m contributes the same fraction q, so the pooled risk is R(q) = Σₘ πₘ Rₘ(q). The pooled risk is therefore monotone whenever each Rₘ is. Theorem 3.1 gives uniformity approximately, so this holds approximately. A shared raw threshold accepts different fractions of each m, so this argument does not apply to raw scores.

Example. Take m = 2 and m = 10 with equal shares, and suppose every raw score at m = 2 exceeds every raw score at m = 10 (a simplified Figure 1(b)). Within each m, risk rises with the accepted fraction:

Accepted top fraction of each m25%50%75%100%
R₂0.220.240.260.30
R₁₀0.040.140.300.46
Pooled, u(x)0.130.190.280.38

A shared raw threshold first accepts only m = 2, so the pooled risk rises from 0.22 to 0.30 at 50% coverage (all of m = 2) and then falls to about 0.25 once the safest m = 10 instances enter. This path is non-monotone although each Rₘ is monotone. With u(x), both counts contribute the same fraction, so the pooled risk is the average of the two rows and rises monotonically.

The required condition, monotone risk within each m, is empirical and we do not prove it (Figures 1(a), 6(a), 7(a)).

Evidence. L_rank over all 66 dataset-judge pairs (Tables 1 and 8); lower is more monotone. u(x) applies the CDF to c(x).

Scoremax SVotec(x)c_KLu(x)
Range0.23–0.420.21–0.410.10–0.280.10–0.230.01–0.05
Mean0.330.300.190.160.03

We have stated this result with its condition as Proposition 3.3 (proof in Appendix C) and replaced the claims that CDF normalization restores monotonicity.

References

  • Angelopoulos et al. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. The Annals of Applied Statistics, 2025.
  • Jung et al. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. ICLR, 2025.
AuthorsOpenReview ↗

Response to Reviewer 7J2h (2/3)

W2 (first part) & Q2: Same-query candidate sets with human labels.

We agree: for m > 2, the labels in Appendix E come from pairwise human comparisons and some candidates can come from other queries, so they are a proxy, not a human preference over the full candidate set.

To answer Q2 directly, we refit the CDFs and re-ran Eq. (6) on TL;DR instances that meet both conditions in Q2. At m = 5, we keep the 768 instances with all candidates from the same query and a label that beat each of the other four candidates in direct pairwise human comparisons. At m = 2, we use 768 randomly sampled original pairs, which humans compared directly.

Evidence. TL;DR, m ∈ {2, 5}; ε = 0.20, δ = 0.10; 1,000 splits; splits with no certified threshold count as 0 coverage.

JudgeMean Cov., c(x)Mean Cov., u(x)GSR, u(x)
MiMo-7B2%26%92.3%
Gemma-3-27B11%42%93.1%
GPT-OSS-120B32%56%93.3%

The coverage gain holds on this subset: u(x) covers 24–31 points more than c(x) for all three judges, with GSR above the 90% target. At m = 10 and 20, almost no instance has a label directly compared with every other candidate, so this check is limited to m ≤ 5.

Naturally occurring sets. We also ran the same test on OASST1 (Köpf et al., 2023). Each prompt has its own human-written replies, all ranked by humans, and the top-ranked reply is the label. With English prompts, m ∈ {2, 3, 4} and about 500 prompts per m, u(x) covers 33% versus 21% for c(x), with GSR 92.7% (means over the three judges). The gain is smaller than on the TL;DR subset, as expected: CDF normalization only aligns scores across m, and m varies less here.

In the revised PDF, we have described the labels for m > 2 as a proxy (Section 4.1 and Appendix E) and added both experiments (Appendix Q, Table 17).

W2 (second part) & Q3 (LTD): Localize-Then-Decide and MACR.

We agree that both methods are closely related, and we have cited and discussed them in Appendix B.

  • Localize-Then-Decide (LTD; Li et al., 2026) first shortlists candidates with a conformal rank quantile, then runs fixed-sequence testing on a margin within the shortlist. It addresses candidate-count distortion at one fixed m at a time by reducing the candidates that compete. CDF normalization instead aligns scores across different m in one calibration set and keeps all candidates.
  • Margin-Adaptive Confidence Ranking (MACR; Jin et al., 2026) learns a confidence ordering for pairwise judgments from about 3,000 additional labeled examples. It changes the ordering of instances, whereas CDF normalization keeps the ordering within each m and aligns scores across m. Because MACR is defined only for m = 2 and needs labels beyond the calibration set, it does not fit a same-budget comparison on mixed m.

Adapting LTD to mixed m. The adapted LTD keeps its shortlist step, with one shortlist size per m, and its default KL margin. As with u(x), one threshold for all m is then set by a single fixed-sequence test on the calibration data pooled over all m. Both thresholds are therefore certified on the pooled accepted set, not on each m, and the comparison contrasts the two corrections for candidate count: shortlisting versus CDF normalization.

Evidence. LTD's shortlist-miss and selection-error levels are both ≈ 0.12. Both candidates are kept at m = 2, so misses can occur only in the three quarters of instances with m ≥ 5, and its overall target agreement is (1 − 0.12)(1 − 0.75 × 0.12) ≈ 0.80, the same as u(x) with ε = 0.20. Both methods use the same stored annotator probabilities, splits and calibration labels. 18 dataset-judge pairs; m ∈ {2, 5, 10, 20}; δ = 0.10; 1,000 splits; splits with no certified threshold count as 0 coverage.

Calibration labels (per m)Adapted LTD, Cov. / GSRu(x), Cov. / GSR
6,000 (≈1,500)33.7% / 94.8%50.2% / 93.1%
2,000 (≈500)28.4% / 93.5%44.7% / 92.8%
800 (≈200)23.0% / 93.3%36.8% / 92.3%

On average, u(x) covers 14–17 points more than adapted LTD at every budget. Both stay above the 90% target, with LTD slightly more conservative.

We have added this comparison to Appendix P (Table 16).

References

  • Jin et al. Margin-Adaptive Confidence Ranking for Reliable LLM Judgement. arXiv:2605.15416, 2026.
  • Köpf et al. OpenAssistant Conversations – Democratizing Large Language Model Alignment. NeurIPS, 2023.
  • Li et al. Localize-Then-Decide Guarantees for LLM Judgments. arXiv:2608.25824, 2026.
AuthorsOpenReview ↗

Response to Reviewer 7J2h (3/3)

Q3 (per-candidate-count thresholds): Comparison under the same calibration budget.

We compared u(x) with one threshold per m, using the same fixed-sequence procedure on c(x) and no CDF. Each of the four per-m tests runs at δ/4 = 0.025, so by a union bound the pooled risk stays below ε with probability at least 1 − δ, matching u(x).

Evidence. 18 dataset-judge pairs; m ∈ {2, 5, 10, 20}; ε = 0.20, δ = 0.10; 1,000 splits; GSR on the pooled accepted set; no certified threshold counts as 0 coverage.

Calibration labels (per m)Per-m thresholds, Cov. / GSRu(x), Cov. / GSR
6,000 (≈1,500)41.4% / 95.6%50.2% / 93.1%
2,000 (≈500)33.2% / 95.9%44.7% / 92.8%
800 (≈200)21.5% / 96.4%36.8% / 92.3%

Both methods meet the 90% target at every budget. On average, u(x) covers 9–15 points more. Each per-m test uses a quarter of the labels at the stricter level δ/4, whereas the single test for u(x) uses all labels at level δ.

The two guarantees differ: per-m thresholds control risk within each m, while u(x) controls only the pooled risk. The gain is therefore in label efficiency when a pooled guarantee suffices, and per-m thresholds remain the right choice when a per-m guarantee is required.

We have added this comparison and stated this distinction in Appendix P (Table 15).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive reviews. They found that the paper identifies "a practical issue" and proposes a lightweight correction (7J2h), that varying numbers of candidate responses are "common in real applications" (1B66), and that the method has "a clear statistical basis" (dBKo) and is evaluated across many datasets and judge models (dBKo, 1B66). We have uploaded a revised PDF with changes in blue; references below are to it, and existing numbering is unchanged.

Main concerns

  • Validity versus efficiency (7J2h, 1B66). We agree that fixed-sequence testing stays valid without monotone risk: Theorem 3.2 holds for any score fixed before testing. CDF normalization improves efficiency: under Eq. (6), u(x) covers 50.2% of judgments on average versus 12% for raw c(x), with GSR 93.1% (18 dataset-judge pairs). The normalized scores are comparable in distribution, not in risk, but the pooled risk becomes approximately a fixed-weight average of the per-m risks, hence approximately monotone whenever they are (Proposition 3.3).
  • Labels for m > 2 (all reviewers). We agree that the constructed labels are a proxy; we added two settings with stronger human labels (results are means over three LLM judges). On a strict TL;DR subset (m ∈ {2, 5}), whose labels beat every other candidate in direct human comparisons, u(x) covers 41.3% versus 15% for c(x), with GSR 92.9%. On OASST1 (m ∈ {2, 3, 4}), whose candidate sets occur naturally and are ranked by humans, it covers 33% versus 21%, with GSR 92.7%. Removing every instance with an other-query candidate barely changes u(x) (coverage 48.6% vs. 50.2%; GSR 92.4%).
  • Baselines (all reviewers). At the same calibration budget, all methods meet the 90% GSR target. On average, u(x) covers 9–15 points more than per-m thresholds (which, unlike u(x), give a per-m guarantee) and 14–17 points more than an adapted Localize-Then-Decide. An unsmoothed empirical CDF covers 0.6–1.3 points less than u(x), so most of u(x)'s gain over per-m thresholds comes from per-m normalization.
  • Practical significance (dBKo). The judge can pick at any m, but its picks often disagree with the benchmark labels, so one must decide which picks humans review. This arises with varying m when LLM judges label preference data in place of human rankers or pick among differential diagnoses, and when coding agents select among test-passing patches. On our benchmarks, at the same guarantee, human review falls from 88% of picks with one raw c(x) threshold to about 50% with u(x).

Changes in the revised PDF

  • Main text. The abstract, introduction and conclusion now say that non-monotone risk lowers efficiency, not validity, and that u(x) yields approximately monotone pooled curves; the introduction adds the scenarios above. Section 3 revises the Remarks after Theorems 3.1 (distributional comparability) and 3.2 (validity conditions), adds Proposition 3.3 and cites Learn then Test. Section 4.1 describes the labels for m > 2 as a proxy and states the threshold rule used in Table 2. A new Section 4.5 summarizes the Eq. (6), human-label, same-query, per-m-threshold and Localize-Then-Decide results, with a small unnumbered table.
  • New appendices. O: results under Eq. (6) (Tables 13–14). P: per-m thresholds, the unsmoothed CDF and adapted Localize-Then-Decide (Tables 15–16). Q: the strict TL;DR subset and OASST1 (Table 17).
  • Extended appendices. B discusses Localize-Then-Decide and Margin-Adaptive Confidence Ranking, C proves Proposition 3.3, and E states the label rule and reports the other-query shares and same-query results.

We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.