acceptodds
Withdrawn from ICLR 2027

Trust the Stream: Online Confidence Calibration for Selective LLM Judges

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are increasingly used as automated judges, motivating selective evaluation methods that certify human agreement by accepting only sufficiently confident predictions. However, with limited calibration data, noisy estimates of the confidence distribution can destabilize the certified threshold and degrade the coverage–agreement trade-off. We introduce Online Confidence Calibration (OCC), which continuously refines the empirical confidence distribution using confidence scores observed during deployment. OCC maps confidence scores to their empirical percentiles, re-scores the fixed calibration set, and re-certifies the acceptance threshold without requiring additional human labels. We prove that the resulting threshold converges to its population counterpart, achieving finite-time exact recovery with high probability and eventual almost-sure lock-in, while preserving the human-agreement guarantee at every round. Across six benchmarks, eleven LLM judges, and four confidence functions, OCC consistently improves threshold stability and calibration quality, certifies stronger agreement at matched coverage than label-free baselines, approaches the population-CDF oracle.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, selective evaluation, online calibration, risk control, distribution-free guarantees

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RbciOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies selective evaluation with LLM judges: a prediction is accepted only if the judge’s confidence exceeds a threshold calibrated on a small human-labeled set to guarantee a target human-agreement rate. In deployment, instances arrive online without human labels. Because annotation is costly, the calibration set is typically small; the authors observe that the empirical confidence distribution estimated from so few samples is noisy, making the certified threshold unstable across calibration draws. To address this, they propose Online Confidence Calibration (OCC), which maps confidences to percentiles under an empirical CDF that is continually refined with unlabeled confidences from the deployment stream, and re-certifies the threshold on the original calibration labels after each update. They show that under i.i.d. data the online threshold eventually matches the one obtained with the true confidence CDF, while the per-round agreement guarantee is retained. Across six benchmarks and eleven judges, OCC reduces threshold variability and certifies higher agreement at matched coverage than label-free baselines.

Critical strengths
  1. A practically motivated problem with a zero-annotation-cost solution. The paper addresses a real bottleneck in selective LLM evaluation: certification quality is limited by a small human-labeled calibration set. OCC exploits unlabeled confidence scores that deployment generates for free, requires no additional annotation, and adds negligible per-round computation compared to a judge forward pass.
  2. Broad and statistically rigorous experiments. The evaluation covers six benchmarks, eleven judges, and four confidence functions across 1,000 random splits. The authors report paired bootstrap intervals (Table 5) and nonempty-run counts with explicit denominators (Table 6). The direction of improvement is consistent across all 18 judge–benchmark combinations in Table 1.
Critical weaknesses
  1. Limited technical contribution. The method essentially recomputes the empirical CDF with more confidence scores obtained from the arriving online unlabeled data. Such an estimate improves with more samples follows directly from standard ECDF consistency, and the certification procedure is inherited unchanged from Jung et al. (2025). Furthermore, most of the gain over Static comes from certifying on CDF-normalized scores at all, even with a CDF estimated from the calibration set alone (72.0→78.0), while online updating adds only 1.8 points (Table 4). Yet Table 1 omits Cal-CDF, which conflates the two effects.

  2. Reducing CDF error does not guarantee a better threshold. The certified threshold has two sources of error: noise in the calibration labels and error in the estimated CDF. These interact: the CDF determines which candidate thresholds are tested, but whether each candidate passes depends on the noisy labels. However, OCC reduces only the CDF error, and the paper implicitly assumes this translates into a better threshold. This is never established: a more accurate placement of candidates can still land on a label-noise spike and stop the sweep earlier. Theorem 3.2 only shows convergence to the threshold under the true CDF, not that this threshold attains higher coverage (the true oracle threshol).

  3. The online setting is not well justified. OCC requires a labeled calibration set drawn from the deployment distribution (Assumption 3.1) previously, which means queries from that distribution must already be available before deployment. Since obtaining unlabeled samples is far cheaper than annotating them, we are able to build the calibration set could equally collect a larger unlabeled pool and estimate the CDF offline. Because OCC’s CDF depends only on the pool size (Eq. 4) and not on the arrival order (App. J), such an offline pool would match OCC’s end-of-stream performance from the first round. This is precisely what the Oracle does, even though the paper labels it as privileged (Table 4). Therefore, the online setting in this paper is not actually practical.

  4. Missing related work. Semi-supervised risk control via PPI (Einbinder et al., arXiv:2412.11174) and SemiCP (Zhou et al., arXiv:2505.21147) already use unlabeled data to stabilize label-limited calibration. Unlike OCC, they also reduce label-side variance. Both predate the deadline and should be discussed and, where applicable, compared.

Questions for authors
  1. Necessity of the online setting. Can the author describe a realistic scenario where a labeled calibration set from the deployment distribution is available, but unlabeled queries from the same distribution are not?
  2. Source of the gains. Could the authors run a ablation experiment in which the calibration percentiles are replaced by sorted i.i.d. Uniform(0,1) draws, with no stream data, and report its frontier alongside Cal-CDF and OCC? Moreover, for n=200 and Δ=0.005, the Cal-CDF percentiles coincide exactly with the grid points. Does OCC’s advantage over Cal-CDF persist under other grid spacings (e.g., Δ=0.0025 or 0.01), where this alignment no longer holds?
  3. Why the true-CDF threshold should be better. Is there any theoretical or empirical argument that the threshold certified under the true CDF attains higher coverage than the one certified under the calibration-only CDF, given the same calibration labels?
  4. Statistical uncertainty of the results. Most tables (e.g., Tables 1, 2, 3, 7, and 8) report only point estimates. Could the authors also provide the confidence intervals or standard deviations?
Reviewer_LtDMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of calibrating decisions of LLM judges, in an online setting that mimics real-world deployment. It extends on the framework of Jung et al. (2025) by re-evaluating an empirical CDF on the newly added unlabeled data, re-normalizing confidence scores, and updating the cutoff threshold. Theoretical results show that the threshold eventually equals the one obtained with the population CDF, and quantifies coverage loss due to the grid approach. Experimentally, the method is shown to be comparable, and demonstrate marginal loss, to the regime which assumes knowing the data in advance. Overall, experiments span over 6 benchmarks and 11 judges, and show gains over the frozen baselines, with the largest tracking improvements for small calibration sets and weaker judges.

Critical strengths

Overall, the considered setup of re-calibrating LLM decisions after deployment, given a stream of unlabeled data is very practical, and is one of the most frequently occurring in real situations. Due to the method's simplicity and practicality, it would be worth taking a look and implementing by a practitioner at the system release, to mitigate miscalibration on real data.

Critical weaknesses
  1. The methodological contribution is on a marginal side. The method combines established components: online empirical-CDF estimation from unlabeled scores, percentile-space thresholding, and re-certification of the threshold with the existing fixed-sequence procedure of Jung et al. (2025) on the same calibration labels, applying them to LLM judges.

  2. The provided artifact doesn’t include essentials to reproduce benchmark numbers. Supplemental materials contain the methods but no baselines, data, or experimental scripts. On what I was able to simulate and reproduce, the method achieved lower gains on top of Cal-CDF, than what’s reported in the paper. My setup might deviate significantly from what should be the actual experiments, so I would appreciate it if the authors update the artifacts to reproduce the results.

Questions for authors
  1. Could the authors update supplemental material with the artifacts to reproduce the base results from the paper?

  2. Could the authors add the Cal-CDF, as the main baseline, to the table 1?

Reviewer_UptWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper extends the selective-evaluation pipeline of Jung et al. (2025): instead of certifying an acceptance threshold on raw judge confidences, it maps confidences to percentiles under an empirical CDF, pools unlabeled deployment confidences into that CDF each round, re-scores the fixed labeled calibration set, and re-runs the fixed-sequence binomial test. Claims: the certified percentile threshold converges to (and exactly locks onto) its population-CDF counterpart in finite time (Thm 3.2, DKW + grid margin), the per-round agreement guarantee is unaffected, and across six benchmarks and eleven judges the method improves the coverage–agreement frontier.

Critical strengths

The problem is real and the writing is careful about what is and is not guaranteed (validity vs. efficiency, GSR vs. nominal risk, i.i.d. scope). The method is trivially cheap and label-free, and the validity argument in the Remark of §3.3 is correct since the scoring map uses confidences only. Experimental hygiene is good: paired bootstrap, disjoint reference set, honest acknowledgment that Cor. B.1 is vacuous, arrival-order and contamination ablations, and the EMP-margin invariance check.

Critical weaknesses
  1. The central mechanism is misdiagnosed, and the incremental gain is bounded by discretization. Fixed-sequence testing depends on scores only through the nested accepted calibration subsets (the authors’ own proof says so).

  2. The headline theory is vacuous at experimental scale. ρ is a minimum over n=200 percentiles against a 201-point grid; its typical value is ≈10−510^{-5}, giving t* around 101010^{10} rounds (median over 2,000 draws), versus T=1,250.

  3. Baseline presentation inflates the result. Table 1 (18 judge–benchmark cells) reports Static vs. W-ECDF vs. OCC and omits Cal-CDF, the paper’s own declared primary control.

Questions for authors

please see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.