- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies selective evaluation with LLM judges: a prediction is accepted only if the judge’s confidence exceeds a threshold calibrated on a small human-labeled set to guarantee a target human-agreement rate. In deployment, instances arrive online without human labels. Because annotation is costly, the calibration set is typically small; the authors observe that the empirical confidence distribution estimated from so few samples is noisy, making the certified threshold unstable across calibration draws. To address this, they propose Online Confidence Calibration (OCC), which maps confidences to percentiles under an empirical CDF that is continually refined with unlabeled confidences from the deployment stream, and re-certifies the threshold on the original calibration labels after each update. They show that under i.i.d. data the online threshold eventually matches the one obtained with the true confidence CDF, while the per-round agreement guarantee is retained. Across six benchmarks and eleven judges, OCC reduces threshold variability and certifies higher agreement at matched coverage than label-free baselines.
Critical strengths
- A practically motivated problem with a zero-annotation-cost solution. The paper addresses a real bottleneck in selective LLM evaluation: certification quality is limited by a small human-labeled calibration set. OCC exploits unlabeled confidence scores that deployment generates for free, requires no additional annotation, and adds negligible per-round computation compared to a judge forward pass.
- Broad and statistically rigorous experiments. The evaluation covers six benchmarks, eleven judges, and four confidence functions across 1,000 random splits. The authors report paired bootstrap intervals (Table 5) and nonempty-run counts with explicit denominators (Table 6). The direction of improvement is consistent across all 18 judge–benchmark combinations in Table 1.
Critical weaknesses
-
Limited technical contribution. The method essentially recomputes the empirical CDF with more confidence scores obtained from the arriving online unlabeled data. Such an estimate improves with more samples follows directly from standard ECDF consistency, and the certification procedure is inherited unchanged from Jung et al. (2025). Furthermore, most of the gain over Static comes from certifying on CDF-normalized scores at all, even with a CDF estimated from the calibration set alone (72.0→78.0), while online updating adds only 1.8 points (Table 4). Yet Table 1 omits Cal-CDF, which conflates the two effects.
-
Reducing CDF error does not guarantee a better threshold. The certified threshold has two sources of error: noise in the calibration labels and error in the estimated CDF. These interact: the CDF determines which candidate thresholds are tested, but whether each candidate passes depends on the noisy labels. However, OCC reduces only the CDF error, and the paper implicitly assumes this translates into a better threshold. This is never established: a more accurate placement of candidates can still land on a label-noise spike and stop the sweep earlier. Theorem 3.2 only shows convergence to the threshold under the true CDF, not that this threshold attains higher coverage (the true oracle threshol).
-
The online setting is not well justified. OCC requires a labeled calibration set drawn from the deployment distribution (Assumption 3.1) previously, which means queries from that distribution must already be available before deployment. Since obtaining unlabeled samples is far cheaper than annotating them, we are able to build the calibration set could equally collect a larger unlabeled pool and estimate the CDF offline. Because OCC’s CDF depends only on the pool size (Eq. 4) and not on the arrival order (App. J), such an offline pool would match OCC’s end-of-stream performance from the first round. This is precisely what the Oracle does, even though the paper labels it as privileged (Table 4). Therefore, the online setting in this paper is not actually practical.
-
Missing related work. Semi-supervised risk control via PPI (Einbinder et al., arXiv:2412.11174) and SemiCP (Zhou et al., arXiv:2505.21147) already use unlabeled data to stabilize label-limited calibration. Unlike OCC, they also reduce label-side variance. Both predate the deadline and should be discussed and, where applicable, compared.
Questions for authors
- Necessity of the online setting. Can the author describe a realistic scenario where a labeled calibration set from the deployment distribution is available, but unlabeled queries from the same distribution are not?
- Source of the gains. Could the authors run a ablation experiment in which the calibration percentiles are replaced by sorted i.i.d. Uniform(0,1) draws, with no stream data, and report its frontier alongside Cal-CDF and OCC? Moreover, for n=200 and Δ=0.005, the Cal-CDF percentiles coincide exactly with the grid points. Does OCC’s advantage over Cal-CDF persist under other grid spacings (e.g., Δ=0.0025 or 0.01), where this alignment no longer holds?
- Why the true-CDF threshold should be better. Is there any theoretical or empirical argument that the threshold certified under the true CDF attains higher coverage than the one certified under the calibration-only CDF, given the same calibration labels?
- Statistical uncertainty of the results. Most tables (e.g., Tables 1, 2, 3, 7, and 8) report only point estimates. Could the authors also provide the confidence intervals or standard deviations?