- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether a statistically valid risk certificate obtained from inexpensive proxy labels can control the target risk that a selective LLM system is meant to satisfy. The authors distinguish a reference-free proxy judge from a reference-aware operational target judge and derive an identity that decomposes the risk gap within the accepted set into false-positive and false-negative disagreement contributions. They also give a sufficient condition for transferring a proxy-risk guarantee to the target and construct finite-sample certificates using audited target labels.
The main experiment uses 1,018 questions from TriviaQA and TruthfulQA with a fixed self-consistency score and Learn-then-Test calibration. At a nominal risk of 0.20, proxy calibration selects a threshold that accepts every answer and has a held-out target risk of 0.250, while target calibration accepts 59.1 percent of answers with a target risk of 0.113. On a separate sample of 400 answers, the reference-aware target agrees with human annotations on 96.75 percent, compared with 74.50 percent for the proxy. Additional experiments vary the evaluator, rubric, calibration size, and target-label audit budget. The low-threshold failure recurs under lenient rubrics across three evaluator families, while the proposed audited corrections are generally less likely to return a certifiable rule than direct target-label calibration.
Critical strengths
The strongest part of the paper is that it separates statistical validity from the validity of the label used to define risk, and supports this distinction with a concrete held-out result. The proxy-selected rule satisfies its calibration certificate, with a Clopper–Pearson upper bound of 0.056, but its held-out target risk is 0.250 against a nominal limit of 0.20. The one-sided exact lower bound is 0.218, so the reported violation is not based only on a point estimate. The same LTT procedure is used for both label sources, and the answers and confidence scores remain fixed. I find the central observation credible, and it matters for LLM systems that rely on automated judges to calibrate when to answer or abstain.
The paper also makes a useful effort to check what its target labels actually measure and to report results that limit its conclusions. The 400-row human study shows substantially higher agreement with the reference-aware target than with the proxy, and the threshold-wise decomposition makes the direction of the mismatch visible. The authors further show that the failure depends on the rubric and task, that some evaluator settings return no solution, and that the audited corrections can be too conservative to certify a rule. They explicitly acknowledge that the main experiment does not demonstrate selection-induced amplification and that the operational target is not universal human ground truth. These qualifications make the empirical findings easier to interpret and prevent the paper from claiming a broader failure than it has measured.
Critical weaknesses
I lean toward rejection because the central finding is convincing, but the theoretical contribution is limited and the experiments do not establish what accepted-set analysis adds beyond global label mismatch.
The theoretical results largely follow from standard probability arguments. Proposition 1 decomposes disagreement between two binary labels, Proposition 2 gives a sufficient but unnecessary condition for transferring risk guarantees, and Proposition 3 combines existing confidence bounds using a union bound. These are valid results, but their novelty beyond noisy-label inference and selective-risk control is unclear. The experiments do little to establish the distinctive role of selection. In Table 1, the proxy-selected threshold accepts every answer, so the target-risk violation occurs without selective filtering. Figure 2 further shows that the risk gap decreases from 0.220 to 0.098 as coverage falls. The authors acknowledge that this does not demonstrate selection-induced amplification, but do not show what accepted-set analysis reveals beyond the mismatch already present in the full population.
My larger concern is that the main evaluator comparison does not isolate the cause of the mismatch. Appendix B.1 shows that the proxy judge is instructed to accept plausibly correct answers even when incomplete or missing caveats, while the target judge requires reference support and rejects partial answers. Reference availability and rubric strictness therefore change together. The human study further shows that the proxy marks 99 of 112 human-incorrect answers as correct. Its agreement of 74.50 percent is only slightly above the 72 percent achieved by always predicting correct on this sample. With such a lenient proxy, the low selected threshold and underestimated target risk are not surprising. Table 2 also shows that stricter reference-free rubrics can produce target risks below 0.20. The experiment establishes a valid failure case, but cannot tell whether missing references or the deliberately lenient rubric drives it, or whether the failure persists with more reliable proxies.
The audited corrections offer little demonstrated practical benefit. In Table 3, direct target-label LTT returns a certifiable rule in 82 percent of draws at budget 400, while Proxy+FP and Residual return no solution. This does not invalidate their guarantees, but the paper identifies no regime in which the additional proxy information improves certification over direct target calibration. I would reconsider my assessment if the authors could isolate reference access from rubric strictness and show a meaningful selection-specific effect beyond global disagreement.
Questions for authors
These are clarifications and focused checks that could affect my assessment.
- Could you separate the effects of reference access and rubric strictness in the main evaluator comparison? In Appendix B.1, the proxy is instructed to accept plausibly correct but incomplete answers, while the target rejects partial answers that are not supported by the reference. A comparison that holds the rubric fixed and changes only reference availability, ideally under both lenient and stricter rubrics, would help identify which factor drives the mismatch. Please report the resulting calibration thresholds, coverage, and held-out target risks. I would also be interested in whether the failure persists with a reference-free proxy that detects a reasonable fraction of human-incorrect answers.
- Could you provide an example or analysis showing that accepted-set disagreement adds information beyond global judge agreement? The main failure occurs at full coverage, and the observed gap decreases as the accepted set becomes smaller. For example, could you compare evaluators with similar global disagreement rates but different distributions of disagreement across confidence scores, and report how these differences affect selected thresholds and target risks? This would help establish the empirical value of the selection-aware theory rather than only the existence of label mismatch.
- Is there a setting in which Proxy+FP or Residual provides a more useful target-risk certificate than direct target-label LTT under the same audit budget? Table 3 shows substantially lower solve probabilities for both corrections, including zero successes at budget 400. Could you report how the component confidence bounds behave across thresholds and explain whether the poor feasibility is mainly caused by the high disagreement rate or by the conservatism of the simultaneous bounds? A comparison under less biased proxy labels would help determine whether these corrections have a useful operating regime.
- Could you report human-target agreement and human-label selective risk at each threshold on the existing 400-row annotation sample, together with the accepted counts? The overall agreement of 96.75 percent supports the operational target, but the paper's main claim concerns risk within selected sets. This analysis would show whether the target remains aligned with human judgments after filtering and whether the conclusions about target-risk control also hold when human labels are used for the selected answers.
5 comments
Follow-up on our response
Dear Reviewer QDbn,
Thank you again for laying out the four checks so clearly. We followed your Q1–Q4 order in our response. In particular, Table 24 gives the threshold-by-threshold human results you asked for, with both sets of judgments on the same 400 answers. If any of the four points could use further clarification, we would be happy to follow up while the discussion is open.
Thank you for the detailed rebuttal and additional experiments. The matched evaluator comparisons and human validation address much of my concern about the experimental design, while the new results also demonstrate a setting where proxy-assisted certification improves coverage.
However, my primary concern remains the limited methodological novelty beyond existing noisy-label inference and risk-control methods. Although the new results show that disagreement location improves risk prediction, they do not sufficiently establish what the accepted-set framework contributes beyond existing score-conditioned risk estimation. I therefore maintain my original rating.
Author Response to Reviewer QDbn
Thank you for your careful review and the four concrete questions. We especially appreciate your point that the original rule accepts every answer. It prompted us to test what disagreement location adds to risk prediction. We address your questions in turn.
Q1. What drives the mismatch? (§5.3, p. 8; Tables 17–18, pp. 25–26; exact messages and parsing, pp. 32–33.) On fixed answers, scores, and splits, we crossed reference access with rubric strictness. Cells give threshold / held-out coverage / held-out risk under one current DeepSeek target.
References changed the DeepSeek rule at fixed wording; wording alone changed the reference-free Qwen rule. Against the non-author consensus, the original proxy detected 13/108 human errors. The lenient reference-free Qwen proxy detected 47/108 (43.52%) yet selected full coverage at 25.93% target risk. Standard reference-free Qwen detected 66/108 and selected 1.0 (Appendix C.9, p. 29). These follow-up risks use current labels; Table 1 retains the frozen original result.
Q2. What does disagreement location add? (§5.4, p. 8; C.4–C.6, pp. 26–27.) We agree the original full-coverage violation needs no selection effect. Before collecting 1,000 new TriviaQA target labels, we fixed two predictors trained on the same separate 400 audit examples. For standard reference-free Qwen, adding the prespecified score split at 0.8 to the proxy verdict reduced validation Brier loss from 0.10205 to 0.08936 on 997 complete pairs (gain 0.01269, 98.33% paired bootstrap interval [0.00555, 0.02017], conditional on training). At 0.8 the predictions were 21.24% versus 15.64%; observed risk was 115/766 = 15.01%. A permutation test preserving the full global confusion matrix and proxy–score relationship gave adjusted p = 0.00015.
An exploratory routing comparison matched global disagreement but selected different thresholds. The joint contrast did not replicate on fresh splits (C.6), and both natural model proxies selected 1.0 on the earlier audit. Our prespecified result is improved prediction of accepted-set risk.
Q3. When do the corrections help? (§5.6, Table 3, p. 9; Tables 21–23, pp. 27–28.) On 4,800 additional TriviaQA questions, the three methods shared 400 target-audit labels and 400 test questions; the corrections also used a 4,000-question proxy pool. At 0.8, the reference-aware Qwen proxy forgave only 3/310 accepted audit answers that the target marked wrong. The direct upper bound was 0.21985. Proxy+FP combined 0.16408 and 0.03204 to give 0.19612; Residual gave 0.19763. Thus both corrections certified 0.8 and covered 309/400 test answers (77.25%, 41 errors); direct LTT selected 1.0 and covered 276/400 (69.00%, 28 errors). At the same threshold, the original full-frame FP rate was 94/708 = 0.133 (Table 4, p. 15), versus 3/310 = 0.00968 in the new audit (Table 21). These estimates describe different pools and proxies. Table 22 shows how the simultaneous bounds combine the component estimates and their uncertainty at each threshold. Table 29 (p. 34) varies proxy-pool size: at audit budget 400, coverage first exceeds direct LTT at 4,000 proxy questions.
Q4. Human outcomes on the original 400 answers (§5.5, Table 2, p. 8; C.9, p. 29). Table 24 now gives both sets of final judgments on the same 400 answers. Agreement is with the frozen operational target; both rates use the number accepted at each threshold.
Both audits used supplied references. On a separate 200-answer audit without supplied references, independent initial agreement was 196/200. After joint adjudication, human error was 19/148 at 0.8 and 13/127 at 1.0 (Table 26, p. 30). These are descriptive human checks on fixed thresholds.
Our theory builds on noisy-label inference and PPI. We study transfer between two label definitions on a data-selected set. Proposition 2 (p. 5) states a sufficient ordering and explains how calibration slack absorbs a positive gap. Q2 and Q3 show what locating and auditing disagreement adds for that set. Two close comparisons are Feng et al. (2026) on imperfect-judge certification and SCOPE (2026) on calibrating abstention in pairwise judging.