acceptodds
Under review as a conference paper at ICLR 2027

When Proxy Calibration Controls the Wrong Risk: Accepted-Set Target Mismatch in Selective LLMs

Abstract

Selective LLM systems often use inexpensive proxy or judge labels for calibration, but a statistically valid certificate only controls the risk defined by those labels. Does proxy-valid selective calibration control the target risk that deployment actually cares about? We derive an accepted-set identity decomposing the proxy–target risk gap into signed disagreement contributions; global judge agreement alone does not establish transfer. At α = 0.20, the held-out main evaluation shows proxy-label calibration selecting λ = 0.2 with coverage 1.000 and held-out target risk 0.250, whereas target-label calibration selects λ = 1.0 with coverage 0.591 and target risk 0.113. A threshold-wise full-frame diagnostic shows that accepted false-positive contributions outweigh false-negative contributions, producing the proxy–target risk gap. In a 400-row blinded validation sample, human judgments agree with the reference-aware operational target on 96.75%, versus 74.50% with the reference-free proxy. The low-threshold failure reproduces under lenient rubrics across DeepSeek, Qwen, and MiniMax, whereas stricter rubrics can select more conservative thresholds or, in one case, return no solution. Increasing target-label audit budgets improves direct feasibility, whereas proxy-assisted corrections can remain conservative. Selective-risk validity is target-relative: deployment claims require measurement validation and target-specific evidence.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Selective Prediction, Large Language Models, Risk Calibration, Proxy Labels, Distribution-Free Risk Control, LLM-as-a-Judge, Measurement Validity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QDbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a statistically valid risk certificate obtained from inexpensive proxy labels can control the target risk that a selective LLM system is meant to satisfy. The authors distinguish a reference-free proxy judge from a reference-aware operational target judge and derive an identity that decomposes the risk gap within the accepted set into false-positive and false-negative disagreement contributions. They also give a sufficient condition for transferring a proxy-risk guarantee to the target and construct finite-sample certificates using audited target labels.

The main experiment uses 1,018 questions from TriviaQA and TruthfulQA with a fixed self-consistency score and Learn-then-Test calibration. At a nominal risk of 0.20, proxy calibration selects a threshold that accepts every answer and has a held-out target risk of 0.250, while target calibration accepts 59.1 percent of answers with a target risk of 0.113. On a separate sample of 400 answers, the reference-aware target agrees with human annotations on 96.75 percent, compared with 74.50 percent for the proxy. Additional experiments vary the evaluator, rubric, calibration size, and target-label audit budget. The low-threshold failure recurs under lenient rubrics across three evaluator families, while the proposed audited corrections are generally less likely to return a certifiable rule than direct target-label calibration.

Critical strengths

The strongest part of the paper is that it separates statistical validity from the validity of the label used to define risk, and supports this distinction with a concrete held-out result. The proxy-selected rule satisfies its calibration certificate, with a Clopper–Pearson upper bound of 0.056, but its held-out target risk is 0.250 against a nominal limit of 0.20. The one-sided exact lower bound is 0.218, so the reported violation is not based only on a point estimate. The same LTT procedure is used for both label sources, and the answers and confidence scores remain fixed. I find the central observation credible, and it matters for LLM systems that rely on automated judges to calibrate when to answer or abstain.

The paper also makes a useful effort to check what its target labels actually measure and to report results that limit its conclusions. The 400-row human study shows substantially higher agreement with the reference-aware target than with the proxy, and the threshold-wise decomposition makes the direction of the mismatch visible. The authors further show that the failure depends on the rubric and task, that some evaluator settings return no solution, and that the audited corrections can be too conservative to certify a rule. They explicitly acknowledge that the main experiment does not demonstrate selection-induced amplification and that the operational target is not universal human ground truth. These qualifications make the empirical findings easier to interpret and prevent the paper from claiming a broader failure than it has measured.

Critical weaknesses

I lean toward rejection because the central finding is convincing, but the theoretical contribution is limited and the experiments do not establish what accepted-set analysis adds beyond global label mismatch.

The theoretical results largely follow from standard probability arguments. Proposition 1 decomposes disagreement between two binary labels, Proposition 2 gives a sufficient but unnecessary condition for transferring risk guarantees, and Proposition 3 combines existing confidence bounds using a union bound. These are valid results, but their novelty beyond noisy-label inference and selective-risk control is unclear. The experiments do little to establish the distinctive role of selection. In Table 1, the proxy-selected threshold accepts every answer, so the target-risk violation occurs without selective filtering. Figure 2 further shows that the risk gap decreases from 0.220 to 0.098 as coverage falls. The authors acknowledge that this does not demonstrate selection-induced amplification, but do not show what accepted-set analysis reveals beyond the mismatch already present in the full population.

My larger concern is that the main evaluator comparison does not isolate the cause of the mismatch. Appendix B.1 shows that the proxy judge is instructed to accept plausibly correct answers even when incomplete or missing caveats, while the target judge requires reference support and rejects partial answers. Reference availability and rubric strictness therefore change together. The human study further shows that the proxy marks 99 of 112 human-incorrect answers as correct. Its agreement of 74.50 percent is only slightly above the 72 percent achieved by always predicting correct on this sample. With such a lenient proxy, the low selected threshold and underestimated target risk are not surprising. Table 2 also shows that stricter reference-free rubrics can produce target risks below 0.20. The experiment establishes a valid failure case, but cannot tell whether missing references or the deliberately lenient rubric drives it, or whether the failure persists with more reliable proxies.

The audited corrections offer little demonstrated practical benefit. In Table 3, direct target-label LTT returns a certifiable rule in 82 percent of draws at budget 400, while Proxy+FP and Residual return no solution. This does not invalidate their guarantees, but the paper identifies no regime in which the additional proxy information improves certification over direct target calibration. I would reconsider my assessment if the authors could isolate reference access from rubric strictness and show a meaningful selection-specific effect beyond global disagreement.

Questions for authors

These are clarifications and focused checks that could affect my assessment.

  1. Could you separate the effects of reference access and rubric strictness in the main evaluator comparison? In Appendix B.1, the proxy is instructed to accept plausibly correct but incomplete answers, while the target rejects partial answers that are not supported by the reference. A comparison that holds the rubric fixed and changes only reference availability, ideally under both lenient and stricter rubrics, would help identify which factor drives the mismatch. Please report the resulting calibration thresholds, coverage, and held-out target risks. I would also be interested in whether the failure persists with a reference-free proxy that detects a reasonable fraction of human-incorrect answers.
  2. Could you provide an example or analysis showing that accepted-set disagreement adds information beyond global judge agreement? The main failure occurs at full coverage, and the observed gap decreases as the accepted set becomes smaller. For example, could you compare evaluators with similar global disagreement rates but different distributions of disagreement across confidence scores, and report how these differences affect selected thresholds and target risks? This would help establish the empirical value of the selection-aware theory rather than only the existence of label mismatch.
  3. Is there a setting in which Proxy+FP or Residual provides a more useful target-risk certificate than direct target-label LTT under the same audit budget? Table 3 shows substantially lower solve probabilities for both corrections, including zero successes at budget 400. Could you report how the component confidence bounds behave across thresholds and explain whether the poor feasibility is mainly caused by the high disagreement rate or by the conservatism of the simultaneous bounds? A comparison under less biased proxy labels would help determine whether these corrections have a useful operating regime.
  4. Could you report human-target agreement and human-label selective risk at each threshold on the existing 400-row annotation sample, together with the accepted counts? The overall agreement of 96.75 percent supports the operational target, but the paper's main claim concerns risk within selected sets. This analysis would show whether the target remains aligned with human judgments after filtering and whether the conclusions about target-risk control also hold when human labels are used for the selected answers.
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer QDbn

Thank you for your careful review and the four concrete questions. We especially appreciate your point that the original rule accepts every answer. It prompted us to test what disagreement location adds to risk prediction. We address your questions in turn.

Q1. What drives the mismatch? (§5.3, p. 8; Tables 17–18, pp. 25–26; exact messages and parsing, pp. 32–33.) On fixed answers, scores, and splits, we crossed reference access with rubric strictness. Cells give threshold / held-out coverage / held-out risk under one current DeepSeek target.

Judge and rubricNo referencesReferences supplied
DeepSeek, lenient0.2 / 100% / 25.93%0.6 / 81.34% / 18.60%
DeepSeek, standard0.2 / 100% / 25.93%0.8 / 69.55% / 14.41%
Qwen, lenient0.2 / 100% / 25.93%1.0 / 59.14% / 10.63%
Qwen, standard1.0 / 59.14% / 10.63%no solution / 0% / undefined

References changed the DeepSeek rule at fixed wording; wording alone changed the reference-free Qwen rule. Against the non-author consensus, the original proxy detected 13/108 human errors. The lenient reference-free Qwen proxy detected 47/108 (43.52%) yet selected full coverage at 25.93% target risk. Standard reference-free Qwen detected 66/108 and selected 1.0 (Appendix C.9, p. 29). These follow-up risks use current labels; Table 1 retains the frozen original result.

Q2. What does disagreement location add? (§5.4, p. 8; C.4–C.6, pp. 26–27.) We agree the original full-coverage violation needs no selection effect. Before collecting 1,000 new TriviaQA target labels, we fixed two predictors trained on the same separate 400 audit examples. For standard reference-free Qwen, adding the prespecified score split at 0.8 to the proxy verdict reduced validation Brier loss from 0.10205 to 0.08936 on 997 complete pairs (gain 0.01269, 98.33% paired bootstrap interval [0.00555, 0.02017], conditional on training). At 0.8 the predictions were 21.24% versus 15.64%; observed risk was 115/766 = 15.01%. A permutation test preserving the full global confusion matrix and proxy–score relationship gave adjusted p = 0.00015.

An exploratory routing comparison matched global disagreement but selected different thresholds. The joint contrast did not replicate on fresh splits (C.6), and both natural model proxies selected 1.0 on the earlier audit. Our prespecified result is improved prediction of accepted-set risk.

Q3. When do the corrections help? (§5.6, Table 3, p. 9; Tables 21–23, pp. 27–28.) On 4,800 additional TriviaQA questions, the three methods shared 400 target-audit labels and 400 test questions; the corrections also used a 4,000-question proxy pool. At 0.8, the reference-aware Qwen proxy forgave only 3/310 accepted audit answers that the target marked wrong. The direct upper bound was 0.21985. Proxy+FP combined 0.16408 and 0.03204 to give 0.19612; Residual gave 0.19763. Thus both corrections certified 0.8 and covered 309/400 test answers (77.25%, 41 errors); direct LTT selected 1.0 and covered 276/400 (69.00%, 28 errors). At the same threshold, the original full-frame FP rate was 94/708 = 0.133 (Table 4, p. 15), versus 3/310 = 0.00968 in the new audit (Table 21). These estimates describe different pools and proxies. Table 22 shows how the simultaneous bounds combine the component estimates and their uncertainty at each threshold. Table 29 (p. 34) varies proxy-pool size: at audit budget 400, coverage first exceeds direct LTT at 4,000 proxy questions.

Q4. Human outcomes on the original 400 answers (§5.5, Table 2, p. 8; C.9, p. 29). Table 24 now gives both sets of final judgments on the same 400 answers. Agreement is with the frozen operational target; both rates use the number accepted at each threshold.

ThresholdOriginal authors (risk; agreement)Non-authors after joint review (risk; agreement)
0.2112/400 (28.00%; 96.75%)108/400 (27.00%; 95.25%)
0.4101/371 (27.22%; 97.04%)97/371 (26.15%; 95.96%)
0.678/330 (23.64%; 97.27%)74/330 (22.42%; 95.45%)
0.854/281 (19.22%; 98.58%)52/281 (18.51%; 97.15%)
1.040/239 (16.74%; 98.74%)38/239 (15.90%; 97.07%)

Both audits used supplied references. On a separate 200-answer audit without supplied references, independent initial agreement was 196/200. After joint adjudication, human error was 19/148 at 0.8 and 13/127 at 1.0 (Table 26, p. 30). These are descriptive human checks on fixed thresholds.

Our theory builds on noisy-label inference and PPI. We study transfer between two label definitions on a data-selected set. Proposition 2 (p. 5) states a sufficient ordering and explains how calibration slack absorbs a positive gap. Q2 and Q3 show what locating and auditing disagreement adds for that set. Two close comparisons are Feng et al. (2026) on imperfect-judge certification and SCOPE (2026) on calibrating abstention in pairwise judging.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer QDbn,

Thank you again for laying out the four checks so clearly. We followed your Q1–Q4 order in our response. In particular, Table 24 gives the threshold-by-threshold human results you asked for, with both sets of judgments on the same 400 answers. If any of the four points could use further clarification, we would be happy to follow up while the discussion is open.

Thank you for the detailed rebuttal and additional experiments. The matched evaluator comparisons and human validation address much of my concern about the experimental design, while the new results also demonstrate a setting where proxy-assisted certification improves coverage.

However, my primary concern remains the limited methodological novelty beyond existing noisy-label inference and risk-control methods. Although the new results show that disagreement location improves risk prediction, they do not sufficiently establish what the accepted-set framework contributes beyond existing score-conditioned risk estimation. I therefore maintain my original rating.

Reviewer_cEy2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when selective calibration using proxy correctness labels fails to control risk under a separate target label. It decomposes the accepted-set risk gap into signed disagreement contributions and gives simultaneous audit-based certificates.

Critical strengths

The distinction between a valid statistical procedure and an unsuitable measurement target is clearly presented. The signed decomposition makes the source of transfer failure easy to inspect, and the experiments separate held-out results from full-pool diagnostics. I also appreciate the human-label check, rubric and evaluator comparisons, explicit no-solution accounting, and equal-audit-budget comparison showing that the proposed corrections can lose to direct target calibration.

Critical weaknesses

Some quibbles with presentation. The main proxy is explicitly prompted to be lenient, whereas the target is prompted to require reference support, so the headline result is partly a deliberately constructed rubric mismatch. The broader rubric study helps, but shared answers from one generator and author annotations using the same references limit generalization and independence of target validation.

Technically, Proposition 2 gives a sufficient transfer condition, not a necessary characterization: target risk can exceed proxy risk while both remain below alpha, so the surrounding language that transfer "must" satisfy this ordering is too strong.

Finally, the proxy-assisted methods rarely or never certify a rule at the tested budgets; this supports caution about those implementations, but does not establish an inherent audit-cost disadvantage for proxy assistance.

Questions for authors
  1. Can you isolate reference availability from rubric strictness using matched prompts, and replicate with another answer generator and independent human annotators?
  2. Can Proposition 2 and its interpretation explicitly distinguish sufficient risk ordering from necessary conditions, including cases where calibration slack absorbs a positive proxy-target gap?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer cEy2

Thank you for pointing out that the original comparison changed both reference access and rubric strictness. Your suggestion to hold the rubric fixed gave us a direct way to test the effect of supplying references. We also appreciate your observation that target risk can exceed proxy risk while both remain below the nominal risk limit α\alpha. We revised Proposition 2 to make clear that the ordering is sufficient and to show how calibration slack can absorb a positive gap.

Q1. Reference access, rubric strictness, and replication (§5.3, p. 8; Tables 17–18, pp. 25–26). We varied reference visibility and lenient/standard wording independently, holding the answers, scores, split, judge, and output format fixed within each comparison. Appendix C.13 (pp. 32–33) reproduces the system messages, both user payloads, parsers, and failure handling. We evaluated all selected rules against one common current DeepSeek target. On the original answer pool:

Judge and rubricWithout references: threshold / coverage / target riskWith references: threshold / coverage / target risk
DeepSeek, lenient0.2 / 100% / 25.93%0.6 / 81.34% / 18.60%
DeepSeek, standard0.2 / 100% / 25.93%0.8 / 69.55% / 14.41%
Qwen, lenient0.2 / 100% / 25.93%1.0 / 59.14% / 10.63%
Qwen, standard1.0 / 59.14% / 10.63%no solution / 0% / undefined

Reference access alone changed the DeepSeek rule under either rubric. With references absent, changing the rubric alone moved Qwen from full coverage to threshold 1.0. The 25.93% full-coverage figure uses fresh target judgments; the original frozen result is 127/509 = 24.95% (Table 1). We repeated the matched comparison on another generator: with Qwen3.5-Flash answers and the same lenient DeepSeek judge, supplying references moved the threshold from 0.6 to 1.0, coverage from 78.39% to 52.85%, and target risk from 98/399 = 24.56% to 40/269 = 14.87%. The size of the effect varies across the five generator pools.

Independent human validation (§§4.3 and 5.5, pp. 6 and 8; Tables 24–26, pp. 29–30). Two non-authors independently reannotated the original 400 answers with the supplied references. Their initial agreement was 78.50%. After joint review, agreement with the operational target was 95.25% overall and 97.07% on the 239 answers accepted at threshold 1.0. Because this check shares the target's references, we also drew 200 answers at random from a separate Qwen3.5-27B test pool. The annotators saw the question and answer, with dataset references, judge verdicts, and scores withheld. They agreed on 196/200 initial judgments. At the previously fixed thresholds of 0.8 and 1.0, final human error was 19/148 (12.84%) and 13/127 (10.24%), with target agreement of 97.30% and 98.43%. The thresholds were fixed before human review. These are sample-level alignment results. The paper records the independent initial labels and the subsequent joint adjudication separately.

Q2. When can positive mismatch be absorbed? (§3.3, Proposition 2, p. 5.) We have corrected the statement and its interpretation. Write Δ=RT−RP\Delta=R_T-R_P at a threshold with positive coverage. The exact condition for target control is Δ≤α−RP\Delta\leq\alpha-R_P. For example, RP=0.10R_P=0.10 and RT=0.15R_T=0.15 both meet α=0.20\alpha=0.20. Requiring RT≤RPR_T\leq R_P at every returnable threshold is sufficient. When the gap is estimated, simultaneous upper bounds for proxy risk and the gap can certify transfer if their sum is at most α\alpha, with confidence allocated jointly. Figure 2b (p. 7) now uses the accepted count for both signed disagreement contributions; the submitted Table 4 already used this denominator, and Table 16 (p. 25) gives the exact counts.

Audit feasibility (§5.6, Table 3, p. 9). We agree that the original result depended on the proxy and the audit setting. With a 4,000-question reference-aware Qwen proxy pool and the same 400 target-audit labels for each method, direct LTT selected threshold 1.0 (69.00% held-out coverage, 10.14% target risk). Both audited corrections selected 0.8 (77.25% coverage, 13.27% target risk). Tables 21–23 (pp. 27–28) show how the false-positive component and its uncertainty enter the bounds. This establishes a benefit at equal target-label count in a specified regime; the additional proxy judgments also carry cost. Table 29 (p. 34) reports solve rates and coverage for all four proxy-pool sizes. The dependence on judge quality connects to recent work on imperfect-judge certification. Feng et al., ICLR 2026

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer cEy2,

We appreciated your suggestion to vary reference access while keeping the rubric fixed. We ran that comparison and reported the results in Tables 17–18. When you have a chance, we would be grateful for your thoughts on whether this addresses the confounding concern you raised.

AuthorsOpenReview ↗

Author Response to Reviewer cEy2

Dear Reviewer cEy2,

You noted that the poor certification feasibility in the original experiments does not establish an inherent audit-cost disadvantage for proxy assistance. We completed an additional check of how feasibility depends on the calibration procedure.

Using the additional TriviaQA audit study described in our response, we applied the same fixed-sequence test to direct target calibration and both audited corrections. Every method used the predetermined threshold order 1.0, 0.8, 0.6, 0.4, 0.2, with a risk limit of 0.20 and an overall error probability of 0.10.

With the reference-aware Qwen proxy and a budget of 200 target-audit labels, mean held-out coverage was 65.75% for direct calibration, 75.03% for Proxy+FP, and 73.31% for Residual. All methods received the same target-audit subsets. The corrections additionally used the existing 4,000-question proxy pool.

The 9.28-percentage-point gain for Proxy+FP shows that the benefit persists when direct calibration also uses this alternative procedure. This retrospective comparison averages 5,000 paired subsamples of the existing 400-question audit reservoir, evaluated on the same separate 400-question test pool.

These results further support your point that audit feasibility depends on the proxy and implementation. They provide another concrete setting in which additional proxy information improves certification at the same target-label budget. Thank you again for helping us clarify this aspect of the paper.

Reviewer_BraQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that Learn-then-Test calibration with a reference-free LLM judge controls proxy risk but not reference-aware target risk, and derives an accepted-set decomposition of the gap together with audited target-risk certificates. On a TriviaQA/TruthfulQA pool, proxy calibration yields held-out target risk 0.250 at \alpha = 0.20, versus 0.113 for target calibration.

Critical strengths
  • A practically relevant point, clearly made. The paper shows that a statistically valid certificate controls only the risk of the label it was calibrated on, and illustrates this with a concrete case where the proxy certificate holds but target risk is 0.25.
Critical weaknesses
  • The propositions largely follow from elementary probability identities and standard confidence bounds, and label-dependent validity is already established in the noisy-label conformal and PPI literature. The proposed corrections also offer little practical benefit: Proxy+FP rarely succeeds, and the residual method never succeeds, compared with an 82% solve rate for direct target LTT at budget 400.

  • The headline proxy-selected rule accepts the entire pool, so its failure reflects global label bias rather than selection-induced amplification. The risk gap decreases under stricter thresholds. Evaluation is also limited to one primary model, a 1,018-answer pool, and a five-point threshold grid.

Questions for authors

See above weaknesses.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer BraQ

Thank you for pointing out that the headline rule accepts every answer and that the original audit results favor direct calibration. We agree on both points. We appreciated your question about what the accepted-set analysis contributes beyond the probability identity itself. It led us to test two practical questions in the revision: whether disagreement location predicts risk for a particular acceptance rule, and when an audited proxy can certify greater coverage than direct calibration at the same target-label budget.

What does selection add? (§5.4, p. 8; Appendix C.4–C.6, pp. 26–27.) At full coverage the original failure is a global label mismatch. For a stricter rule, knowing how many errors the judge makes overall leaves open which errors the rule accepts. We tested this on 1,000 previously proxy-only TriviaQA questions. Before obtaining their target labels, we fixed two predictors trained on the same separate 400 audit examples. Both used the standard reference-free Qwen verdict; one additionally distinguished answers scoring at least 0.8 from those below 0.8. On 997 complete validation pairs, the Brier loss fell from 0.10205 to 0.08936 (paired improvement 0.01269, 98.33% bootstrap interval [0.00555, 0.02017], conditional on the training sample). At threshold 0.8, the two predictors gave 21.24% and 15.64% risk; the observed risk was 115/766 = 15.01%. Thus the location of disagreement helps predict risk for an actual acceptance rule. Calibrated on the earlier audit, both model proxies selected 1.0 for this validation. The evidence supports diagnostic value; the original shrinking gap remains visible in Fig. 2 (p. 7).

When is auditing useful? (§5.6, Table 3, p. 9; Tables 21–23, pp. 27–28.) Before generation, we split 4,800 additional TriviaQA questions into a proxy pool (4,000), a target-audit pool (400), and a test pool (400). Under a standard reference-aware Qwen proxy, only 3 of 310 accepted audit answers at threshold 0.8 were proxy-correct and target-wrong. At that threshold, the upper target-risk bounds were 0.21985 for direct LTT, 0.19612 for Proxy+FP, and 0.19763 for Residual (limit 0.20). Direct calibration selected 1.0 and covered 276/400 test answers (69.00%, 28 target errors). Both corrections selected 0.8 and covered 309/400 (77.25%, 41 errors). The gain is 33 additional held-out answers for the same 400 target-audit labels, using additional proxy judgments. Table 29 (p. 34) shows the size dependence: at budget 400 the corrections match direct coverage for 500, 1,000, and 2,000 proxy questions and exceed it at 4,000. The original 82% direct solve rate and zero correction solves at budget 400 remain in Appendix Table 5 (p. 16). Together the studies show why the amount of proxy data and the frequency of accepted false positives matter.

Theory and scope (§§2–3.4, pp. 2–5). We use the probability identity, residual correction, and simultaneous bounds as established tools. The specific contribution is to connect the certificate for the threshold selected using one label to the risk defined by a second label on that same set. Proposition 2 states why ordering is sufficient and how slack can absorb a positive gap. Proposition 3 and its proof now identify the two disjoint target-error events underlying the upper bound (pp. 5 and 14). Section 2 (p. 3) now discusses Einbinder, Ringel & Romano (TPAMI 2025), who use auxiliary predictions to calibrate a rule for a specified risk more efficiently. We ask when the certificate obtained with one correctness label transfers to another on the selected set. Feng et al. (ICLR 2026) study certification with imperfect judges. Our two experiments quantify what global judge statistics leave unresolved for a selected QA rule and when target audits resolve enough uncertainty to change that rule. Risk control via PPI · Feng et al.

The generator comparison now spans five answer pools. For example, with Qwen3.5-Flash answers and a fixed lenient DeepSeek rubric, supplying references changed the selected threshold from 0.6 to 1.0 and held-out target risk from 24.56% to 14.87%. Tables 18–19 (pp. 25–26) separate the matched prompts from the exploratory original-prompt comparison and show the variation across generators. Finally, with five generations the largest normalized-answer cluster has size 1 through 5. The five positive thresholds exhaust the distinct nonempty acceptance sets on the original pool (§4.1, p. 5; scoring details, p. 20). Generalization to finer scores would require a different scoring construction.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer BraQ,

Thank you again for asking what the accepted-set analysis adds beyond global agreement. If you have a chance to revisit our response before the discussion closes, we would especially welcome your thoughts on the held-out comparison in §5.4, where we tested whether knowing where disagreements occur improves risk prediction.

AuthorsOpenReview ↗

Author Response to Reviewer BraQ

Dear Reviewer BraQ,

Your review raised two concerns about the audited corrections: their limited practical benefit in the original experiments and their relationship to existing noisy-label and PPI methods. We completed an additional comparison addressing these points.

We reused the additional TriviaQA study described in our response, comprising 4,000 proxy-labeled questions, 400 target-audit questions, and a separate 400-question test pool. For the reference-aware Qwen proxy, we compared our implementations with conditional adaptations of the binary and WSR procedures from semi-supervised risk control. Einbinder et al., 2025

To examine dependence on threshold selection, we applied the same fixed-sequence test to every method, using the predetermined order 1.0, 0.8, 0.6, 0.4, 0.2. The risk limit was 0.20 and the overall error probability was 0.10. At a budget of 200 target labels, mean held-out coverage was:

MethodMean coverage
Direct target calibration65.75%
Proxy+FP / SS-RCPS binary, equal component allocation75.03%
Residual73.31%
SS-RCPS binary, fixed 10/90 component allocation75.69%
SS-RCPS WSR74.12%

The 9.28-percentage-point gain for Proxy+FP persists when direct calibration also uses this alternative selection procedure. The comparison additionally clarifies the methodological overlap: with identical component allocations and confidence bounds, Proxy+FP coincides with the binary positive-residual correction.

This retrospective analysis uses 5,000 paired subsamples of the existing audit reservoir and the same separate test pool. It strengthens the evidence for a useful operating regime of proxy-assisted auditing, while making the connection to established methods explicit. Thank you for prompting us to examine both aspects more directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.