acceptodds
Under review as a conference paper at ICLR 2027

The Verification Squeeze: From Verifier Scores to Deployment Decisions

Abstract

Standard LLM verifiers are often evaluated by accuracy or AUROC, but higher scores do not show that accepted outputs meet a target reliability level. For such deployment decisions, what matters is *accepted risk*: the fraction of accepted outputs that are wrong, which depends on the solver population's base error. We call the gap between required and supplied evidence the *verification squeeze*. We introduce the Verification Feasibility Audit (VFA), a two-gate framework. Gate 1 asks whether a verifier-selection procedure provides enough evidence for the population and risk target; Gate 2 asks whether independent calibration data support a fixed acceptance rule. Across five LLMs from four providers, 117 matched benchmark–method comparisons, and a 120-condition calibration design, we find three patterns. (1) Reusing evaluation labels during selection can overstate verifier reliability: measured evidence rises by at the median, and 3 of 40 controlled conditions cross the evidence threshold. (2) Verifier improvements often do not change the audit outcome: none of the audited-infeasible baselines becomes feasible under the methods we test. (3) Calibration support weakens as the statistical claim is strengthened: seven of eight rules meet the empirical accepted-risk target, five pass the prespecified large-sample screen, while finite-sample methods support none of the applicable rules they evaluate at the available sample sizes. These results show that conventional verifier metrics alone are insufficient for deployment decisions. Verifier evaluations should also report the evaluated population, selection protocol, evidence margin, and supported risk statement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Verification, LLM-as-a-Judge, reference-free evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cE1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a important gap in verifier evaluation. A verifier can have good accuracy or AUROC and still fail to make reliable acceptance decisions. The paper therefore studies accepted risk. It asks how often an accepted output is wrong for a particular solver population. The proposed Verification Feasibility Audit has two gates. Gate 1 checks whether a verifier-selection procedure provides enough held-out evidence for a chosen risk and retention target. Gate 2 fixes one rule and asks what separate calibration data can support. The experiments cover label reuse, verifier selection, aggregation, routing, and several calibration procedures.

Critical strengths
  1. It does not stop at verifier accuracy. It asks whether accepted outputs meet a target risk for the tested solver population. The likelihood-ratio margin also shows how far a method remains from that target.

  2. Gate 1 evaluates the search procedure using held-out predictions. Gate 2 then studies one frozen rule on separate calibration data. The paper is also careful to separate empirical success, a large-sample screen, and a finite-sample guarantee.

Critical weaknesses
  1. The main 117-comparison scaling result uses a weaker analysis than the formal Gate 1. The selected rule is held fixed inside each fold, and the final benchmark label is produced from fold-level statuses. Selection is not repeated inside this bootstrap. Therefore, the result is a useful scaling summary, but it is not a full population-level Gate-1 comparison.
  2. The complete two-gate process is only partly shown. At the primary IFEval target, Gate 1 remains uncertain and Gate 2 is not reached. The full pipeline is run only after the risk target is relaxed from 0.10 to 0.20, and the finite-sample methods still give no support. The framework is well motivated, but its end-to-end success is not yet shown.Also Eq. (1) is a rearrangement of standard Bayes, which the paper itself concedes. The label-reuse effect is small (1.13×) and its direction follows by construction.
Questions for authors
  1. Do the main conclusions from the 117 scaling comparisons still hold if each comparison is rerun using the full selection-aware Gate-1 bootstrap?
  2. Is there any pre-specified setting in the current experiments where the complete audit reaches finite-sample-supported?
2 comments
AuthorsOpenReview ↗

Response to Reviewer cE1j (Part 1/2)

Thank you for identifying two gaps: the scaling bootstrap held the selected rules fixed, and no complete audit produced a finite-sample-supported positive result. We have addressed both. We reran all 117 comparisons with selection repeated inside the bootstrap; the estimated gains grow, but the evidence gap remains. A new natural-output GSM8K study at the original 10% risk target certifies a rule that accepts 73.0% of test answers with 4.2% error.

W1. The scaling bootstrap holds the selected rule fixed

We have rerun all 117 comparisons with method and threshold selection repeated inside the bootstrap. You are right that the submitted analysis held those choices fixed and therefore measured uncertainty conditional on the selected rules.

The new analysis uses 2,000 paired parent-task bootstrap replicates. Each replicate repeats score fitting where applicable, baseline/method selection, and threshold selection before pooling the held-out decisions. The baseline and improved method use the same resampled tasks, so their difference is evaluated as a paired comparison. For externally supplied scores, the scores remain fixed and the thresholds are reselected.

Accounting for selection increases the estimated gains but preserves the main evidence-gap finding. We will make this selection-aware analysis the primary result in Section 4.3, update the affected classifications, and retain the submitted estimates as a clearly labeled conditional comparison.

W2(a). The complete two-gate process has not succeeded

The new experiment completes both gates at the original 10% target and certifies a rule that accepts 73.0% of test answers with 4.2% error. This supplies the nontrivial positive case missing from the submitted IFEval walkthrough.

The study uses 1,259 naturally generated GSM8K answers, with separate tasks for PILOT, FIT, CAL, and TEST. Gate 1 returns FEASIBLE with a margin interval of [0.53, 2.12]. FIT then freezes the rule requiring at least two of three judges to vote PASS. Independent CAL gives this rule an exact one-sided 95% risk upper bound of 4.1%, below the 10% target.

Accepting every test answer would incur 25.9% error, so accept-all cannot meet the same risk target. The original Hoeffding test also certifies the rule on the same CAL data. The successful audit therefore does not depend on replacing that test with a sharper bound.

W2(b). Equation (1) is Bayes' rule, and label reuse has an expected effect

The contribution is the measured shortfall between verifier progress and the improvement required for reliable acceptance. We agree that the Bayesian identity is standard. It gives us a common scale on which to compare the required and achieved improvements.

In the submitted Table 13, the 36 audited-infeasible baseline comparisons have a median deficit of 1.97 log units, requiring approximately 7.2× stronger likelihood-ratio evidence. The selection-aware reanalysis shows that the typical gain in the original positive-gain subset remains far smaller than its corresponding deficit. This quantifies the squeeze rather than merely restating the risk condition.

The direction of selection bias is also expected; its effect on the acceptance assessment must still be measured. In the controlled label-access experiment, the macro-median margin difference is 0.124, and 3 of 40 point estimates cross the feasibility boundary. In those comparisons, access to evaluation labels changes whether the verifier appears to meet the evidence requirement. These are reversals of apparent feasibility, not certified deployment decisions. We will foreground these measured consequences in the contribution statement.

AuthorsOpenReview ↗

Response to Reviewer cE1j (Part 2/2)

Q1. Do the conclusions hold after the full selection-aware reanalysis?

Yes. The estimated improvements increase, but most remain insufficient to close the measured evidence deficit. We retain the original 117 comparisons and the original 12-positive-gain subset, so the update does not select a new set of favorable comparisons.

QuantitySelection-aware result
Median updated gain / corresponding submitted deficit, on the original 12-positive-gain comparisons19.0%, versus 12.5% in the submission
One-sided 95% bootstrap upper bound on that median25.0%
Original deficit baselines whose margin upper bound remains below zero36/36
Corresponding improved methods whose upper bound remains below zero35/36

The 19% statistic uses each comparison's submitted log-evidence deficit as its denominator. The 25% bound applies to the median over that original subset, not to every individual improvement.

The analysis also retains positive results. All three Complexity UNCERTAIN-to-FEASIBLE transitions survive. TravelPlanner rationale features reach a point margin of +0.27, but the interval remains UNCERTAIN. There are no strict audited-infeasible-to-FEASIBLE transitions.

Individual labels do change: baseline classifications and improved-method classifications each remain unchanged in 104/117 cases. These are separate counts, not the same 104 unchanged pairs. We will update the full transition tables. Alternative treatments of degenerate bootstrap replicates leave the median-gain upper bound at approximately 25%.

The conclusion now rests on measured gains and their uncertainty, not just on the absence of status crossings. The audit detects improvements while showing that they usually leave a substantial evidence shortfall in the evaluated deficit settings.

Q2. Is there a pre-specified setting that completes the audit with finite-sample support?

The submission did not demonstrate one. The new post-review study completes the audit at the original targets: α=0.10\alpha=0.10, β=0.50\beta=0.50, and δ=0.05\delta=0.05. Its protocol was documented locally rather than publicly preregistered. Solver configuration used separate development checks and a 60-task design set excluded from the reported analysis.

Llama-3.2-3B-Instruct generates one natural GSM8K answer per task, evaluated with our existing forward, critique, and backward GPT-5.4 judge prompts. The remaining 1,259 tasks have fixed, disjoint task-ID-based roles:

StageTasksResult
PILOT / Gate 1380Margin +1.56, interval [0.53, 2.12]; FEASIBLE
FIT190Freeze the rule requiring at least two of three PASS votes
CAL430350 accepted, 8 errors; one-sided 95% risk upper bound 4.1%; observed TPR 98.3%
TEST259189 accepted, 8 errors; 4.2% accepted error, 73.0% coverage, 94.3% observed TPR

Each task contributes one answer. Under the independent-task sampling assumptions, CAL supplies the finite-sample certificate for the frozen rule's population accepted risk. TEST separately measures its behavior; its TPR and coverage are observed quantities. Both exact Clopper–Pearson and the original Hoeffding test support certification. Direct fixed-sequence LTT, using the same total label budget and CAL/TEST sets, selects the same rule on this split.

We also examined how often certification succeeds across splits. In 1,000 additional FIT/CAL repartitions of the same sizes, the frozen-rule procedure certifies in 41% of runs. FIT often selects a looser rule whose full-cohort empirical risk is 9.4%, leaving little room for CAL to certify a risk below 10%. The completed audit provides the requested positive case; the repeated splits characterize the separate issue of certification power.

Reviewer_VG2vOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper points out that LLM verifiers are usually evaluated with metrics such as accuracy, balanced accuracy, and AUROC, which do not answer the deployment question: among the accepted outputs, what fraction is wrong? The authors call this accepted risk. It should depend on the solver's base error rate together with the verifier's TPR and FPR. They propose VFA, a two-gate audit framework. Gate 1 asks whether the verifier-selection procedure supplies enough held-out evidence for a given solver and risk target. Gate 2 fixes one acceptance rule and asks which kind of risk statement (empirical, large-sample, or finite-sample) independent calibration data can support. Empirically, they report three findings: (1) reusing evaluation labels during selection inflates the measured evidence; (2) verifier improvements rarely change the audit outcome; and (3) calibration support shrinks as the claim gets stronger.

Critical strengths
  1. I enjoyed the problem statement and its motivation, especially the introduction's example. Passing a score threshold and meeting a risk target are different things, and the 2% vs. 64% example shows this clearly.
  2. I agree with the authors that deploying a verifier requires understanding its evaluation protocol, configuration, and data size. This matters more as graders and verifiers are built into large-scale online systems, so the problem is timely and important. This paper's contribution makes sense and clear to me.
  3. The math that derives accepted risk, evidence demand, and supply in the Verification Squeeze is clear. The experiments cover many benchmarks, and the two audit case studies add useful support.
Critical weaknesses
  1. I enjoyed the problem setup and the need it addresses for verifier deployment, but the paper is still hard to follow, especially Sections 3.1 and 3.2. Many technical terms are introduced together, and the concrete procedures in FIT, CAL, and TEST are confusing me.
  2. The main text relies heavily on the appendix, often after nearly every paragraph, so the reader has to jump back and forth. I encourage the authors to organize the links between the main paper and the appendix better and to reduce the amount of them.
  3. The role of Gate 2 and its input and output are still unclear to me, especially the three steps FIT, CAL, and TEST. How do they operate together? Algorithm 1 is still hard to follow.
  4. The label-reuse experiment needs more motivation. I agree that reusing evaluation labels during selection inflates the measured evidence, but is this a standard practice?
Questions for authors
  1. I am still confused by the definition of l and u in lines 228 and 229. I think a better introduction for them will be better.
  2. Gate 1 studies whether the evidence is sufficient to support a claim. How are the pilot and reserved data split, and how was this done in the experiments?
  3. The unit of evaluation is not stated clearly. Are results computed on final answers or on independent trajectories?
4 comments
AuthorsOpenReview ↗

Response to Reviewer VG2v (Part 1/2)

Thank you for identifying where the explanation became hard to follow. We will restructure Section 3 to show what the audit does before introducing its statistical quantities, using our new natural-output experiment as a running example. We will upload the revised PDF shortly; the answers below describe each change.

W1. Sections 3.1–3.2 introduce too many terms at once

We will explain the decisions first and introduce each quantity where it is used. We agree that the current order asks readers to learn the notation before knowing what it is for.

Gate 1 asks whether the verifier-selection procedure supplies enough held-out evidence for the risk requirement. Gate 2 asks whether independent calibration supports one fixed acceptance rule. The sequence is: PILOT evaluates the selection procedure; FIT freezes a rule; CAL tests its risk; TEST measures its behavior.

Section 3 and Algorithm 1 will start with this sequence. We will then introduce supply, demand, and the margin when explaining Gate 1, followed by the calibration bound when explaining Gate 2. This also makes clear why different folds may select different rules while the final deployed rule remains fixed.

W2. The reader has to keep returning to the appendix

The main text should contain every step needed to understand the audit's decision. We agree that the present placement of those explanations makes readers reconstruct the procedure.

We will put the data-flow description before Algorithm 1, define the interval endpoints beside the Gate-1 decision rule, and include the new experiment's stage-by-stage results in the main text. The experimental setup will distinguish the separate component studies from complete two-gate runs. Implementation conventions, extra rotations, and extended sensitivity tables will remain in the appendix. We will also consolidate appendix citations rather than interrupting each explanation with another reference.

W3. What are the inputs and outputs of FIT, CAL, and TEST?

FIT chooses the rule, CAL decides whether its risk is supported, and TEST evaluates the unchanged rule. Each uses a separate set of parent tasks.

The new GSM8K experiment makes this concrete. Llama-3.2-3B-Instruct produces one answer per task, and three GPT-5.4 judge prompts vote PASS or FAIL. We retain the original 10% accepted-risk target and 50% correct-answer retention floor.

StageOperationOutput
FIT: 190 tasksChoose the highest-coverage threshold meeting the empirical FIT risk and TPR requirementsFreeze:accept if at least two of three votes are PASS
CAL: 430 tasksApply the frozen rule to independent labeled answers350 accepted, 8 errors; one-sided 95% risk upper bound 4.1%; certified below 10%
TEST: 259 tasksEvaluate the same rule without changing it189 accepted, 8 errors; 4.2% accepted error, 73.0% coverage, 94.3% observed TPR

The purpose of CAL is to check the rule without reusing the labels that made it look best on FIT. CAL does not search for another threshold, and TEST does not decide whether certification passes. The certificate concerns population accepted risk; TPR and coverage above are observed performance measures.

Accept-all has 25.9% test error. The selected rule therefore retains nearly three-quarters of answers while substantially lowering their error rate. We will use this example to annotate Algorithm 1's inputs and outputs.

AuthorsOpenReview ↗

Response to Reviewer VG2v (Part 2/2)

W4. Does evaluation-label reuse occur in practice?

Yes. Table 16 gives concrete examples of evaluation-dependent configuration choices. The Auto-J leaderboard searches tie thresholds for reward-model baselines; SFR-Judge reports the better result from two response orders on non-RewardBench pairwise evaluations; LEXam compares ensembles on the same 50 expert-labeled examples used to report agreement. The practices differ, but each makes evaluation results relevant to the configuration being reported.

Our controlled experiment asks whether label access changes the apparent acceptance decision. Holding the other protocol choices fixed, it produces a 0.124 macro-median margin difference and 3/40 point-margin crossings. In those three comparisons, the measured evidence changes sides of the requirement boundary; these are changes in apparent feasibility, not certified deployments.

The broader held-out study reports a 1.9× median reduction in supply, but also changes other protocol choices. We will keep the two results distinct: the controlled study isolates label access, while the broader study compares complete evaluation procedures.

Q1. What are ℓ and u?

They are the lower and upper endpoints of Gate 1's margin interval: the 5th and 95th percentiles of 2,000 selection-aware bootstrap estimates, forming a nominal central 90% band.

If ℓ>0\ell>0, Gate 1 returns FEASIBLE; if u<0u<0, audited-infeasible; otherwise, UNCERTAIN. For example, the new GSM8K pilot has margin +1.56 and interval [0.53, 2.12], so it passes Gate 1. Independent CAL then tests the final rule's risk. We will define both endpoints immediately before the decision rule.

Q2. How are PILOT and the reserved data split?

Before either gate is evaluated, a deterministic task-ID partition divides the labeled parent tasks into a pilot set PP and a disjoint reserved set CC. Gate 1 uses only PP, rotating five task-grouped folds (3 FIT / 1 SEL / 1 TEST) inside it. Gate 2 uses CC, divided into FIT, CAL, and TEST, and is reached only if Gate 1 returns FEASIBLE. All records from one parent task stay in the same set.

Two experiments use this full split:

RunTasksPP (Gate 1)CC = FIT + CAL + TESTGate-1 outcome
Submitted IFEval walkthrough534115419 = 205 + 102 + 112MEASURE-MORE at α=0.10\alpha=0.10; Gate 2 not reached
New natural-output GSM8K1,259380879 = 190 + 430 + 259FEASIBLE; Gate 2 certifies

The 115 IFEval pilot tasks yield 107 scored records: 90 correct and 17 incorrect. GSM8K first excludes a separate 60-task design set.

The other experiments are component studies and reserve no pilot. The Gate-1 and scaling studies rotate 3 FIT / 1 SEL / 1 TEST folds over the whole task population; the Gate-2 study uses 2 FIT / 2 CAL / 1 TEST folds. They evaluate each gate separately and do not form a sequential certificate. We will move this table and distinction from Appendix B.4 into the main experimental setup.

Q3. Are decisions evaluated on final answers or trajectories?

Most decisions concern an answer, plan, command, or individual claim—not a full agent trajectory. AppWorld is the trajectory-like exception: its verifier input is an API-call trace without application-return information.

The acceptance unit and sampling unit can differ. Several claims may come from one task, so all related records stay together during splitting and task-level resampling. In the new GSM8K experiment, the units coincide: one task, one natural final answer, one acceptance decision. We will state both units for each benchmark in the experimental setup.

Thank you for the rebuttal. The clarifications on FIT, CAL, and TEST are helpful, and I appreciate the GSM8K example, so I am raising my score to 2. However, the framework still reads as a complex assembly of existing components (a known Bayes identity, cross-fitting, bootstrap screening, and standard risk-control tests). The presentation also remains hard to follow, with many named outcomes, newly introduced protocols, and frequent references to the appendix. I do not think the paper reaches the bar for acceptance at this conference.

Thank you for the follow-up and for acknowledging the improvements. We would like to clarify the two remaining concerns.

Contribution. We acknowledge that the statistical components are established. Our contribution concerns the evaluation protocol and the empirical questions it addresses: how much verifier improvements close the evidence gap to a specified risk target, and what independent calibration supports for a fixed rule. The magnitude of those improvements and the remaining deficits are empirical findings; they do not follow from the Bayes identity alone. We would therefore appreciate a more specific assessment of what the protocol and findings add beyond the established ingredients.

Could you clarify whether the concern is that prior work already covers this contribution, or that the additional findings are insufficiently informative? If it is overlap, identifying the closest study and the specific overlap would help us respond directly.

Presentation. Within the page limit, we have placed the core workflow, decision rules, and worked example in the main text, while keeping detailed implementation conventions and extended analyses in the appendix. Section 3 and Tables R1–R2 now explain the data roles and distinguish component studies from complete audits. We take the remaining readability concern seriously.

Could you identify one or two steps or named outcomes that still make the explanation difficult to follow? This would help us make a focused correction.

Reviewer_eFAGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies when a PASS from an LLM verifier can be trusted, measured by accepted risk, the fraction of accepted outputs that are wrong. It rewrites Racc≤αR_{acc} \le \alpha as a condition that TPR/FPR exceeds a demand set by the solver's base error, and builds a two-gate audit called VFA on it. Gate 1 compares the cross-fitted evidence of a verifier-selection procedure with this demand, and Gate 2 assesses one fixed rule on calibration data. Experiments with five LLM judges on about 15 benchmarks report that label reuse inflates evidence, verifier improvements rarely change the audit status, and finite-sample methods support none of the rules.

Critical strengths

Pros

  1. The problem is practical, since what matters when a PASS triggers an action is the error rate among accepted outputs rather than accuracy.
  2. The controlled study in Appendix C.2 changes only access to evaluation labels, which isolates the effect of label reuse cleanly.
Critical weaknesses

Cons

  1. The paper offers no theoretical guarantee, although VFA is presented as an audit for deployment decisions. There is no theorem, the Gate 1 bands are nominal bootstrap percentiles, and the five Gate 2 passes rest on a large-sample screen whose simulated coverage falls to 0.856. No guarantee is stated for the full sequential audit, and Appendix B notes that the component results cannot be combined into one. So neither a bound nor a simulation shows how often VFA approves a rule whose true accepted risk exceeds α\alpha.

  2. The target is a false discovery rate, but the paper does not use that framework. Accepted risk is the positive FDR of the acceptance rule, and Eq. 1 is its Bayesian form [1] rewritten as a likelihood ratio. Up to the add-one correction, the Gate 1 margin is positive exactly when the held-out false discovery proportion is below α\alpha, so the supply and demand view adds nothing beyond that proportion. Conformal selection [2, 3] and Learn-then-Test give finite-sample guarantees for this target, yet none of them is compared, and [2] is mentioned only once in Appendix G.

  3. The role of Gate 1 is unclear. Section 3.2 states that the two gates evaluate different objects and that a Gate 1 pass does not predict or guarantee the rule Gate 2 selects. Gate 1 pools rules that may differ across folds and are chosen by likelihood ratio, while Gate 2 picks a new rule by FIT coverage, so a Gate 1 pass says nothing about the deployed rule. Learn-then-Test already answers the feasibility question with a guarantee by returning an empty set when no threshold meets the target. In the only full run on IFEval in Appendix F.1, reserving a 115-task pilot moves the margin from +1.21 in Table 9 to -0.03, so Gate 1 costs labels and resolution without adding information.

  4. The selection rule in Gate 1 is odd. SEL picks the largest TPR/FPR among rules with TPR at least β=0.5\beta = 0.5, which never uses the risk target and reverses Gate 2, where risk is the constraint and coverage the objective. For a concave ROC curve TPR/FPR falls as the threshold loosens, so the chosen rule sits near TPR=β\mathrm{TPR} = \beta and Gate 1 tests a single operating point fixed by an unexplained β\beta. A verifier that meets α\alpha at a TPR of 0.4 is then judged infeasible. The choice is also noisy, since the 17 wrong outputs in the IFEval pilot are split over five folds and each SEL fold ranks rules by whether about three wrong outputs are accepted.

[1] The positive false discovery rate: A Bayesian interpretation and the q-value. Annals of Statistics, 2003.

[2] Selection by prediction with conformal p-values. JMLR, 2023.

[3] Conformal alignment: Knowing when to trust foundation models with guarantees. NeurIPS, 2024.

Questions for authors

Questions

  1. Table 6 names a generator only for TravelPlanner, while Appendix C.3 and the counts in Table 14 imply one correct and one wrong output per task for BFCL, GSM8K, Temporal, Complexity and MATH-500, so π=0.5\pi = 0.5 by design. The same Bash set has π=0.733\pi = 0.733 and a margin of -2.87 in Table 9 but π=0.375\pi = 0.375 and +0.59 in the scaling pool of Appendix D.1. Since the demand depends only on π\pi, were the wrong outputs sampled from a solver or constructed? If they were constructed, which deployment population do the audit statuses describe?

  2. Section 4.3 reports that no audited-infeasible baseline becomes feasible. In Figure 3, with 100 tasks Gate 1 almost never returns FEASIBLE even at a true margin of +0.75, and most sets in Table 5 have 100 tasks. What result at this sample size could have shown that a verifier improvement works?

  3. Appendix E.2 shows that Hoeffding needs at least 150 calibration tasks whatever the outcomes, and 100 of the 115 rules have fewer. For Temporal, 0 errors in 36 accepted outputs from 36 tasks give an exact binomial upper bound of about 0.08, as used for selective risk in [4], while Hoeffding-Bentkus gives p = 0.580 in Table 14. Is the negative finite-sample result in the abstract a property of the rules or of the chosen bounds?

  4. Appendix B.4 says SEL picks the largest likelihood ratio, and Appendix B.5 applies the add-one FPR only to reported measurements. When a candidate accepts no wrong output on SEL, is its ratio infinite? If so, selection reduces to a tie broken by the lowest threshold, and if not, the correction used on SEL is not described.

[4] Selective classification for deep neural networks. NeurIPS, 2017.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eFAG (Part 1/3)

Thank you for the careful review. We have completed a natural-output end-to-end experiment, rerun all 117 scaling comparisons with selection included in the bootstrap, and compared directly with Learn-then-Test (LTT). These additions address the reliability of the final approval decision, the robustness of the scaling finding, and the information provided by Gate 1.

W1. What guarantees does VFA provide?

When independent CAL uses a valid finite-sample test, VFA approves a rule whose true accepted risk exceeds α\alpha with probability at most δ\delta. We agree that the submission did not state this guarantee explicitly.

FIT freezes the rule before CAL is examined. Conditional on PILOT and FIT, both the rule and the Gate-1 continuation decision are fixed. An unsafe VFA approval therefore requires CAL to falsely certify that rule. Under the calibration procedure's sampling assumptions,

Pr⁡{VFA approves r^ and Racc(r^)>α}≤δ.\Pr\left\lbrace \text{VFA approves }\hat r\ \text{and}\ R_{\mathrm{acc}}(\hat r)>\alpha\right \rbrace\leq\delta.

Gate 1 adds a check before certification; it does not bypass CAL.

Our new experiment demonstrates a useful approval under this guarantee at the original targets: α=0.10\alpha=0.10, β=0.50\beta=0.50, and δ=0.05\delta=0.05. Llama-3.2-3B-Instruct generates natural GSM8K answers, evaluated with our existing forward, critique, and backward GPT-5.4 judge prompts. After separate development checks and exclusion of a 60-task design set, the remaining 1,259 tasks have disjoint roles:

StageTasksResult
PILOT / Gate 1380Supply 10.06 versus demand 2.10; margin +1.56, interval [0.53, 2.12]; FEASIBLE
FIT190Freeze the rule requiring at least two of three PASS votes
CAL430350 accepted, 8 errors; exact one-sided 95% risk upper bound 4.1%
TEST2594.2% accepted error, 73.0% coverage, 94.3% observed TPR

Accept-all has 25.9% test error. Thus, the certified rule provides useful selective acceptance without relaxing the target. The mathematical argument establishes the error-control guarantee; the experiment demonstrates that the complete procedure can produce a nontrivial approval. The original asymptotic screen is separate: its reported 0.856 coverage does not support a finite-sample 95% claim.

W2. Is the demand–supply view just a restatement of FDR?

The identity is standard; our finding is how little verifier scaling closes the required evidence gap. We agree that Equation (1) is Bayes' rule and that a fixed rule's margin carries the same information as its accepted risk. The decomposition makes the required and achieved improvements directly comparable.

In the submission, the 36 audited-infeasible baseline comparisons have a median deficit of 1.97 log units, or approximately 7.2× in LR. After full reselection, the original 12 positive-gain comparisons close a median of 19% of their respective submitted log-evidence deficits, with a one-sided 95% upper bound of 25% on that median. This is the measured squeeze: the typical gain remains small relative to the requirement in those comparisons.

The decomposition also separates two sources of change. On the same population, ΔM=log⁡(Snew/Sbase)\Delta M=\log(S_{\mathrm{new}}/S_{\mathrm{base}}) measures the verifier's evidence gain. Holding TPR and FPR fixed instead, changing solver error prevalence from 5% to 90% raises demand by 171×. Table 9 expresses the supported prevalence as πmax⁡=αS/(1−α+αS)\pi_{\max}=\alpha S/(1-\alpha+\alpha S).

We have also added the requested LTT comparison. With the same natural GSM8K answers, total label budget, and CAL/TEST sets, fixed-sequence LTT certifies four of ten candidates and selects the same two-of-three-votes rule as VFA. Both achieve 4.2% test error at 73.0% coverage on this split.

We will add Storey (2003), conformal selection, and Conformal Alignment to the main-text discussion. Their pFDR and batch-FDR formulations are closely related; the fixed-rule certification here controls the probability of approving a rule whose population accepted risk exceeds the target.

AuthorsOpenReview ↗

Response to Reviewer eFAG (Part 2/3)

W3. What does Gate 1 add beyond Learn-then-Test?

An empty LTT result leaves a practical question unanswered: should we collect more labels or improve the verifier? We tested whether Gate 1 helps distinguish these cases, using 200 task-level runs per population–budget setting. Total label budgets are matched: serial VFA allocates 50%/25%/25% to PILOT/FIT/CAL, while direct LTT allocates 50%/50% to development/CAL.

SettingLTT outcomeGate-1 diagnosis
Bash, Regex, Complexity; budgets 80–210No certified rulesSTOP in 98.5–100% of runs
Natural GSM8K; budgets 80–150Frequently no certified ruleUNCERTAIN in 95–100% of LTT-empty cases

On the first three cohorts, the lowest empirical accepted risks among the tested full-cohort rules are 40.5–62.5%, far above the 10% target. On GSM8K, increasing the budget to 250, 400, and 800 raises LTT certification to 86%, 99%, and 100%. The first result supports improving the tested verifiers; the second supports collecting more evidence. Gate 1 supplies information for this choice that an empty certified set alone does not provide.

Different rules across Gate-1 folds are expected: each fold evaluates the selection procedure on labels it did not use to make that choice. Gate 2 separately fixes the deployment rule on FIT. The earlier IFEval comparison used 505 records versus a 107-record pilot containing only 17 errors; it was not a matched estimate of Gate 1's incremental value.

The new budget study measures the cost as well. At budget 1,259, direct LTT certifies in 100% of runs versus 34% for serial VFA. The latter reserves labels for the pilot and may freeze a high-coverage rule near the risk boundary. These results favor direct LTT for certification power in this feasible cohort and establish Gate 1's separate contribution to diagnosing failed certification.

W4. Why maximize TPR/FPR rather than optimize coverage under the risk target?

Maximizing LR asks how low the accepted risk can be while retaining at least β\beta of correct answers. Coverage optimization is a subsequent choice among eligible rules. For a fixed population,

Racc=[1+LR+1−ππ]−1.R_{\mathrm{acc}}=\left[1+\mathrm{LR}^{+}\frac{1-\pi}{\pi}\right]^{-1}.

Higher LR therefore means lower accepted risk. For the same finite family of population rules, an (α,β)(\alpha,\beta)-feasible rule exists exactly when

\geq \frac{\pi(1-\alpha)}{(1-\pi)\alpha}.$$ **Alpha is not omitted:** it sets the demand that the best eligible LR must reach. This population relationship motivates the selector. Gate 1 then evaluates that selection procedure on held-out data; it does not directly observe the family's population optimum. Gate 2 selects the highest-coverage eligible rule on FIT, and CAL assesses it without retuning. A rule retaining only 40% of correct answers does not meet a requested $\beta=0.50$ floor. For a concave ROC curve, choosing near that floor is consistent with seeking the lowest risk allowed by the retention requirement. The submitted sensitivity analysis reports feasible/infeasible/uncertain counts of **3/8/1, 3/9/0, and 3/9/0** at $\beta=0.3,0.5,0.7$. We agree that few incorrect examples make selection unstable. The selection-aware bootstrap repeats fitting and selection where applicable, rather than treating the winning rule as fixed, so the analysis includes this source of variability.
AuthorsOpenReview ↗

Response to Reviewer eFAG (Part 3/3)

Q1. Were the wrong outputs constructed? Why do the Bash numbers differ?

Yes. We deliberately use constructed correct–incorrect pairs in the atom-level GSM8K, MATH-500, Temporal, BFCL, and Complexity benchmarks to evaluate verifier improvements under controlled conditions. This design has two advantages. First, it provides enough incorrect examples to reliably estimate TPR and FPR, even when a solver rarely makes mistakes. Second, all verifiers evaluate the same candidates with the same error prevalence, allowing us to isolate improvements in verifier evidence rather than differences in the evaluated population.

The resulting audit statuses apply to these benchmark-defined claim populations, with error prevalence π=0.5\pi=0.5, not to the natural outputs of an unspecified solver. This distinction matters because deployment feasibility depends on both verifier performance and solver error prevalence. Our controlled comparisons measure verifier improvements at fixed prevalence; evaluating a specific deployed solver requires its natural error distribution.

The Bash numbers differ because they evaluate different acceptance units (Appendix D.1):

Acceptance unitError prevalenceDemand (α=0.10\alpha=0.10)
Whole command73.3%24.75
Individual atom37.5%5.39

A command is incorrect if any required atom fails. Its higher error prevalence therefore leads to approximately 4.6× greater evidence demand. These results describe different decision populations, not conflicting estimates.

We additionally conducted a complete audit on natural solver outputs to establish deployment relevance. Llama-3.2-3B-Instruct generates one GSM8K answer per task, without constructed errors or correctness-based sampling. Across 1,259 tasks, Gate 1 finds sufficient evidence, and independent CAL certifies the selected rule with a 4.1% risk upper bound. On untouched TEST data, the rule accepts 73.0% of answers with 4.2% error, compared with 25.9% error for accept-all.

The two evaluations are complementary: controlled pairs isolate verifier improvements, while natural solver outputs test whether those improvements support a useful deployment decision.

Q2. What improvement could a 100-task experiment actually detect?

The experiments detect improvements, but 100 tasks may not resolve feasibility near the boundary. A gain must close the baseline deficit to cross zero, and a positive lower margin bound is required for FEASIBLE.

We reanalyzed all 117 comparisons using 2,000 selection-aware bootstrap replicates. On the original 12 positive-gain comparisons, the median gain closes 19% of the submitted deficit (one-sided 95% upper bound: 25%); eight of these 12 are SWE-bench-Verified comparisons. Among the 36 original deficit comparisons, 35 improved methods still have upper margin bounds below zero.

Positive results also survive: six comparisons have positive lower gain bounds, three Complexity cases become FEASIBLE, and TravelPlanner reaches +0.27 but remains UNCERTAIN. The finding concerns insufficient gains, not an inability to detect improvement.

Q3. Are the finite-sample failures caused by the rules or the bounds?

The original failures reflect the tested bounds and sample sizes, not proof that the rules are unsafe. Our Hoeffding test cannot certify with fewer than 150 independent tasks even under ideal observations; 100/115 configurations fall below this threshold.

The reviewer's Temporal calculation is correct at the record level: 36 accepts with zero errors give a CP upper bound of 0.080. It is a valid certificate only if those records are independent, which related claims from one task may not be.

Our new one-answer-per-task experiment achieves certification under both methods: CAL accepts 350 answers with 8 errors, giving a CP upper bound of 0.041; the original Hoeffding test also passes (upper endpoint −0.004). We will clarify the scope of the earlier negative result.

Q4. Is the SEL likelihood ratio infinite when FP=0?

No. SEL already uses the add-one correction

=\widehat{\mathrm{TPR}}_{\mathrm{SEL}} \frac{n_{\mathrm{wrong}}+1}{\mathrm{FP}+1},$$ When FP=0, the denominator is one, so the score remains finite. Among zero-FP candidates evaluated on the same SEL data, higher TPR gives a higher score; tie-breaking applies only when scores are equal. We will state this existing implementation detail explicitly in Appendix B.4. ### References discussed 1. Storey (2003). *The positive false discovery rate: A Bayesian interpretation and the q-value*. 2. Jin and Candès (2023). *Selection by Prediction with Conformal p-values*. 3. Gui, Jin, and Ren (2024). *Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees*. 4. Angelopoulos et al. *Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control*.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We uploaded a revised PDF addressing the concerns raised in our responses to all three reviewers. All additions below are empirical or presentational and use the same targets (α=0.10, β=0.50) and candidate families as the original 117 comparisons. Below is what changed and where, mapped to each reviewer's specific points.

New results

  1. Finite-sample guarantee (Proposition 1, §3.2). We prove Pr{VFA approves r̂ and R_acc(r̂)>α} ≤ η when CAL uses a valid level-η test. Proof in App. B.6. (eFAG W1)

  2. Selection-aware scaling reanalysis is now the primary §4.3 result (Table 1, R3, R4; App. D.5). All 117 comparisons were rerun with method and threshold selection repeated inside a 2,000-replicate bootstrap, rather than held fixed as in the original submission (that original fold-conditional analysis is retained as a labeled comparison in App. D.1–D.4, so both views remain available). Updated headline numbers: 36/36 audited-infeasible baselines and 35/36 improved configurations keep an upper margin bound below zero; the median gain on the original 12 positive-gain comparisons closes 19% of the baseline deficit (one-sided 95% upper bound 25%, vs. 12.5% under the original fixed-rule analysis). (cE1j W1, Q1; eFAG Q2)

  3. Complete end-to-end audit on natural solver outputs (§4.5, Table R2; App. F.3). Llama-3.2-3B-Instruct answers 1,259 natural GSM8K tasks (60 additional tasks held out for design checks). Gate 1 returns FEASIBLE; the frozen two-of-three-vote rule gets a one-sided 95% Clopper–Pearson bound of 4.1% on CAL, also certified by the original Hoeffding test; TEST shows 73.0% coverage at 4.2% error vs. 25.9% for accepting all. This is the complete, finite-sample-supported positive case neither the IFEval nor AppWorld study provided. This study was run after the initial reviews and is disclosed as not publicly preregistered. (cE1j W2(a), Q2; eFAG Q3)

  4. Direct comparison with Learn-then-Test (LTT) (§4.5; App. F.3, Table R5). On the same labels, direct fixed-sequence LTT selects the identical rule. A matched-budget study (no TEST split) shows LTT certifies more often than serial VFA (100% vs. 34% at 1,259 labels); a separate 1,000-repartition study shows serial VFA certifies in only 41% of runs, because FIT's coverage-maximizing choice often lands near the risk boundary. When LTT returns an empty set, Gate 1 distinguishes an infeasible verifier (STOP, 98.5–100% on Bash/Regex/Complexity) from insufficient data (UNCERTAIN, 95–100% on small-budget GSM8K, resolving as budget grows). (eFAG W2, W3)

Restructured and clarified presentation

  1. Section 3 now opens with the data flow (a rule's definition, the pilot/reserved split, what FIT/CAL/TEST each do) before supply, demand, and the margin, using the GSM8K rule as a running example throughout; ℓ and u are now defined immediately before the Gate-1 decision rule. (VG2v W1, W2, Q1)

  2. Data roles are now a main-text table (Table R1, §4.1), distinguishing full two-gate audits (IFEval walkthrough, natural GSM8K) from component studies that evaluate one gate on rotating folds. Acceptance and sampling units are stated explicitly in §4.1 and per-benchmark in Table 5. (VG2v Q2, Q3)

  3. Candidate construction is now explicit (§4.1; App. B.1): atom-level sets (GSM8K, MATH-500, Temporal, BFCL, Complexity, SWE-bench-Verified) pair constructed correct/incorrect outputs at π=0.5 by design; task-level pools add 45/39/39 synthetic negatives to 100 natural solver outputs for GSM8K/Temporal/BFCL respectively. (eFAG Q1)

  4. The SEL selection rule is now written out (App. B.4): LR⁺_SEL = TPR_SEL·(n_wrong+1)/(FP+1), confirming it stays finite even at zero false positives. (eFAG Q4)

  5. Related work (§2.3) now relates accepted risk to the positive false discovery rate (Storey, 2003), conformal selection and Conformal Alignment (Jin & Candès, 2023; Gui et al., 2024), and positions Learn-then-Test as a calibration procedure VFA can use, with Gate 1 as a separate diagnostic step. (eFAG W2)

  6. Scope of the finite-sample negative result (§4.4; App. E.2) is now explicit: it is scoped to the tested Hoeffding bound and the available sample sizes, not a claim that the rules are unsafe — resolved by the §4.5 case, which certifies under both exact Clopper–Pearson and the same Hoeffding test. (eFAG Q3)

We believe these changes directly resolve the concrete, falsifiable concerns raised by all three reviewers. We would welcome the opportunity to discuss any remaining concerns, and respectfully ask reviewers to reconsider their scores in light of these additions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.