- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on a important gap in verifier evaluation. A verifier can have good accuracy or AUROC and still fail to make reliable acceptance decisions. The paper therefore studies accepted risk. It asks how often an accepted output is wrong for a particular solver population. The proposed Verification Feasibility Audit has two gates. Gate 1 checks whether a verifier-selection procedure provides enough held-out evidence for a chosen risk and retention target. Gate 2 fixes one rule and asks what separate calibration data can support. The experiments cover label reuse, verifier selection, aggregation, routing, and several calibration procedures.
Critical strengths
-
It does not stop at verifier accuracy. It asks whether accepted outputs meet a target risk for the tested solver population. The likelihood-ratio margin also shows how far a method remains from that target.
-
Gate 1 evaluates the search procedure using held-out predictions. Gate 2 then studies one frozen rule on separate calibration data. The paper is also careful to separate empirical success, a large-sample screen, and a finite-sample guarantee.
Critical weaknesses
- The main 117-comparison scaling result uses a weaker analysis than the formal Gate 1. The selected rule is held fixed inside each fold, and the final benchmark label is produced from fold-level statuses. Selection is not repeated inside this bootstrap. Therefore, the result is a useful scaling summary, but it is not a full population-level Gate-1 comparison.
- The complete two-gate process is only partly shown. At the primary IFEval target, Gate 1 remains uncertain and Gate 2 is not reached. The full pipeline is run only after the risk target is relaxed from 0.10 to 0.20, and the finite-sample methods still give no support. The framework is well motivated, but its end-to-end success is not yet shown.Also Eq. (1) is a rearrangement of standard Bayes, which the paper itself concedes. The label-reuse effect is small (1.13×) and its direction follows by construction.
Questions for authors
- Do the main conclusions from the 117 scaling comparisons still hold if each comparison is rerun using the full selection-aware Gate-1 bootstrap?
- Is there any pre-specified setting in the current experiments where the complete audit reaches finite-sample-supported?
2 comments
Response to Reviewer cE1j (Part 2/2)
Q1. Do the conclusions hold after the full selection-aware reanalysis?
Yes. The estimated improvements increase, but most remain insufficient to close the measured evidence deficit. We retain the original 117 comparisons and the original 12-positive-gain subset, so the update does not select a new set of favorable comparisons.
| Quantity | Selection-aware result |
|---|---|
| Median updated gain / corresponding submitted deficit, on the original 12-positive-gain comparisons | 19.0%, versus 12.5% in the submission |
| One-sided 95% bootstrap upper bound on that median | 25.0% |
| Original deficit baselines whose margin upper bound remains below zero | 36/36 |
| Corresponding improved methods whose upper bound remains below zero | 35/36 |
The 19% statistic uses each comparison's submitted log-evidence deficit as its denominator. The 25% bound applies to the median over that original subset, not to every individual improvement.
The analysis also retains positive results. All three Complexity UNCERTAIN-to-FEASIBLE transitions survive. TravelPlanner rationale features reach a point margin of +0.27, but the interval remains UNCERTAIN. There are no strict audited-infeasible-to-FEASIBLE transitions.
Individual labels do change: baseline classifications and improved-method classifications each remain unchanged in 104/117 cases. These are separate counts, not the same 104 unchanged pairs. We will update the full transition tables. Alternative treatments of degenerate bootstrap replicates leave the median-gain upper bound at approximately 25%.
The conclusion now rests on measured gains and their uncertainty, not just on the absence of status crossings. The audit detects improvements while showing that they usually leave a substantial evidence shortfall in the evaluated deficit settings.
Q2. Is there a pre-specified setting that completes the audit with finite-sample support?
The submission did not demonstrate one. The new post-review study completes the audit at the original targets: , , and . Its protocol was documented locally rather than publicly preregistered. Solver configuration used separate development checks and a 60-task design set excluded from the reported analysis.
Llama-3.2-3B-Instruct generates one natural GSM8K answer per task, evaluated with our existing forward, critique, and backward GPT-5.4 judge prompts. The remaining 1,259 tasks have fixed, disjoint task-ID-based roles:
| Stage | Tasks | Result |
|---|---|---|
| PILOT / Gate 1 | 380 | Margin +1.56, interval [0.53, 2.12]; FEASIBLE |
| FIT | 190 | Freeze the rule requiring at least two of three PASS votes |
| CAL | 430 | 350 accepted, 8 errors; one-sided 95% risk upper bound 4.1%; observed TPR 98.3% |
| TEST | 259 | 189 accepted, 8 errors; 4.2% accepted error, 73.0% coverage, 94.3% observed TPR |
Each task contributes one answer. Under the independent-task sampling assumptions, CAL supplies the finite-sample certificate for the frozen rule's population accepted risk. TEST separately measures its behavior; its TPR and coverage are observed quantities. Both exact Clopper–Pearson and the original Hoeffding test support certification. Direct fixed-sequence LTT, using the same total label budget and CAL/TEST sets, selects the same rule on this split.
We also examined how often certification succeeds across splits. In 1,000 additional FIT/CAL repartitions of the same sizes, the frozen-rule procedure certifies in 41% of runs. FIT often selects a looser rule whose full-cohort empirical risk is 9.4%, leaving little room for CAL to certify a risk below 10%. The completed audit provides the requested positive case; the repeated splits characterize the separate issue of certification power.
Response to Reviewer cE1j (Part 1/2)
Thank you for identifying two gaps: the scaling bootstrap held the selected rules fixed, and no complete audit produced a finite-sample-supported positive result. We have addressed both. We reran all 117 comparisons with selection repeated inside the bootstrap; the estimated gains grow, but the evidence gap remains. A new natural-output GSM8K study at the original 10% risk target certifies a rule that accepts 73.0% of test answers with 4.2% error.
W1. The scaling bootstrap holds the selected rule fixed
We have rerun all 117 comparisons with method and threshold selection repeated inside the bootstrap. You are right that the submitted analysis held those choices fixed and therefore measured uncertainty conditional on the selected rules.
The new analysis uses 2,000 paired parent-task bootstrap replicates. Each replicate repeats score fitting where applicable, baseline/method selection, and threshold selection before pooling the held-out decisions. The baseline and improved method use the same resampled tasks, so their difference is evaluated as a paired comparison. For externally supplied scores, the scores remain fixed and the thresholds are reselected.
Accounting for selection increases the estimated gains but preserves the main evidence-gap finding. We will make this selection-aware analysis the primary result in Section 4.3, update the affected classifications, and retain the submitted estimates as a clearly labeled conditional comparison.
W2(a). The complete two-gate process has not succeeded
The new experiment completes both gates at the original 10% target and certifies a rule that accepts 73.0% of test answers with 4.2% error. This supplies the nontrivial positive case missing from the submitted IFEval walkthrough.
The study uses 1,259 naturally generated GSM8K answers, with separate tasks for PILOT, FIT, CAL, and TEST. Gate 1 returns FEASIBLE with a margin interval of [0.53, 2.12]. FIT then freezes the rule requiring at least two of three judges to vote PASS. Independent CAL gives this rule an exact one-sided 95% risk upper bound of 4.1%, below the 10% target.
Accepting every test answer would incur 25.9% error, so accept-all cannot meet the same risk target. The original Hoeffding test also certifies the rule on the same CAL data. The successful audit therefore does not depend on replacing that test with a sharper bound.
W2(b). Equation (1) is Bayes' rule, and label reuse has an expected effect
The contribution is the measured shortfall between verifier progress and the improvement required for reliable acceptance. We agree that the Bayesian identity is standard. It gives us a common scale on which to compare the required and achieved improvements.
In the submitted Table 13, the 36 audited-infeasible baseline comparisons have a median deficit of 1.97 log units, requiring approximately 7.2× stronger likelihood-ratio evidence. The selection-aware reanalysis shows that the typical gain in the original positive-gain subset remains far smaller than its corresponding deficit. This quantifies the squeeze rather than merely restating the risk condition.
The direction of selection bias is also expected; its effect on the acceptance assessment must still be measured. In the controlled label-access experiment, the macro-median margin difference is 0.124, and 3 of 40 point estimates cross the feasibility boundary. In those comparisons, access to evaluation labels changes whether the verifier appears to meet the evidence requirement. These are reversals of apparent feasibility, not certified deployment decisions. We will foreground these measured consequences in the contribution statement.