- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PriceCheck, a selective answering framework that controls the error rate of LLM responses and maximize the coverage rate at the same time. PriceCheck first estimates the completeness, leak, and cost of different verification checks and predicts the coverage, risk and cost of different schedules. PriceCheck then selects a schedule with higher coverage under a given selective-risk target. Experiments on math reasoning tasks show higher coverage and satisfied selective risk than baselines, demonstrating the effectiveness of the proposed method.
Critical strengths
- The paper addresses a practically important problem. It goes beyond conventional verifier ranking by considering selective answering under explicit risk control. The proposed framework also provides a systematic way to combine different verification checks.
Critical weaknesses
-
The role of pricing is not sufficiently justified. The paper motivates PriceCheck by emphasizing differences in verification effectiveness and computational cost across checks. However, it remains unclear how the estimated prices, especially computational cost, guide schedule design and contribute to the final performance improvements.
-
The main experiments are conducted on MATH-500, where the generators already achieve approximately 95.7% accuracy, which may not be representative of more challenging settings. Experiments on additional datasets only evaluate price estimation and coverage prediction rather than the complete risk-controlled selection framework. It remains unclear whether the proposed method generalizes to tasks with varying difficulty levels and higher error rates.
Questions for authors
As stated in weaknesses:
-
Could the authors clarify or empirically show how the estimated prices guide schedule design, particularly computational cost?
-
Could the authors evaluate the framework against the main baselines on more challenging datasets with higher base error rates?
11 comments
Response to Reviewer 8CVh (2/6)
W1, Q1. How computational cost guides the schedules.
However, it remains unclear how the estimated prices, especially computational cost, guide schedule design and contribute to the final performance improvements.
Could the authors clarify or empirically show how the estimated prices guide schedule design, particularly computational cost?
Answer. Computational cost guides both early stopping and the choice of certified schedule, and on the harder pool the cost-minimising selector keeps default coverage for a third of the tokens.
A price consists of two fitted rates and a unit cost, and the unit cost is the computational part. It sets the predicted cost of every schedule, the race's saving over the fixed vote and the tokens the cost selector compares. The fitted rates set the predicted coverage and the price order of the tests.
Early stopping. The race recursion predicts the coverage, risk and cost of a vote that stops as soon as its outcome is settled, before the vote runs. Race (2,3) serves the same answers as a fixed vote needing two agreements among the first four draws, and race (3,3) those of three among five, so each race and its fixed vote have identical coverage and risk.
To compare the race with the fixed vote that serves the same answers, we ran a new comparison replaying the fixed vote on the race's re-solve draws.
Table R2. Splits whose maximum-coverage selection at the 1.5% target is a race, against the fixed vote that serves the same answers. Coverage and risk are on the test side and are identical for both rules. Tokens are generated tokens and seconds are estimated accelerator time, both for each answer (Appendix A.7). SHO is source-held-out and QD is question-disjoint.
| Split | Race | Fixed vote | Cov. (%) | Risk (%) | Tokens, race | Tokens, fixed | Sec., race | Sec., fixed |
|---|---|---|---|---|---|---|---|---|
| SHO F0 | (2,3) | 2 of 4 | 78.96 | 0.27 | 13,605 | 21,470 | 20.60 | 32.52 |
| SHO ForwardPref | (2,3) | 2 of 4 | 80.69 | 0.53 | 13,145 | 21,265 | 19.91 | 32.21 |
| SHO STaR | (2,3) | 2 of 4 | 80.56 | 0.54 | 12,927 | 21,075 | 19.58 | 31.92 |
| SHO SelfDistill | (2,3) | 2 of 4 | 79.01 | 0.81 | 12,929 | 20,747 | 19.58 | 31.42 |
| SHO iter2 | (2,3) | 2 of 4 | 79.96 | 0.53 | 14,524 | 23,108 | 22.00 | 35.00 |
| QD D | (2,3) | 2 of 4 | 80.93 | 0.75 | 13,896 | 21,760 | 21.05 | 32.95 |
| QD F | (2,3) | 2 of 4 | 81.06 | 1.06 | 14,184 | 22,681 | 21.48 | 34.35 |
| QD I | (3,3) | 3 of 5 | 79.56 | 0.54 | 17,305 | 26,000 | 26.21 | 39.38 |
| Mean, 8 splits | (2,3) or (3,3) | 2 of 4 or 3 of 5 | 80.09 | 0.63 | 14,064 | 22,263 | 21.30 | 33.72 |
The race serves the same answers as the fixed vote for 37% fewer tokens. The fixed vote costs 1.58 times as much as the race for the same answers (Appendix C.3). The submitted Table 19 also compares the race, listed there as the 8B adaptive vote, with two votes that stop early without a price.
| Whole MATH-500 pool (Table 19) | Generated tokens for each answer | Coverage | Risk |
|---|---|---|---|
| Race, the 8B adaptive vote | 13,427 | 79.8% | 0.54% |
| Adaptive consistency | 18,873 | 80.3% | 0.60% |
| Early-stopping consistency | 22,252 | 80.2% | 0.54% |
The two votes that stop early without a price generate more tokens at about the same coverage and risk, and neither carries a risk guarantee (Appendix B.3).
W1, Q1. Which certified rule to deploy, continued.
how the estimated prices, especially computational cost, guide schedule design
Which rule to deploy. The cost-minimising selector of Section 3.3 takes the certified schedule with the fewest parameter-weighted tokens among those with calibration coverage of at least 0.60. On three of five source-held-out folds it selects a small-first cascade.
| Small-first cascade on the same folds | Result |
|---|---|
| Mean coverage | 77.7% |
| Parameter-weighted cost relative to adaptive vote | under a third |
| Estimated accelerator seconds for each answer, cascade | 13.75 |
| Estimated accelerator seconds for each answer, adaptive vote | 20.39 |
The cost-minimising selector reduces computational cost while retaining coverage (Section 4.1, Appendix C.1).
| Harder pool at 10%, all ten halves | Cost-minimising selection | Maximum-coverage selection |
|---|---|---|
| Coverage | 70.62% | 70.46% |
| Pooled test risk | 4.61% | 3.62% |
| Generated tokens for each answer | 6,979 | 20,921 |
| Parameter-weighted tokens for each answer | 1,466 | 11,513 |
| Estimated accelerator seconds for each answer | 5.85 | 23.63 |
Choosing by cost keeps the coverage of the default for a third of its generated tokens, an eighth of its parameter-weighted tokens and a quarter of its estimated seconds, with test risk still below the target.
Computational cost determines both when voting stops and which certified rule is deployed, saving tokens at equal coverage.
Revised PDF. Appendix I.2.
Response to Reviewer 8CVh (3/6)
W1, Q1. Pricing contributions to final performance.
how the estimated prices, especially computational cost, guide schedule design and contribute to the final performance improvements.
Fitted prices also guide schedule design and certified deployment, and the paper already compares the same votes without a schedule. The certified race produces Table 2 at the 1.5% target on eight of fifteen splits, so its saving applies to the main result (Table 9).
Table 10 certifies the same 8B vote bank without a schedule, as the agreement fraction of three 8B re-solves.
| MATH-500 (Table 10) | PriceCheck | Agreement of three 8B re-solves, no schedule |
|---|---|---|
| Generated tokens for each answer | 15,028 | 16,152 |
| Splits certified at 0.5% and 1.0%, out of 15 | 5 and 15 | 0 and 10 |
| Coverage at 1.5% | 76.09% | 72.23% |
| Wrong answers at matched coverage | 44 | 55 |
At about the same tokens the votes without a schedule certify on fewer splits, serve less and keep more wrong answers. The plain score reaches the schedule's coverage only at sixteen 8B re-solves, about 5.7 times its tokens.
Which rules are tested first. The fitted rates also order the tests, when calibration keeps one answer for each problem.
| Order of the tests over the grid, 1.5% target | Calibration cells licensed, out of 300 |
|---|---|
| Flat Bonferroni | 0 |
| Fixed sequence along a price fitted on problems held out of both sides | 23 |
| Fixed sequence along a price fitted on a different benchmark | 22 |
| Three cheapest-action-first orders, built from costs without a fitted rate | 0 |
Among these orders of the grid only the two that use fitted rates license any cell (Section 5, Appendix C.5, Figure 5). Ordering by cost alone therefore does not recover the certification gain from fitted rates in this comparison.
Prediction before running. Completeness and leak fitted on 100 labelled answers predict schedule coverage through Equations 2 and 3. The same composition uses unit costs to predict each schedule's cost on both token axes.
| Prediction diagnostic | Result |
|---|---|
| Diagnostic schedules (Table 16) | 118 |
| Coverage rank correlation | 0.97 |
| Cascades for cost prediction (Appendix E.5) | 70 |
| Mean signed relative bias, generated tokens | -3.0% |
| Mean signed relative bias, parameter-weighted tokens | -4.9% |
Fitted prices predict both coverage and cost, allowing schedules to be compared before any runs on labelled data.
Certification uses only the labelled calibration counts. A price is used before it, to fix the order of fixed-sequence tests, and after it, to choose among certified schedules, so an error in a fitted price can lower the certified coverage but not the validity of the certificate.
Prices decide which certified rule is deployed and what it costs, saving tokens at equal or higher coverage on the main result.
The paper studies how fitted prices guide schedules and deployment. This matters because the priced schedules save tokens at equal or higher coverage in the main result, as Section “Schedules over priced actions”, paragraph “Where prices are used”, explains. Fitted prices do not enter the certificate, so errors in them can lower certified coverage but leave the guarantee valid.
Revised Section 3.2 states these roles.
Response to Reviewer 8CVh (1/6)
We thank the reviewer for focusing on what pricing does and on harder settings.
Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.
Summary of the answers.