acceptodds
Under review as a conference paper at ICLR 2027

Risk-Controlled Selective LLM Answering by Pricing Label-Free Checks

Abstract

Serving an answer from a large language model requires deciding when to abstain, yet a verifier's ranking accuracy alone does not determine the error rate among served answers. We introduce PriceCheck, which builds a compact family of decision rules from label-free checks such as re-solving a problem. Each check has a price: its agreement rates on correct and incorrect answers and its cost per run. Prices fitted on a small, class-enriched labelled set compose into predictions of a schedule's coverage and cost, guiding which checks to run and when to stop. A calibration test then selects a schedule at a stated selective-risk target. In mathematics, the selected schedules serve 76.1% of answers on average and keep held-out selective risk below 1.5% on all 15 splits. Under the shared testing protocol, PriceCheck serves more answers at that target than reward models, a prompted judge, the generator's confidence and a trained correctness classifier. At matched coverage, it keeps the fewest wrong answers among these scorers. Across 118 diagnostic schedules, price-based coverage predictions have a rank correlation of 0.97 with observed coverage. These results show that choosing how checks are combined and stopped matters alongside how well a verifier ranks answers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
selective prediction, distribution-free risk control, Learn-then-Test, answer verification, large language models, abstention, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8CVhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PriceCheck, a selective answering framework that controls the error rate of LLM responses and maximize the coverage rate at the same time. PriceCheck first estimates the completeness, leak, and cost of different verification checks and predicts the coverage, risk and cost of different schedules. PriceCheck then selects a schedule with higher coverage under a given selective-risk target. Experiments on math reasoning tasks show higher coverage and satisfied selective risk than baselines, demonstrating the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses a practically important problem. It goes beyond conventional verifier ranking by considering selective answering under explicit risk control. The proposed framework also provides a systematic way to combine different verification checks.
Critical weaknesses
  1. The role of pricing is not sufficiently justified. The paper motivates PriceCheck by emphasizing differences in verification effectiveness and computational cost across checks. However, it remains unclear how the estimated prices, especially computational cost, guide schedule design and contribute to the final performance improvements.

  2. The main experiments are conducted on MATH-500, where the generators already achieve approximately 95.7% accuracy, which may not be representative of more challenging settings. Experiments on additional datasets only evaluate price estimation and coverage prediction rather than the complete risk-controlled selection framework. It remains unclear whether the proposed method generalizes to tasks with varying difficulty levels and higher error rates.

Questions for authors

As stated in weaknesses:

  1. Could the authors clarify or empirically show how the estimated prices guide schedule design, particularly computational cost?

  2. Could the authors evaluate the framework against the main baselines on more challenging datasets with higher base error rates?

11 comments
AuthorsOpenReview ↗

Response to Reviewer 8CVh (1/6)

We thank the reviewer for focusing on what pricing does and on harder settings.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Summary of the answers.

  1. How computational cost guides schedules (Weakness 1, Question 1). Computational cost guides early stopping and certified-schedule selection, the two uses measured here. The race serves the same answers as the fixed vote for 37% fewer tokens, and the cost-minimising selector retains default coverage on the harder pool for a third of its generated tokens. Both uses therefore lower cost at equal coverage (Table R2, Appendix I.2).
  2. Contribution to final performance (Weakness 1, Question 1). The race is the certified selection that produces Table 2 on eight of fifteen splits, and against the same 8B votes without a schedule, PriceCheck certifies on more splits and serves more at about the same tokens. The scheduling gains therefore apply to the main certified result (Table 9, Table 10).
  3. Fitted-rate contributions (Weakness 1, Question 1). A price order of the tests licenses calibration cells where orders without a fitted rate license none, and the rates predict each schedule's coverage before it runs. Fitted rates therefore guide test ordering and let schedules be compared before execution (Appendix C.5, Table 16).
  4. Full-pipeline generalization at higher error (Weakness 2, Question 2). The complete pipeline now runs against every main baseline at base errors up to 58.67%, and PriceCheck alone certifies at 5% on the harder pool and at 10% on the question-disjoint halves of three competition benchmarks. The evidence therefore establishes full-pipeline generalization at higher error, beyond price estimation alone (Tables R1 and R5).
AuthorsOpenReview ↗

Response to Reviewer 8CVh (2/6)

W1, Q1. How computational cost guides the schedules.

However, it remains unclear how the estimated prices, especially computational cost, guide schedule design and contribute to the final performance improvements.

Could the authors clarify or empirically show how the estimated prices guide schedule design, particularly computational cost?

Answer. Computational cost guides both early stopping and the choice of certified schedule, and on the harder pool the cost-minimising selector keeps default coverage for a third of the tokens.

A price consists of two fitted rates and a unit cost, and the unit cost is the computational part. It sets the predicted cost of every schedule, the race's saving over the fixed vote and the tokens the cost selector compares. The fitted rates set the predicted coverage and the price order of the tests.

Early stopping. The race recursion predicts the coverage, risk and cost of a vote that stops as soon as its outcome is settled, before the vote runs. Race (2,3) serves the same answers as a fixed vote needing two agreements among the first four draws, and race (3,3) those of three among five, so each race and its fixed vote have identical coverage and risk.

To compare the race with the fixed vote that serves the same answers, we ran a new comparison replaying the fixed vote on the race's re-solve draws.

Table R2. Splits whose maximum-coverage selection at the 1.5% target is a race, against the fixed vote that serves the same answers. Coverage and risk are on the test side and are identical for both rules. Tokens are generated tokens and seconds are estimated accelerator time, both for each answer (Appendix A.7). SHO is source-held-out and QD is question-disjoint.

SplitRaceFixed voteCov. (%)Risk (%)Tokens, raceTokens, fixedSec., raceSec., fixed
SHO F0(2,3)2 of 478.960.2713,60521,47020.6032.52
SHO ForwardPref(2,3)2 of 480.690.5313,14521,26519.9132.21
SHO STaR(2,3)2 of 480.560.5412,92721,07519.5831.92
SHO SelfDistill(2,3)2 of 479.010.8112,92920,74719.5831.42
SHO iter2(2,3)2 of 479.960.5314,52423,10822.0035.00
QD D(2,3)2 of 480.930.7513,89621,76021.0532.95
QD F(2,3)2 of 481.061.0614,18422,68121.4834.35
QD I(3,3)3 of 579.560.5417,30526,00026.2139.38
Mean, 8 splits(2,3) or (3,3)2 of 4 or 3 of 580.090.6314,06422,26321.3033.72

The race serves the same answers as the fixed vote for 37% fewer tokens. The fixed vote costs 1.58 times as much as the race for the same answers (Appendix C.3). The submitted Table 19 also compares the race, listed there as the 8B adaptive vote, with two votes that stop early without a price.

Whole MATH-500 pool (Table 19)Generated tokens for each answerCoverageRisk
Race, the 8B adaptive vote13,42779.8%0.54%
Adaptive consistency18,87380.3%0.60%
Early-stopping consistency22,25280.2%0.54%

The two votes that stop early without a price generate more tokens at about the same coverage and risk, and neither carries a risk guarantee (Appendix B.3).


W1, Q1. Which certified rule to deploy, continued.

how the estimated prices, especially computational cost, guide schedule design

Which rule to deploy. The cost-minimising selector of Section 3.3 takes the certified schedule with the fewest parameter-weighted tokens among those with calibration coverage of at least 0.60. On three of five source-held-out folds it selects a small-first cascade.

Small-first cascade on the same foldsResult
Mean coverage77.7%
Parameter-weighted cost relative to adaptive voteunder a third
Estimated accelerator seconds for each answer, cascade13.75
Estimated accelerator seconds for each answer, adaptive vote20.39

The cost-minimising selector reduces computational cost while retaining coverage (Section 4.1, Appendix C.1).

Harder pool at 10%, all ten halvesCost-minimising selectionMaximum-coverage selection
Coverage70.62%70.46%
Pooled test risk4.61%3.62%
Generated tokens for each answer6,97920,921
Parameter-weighted tokens for each answer1,46611,513
Estimated accelerator seconds for each answer5.8523.63

Choosing by cost keeps the coverage of the default for a third of its generated tokens, an eighth of its parameter-weighted tokens and a quarter of its estimated seconds, with test risk still below the target.

Computational cost determines both when voting stops and which certified rule is deployed, saving tokens at equal coverage.

Revised PDF. Appendix I.2.

AuthorsOpenReview ↗

Response to Reviewer 8CVh (3/6)

W1, Q1. Pricing contributions to final performance.

how the estimated prices, especially computational cost, guide schedule design and contribute to the final performance improvements.

Fitted prices also guide schedule design and certified deployment, and the paper already compares the same votes without a schedule. The certified race produces Table 2 at the 1.5% target on eight of fifteen splits, so its saving applies to the main result (Table 9).

Table 10 certifies the same 8B vote bank without a schedule, as the agreement fraction of three 8B re-solves.

MATH-500 (Table 10)PriceCheckAgreement of three 8B re-solves, no schedule
Generated tokens for each answer15,02816,152
Splits certified at 0.5% and 1.0%, out of 155 and 150 and 10
Coverage at 1.5%76.09%72.23%
Wrong answers at matched coverage4455

At about the same tokens the votes without a schedule certify on fewer splits, serve less and keep more wrong answers. The plain score reaches the schedule's coverage only at sixteen 8B re-solves, about 5.7 times its tokens.

Which rules are tested first. The fitted rates also order the tests, when calibration keeps one answer for each problem.

Order of the tests over the grid, 1.5% targetCalibration cells licensed, out of 300
Flat Bonferroni0
Fixed sequence along a price fitted on problems held out of both sides23
Fixed sequence along a price fitted on a different benchmark22
Three cheapest-action-first orders, built from costs without a fitted rate0

Among these orders of the grid only the two that use fitted rates license any cell (Section 5, Appendix C.5, Figure 5). Ordering by cost alone therefore does not recover the certification gain from fitted rates in this comparison.

Prediction before running. Completeness and leak fitted on 100 labelled answers predict schedule coverage through Equations 2 and 3. The same composition uses unit costs to predict each schedule's cost on both token axes.

Prediction diagnosticResult
Diagnostic schedules (Table 16)118
Coverage rank correlation0.97
Cascades for cost prediction (Appendix E.5)70
Mean signed relative bias, generated tokens-3.0%
Mean signed relative bias, parameter-weighted tokens-4.9%

Fitted prices predict both coverage and cost, allowing schedules to be compared before any runs on labelled data.

Certification uses only the labelled calibration counts. A price is used before it, to fix the order of fixed-sequence tests, and after it, to choose among certified schedules, so an error in a fitted price can lower the certified coverage but not the validity of the certificate.

Prices decide which certified rule is deployed and what it costs, saving tokens at equal or higher coverage on the main result.

The paper studies how fitted prices guide schedules and deployment. This matters because the priced schedules save tokens at equal or higher coverage in the main result, as Section “Schedules over priced actions”, paragraph “Where prices are used”, explains. Fitted prices do not enter the certificate, so errors in them can lower certified coverage but leave the guarantee valid.

Revised Section 3.2 states these roles.

Reviewer_hFxjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Verifier ranking quality alone does not pin down an LLM answering system's selective risk, and the paper asks how to combine and stop label-free checks so as to maximize coverage under a distribution-free risk target. PriceCheck estimates each check's agreement rates on correct and incorrect answers along with its cost, folds them into a fixed family of schedules with predicted coverage and cost, and certifies one schedule using a Bonferroni-corrected binomial Learn-then-Test procedure. On a mathematical-answering benchmark, it consistently certifies the proposed schedules, provides higher coverage while retaining fewer wrong answers than the evaluated alternatives, and predicts schedule coverage accurately; its advantage is especially apparent under stricter risk targets.

Critical strengths
  1. The label-free scheduling perspective is original and offers a useful bridge between verification signals, costs, and statistical certification.
  2. The experimental comparison is thoughtfully designed and makes a visible effort to give competing methods comparable design freedom.
Critical weaknesses
  1. The strongest empirical claims are established in a narrow mathematical-answering setting, leaving cross-domain usefulness uncertain.
  2. The risk guarantee depends on an independence assumption that does not fully match the clustered structure of the evaluation data.
  3. The method's many design and selection choices add complexity and make the practical operating point less straightforward to interpret.
  4. The paper reports ICC 0.45 and 4.74 answers per problem. It supplies problem-count bounds and beta-binomial simulations in Table 12, but states outright that these do not establish a general clustered-data guarantee. The headline all-split risk statement is therefore an answer-level nominal guarantee, not a fully certified statement for the actual clustered evaluation.
  5. The main study is one competition-math family with low base error, one 8B backbone, and five fine-tuned generators in Section 4. Transfer evidence is limited: leaks rise sharply on a harder Minerva pool in Table 15, and the cross-lineage experiment is a single 280-answer Llama pool. Generalization to higher-error or non-math domains is therefore not established.
Questions for authors

See weakness section

10 comments
AuthorsOpenReview ↗

Response to Reviewer hFxj (1/7)

We thank the reviewer for the close reading, for valuing the label-free scheduling perspective, and for noting that the comparison gives competing methods comparable design freedom.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Summary of the answers.

  1. Generality across domains and error rates (Weaknesses 1 and 5). The complete pipeline now runs on harder mathematics with generators from outside the paper's backbone and on programs, with PriceCheck alone certifying the harder pool at the 5% target and AIME at 10%. Its certified rules also keep test risk below target on answers from a 72B generator. Certified coverage therefore extends beyond low-error mathematics and the paper's backbone (Table R1, Table R5 and Table R6, with programs in Table R4).
  2. Guarantee for clustered answers (Weaknesses 2 and 4). A new problem-level certificate is valid under any dependence among answers to one problem, and PriceCheck certifies through it on MATH-500 and a clustered pool of 8426 answers. The risk statement therefore covers the actual clustered evaluation without requiring independence within each problem (Proposition D.1, Table R3).
  3. Practical operating point (Weakness 3). One fixed default runs the whole procedure on the main comparison and both new pools without re-tuning any constant. This fixes the design and selection choices and leaves only the risk target to the user (new Algorithm 1, Table 2, Table R1).

W1, W5. The results hold beyond MATH-500.

The strongest empirical claims are established in a narrow mathematical-answering setting, leaving cross-domain usefulness uncertain.

Generalization to higher-error or non-math domains is therefore not established.

Answer. The results now hold beyond the narrow mathematical setting, at higher error, with generators from outside the paper's backbone, and on programs.

To test whether results generalize to higher-error settings, we ran new experiments on GSM8K and the harder mathematical pool. The harder pool holds 2096 answers to MATH test problems at Levels 4 and 5 outside MATH-500, one for each problem, from DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct and the paper's iter2. Every baseline of Table 2 and the p(True) and log-probability scorers run on it and on GSM8K, and the 23-rule grid, the test of Equation 4 and every constant carry over unchanged from MATH-500.

Limit in the reviewSubmissionRevision
Error rate4.26% base error on MATH-50028.82% on a harder pool, 58.67% on AIME
Model familiesthe paper's 8B backbone and one 280-answer Llama poolfour generators, three from outside the backbone, with a held-out-generator test
Domainmathematicsalso HumanEval and MBPP programs

The revision now tests every limit identified by the reviewer.

Table R1. The complete pipeline on two new pools. Each cell gives the certifying splits and the percent coverage over all splits, a split that does not certify counting as zero, as in Table 2. GSM8K holds 1318 answers at 9.33% base error with ten question-disjoint halves. The harder pool has ten question-disjoint halves and four held-out-generator folds. A dagger marks a single aggregation at 24 levels, in each column the one that certifies the most coverage, which favours the rival.

MethodGSM8K, 5%, halvesHarder pool, 5%, halves5%, held-out generator10%, halves10%, held-out generator
PriceCheck10 / 90.396 / 40.814 / 66.8210 / 70.464 / 68.48
Judge10 / 75.010 / 00 / 010 / 64.234 / 56.89
Judge†10 / 76.100 / 00 / 010 / 68.264 / 68.06
PRM10 / 83.780 / 00 / 01 / 4.922 / 22.15
PRM†10 / 86.840 / 00 / 00 / 02 / 23.84
Math-Shepherd10 / 78.360 / 00 / 00 / 00 / 0
Math-Shepherd†10 / 82.660 / 00 / 00 / 00 / 0
ORM10 / 83.980 / 00 / 00 / 00 / 0
ORM†10 / 85.420 / 00 / 00 / 00 / 0
p(True)0 / 00 / 00 / 00 / 00 / 0
p(True)†0 / 00 / 00 / 00 / 00 / 0
Log-prob0 / 00 / 00 / 00 / 00 / 0
Log-prob†0 / 00 / 00 / 00 / 00 / 0

PriceCheck alone certifies the harder pool at the stricter target, on six halves and every held-out generator.

Harder pool at the 5% targetResult
Certified halves, test coverage68.02%
Certified halves, pooled test risk3.41%
Held-out generators, pooled test risk3.25%

Certification on the harder pool also yields low realised test risk.

AuthorsOpenReview ↗

Response to Reviewer hFxj (2/7)

W1, W5. Test risk and wrong answers on the new pools, continued.

Generalization to higher-error or non-math domains is therefore not established.

Harder pool at 10%, question-disjoint halvesCoverage over all tenPooled test risk
PriceCheck70.46%3.62%
Judge†68.26%6.68%
Judge64.23%6.91%

Where Judge and Judge† also certify, PriceCheck serves the most test answers at about half their test risk.

Wrong answers kept at matched coverage, summed over the halves where PriceCheck certifies, each scorer ranked by its aggregation of highest AUC on the pool, which favours the rival.

ScorerGSM8K, 5%Harder pool, 10%
PriceCheck125267
Judge301542
PRM2281038
Math-Shepherd2671383
ORM1831447
p(True)4361647
Log-prob5592090

At matched coverage it keeps about half as many wrong answers as Judge, the next best scorer, on the harder pool. It keeps fewer than every other scorer on GSM8K.

GSM8K at the 5% targetResult
Pooled test risk2.10%

PriceCheck also serves more than every baseline on GSM8K.

The complete pipeline retains its advantage on harder mathematics and on a different mathematical benchmark.

Revised PDF. new Section 4.5 and new Appendix I.2.


W1, W5. Wider baseline levels, continued.

leaks rise sharply on a harder Minerva pool in Table 15

Generalization to higher-error or non-math domains is therefore not established.

The comparison holds with wider baseline levels, and higher fitted leak affects certified coverage without changing certificate validity.

With their abstention extended to 90%, no baseline certifies at 5% on any half or held-out generator of the harder pool, and at 10% PriceCheck keeps the largest coverage under both level sets.

Coverage at the 10% targetPriceCheckJudge†, levels of Table R1Judge†, levels extended to 90% abstention
Question-disjoint halves70.46%68.26%68.14%
Held-out generators68.48%68.06%66.60%

Each held-out-generator fold certifies on the answers of three generators and tests on the fourth, so these folds test models absent from calibration.

PriceCheck certifies on every held-out generator of the harder pool and of AIME. On the held-out generators at 10%, Judge† comes close to PriceCheck's coverage, but its realised test risk exceeds the target on two of the four. The certificate uses labelled calibration counts, not fitted leaks, so checks with higher leak, as on the Minerva pool of Table 15, can lower certified coverage but not validity.

The certificate does not depend on MATH-500's low error rate or on the paper's backbone.

When wrong answers become frequent, the baselines' scores no longer separate enough correct answers from wrong ones to certify at 5%, while PriceCheck's checks still do.

Revised PDF. Appendix I.2.

AuthorsOpenReview ↗

Response to Reviewer hFxj (3/7)

W1, W5. A 72B generator, continued.

Generalization to higher-error or non-math domains is therefore not established.

To test whether the certified rules carry over to a much larger generator, we ran a new experiment that serves the rules certified in Table R1 unchanged on answers written by Qwen2.5-72B-Instruct, nine times the size of the pool's 8B generators. The calibration halves, the grid, the certified rules and every baseline family are those of Table R1, and nothing is fit on the 72B answers. Its 2089 answers to pool problems carry a base error of 26.33%, and each half tests on the 1042 to 1045 of them that answer its test-side problems.

Table R6. Rules certified on the calibration halves of the harder pool in Table R1, served unchanged on Qwen2.5-72B-Instruct answers to the test-side problems of the same ten question-disjoint halves. Each cell gives the certifying halves and the percent coverage over all ten, a half that does not certify counting as zero. Pooled test risk is over the certifying halves.

Method5%, halves10%, halvesPooled test risk, 5%Pooled test risk, 10%
PriceCheck6 / 42.7510 / 73.212.67%3.09%
Judge0 / 010 / 68.08does not certify7.08%
Judge†0 / 010 / 72.04does not certify7.13%
PRM0 / 01 / 6.80does not certify8.03%
PRM†0 / 00 / 0does not certifydoes not certify
Math-Shepherd, ORM, p(True), Log-prob, each with and without †0 / 00 / 0do not certifydo not certify

On the 72B answers, PriceCheck's test risk stays below the target on every half where its rules are certified, and at 10% it is less than half of Judge†'s while PriceCheck serves the most answers. At 5% only PriceCheck has certified rules to serve.

Scorer, 72B answersWrong answers at matched coverage, 5%Wrong answers at matched coverage, 10%
PriceCheck119236
Judge303584
PRM424781
ORM6691183
Math-Shepherd8591503
Log-prob9101522
p(True)10941829

At matched coverage PriceCheck keeps fewer than half of Judge's wrong answers at both targets.

The rules certified on 7B and 8B generators carry over to a 72B generator, with test risk below target and the lead over every baseline intact.

Revised PDF. New Appendix I.2.


W1, W5. The pipeline also runs on programs.

Generalization to higher-error or non-math domains is therefore not established.

Outside mathematics, the pipeline from fitted prices to the certificate runs on HumanEval and MBPP programs, and PriceCheck certifies on more halves than the Judge with nearly twice its coverage.

The paper's five generators write the HumanEval and MBPP answers again, and the full pool of 1459 answers that close their thinking block are scored with the seven grid members that three backward probes and three re-solves evaluate exactly. The one-answer pool keeps one answer for each of 293 problems, at 8.19% base error, where serving every answer certifies on no half.

To test whether the certificate holds outside mathematics, we ran the complete pipeline and the baselines on HumanEval and MBPP programs.

Table R4. Selective risk at a 10% target on 293 HumanEval and MBPP problems with one pre-assigned answer each, base error 8.19%, on ten question-disjoint halves. A dagger marks the family of 24 thresholds.

MethodHalves certifiedCoverage (%)Test risk (%)Exceedances
PriceCheck1077.002.890
Judge540.694.770
Judge†322.005.330
p(True)317.115.650
p(True)†212.826.560
Program pools, 10% targetPriceCheckJudge
Full pool, problem-level certificate, halves certified10 of 109 of 10
Full pool, wrong answers at PriceCheck's coverage over the ten test halves176215

On programs PriceCheck certifies where serving every answer does not, with nearly twice the Judge's coverage and lower test risk. We also compare wrong answers at PriceCheck's coverage and the accuracy of coverage predictions.

Program diagnosticResult
Full pool, p(True) wrong answers at PriceCheck coverage357
Full pool, Log-prob wrong answers at PriceCheck coverage382
Full pool, coverage-prediction mean absolute error3.61 points
MATH-500 comparison, mean absolute errors (Tables 14 and 16)3.64 and 3.86 points

At matched coverage, PriceCheck keeps fewer wrong programs than both probability scorers, so error filtering also transfers beyond mathematics. Coverage-prediction error on programs is comparable to that on mathematics.

The certificate and the price predictions transfer from mathematics to program synthesis.

Revised PDF. new Appendix I.3, Table R4.

Reviewer_dkaSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PriceCheck, a framework for risk-controlled selective LLM answering. It estimates the reliability and cost of label-free verification checks, combines them into adaptive schedules, and uses statistical testing to certify selective risk. Experiments on mathematical reasoning tasks show improved answer coverage while maintaining low error rates.

Critical strengths

Pricecheck inverstigates how to combine verification checks to maximize coverage under selective-risk constraints and compare multiple verification baselines and examine different risk levels, calibration protocols, and sample dependencies and also achieves higher certified coverage than competing methods at strict risk targets.

Critical weaknesses
  1. The certification relies on independent calibration samples, but several answers come from the same problem (ICC = 0.45). The authors examine this issue through simulations, though a formal guarantee for correlated samples is still missing.
  2. The certification results are less convincing under the nested protocol. While Table 2 reports 76.09% coverage, Table 3b shows that only 10 out of 20 runs are certified, bringing the average coverage down to 38.17% when unsuccessful runs are included. 3 The evaluation is fairly limited, with 500 math problems, five models based on the same 8B backbone, and only 99 incorrect answers. It would be useful to see whether the results hold for other model families and tasks.
Questions for authors
  1. How can the 1.5% risk guarantee hold when answers from the same problem are correlated? Is there a formal guarantee beyond the simulation results?
  2. Why does the nested protocol certify only 10/20 runs, compared with 15/15 in Table 2? How should we interpret the drop in average coverage from 76.09% to 38.17%?
  3. Since Table 9 mainly selects re-solve voting strategies, how much improvement actually comes from the pricing mechanism? Could the authors compare against similar strategies without pricing?
10 comments
AuthorsOpenReview ↗

Response to Reviewer dkaS (1/7)

We thank the reviewer for the precise questions on dependence, the nested protocol and the role of pricing.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Summary of the answers.

  1. Guarantee under correlated answers (Weakness 1, Question 1). Yes, new Proposition D.1 formally certifies selective risk under any dependence among answers to one problem, and through it PriceCheck certifies MATH-500 at a 2% target and a clustered pool of 8426 answers at 10% with the most coverage of any method (Table R3).
  2. Lower nested-protocol coverage (Weakness 2, Question 2). With fewer calibration answers, more nested repeats abstain and count as zero in the all-repeat mean, while certified repeats serve what Table 2 serves (Table 3b, Appendix C.4).
  3. Generality across models and tasks (Weakness 3). On new pools from three generators outside the paper's backbone, on three competition benchmarks, on programs and on answers from a 72B generator, PriceCheck keeps the fewest wrong answers at matched coverage (Table R1, Table R4, Table R5 and Table R6).
  4. Contribution beyond unpriced voting (Question 3). The same re-solve votes certified without a schedule certify on fewer splits at about the same tokens, and the priced race serves the same answers as the fixed vote with 37% fewer tokens (Table 10, Table R2).

W1, Q1. A formal guarantee for correlated answers.

The authors examine this issue through simulations, though a formal guarantee for correlated samples is still missing.

How can the 1.5% risk guarantee hold when answers from the same problem are correlated? Is there a formal guarantee beyond the simulation results?

Answer. Yes, new Proposition D.1 is a formal guarantee for correlated answers to one problem, and through it PriceCheck certifies the clustered harder pool with more coverage than any baseline.

To test whether the formal guarantee certifies correlated answers, we ran a new experiment retaining all answers from each problem in the harder pool. New Proposition D.1 is a formal guarantee for pools that keep several correlated answers to one problem. It certifies the answer-weighted selective risk through the problem-level loss Li=(Ei−αNi+αM)/ML_i = (E_i - \alpha N_i + \alpha M)/M, where problem ii has NiN_i served answers, EiE_i of them wrong, and at most MM answers. Each LiL_i lies in [0,1][0,1], and its mean is at most α\alpha exactly when the answer-weighted selective risk is at most α\alpha. A betting test of that mean over i.i.d. problems is therefore valid by Ville's inequality under any dependence among the answers to one problem.

We checked the certificate in simulations at the least favourable null.

Simulation setting or outcomeValue
Intraclass correlations0.45, 0.8 or 1.0
Settings60
Nominal level5%
True-null rejection rateat most 4.9%

True-null rejection remains below the nominal level under the simulated correlations.

Table R3. The problem-level certificate of Proposition D.1 on the harder pool with all 8426 answers of its four generators, up to four for each of 2276 problems, at 28.55% base error (new Appendix D.3). Each held-out-generator fold tests on one generator after certifying on the others. Every method uses the same betting test, order and splits, PriceCheck with its fixed 23-rule grid and each baseline with thresholds at calibration quantiles as in Table 2. Cells read as in Table R1 below, and a dagger marks a single aggregation at 24 levels, in each column the one that certifies the most coverage, which favours the rival.

Method5%, halves10%, halves10%, held-out generator
PriceCheck7 / 47.3010 / 70.594 / 69.02
Judge0 / 07 / 43.063 / 40.03
Judge†0 / 08 / 53.883 / 45.24
PRM0 / 04 / 20.162 / 22.50
PRM†0 / 02 / 10.510 / 0
Math-Shepherd, ORM, p(True), Log-prob, each with and without †0 / 00 / 00 / 0

PriceCheck certifies the most coverage when answers to one problem are correlated, and uniquely certifies the stricter target. With baseline levels widened, it still certifies the most coverage in every column (Appendix D.3).

Clustered harder-pool conditionPriceCheck pooled test riskStrongest baseline pooled test risk
10%, halves3.23%6.39% (Judge†)
5%, halves2.99%none certifies
10%, held-out generators3.48%6.09% (Judge†)

Correlation among the answers to one problem does not remove the result, because under a guarantee that allows any such correlation PriceCheck still certifies with the most coverage of any method and keeps test risk below target.

Revised PDF. Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer dkaS (2/7)

W1, Q1. MATH-500 sample size and the 1.5% target.

How can the 1.5% risk guarantee hold when answers from the same problem are correlated?

On MATH-500, PriceCheck's 1.5% schedules stay below target on all 15 splits, and Proposition D.1 certifies its 2% target on all five source-held-out folds.

Equation 4 is valid under the i.i.d. condition of the risk-control methods it builds on (Bates et al., 2021, Angelopoulos et al., 2025), and every baseline is certified with the same test. For pools with several answers to one problem, the new certificate above takes the problem as the i.i.d. unit, as conformal language modeling does when it samples several outputs for each prompt (Quach et al., 2024).

Each new pool in Table R1 keeps one answer for each problem and draws its halves over problems, and the 23-rule grid and every constant carry over unchanged from MATH-500. Both conditions of Equation 4, independent calibration answers and a family fixed independently of them, therefore hold for PriceCheck on the question-disjoint halves, and the held-out-generator folds add a generator that calibration never sees.

Problem-level certification diagnosticResult
Problem orders, every fold certified under each20
Test coverage71.31%
Errors among served answersone among 1659

All checked problem orders yield certification with low realised test error.

Problems needed, or available, to certify one of 23 rules at 1.5%Problems
Needed by any valid test whose p-value rises with each problem's loss, every answer served without error406
Needed when a schedule serves 80% of each problem's answers without error507
Largest MATH-500 source-held-out fold491
MATH-500 question-disjoint half245
Harder-pool question-disjoint half1138

No MATH-500 fold or half has enough problems for a schedule that serves most answers without error, while a half of the harder pool has more than twice the problems needed.

Counting each kept problem once, the problem-count bound keeps every policy at its operating point below the trained reward model (Table 12). The beta-binomial simulations of Appendix D.1 use the same correlations as in the certificate simulations.

Problem-count result or settingValue
Trained reward model, bound at its operating point2.14%
Target for unanimous three-vote and small-first cascade1.5%
Unanimous three-vote, bound1.20%
Small-first cascade, bound1.45%
Nominal level and true-null rejection rate5% and at most 3.6%
Bonferroni level and true-null rejection rate0.22% and at most 0.12%

Both named policies stay below target and the problem-count construction rejects a true null below nominal levels.

The 1.5% result of Table 2 stands on MATH-500, and the problem-level certificate reaches 2% there because a MATH-500 fold has too few problems for 1.5%, not because of the method.

Revised PDF. new Appendix D.3.


W2, Q2. Why fewer nested repeats certify.

While Table 2 reports 76.09% coverage, Table 3b shows that only 10 out of 20 runs are certified, bringing the average coverage down to 38.17% when unsuccessful runs are included.

Why does the nested protocol certify only 10/20 runs, compared with 15/15 in Table 2? How should we interpret the drop in average coverage from 76.09% to 38.17%?

Answer. The lower all-repeat mean counts abstaining repeats as zero, while certified repeats serve what Table 2 serves.

We compare coverage among certified repeats with coverage averaged over all repeats.

Table 3b learns the family as well as certifying, and uses fewer answers than Table 2.

Nested protocol (Table 3b)PriceCheckReward-model arm
Certified repeats out of 20 at 1.5%108
Mean coverage of the certified repeats at 1.5%76.33%64.18%
Mean coverage over all 20 repeats at 1.5%, an abstaining repeat counting as zero38.17%25.67%
Certified repeats out of 20 at 1.0%50

Certified PriceCheck repeats serve about what Table 2 serves. PriceCheck also certifies more often and serves more than the reward-model arm under the same protocol. It serves more on each of the four repeats where both arms certify (Section 4.3).

Section 3.3 sets the primary target to the strictest level at which half of the nested repeats certify.

Nested-protocol diagnosticResult
Certified-repeat pooled test risk0.63%
More permissive target (Table 11)2.0%
Repeats certified there13 of 20
Coverage there78.2%

Certifying half the repeats determines the target. The abstaining repeats explain the lower all-repeat mean.

AuthorsOpenReview ↗

Response to Reviewer dkaS (3/7)

W2, Q2. Fewer calibration answers in the nested protocol, continued.

How should we interpret the drop in average coverage from 76.09% to 38.17%?

Nested protocol, Table 3bTable 2
Familylearned from 196 of the 491 problemsthe fixed 23-rule grid
Calibration side147 problems, 693.5 answers on average1150 to 1169 or 1857 to 1865 answers (Appendix A.9)

Learning the family leaves fewer answers for calibration, which makes certification harder.

The new repeat-level diagnosis in Appendix C.4 compares calibration counts with the exact test's requirements in the ten repeats that abstain.

Calibration errors of the best memberCalibration answers it servesAnswers the exact test needs
1431506
2412 to 541653 to 671
5520 to 535985 to 1006

No member of the re-selected family has zero calibration errors. In each of these repeats the certificate abstains because the best member serves fewer calibration answers than the exact test needs at its error count, not because a certified rule serves less.

The nested protocol is a stricter test that certifies on fewer answers, so it abstains more often, and when it certifies it serves what Table 2 serves.

Revised PDF. Appendix C.4.


W3. Other model families and tasks.

The evaluation is fairly limited, with 500 math problems, five models based on the same 8B backbone, and only 99 incorrect answers. It would be useful to see whether the results hold for other model families and tasks.

Answer. Yes, the results hold on other model families and tasks, and PriceCheck keeps the fewest wrong answers at matched coverage on every new pool.

To test whether results hold for other model families and tasks, we added GSM8K and the harder pool with the grid, test and constants unchanged. The harder pool uses one answer for each problem from DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct and the paper's iter2. Three of the four generators come from outside the paper's 8B backbone. Each of four folds holds one generator out.

PoolAnswers or problemsWrong answersSelection
Harder pool2096 answers604MATH test Levels 4 and 5 outside MATH-500
MATH-500 comparison2326 answers99the 500 MATH-500 problems, five generators fine-tuned from one 8B backbone
GSM8K1318 problems123GSM8K test, one answer from the paper's iter2 for each problem

The harder pool tests many more wrong answers and generators outside the original backbone.

Table R1. The complete pipeline on two new pools. Each cell gives the certifying splits and the percent coverage over all splits, a split that does not certify counting as zero, as in Table 2. GSM8K holds 1318 answers at 9.33% base error with ten question-disjoint halves. The harder pool, 2096 answers at 28.82% base error, has ten question-disjoint halves and four held-out-generator folds. A dagger marks a single aggregation at 24 levels, in each column the one that certifies the most coverage, which favours the rival.

MethodGSM8K, 5%, halvesHarder pool, 5%, halves5%, held-out generator10%, halves10%, held-out generator
PriceCheck10 / 90.396 / 40.814 / 66.8210 / 70.464 / 68.48
Judge10 / 75.010 / 00 / 010 / 64.234 / 56.89
Judge†10 / 76.100 / 00 / 010 / 68.264 / 68.06
PRM10 / 83.780 / 00 / 01 / 4.922 / 22.15
PRM†10 / 86.840 / 00 / 00 / 02 / 23.84
Math-Shepherd10 / 78.360 / 00 / 00 / 00 / 0
Math-Shepherd†10 / 82.660 / 00 / 00 / 00 / 0
ORM10 / 83.980 / 00 / 00 / 00 / 0
ORM†10 / 85.420 / 00 / 00 / 00 / 0
p(True)0 / 00 / 00 / 00 / 00 / 0
p(True)†0 / 00 / 00 / 00 / 00 / 0
Log-prob0 / 00 / 00 / 00 / 00 / 0
Log-prob†0 / 00 / 00 / 00 / 00 / 0

PriceCheck alone certifies the harder pool's stricter target, on six question-disjoint halves and every held-out generator. At the looser target, it serves the most test answers on the halves, and on GSM8K it serves more than every baseline.

Wrong answers kept at matched coverage, summed over halves where PriceCheck certifies, each scorer at its highest-AUC aggregation.

ScorerGSM8K, 5%Harder pool, 10%
PriceCheck125267
Judge301542
PRM2281038
Math-Shepherd2671383
ORM1831447
p(True)4361647
Log-prob5592090

At matched coverage PriceCheck keeps about half as many wrong answers as Judge, the next best scorer, on the harder pool. It keeps fewer than every other scorer on GSM8K.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (1/4)

General response

We thank the reviewers, who find the label-free scheduling perspective a useful bridge between verification signals, costs and statistical certification (hFxj), the framework a systematic way to combine checks (8CVh), and the certified coverage above competing methods at strict risk targets (dkaS). Each shared request now has new results in the revised PDF, summarised below.

1. Generality across error rates, models and tasks.

Answer. PriceCheck certifies on every new pool and uniquely certifies the harder pool and competition benchmarks at their stated targets.

Every pool keeps the 23-rule grid, test and constants of MATH-500 unchanged. The harder pool and competition benchmarks use four generators, three from outside the paper's 8B backbone.

Unique certificationTarget
Harder pool5%
Three competition benchmarks, question-disjoint halves10%

PriceCheck uniquely certifies the harder pool and the competition benchmarks at these targets.

PoolBase errorTargetPriceCheck, splits certified and coverageStrongest baseline
MATH Levels 4 and 5 outside MATH-500, halves28.82%5%6 of 10, 40.81%none certifies
Same pool, held-out generators28.82%5%4 of 4, 66.82%none certifies
GSM8K, halves9.33%5%10 of 10, 90.39%PRM†, 10 of 10, 86.84%
AIME, halves58.67%10%10 of 10, 35.52%none certifies
OlympiadBench, halves49.81%10%9 of 10, 38.32%none certifies
Omni-MATH, halves58.49%10%4 of 10, 10.64%none certifies
HumanEval and MBPP programs, halves8.19%10%10 of 10, 77.00%Judge, 5 of 10, 40.69%

PriceCheck certifies on every pool, and unique certification extends to pools with much higher base error. At these targets, baselines certify on low-error pools but fail on high-error pools.

Harder pool targetPriceCheck pooled test riskJudge† pooled test risk
10%3.62%6.68%
10%, the same rules served on Qwen2.5-72B-Instruct answers3.09%7.13%

On the harder pool, PriceCheck also serves more test answers at about half the test risk of Judge† (Table R1). Served unchanged on answers from Qwen2.5-72B-Instruct, nine times the size of the pool's 8B generators, the same certified rules keep this lead (Table R6). On the competition benchmarks, no baseline certifies on any half, even with abstention extended to 90%.

The certificate and PriceCheck's lead over the baselines do not depend on MATH-500's low base error, on the paper's backbone, on generator size or on mathematics.

Revised PDF. Table R5, Appendix I.2.


2. A formal guarantee for clustered answers.

Answer. New Proposition D.1 certifies selective risk under any dependence among answers to one problem, with PriceCheck achieving the most coverage.

It takes the problem as the i.i.d. unit and certifies the same answer-weighted risk through a problem-level loss.

MATH-500, problem-level certificateResult
Target2%
Source-held-out folds certifiedall five
Problem orders checked20
Test coverage71.31%
Test errors among served answersone among 1659

The certificate applies to the clustered pool under every checked problem order.

Harder pool, all 8426 answers, up to four for each problemPriceCheckStrongest baseline
10% target, question-disjoint halves10 of 10, 70.59%Judge†, 8 of 10, 53.88%
10% target, held-out generators4 of 4, 69.02%Judge†, 3 of 4, 45.24%
5% target, question-disjoint halves7 of 10, 47.30%none certifies

PriceCheck keeps its certificate and its lead when the answers to one problem are correlated. With baseline levels widened it still certifies the most coverage in every column.

The pools of Table R1 and the one-answer program pool keep one answer for each problem and a grid fixed independently of the calibration answers, so both conditions of Equation 4 hold for PriceCheck on their question-disjoint halves.

The guarantee no longer needs independence among answers to one problem.

Revised PDF. Table R3, Appendix D.3.

AuthorsOpenReview ↗

General Response (2/4)

General response, continued

3. Pricing versus the same unpriced votes.

Answer. Table 9 reports which certified schedule PriceCheck deploys on each split. It is not the comparison against the same votes without pricing, which Table 10 provides. Pricing lets the same re-solve votes serve more answers at about the same tokens, or the same answers at fewer tokens.

We compare identical re-solve votes with and without schedules, including the fixed vote that serves the same answers as each race.

Same re-solve votes, with and without pricingWith pricingWithout pricing
MATH-500 at 1.5%, coverage at about the same tokens (Table 10)76.09%72.23%
Eight race splits, generated tokens for each answer for the same answers (Table R2)14,06422,263

The priced race serves the same answers as the equivalent fixed vote with 37% fewer generated tokens. Without a schedule, the plain 8B agreement score reaches the schedule's coverage only at sixteen votes (Section 4.1, Table 10).

Prices also predict coverage before a schedule runs. The cost-minimising selector chooses by unit cost.

MeasurementResult
Unscheduled score, tokens relative to scheduleabout 5.7 times
Harder-pool coverage prediction, mean absolute error over 23 schedules on question-disjoint halves2.92 points
Cost-minimising selection, harder pool at 10%, all ten halves70.62% coverage
Generated tokens for each answer, cost-minimising selection6,979
Generated tokens for each answer, maximum coverage20,921

Choosing by cost retains coverage while reducing generated tokens. Coverage prediction also lets schedules be compared before their votes run, so pricing guides design as well as deployment.

The paper studies how to schedule the same label-free checks. This matters because pricing decides which votes run and when they stop, saving tokens at equal or higher coverage, as Section “Schedules over priced actions”, paragraph “Where prices are used”, explains.

Revised PDF. Appendix I.2.


4. The nested protocol.

Answer. The lower all-repeat mean comes from repeats that abstain, and a repeat that certifies serves what Table 2 serves.

The nested protocol averages coverage over every repeat, including those that abstain.

Nested protocol at 1.5% (Table 3b)PriceCheckReward-model arm
Certified repeats out of 20108
Mean coverage over all 20 repeats, an abstaining repeat counting as zero38.17%25.67%
Mean coverage of the certified repeats76.33%64.18%

PriceCheck certifies more repeats and serves more than the reward-model arm under the same protocol. Its certified repeats serve about what Table 2 serves.

Nested-protocol diagnosticResult
Pooled test risk of certified repeats0.63%
Calibration answers per repeat, mean693.5
Calibration answers in Table 2, minimum1150

The other ten repeats abstain because each repeat has fewer calibration answers. The nested protocol certifies fewer repeats without lowering coverage among the repeats that certify.

The nested protocol abstains more often because it certifies on fewer answers, and when it certifies it serves what Table 2 serves.

Revised PDF. Appendix C.4.

Tables R1 and R5 and the revision summary follow in the next comment.

AuthorsOpenReview ↗

General Response (3/4)

Tables R1 and R5, and revision summary

Answer. PriceCheck uniquely certifies the harder pool at the strict target and keeps fewer wrong answers at matched coverage.

To test whether the complete pipeline generalizes to higher error rates and other generators, we ran new experiments on GSM8K and the harder pool.

Table R1. The complete pipeline on two new pools, with MATH-500's grid, test and constants. Each cell gives the certifying splits and the percent coverage over all splits, a split that does not certify counting as zero, as in Table 2. GSM8K holds 1318 answers with ten question-disjoint halves. The harder pool of 2096 answers has ten question-disjoint halves and four held-out-generator folds. A dagger marks a single aggregation at 24 levels, in each column the one that certifies the most coverage, which favours the rival.

MethodGSM8K, 5%, halvesHarder pool, 5%, halves5%, held-out generator10%, halves10%, held-out generator
PriceCheck10 / 90.396 / 40.814 / 66.8210 / 70.464 / 68.48
Judge10 / 75.010 / 00 / 010 / 64.234 / 56.89
Judge†10 / 76.100 / 00 / 010 / 68.264 / 68.06
PRM10 / 83.780 / 00 / 01 / 4.922 / 22.15
PRM†10 / 86.840 / 00 / 00 / 02 / 23.84
Math-Shepherd10 / 78.360 / 00 / 00 / 00 / 0
Math-Shepherd†10 / 82.660 / 00 / 00 / 00 / 0
ORM10 / 83.980 / 00 / 00 / 00 / 0
ORM†10 / 85.420 / 00 / 00 / 00 / 0
p(True)0 / 00 / 00 / 00 / 00 / 0
p(True)†0 / 00 / 00 / 00 / 00 / 0
Log-prob0 / 00 / 00 / 00 / 00 / 0
Log-prob†0 / 00 / 00 / 00 / 00 / 0

Only PriceCheck certifies the harder pool at its strict target of 5%, and it serves more than every baseline on GSM8K.

Wrong answers kept at matched coverage, summed over the halves where PriceCheck certifies, each scorer ranked by its aggregation of highest AUC on the pool, which favours the rival.

ScorerGSM8K, 5%Harder pool, 10%
PriceCheck125267
Judge301542
PRM2281038
Math-Shepherd2671383
ORM1831447
p(True)4361647
Log-prob5592090

At matched coverage at 10% PriceCheck keeps about half as many wrong answers as Judge, the next best scorer, on the harder pool.

The complete pipeline retains its advantage at matched coverage on both new pools, answering the request for a comparison beyond the original benchmark. The revised PDF reports this comparison in the section “The complete pipeline on harder problems and further generators”.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.