acceptodds
Withdrawn from ICLR 2027

Audit-Certified Conformal Routing for Reliable Hybrid Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hybrid post-training combines scalar-reward updates with distillation, but noisy reward estimates can route examples into wrong-direction updates. We study a split-conformal sign gate with an independent audit of wrong-sign risk among routed examples. A single global conformal radius induces exactly a magnitude threshold; its contribution is a finite-sample marginal interpretation of that threshold, not a different sample ranking. Our evaluation separates calibration, audit, routing diagnosis, policy fitting, and final testing. On GSM8K, the global gate obtains accuracy, while studentized routing obtains , versus for an equally audited magnitude gate; sample-adaptive conformal variants are also evaluated on MATH, HumanEval, and two additional 3B backbones. A separate group-aware construction addresses the sign of the centered advantage used by GRPO. These guarantees concern routing on the stated distribution, not optimization gains or robustness to arbitrary policy drift.

Primary area:
reinforcement learning
Keywords:
conformal prediction, reinforcement learning, LLM post-training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_d5XROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper "Audit-Certified Conformal Routing for Reliable Hybrid Training" aims to improve post-training of LLMs through a router, that decides whether for a given training sample the model should be trained on the guidance of a reinforcement learning reward signal (e.g. GRPO) or on a distillation loss based on a larger teacher model. For this, the authors employ conformal prediction to bound the predictions of the reward model with uncertainty intervals and only employ the reward loss, when the whole interval lies above a chosen threshold τ\tau or below −τ-\tau. The evaluation covers reliability of the resulting router based on conformal predictions and final performance of the trained models on Qwen and Llama models and the GSM8K, MATH and HumanEval datasets. Across all experiments, the authors report modest performance improvements between 0.6% and 0.9%.

Critical strengths

Strenghts:

  • the paper addresses a relevant and timely problem, i.e., that unreliable rewards may reinforce mistakes of models in post training.
  • the idea of using a router with conformal predictions seems to be novel and is certainly interesting, while the statistical tools are well-established and have been extensively used in LLM contexts.
Critical weaknesses

Weaknesses

  • while the idea is interesting it is certainly not groundbreaking and many similar ideas already exist in the literature. Furthermore, it seems like the method enables only moderate improvements on small-scale models. So overall, the method and the performance improvements are largely incremental.
  • in GRPO, a sample can also get a negative reward when the reward model assigned a positive reward, but the other answers in the group get far lager rewards. Thus, it is very questionable whether the router objective connects to the actual updates done during GRPO.
  • The results are questionable, e.g., for QWEN 2.5 1.5B Instruct, the best result on MATH in the paper is 29.8% while the official model card reports 55.2% and the best result on GSM8K in the paper is 50.7% while the model card reports 73.2% (see https://qwen.ai/blog?id=qwen2.5-llm). Also, an experiment investigating the improved performance of the learned uncertainty model with and without conformal calibration to understand the contribution of each building block would be helpful.

Minor Comments

  • the paper appears to be missing a conclusions section.
  • the paper appears to be written with heavy LLM use. before publication, i would whole-heartedly suggest a rewrite to have a story and to make the paper better understandable.
Questions for authors

Questions:

  • What is the performance of the employed student checkpoints before training (since QWEN 2.5B results appear to be far better on the published model card)?
  • How can the proposed method help when GRPO is used? Isn't the absolute magnitude of the reward unimportant in this context, but the distribution of rewards within the considered group?
  • Does a gate with conformal routing outperform a gate that uses the same learned uncertainty model but without conformal calibration?
2 comments
AuthorsOpenReview ↗

Response to Reviewer d5XR – part 1

Thank you for the review. The table summarizes our answers.

Your pointOur answerWhere in the revision
Gains of 0.6–0.9% are incrementalWe agree that the accuracy gains are small. The main contribution is the certified error rate of the routed updates.Abstract; Section 7
A positive reward can give a negative GRPO advantageCorrect. The group-aware certificate covers the centered advantage and lowers wrong centered signs among routed examples from 27.0% to 8.8%.Section 4.4, Table 7
Qwen2.5-1.5B scores far below the model cardThe gap comes from our local evaluation harness. In a common harness, the same checkpoint scores 73.2% on GSM8K and 55.0% on MATH.Section 5.3, Table 5; Appendix K
Learned uncertainty with and without calibrationAdded. With the same scale model, calibration and the audit add 8 of 1,319 problems over the raw score, and calibration adds 5 over a directly audited cutoff.Section 5.2, Table 4
No conclusion sectionAdded.Section 7
Reads as heavily LLM-writtenThe main text is rewritten in plain language.Sections 1–8

How do the checkpoints score before training?

We evaluated the untouched instruction checkpoints, on the same checkpoint bytes, in our local harness and in a common benchmark harness. The Qwen2.5 report lists 73.2% on GSM8K and 55.2% on MATH for Qwen2.5-1.5B-Instruct (Table 10).

Qwen2.5-1.5B-Instruct, untouchedGSM8KMATHHumanEval
Our local harness628/1,319 (47.6%)1,370/5,000 (27.4%)102/164 (62.2%)
Common benchmark harness965/1,319 (73.2%)2,750/5,000 (55.0%)101/164 (61.6%)

The common harness reproduces the report on the math benchmarks, so the low absolute scores in our tables reflect the local harness, not a weak checkpoint. These runs do not identify which prompt, decoding-limit, or answer-extraction choice of the local harness causes the gap. Within each harness, prompts, decoding, extraction, and scoring are fixed for all methods. Under the local harness, distillation-only training scores below the untouched checkpoint, 45.0% against 47.6% on GSM8K. Distillation-only training is therefore not an initialization control, and the untouched checkpoint is the right reference. Appendix K reports all four checkpoints in both harnesses.

Under the common harness, with the same 2,000-step horizon, teacher, data, and audit-label budget, every audited router improves on the untouched checkpoint (single-checkpoint counts):

Qwen2.5-1.5B, common harnessGSM8K / 1,319MATH / 5,000HumanEval / 164
Untouched instruction checkpoint965 (73.2%)2,750 (55.0%)101 (61.6%)
Directly audited learned-scale score973 (73.8%)2,772 (55.4%)110 (67.1%)
Studentized conformal + audit977 (74.1%)2,787 (55.7%)112 (68.3%)
CQR calibration + audit976 (74.0%)2,779 (55.6%)111 (67.7%)

Studentized routing gains 12 GSM8K, 37 MATH, and 11 HumanEval problems over the untouched checkpoint. Over the directly audited score, it gains 4, 15, and 2. These are single-checkpoint counts without seed-level uncertainty, so they do not establish significance.

Please see the second part of our response for the GRPO update, the calibration ablation, the effect size, and the writing.

How does the method help GRPO?

The router decides which examples receive the GRPO update and which receive distillation. You are right that a positive raw reward can give a negative advantage after group centering, so the raw-sign certificate does not cover the GRPO update. Section 4.4 adds a certificate for the centered quantity. For the active reward-routed set AGA_G of a group, it compares

\quad\text{with}\quad A_i^\star=R_i-\frac{1}{|A_G|}\sum_{j\in A_G}R_j .$$ A group-level conformal quantile of the largest error in the group gives simultaneous intervals for all its members. A group uses the reward branch only when at least two examples remain and every interval excludes zero. Otherwise the whole group distills. Positive standard-deviation scaling preserves sign, so the certificate covers the centered and rescaled advantage that GRPO uses. The probability of any wrong certified sign in a new exchangeable group is at most the group error level. It does not certify the clipped policy gradient. | Routing rule | Routed examples | Wrong centered sign among routed | Accuracy | |---|---:|---:|---:| | Raw-sign gate | 66.3% | 27.0% | 50.0 ± 0.5 | | Group-aware whole-group certificate | 43.5% | 8.8% | 50.2 ± 0.5 | | Oracle normalized-direction gate | 61.5% | 0.0% | 52.6 ± 0.4 |
AuthorsOpenReview ↗

Response to Reviewer d5XR – part 2

The group-aware gate cuts wrong centered signs among routed examples from 27.0% to 8.8% and routes fewer examples. Its accuracy of 50.2, against 50.0, does not show an optimization gain. The oracle gate uses the true centered directions and is an upper reference.

Does the conformal gate beat the same uncertainty model without calibration?

Yes, by a small margin. Table 4 holds the learned scale model fixed and varies only calibration and the audit (single-checkpoint counts on 1,319 GSM8K problems):

Same learned scale modelCorrectFresh routed / 600Fresh wrong / routed
Uncalibrated width, no audit662 (50.2%)422 (70.3%)23.5%
Direct cutoff + audit665 (50.4%)388 (64.7%)17.3%
Studentized conformal + audit670 (50.8%)389 (64.8%)17.5%

Without calibration and the audit, the scale model routes 70.3% of examples with a 23.5% wrong-sign rate. An audited cutoff on the same score lowers the rate to 17.3%. Calibration then adds 5 problems, at a rate of 17.5%, and gives the cutoff a finite-sample interpretation. The learned quantile model shows the same pattern: 661 without calibration or audit, 666 with a direct audited offset, and 669 with CQR.

Are the gains too small to matter?

We agree that the accuracy gains are small. Adaptive routing beats an equally audited magnitude gate by 0.9 points on GSM8K, with paired 95% interval [0.3, 1.5], and by 0.4 to 0.9 points in each of six 3B backbone–task cells, where each per-cell interval excludes zero before multiplicity adjustment (Table 16). The main contribution is an explicit error budget for the reward branch. In the primary GSM8K experiment, studentized routing has a 17.5% wrong-sign rate among routed examples, against 22.3% for the audited magnitude gate and 27.9% for the uncertainty threshold, and the audit bounds the rate at 0.25 on the audit distribution. The revised abstract and Conclusion present the method this way.

Conclusion and writing

Section 7 is a new Conclusion, and Section 8 lists the limitations. The main text has been rewritten in plain language, with every term defined before use and a worked example in Section 3. AI assistance is disclosed in the paper.

Please let us know whether these results answer your questions.

Reviewer_s47JOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to route training examples between reward-based updates and teacher distillation when reward estimates are unreliable. It constructs conformal intervals around predicted rewards, routes examples to the reward branch when the interval supports a definite sign, and uses an independent audit to control conditional wrong-sign risk among routed examples.

Critical strengths

The paper provides mathematical proofs for its routing guarantees. It also includes a range of experiments across multiple tasks and model backbones, with comparisons of routing strategies, ablation studies, and evaluations under policy drift.

Critical weaknesses

The manuscript is very difficult to understand. It introduces numerous specialized terms without sufficient explanation (e.g., prompt-ID-disjoint reward-model fitting, conditional wrong-sign rate ), and the notation and variables in the equations are often insufficiently explained or contextualized.

Although the problem setup defines the reward estimator, signed target, and routing decision, it does not clearly articulate the technical challenges that make this problem nontrivial or explain why existing routing strategies are insufficient. It is also unclear how representative this setup is of hybrid reward–distillation training more broadly, particularly its reliance on trusted signed targets for calibration and auditing. As a result, the reader is left without a clear understanding of what fundamental difficulty the proposed method resolves or how broadly its formulation applies.

Questions for authors

What are the main technical challenges in the proposed problem, and why are existing magnitude- or uncertainty-based routing strategies insufficient?

Could the authors provide an intuitive, step-by-step example showing how the reward estimate, conformal interval, audit, and final routing decision fit together?

Which components constitute methodological advances over existing conformal prediction and risk-auditing techniques?

How broadly does the setup apply to hybrid reward–distillation training, particularly when trusted signed targets are unavailable or costly to obtain?

2 comments
AuthorsOpenReview ↗

Response to reviewer s47J – part 1

Thank you for the review. We agree that the submitted text was hard to read, and the main text is rewritten. The table summarizes our answers.

Your pointOur answerWhere in the revision
Undefined termsSection 2 defines routed examples, wrong-sign events, and the marginal and conditional wrong-sign rates. Section 3 defines prompt-ID-disjoint splits.Sections 2–3
The technical challenge is not statedSection 1 states it: a low average error rate can hide a high error rate among the examples the router selects.Section 1
Reliance on trusted signed targetsTrusted targets are needed only for calibration and audit, 800 to 1,750 labeled examples per task. The paper now discusses their cost and sources.Sections 7–8

The terms

A routed example is one sent to the reward branch instead of distillation. A wrong-sign event is a routed update whose direction disagrees with the trusted target. The marginal wrong-sign rate counts wrong-sign updates among all examples. The conditional wrong-sign rate counts them among routed examples only, which are the updates that training actually applies. Prompt-ID-disjoint means that no prompt appears in more than one data role (reward-model fitting, calibration, audit, routing test, policy fitting, optimization development, or final test), even through different completions.

What are the main technical challenges, and why are magnitude- or uncertainty-based routers insufficient?

There are three challenges.

  1. A large predicted reward is not necessarily a reliable update direction.
  2. A low average error rate can hide a high error rate in the selected subset. Suppose 15 of 100 examples have the wrong reward sign, a marginal rate of 15%. A router that selects exactly those 15 has a conditional rate of 100%.
  3. GRPO centers rewards within a group, so a correct raw sign can still give a wrong advantage sign.

A magnitude or uncertainty score can rank examples usefully. The score alone, however, gives no finite-sample statement about the error rate of the subset it selects. The audit supplies that statement, and it can be applied to any candidate rule. We apply it to the magnitude baseline as well. Example-adaptive widths also route better than both heuristics (Table 2). Accuracy is over ten seeds, and the routing columns come from one checkpoint:

RouterGSM8K accuracyRoutedWrong sign among routed
Uncertainty threshold, five dropout passes49.1 ± 0.666.3%27.9%
Audit-tuned magnitude49.8 ± 0.565.8%22.3%
Global conformal, identical to a magnitude cutoff50.0 ± 0.566.3%20.1%
Studentized conformal50.7 ± 0.564.8%17.5%

Studentized routing improves on the audited magnitude gate by 0.9 points, with paired 95% interval [0.3, 1.5], and on the uncertainty threshold by 1.6 points, with interval [1.0, 2.2].

A step-by-step example

Section 3 now walks through one example.

  1. Calibrate. Calibration residuals give the radius, here 0.30 at α = 0.15.
  2. Route. With margin τ = 0.10, an estimate of 0.80 gives the interval [0.50, 1.10], which lies above τ, so the example receives a positive reward update. An estimate of 0.15 gives [−0.15, 0.45], which overlaps the indifference region, so the example uses distillation.
  3. Audit. Before training, the candidate rule is checked on independent labeled examples. On GSM8K, the selected rule routes 397 audit examples, and 78 of them have the wrong sign. After correcting for three candidate margins, the upper bound on its conditional wrong-sign rate is 0.243, below the budget of 0.25.
  4. Train. Training applies the rule to new completions without their trusted targets. If every candidate fails the audit, training uses distillation only.

Please see the second part of our response for the methodological contributions and the cost of trusted targets.

Which parts are methodological advances?

Conformal calibration and binomial audits are established tools, and we do not claim them as new. The advances lie in how they are combined for training-time routing.

  1. A routing event for reward updates, with a finite-sample bound on marginal wrong-sign updates (Proposition 5).
  2. An audit over a finite family of candidate rules that bounds the wrong-sign rate among routed examples after the margin is selected (Theorem 6).
  3. A group-aware certificate for the sign of the centered advantage that GRPO uses (Proposition 7).
  4. An evaluation that separates the guarantee from the score, with an identity control, directly audited thresholds on the same learned scores, and untouched checkpoints (Section 5).
AuthorsOpenReview ↗

Response to reviewer s47J – part 2

How broadly does it apply when trusted targets are costly or unavailable?

Trusted targets are needed for calibration and audit, not for every training completion. The primary protocol uses 800 calibration and 600 audit prompts on GSM8K, 1,000 and 750 on MATH, and 400 and 400 on APPS, in addition to the reward-model fitting data. Math targets use exact correctness, and code targets use test pass rates. Targets can also come from verifiers, execution, or adjudicated preferences. With noisy preference labels, the guarantee concerns agreement with that reference. In the TL;DR pilot, studentized routing wins 600 of 1,087 adjudicated comparisons against distillation, against 590 for magnitude routing, and we do not claim this difference is significant.

If no trusted direction can be collected, the certificate cannot be established. An uncertainty heuristic still works in that case, but without a certificate. Policy drift adds a recurring cost. In the GSM8K drift experiment, each adaptation protocol uses 2,400 new trusted labels, and periodic recalibration reaches a conditional wrong-sign rate of 22.1% at the end of training, against 43.5% for one-shot calibration. Sections 7 and 8 now state these costs.

Please let us know whether these results answer your questions.

Reviewer_GLtJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses a routing problem in RL that between reward-based updates and teacher distillation, the learned rewards are unreliable. The method uses conformal prediction to decide when a reward estimate is reliable enough for training and otherwise falls back to teacher distillation. A separate dataset is used to check how often the selected reward estimates have the wrong sign. The paper also studies whether adapting the intervals to each example improves routing decisions. Experiments on math and coding tasks show small but consistent improvements over selecting examples based on the size of their predicted rewards.

Critical strengths

The method uses a separate dataset to check how often the rewards it chooses to trust are wrong. This makes its reliability claim more relevant to the samples actually used for reward-based training.

Critical weaknesses
  1. The contribution beyond existing methods is unclear. The basic method is equivalent to selecting samples with large predicted rewards. The improved version also learns which predictions are more reliable, but the experiments do not compare against a simple threshold using that same reliability information. It is therefore unclear whether the gains come from conformal calibration or simply from a better confidence score. This missing comparison makes the paper's main contribution difficult to be established.

  2. The audit results are difficult to reconcile with the stated calibration setup. The paper specifies 85% nominal interval coverage, but, e.g., Table 18 reports 216 and 222 incorrect reward directions among 750 MATH audit examples. Since each such error must fall outside the predicted interval, these counts imply coverage of at most 71.2% and 70.4%. Given the stated sample sizes, these gaps are difficult to explain by ordinary sampling variation if calibration and audit data follow the same distribution. The authors need to reconcile these results.

Questions for authors
  1. How do the Table 18 error counts reconcile with the 85% nominal coverage specified in Appendix E? Were calibration and audit examples generated using the same model and sampling settings?
  2. Does conformal calibration improve over a directly audited threshold when both methods adopt the same other settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer GLtJ – Part 1

Thank you the careful review. The table summarizes our answers, and the direct answers follow.

Your pointOur answerWhere in the revision
The base method equals thresholding large predicted rewardsCorrect. With one global radius, the gate is exactly a threshold on the reward magnitude, and the abstract now says so. Example-adaptive widths change the ranking.Abstract; Section 4.3
No comparison with a direct threshold on the same reliability scoreAdded. With everything else equal, calibration adds 5 of 1,319 problems for studentized routing and 3 for CQR.Section 5.2, Table 4
Table 18 counts cap coverage at 71.2% and 70.4%Correct. They do not reconcile with an 85% coverage reading, and the paper now says so. A joint validation reaches 85.2% and 85.3% audit coverage.Section 5.4; Appendices E.1 and L
Same model and sampling settings for calibration and audit?For the 3B audit, the saved records do not establish this. The joint validation fixes one generator and one decoding setting for both.Section 5.4; Appendix L

Does conformal calibration beat a directly audited threshold with all other settings equal?

Only slightly. With a global radius the gate is

Aα,τ(x,y)=1{∣r^(x,y)∣≥q^α+τ},A_{\alpha,\tau}(x,y)=\mathbf{1}\lbrace|\hat r(x,y)|\ge \hat q_\alpha+\tau\rbrace,

which is a magnitude threshold. The studentized gate is also a threshold, on the normalized score (∣r^∣−τ)/s(|\hat r|-\tau)/s. A direct threshold on that score at the same cutoff gives the same mask. Calibration supplies a finite-sample interpretation of the cutoff, and the audit can certify either implementation. Neither guarantee implies that the conformal version must score higher, so Section 5.2 tests it directly.

All rows share the frozen reward, scale, and quantile models, the teacher, the 800 calibration labels, the 600 independent audit labels, the training data, and the 2,000-step budget. Each audited procedure tests three cutoffs fixed before the audit and keeps the largest routed set that passes the same audit with β = 0.25 and δ = 0.05. A fresh set of 600 examples measures routing. Task results are single-checkpoint counts on the 1,319 GSM8K test problems, not seed averages.

MethodCorrect / 1,319Fresh coverage / 600Fresh routed / 600Fresh wrong / routed
Learned scale, uncalibrated width, no audit662 (50.2%)468 (78.0%)422 (70.3%)99/422 (23.5%)
Same learned-scale score, direct cutoff + audit665 (50.4%)519 (86.5%)388 (64.7%)67/388 (17.3%)
Studentized conformal + audit670 (50.8%)516 (86.0%)389 (64.8%)68/389 (17.5%)
Copied cutoff at the studentized setting670 (50.8%)516 (86.0%)389 (64.8%)68/389 (17.5%)
Raw learned quantile interval, no calibration or audit661 (50.1%)477 (79.5%)419 (69.8%)96/419 (22.9%)
Same learned quantiles, direct offset + audit666 (50.5%)516 (86.0%)391 (65.2%)70/391 (17.9%)
CQR calibration + audit669 (50.7%)514 (85.7%)393 (65.5%)72/393 (18.3%)

Studentized routing gains 5 of 1,319 problems (0.38 points) over the directly audited cutoff on the same score, and CQR gains 3 (0.23 points) over the directly audited offset on the same quantiles. The direct controls have slightly lower wrong-sign rates, so the comparison is a tradeoff between accuracy and wrong-sign rate. The copied cutoff ties exactly because it is the same rule. Under the common benchmark harness, the studentized gains are 4 of 1,319 GSM8K, 15 of 5,000 MATH, and 2 of 164 HumanEval problems (Table 5). None of these differences is established as significant. These single-checkpoint counts are separate from the ten-seed comparison of Table 2.

Please see the next part for how each audited procedure selected its cutoff.

AuthorsOpenReview ↗

Response to Reviewer GLtJ – Part 2

How each audited procedure selected its cutoff

The audit fixes τ = 0.10. The direct scale procedure thresholds the normalized score at 0.90, 1.10, and 1.30. Studentization uses calibrated radius 1.08 and candidates 1.08, 1.18, and 1.28. For learned quantiles, the direct interval adds offsets of 0, 0.10, and 0.20. CQR uses radius 0.09 plus the same additions. All families are fixed without audit labels, and they match in size and budget. Upper bounds are exact one-sided Clopper–Pearson bounds with per-candidate failure probability 0.05/3.

ProcedureCutoff or offsetAudit routed / 600Audit wrong / routedExact upper boundDecision at 0.25
Direct scale score0.90438105/4380.2862Reject
Direct scale score1.1038666/3860.2159Select
Direct scale score1.3034148/3410.1859Pass, fewer routed
Studentized radius1.0838967/3890.2171Select
Studentized radius1.1836560/3650.2101Pass, fewer routed
Studentized radius1.2834650/3460.1897Pass, fewer routed
Direct quantile offset0.00440107/4400.2897Reject
Direct quantile offset0.1039171/3910.2271Select
Direct quantile offset0.2034650/3460.1897Pass, fewer routed
CQR offset0.0939272/3920.2293Select
CQR offset0.1937060/3700.2074Pass, fewer routed
CQR offset0.2933043/3300.1751Pass, fewer routed

The simultaneous guarantee holds within each three-candidate procedure. Selecting among all four procedures would need a wider correction. Both conformal families keep their first candidate, so removing the audit at those settings leaves the masks and task counts unchanged. The audit supplies the certificate.

How do the Table 18 counts reconcile with Appendix E?

They do not, and the revision says so. Every wrong-sign event is an interval miss, so the τ = 0 counts bound audit coverage from above:

3B cellWrong signs at τ = 0Coverage at most
Qwen2.5-3B, GSM8K140/60076.7%
Qwen2.5-3B, MATH216/75071.2%
Qwen2.5-3B, APPS105/40073.8%
Llama-3.2-3B, GSM8K153/60074.5%
Llama-3.2-3B, MATH222/75070.4%
Llama-3.2-3B, APPS102/40074.5%

Split-conformal coverage averages over calibration draws, so a single audit sample can fall below 85%. Gaps this large, however, are hard to explain by sampling variation if calibration and audit follow one distribution. The audit used base-student completions before post-training. The saved records do not establish that calibration used the same generator, checkpoint, and decoding settings, so we cannot reconcile the counts after the fact. The revision therefore no longer reads the 3B audit as evidence of 85% coverage (Section 5.4 and Section 8).

The conditional-risk audit still holds, because it does not use conformal coverage. It needs only i.i.d. audit data that played no part in building the candidates. For the selected 3B MATH margins, 118 wrong signs among 415 routed examples and 110 among 404 give exact upper bounds of 0.3343 and 0.3224, below the 3B budget of 0.40.

Did calibration and audit use the same model and sampling settings?

In the 1–1.5B primary experiments, a separate fresh routing test agrees with the audit. Fresh coverage is 86.0%, 85.5%, and 86.5% on GSM8K, MATH, and APPS, every wrong sign is an interval miss, and the audit bounds of 0.243, 0.241, and 0.233 meet the budget of 0.25 (Table 6). For the 3B audit, the records do not establish matched settings, as explained above.

To test the matched case directly, the joint validation freezes each 3B reward model, scale model, and base-student generator. It then draws 1,000 calibration and 750 audit prompt–completion pairs from the same population, with greedy decoding, top-p = 1, and at most 1,024 completion tokens. α = 0.15 and the margins 0, 0.05, and 0.10 are fixed before the audit, and the 851st ordered residual defines the radius.

MATH cellStudentized radiusAudit covered / 750Routed at margin 0.05Wrong / routedExact upper bound
Qwen2.5-3B1.06639 (85.2%)45076/450 (16.9%)0.2101
Llama-3.2-3B1.11640 (85.3%)43878/438 (17.8%)0.2206

Every wrong sign falls among the 111 and 110 interval misses. Margin 0 fails the 0.25 budget in both cells, and margin 0.05 is selected. The validation covers MATH only. It shows that audit coverage is close to nominal when calibration and audit share a distribution, and it does not recover the original 3B records (Appendix L).

Please let us know whether these results answer your questions.

I appreciate the authors for adding the requested comparison with a directly audited threshold using the same reliability score. However, the new comparisons show little difference between conformal calibration and direct thresholding on the same reliability score. The authors also could not confirm whether the original calibration and audit samples were generated under the same settings. My concerns are therefore partially addressed. The paper should make more modest claims about the benefits using calibration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for careful and direct reviews. The main questions were whether the method is more than a threshold on the predicted reward, whether calibration beats a directly audited threshold on the same score, why the 3B MATH audit counts imply coverage below 85%, how the method relates to the GRPO advantage, why absolute scores sit below the model card, and whether the paper is readable. The revised paper answers each one.

  • Calibration against a direct threshold on the same score. With the same learned scale, labels, and audit, studentized conformal routing solves 670 of 1,319 GSM8K problems, against 665 for a directly audited threshold. Calibration adds 5 problems. Its value is a finite-sample error interpretation of the cutoff, and the audit supplies the certificate on the routed subset (Section 5.2, Table 4). The abstract now states this result.
  • The 3B audit does not support 85% coverage, and the paper now says so. Every wrong sign is an interval miss, so the τ = 0 counts of the 3B audit bound its coverage at 70.4% to 76.7% across the six cells (Table 18). A new joint validation draws calibration and audit from the same frozen generator on MATH. Its audit coverage is 85.2% and 85.3%, and every wrong sign is an interval miss (Appendix L). The conditional-risk audit does not depend on coverage.
  • GRPO's centered advantage. The raw-sign theorem does not certify the sign after group centering. The group-aware certificate of Section 4.4 does, and it lowers wrong centered signs among routed examples from 27.0% to 8.8%, while routing 43.5% of examples instead of 66.3% (Table 7).
  • Untouched checkpoints. The same Qwen2.5-1.5B-Instruct checkpoint scores 47.6% on GSM8K and 27.4% on MATH in our local harness, and 73.2% and 55.0% in a common benchmark harness, against 73.2% and 55.2% in the Qwen2.5 report. Under the common harness, all audited routers improve on the untouched checkpoint (Section 5.3, Table 5).
  • Readability. The main text is rewritten. Section 2 defines routed examples, wrong-sign events, and the marginal and conditional wrong-sign rates. Section 3 defines prompt-ID-disjoint splits and works through an example. Section 7 is a new Conclusion, and Section 8 discusses the cost of trusted targets.

Revisions at a glance

New result or changeRequested byWhere in the revision
Conformal calibration against direct audited thresholds on the same learned scoresGLtJ, d5XRSection 5.2, Table 4; Appendix J
Coverage implied by the 3B audit, and a joint calibration-and-audit validationGLtJSection 5.4; Appendices E.1 and L
Untouched checkpoints in two harnesses, and training under the common harnessd5XRSection 5.3, Table 5; Appendix K
Group-aware certificate for the GRPO advantage, with its costd5XRSection 4.4, Table 7
Definitions of routed, wrong-sign, marginal, and conditional ratess47JSection 2
Technical challenge stated, and a worked routing examples47JSections 1 and 3
Cost and sources of trusted targetss47JSections 7 and 8
Conclusion sectiond5XRSection 7
Main text rewrittens47J, d5XRSections 1–8
Six routing values in Table 2 corrected by 0.1 point to match their countsOur own checkTable 2

Table and section numbers in our replies refer to the revised paper. Table 18 and Appendix E keep their numbers.

What the paper claims

The method decides which training examples receive a reward-based update. An example is routed to the reward branch only when a split-conformal interval around its estimated reward clears a margin, and an independent audit bounds the wrong-sign rate among routed examples. With one global radius, the gate is exactly a magnitude threshold, so calibration does not change the ranking. Example-adaptive interval widths do change it, and they add 0.4 to 0.9 points over an equally audited magnitude gate in the tested settings. The guarantees concern the routing event on the stated distribution. They do not promise optimization gains or robustness to arbitrary policy drift.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.