- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper "Audit-Certified Conformal Routing for Reliable Hybrid Training" aims to improve post-training of LLMs through a router, that decides whether for a given training sample the model should be trained on the guidance of a reinforcement learning reward signal (e.g. GRPO) or on a distillation loss based on a larger teacher model. For this, the authors employ conformal prediction to bound the predictions of the reward model with uncertainty intervals and only employ the reward loss, when the whole interval lies above a chosen threshold or below . The evaluation covers reliability of the resulting router based on conformal predictions and final performance of the trained models on Qwen and Llama models and the GSM8K, MATH and HumanEval datasets. Across all experiments, the authors report modest performance improvements between 0.6% and 0.9%.
Critical strengths
Strenghts:
- the paper addresses a relevant and timely problem, i.e., that unreliable rewards may reinforce mistakes of models in post training.
- the idea of using a router with conformal predictions seems to be novel and is certainly interesting, while the statistical tools are well-established and have been extensively used in LLM contexts.
Critical weaknesses
Weaknesses
- while the idea is interesting it is certainly not groundbreaking and many similar ideas already exist in the literature. Furthermore, it seems like the method enables only moderate improvements on small-scale models. So overall, the method and the performance improvements are largely incremental.
- in GRPO, a sample can also get a negative reward when the reward model assigned a positive reward, but the other answers in the group get far lager rewards. Thus, it is very questionable whether the router objective connects to the actual updates done during GRPO.
- The results are questionable, e.g., for QWEN 2.5 1.5B Instruct, the best result on MATH in the paper is 29.8% while the official model card reports 55.2% and the best result on GSM8K in the paper is 50.7% while the model card reports 73.2% (see https://qwen.ai/blog?id=qwen2.5-llm). Also, an experiment investigating the improved performance of the learned uncertainty model with and without conformal calibration to understand the contribution of each building block would be helpful.
Minor Comments
- the paper appears to be missing a conclusions section.
- the paper appears to be written with heavy LLM use. before publication, i would whole-heartedly suggest a rewrite to have a story and to make the paper better understandable.
Questions for authors
Questions:
- What is the performance of the employed student checkpoints before training (since QWEN 2.5B results appear to be far better on the published model card)?
- How can the proposed method help when GRPO is used? Isn't the absolute magnitude of the reward unimportant in this context, but the distribution of rewards within the considered group?
- Does a gate with conformal routing outperform a gate that uses the same learned uncertainty model but without conformal calibration?
2 comments
Response to Reviewer d5XR – part 2
The group-aware gate cuts wrong centered signs among routed examples from 27.0% to 8.8% and routes fewer examples. Its accuracy of 50.2, against 50.0, does not show an optimization gain. The oracle gate uses the true centered directions and is an upper reference.
Does the conformal gate beat the same uncertainty model without calibration?
Yes, by a small margin. Table 4 holds the learned scale model fixed and varies only calibration and the audit (single-checkpoint counts on 1,319 GSM8K problems):
| Same learned scale model | Correct | Fresh routed / 600 | Fresh wrong / routed |
|---|---|---|---|
| Uncalibrated width, no audit | 662 (50.2%) | 422 (70.3%) | 23.5% |
| Direct cutoff + audit | 665 (50.4%) | 388 (64.7%) | 17.3% |
| Studentized conformal + audit | 670 (50.8%) | 389 (64.8%) | 17.5% |
Without calibration and the audit, the scale model routes 70.3% of examples with a 23.5% wrong-sign rate. An audited cutoff on the same score lowers the rate to 17.3%. Calibration then adds 5 problems, at a rate of 17.5%, and gives the cutoff a finite-sample interpretation. The learned quantile model shows the same pattern: 661 without calibration or audit, 666 with a direct audited offset, and 669 with CQR.
Are the gains too small to matter?
We agree that the accuracy gains are small. Adaptive routing beats an equally audited magnitude gate by 0.9 points on GSM8K, with paired 95% interval [0.3, 1.5], and by 0.4 to 0.9 points in each of six 3B backbone–task cells, where each per-cell interval excludes zero before multiplicity adjustment (Table 16). The main contribution is an explicit error budget for the reward branch. In the primary GSM8K experiment, studentized routing has a 17.5% wrong-sign rate among routed examples, against 22.3% for the audited magnitude gate and 27.9% for the uncertainty threshold, and the audit bounds the rate at 0.25 on the audit distribution. The revised abstract and Conclusion present the method this way.
Conclusion and writing
Section 7 is a new Conclusion, and Section 8 lists the limitations. The main text has been rewritten in plain language, with every term defined before use and a worked example in Section 3. AI assistance is disclosed in the paper.
Please let us know whether these results answer your questions.
Response to Reviewer d5XR – part 1
Thank you for the review. The table summarizes our answers.
How do the checkpoints score before training?
We evaluated the untouched instruction checkpoints, on the same checkpoint bytes, in our local harness and in a common benchmark harness. The Qwen2.5 report lists 73.2% on GSM8K and 55.2% on MATH for Qwen2.5-1.5B-Instruct (Table 10).
The common harness reproduces the report on the math benchmarks, so the low absolute scores in our tables reflect the local harness, not a weak checkpoint. These runs do not identify which prompt, decoding-limit, or answer-extraction choice of the local harness causes the gap. Within each harness, prompts, decoding, extraction, and scoring are fixed for all methods. Under the local harness, distillation-only training scores below the untouched checkpoint, 45.0% against 47.6% on GSM8K. Distillation-only training is therefore not an initialization control, and the untouched checkpoint is the right reference. Appendix K reports all four checkpoints in both harnesses.
Under the common harness, with the same 2,000-step horizon, teacher, data, and audit-label budget, every audited router improves on the untouched checkpoint (single-checkpoint counts):
Studentized routing gains 12 GSM8K, 37 MATH, and 11 HumanEval problems over the untouched checkpoint. Over the directly audited score, it gains 4, 15, and 2. These are single-checkpoint counts without seed-level uncertainty, so they do not establish significance.
Please see the second part of our response for the GRPO update, the calibration ablation, the effect size, and the writing.
How does the method help GRPO?
The router decides which examples receive the GRPO update and which receive distillation. You are right that a positive raw reward can give a negative advantage after group centering, so the raw-sign certificate does not cover the GRPO update. Section 4.4 adds a certificate for the centered quantity. For the active reward-routed set AG of a group, it compares
\quad\text{with}\quad A_i^\star=R_i-\frac{1}{|A_G|}\sum_{j\in A_G}R_j .$$ A group-level conformal quantile of the largest error in the group gives simultaneous intervals for all its members. A group uses the reward branch only when at least two examples remain and every interval excludes zero. Otherwise the whole group distills. Positive standard-deviation scaling preserves sign, so the certificate covers the centered and rescaled advantage that GRPO uses. The probability of any wrong certified sign in a new exchangeable group is at most the group error level. It does not certify the clipped policy gradient. | Routing rule | Routed examples | Wrong centered sign among routed | Accuracy | |---|---:|---:|---:| | Raw-sign gate | 66.3% | 27.0% | 50.0 ± 0.5 | | Group-aware whole-group certificate | 43.5% | 8.8% | 50.2 ± 0.5 | | Oracle normalized-direction gate | 61.5% | 0.0% | 52.6 ± 0.4 |