- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the RADAR framework, which divides the processing of LLM confidence into three stages: Repair, Adopt, and Diagnose. Theoretically, the authors decompose the squared risk of scalar recalibration into outcome noise, information not included in the original score, mapping error, and the pooling gap introduced by monotonic constraints, thus demonstrating that scalar recalibration cannot recover information compressed by the score. In the Repair stage, the authors propose PAIC, which uses a source-trained correctness probe to fit a monotonic mapping of the original confidence onto unlabeled target prompts. In the Adopt stage, a paired betting-based test is proposed to control the probability of incorrectly deploying no-improvement candidates. In the Diagnose stage, a prompt-clustered kernel test is proposed to detect residual errors associated with given features and handle the dependencies between multiple answers under the same prompt. Experiments covering various models, tasks, and distribution shifts show that PAIC can reduce Brier risk in some settings, but information from richer predictors may be lost after projection back to the scalar score, and reduced prediction risk does not necessarily improve actual review decisions.
Critical strengths
1.The delineation of the scalar recalibration capability boundary is clear and important. Risk decomposition clearly distinguishes between mapping errors that can be corrected through remapping and information lost in the original score, and further analyzes the pooling gap caused by monotonicity. This explains why a score that can achieve calibration still systematically fails on identifiable subgroups.
2.RADAR separates fitting, deployment, and auditing, which has strong practical implications. In particular, the paired adoption procedure and prompt-clustered audit avoid using the same probe for self-validation and incorrectly treating multiple responses to the same prompt as independent samples. The paper also reports results showing that PAIC has no benefit or reduces downstream utility, making the conclusions more credible than simply showing positive benchmark gains.
Critical weaknesses
1.The core assumptions of PAIC are difficult to verify in its primary use cases. PAIC can fit without target labels, but this guarantee depends on a sufficiently small target-domain score-conditional probe bias, which itself typically requires target labels for evaluation; the final adoption phase also still requires independent target labels. The paper needs to more clearly explain when PAIC offers a substantial advantage in label cost or deployment performance compared to directly waiting for a small number of target labels before label-based recalibration.
2.The main adoption results provide weak support for PAIC itself, and its overall external validity is limited. In cached raw-score allocations, the vast majority of adoptions in paired betting chose richer direct predictors instead of PAIC. Therefore, these results primarily validate the statistical power of the adoption test and do not adequately demonstrate that PAIC is a frequently worthwhile repair in practical deployments. The experiments are also primarily based on cached responses, a limited-scale open-source model, and several overlapping refits; some results only provide pointwise or descriptive uncertainty.
Questions for authors
1.Could you report the adoption results separately, including only the original score and each PAIC variant, excluding richer direct predictors? For example, the PAIC certification rate, beneficial adoption rate, false adoption rate, and the target labels required to achieve a given adoption power. This result directly impacts my assessment of the actual value of PAIC.
2.Does the paper’s indistinguishability result imply that score-conditional probe bias cannot, in general, be diagnosed from unlabeled target data alone? If so, could the authors clarify the additional assumptions or labeled evidence that practitioners would need before relying on PAIC? It would also be helpful to report, using held-out labels, the empirical relationship between score-conditional bias and PAIC’s risk improvement.
8 comments
Response to Reviewer rfhz (Part 2/3)
W2 (continued). The main adoption results chose richer predictors, not PAIC. External validity: cached responses, limited-scale models, overlapping refits, pointwise uncertainty.
The full candidate family. You are right that when richer predictors are candidates and their inputs can be served, the betting test usually adopts them (Appendix B.10); PAIC is for deployments that can serve only the confidence, as with a hosted model (page 4, lines 186–191). On HLE, where PAIC beats the direct probes, the same rule adopted a PAIC repair in 13,072 trials and a richer predictor in 3,776 (Appendix F.3). In both experiments with the full family, 69,000 trials, the test never adopted a score that was worse on held-out prompts (Tables 15, 32), and in 168,000 simulated decisions with nothing to gain it never selected a harmful candidate. The theory says why: the adopted score is safe and near-best among the candidates that can be detected (Corollary A.37).
External validity. The revision extends the same protocol to fresh generation with a newer model (Qwen3.5-9B, 6,000 answers; page 9, lines 470–476), to a harder benchmark (HLE; Appendix F.3, pages 71–72, lines 3829–3858), and to the end-to-end pipeline of Q1, in which every stage uses its own prompts. The breadth study includes Qwen2.5-32B (Appendix C, pages 63–64, lines 3348–3420). Two further experiments from the discussion period, with a second new model (Gemma-4-12B-it) and on long-form question answering (ASQA), are reported in our response to Reviewer 5WG2. The new comparisons report simultaneous intervals, and the bias analysis of Q2 uses a cluster bootstrap over the 14 combinations of shift and model, because the refits of a combination overlap (Appendix F.1, page 68, lines 3632–3647).
Q2. Does the indistinguishability result mean that the probe's bias cannot be diagnosed from unlabeled target data? What must practitioners assume or label? Please report the empirical relation between this bias and PAIC's risk improvement.
Yes: in general it cannot, and Proposition 3.2.2 makes this precise (page 6, lines 270–274; proof in Appendix A.12, page 40, lines 2122–2147). It turns the indistinguishability result of the submission into a statement about the deployment decision: it constructs two populations in which every unlabeled quantity, the probe's predictions included, has the same distribution, while the same PAIC repair lowers risk by in one and raises it by in the other. Every label-free rule therefore has worst-case regret of at least . Unlabeled data do bound the size of the change (Proposition A.26, page 39, lines 2065–2088), but not its sign.
What practitioners need. Without target labels, only an assumption helps: under covariate shift with a bounded density ratio, the ratio times the probe's excess risk on source data bounds the bias (Remark A.30, page 40, lines 2148–2159). This requires correctness given the inputs to behave as on the source, which unlabeled data cannot check. Otherwise a small labeled target set suffices: to certify the repair (about labels), to estimate the bias directly, or to remove its constant part at rate with labels (Corollary 3.2.3, page 6, lines 282–295).
The empirical relation, with held-out labels. By Corollary 3.2.1, PAIC's gain is the headroom (what the shift left to repair) minus the probe's squared bias. So what PAIC loses against a fit on target labels, its shortfall, should follow the bias, and it does; the probe's accuracy does not (Appendix F.1, page 68, lines 3649–3660; 84 cases: 42 shift experiments, two probes; cluster-bootstrap intervals):
| Quantity | Result |
|---|---|
| PAIC's shortfall against the probe's bias | Spearman 0.93 [0.78, 0.97], slope 0.96 |
| The same, within the main benchmark (small bias) | 0.59 [0.28, 0.78] |
| PAIC's shortfall against the probe's MSE | 0.35 [−0.11, 0.77] |
| PAIC's risk change against the bias, adjusted for headroom | Partial Spearman 0.46 [0.21, 0.61]: a larger bias, a smaller improvement |
| Controlled test: each probe's MSE raised by 0.010 | PAIC's risk rises by 0.0131 if the added error varies with the score, by 0.0012 if it cancels within score bins (larger in 83 of 84 cases); over 504 perturbed probes its change tracks the change in bias (0.92), not in MSE (0.38) (Appendix F.2, page 71, lines 3780–3791) |
At the setting level, the shortfall in Table 1 matches the bias to within 0.002 where the bias is small and to within 0.008 on HLE, where it reaches 0.15 (page 8, lines 423–431; Figure 3).
Response to Reviewer rfhz (Part 3/3)
W1. The probe's bias needs target labels to check, and adoption needs labels too. When does PAIC beat simply waiting for a few target labels and recalibrating?
Before target labels exist and while they are few. With none, PAIC is already close to supervised recalibration; and the labels that do arrive make it better (Appendix F.6, page 79, lines 4245–4255; Appendix F.8; Tables 41, 44).
| Target labels at hand | Wait and recalibrate on them | PAIC |
|---|---|---|
| None | Not possible: the shifted confidence stays in use | Repaired at once: 95% and 80% of the gain of a 750-label fit on MATH, 59–97% of the gap to a 250-label fit on HLE, a median 87% over the six shift settings |
| 100 | Fitted (risk 0.2304 for Llama, 0.1464 for Qwen), not yet certified | With no labels, already level (0.2294, 0.1468); the 100 labels are free for the decision, or correct the probe: then better by 0.005 for Llama, level for both Qwen models |
| 250 to 750 | Certified from 250 labels (Llama), 500 (Qwen3.5-9B), not within 750 (Qwen2.5) | Certified from 250, 250 and 750 labels |
Two results explain the table. Any calibration must be certified on labels independent of those used to fit it (Proposition 3.3), so PAIC, which uses none to fit, spends every target label on the decision. And a few labels remove the constant part of the probe's bias at rate , against for refitting the calibration (Corollary 3.2.3): shifting the probe lowers PAIC's risk in all 16 cases tested (page 10, lines 501–505). The corollary also says when to do which: shift the probe when its bias is nearly constant across scores, refit the calibration otherwise. With many labels the two routes meet, and the same test adopts whichever is better.
We sincerely thank the reviewer for their constructive feedback and thoughtful suggestions. We hope that our responses and additional experimental results have adequately addressed the concerns raised. We would be happy to provide any further clarifications or answer additional questions. If you find that your concerns have been satisfactorily addressed, we would be grateful if you could kindly reconsider and update your evaluation score accordingly.
Response to Reviewer rfhz (Part 1/3)
Thank you for the careful review and for two precise questions. Q1 asked for exactly the analysis that isolates PAIC's value, and answering it has made the paper's case clearer. Page and line numbers refer to the revised manuscript.
Done, and the result supports PAIC itself, not only the test: with PAIC repairs as the only candidates, the betting test certifies PAIC in 33 of 42 shift experiments, and in up to 72% of trials on the 30 experiments of the submission; no adoption was a false one (Section 4.3, page 10, lines 489–499).
(a) The 30 experiments of the submission that repair the model's own confidence, with the three PAIC repairs as the only candidates against the original score (Appendix F.5, pages 78–79, lines 4205–4215). A trial is one random draw of the labeled examples:
Every adoption was beneficial: each adopted repair has lower risk on the test pool, and the 10 of 90 repairs that do not improve were never certified. Under the stricter count of the appendix, where the gain must also be supported by the adopted repair's own test interval, the rates are 15.1%, 23.4%, 28.1% and 28.2%, and 76.9% with 500 labels on the 11 experiments that contain such a repair.
(b) An end-to-end pipeline on 42 shift experiments, with PAIC repairs as the only candidates and every stage (repair, test, audit) on its own prompts (Appendix F.4, page 76, lines 4059–4071): PAIC is certified in 33 (15 of 18 in the main benchmark, 18 of 24 into or within Humanity's Last Exam, HLE), and all 33 certified repairs are better on held-out test prompts (intervals below zero). Against the calibration in use, which was fitted on source labels, the deployed confidence lowers Brier risk by 0.1025 on average over the 24 HLE experiments and by 0.0136 over the 18 of the main benchmark (Table 35). Nothing is adopted in the six experiments where there is almost nothing to repair (headroom 0.003–0.005).
(c) A newer model (Appendix F.8, page 81, lines 4371–4373): on Qwen3.5-9B, the betting test certifies PAIC from 250 labels.
Labels for a given power: about 6 divided by the gain, where the gain is how much the repair lowers the Brier risk. Corollary 3.3.1 (page 7, lines 334–348) puts the count at the order of log(1/α)/gain. A test on 77 improving PAIC repairs, with gains from 0.0015 to 0.21, measures it (Appendix F.11, page 86, lines 4609–4621): labels scale as gain−0.97 (95% interval −1.07 to −0.92), about 6/gain for 50% power and 7/gain for 80%. The corollary's bound was never violated, and repairs that do not improve were certified in at most 0.025% of draws. The law also explains (c): under a second, more lenient grader the gain is a sixth as large, and 750 labels are then not yet enough. Because the test is valid at every sample size (Proposition A.36), labeling can stop at the first certification, so the gain need not be known in advance.