- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies confidence estimation for reasoning LLMs without correctness labels and with limited sampling budgets. BASC uses agreement between an anchor response and an independent probe as a binary observation of the anchor answer's latent self-consistency. It either recalibrates an existing score or first learns an agreement predictor from response representations, followed by isotonic recalibration on an independent split. An existing binary-isotonic calibration guarantee transfers to latent self-consistency through a conditional-expectation identity. Correctness calibration is evaluated empirically rather than guaranteed. Experiments cover 36 model-dataset combinations spanning nine reasoning models and five datasets, including distribution shifts, selective prediction, and an external-embedding variant.
Critical strengths
-
A practical low-budget design. BASC requires no correctness labels or updates to the underlying LLM. Its lowest-budget setting uses two responses per calibration query and one at deployment.
-
Improved calibration over a standard baseline. Across 36 model-dataset combinations, BASC-L () reduces average correctness ECE from 0.197 to 0.083 and Brier score from 0.222 to 0.162 compared with token probabilities (Table 1).
Critical weaknesses
-
Incremental contribution relative to existing uncertainty-amortization frameworks. Semantic Entropy Probes (Kossen et al., 2024, Section 4) already construct sampling-derived semantic-entropy supervision offline, train lightweight hidden-state probes without correctness labels, and estimate uncertainty from a single generation at deployment. BASC's main increment is to replace costlier per-query targets with pooled binary-agreement observations and combine score learning with held-out isotonic recalibration. Its calibration bound is also transferred from existing binary-isotonic theory. These are useful refinements, but the conceptual advance appears concentrated in the supervision budget and calibration combination rather than a new uncertainty-amortization framework.
-
The headline efficiency comparison does not isolate the benefit of the proposed supervision. Figure 3(a) supports the tenfold calibration-budget advantage by comparing two-stage BASC with single-stage ridge estimators, although held-out recalibration itself substantially reduces ECE. Figure 3(b) controls for this stage, but reports AUROC against samples per query rather than correctness ECE and Brier against total decoding budget. A budget-matched comparison with the same recalibration stage is needed to establish how much of the claimed calibration-efficiency advantage comes from the proposed supervision rather than standard recalibration.
Questions for authors
- Joint error analysis of BASC, self-consistency, and correctness: Table 1 and Figure 5(a) already assess self-consistency (SC) as a correctness proxy, and Table 1 reports correctness calibration for BASC. However, the results do not separately quantify BASC-to-SC calibration error and the conditional proxy discrepancy in Equation 7 / Appendix A.1, leaving their contributions to the remaining correctness error unclear. A joint analysis for the same final confidence would help connect the proxy guarantee to the empirical correctness results. With indicating anchor correctness and denoting latent SC, the relevant quantities are:
Model-dataset breakdowns using common confidence bins and uncertainty estimates would make this comparison more informative. In particular, the mean , , and within each bin would help distinguish imperfect BASC-to-SC calibration from SC-to-correctness mismatch, and reveal whether their signed contributions reinforce or cancel. The existing 100-probe pool could support this analysis on held-out test examples.
-
Comparison in Figure 3(a): It remains unclear whether BASC-L's lower ECE with 800 decodes than the ridge baselines with 8,000 decodes would persist if RR-SC used the same held-out isotonic stage under a matched total decoding budget. Correctness ECE and Brier curves against total calibration decodes, counting both stages, would help separate the benefit of the proposed supervision from that of recalibration.
-
Confidence-based decisions: Section 3.3 and Figure 4 already evaluate selective prediction at fixed abstention rates, including confidence-accuracy gaps for the selected and rejected sets. The introduction motivates calibration as a way to decide "when a model can be trusted to answer, when it should defer" (Section 1, lines 029-031). An analysis at fixed numerical confidence thresholds, such as accepting answers only when , would better connect these results to that deployment motivation. Accepted accuracy and coverage, with sample counts and uncertainty estimates, on the existing QA tasks and distribution shifts would clarify the reliability and coverage obtained when using the same confidence cutoff rather than fixing the abstention rate.
4 comments
author rebuttal 2
Q4: Selective Prediction with Fixed Thresholds
Our original selective prediction evaluation follows the common convention of fixing coverage (or abstention) rates, which allows methods to be compared directly through a single metric, accepted accuracy. Fixing numerical confidence thresholds instead requires jointly evaluating accuracy and coverage, since different methods may accept substantially different fractions of examples.
We agree that this complementary evaluation is valuable for deployment decisions. We have therefore added a fixed confidence threshold experiment (Appendix Figure 22), evaluating thresholds . We report accepted accuracy, mean accepted confidence, and coverage on in-distribution and OOD settings. Our results show that BASC provides more effective confidence-based selection than the single-generation baselines. At , BASC-L accepts 39–41% of QA answers with 64–69% accuracy, compared with approximately 53% accuracy for raw token probabilities and verbalized confidence, which accept substantially more answers. BASC-L's accepted accuracy generally increases with the threshold, while coverage decreases. BASC does exhibit some overconfidence relative to correctness, consistent with our new error decomposition (Appendix D.1.2). These results demonstrate the practical utility of BASC for confidence-based decisions while clarifying the limitations of its self-consistency calibration guarantee.
Note: API-only experiments
At the request of Reviewer MYqJ we have added additional experiments with GPT-5-mini to Appendix D.1.1, further demonstrating that BASC extends to API-only models.
Thank you for the detailed response and additional experiments. The matched-budget comparison(Q2) addresses my concern about separating score learning from recalibration, joint-error analysis(Q3) shows that BASC calibrates well to self-consistency.
However, I remain unconvinced that these additions establish a sufficiently substantial contribution beyond existing work. My point is that BASC can calibrate welll to SC, but SC is itself only a surrogate for correctness. So good SC calibration does not necessarily make it a reliable probability of correctness.
I appreciate the authors’ additional experiments and analysis to address my concerns and clarify these limitations.
We thank the reviewer for the continued engagement. We are glad that our matched-budget comparison (Q2) and error decomposition (Q3) addressed those concerns.
We agree that calibration to self-consistency does not guarantee calibration to correctness, and we explicitly acknowledge and quantify this limitation. However, reliance on imperfect proxies is inherent to methods for unsupervised uncertainty estimation and calibration. For example, DACA (NeurIPS 2025) uses base-model confidence as an unlabeled signal, while SEP and Self-Calibration (ICLR 2026) use sampling-derived uncertainty estimates. These methods establish their practical value through empirical evaluation against correctness, rather than guarantees that their proxies are equivalent to correctness.
BASC addresses an important computational bottleneck within this established research paradigm. Our contribution is not to eliminate the limitations of the underlying uncertainty proxy, but to substantially improve the efficiency of learning from it. By learning directly from pooled binary agreement observations, BASC eliminates the need for costly per-example uncertainty targets, reducing supervision-generation requirements by 5× relative to SEP and 16× relative to Self-Calibration. Importantly, we evaluate correctness calibration directly, demonstrating improvements in ECE (0.197 to 0.083) and Brier score (0.222 to 0.162) across 36 settings in our original submission. We have further demonstrated improvements on a proprietary model (GPT-5-mini) across three QA datasets during rebuttal.
Overall, we believe that substantially improving the efficiency of an established and useful approach through a distinct, theoretically grounded supervision strategy represents a meaningful methodological contribution.
author rebuttal
We thank the reviewer for taking the time to read and offer feedback on our submission. Please see below regarding responses to the concerns raised in your review.
Q1: Contribution relative to existing uncertainty-amortization frameworks
We thank the reviewer for highlighting this distinction. We agree that amortizing multi-sample uncertainty estimates into single-generation predictors is not itself new. However, our contribution differs from SEP in two important respects.
First, SEP focuses on predicting semantic entropy for hallucination detection, rather than producing calibrated confidence estimates. On the other hand, BASC provides a general framework for unsupervised confidence recalibration. While SEP could itself serve as a Stage 1 scoring function within BASC, it does not itself produce calibrated confidence, which is a distinct objective from hallucination detection.
Second, and more importantly, BASC addresses a different axis of efficiency than previous methods by reducing offline sampling costs, rather than only test-time costs. SEP constructs per-query semantic entropy targets from multiple generations (10 in its experiments). BASC instead requires only 2 responses per calibration query, without estimating accurate per-query uncertainty targets.
Overall, BASC's contribution goes beyond a 5x reduction in sampling. BASC introduces a different approach to supervision, removing a central computational bottleneck of sampling-based uncertainty amortization, while enabling label-free recalibration of arbitrary uncertainty scores with a distribution-free guarantee for latent self-consistency.
Q2: Isolating Stage 2 in headline efficiency comparison
We thank the reviewer for raising this concern. We agree that the original comparison did not isolate the contribution of Stage 1 score learning from Stage 2 recalibration.
To address this, we have added BASC-RR-SC to Figure 3(a), which applies the same held-out binary isotonic recalibration to a ridge-regression predictor trained on empirical self-consistency. We use identical data splits and decoding budgets for BASC-RR-SC and BASC-L: 200 Stage 1 queries with k responses each, followed by 200 Stage 2 queries with two responses each, for a total of 200k+400 decodes.
The new results show that BASC-RR-SC and BASC-L achieve comparable ECE (Figure 3(a)) and Brier scores (Appendix Figure 11) at matched budgets. This establishes that much of the calibration-efficiency advantage over single-stage baselines comes from binary recalibration, rather than the particular Stage 1 learner. However, BASC-L's binomial-logistic Stage 1 achieves significantly higher AUROC at small k (Figure 3(b)), demonstrating an additional benefit in discrimination (this is also reflected in a slightly better Brier score).
These results reinforce our central contribution, that binary recalibration provides efficient, unsupervised recalibration for different scoring functions, while our binomial-logistic learner offers additional discrimination under limited sampling.
Q3: Joint Error Analysis of BASC and Self-Consistency
We thank the reviewer for this suggestion. We have added a detailed analysis in Appendix D.1.2, using the existing 100-probe evaluation pool to decompose an upper bound on BASC's correctness calibration error into two components, calibration error with respect to self-consistency and the gap between self-consistency and correctness.
Across the full 36 settings, we find that BASC-L is well calibrated to self-consistency (ECE 0.044), while the self-consistency/correctness discrepancy (0.069) accounts for 61% of the decomposed gap. This discrepancy is substantially larger on QA than math (0.103 vs. 0.034). We report the decomposition, including signed contributions, in the aggregate as well as across all 36 model/dataset combinations (Tables 8-9). We further characterize highly self-consistent but incorrect answers in Figure 13, finding that 15–30% of answers with estimated self-consistency at least 0.9 are incorrect for several QA model/dataset combinations, with 95% confidence intervals.
These findings clarify an important limitation. While BASC effectively calibrates to self-consistency, correctness calibration is still contingent on the quality of self-consistency as a correctness proxy. We have incorporated these findings into Section 3.1 and Limitations.