acceptodds
Under review as a conference paper at ICLR 2027

Binary Agreement for Efficient Unsupervised Confidence Calibration of Reasoning LLMs

Abstract

Reasoning language models often lack the calibrated confidence needed for reliable deployment, particularly when correctness labels are unavailable and sampling budgets are limited. We introduce Binary Agreement for Self-consistency Calibration (BASC), which uses self-consistency as an unlabeled proxy for correctness without constructing costly self-consistency targets for each calibration query. Our key observation is that agreement between an anchor response and a single independent probe is a Bernoulli observation whose conditional mean is the anchor's latent self-consistency. Pooling these observations across unlabeled queries, BASC supports both recalibrating existing confidence scores and learning more discriminative scores before recalibration, using as few as two responses per calibration query and only one at deployment. The use of binary isotonic recalibration provides a distribution-free calibration guarantee with respect to latent self-consistency. Across 5 tasks and 9 reasoning models, BASC substantially improves calibration with respect to correctness.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLMs, calibration, UQ, reliable ML, trustworthy ML

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PjhEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies confidence estimation for reasoning LLMs without correctness labels and with limited sampling budgets. BASC uses agreement between an anchor response and an independent probe as a binary observation of the anchor answer's latent self-consistency. It either recalibrates an existing score or first learns an agreement predictor from response representations, followed by isotonic recalibration on an independent split. An existing binary-isotonic calibration guarantee transfers to latent self-consistency through a conditional-expectation identity. Correctness calibration is evaluated empirically rather than guaranteed. Experiments cover 36 model-dataset combinations spanning nine reasoning models and five datasets, including distribution shifts, selective prediction, and an external-embedding variant.

Critical strengths
  1. A practical low-budget design. BASC requires no correctness labels or updates to the underlying LLM. Its lowest-budget setting uses two responses per calibration query and one at deployment.

  2. Improved calibration over a standard baseline. Across 36 model-dataset combinations, BASC-L (k=2k=2) reduces average correctness ECE from 0.197 to 0.083 and Brier score from 0.222 to 0.162 compared with token probabilities (Table 1).

Critical weaknesses
  1. Incremental contribution relative to existing uncertainty-amortization frameworks. Semantic Entropy Probes (Kossen et al., 2024, Section 4) already construct sampling-derived semantic-entropy supervision offline, train lightweight hidden-state probes without correctness labels, and estimate uncertainty from a single generation at deployment. BASC's main increment is to replace costlier per-query targets with pooled binary-agreement observations and combine score learning with held-out isotonic recalibration. Its calibration bound is also transferred from existing binary-isotonic theory. These are useful refinements, but the conceptual advance appears concentrated in the supervision budget and calibration combination rather than a new uncertainty-amortization framework.

  2. The headline efficiency comparison does not isolate the benefit of the proposed supervision. Figure 3(a) supports the tenfold calibration-budget advantage by comparing two-stage BASC with single-stage ridge estimators, although held-out recalibration itself substantially reduces ECE. Figure 3(b) controls for this stage, but reports AUROC against samples per query rather than correctness ECE and Brier against total decoding budget. A budget-matched comparison with the same recalibration stage is needed to establish how much of the claimed calibration-efficiency advantage comes from the proposed supervision rather than standard recalibration.

Questions for authors
  1. Joint error analysis of BASC, self-consistency, and correctness: Table 1 and Figure 5(a) already assess self-consistency (SC) as a correctness proxy, and Table 1 reports correctness calibration for BASC. However, the results do not separately quantify BASC-to-SC calibration error and the conditional proxy discrepancy in Equation 7 / Appendix A.1, leaving their contributions to the remaining correctness error unclear. A joint analysis for the same final confidence CC would help connect the proxy guarantee to the empirical correctness results. With ZZ indicating anchor correctness and SS denoting latent SC, the relevant quantities are:

CES(C)=E[∣E[S∣C]−C∣]\mathrm{CE}_S(C)=\mathbb{E}[|\mathbb{E}[S\mid C]-C|]

CEZ(C)=E[∣E[Z∣C]−C∣]\mathrm{CE}_Z(C)=\mathbb{E}[|\mathbb{E}[Z\mid C]-C|]

ΔZ,S(C)=E[∣E[Z−S∣C]∣]\Delta_{Z,S}(C)=\mathbb{E}[|\mathbb{E}[Z-S\mid C]|]

Model-dataset breakdowns using common confidence bins and uncertainty estimates would make this comparison more informative. In particular, the mean CC, SS, and ZZ within each bin would help distinguish imperfect BASC-to-SC calibration from SC-to-correctness mismatch, and reveal whether their signed contributions reinforce or cancel. The existing 100-probe pool could support this analysis on held-out test examples.

  1. Comparison in Figure 3(a): It remains unclear whether BASC-L's lower ECE with 800 decodes than the ridge baselines with 8,000 decodes would persist if RR-SC used the same held-out isotonic stage under a matched total decoding budget. Correctness ECE and Brier curves against total calibration decodes, counting both stages, would help separate the benefit of the proposed supervision from that of recalibration.

  2. Confidence-based decisions: Section 3.3 and Figure 4 already evaluate selective prediction at fixed abstention rates, including confidence-accuracy gaps for the selected and rejected sets. The introduction motivates calibration as a way to decide "when a model can be trusted to answer, when it should defer" (Section 1, lines 029-031). An analysis at fixed numerical confidence thresholds, such as accepting answers only when C≥0.9C\ge0.9, would better connect these results to that deployment motivation. Accepted accuracy and coverage, with sample counts and uncertainty estimates, on the existing QA tasks and distribution shifts would clarify the reliability and coverage obtained when using the same confidence cutoff rather than fixing the abstention rate.

4 comments
AuthorsOpenReview ↗

author rebuttal

We thank the reviewer for taking the time to read and offer feedback on our submission. Please see below regarding responses to the concerns raised in your review.

Q1: Contribution relative to existing uncertainty-amortization frameworks

We thank the reviewer for highlighting this distinction. We agree that amortizing multi-sample uncertainty estimates into single-generation predictors is not itself new. However, our contribution differs from SEP in two important respects.

First, SEP focuses on predicting semantic entropy for hallucination detection, rather than producing calibrated confidence estimates. On the other hand, BASC provides a general framework for unsupervised confidence recalibration. While SEP could itself serve as a Stage 1 scoring function within BASC, it does not itself produce calibrated confidence, which is a distinct objective from hallucination detection.

Second, and more importantly, BASC addresses a different axis of efficiency than previous methods by reducing offline sampling costs, rather than only test-time costs. SEP constructs per-query semantic entropy targets from multiple generations (10 in its experiments). BASC instead requires only 2 responses per calibration query, without estimating accurate per-query uncertainty targets.

Overall, BASC's contribution goes beyond a 5x reduction in sampling. BASC introduces a different approach to supervision, removing a central computational bottleneck of sampling-based uncertainty amortization, while enabling label-free recalibration of arbitrary uncertainty scores with a distribution-free guarantee for latent self-consistency.

Q2: Isolating Stage 2 in headline efficiency comparison

We thank the reviewer for raising this concern. We agree that the original comparison did not isolate the contribution of Stage 1 score learning from Stage 2 recalibration.

To address this, we have added BASC-RR-SC to Figure 3(a), which applies the same held-out binary isotonic recalibration to a ridge-regression predictor trained on empirical self-consistency. We use identical data splits and decoding budgets for BASC-RR-SC and BASC-L: 200 Stage 1 queries with kk responses each, followed by 200 Stage 2 queries with two responses each, for a total of 200k+400200k+400 decodes.

The new results show that BASC-RR-SC and BASC-L achieve comparable ECE (Figure 3(a)) and Brier scores (Appendix Figure 11) at matched budgets. This establishes that much of the calibration-efficiency advantage over single-stage baselines comes from binary recalibration, rather than the particular Stage 1 learner. However, BASC-L's binomial-logistic Stage 1 achieves significantly higher AUROC at small kk (Figure 3(b)), demonstrating an additional benefit in discrimination (this is also reflected in a slightly better Brier score).

These results reinforce our central contribution, that binary recalibration provides efficient, unsupervised recalibration for different scoring functions, while our binomial-logistic learner offers additional discrimination under limited sampling.

Q3: Joint Error Analysis of BASC and Self-Consistency

We thank the reviewer for this suggestion. We have added a detailed analysis in Appendix D.1.2, using the existing 100-probe evaluation pool to decompose an upper bound on BASC's correctness calibration error into two components, calibration error with respect to self-consistency and the gap between self-consistency and correctness.

Across the full 36 settings, we find that BASC-L is well calibrated to self-consistency (ECE 0.044), while the self-consistency/correctness discrepancy (0.069) accounts for 61% of the decomposed gap. This discrepancy is substantially larger on QA than math (0.103 vs. 0.034). We report the decomposition, including signed contributions, in the aggregate as well as across all 36 model/dataset combinations (Tables 8-9). We further characterize highly self-consistent but incorrect answers in Figure 13, finding that 15–30% of answers with estimated self-consistency at least 0.9 are incorrect for several QA model/dataset combinations, with 95% confidence intervals.

These findings clarify an important limitation. While BASC effectively calibrates to self-consistency, correctness calibration is still contingent on the quality of self-consistency as a correctness proxy. We have incorporated these findings into Section 3.1 and Limitations.

AuthorsOpenReview ↗

author rebuttal 2

Q4: Selective Prediction with Fixed Thresholds

Our original selective prediction evaluation follows the common convention of fixing coverage (or abstention) rates, which allows methods to be compared directly through a single metric, accepted accuracy. Fixing numerical confidence thresholds instead requires jointly evaluating accuracy and coverage, since different methods may accept substantially different fractions of examples.

We agree that this complementary evaluation is valuable for deployment decisions. We have therefore added a fixed confidence threshold experiment (Appendix Figure 22), evaluating thresholds τ∈{0.5,0.6,0.7,0.8,0.9,0.95}\tau\in\{0.5,0.6,0.7,0.8,0.9,0.95\}. We report accepted accuracy, mean accepted confidence, and coverage on in-distribution and OOD settings. Our results show that BASC provides more effective confidence-based selection than the single-generation baselines. At τ=0.7\tau=0.7, BASC-L accepts 39–41% of QA answers with 64–69% accuracy, compared with approximately 53% accuracy for raw token probabilities and verbalized confidence, which accept substantially more answers. BASC-L's accepted accuracy generally increases with the threshold, while coverage decreases. BASC does exhibit some overconfidence relative to correctness, consistent with our new error decomposition (Appendix D.1.2). These results demonstrate the practical utility of BASC for confidence-based decisions while clarifying the limitations of its self-consistency calibration guarantee.

Note: API-only experiments

At the request of Reviewer MYqJ we have added additional experiments with GPT-5-mini to Appendix D.1.1, further demonstrating that BASC extends to API-only models.

Thank you for the detailed response and additional experiments. The matched-budget comparison(Q2) addresses my concern about separating score learning from recalibration, joint-error analysis(Q3) shows that BASC calibrates well to self-consistency.

However, I remain unconvinced that these additions establish a sufficiently substantial contribution beyond existing work. My point is that BASC can calibrate welll to SC, but SC is itself only a surrogate for correctness. So good SC calibration does not necessarily make it a reliable probability of correctness.

I appreciate the authors’ additional experiments and analysis to address my concerns and clarify these limitations.

We thank the reviewer for the continued engagement. We are glad that our matched-budget comparison (Q2) and error decomposition (Q3) addressed those concerns.

We agree that calibration to self-consistency does not guarantee calibration to correctness, and we explicitly acknowledge and quantify this limitation. However, reliance on imperfect proxies is inherent to methods for unsupervised uncertainty estimation and calibration. For example, DACA (NeurIPS 2025) uses base-model confidence as an unlabeled signal, while SEP and Self-Calibration (ICLR 2026) use sampling-derived uncertainty estimates. These methods establish their practical value through empirical evaluation against correctness, rather than guarantees that their proxies are equivalent to correctness.

BASC addresses an important computational bottleneck within this established research paradigm. Our contribution is not to eliminate the limitations of the underlying uncertainty proxy, but to substantially improve the efficiency of learning from it. By learning directly from pooled binary agreement observations, BASC eliminates the need for costly per-example uncertainty targets, reducing supervision-generation requirements by 5× relative to SEP and 16× relative to Self-Calibration. Importantly, we evaluate correctness calibration directly, demonstrating improvements in ECE (0.197 to 0.083) and Brier score (0.222 to 0.162) across 36 settings in our original submission. We have further demonstrated improvements on a proprietary model (GPT-5-mini) across three QA datasets during rebuttal.

Overall, we believe that substantially improving the efficiency of an established and useful approach through a distinct, theoretically grounded supervision strategy represents a meaningful methodological contribution.

Reviewer_cMvcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Binary Agreement for Self-consistency Calibration (BASC), an unlabeled approach to confidence estimation for reasoning LLMs. Agreement between an anchor response and an independent probe provides a Bernoulli observation of latent self-consistency. These observations either recalibrate an existing confidence score or train a score predictor followed by held-out isotonic recalibration. The method requires only one response at deployment. The paper transfers a binary-isotonic calibration guarantee to latent self-consistency and evaluates correctness calibration, discrimination, distribution shift, and selective prediction across 36 model–dataset combinations.

Critical strengths

The method makes effective use of inexpensive agreement supervision. The identity E[B∣U]=E[S∣U]E[B\mid U]=E[S\mid U] explains why accurate self-consistency estimates for every calibration query are unnecessary. Separating score learning from held-out recalibration produces a simple pipeline with low sampling requirements. The paper also clearly distinguishes its guarantee for self-consistency from calibration with respect to correctness.

The experiments establish usefulness beyond a reduction in ECE alone. Table 1 reports improvements in ECE, Brier score, and, for BASC-L with k=5k=5, AUROC relative to token probabilities. Distribution-shift and selective-prediction experiments provide additional evidence that the learned scores are informative. The comparison with test-time self-consistency also transparently shows the discrimination sacrificed in exchange for single-response deployment.

Critical weaknesses

The practical boundary between self-consistency calibration and correctness calibration needs further characterization. Equation 7 and Appendix A.1 correctly retain an uncontrolled discrepancy between these targets. This is an acknowledged limitation rather than an error in the guarantee. However, the empirical analysis does not sufficiently characterize the confidently repeated errors for which agreement becomes a poor correctness proxy. This matters for interpreting how far the results support reliable deployment beyond the evaluated settings.

The comparison with continuous-target predictors does not fully isolate the source of the calibration advantage. Figure 3(a) compares two-stage BASC with single-stage ridge predictors. Figure 3(b) applies the same second-stage recalibration to ridge-based scores, but reports AUROC rather than ECE and Brier score. Consequently, the large calibration advantage in Figure 3(a) partly reflects access to held-out recalibration. A matched comparison would clarify the additional benefit of binary/binomial score learning over a shared recalibration stage.

Questions for authors

Can you report ECE and Brier score for RR-SC followed by the same held-out isotonic stage, with matched total decoding budgets and explicitly stated allocations between score learning and recalibration?

Using the existing evaluation pools, can you quantify the correctness–self-consistency discrepancy by task or confidence bin, particularly for highly self-consistent but incorrect answers? This would clarify the main practical limitation without requiring additional benchmarks.

At the actual Stage-2 sample sizes, does the finite-sample result yield an informative numerical bound? If not, please distinguish the theoretical calibration guarantee from a practically informative finite-sample certificate.

3 comments
AuthorsOpenReview ↗

author rebuttal

We thank the reviewer for taking the time to review our submission and offer feedback. In response to the concerns raised in the review, we have added new experiments and analyses. Please see below.

Q1: Results for RR-SC followed by the same held-out isotonic stage

We have added BASC-RR-SC to Figure 3(a), which applies our binary isotonic recalibration to the ridge-regression self-consistency predictor, with Brier scores in Appendix Figure 11. We use identical data splits and decoding budgets for BASC-RR-SC and BASC-L: 200 Stage 1 queries with kk responses each, followed by 200 Stage 2 queries with two responses each, for a total of 200k+400200k+400 decodes.

BASC-RR-SC achieves comparable ECE and Brier scores to BASC-L at matched budgets, demonstrating that our binary recalibration framework is effective with different Stage 1 scoring functions. However, BASC-L's binomial-logistic Stage 1 achieves higher AUROC at small kk (Figure 3(b)), demonstrating an additional advantage in discrimination under limited sampling (which is also reflected in a slightly better Brier score). We have clarified these points in Section 3.1.

Q2: Quantifying the correctness–self-consistency discrepancy

Thank you for this constructive feedback, we agree that understanding this discrepancy is key to the reliable deployment of BASC.

To address this concern, we have added a detailed decomposition of the upper bound on correctness calibration error in Appendix D.1.2 (Tables 8–9), using common confidence bins across all 36 model/dataset combinations. We find that BASC-L is well calibrated to self-consistency (ECE=0.044), while the self-consistency/correctness discrepancy (ECE=0.069) is the larger component of the correctness calibration error bound. This discrepancy is particularly pronounced on QA (0.103) vs. math (0.034). Finally, we also directly examine highly self-consistent but incorrect answers (Figure 13), finding that 15–30% of answers with estimated self-consistency at least 0.9 are incorrect for several QA model/dataset combinations. These results clarify when self-consistency is a poor proxy for correctness and the practical limits of BASC's guarantee.

Q3: Informativeness of the finite-sample bound

We agree that the theoretical guarantee does not provide a practically informative numerical certificate at our calibration sample sizes. At Stage-2 calibration set size of 200 or 400, the high-probability rate factors are approximately 1.19 and 1.10, respectively (with δ=0.05\delta=0.05), before multiplication by a constant factor. The guarantee establishes distribution-free convergence to calibration with respect to latent self-consistency, rather than a tight numerical bound at our experimental sample sizes. We have explicitly clarified this distinction in the revised Limitations section.

Note: API-only experiments

At the request of Reviewer MYqJ we have added additional experiments with GPT-5-mini to Appendix D.1.1, further demonstrating that BASC extends to API-only models.

Reviewer_cMvcOpenReview ↗

Follow-up on author response

Thank you for the detailed response. The matched-budget BASC-RR-SC comparison addresses my concern about attribution: the reported ECE/Brier results are comparable, while BASC-L improves discrimination at small k. The decomposition and high-self-consistency error rates also clarify the gap between self-consistency and correctness. Finally, the reported rate factors make the limited numerical usefulness of the finite-sample bounds at 200/400 calibration samples explicit.

These responses largely address my three questions. I continue to view the proxy mismatch and loose bounds as limitations, rather than theoretical errors, and find the empirical results valuable. Please retain these qualifications in the paper.

AI assistance was used to organize the rebuttal evidence and draft this follow-up.

Thank you, we're glad that our additional experiments and analyses have largely addressed your concerns. We will definitely keep the relevant analyses and qualifications in the paper. If you have any remaining questions or concerns, we would be happy to address them. Otherwise, given that your main concerns have been addressed, we would appreciate it if you would consider raising your overall score.

Reviewer_MYqJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a nice simple method for calibrating LLM confidence scores using unlabeled examples, with some theoretical justification and with a range of experimental results on various Q/A datasets with a variety of modern LLMs. The experimental results show, that (i) self-consistency is correlated with correctness (agreeing with prior work), and (ii) the proposed approach BASC-L appears to improve calibration and AUROC measures, for the datasets and models and baselines used in the experiments

Critical strengths
  • the paper addresses an important problem, that of calibration of LLM confidence

  • the method is simple to implement and label-efficient: this gives it a chance of having impact in real-world problems

  • the experiments are comprehensive

  • the paper is clearly written

Critical weaknesses

The baselines seems a little thin. Would it make sense to also include methods like DACA (Luo et al., 2025) and BaseCal (Tan et al., 2026) and Self-Calibration (Huang et al., 2025)? And ConfTuner (Li et al., NeurIPS 2025) could perhaps be included as another supervised alternative: and/or you could try a variant of ConfTuner in unsupervised mode, using agreement pseudo-labels instead of correctness labels.

Questions for authors

Well, the instructions say to only ask questions whose answers could materially affect your assessment, but I'm not sure I have any of these.

However, i do have a number of suggestions that the authors might find useful for later improving the paper - and since I don't see anywhere else to enter this information, I will enter this here.

  • terminology in places could be clearer. For example, line 115, make it clear that "correctness label" is referring to Z in {0,1}, and is different from the true label. Also, make it clearer how you get from "y" to Z: there must be some true label (or value) y_true implicit here (??).... and Z = I(y = y_true)?? Or is \hat{y} a label that is extracted (somehow) from the string y, and Z = I(\hat{y} = y_true). This level of detail is important to indicate the scope of the approach that follows.

  • the text on lines 118 to 122 could be clearer: "For an anchor response y = (r, a) to prompt x, let Y ′ = (R′,A′) ∼ ˜pθ(· | x) be an independent response drawn from a fixed probe sampling distribution ˜pθ. While the anchor and probe distributions may be the same, separating them allows low-entropy decoding at deployment while using more diverse probe samples to assess uncertainty." Be more precise and clearer here in terms of what is meant by an "anchor response" (why is it special as an "anchor"?) and a "probe distribution". Don't necessarily assume that all readers are familiar with this terminology (and within a few months these terms may go out of style anyway): its good to define these terms clearly in a self-contained way for the reader.

  • for Related Work: there's a recently-published method called CONCORD that shares some similarities with the BASC approach: see https://openreview.net/pdf?id=ToJd6W2TJc. This was just published at the COLM 2026 conference (and doesn't seem to have an earlier arXiv version), so its fair that the authors of this paper under review would not have seen it. However, its probably worth at least discussing under Related Work in the final version of this paper and (if the authors have time and energy) would be interesting to compare it with BASC in the experimental results (but not required). I suspect BASC would perform better, but that's just a guess.

  • does aggregated reliability in Fig 2(a) make sense in general? has it been used before? could the aggregation hide important weaknesses of an approach? I didn't have time to think about this in detail but would appreciate some additional discussion on whether this way of plotting reliability might have potential flaws.

  • is there a reason not to report AUROC for the distribution shift results? (apologies if this is reported somewhere and I missed it)

  • For the final version of your paper, make your figures and text in the figures a bit bigger for the reader: the fonts are very small

  • Colors of 1-stage and BASC-L in Fig 3(a) could be differentiated better (they are too similar, to my eyes at least)

  • caption in Table 2 says "Our method offers reliable performance even when calibration data does not match test data". You may want to change this statement: your method performs better than your baselines, but the absolute values of ECE and Brier scores are still relatively high.

  • you mention "API-only"/closed-models, but don't actually evaluate this on a real closed model in the paper (unless I missed it): would be nice to add an actual experiment here with a real closed model.

  • in Limitations the authors point out that BASC only tracks self-consistency, which (as is well-known) is not necessarily correlated with calibrated uncertainty. It would be helpful to the reader to see experiments (for some model-dataset combination) that illustrates this failure mode. In fact I would find the paper more useful and valuable if the authors spent some time exploring if there are regimes where their method "breaks" (if indeed it does break). You could add this for the camera-ready version if the paper is accepted.

3 comments
AuthorsOpenReview ↗

author rebuttal

We thank the reviewer for their positive assessment and constructive suggestions. We have made several revisions to address these comments. Please see below for responses to each concern.

Q1: Baseline coverage

We did consider DACA and BaseCal, as noted in Related Work paragraph 2, but found that their underlying base-model-derived confidence signals perform poorly on our reasoning models (Appendix C). We have clarified this comparison in an updated Related Work. We have also clarified our baseline selection criteria: our primary comparisons focus on methods that require no correctness labels, no modifications to the underlying LLM, and only a single generation at deployment.

Although the full Self-Calibration method of Huang et al. (2025) falls outside this setting, we evaluate its confidence-weighted self-consistency target (RR-SSC) in Figure 3(a), finding that it performs worse than standard self-consistency (RR-SC), despite requiring a more costly heuristic target. ConfTuner similarly requires training the underlying LLM, which falls outside the scope of our setting.

Q2: Method terminology

Thank you for these suggestions. We have clarified the distinction between correctness labels and ground truth answers, as well as the definitions and roles of anchor and probe responses in Section 2.

Q3: CONCORD as related work

We agree that this work has some relation to ours, although the expensive reinforcement learning would not be applicable in our resource-constrained settings. We have highlighted this as a concurrent work in our revised manuscript.

Q4: Aggregated reliability diagrams

We believe the aggregated reliability diagram provides a useful overall view of calibration (e.g., if the model were badly calibrated in the aggregate that would be concerning), but agree that aggregation can obscure miscalibration in individual settings. We have acknowledged this limitation in Section 3.1 and pointed readers to the per-model and per-dataset reliability diagrams that were included in Appendix Figures 9–10 of the submission.

Q5: AUROC under distribution shift

We originally presented ECE and Brier score, which capture complementary aspects of probabilistic prediction quality, whereas AUROC specifically measures discrimination. For completeness, we have added AUROC results for all distribution-shift experiments in Appendix D.3 and referenced them in Section 3.2. BASC-L outperforms the unsupervised single-generation baselines, although the discrimination gains are smaller than the calibration gains.

Q6: Figure legibility

We have enlarged the main paper figures and their text to improve readability, and changed the color of the 1-stage baseline in Figure 3(a).

Q7: Table 2 caption

We agree and have revised the caption to describe the relative performance improvements without overstating absolute reliability under distribution shift.

Q8: API-only and closed-model evaluation

We thank the reviewer for this suggestion. We have updated our paper with experiments using GPT-5-mini on three QA datasets (Appendix D.1.1). We compare two variants against raw verbalized confidence (VC): BASC-VC, which directly recalibrates VC using binary agreement, and BASC-L, which first learns a score before recalibration. Both BASC variants improve ECE across all three datasets and reduce average Brier score relative to raw verbalized confidence. BASC-VC achieves the lowest average ECE and Brier score, while BASC-L also substantially improves calibration, although its Brier score is worse than verbalized confidence on TriviaQA. These results demonstrate that BASC extends to API-only models without correctness labels or access to model internals.

Q9: Failure modes of self-consistency

We agree that understanding these failure modes is important. We have added a detailed analysis in Appendix D.1.2, including a decomposition of correctness calibration error (Tables 8–9) and an examination of highly self-consistent but incorrect answers (Figure 13). We find that BASC is well calibrated to self-consistency (error of 0.044), but that self-consistency can be a less reliable proxy for correctness on QA tasks than Math. We now discuss these findings in Section 3.1 and Limitations.

Thanks for the detailed response. I very much appreciate the various clarifications and additional experiments.

Thank you, we are glad that you found our clarifications and additional experiments helpful. If you have any remaining questions or concerns, we would be happy to address them during the remaining discussion period. If your main concerns have been addressed, particularly regarding baseline selection and coverage, we would appreciate it if you would consider raising your overall score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.