acceptodds
Under review as a conference paper at ICLR 2027

On Calibration of Large Language Models: From Response To Capability

Abstract

Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce **capability calibration, a new evaluation framework** for measuring how well query-level confidence aligns with a model’s expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Confidence Calibration, Confidence Estimator Evaluation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GAJeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies capability calibration: estimating a model's probability of answering a query correctly under its generation distribution, rather than estimating the correctness of a particular generated response. It formalizes this target, compares confidence estimators, and investigates applications to pass@k\mathrm{pass}@k forecasting and inference-budget allocation. The idea is reasonable and intuitive, but my main concerns are how much the framework adds beyond earlier query-success prediction and adaptive sampling, and whether the probability estimates are sufficiently robust and precise to support the practical forecasting claims.

Critical strengths
  1. Clear reporting of estimator limitations. The paper reports poor learned pass@k\mathrm{pass}@k forecasts and out-of-domain transfer failures. These results identify where the estimators remain unreliable and expose the limits of the proposed applications.
Critical weaknesses
  1. Limited incremental conceptual contribution. Previous literature has explored related ideas (Kadavath et al.; Damani et al.), which the paper acknowledges. I find the claimed conceptual advance insufficiently established. Under the response-calibration definition on p. 2, S=μ(X)S = \mu(X) is also response-calibrated. Comparing Oracle-RC with Oracle-CC therefore does not establish that all response-calibrated scores are unsuitable for pass@k\mathrm{pass}@k forecasting, as the broader claim on p. 5 suggests. The evaluation framework remains useful, but its conceptual scope needs qualification.

  2. Practical forecasting remains insufficiently supported. The sampling-uncertainty and aggregate-stability analyses do not establish precise query-level probabilities near zero before answer generation or reliable large-kk forecasts. Learned forecasts show substantial errors in several settings. Other settings perform better and the allocation gains are useful, but reliable forecasting across the evaluated settings remains unestablished.

Questions for authors
  1. Calibration definitions and evaluation precision. How does the p. 5 assertion reconcile with the p. 2 definition, under which S=μ(X)S = \mu(X) is response-calibrated? The query-only comparison uses repeated labels for CC and a single label for RC. With fixed query-only scores evaluated on the same queries, repeated labels and weights, their Brier scores differ by an estimator-independent term and give identical rankings. Does the observed selection advantage therefore reflect improved precision from repeated labels?

  2. Independent forecasts and budget assumptions. Do Oracle-CC and the reference pass@k\mathrm{pass}@k curves use independent response pools? What uncertainty do the existing draws imply for rare-success and large-kk forecasts? Although confidence-estimation costs are discussed, does the allocation budget include confidence-generation samples and their costs? What success-verification or answer-selection information is assumed, given that generating a correct answer does not establish that the system can identify it?

5 comments

Thank you for your review. We address your concerns below.

Weakness 1

Under the response-calibration definition on p. 2, S=μ(X)S = \mu(X) is also response-calibrated.

In practice, response calibration is conditioned on the correctness of one generated output C(x,yi)C(x,y_i), which is binary, so a perfectly capability-calibrated confidence estimator is not guaranteed to be response-calibrated unless the model is deterministic.

Comparing Oracle-RC with Oracle-CC therefore does not establish that all response-calibrated scores are unsuitable for forecasting, as the broader claim on p. 5 suggests.

We agree that this does not rule out all response-calibrated scores. However, the optimal response-conditioned estimator under the RC is Oracle-RC. As shown in Table 2, even a non-oracle CC-trained probe can outperform Oracle-RC, demonstrating that CC is a more suitable evaluation target.

The evaluation framework remains useful, but its conceptual scope needs qualification.

We agree that the concept of “expected accuracy” is not new, but prior works have only considered it in different application-specific formulations. Our formulation unifies separate fields and applications into one objective: accurately estimating query-level confidence. This unification is operational rather than merely terminological, since capability calibration provides a practical target for future confidence estimators to pursue. By hill-climbing this objective, increasingly capability-calibrated confidence estimators emerge, enabling improvements across all of these applications.

Weakness 2

  • First, our primary contribution is an evaluation framework, not a new confidence estimator that guarantees accurate forecasting. The proposed capability calibration points out the limitations of current confidence estimators. We explicitly acknowledge the limitations of existing estimators in the Conclusion. These limitations motivate developing stronger CC estimators rather than undermine CC as an evaluation framework.
  • Second, the sampling analyses establish the reliability of our evaluation, not the precision of individual learned estimators.

Question 1.1

How does the p. 5 assertion reconcile with the p. 2 definition,...?

As clarified in the earlier response, RC conditions on input xx and a sampled response yy. Thus, for applications that require query-level confidence rather than response-level confidence, RC is a misaligned evaluation framework.

Question 1.2

The query-only comparison uses repeated labels for CC and a single label for RC. With fixed query-only scores evaluated on the same queries, repeated labels and weights, their Brier scores differ by an estimator-independent term and give identical rankings. Does the observed selection advantage therefore reflect improved precision from repeated labels?

  • For a query-level confidence score, to the best of our knowledge, no one has evaluated RC with repeated responses and taken the expectation (average) of the loss. Even if one does, Theorem 1 shows that two evaluation losses are inequivalent.
  • Second, RC’s label is a single response’s correctness, which is binary. For query-level confidence of a stochastic model, which aims to estimate “how likely can the LLM answer this query correctly?”, the binary correctness of a response is generally the wrong target.
  • The selection differences in Appendix D.3 reflect the higher variance of single-response evaluation, which only happens in response calibration. Moreover, this does not negate the distinction between RC and CC as evaluation targets, particularly for response-conditioned estimators.

Question 2.1

Do Oracle-CC and the reference curves use independent response pools?

Oracle-CC is calculated based on the samples of the reference curves; that is why it’s called Oracle-CC. The Oracle-CC baseline is a sanity check.

Question 2.2

What uncertainty do the existing draws imply for rare-success and large-forecasts?

While our experiments establish stable calibration evaluations (Appendix B.3), precise rare-success and large-k forecasts may require additional samples. This is a limitation of finite-sample estimation rather than the CC framework itself.

Question 2.3

does the allocation budget include confidence-generation samples and their costs?

Verbalized confidence does consume the sampling budget. We have updated the results in the revision. While verbalized confidence as the budget allocator’s performance is weak in low-budget scenarios, it still outperforms uniform allocation in high-budget scenarios.

Question 2.4

What success-verification or answer-selection information is assumed, given that generating a correct answer does not establish that the system can identify it?

We follow the answer verification procedures of these datasets. Verifier quality can affect the calibration target, but this limitation is shared by both RC and CC, as both rely on correctness evaluation.

Thank you for the clarifications, particularly regarding Oracle-CC’s role as a sanity check and the reported revision to budget accounting. I also appreciate the distinction between an evaluation framework and an accurate estimator. However, my main concerns about the contribution and insight remain:

  • Dependence on a reliable correctness criterion. CC assumes that correctness can be meaningfully defined and reliably evaluated. This is reasonable for some benchmarks but does not hold for many broader tasks. Its interpretation as model capability therefore needs to remain qualified by the evaluation criterion.

  • Limited additional conceptual insight. Where reliable correctness evaluation is available, I remain unclear what substantive understanding CC adds beyond existing query-level success-probability estimation. The applications to repeated sampling and budget allocation illustrate the usefulness of that probability, but do not sufficiently establish the additional value of the proposed framework.

  • A gap between the demonstrated predictor and the broader capability interpretation. The experiments primarily show that a probe trained for a particular model can use query-stage hidden states to estimate success rates. Although predicting an average over future sampling trajectories is well-defined, it remains unclear how reliably these representations support such prediction across settings. Their dependence on model parameters does not by itself establish a general ability to infer model capability. The current results therefore provide limited support for the broader predictive promise.

I do not expect the paper to solve general capability prediction. My remaining concern is whether, within its demonstrated scope, the framework provides sufficiently new conceptual or empirical insight beyond organizing an existing prediction target.

AuthorsOpenReview ↗

Response to follow questions

Thank you for your timely response. We address your follow-up questions below.

Question 1

Dependence on a reliable correctness criterion.

  • We agree that measuring CC depends on verifying response correctness. However, as clarified earlier, RC also relies on correctness verification. We acknowledge that reliable correctness verification remains an ongoing challenge for robust confidence calibration, but it is beyond the scope of this work. This work aims to propose CC, which is a useful calibration framework for LLMs.
  • We respectfully clarify that CC is an evaluation framework, not a method for inferring general model capability. We use the term “capability” calibration to distinguish it from “response” calibration: CC calibrates confidence estimates against the probability of correctness across an LLM's outputs, which reflects its capability, rather than the correctness of a single response.

Question 2

Limited additional conceptual insight.

I remain unclear what substantive understanding CC adds beyond existing query-level success-probability estimation.

My remaining concern is whether, within its demonstrated scope, the framework provides sufficiently new conceptual or empirical insight beyond organizing an existing prediction target.

  • For the LLMs downstream applications, yes, several applications are estimating query-level success probability. However, to the best of our knowledge, this is the first work that formulates expected accuracy as the calibration target for LLMs. We have discussed why the proposed framework matters to the downstream applications in our earlier response to Weakness 1.
  • For the confidence calibration community, we believe that setting the right evaluation is more important than developing new methods that target a wrong goal. Section 4 shows the importance of query-level confidence, which also demonstrates why hill-climbing the right evaluation metric is important.
  • We believe that systematically evaluating existing estimators under an important calibration objective is itself a meaningful contribution to the community. Guo et al. (2017) [1] provide a relevant precedent: their systematic empirical analysis revealed that modern neural networks are often poorly calibrated and identified an existing technique, temperature scaling, as an effective method. Similarly, we provide a systematic analysis of capability calibration and identify linear probing and isotonic regression as effective existing methods.

While some applications implicitly estimate expected accuracy, our work provides a generalizable framework that is independent of these applications.

  • (Section 4.1) The existing pass@k prediction work, Kazdan et al. (2025), estimates expected accuracy by sampling a subset of outputs. However, it does not connect this task to confidence calibration or recognize that linear probing, an established confidence estimator, can simulate the scaling without sampling.
  • (Section 4.2) Damani et al. (2025) estimate expected accuracy from LLM hidden representations for inference budget allocation. However, they do not connect this task to confidence calibration or recognize that verbalized confidence, a established confidence estimator, can perform well even for black-box LLMs.
  • Other applications mentioned in Section 4.3, such as model routing and query refinement, may naturally benefit from capability calibration.

This evidence demonstrates why a unified framework for query-level confidence estimation and evaluation is useful for downstream applications of LLMs.

Question 3

A gap between the demonstrated predictor and the broader capability interpretation.

As clarified earlier, capability calibration does not aim to measure a model’s capability. Section 5 aims to test the practical achievability of CC, so we test the existing confidence estimators, including the linear probe. Since our main contribution is proposing a new calibration framework, improving existing confidence estimators is not the main focus of this work. We agree that the model’s representation may not be the only source to estimate capability-calibrated query-level confidence, so as we stated in the Conclusion, we envision future work developing confidence estimators that push the frontier of capability calibration.


References

[1] On Calibration of Modern Neural Networks, ICML 2017

Reviewer_skBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The main idea of this paper is to assess confidence estimation under probabilistic sampling, which provides better insights than deterministic decoding. For this purpose, they introduce a metric called capability calibration that describes how close an existing confidence estimator is estimating the expected accuracy of a task.

Critical strengths

The problem of confidence estimation in LLMs is sufficiently well-motivated. The paper provides sufficiently many empirical experiments with interesting insights, provided in clear figures and tables that are easy to read and understand. Generally, large parts of the text read human-written and polished, which makes the paper easy to follow. Figure 1 is useful to better understand their approach.

Critical weaknesses

Weak contributions. The core technical and theoretical contributions in this paper (in particular Section 3) are rather weak. Capability calibration is just a metric that describes how close an existing confidence estimator is estimating the expected accuracy of a task, instead of the accuracy under deterministic decoding. Theorem 1 essentially states that using expectation vs deterministic decoding yield different metrics, but this is actually already clear by definition. The theorem further relates the two, but the broader impact of this theoretical statement remains unclear [in particular the impact of the statement that the probabilistic metric (CC) decouples the model's output variance from the deterministic metric (RC)]. The experiments also do not refer back to Theorem 1, it remains unclear why the Theorem is needed.

Missing clarity. The use-cases of the proposed framework remain unclear in the paper, where the evaluation framework is introduced "to stress-test existing confidence estimators" (L088), "for calibration of stochastic models" (L136), "for confidence estimation" (L483), etc. Could you please clearly state what your evaluation framework is evaluating exactly, what you use it for in your expeirments, and what you believe it can be used for beyond what you already show in your experiments? Clearly separating this would be very helpful for the reader.

Closely related, could you please elaborate why your method is called "calibration" (in capability calibration), given that your proposed approach does not actually calibrate anything?

Missing related work. This work misses a thorough related work section, the current related work section appears incomplete and mixes background & preliminaries with references to related works, which makes it more difficult to follow carefully. Can you please contextualize how your work relates to conformal calibration, e.g. [1,2]? Further, the paper contains claims without reference: "While response-level confidence and query-level confidence are equivalent for deterministic models [...], the two are fundamentally misaligned in LLMs due to the stochastic nature of text generation." (L 047) - please make clear if this is a known fact or a statement you claim/prove in your paper. Further, the mismatch between deterministic decoding and probabilistic generation has been described in the LLM evaluation literature before, see e.g. [3]. Can you describe how your empirical observation differs from [3]?

Missing limitations. The paper is missing a careful discussion of the limitations of the evaluation framework and contributions. For example, in which scenarios is your approach limited, when should we use different frameworks? What is not working so far, what could be addressed in future work? This would help the reader to better assess whether your method is useful for their use-case. The current discussion section 6 reads more like a conclusion, not a discussion, and is not sufficient.

Unpolished presentation. While large parts of the text itself appear polished, the paper's presentation would generally benefit from more polishing, some parts are currently unnecessarily difficult to follow, this includes mainly:

  • Presentation order: section 3 presents the framework, section 4 shows first experiments, section 5 instantiates the framework, and in section 5 shows experiments again. I would suggest to separate the framework from experiments more carefully. I would also rename the last section to conclusion, instead of calling it discussion.
  • Overloaded notation: Some notation is used more than once, for example N in Theorem 1 represents the number of input samples, whereas in the definition of the expected accuracy, N stands for the number of samples from the model (Line 161).
  • Unclear wording. Theorem 1 is about "calibration loss functions" (L202) and the paragraph following Theorem 1 makes statements about "strictly convex and differentiable losses". The problem is, this is the first time the paper refers to L^response and L^capability as losses. It remains unclear why they are suddenly losses with potential properties such as convexity, given that they have been introduced as simple scores in section 3.1 before.

Overall, my current recommendation is weak reject based on weaknesses 1, 2, 3 and 4.


References

[1] Mohri et al., Language Models with Conformal Factuality Guarantees, ICML 2024.

[2] Quach et al., Conformal Language Modeling, ICLR 2024

[3] Scholten et al., A Probabilistic Perspective on Unlearning and Alignment for Large Language Models, ICLR 2025.

Questions for authors

Main questions see weaknesses.

Minor: What does L stand for in the column 'cost' in table 3?

5 comments

Thank you for your review. We address your concerns below.

Weakness 1: Weak contributions

We respectfully disagree with this assessment. We believe that establishing the capability calibration framework and empirical study are significant contributions to the community.

  • Without capability calibration, the evaluation of query-level confidence will always be misaligned under existing response calibration. We believe that setting the right evaluation is more important than developing new methods that target a wrong goal.
  • While the idea of expected accuracy appears in several applications, our formulation unifies seemingly separate fields and applications into one underlying objective: accurately estimating query-level confidence. This unification is operational rather than merely terminological, since capability calibration provides a practical target for future confidence estimators to pursue. By hill-climbing this objective, increasingly capability-calibrated confidence estimators emerge, enabling improvements across all of these applications.
  • Theorem 1 shows a simple yet beautiful result: even if one takes infinite evaluations under response calibration, the results still differ from capability calibration. The difference is exactly the output variance, which is independent of the confidence estimation. Theorem 1 helps readers understand their difference.

Weakness 2: Missing clarity.

We respectfully disagree that the purpose of our framework is unclear. Confidence calibration is a well-established evaluation framework in machine learning [1], and our work follows this established formulation.

Closely related, could you please elaborate why your method is called "calibration" (in capability calibration), given that your proposed approach does not actually calibrate anything?

We believe the Introduction (L36) clearly defines “calibration” in a manner consistent with the existing literature. Confidence calibration measures how well estimated confidence aligns with the probability of correctness. It is an evaluation framework for confidence estimation of a given model, not a method.

what you believe it can be used for beyond what you already show in your experiments?

This is discussed in Section 4.3.

Weakness 3: Missing related work

Can you please contextualize how your work relates to conformal calibration?

Conformal calibration aims to construct prediction sets with statistical coverage guarantees. Confidence calibration [1] evaluates whether estimated confidence aligns with correctness probability. Therefore, capability calibration addresses a fundamentally different objective from conformal calibration.

Further, the paper contains claims without reference: "While response-level confidence and query-level confidence are equivalent for deterministic models [...], the two are fundamentally misaligned in LLMs due to the stochastic nature of text generation." (L 047)

This is a contribution of our work. Lines 49–53 explain the intuition, and Section 3.2 formally establishes the distinction between RC and CC.

Can you describe how your empirical observation differs from [3]?

[3] argues that unlearning and alignment require considering the LLM output distribution rather than only responses generated by greedy decoding. However, this is distinct from confidence calibration.

Weakness 4: Missing limitations

The limitation of capability calibration is discussed throughout the paper. As requested, we added a limitation section in the Conclusion of the revision. We have discussed when to use response calibration as the evaluation, and when to use capability calibration in Section 4.3. As it depends on the applications, we do not think this is a limitation of capability calibration.

Weakness 5: Unpolished presentation

Presentation order.

Section 3 introduces Capability Calibration (CC) and examines how it differs from existing response calibration, both theoretically and empirically. Section 4 demonstrates the practical applications of CC, while Section 5 examines the extent to which current confidence estimators can achieve CC. We believe this structure helps readers understand CC and its practical value.

Overloaded notation.

Thanks for pointing this out. We have fixed this in the revision.

Unclear wording.

We respectfully disagree that our introduction of capability calibration is unclear. Confidence calibration [1] defines losses like ECE and Brier score to quantify the performance of confidence estimations. Capability calibration uses expected accuracy as the target.

Minor Question

What does L stand for in the column 'cost' in table 3?

LL denotes the number of generated tokens, representing the autoregressive decoding cost. We have clarified this notation in the caption of Table 3.


References

[1] On Calibration of Modern Neural Networks, ICML 2017

Thank you for the discussion and provided clarifications, in particular regarding related work. While I'm still reading the discussion with other reviewers, I would have follow-up questions.

You mention [1] as work that establishes confidence calibration, but the difference is that they actually propose an approach to calibrate confidence (temperature scaling), while your paper only proposes an evaluation framework. Would temperature scaling transfer to your setting? Moreover, you also mention that expected accuracy already appears in several applications, so the impact of this work (beyond generalizing across fields) remains a bit unclear to me. Theorem 1 might help the reader to understand that there is a difference between "capability calibration" and "response calibration", but (1) that the two are different concepts is already clear by their definition, (2) you never elaborate on the impact of your statement that the difference between the two is the output variance, and (3) you never refer back to this theorem in your empirical study (for example, I believe you do not quantify the output variance in your empirical studies). Could you please comment on the impact of theorem 1 beyond helping the reader to see that there is a difference between the two concepts? This would help to better assess the main contribution.

[1] On Calibration of Modern Neural Networks, ICML 2017

AuthorsOpenReview ↗

Response to follow questions

Thank you for your timely response. We address your follow-up questions below.

Question 1

You mention [1] as work that establishes confidence calibration, but the difference is that they actually propose an approach to calibrate confidence (temperature scaling), while your paper only proposes an evaluation framework.

We respectfully disagree with this assessment.

  • First, [1] does not propose temperature scaling. As discussed in [1], they refer to Platt scaling (Platt et al., 1999) as temperature scaling, and it is used in “knowledge distillation (Hinton et al., 2015) and statistical mechanics (Jaynes, 1957).”
  • Second, [1] finds that temperature scaling is effective for response calibration; we find that (1) linear probe and (2) isotonic regression are effective for capability calibration. Please see Section 5 for more discussion. Section 5 adapts existing confidence estimators for capability calibration, while [1] tests them for response calibration.

Question 2

Moreover, you also mention that expected accuracy already appears in several applications, so the impact of this work (beyond generalizing across fields) remains a bit unclear to me.

As we mentioned in our earlier response, this work is the first to formulate expected accuracy as the calibration target for LLMs. This is a new challenge for confidence estimators, as this work is the first to identify that query-level confidence is different from response-level confidence for LLMs.

While some applications implicitly estimate expected accuracy, our work provides a generalizable framework that is independent of these applications.

  • (Section 4.1) The existing pass@k prediction work, Kazdan et al. (2025), estimates expected accuracy by sampling a subset of outputs. However, it does not connect this task to confidence calibration or recognize that linear probing, an established confidence estimator, can simulate the scaling without sampling.
  • (Section 4.2) Damani et al. (2025) estimate expected accuracy from LLM hidden representations for inference budget allocation. However, they do not connect this task to confidence calibration or recognize that verbalized confidence, a established confidence estimator, can perform well even for black-box LLMs.
  • Other applications mentioned in Section 4.3, such as model routing and query refinement, may naturally benefit from capability calibration.

This evidence demonstrates why a unified framework for query-level confidence estimation and evaluation is useful for downstream applications of LLMs.

Question 3

Could you please comment on the impact of theorem 1 beyond helping the reader to see that there is a difference between the two concepts?

Theorem 1 is intended to theoretically characterize the distinction between RC and CC, rather than serve as a standalone contribution.

(1) that the two are different concepts is already clear by their definition,

Thank you for recognizing that Section 3.1 is clear enough, but some readers might wonder: “What if we take repeated labels for RC; will the results be the same?” (see Reviewer GAJe Question 1), so we need Theorem 1.

(2) you never elaborate on the impact of your statement that the difference between the two is the output variance

Beyond establishing their difference, it quantifies how RC Brier loss conflates query-level confidence estimation error with irreducible output variance. The output variance is independent of the confidence estimation sis_i; it is from the model’s stochasticity.

(3) you never refer back to this theorem in your empirical study

We do not claim that Theorem 1 independently leads to a new estimation method. Empirical sections aim to show the practical impact and achievability of CC. We empirically validate the differences between RC and CC in Figure 2 and Section 5.1.

This would help to better assess the main contribution.

We would like to clarify again that Theorem 1 is not the only contribution of the paper. Our main contributions are (1) defining capability calibration, (2) establishing how it differs from existing response calibration, (3) demonstrating its importance for downstream applications, and (4) evaluating the extent to which existing confidence estimators achieve it.

Reviewer_cmQEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that response-level calibration is misaligned with how LLMs are used under stochastic decoding, where single-response correctness is a noisy proxy for a model’s ability on a query. It introduces capability calibration (CC): a query-level confidence score is evaluated against the model’s expected accuracy over its output distribution. A squared-loss decomposition relates CC to response calibration (RC). CC is shown to be better suited to pass@k estimation and inference budget allocation. Verbalized confidence, P(True) and linear probes are adapted to the CC target and evaluated on three LLMs and seven datasets using Brier score, with expected accuracy estimated from 100 samples per query. I understand that probes offer the best cost–performance trade-off.

Critical strengths
  1. I believe separating capability calibration from response calibration addresses a real mismatch between how calibration is evaluated and how LLMs are used with repeated sampling. Establishing CC as a standard target could influence future evaluation practice.
  2. The decomposition showing that the gap between RC and CC loss equals the variance of response correctness explains intuitively why RC penalizes output stochasticity.
  3. I feel the result that simple probes give strong CC estimates at negligible online cost is useful for practitioners, particularly for pre-generation pass@k and budget forecasting.
Critical weaknesses
  1. The pass@k study compares only Oracle-RC, Oracle-CC and Probe. The small-sample beta-distribution method of Kazdan et al. (2025) is discussed but not evaluated. I feel this makes it hard to judge the practical gain from CC-trained estimators over existing pipelines.
  2. I believe the decomposition, while clarifying, is a direct consequence of squared loss, so the contribution rests mainly on the framing and the empirical study.
  3. Estimating expected accuracy requires about 100 samples per query (up to 24 GPU-hours per dataset, Appendix B.7). I feel this may limit how readily the framework can be adopted for new models or domains.
Questions for authors

Clarifications:

  1. How would the probe-based pass@k predictions compare with Kazdan et al. (2025) when only a small number of samples is available?
  2. Appendix B.5 shows stability across temperatures. Does this also hold across prompt variants, and is the lower instance-level correlation on GSM8K a concern for math tasks?
  3. Do trained probes transfer across models or domains without retraining?
  4. How sensitive are the CC results to the correctness verifier, especially for math and open-ended QA?

Suggestions:

  1. Semantic Entropy Probes (Kossen et al., 2024) might be a natural, inexpensive baseline for CC estimation.
  2. It might help readers if the key takeaways from Appendices B.2 and B.5 were summarized in the main text.
1 comment

Thank you for your careful reading and thorough review.

In this rebuttal, we (1) clarify the paper’s contributions and (2) conduct new experiments. We address your concerns below.

Weakness 1 & Clarification 1: Comparison with Kazdan et al. (2025)

The small-sample beta-distribution method of Kazdan et al. (2025) is discussed but not evaluated.

How would the probe-based pass@k predictions compare with Kazdan et al. (2025) when only a small number of samples is available?

  • To clarify, Section 4 aims to show why capability calibration is the appropriate goal for these applications, not to claim that current confidence estimation methods can fully achieve it.
  • We compare linear probe with the method of Kazdan et al. (2025) in the revised paper’s Appendix D.1.3. Since linear probe does not require samples like the method in Kazdan et al. (2025), we use Bayes ’ rule to update its pass@k simulation. The results show that there is no clear winner. The superior pass@k simulation performance of well-calibrated probes demonstrates the potential of capability-calibrated confidence estimators in these applications.

Weakness 2: Main contribution

I believe the decomposition, while clarifying, is a direct consequence of squared loss, so the contribution rests mainly on the framing and the empirical study.

  • Decomposition shows a simple yet beautiful result: even if one takes infinite evaluations under response calibration, the results still differ from capability calibration. The difference is exactly the output variance due to stochasticity. We discuss CC vs RC beyond square loss in Appendix A.3. These theoretical results show that CC and RC are fundamentally different, and we discuss their practical implications in Section 4.
  • We believe that establishing the capability calibration framework and empirical study are significant contributions to the community. Without capability calibration, the evaluation of query-level confidence will always be misaligned under response calibration. Our formulation unifies seemingly separate fields and applications into one underlying objective: accurately estimating query-level confidence. This unification is operational, since capability calibration provides a practical target for future confidence estimators to pursue. By hill-climbing this objective, increasingly capability-calibrated confidence estimators emerge, enabling improvements across all of these applications.

Weakness 3: Obtaining CC’s target

Estimating expected accuracy requires about 100 samples per query... I feel this may limit how readily the framework can be adopted for new models or domains.

As discussed in Appendix B.1, expected accuracy remains largely stable with fewer than 100 samples. Appendix B.3 also shows that the ranking of confidence estimators remains stable with fewer samples. We use 100 samples per query in the main results to ensure rigorous evaluation, not because of a limitation of the framework.

Clarification 2: Stability across temperatures and prompt variants

Appendix B.5 shows stability across temperatures. Does this also hold across prompt variants, and is the lower instance-level correlation on GSM8K a concern for math tasks?

CC is defined under a fixed inference configuration, including the prompt and decoding settings. While prompt variations may affect expected accuracy, these settings are typically determined before deployment and remain fixed afterward. If the configuration changes, capability estimates should be re-evaluated. Regarding GSM8K, despite lower rank correlations, the rankings are stable for confidence estimators, which is important for selecting good confidence estimators.

Clarification 3: Linear probes transferability

Do trained probes transfer across models or domains without retraining?

Please see Appendix C.4 for details. Linear probes transfer within the same domain. Different models have different activation sizes and target distributions (Table 8), so probes trained on one model do not transfer to another.

Clarification 4: Sensitivity to correctness verifier

How sensitive are the CC results to the correctness verifier, especially for math and open-ended QA?

Verifier quality can affect the calibration target, but this limitation is shared by both RC and CC, as both rely on correctness evaluation.

Suggestion 1: A useful confidence estimator

Semantic Entropy Probes (Kossen et al., 2024) might be a natural, inexpensive baseline for CC estimation.

Thanks for pointing this out. We have included this work in the related work section (Appendix C.2). We welcome any confidence estimators to pursue the proposed capability calibration, which is an evaluation framework.

Suggestion 2: Summarizing takeaways

It might help readers if the key takeaways from Appendices B.2 and B.5 were summarized in the main text.

Thanks for the suggestion. We have updated Section 5.1 and Section 5.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.