- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies capability calibration: estimating a model's probability of answering a query correctly under its generation distribution, rather than estimating the correctness of a particular generated response. It formalizes this target, compares confidence estimators, and investigates applications to forecasting and inference-budget allocation. The idea is reasonable and intuitive, but my main concerns are how much the framework adds beyond earlier query-success prediction and adaptive sampling, and whether the probability estimates are sufficiently robust and precise to support the practical forecasting claims.
Critical strengths
- Clear reporting of estimator limitations. The paper reports poor learned forecasts and out-of-domain transfer failures. These results identify where the estimators remain unreliable and expose the limits of the proposed applications.
Critical weaknesses
-
Limited incremental conceptual contribution. Previous literature has explored related ideas (Kadavath et al.; Damani et al.), which the paper acknowledges. I find the claimed conceptual advance insufficiently established. Under the response-calibration definition on p. 2, is also response-calibrated. Comparing Oracle-RC with Oracle-CC therefore does not establish that all response-calibrated scores are unsuitable for forecasting, as the broader claim on p. 5 suggests. The evaluation framework remains useful, but its conceptual scope needs qualification.
-
Practical forecasting remains insufficiently supported. The sampling-uncertainty and aggregate-stability analyses do not establish precise query-level probabilities near zero before answer generation or reliable large- forecasts. Learned forecasts show substantial errors in several settings. Other settings perform better and the allocation gains are useful, but reliable forecasting across the evaluated settings remains unestablished.
Questions for authors
-
Calibration definitions and evaluation precision. How does the p. 5 assertion reconcile with the p. 2 definition, under which is response-calibrated? The query-only comparison uses repeated labels for CC and a single label for RC. With fixed query-only scores evaluated on the same queries, repeated labels and weights, their Brier scores differ by an estimator-independent term and give identical rankings. Does the observed selection advantage therefore reflect improved precision from repeated labels?
-
Independent forecasts and budget assumptions. Do Oracle-CC and the reference curves use independent response pools? What uncertainty do the existing draws imply for rare-success and large- forecasts? Although confidence-estimation costs are discussed, does the allocation budget include confidence-generation samples and their costs? What success-verification or answer-selection information is assumed, given that generating a correct answer does not establish that the system can identify it?
5 comments
Thank you for the clarifications, particularly regarding Oracle-CC’s role as a sanity check and the reported revision to budget accounting. I also appreciate the distinction between an evaluation framework and an accurate estimator. However, my main concerns about the contribution and insight remain:
-
Dependence on a reliable correctness criterion. CC assumes that correctness can be meaningfully defined and reliably evaluated. This is reasonable for some benchmarks but does not hold for many broader tasks. Its interpretation as model capability therefore needs to remain qualified by the evaluation criterion.
-
Limited additional conceptual insight. Where reliable correctness evaluation is available, I remain unclear what substantive understanding CC adds beyond existing query-level success-probability estimation. The applications to repeated sampling and budget allocation illustrate the usefulness of that probability, but do not sufficiently establish the additional value of the proposed framework.
-
A gap between the demonstrated predictor and the broader capability interpretation. The experiments primarily show that a probe trained for a particular model can use query-stage hidden states to estimate success rates. Although predicting an average over future sampling trajectories is well-defined, it remains unclear how reliably these representations support such prediction across settings. Their dependence on model parameters does not by itself establish a general ability to infer model capability. The current results therefore provide limited support for the broader predictive promise.
I do not expect the paper to solve general capability prediction. My remaining concern is whether, within its demonstrated scope, the framework provides sufficiently new conceptual or empirical insight beyond organizing an existing prediction target.
Response to follow questions
Thank you for your timely response. We address your follow-up questions below.
Question 1
Dependence on a reliable correctness criterion.
- We agree that measuring CC depends on verifying response correctness. However, as clarified earlier, RC also relies on correctness verification. We acknowledge that reliable correctness verification remains an ongoing challenge for robust confidence calibration, but it is beyond the scope of this work. This work aims to propose CC, which is a useful calibration framework for LLMs.
- We respectfully clarify that CC is an evaluation framework, not a method for inferring general model capability. We use the term “capability” calibration to distinguish it from “response” calibration: CC calibrates confidence estimates against the probability of correctness across an LLM's outputs, which reflects its capability, rather than the correctness of a single response.
Question 2
Limited additional conceptual insight.
I remain unclear what substantive understanding CC adds beyond existing query-level success-probability estimation.
My remaining concern is whether, within its demonstrated scope, the framework provides sufficiently new conceptual or empirical insight beyond organizing an existing prediction target.
- For the LLMs downstream applications, yes, several applications are estimating query-level success probability. However, to the best of our knowledge, this is the first work that formulates expected accuracy as the calibration target for LLMs. We have discussed why the proposed framework matters to the downstream applications in our earlier response to Weakness 1.
- For the confidence calibration community, we believe that setting the right evaluation is more important than developing new methods that target a wrong goal. Section 4 shows the importance of query-level confidence, which also demonstrates why hill-climbing the right evaluation metric is important.
- We believe that systematically evaluating existing estimators under an important calibration objective is itself a meaningful contribution to the community. Guo et al. (2017) [1] provide a relevant precedent: their systematic empirical analysis revealed that modern neural networks are often poorly calibrated and identified an existing technique, temperature scaling, as an effective method. Similarly, we provide a systematic analysis of capability calibration and identify linear probing and isotonic regression as effective existing methods.
While some applications implicitly estimate expected accuracy, our work provides a generalizable framework that is independent of these applications.
- (Section 4.1) The existing pass@k prediction work, Kazdan et al. (2025), estimates expected accuracy by sampling a subset of outputs. However, it does not connect this task to confidence calibration or recognize that linear probing, an established confidence estimator, can simulate the scaling without sampling.
- (Section 4.2) Damani et al. (2025) estimate expected accuracy from LLM hidden representations for inference budget allocation. However, they do not connect this task to confidence calibration or recognize that verbalized confidence, a established confidence estimator, can perform well even for black-box LLMs.
- Other applications mentioned in Section 4.3, such as model routing and query refinement, may naturally benefit from capability calibration.
This evidence demonstrates why a unified framework for query-level confidence estimation and evaluation is useful for downstream applications of LLMs.
Question 3
A gap between the demonstrated predictor and the broader capability interpretation.
As clarified earlier, capability calibration does not aim to measure a model’s capability. Section 5 aims to test the practical achievability of CC, so we test the existing confidence estimators, including the linear probe. Since our main contribution is proposing a new calibration framework, improving existing confidence estimators is not the main focus of this work. We agree that the model’s representation may not be the only source to estimate capability-calibrated query-level confidence, so as we stated in the Conclusion, we envision future work developing confidence estimators that push the frontier of capability calibration.
References
[1] On Calibration of Modern Neural Networks, ICML 2017
Rebuttal
Thank you for your review. We address your concerns below.
Weakness 1
In practice, response calibration is conditioned on the correctness of one generated output C(x,yi), which is binary, so a perfectly capability-calibrated confidence estimator is not guaranteed to be response-calibrated unless the model is deterministic.
We agree that this does not rule out all response-calibrated scores. However, the optimal response-conditioned estimator under the RC is Oracle-RC. As shown in Table 2, even a non-oracle CC-trained probe can outperform Oracle-RC, demonstrating that CC is a more suitable evaluation target.
We agree that the concept of “expected accuracy” is not new, but prior works have only considered it in different application-specific formulations. Our formulation unifies separate fields and applications into one objective: accurately estimating query-level confidence. This unification is operational rather than merely terminological, since capability calibration provides a practical target for future confidence estimators to pursue. By hill-climbing this objective, increasingly capability-calibrated confidence estimators emerge, enabling improvements across all of these applications.
Weakness 2
Question 1.1
As clarified in the earlier response, RC conditions on input x and a sampled response y. Thus, for applications that require query-level confidence rather than response-level confidence, RC is a misaligned evaluation framework.
Question 1.2
Question 2.1
Oracle-CC is calculated based on the samples of the reference curves; that is why it’s called Oracle-CC. The Oracle-CC baseline is a sanity check.
Question 2.2
While our experiments establish stable calibration evaluations (Appendix B.3), precise rare-success and large-k forecasts may require additional samples. This is a limitation of finite-sample estimation rather than the CC framework itself.
Question 2.3
Verbalized confidence does consume the sampling budget. We have updated the results in the revision. While verbalized confidence as the budget allocator’s performance is weak in low-budget scenarios, it still outperforms uniform allocation in high-budget scenarios.
Question 2.4
We follow the answer verification procedures of these datasets. Verifier quality can affect the calibration target, but this limitation is shared by both RC and CC, as both rely on correctness evaluation.