- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors tackle the problem of black-box LLM confidence calibration. They propose a method called PairScore, which prompts the LLM with a pair of two answers (to different questions) and asks the LLM to verbalize its confidence on each one. It does this for 5N pairs where there are N answered questions, and computes an answer’s confidence as the average of the 5 confidences for that answer. They show that this pairwise scoring method often outperforms the baselines AbsScore (pointwise scoring) and PairChoice (pairwise ranking).
Critical strengths
- Proposes a well-motivated method PairScore that addresses limitations of the baselines AbsScore and PairChoice, and shows better results.
Critical weaknesses
- Missing baselines severely limit the significance of the work. How does PairScore compare with self-consistency (Xiong et al., 2024) and DINCO (Wang and Stengel-Eskin, 2026)?
References:
Xiong et al., 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.
Wang and Stengel-Eskin, 2026. Calibrating Verbalized Confidence with Self-Generated Distractors.
Questions for authors
None
3 comments
Thank you for taking the time to run additional experiments to address my comments.
The implementation of self-consistency seems to be off. Self-consistency examines the consistency among sampled generations (Xiong et al., 2024), rather than sampled verifications of a fixed answer as you do here. The baselines SC and DINCO are also evaluated at a less than typical budget; they would definitely improve with more budget, but it's unclear whether the same is true for the proposed PairScore. Nonetheless, I do see that under the considered setting of limited compute and output tokens costing about twice as much input tokens, PairScore may be competitive with the baselines (although the nonstandard implementation of SC makes this unclear). I will increase my score to a 2.
Thank you for revisiting the results and increasing your score.
We agree that verification sampling differs from generation-based self-consistency. As specified in Section 5.1 and Appendix B.2, our standalone baseline adapts the vote to binary verification. This evaluates consistency in judging a supplied claim, rather than consistency among independently generated answers. Our DINCO-NVC-SC evaluation includes fresh answers or passages, although it does not isolate generation-based SC as a standalone baseline.
We appreciate your recognition of PairScore’s competitiveness under limited compute. Our assumed 2–2.5× output-token price premium is modest relative to commercial API pricing. For example, Claude models charge five times as much for output tokens as for input tokens. This underscores the practical relevance of the setting.
Author Response
Thank you for the feedback. Our response is as follows.
W1: Self-consistency and DINCO baselines
Our focus is on studying the two elicitation mechanisms for confidence: choice, and score-based. Experiments presented in the paper are focused on highlighting the differences between them. However, based on your suggestion, we have also included comparisons with self-consistency (Xiong et al., 2024) and DINCO (Wang and Stengel-Eskin, 2026) in the claim verification settings. We discuss the setup and results briefly here. Please see Section 5.1 and Figure 3 for further details.
Setup
We test whether PairScore ranks claims better at comparable inference cost. We evaluate all methods on the same claims across six models and four datasets. We measure ranking quality with AUROC and evaluate the confidence probabilities with Brier score and ECE.
Self-consistency uses the fraction of sampled true-or-false judgments labeled true. For DINCO-NVC, we use the authors’ numerical variant of normalized verbalized confidence (NVC). The model scores the claim and its generated distractors separately. NVC uses the distractor scores to normalize the claim’s confidence, giving less weight to redundant distractors and more weight to those that contradict the claim. DINCO-NVC-SC averages NVC with a self-consistency score from fresh generations. Generating and checking those samples adds to the cost.
We price input and output tokens separately and apply the same rates to every method on a given model. Depending on the model family, output tokens cost 2–2.5 times as much as input tokens. We set the cost of five AbsScore calls per claim to 1×.
Results
The first four rows compare the methods at their tested budgets closest to 1×. For self-consistency, this corresponds to four true-or-false judgments per claim. We also show the two lowest DINCO-NVC budgets. The PairScore result at 1.59× is the closest cost match to DINCO-NVC with up to two distractors (1.53×).
At similar cost, PairScore ranks claims better than self-consistency and DINCO-NVC. It also gives lower Brier score and ECE in both comparisons. Figure 3 in Section 5.1 shows how the three metrics change across the tested budgets.
DINCO-NVC-SC gives better calibration at a higher cost. At the authors’ standard setting of five distractors and five consistency samples, it costs 11.18× the AbsScore reference. Generating and checking the consistency samples accounts for about four-fifths of that estimated cost. DINCO-NVC-SC reaches 77.06 mean AUROC, with a lower Brier score (0.176) and ECE (0.143) than PairScore. Wang and Stengel-Eskin, Sections 2.3 and 3.1
Overall, the significance of our work is in showing how to get more informative confidence estimates from pairwise comparisons. Shrivastava et al. (2025) opened this direction by asking which of two items a model was more confident in as an alternative to scoring items in isolation. We build on that idea by asking what a choice leaves out. When nearly every claim loses to a clearly true partner, for example, the resulting preferences do little to distinguish true claims from false ones. PairScore keeps the comparison context and lets the model express how confident it is in each claim. In our claim-verification experiments, PairScore ranks claims better on average than PairChoice at comparable cost.