acceptodds
Under review as a conference paper at ICLR 2027

Don’t Be Choosy: Scoring over Choosing for Verbalized LLM Confidence

Abstract

Accurate uncertainty estimation is critical for the reliable deployment of large language models (LLMs). A simple and widely used approach is to ask a model to verbalize its confidence on each item, referred to as AbsScore henceforth. However, AbsScores can be poorly discriminative, exhibiting overconfidence or collapsing to a small set of values. Recent work proposed pairwise elicitation as an alternative, asking the model which of two items it is more confident about. While effective, this formulation forces a preference even when confidences are similar and discards information about the magnitude of the confidence difference. To overcome these limitations, we introduce PairScore, a simple alternative: present two items together, but ask the model to score both rather than choose between them. This preserves graded confidence while retaining the benefits of pairwise elicitation, and the resulting scores can be aggregated directly without fitting a separate ranking model. PairScore improves confidence ranking over AbsScore and PairChoice in most claim-verification settings, while gains in best-of- response selection depend on the task. PairScore also improves calibration and reduces sensitivity to presentation order and comparison partners. We extend both formats beyond pairs: -score jointly scores items and -rank returns a full ordering of them. Scoring improves as more items share a prompt, whereas ranking peaks at small and then declines; -score matches or exceeds -rank at every tested.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Claim verification, verbalized confidence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NMzDOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors tackle the problem of black-box LLM confidence calibration. They propose a method called PairScore, which prompts the LLM with a pair of two answers (to different questions) and asks the LLM to verbalize its confidence on each one. It does this for 5N pairs where there are N answered questions, and computes an answer’s confidence as the average of the 5 confidences for that answer. They show that this pairwise scoring method often outperforms the baselines AbsScore (pointwise scoring) and PairChoice (pairwise ranking).

Critical strengths
  • Proposes a well-motivated method PairScore that addresses limitations of the baselines AbsScore and PairChoice, and shows better results.
Critical weaknesses
  • Missing baselines severely limit the significance of the work. How does PairScore compare with self-consistency (Xiong et al., 2024) and DINCO (Wang and Stengel-Eskin, 2026)?

References:

Xiong et al., 2024. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.

Wang and Stengel-Eskin, 2026. Calibrating Verbalized Confidence with Self-Generated Distractors.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Author Response

Thank you for the feedback. Our response is as follows.

 

W1: Self-consistency and DINCO baselines

Our focus is on studying the two elicitation mechanisms for confidence: choice, and score-based. Experiments presented in the paper are focused on highlighting the differences between them. However, based on your suggestion, we have also included comparisons with self-consistency (Xiong et al., 2024) and DINCO (Wang and Stengel-Eskin, 2026) in the claim verification settings. We discuss the setup and results briefly here. Please see Section 5.1 and Figure 3 for further details.

Setup

We test whether PairScore ranks claims better at comparable inference cost. We evaluate all methods on the same claims across six models and four datasets. We measure ranking quality with AUROC and evaluate the confidence probabilities with Brier score and ECE.

Self-consistency uses the fraction of sampled true-or-false judgments labeled true. For DINCO-NVC, we use the authors’ numerical variant of normalized verbalized confidence (NVC). The model scores the claim and its generated distractors separately. NVC uses the distractor scores to normalize the claim’s confidence, giving less weight to redundant distractors and more weight to those that contradict the claim. DINCO-NVC-SC averages NVC with a self-consistency score from fresh generations. Generating and checking those samples adds to the cost.

We price input and output tokens separately and apply the same rates to every method on a given model. Depending on the model family, output tokens cost 2–2.5 times as much as input tokens. We set the cost of five AbsScore calls per claim to 1×.

Results

The first four rows compare the methods at their tested budgets closest to 1×. For self-consistency, this corresponds to four true-or-false judgments per claim. We also show the two lowest DINCO-NVC budgets. The PairScore result at 1.59× is the closest cost match to DINCO-NVC with up to two distractors (1.53×).

 

Method  Cost (↓)  AUROC, % (↑)  Brier (↓)  ECE (↓)
AbsScore1.00×75.480.2300.230
PairChoice1.06×72.930.2330.199
PairScore0.95×78.550.1930.176
Self-consistency1.06×73.650.2160.206
PairScore (DINCO-NVC comparison)1.59×78.920.1900.168
DINCO-NVC (up to 1 distractor)1.35×75.030.2200.209
DINCO-NVC (up to 2 distractors)1.53×74.910.2090.196

 

At similar cost, PairScore ranks claims better than self-consistency and DINCO-NVC. It also gives lower Brier score and ECE in both comparisons. Figure 3 in Section 5.1 shows how the three metrics change across the tested budgets.

DINCO-NVC-SC gives better calibration at a higher cost. At the authors’ standard setting of five distractors and five consistency samples, it costs 11.18× the AbsScore reference. Generating and checking the consistency samples accounts for about four-fifths of that estimated cost. DINCO-NVC-SC reaches 77.06 mean AUROC, with a lower Brier score (0.176) and ECE (0.143) than PairScore. Wang and Stengel-Eskin, Sections 2.3 and 3.1

Overall, the significance of our work is in showing how to get more informative confidence estimates from pairwise comparisons. Shrivastava et al. (2025) opened this direction by asking which of two items a model was more confident in as an alternative to scoring items in isolation. We build on that idea by asking what a choice leaves out. When nearly every claim loses to a clearly true partner, for example, the resulting preferences do little to distinguish true claims from false ones. PairScore keeps the comparison context and lets the model express how confident it is in each claim. In our claim-verification experiments, PairScore ranks claims better on average than PairChoice at comparable cost.

Thank you for taking the time to run additional experiments to address my comments.

The implementation of self-consistency seems to be off. Self-consistency examines the consistency among sampled generations (Xiong et al., 2024), rather than sampled verifications of a fixed answer as you do here. The baselines SC and DINCO are also evaluated at a less than typical budget; they would definitely improve with more budget, but it's unclear whether the same is true for the proposed PairScore. Nonetheless, I do see that under the considered setting of limited compute and output tokens costing about twice as much input tokens, PairScore may be competitive with the baselines (although the nonstandard implementation of SC makes this unclear). I will increase my score to a 2.

Thank you for revisiting the results and increasing your score.

We agree that verification sampling differs from generation-based self-consistency. As specified in Section 5.1 and Appendix B.2, our standalone baseline adapts the vote to binary verification. This evaluates consistency in judging a supplied claim, rather than consistency among independently generated answers. Our DINCO-NVC-SC evaluation includes fresh answers or passages, although it does not isolate generation-based SC as a standalone baseline.

We appreciate your recognition of PairScore’s competitiveness under limited compute. Our assumed 2–2.5× output-token price premium is modest relative to commercial API pricing. For example, Claude models charge five times as much for output tokens as for input tokens. This underscores the practical relevance of the setting.

Reviewer_SdLqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PairScore, a confidence estimation approach that asks an LLM to score two items jointly rather than independently scoring (AbsScore) or performing a binary choice (PairChoice). Scores from multiple pairings are averaged. Across various models and claim-verification datasets, PairScore usually improves confidence ranking, especially for smaller models. It also explores calibration, response selection, and scoring larger groups of items. Especially, further extensions include k-score and k-rank, which include multiple items in one judgment, and the scoring-based approach is shown to be overall better.

Critical strengths
  • The proposed approach is simple and practical, it only requires changing the evaluation prompt and format.
  • The proposed method is evaluated on a wide range of models and datasets, with detailed analysis to show the patterns of different approaches.
Critical weaknesses
  • The source of the gains is still somewhat questionable. At the same call budget, PairScore obtains roughly twice as many item evaluations as AbsScore, which raises some doubts on the true source of the improvement.
  • The experiments are mainly on claim verification tasks, it is unclear whether the proposed approach can be generalized to more applications where there are needs to obtain model confidence.
Questions for authors
  • It seems that most of the explored models are relatively small, I'm wondering whether similar patterns will still hold for larger models?
  • In Figure 5, in some cases, k-score and k-rank seem to perform similarly, are there any insights on why the patterns in these cases are different to others?
  • I'm wondering if one enhances the PairChoice approach beyond binary choice, (for example, into multiple categories: A is much better than B, A is better than B, A is roughly the same as B, and so on), how will it perform?
2 comments
AuthorsOpenReview ↗

Author Response [Part 1]

Thank you for the thoughtful review. We appreciate your comments on the simplicity of PairScore. We have updated the draft.  

W1. Inference cost and the number of scores per claim

We compare the methods by pricing their input and output tokens. The first three rows use the tested budget closest to the cost of five AbsScore calls per claim (1×) for each method. In the last row, we reduce PairScore to four scores per claim on average to test whether its advantage persists when it produces fewer scores than AbsScore.

 

Method  Cost (↓)  AUROC, % (↑)  Brier (↓)  ECE (↓)
AbsScore1.00×75.480.2300.230
PairChoice1.06×72.930.2330.199
PairScore0.95×78.550.1930.176
PairScore (fewer scores per claim)0.64×77.420.2000.182

 

PairScore achieves higher mean AUROC than AbsScore (77.42% versus 75.48%) with fewer scores per claim (four versus five) and 36% lower estimated cost. Thus, obtaining more item evaluations cannot by itself explain the observed advantage. At comparable cost, PairScore also achieves higher mean AUROC and lower Brier score and ECE than both AbsScore and PairChoice. Figure 3 in the revised manuscript shows how all three metrics change with cost.

In addition, Section 2 illustrates why the format of a judgment matters. When almost every claim loses to an obviously true partner, the choices tell us little about which claims are true. PairScore can still distinguish those claims by the scores they receive. Appendix D gives the corresponding comparison with AbsScore. Claims that receive the same score in isolation can receive different scores when judged alongside another claim.

 

W2. Does PairScore help beyond claim verification?

The original submission evaluated PairScore beyond claim verification in three distinct uses of model judgment: selecting among generated answers, deciding whether studies meet review-specific inclusion criteria, and ranking research proposals against expert assessments. In response to your concern, we added a fourth application in which a small model uses its confidence to decide which questions to pass to a larger model.

In a model cascade, keeping the small model’s answer saves a call to the larger one (Chen et al., 2023, Gupta et al., 2024). Qwen3-1.7B answered 500 ARC-Challenge questions. We used each confidence ranking to defer the lowest-ranked answers first and measured the cost at the first tested deferral rate where the cascade reached 90% accuracy.

Estimated cost to reach 90% accuracy (1× = the larger model answering all 500 questions)

 

Larger model  Random routing cost (↓)  AbsScore cost (↓)  PairChoice cost (↓)  PairScore cost (↓)
Qwen3-32B0.80×0.81×0.86×0.68×
Gemma-3-27B-IT0.91×0.96×1.05×0.81×

 

PairScore reached 90% accuracy at the lowest estimated cost with both larger models—21% lower than PairChoice for Qwen3-32B and 23% lower for Gemma-3-27B-IT, including confidence elicitation. We estimate cost by weighting each model’s input and output tokens by its parameter count. Figure 7 in Section 5.3 shows the Qwen3-32B routing curves.

The original submission also evaluates three applications:

  • Selecting an answer. Each model generated eight answers to a question and selected one. On HumanEval+ and AlpacaEval, PairScore selected better answers than AbsScore for all three generators. PairScore and PairChoice each led in three of those six comparisons. On math problems, majority voting generally worked better.
  • Screening studies for systematic reviews. This task requires judging whether a study meets a particular review’s inclusion criteria, rather than whether a statement is factually true. We ranked titles and abstracts from five published reviews spanning biology, medicine, ecology and conservation, and evaluated the rankings against the reviews’ final inclusion decisions. Across two models, PairScore had higher AUROC than PairChoice in eight of ten settings, tied in one, and was lower in one.
  • Ranking research proposals. We measured how often Claude Sonnet 4.5 preferred the same proposal as the experts. When the model gave a short answer and saw each pair in one order, agreement was higher with PairScore than with PairChoice (60.7% versus 50.6%). When the model could explain its judgment and saw both orders, the two methods agreed with experts at similar rates on matched pairs (58.7% versus 57.5%).

Together, these experiments extend the evaluation from factual correctness to answer selection, review-specific relevance and expert-assessed scientific quality.

AuthorsOpenReview ↗

Author Response [Part 2]

Q1. Does the advantage persist on larger models?

We extended the claim-verification experiment to Qwen3-8B and Qwen3-14B. For each model and dataset, we used the cost of five AbsScore calls per claim as the reference and took as many complete rounds of each method as fit within 110% of that budget. The table gives mean AUROC (%) across the same four datasets.

 

Qwen3 size  AbsScore AUROC (↑)  PairChoice AUROC (↑)  PairScore AUROC (↑)
1.7B66.860.573.9
4B77.170.378.8
8B80.078.683.1
14B82.780.184.5

 

PairScore’s advantage persists beyond the smallest models: it achieves higher mean AUROC than both baselines at every Qwen3 size tested and outperforms PairChoice on all four datasets at both 8B and 14B. Figure 4 in the revised paper shows the results by dataset.

 

Q2. k-score vs k-rank?

The gap between k-rank and k-score may depend on how difficult the claims are for the model to judge. On CREAK and FEVER, both models distinguish true from false claims fairly well when they score one claim at a time (AbsScore), and k-rank comes close to k-score on average. On TruthfulQA, where single-claim AUROC is lowest, both models do better with k-score at every group size we tested.

The table gives mean AUROC (%) across Qwen3-4B (Figure 5) and Phi-4-mini (k-wise appendix). For k-rank and k-score, we average over all six group sizes.

 

Dataset  AbsScore AUROC (↑)  k-rank AUROC (↑)  k-score AUROC (↑)
CREAK91.490.191.5
FEVER82.881.884.3
TruthfulQA70.972.880.8

 

Across these settings, k-score gives comparable or better rankings on average, with more flexibility in how the model expresses confidence. For example, it can give two claims the same score when it finds them equally plausible. With k-rank, it must place one ahead of the other. Having to sort the whole list may also make ranking harder as the group grows. These patterns are consistent with task difficulty influencing the gap.

 

Q3. What if PairChoice can express the strength of its preference?

We tested your suggestion in the fixed-partner experiment in Section 2 (Figure 2). There, we compare every claim with the same statement and ask how well the resulting judgments rank true claims above false ones. If two claims both lose to that statement, PairChoice gives them the same outcome. Graded choice could still distinguish them if the model expresses a stronger preference in one comparison than in the other.

We gave the five responses equally spaced weights from 0 (strongly favor the partner) to 1 (strongly favor the claim), with 0.5 for a tie. For each of the five partners, we ranked claims by their average weight over three judgments. The table gives mean AUROC (%) across two models on CREAK and Climate-FEVER.

 

Comparison partner  PairChoice AUROC (↑)  Graded choice AUROC (↑)  PairScore AUROC (↑)
Average over all five partners68.664.683.6
“The Earth orbits the Sun” (true)58.954.384.9
“Two plus two equals five” (false)68.672.982.7

 

Grading improved on PairChoice against the clearly false partner. PairScore ranked true claims above false ones more reliably than graded choice for every partner on both models and both datasets. Allowing ties and preference strength therefore did not close the gap in this setting. Figure 2 shows the same experiment counting wins, ties, and losses.The appendix also reports the analysis using all five grades.

We also tested graded choice in the order-reversal experiment in Section 2 (Figure 1), where the model sees the same two city statements in both orders. Across five models, graded choice changed its preference more often on average than PairChoice for all three kinds of pairs (both true, both false, and one of each). We counted a change if the model preferred a different claim or moved between a preference and a tie. A weaker or stronger preference for the same claim counted as agreement.

Reviewer_NTL9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares three ways to elicit verbalized confidence from LLMs: AbsScore (score one item 0–100), PairChoice (present two items, force a binary preference, aggregate via Bradley–Terry), and the proposed PairScore (present two items, ask for a numerical score for each, aggregate by averaging). The authors argue PairChoice is order-sensitive when a tie is correct and is dominated by an extreme-confidence partner, and that PairScore avoids both while keeping graded confidence. Validation covers four claim-verification datasets with eight models (AUC-ROC, Brier/ECE), a k-wise extension (k-score vs. k-rank), best-of-N selection on seven benchmarks, and two extra ranking applications (systematic-review study selection, research-proposal ranking).

Critical strengths
  1. The Section 2 diagnostics cleanly isolate two real failure modes of PairChoice—order sensitivity when a tie is correct, and partner dominance under extreme-confidence anchors. This is the most convincing part and is directly useful for practitioners choosing an elicitation format.
  2. The k-score vs. k-rank contrast is non-obvious and practically relevant: k-score improves and plateaus up to k=20, while k-rank peaks early and degrades. This gives actionable guidance for listwise ranking and best-of-N pipelines.
Critical weaknesses
  1. The central comparison is confounded. PairScore differs from PairChoice not only in scoring vs. choosing, but also in aggregation (mean vs. Bradley–Terry), output length, parsing, and prompt wording. No ablation holds aggregation fixed, so the claim that the format itself drives the gain is not established. This matters because the format change is the paper's main selling point.
  2. The gains are inconsistent and the headline is overclaimed. PairScore falls below AbsScore(5×) in some settings by up to 7 points; best-of-N gains are task-dependent and majority vote is stronger on math/QA; the research-proposal advantage is non-significant under equal scaffolding (58.7% vs. 57.5%, CI includes zero, p=.503). The abstract does not sufficiently foreground these boundary conditions, which weakens the significance claim for ICLR.
Questions for authors
  1. Can you ablate aggregation—e.g., fit Bradley–Terry to PairScore-derived preferences, or feed PairChoice outcomes into a graded aggregation—to separate format from aggregation?
  2. At k=20, k-rank parse rate is 71% vs. 99% for k-score. How much of the k-score > k-rank gap is due to parsing/output length rather than the format itself?
  3. Given the non-significant CoT result, is PairScore's advantage regime-specific (terse, single-order, low-token prompts)? If so, should the abstract say so explicitly?
  4. How does PairScore compare to post-hoc recalibration of AbsScore (temperature scaling, isotonic) for the calibration claims, especially on TruthfulQA where ECE stays near 0.3?
  5. Does the fixed-partner robustness result replicate on the API models, or only on Qwen3-4B and Phi-4-mini?
2 comments
AuthorsOpenReview ↗

Author Response [Part 1]

Thank you for the feedback. We respond to each of your questions below.

 

W1 / Q1 (a): Does the advantage persist when aggregation is held fixed?

In the main experiment, we fit Bradley–Terry to PairChoice’s judgments and average the scores each claim receives from PairScore. But the two scores also tell us which claim the model prefers. If asking for scores gives a more reliable preference than asking for a choice, we should see a better ranking even when we aggregate those preferences in the same way.

We reanalyzed the responses from 2,500 calls per format in each setting. For PairScore, we treated the higher-scored claim as the winner and omitted ties. We then ranked claims from each format in two ways. Win rate is the fraction of comparisons a claim wins, equivalent to averaging wins (1) and losses (0). Bradley–Terry also accounts for which claims it faced. The table gives mean AUROC (%) across six models and four datasets.

 

Elicitation  Win rate AUROC (↑)  Bradley–Terry AUROC (↑)  Mean score AUROC (↑)
PairChoice72.1672.93—
PairScore77.3276.5678.92

 

PairScore’s advantage persists with aggregation held fixed: 5.16 AUROC points with win rate and 3.63 points with Bradley–Terry. Thus, the choice of aggregation rule alone does not explain the gap. Retaining the numerical scores and averaging them gave the highest mean AUROC. Figure 6 in Section 5.2 shows this comparison, and Appendix Table 7 gives the results for each model and dataset.

 

W1 / Q1 (b): Prompt wording

In the claim-verification experiments, we used opening 1 for PairChoice and opening 2 for PairScore (Appendix J). If that wording explains the gap, we would expect it to shrink when both formats use the same opening question. We therefore tested each format with both openings.

  • Prompt opening 1 asks “Which of the following statements are you more confident of being correct?”
  • Prompt opening 2 asks “How confident are you that each of the following statements is factually correct?”

With either opening, PairChoice asked for A or B and PairScore asked for a score from 0 to 100 for each claim. Each format used its own set of claim pairs, with the pairs and their order held fixed across openings. Entries are mean AUROC (%) across four models and four datasets.

 

Answer requested  Prompt opening 1 AUROC (↑)  Prompt opening 2 AUROC (↑)
PairChoice (A or B)69.4464.30
PairScore (two scores)75.4476.47

 

PairScore achieves higher mean AUROC under either shared opening question, so the original difference in opening wording does not by itself explain the advantage. Each method also performs better with its original opening.

 

W2 / Q3: Scope of the gains, best-of-N results, and the reasoning regime

We agree that evidence for PairScore is strongest when models give direct answers, particularly in claim verification.The revised abstract explicitly qualifies the claim-verification results as using “direct answers."

During the rebuttal, we added a model-cascading experiment to test whether these direct confidence judgments translate into a practical benefit: deciding which of a small model’s answers to pass to a larger one. Qwen3-1.7B answered 500 ARC-Challenge questions. We sent its lowest-confidence questions to Qwen3-32B in one experiment and to Gemma-3-27B-IT in the other. With Qwen3-32B, the cascade reached 90% accuracy by referring 50% of the questions with PairScore and 70% with PairChoice (Figure 7).

Estimated cost to reach 90% accuracy (1× = the larger model answering all 500 questions)

 

Larger model  Random routing cost (↓)  AbsScore cost (↓)  PairChoice cost (↓)  PairScore cost (↓)
Qwen3-32B0.80×0.81×0.86×0.68×
Gemma-3-27B-IT0.91×0.96×1.05×0.81×

 

PairScore reached 90% accuracy at the lowest estimated cost with either larger model. The estimate includes the confidence calls and weights each model’s input and output tokens by its parameter count.

Overall, scoring both items can give a more useful confidence ranking than asking the model to choose between them. That ranking helped identify true claims, bring relevant studies to the top of a screening list, and decide which questions to pass to a larger model. The revised abstract reflects that gains in answer selection and scientific document ranking depend on the task and how the model makes its judgments.

AuthorsOpenReview ↗

Author Response [Part 2]

Q2: Do parsing failures explain the gap between k-score and k-rank?

We reran the experiment on the same groups of 20 claims with revised ranking instructions and output limits. Across two models (Phi-4-mini and Qwen3-4B) and three datasets (CREAK, FEVER, and TruthfulQA), 88% of rankings and over 99% of score lists were complete. An incomplete ranking leaves out at least one claim. For example, if k-rank lists 19 of the 20 claims, we know the order of those 19. We compared the methods three ways.

  1. Each method’s complete replies. We keep every complete score list and every complete ranking, even when only one method answered a group in full.
  2. Same groups, complete rankings. We keep a group only if k-score gave all 20 scores and k-rank ranked all 20 claims.
  3. Same groups, partial rankings allowed. We also keep groups with all 20 scores and a usable partial ranking. If k-rank lists 19 claims, we use the order of those 19.

 

Comparison  k-score mean AUROC (↑)  k-rank mean AUROC (↑)  Difference (points)
Each method’s complete replies85.9480.485.46
Same groups, complete rankings85.7480.555.19
Same groups, partial rankings allowed85.9680.994.97

 

On the same groups with complete outputs from both methods, k-score retains a 5.19-point AUROC advantage. Allowing partial rankings leaves a 4.97-point gap. Thus, parsing failures do not fully explain the advantage at k=20 (Figure 5 and Appendix F).

 

Q4: Comparison with post-hoc recalibration

Our focus is on confidence estimates that can be used directly, since fitting a post-hoc correction requires labeled data. Following your suggestion, we tested whether calibrating AbsScore on labeled TruthfulQA claims closes the gap with PairScore.

For each model, we randomly chose 100 of the 500 TruthfulQA claims to fit the corrections and evaluated both methods on the remaining 400. We repeated this split 100 times and averaged the results across splits and the six models.

 

Correction  AbsScore ECE (↓)  PairScore ECE (↓)  AbsScore Brier (↓)  PairScore Brier (↓)
None0.3370.2440.2920.212
Temperature scaling0.2730.2540.2220.195
Isotonic regression0.0620.0650.1430.129
Platt scaling0.0550.0520.1400.124

 

On TruthfulQA, AbsScore calibrated with isotonic regression had lower ECE than raw PairScore. Applying isotonic regression or Platt scaling to both methods brings their ECE close, and PairScore retains a lower mean Brier score.

 

Q5: Fixed-partner robustness on API models

PairScore’s ranking is less sensitive to the comparison partner across all three API models on both datasets. We used the same claims and partners as in Figure 2 and ranked the claims separately against each partner. Each entry gives the mean AUROC (%) across the five partners, with the range in parentheses.

 

Dataset  API model  PairChoice AUROC (↑)  + five grades AUROC (↑)  PairScore AUROC (↑)
Climate-FEVERGPT-4o-mini63.9 (53.6–78.5)60.5 (51.3–79.4)87.3 (85.4–88.9)
Climate-FEVERGPT-4.1-nano62.0 (55.4–72.2)59.2 (54.4–63.2)80.3 (77.7–82.2)
Climate-FEVERGemini-2.5-Flash-Lite71.4 (50.0–78.7)68.0 (53.6–74.8)87.8 (87.3–89.0)
CREAKGPT-4o-mini73.0 (51.7–95.3)67.1 (50.0–92.4)96.8 (95.9–97.4)
CREAKGPT-4.1-nano76.4 (50.0–90.8)72.2 (52.1–84.7)93.5 (91.6–94.5)
CREAKGemini-2.5-Flash-Lite79.1 (50.0–90.4)70.4 (56.2–90.4)96.5 (96.0–96.7)

 

The clearest example is the partner “The Earth orbits the Sun.” Almost every claim loses that comparison, so the choices tell us little about which claims are true. PairScore can still distinguish two claims that both lose. Against this partner, mean AUROC was close to chance for either choice format and above 90% for PairScore. Section 2 points to the per-partner results in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.