- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the limits of verbal confidence monitoring in four frontier LLMs across 15 benchmarks (38,238 items) under pre-answer, answer-time, and post-answer review conditions scored against fixed model responses. The authors find that task accuracy and error discrimination (AUROC) frequently dissociate, and they introduce a pair-level decomposition showing that pooled AUROC is largely driven by cross-tier comparisons between easy correct items and hard incorrect items—which a question-only difficulty forecast can already separate—while discrimination collapses toward chance within the hard item tier. Cross-model evaluation experiments between GPT and Gemini further suggest that peer review primarily tracks inter-model agreement rather than independent error detection.
Critical strengths
The experimental design and the pair-level decomposition of AUROC into within-easy, within-hard, and cross-tier pairs provide a clear and valuable methodological perspective on LLM calibration. Holding the generated answer fixed across pre-answer, answer-time, and post-answer conditions cleanly isolates changes in confidence judgment from changes in task performance. Demonstrating that high pooled AUROC scores are largely explained by coarse item-difficulty separation rather than fine-grained error detection is an important takeaway for works using verbal confidence or LLM-as-judge signals. The empirical scale of the evaluation—spanning four frontier models, 15 benchmarks, and nearly 40,000 items—is thorough and well executed. The authors also include extensive sensitivity analyses in the appendix, including alternative difficulty references, within-benchmark stratifications, and a neat label-free overconfidence bound based on mutually exclusive answers.
Critical weaknesses
- The central empirical claims—that error discrimination collapses on hard items and that shared errors survive cross-model review—are heavily confounded by answer-key noise and format-failure selection bias in the hard tier. Roughly 60% of the reference-hard pool comes from HLE, SimpleQA, and MMLU-Pro, which rely on LLM judges or have known key noise; on hard MMLU-Pro items, AUROC even falls well below chance (0.33–0.45), a clear sign that models are confidently right on mislabeled items. The authors' own blinded audit disagreed with the benchmark key on 9 of 17 items and left 13 of 16 high-confidence shared errors unresolved. Meanwhile, format failures in Gemini and Grok disproportionately drop hard items (e.g., Gemini keeps only 7.3% of hard MathVision items vs. 43.0% of easy ones). Because mislabeled "hard errors" mechanically pull within-hard AUROC toward 0.5 and inflate shared-error rates, it remains unclear how much of the reported collapse reflects genuine monitoring limits versus ground-truth noise and selection bias.
- The headline accuracy–AUROC dissociation highlighted in the abstract and introduction (96.8% accuracy vs. 0.516 answer-time AUROC on MATH-500) is largely an artifact of prompt field ordering under low reasoning effort. Because the JSON schema requests
answerandconfidencebefore the writtensolutionunderreasoning_effort = low, the model outputs its confidence before generating the step-by-step derivation, and 13 of the 14 errors caught during post-answer review are simple contradictions between the initial answer field and the subsequent solution text. Evaluating a single prompted verbal-confidence schema under pre-solution ordering is too narrow to establish general limits on metacognitive monitoring in LLMs.
Questions for authors
- If benchmarks with model judges or known key noise (HLE, SimpleQA, MMLU-Pro, and LiveMathBench) are excluded simultaneously, does within-hard AUROC on the remaining deterministically graded benchmarks still collapse to the 0.49–0.63 range?
- How does Gemini's within-hard AUROC change when evaluated only on the common subset of items where all four models produced valid structured outputs?
- On MATH-500 and a few other reasoning benchmarks, does the accuracy–AUROC dissociation persist if answer-time confidence is requested after the written solution field or under medium/high reasoning effort?
3 comments
Response (Part II)
Q3 & Weakness 2: Output field order (solution-first) and reasoning effort (2 × 2 ablation + trace analysis)
Q3: "On MATH-500 and a few other reasoning benchmarks, does the accuracy-AUROC dissociation persist if answer-time confidence is requested after the written solution field or under medium/high reasoning effort?"
On MATH-500, solution-first prompting reduces errors from 16 to 2 and eliminates answer-solution contradictions (13 → 0), confirming your mechanism; on MuSR, Answer-time AUROC remains near chance even under solution-first prompting and medium reasoning effort (0.505 overall across 183 errors; 0.457 within-hard across 128 hard questions including 66 errors; Appendix H, Tables 26-27).
-
Recorded pre-output
reasoning_tokens(rt) in the archived 15-benchmark run (Table 26): On archived MATH-500 (loweffort, N = 497), GPT-5.6 Sol emittedrt = 0on 163 questions andrt > 0on 334 questions: 12 of the 13 answer-solution contradictions occurred whenrt = 0(AUROC = 0.500), dropping to 1/334 whenrt > 0(98.8% accuracy, AUROC = 0.709). Observational filtering across the archived 15-benchmark suite shows that GPT-5.6 Sol and Grok 4.6 already emitrt > 0on 97.6% (mean 500 tokens) and 100% (mean 1,214 tokens) of reference-hard questions, and restricting tort >= 256still leaves within-hard AUROC (0.681 / 0.630) well below within-easy AUROC (0.840 / 0.880). -
Controlled 2 × 2 generation experiment (
answer-first vs.solution -> answer -> abstain -> confidence×lowvs.mediumeffort; archived baseline vs. three new arms, 4,362 new calls on GPT-5.6 Sol; full 95% CIs in Appendix H, Table 27):
| Task | Order, Effort | Err / N | Acc% | rt>0% | Ans≠Sol | AT AUROC | Hard AUROC |
|---|---|---|---|---|---|---|---|
| MATH-500 | ans-first, low | 16 / 497 | 96.8 | 67.2 | 13 | 0.516 | 1.000 |
| MATH-500 | sol-first, low | 2 / 498 | 99.6 | 62.9 | 0 | 0.997 | 1.000 |
| MATH-500 | ans-first, med | 8 / 499 | 98.4 | 71.5 | 5 | 0.532 | 1.000 |
| MATH-500 | sol-first, med | 2 / 499 | 99.6 | 71.1 | 0 | 0.997 | 0.750 |
| GPQA Dia. | ans-first, low | 13 / 195 | 93.3 | 96.4 | n/a | 0.848 | 0.465 |
| GPQA Dia. | sol-first, low | 19 / 198 | 90.4 | 99.0 | n/a | 0.772 | 0.417 |
| GPQA Dia. | ans-first, med | 15 / 198 | 92.4 | 98.0 | n/a | 0.855 | 0.583 |
| GPQA Dia. | sol-first, med | 16 / 198 | 91.9 | 99.0 | n/a | 0.855 | 0.750 |
| MuSR | ans-first, low | 204 / 756 | 73.0 | 84.7 | n/a | 0.546 | 0.533 |
| MuSR | sol-first, low | 197 / 756 | 73.9 | 95.6 | n/a | 0.534 | 0.518 |
| MuSR | ans-first, med | 180 / 756 | 76.2 | 93.1 | n/a | 0.501 | 0.496 |
| MuSR | sol-first, med | 183 / 756 | 75.8 | 98.5 | n/a | 0.505 | 0.457 |
Note: Reference-hard counts are n = 3 (MATH-500), n = 17 (GPQA Diamond), and n = 128 (MuSR); with only 2 overall errors under MATH-500 sol-first and n = 3 hard items on MATH-500, those AUROCs are shown for completeness only.
- In the revised manuscript (Abstract, Section 2.3, Appendix H), we explicitly attribute MATH-500's answer-time failure to premature commitment:
solution-first prompting cuts errors from 16 to 2 and eliminates all observed answer-solution contradictions (13 → 0), whereasmediumeffort alone underanswer-first still leaves 8 errors (mean 99.4% confidence) and 5 contradictions at 100% confidence (AUROC = 0.532). Manual inspection of the two archived MATH-500 errors that retained 98-99% review confidence confirmed one genuine geometry error (E11) and onearccotprincipal-value convention ambiguity (E16; Appendix I). On GPQA Diamond (n = 17 hard questions), Hard AUROC reaches 0.750 only when both interventions are combined, with a wide 95% CI [0.500, 0.967] touching 0.5 (Table 27). - On MuSR, where error counts remain substantial, even combining
mediumreasoning effort (98.5% of responses with prior internal reasoning) and a complete written solution beforeanswerandconfidence(solution-first/medium) yields 75.8% accuracy while Answer-time AUROC stays at 0.505 (0.457 within-hard; 95.6% mean confidence across its 183 errors). Thus, output ordering does not explain all weak Answer-time error ranking.
We hope these results can fully address your concerns. We are glad to have deeper discussion if you have any further questions.
Official Comment by Reviewer HsZo
Thank you for your thoughtful response and for the work you put into addressing my questions during the rebuttal period. After considering your response, I’ve decided to keep my original score.
Response (Part I)
We thank you for your careful review and the three testable questions (Q1-Q3), which we address below and in Appendices G-I (Tables 22-27).
Q1 & Weakness 1a
Yes: within-hard AUROC across the 11 remaining deterministic benchmarks is 0.513-0.608 at Answer-time and 0.508-0.576 at Post-answer, squarely inside 0.49-0.63 (Appendix G, Table 22).
Across these 11 deterministic benchmarks (22,906 planned questions/model), reference-hard questions form 8.6-14.1% of retained answers yet hold 52.7-66.7% of all errors; within-hard pairs form 0.6-2.9% of correct-error pairs vs. 50.7-63.7% for cross-tier pairs; and within individual benchmarks, within-hard AUROC < within-easy AUROC in 28/34 (Answer-time) and 29/34 (Post-answer) estimable cells (full 95% bootstrap CIs in Appendix G, Table 22):
Q2 & Weakness 1b
Gemini's within-hard AUROC differs from its full-cohort estimates by at most 0.0121 on the 15-task common intersection, and sits at 0.506-0.510 when Q1 and Q2 are combined (Appendix G, Tables 23-24).
Even as review lifts Gemini's within-easy AUROC from 0.703 to 0.937 on this exact shared set, its within-hard AUROC remains 0.510 → 0.506.