acceptodds
Under review as a conference paper at ICLR 2027

On the Limits of Metacognitive Monitoring in LLMs

Abstract

Reliable decisions depend on recognizing when an answer may be wrong. In biological cognition, metacognitive monitoring can dissociate from task performance, raising the question of how closely solving and judging are linked in language models. Here we study the confidence reports of four frontier models across 15 benchmarks. High task accuracy can coexist with weak error discrimination: a model solves 97% of competition mathematics problems while its answer-time confidence ranks correct answers above errors barely better than chance. Confidence separates correct answers from errors more effectively on questions solved by a separate reference model, while review brings limited improvement on reference-hard questions. Aggregate discrimination also rewards ranking correct answers on easy questions above errors on hard ones, which question-only forecasts already do well. Cross-evaluation helps most where the evaluator answered correctly, and errors shared by the two models usually retain high confidence. Hard questions and shared errors remain difficult targets for prompted self-review and peer oversight, even in models with strong problem-solving performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Metacognition, Error Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HsZoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the limits of verbal confidence monitoring in four frontier LLMs across 15 benchmarks (38,238 items) under pre-answer, answer-time, and post-answer review conditions scored against fixed model responses. The authors find that task accuracy and error discrimination (AUROC) frequently dissociate, and they introduce a pair-level decomposition showing that pooled AUROC is largely driven by cross-tier comparisons between easy correct items and hard incorrect items—which a question-only difficulty forecast can already separate—while discrimination collapses toward chance within the hard item tier. Cross-model evaluation experiments between GPT and Gemini further suggest that peer review primarily tracks inter-model agreement rather than independent error detection.

Critical strengths

The experimental design and the pair-level decomposition of AUROC into within-easy, within-hard, and cross-tier pairs provide a clear and valuable methodological perspective on LLM calibration. Holding the generated answer fixed across pre-answer, answer-time, and post-answer conditions cleanly isolates changes in confidence judgment from changes in task performance. Demonstrating that high pooled AUROC scores are largely explained by coarse item-difficulty separation rather than fine-grained error detection is an important takeaway for works using verbal confidence or LLM-as-judge signals. The empirical scale of the evaluation—spanning four frontier models, 15 benchmarks, and nearly 40,000 items—is thorough and well executed. The authors also include extensive sensitivity analyses in the appendix, including alternative difficulty references, within-benchmark stratifications, and a neat label-free overconfidence bound based on mutually exclusive answers.

Critical weaknesses
  1. The central empirical claims—that error discrimination collapses on hard items and that shared errors survive cross-model review—are heavily confounded by answer-key noise and format-failure selection bias in the hard tier. Roughly 60% of the reference-hard pool comes from HLE, SimpleQA, and MMLU-Pro, which rely on LLM judges or have known key noise; on hard MMLU-Pro items, AUROC even falls well below chance (0.33–0.45), a clear sign that models are confidently right on mislabeled items. The authors' own blinded audit disagreed with the benchmark key on 9 of 17 items and left 13 of 16 high-confidence shared errors unresolved. Meanwhile, format failures in Gemini and Grok disproportionately drop hard items (e.g., Gemini keeps only 7.3% of hard MathVision items vs. 43.0% of easy ones). Because mislabeled "hard errors" mechanically pull within-hard AUROC toward 0.5 and inflate shared-error rates, it remains unclear how much of the reported collapse reflects genuine monitoring limits versus ground-truth noise and selection bias.
  2. The headline accuracy–AUROC dissociation highlighted in the abstract and introduction (96.8% accuracy vs. 0.516 answer-time AUROC on MATH-500) is largely an artifact of prompt field ordering under low reasoning effort. Because the JSON schema requests answer and confidence before the written solution under reasoning_effort = low, the model outputs its confidence before generating the step-by-step derivation, and 13 of the 14 errors caught during post-answer review are simple contradictions between the initial answer field and the subsequent solution text. Evaluating a single prompted verbal-confidence schema under pre-solution ordering is too narrow to establish general limits on metacognitive monitoring in LLMs.
Questions for authors
  1. If benchmarks with model judges or known key noise (HLE, SimpleQA, MMLU-Pro, and LiveMathBench) are excluded simultaneously, does within-hard AUROC on the remaining deterministically graded benchmarks still collapse to the 0.49–0.63 range?
  2. How does Gemini's within-hard AUROC change when evaluated only on the common subset of items where all four models produced valid structured outputs?
  3. On MATH-500 and a few other reasoning benchmarks, does the accuracy–AUROC dissociation persist if answer-time confidence is requested after the written solution field or under medium/high reasoning effort?
3 comments
AuthorsOpenReview ↗

Response (Part I)

We thank you for your careful review and the three testable questions (Q1-Q3), which we address below and in Appendices G-I (Tables 22-27).

Q1 & Weakness 1a

If benchmarks with model judges or known key noise (HLE, SimpleQA, MMLU-Pro, and LiveMathBench) are excluded simultaneously, does within-hard AUROC on the remaining deterministically graded benchmarks still collapse to the 0.49-0.63 range?

Yes: within-hard AUROC across the 11 remaining deterministic benchmarks is 0.513-0.608 at Answer-time and 0.508-0.576 at Post-answer, squarely inside 0.49-0.63 (Appendix G, Table 22).

Across these 11 deterministic benchmarks (22,906 planned questions/model), reference-hard questions form 8.6-14.1% of retained answers yet hold 52.7-66.7% of all errors; within-hard pairs form 0.6-2.9% of correct-error pairs vs. 50.7-63.7% for cross-tier pairs; and within individual benchmarks, within-hard AUROC < within-easy AUROC in 28/34 (Answer-time) and 29/34 (Post-answer) estimable cells (full 95% bootstrap CIs in Appendix G, Table 22):

ModelNHardPre HardAT EasyAT HardPost HardAT Pooled
GPT-5.6 Sol18,8272,6600.5450.8010.5760.5760.798
Qwen3.8-27B17,8932,4300.5920.8340.6080.5720.826
Gemini 3.8 Flash10,2088770.5140.7250.5130.5080.781
Grok 4.614,3301,9900.5640.8830.6010.5760.854
  • Crossed-evaluation & LiveBench sensitivity (Table 25): In our 9-task crossed set (which already excludes HLE, SimpleQA, and LiveMathBench), excluding MMLU-Pro (N = 8,227; 292 same-wrong) and excluding both MMLU-Pro and LiveBench (N = 7,918; 278 same-wrong) keep agreement gaps at +9.2 pp (GPT) and +20.7 pp (Gemini), probability-sum violations at 80.1% / 89.1%, and shared-error flagging below 50% by either judge at 9.6% / 7.5% (without MMLU-Pro) and 7.9% / 5.0% (without MMLU-Pro & LiveBench) for GPT-authored / Gemini-authored responses (> 90% of each source's shared errors remain unflagged).
  • Manual MMLU-Pro audit (Appendix I): Adjudicating all 28 sampled reference-hard MMLU-Pro questions confirmed your diagnosis: among the 16 high-confidence shared-error questions, 8 have wrong keys (e.g., fiddler crab circatidal rhythms, Wien's law scaling), 7 are ambiguous/underspecified, and 1 (K25) has an upheld key where all four models erred (9 wrong keys, 15 ambiguous, 4 upheld keys across all 28). Primary labels remain untouched alongside the 11-task exclusion above.

Q2 & Weakness 1b

How does Gemini's within-hard AUROC change when evaluated only on the common subset of items where all four models produced valid structured outputs?

Gemini's within-hard AUROC differs from its full-cohort estimates by at most 0.0121 on the 15-task common intersection, and sits at 0.506-0.510 when Q1 and Q2 are combined (Appendix G, Tables 23-24).

  • 15-task suite intersection (N = 20,295; 2,766 hard across 14 represented tasks, 0 from LongBench v2; Table 23): Gemini's within-hard AUROCs are Pre 0.483 [0.455, 0.513], AT 0.498 [0.469, 0.528], and Post 0.522 [0.491, 0.552] (vs. 0.487, 0.486, 0.524 on its full 25,475-question set; max shift across all 4 models and stages is 0.0414).
  • Q1 + Q2 combined (N = 8,071; 655 hard across 10 represented deterministic tasks; 95% CIs in Table 24):
ModelAT EasyAT HardPost EasyPost HardPost Pooled
GPT-5.6 Sol0.7870.5300.8180.5180.798
Qwen3.8-27B0.8450.5770.8680.5420.837
Gemini 3.8 Flash0.7030.5100.9370.5060.890
Grok 4.60.8840.5440.8970.5730.855

Even as review lifts Gemini's within-easy AUROC from 0.703 to 0.937 on this exact shared set, its within-hard AUROC remains 0.510 → 0.506.

AuthorsOpenReview ↗

Response (Part II)

Q3 & Weakness 2: Output field order (solution-first) and reasoning effort (2 × 2 ablation + trace analysis)

Q3: "On MATH-500 and a few other reasoning benchmarks, does the accuracy-AUROC dissociation persist if answer-time confidence is requested after the written solution field or under medium/high reasoning effort?"

On MATH-500, solution-first prompting reduces errors from 16 to 2 and eliminates answer-solution contradictions (13 → 0), confirming your mechanism; on MuSR, Answer-time AUROC remains near chance even under solution-first prompting and medium reasoning effort (0.505 overall across 183 errors; 0.457 within-hard across 128 hard questions including 66 errors; Appendix H, Tables 26-27).

  1. Recorded pre-output reasoning_tokens (rt) in the archived 15-benchmark run (Table 26): On archived MATH-500 (low effort, N = 497), GPT-5.6 Sol emitted rt = 0 on 163 questions and rt > 0 on 334 questions: 12 of the 13 answer-solution contradictions occurred when rt = 0 (AUROC = 0.500), dropping to 1/334 when rt > 0 (98.8% accuracy, AUROC = 0.709). Observational filtering across the archived 15-benchmark suite shows that GPT-5.6 Sol and Grok 4.6 already emit rt > 0 on 97.6% (mean 500 tokens) and 100% (mean 1,214 tokens) of reference-hard questions, and restricting to rt >= 256 still leaves within-hard AUROC (0.681 / 0.630) well below within-easy AUROC (0.840 / 0.880).

  2. Controlled 2 × 2 generation experiment (answer-first vs. solution -> answer -> abstain -> confidence × low vs. medium effort; archived baseline vs. three new arms, 4,362 new calls on GPT-5.6 Sol; full 95% CIs in Appendix H, Table 27):

TaskOrder, EffortErr / NAcc%rt>0%Ans≠SolAT AUROCHard AUROC
MATH-500ans-first, low16 / 49796.867.2130.5161.000
MATH-500sol-first, low2 / 49899.662.900.9971.000
MATH-500ans-first, med8 / 49998.471.550.5321.000
MATH-500sol-first, med2 / 49999.671.100.9970.750
GPQA Dia.ans-first, low13 / 19593.396.4n/a0.8480.465
GPQA Dia.sol-first, low19 / 19890.499.0n/a0.7720.417
GPQA Dia.ans-first, med15 / 19892.498.0n/a0.8550.583
GPQA Dia.sol-first, med16 / 19891.999.0n/a0.8550.750
MuSRans-first, low204 / 75673.084.7n/a0.5460.533
MuSRsol-first, low197 / 75673.995.6n/a0.5340.518
MuSRans-first, med180 / 75676.293.1n/a0.5010.496
MuSRsol-first, med183 / 75675.898.5n/a0.5050.457

Note: Reference-hard counts are n = 3 (MATH-500), n = 17 (GPQA Diamond), and n = 128 (MuSR); with only 2 overall errors under MATH-500 sol-first and n = 3 hard items on MATH-500, those AUROCs are shown for completeness only.

  • In the revised manuscript (Abstract, Section 2.3, Appendix H), we explicitly attribute MATH-500's answer-time failure to premature commitment: solution-first prompting cuts errors from 16 to 2 and eliminates all observed answer-solution contradictions (13 → 0), whereas medium effort alone under answer-first still leaves 8 errors (mean 99.4% confidence) and 5 contradictions at 100% confidence (AUROC = 0.532). Manual inspection of the two archived MATH-500 errors that retained 98-99% review confidence confirmed one genuine geometry error (E11) and one arccot principal-value convention ambiguity (E16; Appendix I). On GPQA Diamond (n = 17 hard questions), Hard AUROC reaches 0.750 only when both interventions are combined, with a wide 95% CI [0.500, 0.967] touching 0.5 (Table 27).
  • On MuSR, where error counts remain substantial, even combining medium reasoning effort (98.5% of responses with prior internal reasoning) and a complete written solution before answer and confidence (solution-first/medium) yields 75.8% accuracy while Answer-time AUROC stays at 0.505 (0.457 within-hard; 95.6% mean confidence across its 183 errors). Thus, output ordering does not explain all weak Answer-time error ranking.

We hope these results can fully address your concerns. We are glad to have deeper discussion if you have any further questions.

Reviewer_HsZoOpenReview ↗

Official Comment by Reviewer HsZo

Thank you for your thoughtful response and for the work you put into addressing my questions during the rebuttal period. After considering your response, I’ve decided to keep my original score.

Reviewer_BpvVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper measures the relationship between confidence and accuracy in four frontier LLMs on 15 benchmarks and two metacognitive metrics. It probes for confidence judgments before, during, and after answering, and cross-references those with distinct model judges. It reproduces a number of known dissociations and correlations.

Critical strengths

The authors are to be commended for using a large and diverse battery of tests. The figures are clear, the design is careful, the findings are described thoroughly and transparently, and the analyses appear to be robust and correct.

Critical weaknesses

The paper's core findings (that accuracy and confidence discriminability can dissociate, that difficulty generally correlates with poor confidence discriminability, that models are more confident on easier questions, that cross-evaluation helps when the two models have complementary knowledge) have been demonstrated before, in papers the authors mostly cite. Bringing these together under one protocol, as the authors do, is a contribution, but a minor one.

The text is sometimes hard to follow, and reads LLM-y.

Questions for authors

Can you offer concrete examples of practical applications of these findings?

2 comments

Thank you for reviewing our manuscript and raising questions about novelty and writing. We hope the following clarification and the revised manuscript can address your concerns.

On novelty: you are right that the four qualitative patterns you listed were noted in earlier papers we cite. What those studies did not show, and what surprised us most in our own data, is how badly standard evaluation metrics hide hard-question failures:

  • Pooled AUROC tracks question difficulty, not hard errors (Section 2.4, Figure 3): Reference-hard questions hold 47% to 77% of all errors (Table 6), yet hard-hard pairs get ≤ 4% of the pooled AUROC weight (even perfect within-hard ranking lifts pooled AUROC by < 0.02), so a Pre-answer guess from the question alone matches Answer-time pooled AUROC within 0.022.
  • Fresh review catches easy slips, not hard errors (Sections 2.3-2.4, Figure 3C, Table 24): Holding the recorded answer fixed shows that fresh-context review flips upside-down error confidence and cleans up easy slips (e.g., Gemini within-easy AUROC 0.703 → 0.937), yet stalls on hard questions (0.510 → 0.506).

Can you offer concrete examples of practical applications of these findings?

Yes. These limits directly change how confidence should be wired into real LLM systems (Section 4; Appendices E & K):

  1. Cascade routine queries; route hard ones upfront (Appendix K, Table 30). Deferring Qwen3.8-27B at Post-answer < 70% escalates < 10% (9.8%) of queries to GPT-5.6 Sol and lifts overall accuracy 70.6% → 74.6%. Yet a separate ~10%-budget hard-tier test (Post-answer < 15%) barely moves (16.0% → 17.9% vs. 25.6% oracle), motivating upfront Pre-answer routing to tools or humans.
  2. Use two models to break ties, never agreement for safety (Section 2.5, Appendices E.6-E.7). Label-free peer scoring resolves 75.7% of single-solver disagreements (vs. 62.6% Answer-time self-confidence) and closes 45% of the gap to the union ceiling across all items (Table 12), yet leaves > 90% of each model's matching wrong answers unflagged below 50% confidence (Table 25).
  3. Audit overconfidence with zero labels (Figure 4B, Appendix E.4). Separately rating provably mutually exclusive candidate answers (mean probability sum sˉ\bar{s}) bounds mean overconfidence on those pairs by (sˉ−1)/2(\bar{s}-1)/2, exposing > 22% (GPT) and > 15 % (Gemini) overconfidence on conflicting multiple-choice items with zero human labels.

Finally, we appreciate your fair point on the writing: trying to pack 15 benchmarks of per-model numbers and intervals into 10 pages made parts of our original text overly compressed and mechanical to read. We have stripped out more than half of the inline number readouts from the main text and moved the intervals into the figures and tables so the paper reads much more naturally.

I appreciate the additional information, but I'm still not seeing the significance of this. Hard-hard pairs are rare in your set, so naturally will not contribute much to AUROC, but stratified comparisons are standard in the literature. Likewise, it would be expected that the same model reviewing its own answer in a fresh context would have a hard time recognizing errors on questions it couldn’t solve, and an easy time recognizing errors on easy questions it got wrong, in the same way another rollout would likely get them right. This is especially true when the error occurs in a protocol where the answer precedes an explanation that contradicts it. Finally, those applications are not novel and follow from known results. The paper is not a bad one, but its significance just doesn't rise to the level of an ICLR conference paper, so I retain my ratings.

Reviewer_LFcCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper reports on an empirical analysis of confidence reports across different models and benchmarks. It seeks to shed light on which mistakes are revealed by confidence, comparing pre-answer confidence, answer-time confidence, and a new post-answer confidence where the model reviews its own question and answer in a fresh context without access to the reasoning trace.

Based on a large run of experiments, the paper then provides an extensive discussion of results.

Critical strengths

The new post-answer confidence where the model reviews its own question and answer in a fresh context without access to the reasoning trace is potentially novel.

The most interesting finding might be regarding whether another model can identify errors in other model's outputs.

Critical weaknesses

The paper, based on a large run of experiments, provides an extensive discussion of results. However, none of the discussed points appear to be truly novel insights.

For example, a key claim is that high task accuracy can coexist with weak error discrimination, which however has already been widely known for a while now.

Hence, it is unclear what the key novel insight in this paper.

Questions for authors

What are the genuinely novel insights?

1 comment

Thank you for reviewing our manuscript and raising the question about novelty. We hope the following clarification and the revised manuscript can address your concerns.

What are the genuinely novel insights?

We agree that high task accuracy coexisting with weak error discrimination is already known (e.g., Xiong et al., 2024); in the revised manuscript, that observation serves only as a motivating starting point.

Our primary novel insight is a quantitative explanation of why our pooled Answer-time AUROC across 15 benchmarks is high (0.81-0.87) even when error ranking fails on the hard questions (Section 2.4, Figure 3):

  • Hard questions hold 47-77% of all errors (Table 6), yet error ranking inside them drops to 0.49-0.63 (0.51-0.61 across the 11 deterministic benchmarks; Figure 3A, Table 22).
  • Pooled AUROC misses this because hard-hard pairs are ≤ 4% of its denominator (Figure 3B): Models rarely answer hard questions correctly, so holding other pair types fixed, even perfect (1.0) within-hard ranking lifts pooled AUROC by < 0.02.
  • Cross-tier pairs dominate instead: a question-only Pre-answer forecast matches Answer-time pooled AUROC within 0.022 (Figure 3A). High Answer-time AUROC tracks question difficulty rather than answer errors (Post-answer review can still raise pooled AUROC via easy questions; Appendix F.4).

This fixed-answer, difficulty-stratified protocol also clarifies the two parts of our design that you highlighted as potentially novel and most interesting:

  • Fresh-context review catches easy slips, not hard errors (Sections 2.3-2.4, Figure 3C): At answer time, every model is more confident in easy-question errors than hard ones; fresh review of the same answer flips this across all four models (e.g., Gemini within-easy AUROC 0.703 → 0.937, within-hard 0.510 → 0.506 on the strict deterministic 4-model set; Table 24).
  • Peer review settles disagreements, not shared errors (Section 2.5, Figure 4): Under source-neutral prompts, peer confidence tracks answer agreement, resolving 75.7% of single-solver disagreements (vs. 62.6% Answer-time self-confidence; 45% union-gap recovery across all items; Appendix E.6) while leaving > 90% of each model's matching wrong answers unflagged below 50% confidence (Appendix E.7; Table 25).

To conclude, our work highlights an under-explored challenge for tasks that require metacognitive capacity, such as autonomous research and high-stakes delegation -- even frontier LLMs struggle to estimate confidence on hard problems, a blind spot that is hidden by common metrics in the literature and calls for action.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We are grateful to all three reviewers for the careful evaluation and consistent recognition of our comprehensive experimental design (Soundness: 3, 3, 2). Meanwhile, looking across the divergence in evaluating the informativeness of our work, a key question is how our study goes beyond the prior work on LLM metacognition. In the original draft, too many per-model numbers in the main text obscured how our fixed-answer, difficulty-stratified protocol differs from existing setups. As a result, our core contributions did not stand out clearly in form of the original submission, which are two-fold:

(1) Up-to-date results on four frontier models from different providers on 15 benchmarks (LLMs are iterating rapidly, but surprisingly the metacognitive monitoring ability still fails catastrophically on hard problems). This serves as a comprehensive baseline for future metacognition studies. (2) Two counter-intuitive and novel mechanisms established by the experiments that were not clearly stated in previous literature, as stated below:

  • Answer-time confidence barely improves the pooled AUROC over a question-only Pre-answer forecast (Section 2.4, Figure 3). Prior studies evaluate verbal or learned calibration at the task or dataset level (Kadavath et al., 2022; Tian et al., 2023; Xiong et al., 2024) and examine how verification varies with question priors or difficulty (Kapoor et al., 2024; Tan et al., 2025; Zhou et al.,2026), yet do not explain why pooled AUROC stays high (0.81-0.87 in our cohorts) when hard-question ranking fails. When 47-77% of all errors are on the hard questions (Table 6) and confidence fails to rank them, one would expect the overall AUROC to be low and Answer-time confidence to be much higher than a question-only forecast. Neither expectation is true on the matched full-suite cohorts, because very few hard questions are answered correctly and pairs of two hard questions are ≤ 4% of the AUROC denominator. On the exact same recorded answers, a question-only Pre-answer forecast is within 0.022 AUROC of Answer-time confidence for every model, which indicates that a high pooled AUROC is not evidence of error detection within the hard questions.

  • At answer time, models are more confident in errors on easy questions than on hard questions, and fresh review mainly helps on the reference-easy questions (Sections 2.3-2.4, Figure 3C). Unlike self-correction pipelines that rewrite the answer during review (Huang et al., 2024; Kamoi et al., 2024) or unstratified separate-session probing (Kadavath et al., 2022; Xiong et al., 2024), holding the answer fixed across external difficulty strata reveals an upside-down pattern. Intuitively, a careless error on an easy question should receive lower confidence than an error on a hard question, yet the opposite is observed at Answer-time across all four models on the matched full-suite cohorts. When we review the exact same fixed answer in a fresh context (Post-answer), the error-confidence ordering is reversed for all four models and easy-question errors are flagged, while within-hard discrimination remains weak on the deterministic benchmarks (Tables 22 and 24). Separately, our 2 by 2 generation experiment confirms that solution-first prompting removes self-contradictions on MATH-500, whereas the Answer-time AUROC on MuSR is still near chance level after step-by-step solutions and medium effort (Table 27).

To present these central contributions of our study in a more clear and straightforward form, we shortened the main-text numeric readouts and moved detailed intervals to the figures and tables, with new control and deployment analyses in Appendices G-K (new analyses and substantive additions marked in blue). We hope this clearer framing makes our core contributions much easier to see.

Moreover, we carefully address all technical and practical questions in our individual responses below (for example, after simultaneously excluding the four disputed/model-judged benchmarks and matching valid outputs across all four models, within-hard AUROC remains 0.51-0.58 at both answer time and review).

Finally, we would like to thank the reviewers again for insightful comments that have greatly improved the presentation and completeness of our study. We are more than happy to discuss any further feedbacks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.