acceptodds
Under review as a conference paper at ICLR 2027

LLMs learn different forms of metacognition when trained to predict their own accuracy

Abstract

Large language models are trained to always produce an answer, regardless of whether they possess the relevant knowledge, which leads them to fabricate facts. Prior work has shown that LLMs' confidence estimates correspond poorly to their actual performance, and that fine-tuning can substantially improve them. However, what models actually learn during such training remains poorly understood. We investigate how LLMs acquire metacognitive monitoring, the ability to know what one knows, by training 10 open-weight LLMs to predict their own accuracy on factual multiple-choice questions before answering them. We find that trained confidence reflects two distinct signals. While on questions close to the training data, it tracks the model's true accuracy, in other domains, it instead tracks output consistency: the concentration of the model's answer distribution. Output consistency tracking emerges early in training and generalizes across datasets, whereas accuracy tracking develops later and remains local to the training distribution. These results suggest that calibration training may not teach models to generally detect errors they commit confidently, and they raise broader questions about the nature of metacognition in artificial systems.

Primary area:
applications to neuroscience & cognitive science
Keywords:
metacognition, large language models, calibration, self-consistency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8PPMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what calibration training actually teaches LLMs by training to predict accuracy on factual multiplechoice questions across 10 open-weight models and multiple datasets. The authors find that verbalized confidence shifts from nearly uncorrelated with accuracy before training (r=0.04) to strongly correlated after (r=0.73), and that trained confidence carries two distinct signals: true-accuracy tracking on questions near the training distribution and output-consistency tracking on distant ones.

Critical strengths
  1. The paper focuses from reporting improved calibration metrics to asking what calibration training actually teaches the model, i.e.whether the signal reflects genuine self-knowledge or an artifact of the answer distribution. This is consequential because it addresses a fundamental gap in the literature and reframes the problem in a way that yields more interpretable insights.

  2. Direct and convincing core observations. The U-shaped confidence–accuracy relation and the finding that trained confidence never falls below chance even on confidently incorrect answers are clearly visible in the figures and do not depend on contested statistical machinery. This is consequential because these results stand on their own as robust empirical findings, regardless of how one interprets the deeper mechanistic claims.

Critical weaknesses
  1. The main “accuracy tracking generalizes” claim rests on a test split that’s almost a copy of training. The held-out category shares ranges digit counts and prompt templates with the training categories with only the operator symbol changes. So this isn’t a very OOD transfer; it’s interpolation within the same generative process. This is consequential because it is the only setting in which accuracy tracking appears to transfer, yet the abstract and introduction present it as broader category-level generalization; without stronger evidence, the central mechanistic claim is not well supported.

  2. The setup doesn’t actually produce a system that both answers and reports confidence. The LoRA makes the network unable to answer questions, while the accuracy consistency targets come from the base model. So there’s no single model doing both things at once, which undercuts the abstention motivation and makes it hard to say what “the LLM learned.” This is consequential because the paper’s framing depends on interpreting the result as something the base model has learned about its own accuracy, but the missing frozen-backbone probe control leaves open the simpler alternative that the LoRA merely reconstructs the base answer distribution internally.

Questions for authors
  1. If MATH's test category is replaced with a genuinely disjoint one (different magnitude ranges, mixed operators, parenthesized expressions), does accuracy tracking survive?
2 comments

We thank the reviewer for their careful reading and constructive feedback. We particularly appreciate their recognition that the paper shifts the focus from calibration metrics to what calibration training actually teaches the model. We also appreciate their observation that the core findings are visible directly in the figures: the U-shaped confidence–accuracy relation, and the fact that trained confidence remains high on confidently incorrect answers.

One small clarification: after training, confidence is no longer verbalized but read from a linear probe on the model’s hidden states. The comparison between verbalized confidence before training (r = 0.04) and probe-based confidence after training (r = 0.73) serves as a sanity check that fine-tuning succeeded, not as a comparison between equivalent elicitation formats.

W1. We agree with the reviewer’s description of the MATH test split: it is close to the training categories by design, and accuracy tracking on it could be seen as interpolation within the same generative process. This is precisely our point: across all our experiments the two MATH datasets are the only ones where the model is able to predict its true accuracy consistently beyond the training questions and it is the one closest to the training data. As soon as test questions are more distant, from held-out topics in MMLU-Pro and MedMCQA or from other datasets, trained confidence tracks output consistency instead (Fig. 3) even if the model was trained on MATH. Our paper does not claim that true accuracy tracking generalizes. As stated in the abstract, accuracy tracking “remains local to the training distribution”. The generalization we report concerns output consistency tracking. We recognize, however, that phrases such as “held-out category” and “transfers to unseen domains” (Introduction) may suggest that the MATH test split is more out-of-distribution than intended, and we have clarified them in the revised version.

W2. We agree that our setup does not produce a single model that both answers and predicts its accuracy, and we state this in Section 2.4. This is a deliberate design choice. Training a model to answer and to predict its accuracy at the same time means predicting a moving target: every update changes the model’s accuracy, including on held-out questions. Keeping the base model frozen and training only the adapters to predict its accuracy ensures that true accuracy and output consistency remain fixed throughout training, which allows a clean comparison of which signal the trained confidence tracks. The same frozen weights can still be used in practice, with adapters disabled to answer and enabled to report confidence. We agree that a jointly trained model would match the abstention motivation more closely, and that “predicting one’s own accuracy” should be understood here as an adapted model predicting the accuracy of its frozen base. Our aim is to study what models learn when trained to predict their accuracy, not to build the best-performing confidence estimator. We will make this explicit in the limitations.

We also agree that a probe on the frozen backbone would indicate whether the consistency signal is already linearly available in the base model or computed by the adapters. Our claim concerns what the trained confidence tracks, not how it is computed. The middle-probe experiment (Appendix G) shows that consistency tracking does not simply leak from the output logits which is a necessary control for our claim; localizing the signal more precisely in the network is a question we leave to future work.

Answer to Questions. We thank the reviewer for this suggestion. We designed the MATH dataset to contain single-operation questions that can be answered fairly directly, without requiring complex chain-of-thought, consistent with our focus on knowledge-like recall rather than multi-step reasoning (Section 2). Mixed operators or parenthesized expressions risk introducing such multi-step reasoning to solve the problem. Instead, we propose future work to test the same hypothesis from the other direction. Creating test splits for MMLU-Pro and MedMCQA that are closer to their training data. Combined with the current topic-disjoint test splits, this provides both near and far test sets within each dataset, which is the contrast the reviewer asks for. The extreme version of the question is already answered by our cross-dataset results: MATH-trained models evaluated on MMLU-Pro and MedMCQA track output consistency rather than accuracy (Fig. 3). The new splits would add data at low to moderate distances from the training set, and test whether the MATH results replicate on the other datasets. Based on these results, we predict that on a more distant MATH test set, accuracy tracking would weaken in favor of consistency tracking. Given the short discussion period, we could not implement this new condition yet and leave the question open for future work.

The rebuttal clarifies the intended scope but does not resolve my main concerns. The only evidence of accuracy tracking beyond the training questions comes from a near-in-distribution MATH split, while the cross-dataset results confound distributional distance with changes in task, domain, and difficulty. Moreover, the additional controlled experiments and the frozen-backbone probe needed to clarify what calibration training actually teaches are left to future work. The central claims therefore remain insufficiently supported and are substantially more limited than the presentation suggests. This also weakens the paper’s novelty: the remaining contribution is largely local accuracy interpolation and consistency tracking under changing experimental conditions.

Reviewer_ZyG4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors seek to understand what LLMs learn when trained in metacognitive monitoring by fine-tuning 10 open-weight models to predict their accuracy on a pair of knowledge and reasoning multiple-choice question answering benchmarks, as well as custom mathematical calculation multiple-choice questions. They find that the trained models can predict their accuracy on test samples close to the training distribution, but on samples far from that predictions are poor on low-accuracy questions, and instead track consistency. They further report that the consistency tracking appears early in training, and accuracy tracking only later OOD.

Critical strengths

Metacognitive monitoring is an important skill with relevance to LLM applications and safety, so it's worth understanding how it might develop with training. The authors are to be commended for testing a breadth of models, and for careful dataset splits. The main text is clearly written, with informative figures.

Critical weaknesses

Most highlighted findings strike me as inherent to the design. A model trained to predict its accuracy can learn generalizable surface cues of difficulty or memorize the training data. The MATH dataset plausibly offers the former, so the models perform well on that in-distribution test set. The models likely have to rely more on the latter in the other (factual) datasets, which is why they don't generalize as well OOD (and is why they see training set accuracy tracking developing later in those).

After a model has learned to align confidence and accuracy, its consistency will track accuracy above chance in multiple-choice questions, but below chance consistency may increase again, as the correct answer is excluded, explaining the bow shape in Figure 2, bottom right.

The authors may be correct that when surface cues were not available the models learned to monitor their own uncertainty, and that this is why confidence tracked consistency in such cases, but the necessary mechanistic interpretability work to establish this was not done.

The apparent trend in Figure 4 comes from one cluster, so it simply recapitulates the fact that test accuracy tracking is only high when MATH-trained models are evaluated on MATH, as was shown in Figure 3.

Questions for authors

Have you tried to predict accuracy from surface features in any of the datasets?

2 comments

We thank the reviewer for their feedback and for acknowledging the breadth of models tested, the careful dataset splits, and the clarity of the paper.

W1. We agree that accuracy prediction near the training data may rely on surface cues of difficulty (MATH), and that for factual datasets accuracy on training questions can be learned through memorization. However, this account explains only where accuracy tracking succeeds or fails. It does not explain what the model does instead. Out of distribution, trained confidence does not collapse to a dataset-level mean or to noise: it consistently tracks output consistency. This holds even for models trained on MATH and evaluated on medical or scientific questions, where neither arithmetic surface cues nor memorization are available (Fig. 3). This is the main result of the paper, and it is not implied by the training objective, which targets accuracy. While accuracy tracking in one condition may follow from the design, the contribution lies in comparing it with the conditions where it does not hold, and in identifying what replaces it.

W2. We agree with the reviewer’s explanation. Because accuracy and output consistency are related by a U-shaped function, tracking one signal necessarily produces a non-linear relation with the other. This relationship was already described in Section 3.2. We would like to emphasize that the model was trained to predict its true accuracy, so the most expected outcome would be a linear relation between trained confidence and accuracy everywhere. Both outcomes are plausible a priori, as the reviewer points out. The question the paper addresses is which one holds, and under which conditions, and the answer is not given by the design. Δr is used precisely to discriminate between them across conditions.

W3. This paper is a behavioral study of what LLMs predict when trained to predict their own performance, and of how this relates to theories in cognitive science (we would like to highlight that the paper was submitted to the cognitive science track). Its aim is not to provide a mechanistic account of how models internally compute these predictions from input text or what cues they follow. Our main point is that even when explicitly trained to predict their accuracy, models track their output consistency away from their training data, consistent with findings on human confidence (Caziot & Mamassian, 2021). Mechanistic interpretability work would complement this approach well but falls outside the scope of this paper.

W4. The trend in Fig. 4 is not driven solely by the in-distribution MATH cluster: when this cluster is removed, the correlation decreases from r = 0.49 to r = 0.15 but persists. The decrease is expected, since this cluster is where accuracy tracking dominates and thus where Δe is expected to be the lowest. To control for differences between datasets, here are two complementary comparisons:

  • Same test questions, different training sets. For MedMCQA test questions, mean Δe = 0.00 at distance ≈ 0.46 for models trained on MedMCQA, vs. 0.05 at ≈ 0.71 for models trained on MATH10. Because the test questions are identical, this difference is not explained by properties of the test set.
  • Same training set, different test sets. For models trained on MedMCQA, mean Δe = 0.00 on the MedMCQA test split (distance ≈ 0.46), vs. 0.08 on MATH10 (distance ≈ 0.64). Here the difference is not explained by properties of the training set.

In both cases, Δe increases with distance to the training data. We suggested future work could add test splits for MMLU-Pro and MedMCQA constructed to be closer to their training data. This will provide additional clusters at low distance and replicate the MATH findings on other datasets, which we agree could be a good addition to the current results.

Answer to Questions. We have not tested surface-feature predictors. Accuracy tracking near the training data may well rely on such cues, which is compatible with our interpretation of it as learned from observing one’s performance on similar questions. Our main claim does not depend on how accuracy is predicted near the training data, whether through surface cues or deeper features. It depends on what the trained confidence tracks as the distance between training and test questions increases. Identifying the cues used to predict accuracy near the training data falls outside the scope of this question.

Regarding W1, I agree that consistency tracking isn't trivial, although it is not surprising. Regarding W4, the remaining effect isn't large, and both comparisons contrast same-domain with cross-domain training, so they don’t show a graded effect of distance. Regarding the question, the surface cue-based account, if true, would undermine the distance-based account you offer.

I maintain my score.

Reviewer_SZK3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what a language model learns when it is trained to predict its own accuracy. Ten open models receive LoRA adapters and a linear probe trained to predict, before answering, the probability they assign to the correct option on multiple-choice questions from science, medicine and arithmetic, with train and test split by topic. Trained confidence correlates with accuracy on the test splits, but the relationship is the following: confidence stays high when the model is confidently wrong. Trained confidence correlates more linearly with the concentration of the model's answer distribution, which the paper calls output consistency. Accuracy tracking appears only on the arithmetic dataset, whose test category lies between training categories, and does not transfer across datasets. Consistency tracking appears within the first training steps, transfers everywhere, and survives moving the probe to the middle of the network. The paper interprets the two as first-person and third-person metacognition and concludes that calibration training does not teach models to detect their confident errors out of distribution.

Critical strengths
  1. The experiments are careful. Topic-disjoint splits with matched accuracy, ten models, five datasets, five runs each, a cross-dataset grid, a mid-layer probe control and training-time curves. The result that accuracy tracking stays local to the training distribution is a useful warning for anyone reporting calibration on a random split.
Critical weaknesses
  1. The obvious baseline is missing. The only untrained comparison is verbalized confidence parsed from five greedy tokens, which gives a correlation of 0.04. The paper's own finding is that the probe learns output consistency, yet it never reports how well consistency itself, or max option probability, correlates with accuracy. If that number is close to the trained probe's on far datasets, training adds nothing out of distribution and the claim that fine-tuning substantially improves confidence estimation is against a strawman.

  2. The contribution is incremental. Cross-task transfer of accuracy probes was measured in Kadavath et al. (2022) and Orgad et al. (2025), prospective prediction of QA accuracy from hidden states in Gottesman and Geva (2024), and dataset-specific uncertainty directions that transfer only within math in Cohen et al. (2025). None of these is discussed. What the paper adds is a clean demonstration that a trained probe converges to output entropy outside its training topics. The metacognition framing and the human analogy are interpretation and are not tested.

Questions for authors
  1. Within a fixed level of output consistency, does trained confidence still correlate with accuracy out of distribution?
  2. Does a probe trained on one model predict a second model's accuracy on the same questions?
  3. Does accuracy tracking transfer to a held-out dataset when the probe is trained on all five training splits together?
3 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We thank the reviewer for their careful reading and for acknowledging the experimental design: topic-disjoint splits, ten models with five runs each, the cross-dataset grid, and the mid-layer control.

W1. We agree that this baseline should be reported, and we have added it in Appendix: the correlation of output consistency (and of max option probability) with true accuracy on every test split, alongside trained confidence. We would like to clarify the claim of the paper, which this baseline supports rather than undermines. Our claim is not that training produces a confidence estimate better than output consistency. Our claim is that a model trained to predict its true accuracy learns, away from its training distribution, to output something closer to its output consistency than to its true accuracy. The new baseline shows this directly:

  • Far from the training data (cross-dataset and MMLU-Pro/MedMCQA test splits), trained confidence predicts accuracy only slightly better than consistency alone. For example, averaged over the 10 models on MMLU-Pro10, r(confidence, accuracy) = 0.58 vs. r(consistency, accuracy) = 0.56. At the same time, trained confidence correlates more strongly with consistency than with accuracy: r(confidence, consistency) = 0.69 vs. r(confidence, accuracy) = 0.58, a difference of 0.11. Training therefore adds little accuracy information beyond consistency: the model has learned consistency as a proxy for accuracy.
  • Near the training data (MATH test splits), trained confidence clearly exceeds the consistency baseline. Averaged over the 10 models on MATH10, r(confidence, accuracy) = 0.86 vs. r(consistency, accuracy) = 0.62. Trained confidence also correlates more strongly with accuracy than with consistency: r(confidence, accuracy) = 0.86 vs. r(confidence, consistency) = 0.64, a difference of 0.22. This is the accuracy tracking we describe.

The setting is also prospective: the probe reads the hidden state of the confidence prompt, in which the model is never asked to answer. Output consistency is therefore not directly available to it, and recovering it is itself non-trivial, we will make this more clear in the main text. Furthermore even when decoding happens at the middle of the network (cf Appendix H) predictions still match with consistency.

Regarding the sentence “fine-tuning substantially improves confidence estimation” (Section 3.1 and Contributions): the comparison with verbalized confidence (r = 0.04 before vs. 0.73 after training) was intended as a sanity check that fine-tuning succeeded, as stated in Section 3.1, not as a claim that trained confidence outperforms output consistency. To avoid this reading, we have rephrased it as “substantial improvement on the verbalised baseline”.

AuthorsOpenReview ↗

Rebuttal 2/3

W2. We thank the reviewer for these references, which we now discuss:

  • Kadavath et al. (2022), beyond our experimental setup: fine-tuned P(IK) generalizes only partially across tasks, with poor out-of-distribution calibration yet does not discuss what the model learns nor why it does not generalise out of distribution. Our results suggest an account of this, namely that the part that transfers may be consistency tracking and is consistent with recent cognitive science findings in humans.
  • Orgad et al. (2025): error-detection probes transfer only between tasks requiring similar skills, and out of distribution they rarely exceed a logit-based baseline. This is consistent with our results. We go further by showing what an accuracy-trained confidence converges to and under which conditions.
  • Gottesman & Geva (2024): a precedent for prospective performance prediction from hidden states, though at the level of a general subject rather than individual questions.
  • Cohen et al. (2025): dataset-specific uncertainty directions that transfer only between closely related datasets, mirroring the local accuracy tracking we observe.

These works establish that confidence or error probes transfer poorly. Kadavath et al. fine-tune the model; the other three probe frozen representations. None of them characterizes what an accuracy-trained confidence estimator actually tracks. Our contributions relative to this literature are:

  • Decomposition: Trained confidence decomposes into two signals: true accuracy and output consistency. Consistency dominates out of distribution, including on questions the model answers confidently but wrongly.
  • Distance dependence: At the level of individual questions (and not topics), distance to the training data predicts which signal confidence tracks (Fig. 4).
  • Training dynamics: Consistency tracking emerges early; accuracy tracking develops later and remains local (Appendix F).
  • Logit-leak control: Consistency tracking persists with a mid-layer probe, so it is not a leak from the output logits (Appendix G).

The metacognition framing and the human analogy are presented as an interpretation of our findings, connecting them to closely related questions in the cognitive science literature (e.g., Caziot & Mamassian, 2021, on confidence and self-consistency). This is relevant to the track to which the paper was submitted (applications to neuroscience and cognitive sciences). We agree that this interpretation is not tested directly, and we have clarified in the Discussion that it is a proposed interpretation rather than an empirical finding.

AuthorsOpenReview ↗

Rebuttal 3/3

Answer to questions:

Q1. We thank the reviewer for this question, which directly tests our decomposition. We binned test questions into quintiles of output consistency and computed the correlation between trained confidence and accuracy within each bin, for each model. We compare two conditions: out of distribution (training and test datasets differ) and near distribution (MATH-trained models evaluated on MATH test splits).

Out of distribution: within consistency bins, r(confidence, accuracy) = 0.17 ± 0.02 on average (from 0.10 ±0.02 in the lowest quintile to 0.30 ±0.04 in the highest), compared with r = 0.52 ± 0.06 without binning. Most of the correlation between trained confidence and accuracy therefore comes from consistency. Once consistency is held approximately constant, only a weak accuracy signal remains.

Near distribution (MATH → MATH): within bins, r = 0.65 ±0.07 on average (from 0.51 ±0.08 to 0.81 ±0.07), close to r = 0.81 ±0.04 without binning. Here, trained confidence carries accuracy information beyond consistency. The within-bin correlation is highest in the top quintile. Because quintiles are wide, consistency still varies within each bin, and part of this value may reflect that residual variation. Overall, this analysis confirms the decomposition: away from the training data, trained confidence carries little accuracy information beyond output consistency, whereas near the training data it tracks accuracy directly. We will add this analysis to the appendix (within-bin correlations per condition, SEM across models). See appendix I.

Q2 - Does a probe trained on one model predict a second model's accuracy on the same questions? We ran this experiment but did not include it, as it falls outside the main question of the paper. A probe trained to predict model A’s own accuracy predicts it better than a probe trained on model A to predict another model B’s accuracy on the same questions. Self-prediction thus outperforms cross-prediction, indicating that trained confidence captures model-specific information, not only question difficulty shared across models.

Q3 - Does accuracy tracking transfer to a held-out dataset when the probe is trained on all five training splits together? We did not run it in this version of the paper but this is an interesting suggestion and we included it as future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response 1/2

We thank the three reviewers for their careful reading and constructive feedback. All three recognized the care of the experimental design: topic-disjoint splits with matched accuracy, ten models with five runs each, the cross-dataset evaluation, and the mid-layer control. Reviewers SZK322 and 8PPM also noted the practical relevance of our main observation, that trained confidence remains high on confidently incorrect answers away from the training distribution. We summarize here the main points raised and how we addressed them.

  1. Main claim. Some comments suggested that the paper claims accuracy tracking generalizes. Our claim is the opposite. Models trained to predict their accuracy track it only near their training data; farther away, trained confidence tracks output consistency instead, even for models trained on MATH and evaluated on medical or scientific questions (Fig. 3). The MATH test split is close to the training data by design, which is why it is the only setting where accuracy tracking appears beyond the training questions. We clarified some phrases (“held-out category”, “transfers to unseen domains”) that may have suggested otherwise throughout the paper.

  2. Output consistency baseline (Reviewer SZK3). We added the correlation of output consistency with accuracy on every test split. Far from the training data, trained confidence correlates with accuracy only slightly better than consistency alone, while correlating more strongly with consistency than with accuracy. This supports our claim that the model learns consistency as a proxy for accuracy. The comparison with verbalized confidence serves only as a sanity check that fine-tuning succeeded, indeed the trained model does not report its confidence verbally but through a LoRA adapter and a trained probe. These results are now discussed in appendix H.

  3. Distance dependence (Reviewer ZyG4). The trend in Fig. 4 is not driven solely by the in-distribution MATH cluster: without it, the correlation decreases from r = 0.49 to r = 0.15 but persists. To control for dataset differences, we compared identical test questions evaluated by models trained on different datasets, and identical training sets evaluated on different test sets. In both cases, Δe increases with distance to the training data.

  4. Related work (Reviewer SZK3). We now discuss Kadavath et al. (2022), Orgad et al. (2025), Gottesman & Geva (2024), and Cohen et al. (2025). These works show that confidence and error probes transfer poorly across tasks. None of them characterizes what an accuracy-trained confidence estimator actually tracks, how this depends on distance to the training data, or how the two signals emerge during training, which we believe are our original scientific contributions, including in comparison with these papers cited by reviewers. These references are now briefly discussed in the introduction and we added related work discussions in Appendix A to highlight how our work differentiates from these papers.

  5. Design choices (Reviewers ZyG4 and 8PPM). We would like to clarify the design to make it clear to all reviewers: we keep the base model frozen so that true accuracy and output consistency remain fixed during training, allowing a clean comparison of which signal trained confidence tracks. Our study is behavioral: it characterizes what trained confidence tracks rather than how it is computed internally. Mechanistic localization is left to future work. We added clarifications about design choices throughout the paper.

AuthorsOpenReview ↗

General Response 2/2

  1. Interpretation. The metacognition framing and the comparison with human confidence are presented as an interpretation of our findings, in line with the cognitive science track to which the paper was submitted. Recent work in cognitive science has shown that self-consistency explains human confidence better than accuracy on some tasks (Caziot & Mamassian, 2021), opening a debate about the nature of confidence and the signal it actually tracks. Our work contributes to this debate in two ways. First, it investigates to what extent LLMs can be trained to predict their own accuracy, as well as whether they show behavioral signatures similar to those in humans. Second, it allows experiments that are impossible with human participants. In particular, it tests whether consistency-based confidence is a solution that learning systems converge to, or merely an “accident” of biological evolution. Our results point toward the former: even when explicitly trained to predict their true accuracy, LLMs still converge to a form of confidence closer to consistency away from their training data, replicating these cognitive science findings in a new kind of system explicitly trained to predict true accuracy and not consistency. These results suggest that this solution could be optimal in general and could motivate future work to verify this hypothesis further. We updated the discussion to make this discussion and the positioning of the paper more apparent.

  2. Future work. Reviewers asked, in different forms, for more evidence at low and moderate distances from the training data. Given the short discussion period we could not run this additional experiment. We propose to add test splits for MMLU-Pro and MedMCQA closer to their training data for future work. Combined with the current topic-disjoint splits, this would provides near and far test sets within each dataset and allows us to test whether the MATH results replicate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.