acceptodds
Under review as a conference paper at ICLR 2027

Introspection or Post-Hoc Rationalization? A Cross-Model and Cross-Domain Study of LLM Metacognition

Abstract

Recent literature has increasingly studied the metacognitive and introspective capabilities of Large Language Models (LLMs), suggesting they may possess access to their internal states. However, a self-report can be *accurate* without being introspective. An accurate report may be caused by truly accessing the internal state (*introspection*), by the model recomputing that state in a new context (*self-simulation*), or by exploiting information that merely correlates with its internals (*post-hoc rationalization*), a route also known in LLMs as *causal bypassing*. In this paper, we empirically evaluate current introspection benchmarks under the core thesis that accuracy is *not* access. Because these evaluations score self-reports by accuracy, which alone cannot distinguish these routes, we introduce two complementary methodologies and apply them to test instruction-tuned LLMs from families, spanning B to B parameters. First, a *cross-model test* asks whether the rationalization route is available by giving external reporters exactly the inputs the self-reporter receives. We found that external reporters match self-accuracy, indicating the task is solvable via rationalization. A small residual self-advantage remains in some cases, which is compatible with a privileged route but cannot distinguish introspection from self-simulation. Second, an *introspection generalization framework* asks whether training on these evaluations builds a general route to internal states or only task-specific heuristics. Across task families and datasets, in-domain fine-tuning gains largely fail to transfer across tasks. Together, these results do not claim models cannot introspect, rather, they show that current evaluations cannot tell the routes apart, and that training on introspection tasks strengthens task-specific rationalization rather than privileged access.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Introspection, Metacognition

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_8KeoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper implements a number of paradigms that various other researchers in the field have created, on a range of different model sizes and families. It tests privileged self access, through measuring whether other models can predict self-report labels as well as the model itself, and generalization of fine-tuning on one introspection task to others, on the logic that failure means the training taught shortcuts. They authors report that models mostly do not outperform external observers, and that training mostly does not transfer across tasks. The authors conclude that the evaluations cannot establish introspection.

Critical strengths

The paper adopts a sound definition of introspection, and critical appraisals of reported results in the field are welcome. It includes an impressive breadth of tests and a diverse set of models.

Critical weaknesses

Such a paper's conclusions can only be as strong as the paradigms it uses, and there are far too many for me to validate either their logic as valid tests of introspection or the correctness of their implementation. In general these paradigms are not widely used "benchmarks" as the paper claims, but are custom made for specific investigations.

I take it as given that there are numerous overclaims about introspection in the literature, but this paper seems to take the opposite tack, either shrugging off or ignoring the four positive self-access tasks, the numerous off-diagonal transfers, and the fact that Qwen-27B in particular looks to be relatively robust in terms of transferring to other tasks, except for steering.

Questions for authors

You suggest that Qwen-27B's "large gain may reflect answer-format learning" but say you didn't test it. Why not? If you want to claim that "training on introspection tasks strengthens task-specific rationalization rather than privileged access" it seems important to settle whether the gains are indeed task-specific.

3 comments

We thank the reviewer for the critical reading and for appreciating both our definition of introspection and the breadth of our evaluation. Below, we address the questions and concerns. If those address or mitigate your concerns, we would appreciate it if you could update the score accordingly.

[W1] R: "there are far too many paradigms for me to validate either their logic as valid tests of introspection or the correctness of their implementation."

We understand that the number of tasks makes them hard to check within a review. Regarding implementation, each task is taken from the literature and follows the original data and label definitions. Appendix C maps every reported number to the command that produces it, so that any single paradigm can be checked independently.

Regarding the logic of the paradigms, our experiments take existing introspection tasks and ask whether each one, as defined in the paper that introduced it, can be solved without privileged access.

R: "In general these paradigms are not widely used "benchmarks" as the paper claims, but are custom made for specific investigations."

We agree that these tasks are not widely used "benchmarks", and now call them "evaluation paradigms" throughout the paper to reflect the reviewer's suggestion. They are, however, the paradigms on which published introspection claims rest, which is why we test them.

[W2] R: "I take it as given that there are numerous overclaims about introspection in the literature, but this paper seems to take the opposite tack, either shrugging off or ignoring the four positive self-access tasks, the numerous off-diagonal transfers, and the fact that Qwen-27B in particular looks to be relatively robust in terms of transferring to other tasks, except for steering."

Our aim is not to argue that models cannot introspect, but to identify which positive results current paradigms can attribute to privileged access. We agree the submission discussed the positive evidence too briefly, and the revision adds a section, "What positive evidence survives?" (Sec. 5.3), which examines it. Below we discuss the positive evidence the reviewer listed:

Four positive self-access tasks. On 1st char, 1st word, among, and ethical stance, the self-reporter keeps a residual advantage over external reporters (+4.4 on average), although external reporters still learn these tasks well without privileged access. Our test cannot attribute this advantage to introspection rather than self-simulation or familiarity with one's own outputs. The submission noted this only briefly (abstract and Sec. 5.1). The revision discusses it explicitly (Sec. 5.1 and 5.3) and corrects the Conclusions, which mentioned only self-simulation, to state that a small privileged component remains possible.

Qwen3.5-27B. Its broad transfer came from an error with the thinking mode in the baseline of Qwen models (see Q1 for details). With the corrected baseline it no longer stands out.

Off-diagonal transfers. With the corrected baseline, none of the seven significant off-diagonal gains of the submission remains significant (App. B.9). The clearest remaining transfers are between tasks that share their target (e.g., predicting the first word fixes the first character), which follows from learning regularities of the model's outputs that external reporters also learn (Sec. 5.1). The largest remaining gains (Ministral-3-14B, into confidence reporting tasks) are a dataset-specific response bias: they come from the same few confidence datasets whichever task was trained, while on other datasets the same adapters fall below the majority class.

[Q1] R: "You suggest that Qwen-27B's "large gain may reflect answer-format learning" but say you didn't test it. Why not?"

Thank you for spotting this outlier and pressing this point. Re-testing it revealed an error in our zero-shot baseline, which we have now corrected. The script for computing the zero-shot baseline did not disable Qwen's default thinking mode, while the training script correctly did. Therefore, the untrained Qwen models (i.e. the zero-shot baselines rows in Tables 1 and 35) ran out of tokens while reasoning before giving an answer, and scored 0% on most tasks. This made off-diagonal gains look big but they actually are not. We also re-checked all other model configurations and no other model is affected (Gemma-4 defaults to thinking off, Llama and Ministral have no thinking mode). All the other numbers in the tables (e.g. fine-tuned results and the cross-model test) are also correct and unchanged. We have corrected Tables 1 and 35 and App. B.9.

With the corrected baseline, the off-diagonal gain disappears: after training on most other tasks, Qwen3.5-27B performs like the untrained model, and no Qwen transfer remains a significant gain (App. B.9). The in-task gains remain, so the gains are task-specific.

I appreciate the authors' commitment to truth, but the fact that they discovered a coding error that substantially changed some results in response to my question does not allay my concerns about my (and anyone's) ability to validate such a large number of paradigms. Therefore, I maintain my score.

We thank the reviewer for the reply. We would like to clarify that the issue was a configuration setting rather than a coding error, affected only the zero-shot baseline rows of the Qwen models, and did not change the conclusions of the paper, but in fact strengthened them.

Regarding verifiability, could the reviewer suggest concrete steps that would make the implementation easier to validate? We have already released all the code, which follows the original implementation of each paradigm, and Appendix C maps every reported number to the command that produces it. We would like to make sure that we don't face a similar criticism in future revisions.

Reviewer_dJh1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary
  • Existing introspection evaluations cannot distinguish between introspection, which requires privileged access to internal states, and rationalization, which is a shortcut drawing on internal states and inputs.
  • The authors argue that some previously reported introspection can be explained as rationalization, since other LLMs and embedding-based classifiers achieve performance comparable to self-prediction.
  • Fine-tuning an LLM on one introspection task does not generalize to most other introspection tasks, which suggests that most of the routes acquired through introspection training are task-specific.
Critical strengths
  1. Prior studies on introspection in LLMs have each used their own definitions and experimental setups, so their conclusions have been scattered. A unified analysis of these studies will be valuable for future research.
Critical weaknesses
  1. The paper presents introspection and self-simulation as separate routes, but I found the distinction hard to follow. Based on the notion of introspection in psychology (L41–43), self-simulation appears to be an instance of introspection. If your definition (L55–57) separates the two, how does it differ from what psychology calls introspection?
  2. I question whether training the external LLMs with LoRA in the cross-model test is a fair comparison. It may make the behavior of the external LLM unnecessarily similar to that of the original LLM, and so underestimate the original LLM's introspective ability.
Questions for authors
  1. Related to Weakness 2: could using external LLMs that behave similarly to the original LLM, for example because of similar training data, lead to underestimating its introspective ability?
2 comments

We thank the reviewer for the careful reading, for recognising the benefits of having a unified analysis for introspection tasks, and for the positive feedback. We hope the following clarifications address their questions.

[W1] R: "The paper presents introspection and self-simulation as separate routes [...] If your definition (L55-57) separates the two, how does it differ from what psychology calls introspection?"

The setup of our paper is aligned with the psychological definition in L41-43 where introspection is privileged access to one's own states, and self-simulation is indeed an instance of it. In other words, our evaluation distinguishes privileged vs. non-privileged access (L60-62). Under it, introspection and self-simulation are both privileged, and rationalization is not.

In L55-57 we use "introspection" in the narrower sense adopted in recent LLM work (Lindsey, 2025), which additionally requires the report to be grounded (i.e. causally dependent on the internal state that occurred in the original computation). Self-simulation does not meet this criterion, as the model re-runs the computation in the reporting context with the same weights and reports on the re-run, so the report depends on its weights but not on the original state. For example, suppose a model answered a question X with "Cat" and was then asked how confident it was. It could read out the uncertainty represented when it produced "Cat" (introspection), or re-pose X and check how strongly it now favors "Cat" (self-simulation).

Both routes produce the same report and both outperform an external reporter, so behavioral evaluation cannot tell them apart. Our results therefore focus on establishing if privileged access is used or not, i.e., introspection in the broad sense, and make no claim about which route a model uses. Testing grounding would require intervening on the original state while keeping the input fixed and we leave it to future work.

[W2] R: "I question whether training the external LLMs with LoRA in the cross-model test is a fair comparison. It may make the behavior of the external LLM unnecessarily similar to that of the original LLM, and so underestimate the original LLM's introspective ability.”

We take this opportunity to clarify that the cross-model test does not estimate the source's introspective ability. It asks whether the task requires privileged access (L174). If an external reporter can reach the source's accuracy from (input, label) pairs alone, including by learning to behave like the source from those labels, then the task can be solved without access to the source, and success on it is not evidence of introspection. This does not imply that the source lacks introspective ability, as we state in the Conclusions (L441: "LLMs may still introspect, but accuracy on these evaluation paradigms cannot establish it").

Furthermore, the comparison is symmetric, because the self-reporter is fine-tuned on the same pairs with the same procedure. So it has everything the external reporter has, plus whatever access it has to its own states. Any similarity that fine-tuning creates is available to both.

Additionally, the LoRA data do not contain the source's responses, only a property of them, such as a confidence label or the first character or first word of the answer. For instance, for "What is the capital of France?" the reporter learns the source's confidence label, not what the source answered. It never sees the source's activations. The reporter is thus trained to predict a property of the source's behavior, not to reproduce it.

[Q1] R: "could using external LLMs that behave similarly to the original LLM, for example because of similar training data, lead to underestimating its introspective ability?"

Pre-training corpora are not disclosed for any of the models we study, so overlap cannot be controlled directly. What can be controlled is the choice of reporters, which is why the cross-model test spans four model families.

Following the reviewer’s question, we tested whether reporters from the same family have an advantage compared to other-family external models. Using the same analysis we apply to the self-advantage (Sec. 5.1), we compute for each reporter-source pair the gain. This measures how many points more accurate the reporter is than expected (given how well it reads other sources and how readable the source is to others). All reporters learn to predict the source well, and the gains follow a clear ordering: the self-reporter is best (+4.4 pp), same-family reporters have a slight advantage (+1.3), and different-family reporters none (0.0). The nature of the self-advantage remains open: it may reflect privileged access, but also self-simulation or familiarity with one's own output distribution (Sec. 5.3). The same-family advantage is small enough not to affect our results and thus similarity between models does not lead us to underestimate the self-reporter (new paragraph in App. A.5).

Thank you for your detailed response. The authors' explanations and additional experiments have addressed my concerns, so I'll increase my soundness score and maintain my positive rating.

Comments on the paper

5 comments

We thank the reviewer for a careful and constructive review. Many of the points raised are requests for additional analyses or clarification, and we found them helpful for strengthening the paper. Below, we answer each point and report the new analyses.

[W1] R: "the abstract concludes that training strengthens task-specific rationalization "rather than privileged access."

We agree, Sec. 5.3 (5.4 in the revision) already stated this conclusion. We now align the abstract and Sec. 3.1 as well.

[W2] R: "Please use a consistent checkpoint, evaluation split, and scoring definition within each transfer comparison."

The steering numbers the reviewer cites measure different capabilities: detection of an injected concept (Tab. 1 and 35), naming of held-out concepts (App. B.8), and a composite score that requires both (Tab. 46). Within each transfer matrix, however, the diagonal, the off-diagonal cells and the zero-shot baseline are scored with the same metric, so every diagonal-versus-off-diagonal comparison is internally consistent. We have revised the captions so that each states which metric it reports, and we will further align the steering results across tables in the camera-ready version.

R: "A failure to transfer can therefore reflect difficulty following a new reporting format."

We tested this directly by measuring how often each answer is in the requested format, for the models with the largest off-diagonal effects (Qwen3.5-9B, Qwen3.5-27B, Llama-3.1-8B; new App. B.10). In most cells format is not the issue: off-diagonal answers are well formed in 93.7-100% of cases on average, so transfer failures generally occur with well-formed answers. Format failures are confined to adapters trained on self-prediction tasks, mostly word tasks, which keep answering with a word when a character or True/False is requested.

[W3] R: "…the word-task parser drops single-character outputs, so a correct first word such as "a" can never receive credit."

We fixed the parser so that single-character words are credited, and re-scored the comparisons. In the generalization matrices the fix changes no cell, because the fine-tuned models never answer a word task with a single character. In the cross-model test it matters for one source, Gemma-4-31B, whose 1st-word labels are often "I". Since Gemma-4-31B is never a reporter, the shift is the same for every reporter reading it, and the self advantage is essentially unchanged (−0.05 points). We will update the affected cells in the camera-ready version.

R: "24-55% of the medals test questions also occur verbatim in training."

We re-scored medals on the deduplicated test split, removing every test question that also appears in training. The static classifiers lose 6-12 points but still beat the majority class for 7 of 8 models, and still match or beat the fine-tuned models where the comparison is possible. The cross-model self advantage on medals stays small (from +0.9 to +0.4, App. B.4).

R: "[…] whether self-prediction and confidence labels are refreshed after fine-tuning."

No. The labels are those of the base model, because the base model is the object of study. As the reviewer notes, fine-tuning may change the model's own answers, so we measured it (App. A.5, Table 57). After fine-tuning, Qwen3.5-9B's answers change on 3-7% of rows for multiple-choice tasks and on 26-65% for free-text tasks. On the changed rows only, we ask whether the fine-tuned model predicts its new answer or its old one, and compare with the external reporters, which were trained on the same old labels and never saw the fine-tuned model. The self-reporter matches its new answers no more often than the external reporters do, and its advantage lies only in the old answers it was trained on (e.g. 1st char: new answers 33% vs. 29-34%, old answers 46% vs. 25-29%). Scoring against the original labels therefore does not disadvantage the self-reporter.

[W4] R: "Please report paired confidence intervals for the main self-cross differences and define what difference would count as practically negligible."

We agree, and we have changed both the analysis and the wording (Sec. 5.1, App. A.5). We now use a paired, hierarchical bootstrap: the self-reporter and the external reporters are compared on the same evaluation rows, resampling both training seeds and rows. We define ±2 points as practically negligible (about the between-seed standard deviation of a single LoRA cell; ±3 gives identical verdicts) and claim parity only when the 90% interval lies within this margin. Over the seven main datasets, self exceeds cross by +4.3 points (95% CI [+3.1, +5.5]). On 2nd char, math and medals, self and cross are equivalent (+0.6, 90% CI [-0.6, +1.7]); on the other four datasets the self advantage is clearly non-zero (+7.2, 95% CI [+5.3, +9.0]).

R: "…especially for activation-monitoring LoRA results based on 100 test examples and one repetition."

We agree that this comparison is limited on its own; it was an additional check. The activation-monitoring conclusion rests on the in-context results, where self and cross differ by less than half a point on both datasets. The LoRA comparison points the same way: self and cross differ by less than two points, which rules out a large self advantage but cannot establish parity at a fine margin.

R: "For transfer, uncertainty should include the sampled evaluation examples as well as training seeds."

The tests of App. B.9 previously captured only the variation across training seeds; they now also include the sampling error of the evaluation examples. The latter dominates (median standard error 2.6 vs. 0.35 points). With it, no off-diagonal gain is significant in any model, all significant off-diagonal effects are drops, and in-task gains remain significant where they are large. Sec. 5.2 now reports these results.

R: "The residual self-advantage […] is a result to explain. The disappearance of the advantage […] is not decisive"

We agree on both points. The revision discusses the residual in a dedicated section (Sec. 5.3) and no longer uses the in-context results as evidence against it. Three analyses bear on it: it appears only on generation-dependent tasks and does not grow from an 8B to a 14B reporter; same-family reporters share part of it (+1.3 vs. +4.4 points for the self pair); and the label-refresh test (W3) shows that the self-reporter's advantage lies in the behaviour it was trained on rather than in its current behaviour. It is compatible with a small privileged component, but also with self-simulation or familiarity with one's own outputs, and our test cannot separate these.

We thank again all three reviewers for their detailed and constructive reviews. We have uploaded a revised PDF that integrates their feedback and the additional analyses they asked for. The main additions are:

New Sec. 5.3, "What positive evidence survives?" (8Keo, YBTm): a dedicated discussion of the residual self-advantage of the cross-model test and of the remaining off-diagonal gains of the generalization test.

Uncertainty and equivalence (YBTm): a paired, hierarchical bootstrap for the self-cross comparison with a ±2-point equivalence margin (Sec. 5.1), and transfer tests that include the sampling error of the evaluation examples as well as training seeds (Sec. 5.2, App. B.9).

Additional controls:

  • same-family vs. different-family reporters (dJh1, App. A.5);
  • label refresh after fine-tuning (YBTm, App. A.5);
  • deduplicated medals (YBTm, App. B.4);
  • answer-format validity of the fine-tuned checkpoints (YBTm, 8Keo, new App. B.10).

We also made the following clarifications and corrections:

  • The abstract, Sec. 3.1 and the Conclusions now state more precisely what the tests show (YBTm, 8Keo), and we refer to the tasks as "evaluation paradigms" rather than "benchmarks" (8Keo).
  • While addressing 8Keo's question on Qwen3.5-27B, we corrected the zero-shot baseline of the Qwen models (Tables 1 and 35, App. B.9), which removes Qwen's apparent cross-task transfer, while the other numbers are unaffected by this correction.

None of these changes alters the main conclusions, and the new analyses support them and make their scope more precise.

We answer each reviewer in detail in the individual threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.