acceptodds
Under review as a conference paper at ICLR 2027

A Mechanistic Study of Language Model Introspection

Abstract

Large language models (LLMs) can sometimes report perturbations to their internal activations—even when the input provides no evidence that an intervention occurred. How do models detect and localize such internal changes? We study this question using a controlled task that keeps the input text fixed. We either inject a concept vector into the hidden state at one of ten token positions or apply no intervention. The model is asked to identify the perturbed position or report that no intervention occurred. Across three model families, we identify two small groups of attention heads with distinct roles in introspective reporting. Middle-layer *gate heads* influence whether the model reports a change, while *router heads* in a later layer help select the position to report. Interventions on gate heads can suppress position reports even when router heads supply location information. We further examine why reporting accuracy varies across concepts. Concept vectors that are localized more accurately produce stronger attention-score and output responses in gate heads, which is associated with better alignment of the induced key and value changes in their QK and OV computations. Together, these findings identify attention-head mechanisms supporting introspective detection and localization.

Primary area:
interpretability and explainable AI
Keywords:
LLM introspection, mechanistic interpretability, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BcFoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper attempts to provide a mechanistic attribution of introspection, namely, the ability of an LLM to actively detect whether it has been affected by activation steering. The authors identify several attention heads, establish their causal influence on introspective outputs, and present some scattered observations about these attention heads.

Critical strengths
  1. The paper selects a rather interesting task, namely, introspection.

  2. The experiments are reasonably solid. However, this rigor cannot conceal the fatal shortcomings in scientific contribution that I discuss in the weaknesses.

Critical weaknesses
  1. This paper is highly representative of what has been called "Slopterpretability" at COLM, namely:

    1. Carves off a narrow question: e.g., what is the mechanism behind refusal in multi-turn jailbreaks? (not a real submitted paper)
    2. Performs interpretability analysis using off-the-shelf tools, like circuit discovery or SAEs
    3. Reports the results and draws conclusions, sometimes with exaggeration
      (https://gregdurrett.github.io/colm2026-blog/ai-papers.html)
    • My criticism of this paradigm is that it does not provide substantial insights or generalizable knowledge. We must be wary of the fact that research following this paradigm essentially uses one black box, the attention head (note that we still lack effective tools for understanding what an attention head actually computes), to explain another black box. It does not yield computational principles that humans can understand (considering that mechanistic interpretability aims to reduce a computational process to human-understandable pseudocode, merely chaining attention heads together is clearly not an ideal reduction), nor does it yield more generalizable knowledge. For example, just to give one possible direction, a genuinely valuable piece of generalizable knowledge might be: "Why do Transformers exhibit this kind of introspection in the first place? Presumably, such injection scenarios are entirely absent from the training data, so why does this capability emerge? What deeper properties of the training process does this reveal?"

    • This criticism manifests itself in many concrete ways. For example, and particularly in the "QKOV" analysis of the alignment between the singular vectors of attention-head parameter matrices and the perturbation directions, the reported phenomenon is simply an inevitable property of attention-head computation. Perturbations aligned with the singular vectors of attention-head parameter matrices will naturally have a greater effect on the computation, as a direct mathematical consequence. Experimentally verifying this merely confirms well-known principles of linear algebra and does not provide us with additional knowledge. This also explains what I personally mean by a "lack of novelty": the issue is not how new or old the investigated phenomenon is, but whether we provide new, systematic scientific knowledge rather than merely local or scattered observations of a phenomenon.

  2. Beyond the novelty problem inherent in the aforementioned research paradigm, note that prior work has already used dictionary-learning tools to identify what the authors call "Gate Features," which determine whether an injected perturbation is present. Intuitively, if such features exist, there must also be a set of attention heads that produce them, because covariance along a particular direction in the hidden space cannot arise out of thin air. There must be some attention heads whose outputs (the so-called OV circuits) are aligned with that direction. Such features are also likely to be aligned with the inputs of certain attention heads (the so-called QK circuits); otherwise, a dangling feature that has no influence on the output would likely be discarded by normalization. Therefore, from an information-theoretic perspective, given the findings of prior work, the experiments in this paper can not further reduce our uncertainty about how LLMs operate or behave to an even lesser extent.

  3. Although I do not believe this paper is entirely AI-generated, and it clearly contains some human refinement—which is already remarkably rare in this ICLR cycle—many unpleasant and difficult-to-read characteristics of AI-generated writing have still not been thoroughly eliminated. However, this point is not critical, but I offer a bonus for the "not entirely AI-generated", as I believe Weakness 1 alone already constitutes sufficient grounds for rejecting this paper.

Questions for authors

See weaknesses.

6 comments

Response to Weakness 1.1: Novelty and Generalizability

We would like to respectfully clarify that our task is not simply Slopterpretability. Operating at the fine-grained attention-head level does not imply a lack of novelty; well-known works in the interpretability field have likewise understood LLMs in this way.

  • In Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, Wang et al. studied a single controlled task and classified the different heads.
  • In Retrieval Head Mechanistically Explains Long-Context Factuality (ICLR 2025), Wu et al. likewise identified retrieval heads on a retrieval task.

We define an objective, controllable task that satisfies the requirements of introspection, and provide a detailed analysis of the entire mechanism by which an LLM reads whether it has been intervened upon and outputs the final position of the intervention. Therefore, using specific attention heads to explain certain functions also provides a possible avenue for humans to understand LLMs.

Furthermore, we test our method across three model families and find it effective on all of them. In the interpretability literature, validating a general framework simultaneously across different model families is far from trivial, which indicates that our analysis approach has strong generalization and universality.

Response to Weakness 1.2: Human-Understandable Computational Mechanism

We thank the reviewer for raising questions about our detection process, which we did not express very clearly in the paper. We analyze the attention patterns of the router head under gate on/off conditions. When the gate is off, the router head's attention to the positions it previously attended to sharply decreases, while its attention to other positions increases uniformly. This illustrates the circuit relationship between the two components: the gate must be on for the router to take effect. This reflects the overall circuit process. Specifically, we can summarize the mechanism as:

if gate is closed:
    output none
else:
    output the position selected by the router

This simplified form clarifies how we understand the mechanism, and therefore we have not failed to address the actual detection process.

Response to Weakness 1.3: Mechanistic Understanding

We thank the reviewer for this deep question, which we fully agree is important. Understanding why such self-monitoring emerges despite its absence from pre-training data would indeed reveal something fundamental about the training process; to our knowledge, this remains an open problem across the field. Our work addresses the more proximal how reducing the mechanism to causally-validated gate/router heads which we see as a necessary first step toward that deeper question. We are actively pursuing this direction as future work, and thank the reviewer for pushing us to state it explicitly.

Response to Weakness 1.4: QKOV analysis

We perform SVD on the matrices ΔK\Delta K and WO(ΔV)⊤W_O(\Delta V)^\top that induce the activation change, which is a routine operation in linear algebra. However, we would like to emphasize that removing the top five OV singular modes has an effect close to removing the full value-change term, providing a concise functional explanation supported by causal intervention, whereas removing the remaining modes has a smaller impact on localization accuracy (Appendix C.2). This reveals a low-rank, causally interpretable mechanism through which introspective behavior can be directly and efficiently manipulated, rather than requiring diffuse modification of the entire value-change term. Moreover, the difference in alignment between the high-accuracy and low-accuracy concept groups is larger than the difference in mode magnitude. These results are not directly guaranteed by SVD identities.

response to Weakness 2: feature or head comes first

We would like to first emphasize a key point: at the final token position, attention transfers the perturbation information first, and only afterwards does the MLP read out the resulting feature. Macar et al. call their analysis results "Gate Features," whereas we call ours "gate heads"; we acknowledge the naming similarity and fully agree that "if such features exist, there must also be a set of attention heads that produce them." However, Macar et al. examine MLP features at the final prompt position, whereas in a Transformer only attention can transfer information across positions. The information at the final position must therefore arrive via attention in earlier layers, meaning the MLP features they analyze with a transcoder are merely a downstream readout of this attention-transferred information. Therefore, this causally-validated, rule-like circuit of gate and router heads provides a possible avenue for humans to understand how the model detects and reports intervention.

Furthermore, we use a straight-through estimator to study component-level joint optimization, resolving the limitation of per-head attribution that ignores synergistic effects, whereas Macar et al. select features based on attribution scores and overlook the joint synergistic effects across different layers.

response to Weakness 3: writing We acknowledge that our writing is not sufficiently clear in this regard, and we will revise the content and presentation of the paper in the subsequent version.

Thanks for your reply (with sincere human language)!

  1. I am not fully convinced by your defense of the novelty of this work. We cannot simply argue that because similar approaches have been used in the past, they are therefore immune to criticism. Moreover, the works on discovery heads or steering vectors that you cited are already several years old, which is quite outdated for such a rapidly evolving field (this is minor). Attempts to extract certain heads, circuits, or intermediate computational results are essentially using one black box to explain another. They do not really explain how the computation works.

  2. The same concern applies to your defense of the computational mechanism. I understand that this weakness may put you in a difficult position and force you to speculate about a mechanism, such processing may have a negative psychological impact. However, please note that one of the key components of your proposed mechanism is the "gate". The current experiments (I think) do not explain where this gate comes from. It remains an oracle produced by a black box. Therefore, what you have presented is not really a complete computational mechanism, but rather the final stage of one.

  3. I will respond to your rebuttal of my third point a little later.

  4. Please note that ICLR allows authors to update their PDFs during the rebuttal period. Therefore, rather than merely promising to revise the paper, you should actually make those revisions. Otherwise, it will be difficult for me to reconsider my evaluation.

Reviewer_XWvWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper explores the granularity and quality of intervention self-report in language models. The extent to which language models can articulate properties of their intermediate computation has been a topic of recent interest, this paper studies a specific instantiation of intervention detection by language models. To that end, they provide a set-up where the language model is provided with 10 tokens in context and the task is to detect whether any of those tokens were intervened on, if yes then which. The models show non-trivial (well above change) accuracy on the task, therefore they study the mechanisms that lead to the task performance.

The authors pose a two component circuit for this task, first a determination of whether an injection has happened is made and this followed by the determination of the position of the intervention. They provide supporting evidence for this two-component structure using attention-head analysis and localizing the different components to a group of heads (separately).

Furthermore, the paper notes that model accuracy for self-report depends on the injection concept and conduct analysis to explain this dependence.

Critical strengths
  1. The paper provides an interesting new granularity and task to study how models detect steering interventions.
  2. The task is reasonably specified with clear interpretations of model accuracies and chance accuracies.
Critical weaknesses
  1. The paper does not answer any of the mechanistic questions posed. A general criticism that holds for all of the mechanistic claims is that they never separate "detection" from literal "reporting". For instance, the evidence they show for the heads they identify for the model reporting "yes" could be explained by those heads just being responsible for promoting the tokens associated with a positive report once a determination has already been made earlier in the model. If this easier hypothesis is true, then we would expect positive report to be supressed if those heads are supressed and the heads can be used to simply increase the likelihood of "yes". The crux of the question is not about which heads are responsible for a positive report because that question simply collapses to which heads are responsible to say "yes". The more important question, unanswered by their analysis, is about the actual detection process. None of their experiments make any progress towards this.
  2. Similarly, the authors pose the question about the mechanistic analysis of model accuracy on the task and its relationship to the steering-concepts. The analysis looks into the change induced by a steering vector on the KQ and OV matrices. Specifically, they note that for the concepts where accuracy was high, the alignment (of injected query) is more important than the absolute size change in the key matrix. It is possible that this is a circular finding about attention itself; by just comparing cases where the model failed v/s where it succeeded -- the authors risk making a circular claim. They need to at least make a prediction about concept vectors not used in the analysis, this claim would be more convincing if they could provide (or even learn) a threshold and show that the alignment of an unseen steering vector can be used to predict self-report performance on that concept.
Questions for authors

Would it be possible to define an experiment on the detected circuits that increases the accuracy of the model? As opposed to increasing the probability of a particular report.

6 comments

Response to Weakness 1

1. Response to the Concern About Independent Detection and Reporting Mechanisms

We argue that detecting and reporting are functionally distinguishable but need not be realized by fully independent mechanisms. The key question is what the model detects and whether its answer is grounded in access to internal information about that object.

Macar et al., Mechanisms of Introspective Awareness, likewise do not require detecting and reporting to be treated as coupled mechanisms. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small also analyzes internal mechanisms through the logit trends of reporting.

In our paper, we analyze the model's detecting and reporting processes by observing its internal representations. As shown in Figure 2a, the model's realization that it has been intervened upon is a gradual process, not accomplished through a single layer. Accordingly, we select the key gate heads in these layers to explain this phenomenon.

We then localize the key gate heads in the relevant layers and verify their role in the circuit by intervening on them and observing changes in the attention patterns of the downstream router heads. Therefore, our mechanistic analysis involves not only changes in the final report, but also the cross-layer evolution of internal signals and the functional dependence between the gate and the router.


2. Response to the Concern About the Detection Process

We thank the reviewer for raising questions about our detection process, which we did not express very clearly in the paper. We analyze the attention patterns of the router head under gate on/off conditions.

When the gate is off, the router head's attention to the positions it previously attended to sharply decreases, while its attention to other positions increases uniformly.

This illustrates the circuit relationship between the two components: the gate must be on for the router to take effect. This reflects the overall circuit process. Specifically, we can summarize the mechanism as:

if gate is closed:
    output none
else:
    output the position selected by the router

This simplified form clarifies how we understand the mechanism, and therefore we have not failed to address the actual detection process.


3. Response to the Concern About “Yes” Semantics in Gate Heads

We thank the reviewer for raising the concern that the gate heads we selected may simply carry the semantics of "Yes." However, we demonstrate through experiments that the gate heads we select do not promote or suppress "Yes" responses.

When we reuse the gate heads selected in the paper and reverse the required output, instructing the model to output "No" when injected and "Yes" when not injected, turning the gate on (promoting the output of these heads) while the LLM is not injected causes the model to say "No" (see Table 1). In other words, these gates do not promote "Yes"-type affirmative responses; rather, they make the model believe it has been injected.

CombinationQwenLlamaGemma
Digits (ordered)70.58%84.24%97.48%
Digits (shuffled)60.34%100%99.39%
Letters (ordered)52.45%100%97.74%
Letters (shuffled)55.22%100%99.09%
Words (ordered)64.06%99.97%98.80%
Words (shuffled)67.90%100%99.62%

Table 1. Gate on without injection: No-response rates (%).

Similarly, when we require the model to reply "Yes" when not injected and suppress the output of these heads while the LLM is injected, the LLM still outputs "Yes" (see Table 2), indicating that suppressing these heads does not suppress "Yes"-type affirmative responses. Therefore, the gate heads we identified genuinely reflect whether the model is aware of its own internal state, rather than merely encoding an affirmative or negative semantic bias.

CombinationQwenLlamaGemma
Digits (ordered)94.54%70.07%92.33%
Digits (shuffled)79.12%4.01%72.30%
Letters (ordered)89.12%83.07%91.69%
Letters (shuffled)87.04%50.72%72.13%
Words (ordered)75.14%82.31%92.22%
Words (shuffled)49.43%22.38%72.09%

Table 2. Gate off with injection: Yes-response rates (%).

Response to Weakness 2

We thank the reviewer for raising the concern that our QK analysis may involve a circular claim. We therefore supplement the following experiment. On held-out concepts, we use the midpoint of the average values of the QK metric ∣ψ1⊤qI∣|\psi_1^\top q^I| and the OV metric ∣cos⁡(aI,xk)∣|\cos(a^I, x_k)| (for k=1,2,…,5k=1,2,\dots,5) defined in Section 5, computed over the original introspective and non-introspective groups, as the threshold, and classify 100 new "intro" concepts and 100 new "nonintro" concepts. The selection of the gate heads and the threshold is entirely independent of the new concepts. The new introspective and non-introspective concept words are randomly sampled from the entire vocabulary, excluding the concepts used in the original paper.

The QK/OV accuracies for Qwen, Llama, and Gemma are 97.5/98.5%, 66.0/81.0%, and 92.5/99.5%, respectively. This shows that our QK/OV analysis can fundamentally distinguish introspective from non-introspective concepts, indicating that introspective and non-introspective words are indeed separated by the gate head within the model.

Response to Question

Using the router heads mentioned in our paper, during inference we concentrate the attention at the final prompt position of each identified head entirely onto the position with the highest weight in the sequence. This optimization relies solely on the model's own inference process. Moreover, for each of the three models we intervene on only 1–2 heads, and this intervention improves the localization accuracy of Qwen, Llama, and Gemma by 5.40, 4.32, and 4.68 percentage points, respectively, across the six labels.

please provide the numbers for these tables where you do the injection but don't do anything to the attention heads (in your reversed setting)

Reviewer_FLQkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how language models detect and locate activation perturbations. It injects a concept vector at one of ten token positions and asks the model to identify that position or report no intervention. Across three model families, activation-patching experiments identify attention heads that influence whether a perturbation is reported and separate heads that influence the reported location. Further analysis associates successful localization with stronger alignment in the former heads’ attention computations. The authors interpret these findings as a mechanism supporting introspective reporting.

Critical strengths

N/A

Critical weaknesses

Missing many citations. I feel like the authors do not interact with the field in any meaningful depth.

The results are highly incremental. The paper replicates prior work on introspection, with detecting introspection from a steering vector, then looks for attention heads that are associated with 'realization' about introspection. The paper doesn’t convincingly distinguish its proposed introspection mechanism from ordinary answer selection.

Macar et al. already report evidence-carrier/gate mechanisms and a partial separation between detection and concept identification. This paper adds position localization, native attention-head analysis, and useful crossed interventions. Those are legitimate contributions, but “two components underpin introspection” is not itself new

Questions for authors

N/A

3 comments

Response to Weakness 1: Missing Citations

We sincerely thank the reviewer for pointing out the insufficient discussion of related work. Section 2 of the current manuscript already covers several research directions directly relevant to this paper: empirical evidence on model introspection and self-reporting (Binder et al.; Lindsey), critical evaluations of introspective explanations (Singh et al.; Hahami et al.), studies of introspection mechanisms (Macar et al.), as well as activation steering, attention-head circuits, and causal intervention methods. Appendix A.1 further discusses the relationship between introspection criteria and the task design of this work. Due to space constraints in the main text, our introduction and comparison of these works are relatively brief. We will expand this discussion in the revised version and add citations to papers such as Betley et al.'s "Tell me about yourself: LLMs are aware of their learned behaviors" and Plunkett et al.'s "Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions." We also welcome the reviewer to point out any omissions they consider particularly important, so that we can further improve the related work.

Response to Weakness 2: Contribution

We argue that answer selection and functional introspection are not mutually exclusive explanations. The key question is what the model detects and whether its answer is grounded in access to internal information about that object. Binder et al., Looking Inward: Language Models Can Learn About Themselves by Introspection, agree that the internal mechanisms that ordinarily produce answers can be reused, without requiring a separate introspection module. Likewise, Macar et al., Mechanisms of Introspective Awareness, do not require the gate to be a pure detection module independent of answer output. Nevertheless, our experiments provide evidence against some simpler anomaly-selection explanations for the observed reporting behavior.

  • When we replace the word at the injection position with an injected word, the model does not believe it has been injected (Appendix A.5), ruling out anomalies caused by the injected word itself.
  • Although the gate head and router head were selected under the single Ordered Digits setting (sequential digits 0–9), their function nevertheless generalizes to Letters (A–J), Words (one–ten), and randomly permuted order mappings, demonstrating that this is not a simple position-selection mechanism (Appendix A.18).

Furthermore, based on the values in Table 2 and Figure 3, we can approximately abstract the mechanism as:

if gate is closed:
    output none
else:
    output the position selected by the router

This simplified if–else decision framework demonstrates that the gating mechanism performs second-order modulation of the expression of internal state information, directly supporting the functional metacognitive interpretation in our paper (Appendix A.1) and satisfying Lindsey's definition of introspection.

Response to Weakness 3

Our task is not a simple reproduction of prior work. Beyond the improvements over prior work that you mentioned, and subject to the requirements of introspection (Appendix A.1), we also operationalize intervention detection and localization as evaluation tasks with explicit labels, thereby improving the objectivity and controllability of the evaluation. Initially, we also adopted the approach of Macar et al., i.e., using an LLM judge to perform binary classification on free-form generated text. We found that this approach suffers from several problems.

  • Specifically, free-form generation tends to produce outputs that do not fully comply with the instructions. For example, when generating subsequent tokens, the model may "externally infer" that it has been intervened upon by reading the anomalous text it has already generated, making it difficult to assess whether it truly detects its own injection.

  • Moreover, LLM judges introduce a certain degree of error and misclassification, which compromises the accuracy of the experimental evaluation and, in turn, the subsequent analysis of the mechanisms. Therefore, we adopted the task setting used in the current paper. This safeguards the rigor of the subsequent mechanism studies and avoids the problem of biased evaluation.

It is true that the idea that "two components underpin introspection" is not new, but we believe this does not imply a lack of novelty. Well-known works in the interpretability field have likewise understood LLMs in this way: Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small obtained a three-component result on the IOI task, and Locating and Editing Factual Associations in GPT obtained a two-component result for knowledge editing. Moreover, although Macar et al.'s work looks similar to ours at first glance, our two-stage analysis differs from Macar et al.'s use of external interpretability tools.

  • Our focus on attention is motivated by the information-transfer process required to report internal perturbations: the model outputs the final answer and the intervention at different context positions, and in a Transformer model only attention can transfer such information across positions. Therefore, directly studying attention for causal attribution directly reflects the model's own computational process, and it is the direct component that transfers the perturbation information to the final answer position. More importantly, the MLP features at the final answer position only emerge after the information has been transferred through attention. Macar et al., by contrast, primarily use a transcoder to analyze MLP features at the final answer position, with less emphasis on how information moves across token positions.
  • Furthermore, we use a straight-through estimator to study component-level joint optimization, resolving the limitation of per-head attribution that ignores synergistic effects, whereas Macar et al. select features based on attribution scores and overlook the joint synergistic effects across different layers.

Therefore, our method and analysis approach are not entirely identical.

Dear Reviewer FLQk

We are grateful for the time you spent on our paper and for the helpful comments in your review.

We have posted a reply that goes through each of your points. Since the author response period is drawing to a close, we would appreciate it if you could look it over when you have a moment.

If any part of the reply remains unclear, or you wish us to clarify or check anything further, please tell us. We will do what we can in the time that is left.

Thank you for your consideration.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.