- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper attempts to provide a mechanistic attribution of introspection, namely, the ability of an LLM to actively detect whether it has been affected by activation steering. The authors identify several attention heads, establish their causal influence on introspective outputs, and present some scattered observations about these attention heads.
Critical strengths
-
The paper selects a rather interesting task, namely, introspection.
-
The experiments are reasonably solid. However, this rigor cannot conceal the fatal shortcomings in scientific contribution that I discuss in the weaknesses.
Critical weaknesses
-
This paper is highly representative of what has been called "Slopterpretability" at COLM, namely:
- Carves off a narrow question: e.g., what is the mechanism behind refusal in multi-turn jailbreaks? (not a real submitted paper)
- Performs interpretability analysis using off-the-shelf tools, like circuit discovery or SAEs
- Reports the results and draws conclusions, sometimes with exaggeration
(https://gregdurrett.github.io/colm2026-blog/ai-papers.html)
-
My criticism of this paradigm is that it does not provide substantial insights or generalizable knowledge. We must be wary of the fact that research following this paradigm essentially uses one black box, the attention head (note that we still lack effective tools for understanding what an attention head actually computes), to explain another black box. It does not yield computational principles that humans can understand (considering that mechanistic interpretability aims to reduce a computational process to human-understandable pseudocode, merely chaining attention heads together is clearly not an ideal reduction), nor does it yield more generalizable knowledge. For example, just to give one possible direction, a genuinely valuable piece of generalizable knowledge might be: "Why do Transformers exhibit this kind of introspection in the first place? Presumably, such injection scenarios are entirely absent from the training data, so why does this capability emerge? What deeper properties of the training process does this reveal?"
-
This criticism manifests itself in many concrete ways. For example, and particularly in the "QKOV" analysis of the alignment between the singular vectors of attention-head parameter matrices and the perturbation directions, the reported phenomenon is simply an inevitable property of attention-head computation. Perturbations aligned with the singular vectors of attention-head parameter matrices will naturally have a greater effect on the computation, as a direct mathematical consequence. Experimentally verifying this merely confirms well-known principles of linear algebra and does not provide us with additional knowledge. This also explains what I personally mean by a "lack of novelty": the issue is not how new or old the investigated phenomenon is, but whether we provide new, systematic scientific knowledge rather than merely local or scattered observations of a phenomenon.
-
Beyond the novelty problem inherent in the aforementioned research paradigm, note that prior work has already used dictionary-learning tools to identify what the authors call "Gate Features," which determine whether an injected perturbation is present. Intuitively, if such features exist, there must also be a set of attention heads that produce them, because covariance along a particular direction in the hidden space cannot arise out of thin air. There must be some attention heads whose outputs (the so-called OV circuits) are aligned with that direction. Such features are also likely to be aligned with the inputs of certain attention heads (the so-called QK circuits); otherwise, a dangling feature that has no influence on the output would likely be discarded by normalization. Therefore, from an information-theoretic perspective, given the findings of prior work, the experiments in this paper can not further reduce our uncertainty about how LLMs operate or behave to an even lesser extent.
-
Although I do not believe this paper is entirely AI-generated, and it clearly contains some human refinement—which is already remarkably rare in this ICLR cycle—many unpleasant and difficult-to-read characteristics of AI-generated writing have still not been thoroughly eliminated. However, this point is not critical, but I offer a bonus for the "not entirely AI-generated", as I believe Weakness 1 alone already constitutes sufficient grounds for rejecting this paper.
Questions for authors
See weaknesses.
6 comments
response to Weakness 2: feature or head comes first
We would like to first emphasize a key point: at the final token position, attention transfers the perturbation information first, and only afterwards does the MLP read out the resulting feature. Macar et al. call their analysis results "Gate Features," whereas we call ours "gate heads"; we acknowledge the naming similarity and fully agree that "if such features exist, there must also be a set of attention heads that produce them." However, Macar et al. examine MLP features at the final prompt position, whereas in a Transformer only attention can transfer information across positions. The information at the final position must therefore arrive via attention in earlier layers, meaning the MLP features they analyze with a transcoder are merely a downstream readout of this attention-transferred information. Therefore, this causally-validated, rule-like circuit of gate and router heads provides a possible avenue for humans to understand how the model detects and reports intervention.
Furthermore, we use a straight-through estimator to study component-level joint optimization, resolving the limitation of per-head attribution that ignores synergistic effects, whereas Macar et al. select features based on attribution scores and overlook the joint synergistic effects across different layers.
response to Weakness 3: writing We acknowledge that our writing is not sufficiently clear in this regard, and we will revise the content and presentation of the paper in the subsequent version.
Thanks for your reply (with sincere human language)!
-
I am not fully convinced by your defense of the novelty of this work. We cannot simply argue that because similar approaches have been used in the past, they are therefore immune to criticism. Moreover, the works on discovery heads or steering vectors that you cited are already several years old, which is quite outdated for such a rapidly evolving field (this is minor). Attempts to extract certain heads, circuits, or intermediate computational results are essentially using one black box to explain another. They do not really explain how the computation works.
-
The same concern applies to your defense of the computational mechanism. I understand that this weakness may put you in a difficult position and force you to speculate about a mechanism, such processing may have a negative psychological impact. However, please note that one of the key components of your proposed mechanism is the "gate". The current experiments (I think) do not explain where this gate comes from. It remains an oracle produced by a black box. Therefore, what you have presented is not really a complete computational mechanism, but rather the final stage of one.
-
I will respond to your rebuttal of my third point a little later.
-
Please note that ICLR allows authors to update their PDFs during the rebuttal period. Therefore, rather than merely promising to revise the paper, you should actually make those revisions. Otherwise, it will be difficult for me to reconsider my evaluation.
We would like to respectfully clarify that our task is not simply Slopterpretability. Operating at the fine-grained attention-head level does not imply a lack of novelty; well-known works in the interpretability field have likewise understood LLMs in this way.
We define an objective, controllable task that satisfies the requirements of introspection, and provide a detailed analysis of the entire mechanism by which an LLM reads whether it has been intervened upon and outputs the final position of the intervention. Therefore, using specific attention heads to explain certain functions also provides a possible avenue for humans to understand LLMs.
Furthermore, we test our method across three model families and find it effective on all of them. In the interpretability literature, validating a general framework simultaneously across different model families is far from trivial, which indicates that our analysis approach has strong generalization and universality.
We thank the reviewer for raising questions about our detection process, which we did not express very clearly in the paper. We analyze the attention patterns of the router head under gate on/off conditions. When the gate is off, the router head's attention to the positions it previously attended to sharply decreases, while its attention to other positions increases uniformly. This illustrates the circuit relationship between the two components: the gate must be on for the router to take effect. This reflects the overall circuit process. Specifically, we can summarize the mechanism as:
This simplified form clarifies how we understand the mechanism, and therefore we have not failed to address the actual detection process.
We thank the reviewer for this deep question, which we fully agree is important. Understanding why such self-monitoring emerges despite its absence from pre-training data would indeed reveal something fundamental about the training process; to our knowledge, this remains an open problem across the field. Our work addresses the more proximal how reducing the mechanism to causally-validated gate/router heads which we see as a necessary first step toward that deeper question. We are actively pursuing this direction as future work, and thank the reviewer for pushing us to state it explicitly.
We perform SVD on the matrices ΔK and WO(ΔV)⊤ that induce the activation change, which is a routine operation in linear algebra. However, we would like to emphasize that removing the top five OV singular modes has an effect close to removing the full value-change term, providing a concise functional explanation supported by causal intervention, whereas removing the remaining modes has a smaller impact on localization accuracy (Appendix C.2). This reveals a low-rank, causally interpretable mechanism through which introspective behavior can be directly and efficiently manipulated, rather than requiring diffuse modification of the entire value-change term. Moreover, the difference in alignment between the high-accuracy and low-accuracy concept groups is larger than the difference in mode magnitude. These results are not directly guaranteed by SVD identities.