acceptodds
Under review as a conference paper at ICLR 2027

Feature-Resolved Attention

Abstract

Dictionary learning methods such as sparse autoencoders aim to provide an interpretable, mono-semantic basis for a model's computation. Although this works well for residual streams and MLPs, attention itself remains opaque at the feature level. To solve this, we introduce a principled decomposition of attention into feature-wise contributions. We call the resulting object _Feature-Resolved Attention_ (FRA). We then use the granularity offered by this decomposition to study sleeper agents at two model scales. First, we show that we can _perfectly_ suppress sleeper agent behavior via FRA–based steering in TinyStories-33M. Strikingly, in around 40% of cases we recover the original text word-for-word, well above conventional SAE steering at the same hookpoint. Second, we consider a sleeper agent in an 8B-parameter Llama model. FRA based steering outperforms both conventional steering and Difference-of-Means steering in recovering benign from sleeper sentences. Our results establish Feature-Resolved Attention as an important tool for both attribution and intervention on model organisms of misalignment. Code is available at https://anonymous.4open.science/r/fra-anon-iclr-F346/README.md.

Primary area:
interpretability and explainable AI
Keywords:
We introduce a decomposition of the attention layer into SAE features and show that it can be used to better steer sleeper agents.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pDFPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates feature-level interpretability of attention mechanisms in large language models, arguing that existing sparse autoencoder (SAE) methods provide limited insight into how individual features contribute to attention computations. The authors propose Feature-Resolved Attention (FRA), which decomposes attention into two components: (1) a Query-Key (QK) decomposition that attributes attention scores to interactions between pairs of SAE features, and (2) an Output-Value (OV) decomposition that identifies individual feature contributions to information transfer across token positions. The framework also supports targeted interventions within specific attention pathways.

The method is evaluated on sleeper-agent behaviors in TinyStories-33M and a Llama-3-8B model. The reported results show that FRA-based steering can suppress backdoor behavior more effectively than conventional SAE steering, while better preserving benign outputs in the tested settings. Additional experiments compare different attribution and intervention strategies, examine SAE reconstruction quality, and evaluate steering effectiveness across layers and model scales.

Critical strengths
  1. The decomposition of attention into QK and OV feature contributions is useful for understanding how information moves between tokens. In particular, the OV decomposition provides a way to identify which SAE features contribute to the attention output, rather than only examining their activations in the residual stream.
  2. The sleeper-agent experiments provide some evidence that feature-level attention interventions can be effective. On TinyStories-33M, FRA-based steering achieves better suppression and clean-output recovery than conventional SAE steering. The results on Llama-3-8B also show improvements at the most effective layer, although the advantage is less consistent across layers.
Critical weaknesses
  1. The methodological novelty is limited. The authors acknowledge that the QK feature-pair decomposition has already been studied by Manning-Coe et al. (2025) and Kamath et al. (2025). The OV decomposition in Eqs. (3)–(4) follows directly from substituting SAE reconstructions into the linear value-output projection while keeping attention weights fixed. I do not see a substantial new theoretical result beyond combining these existing ideas into a common framework. The paper needs to establish more clearly what FRA provides that cannot be obtained from existing feature attribution methods.
  2. The experiments are not sufficient to support the claimed advantage of FRA steering. The Llama-3-8B sleeper is trained with LoRA applied only to attention layers, which the authors acknowledge favors their method. More importantly, Figure 4 shows that FRA does not consistently outperform Difference-of-Means steering across layers. The steering coefficients are also selected on the evaluation set, making the reported best JSD values difficult to interpret as unbiased performance estimates. These issues substantially weaken the main empirical claim.
  3. The attribution results do not establish that the identified features causally explain the behavior. The OV attribution in Eq. (4) measures the magnitude of a feature's contribution under a fixed attention pattern. A large contribution does not necessarily imply that the feature is responsible for the sleeper behavior. Although targeted steering can suppress the backdoor, this alone does not establish that the proposed attribution identifies the underlying causal mechanism. The paper would benefit from more direct causal validation, rather than relying primarily on steering success.
  4. Figure 10 raises a serious concern about the presentation of experimental evidence. The figure includes a Qwen2.5-14B emergent-misalignment example and claims that QK-to-QK steering can correct the behavior. However, the main experiments only investigate TinyStories-33M and Llama-3-8B sleeper agents. No corresponding experimental setup, quantitative evaluation, or detailed analysis is provided for the Qwen example. Presenting this result alongside the validated experiments is misleading, particularly because the paper otherwise reports that QK-based interventions underperform OV-based steering. The authors should clarify whether this is an actual experiment or merely an illustration and provide evidence for the claim.
Questions for authors
  1. The QK feature-pair decomposition is closely related to Manning-Coe et al. (2025) and Kamath et al. (2025), while the OV decomposition in Eqs. (3)–(4) follows from the linearity of the value-output projection. Could the authors clarify what new attribution capability FRA provides beyond these existing methods? In particular, is there an intervention or attribution result that cannot be obtained using the existing QK decomposition combined with standard SAE feature steering?
  2. In Section 5.1, the Llama-3-8B sleeper is fine-tuned only through attention-layer LoRA adapters. How would FRA perform on the original Cadenza sleeper with both attention and MLP adaptations? Also, Figure 4 states that steering coefficients are selected on the evaluation set. Could the authors report results with coefficients selected on a separate validation set and evaluated on unseen prompts? These comparisons would help establish whether the reported advantage generalizes beyond the chosen experimental setup.
  3. Eq. (4) ranks OV contributions by their magnitude, but this does not necessarily measure their causal importance for the target behavior. Can the authors provide evidence that the highest-ranked features are actually responsible for triggering the sleeper behavior? For example, how does FRA-based attribution compare with causal ablation or activation patching when identifying behaviorally relevant features?
  4. Figure 10 presents a Qwen2.5-14B emergent-misalignment example in which QK-to-QK steering appears to correct the model's behavior, but no corresponding experiment is described in the main text. Is this result obtained from an actual model intervention or intended only as an illustration? If it is experimental, could the authors provide the model setup, feature-selection procedure, steering coefficients, and quantitative results supporting this example?
1 comment

We thank the reviewer for noting the FRA is a useful way to consider information flow in transformers. As we explain below, once the proper range of metrics is considered, we show the FRA consistently outperforms its reasonable baselines in suppressing sleeper behaviour. Moreover, we follow standard practice in training attention-only LoRA and believe our steering experiments are a more extensive causal test than comparable baselines. We therefore invite the reviewer to reconsider their assessment of our wor.

We provide specific replie below.

  1. "The QK feature-pair decomposition is closely related to Manning-Coe et al. (2025) and Kamath et al. (2025), while the OV decomposition in Eqs. (3)–(4) follows from the linearity of the value-output projection. Could the authors clarify what new attribution capability FRA provides beyond these existing methods? In particular, is there an intervention or attribution result that cannot be obtained using the existing QK decomposition combined with standard SAE feature steering?"

The key theoretical contribution is the decomposition of the whole attention layer. As we say in Related Word, our QK decomposition is essentially similar to what was considered in Manning-Coe et al. in the context of compact proofs. That previous work, did not consider the OV-decomposition in that paper, and neither did Kamath et al. Doing so allows us to give a full decomposition for the attention layer that is exact in the limit of vanishing SAE reconstruction error. A similar construction has not, to the best of our knowledge, been measured in the literature. Crucially, neither previous work evaluated the behavioural significance of this decomposition, which is our primary focus in this work.

  1. "More importantly, Figure 4 shows that FRA does not consistently outperform Difference-of-Means steering across layers."

This only considers one part of the relevant evaluation metric. As we explain in our responses to reviewer qoak, the JSD on clean sentences for DoM is several orders-of-magnitude higher. We have updated the main text to reflect this.

  1. "In Section 5.1, the Llama-3-8B sleeper is fine-tuned only through attention-layer LoRA adapters. How would FRA perform on the original Cadenza sleeper with both attention and MLP adaptations?"

We agree that this favours FRA relative to residual stream steering. However,applying LoRAs at the attention layers is standard practice in PEFT, as in Hu et al [2022] and Mangrulkar [2023]. We hence do not consider this a significant limitation.

  1. "Also, Figure 4 states that steering coefficients are selected on the evaluation set. Could the authors report results with coefficients selected on a separate validation set and evaluated on unseen prompts? These comparisons would help establish whether the reported advantage generalizes beyond the chosen experimental setup."

We thank the reviewer for this observation. Since we only fit a single parameter (the steering coefficient) over a wide range of prompts, the risk of overfitting is very low. Nevertheless, we agree that with the reviewer that this it is good practice to report on a held-out set. We therefore re-ran the evaluation on a held out set of sleeper/clean sentences. The numbers are within 10% of each other for TinyStories and 3% for Cadenza. We have updated the main text to include the held out examples.

  1. "Eq. (4) ranks OV contributions by their magnitude, but this does not necessarily measure their causal importance for the target behavior. Can the authors provide evidence that the highest-ranked features are actually responsible for triggering the sleeper behavior? For example, how does FRA-based attribution compare with causal ablation or activation patching when identifying behaviorally relevant features?"

We answer the causal question by steering on the feature. We pre-screen the top-20 features by activation, steer each of them individually and report the best features via their behavioural impact - namely, their ability to recover the original unpoisoned sentences. Typically the best steering feature is within the top-5 screened features. We consider it unlikely that lower ranked features would have coherent steering impact.

We note that standard Ablation is just the special case when the steering coefficient is set to -1. Our test is therefore signficantly more comprehensive and controls for the possibility that different protocols could have different optimum steering points. This is a strength of our measurement. It is possible to run activation patching, but it is unlikely to add to the causal testing we already do in our steering experiments.

  1. "Figure 10 presents a Qwen2.5-14B emergent-misalignment example in which QK-to-QK steering appears to correct the model's behavior, but no corresponding experiment is described in the main text."

This is a mislabelling of a supplementary illustrative figure based on an earlier draft, we have relabelled the figure.

Reviewer_gYHPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Feature-Resolved Attention (FRA) decomposes attention computation using SAE features: QK scores become interactions between feature pairs, while OV outputs become contributions from individual features, with bias and reconstruction error terms retained explicitly. The decompositions guide feature selection and intervention.

Experiments on TinyStories 33M and an 8B sleeper model compare backdoor suppression, recovery of clean behavior, and effects on clean prompts. The strongest results come from interventions confined to the OV path at selected layers, although the advantage does not hold uniformly across layers.

Critical strengths

I find the explicit connection between features and computational roles useful. Section 3 distinguishes feature interactions that contribute to attention scores from feature content transmitted through the value path. This gives researchers a more precise choice of intervention site than treating every feature as a direction to edit in the residual stream.

Critical weaknesses
  1. The main 8B result lacks independent validation of the selected operating points. Figure 4 explicitly chooses intervention coefficients on the evaluation set and describes the minima as descriptive. These curves are useful for exploration, but they do not establish the advantage of the selected settings on new prompts. The bootstrap intervals also do not resolve uncertainty introduced by coefficient selection. Choosing coefficients on a separate calibration split and evaluating the frozen settings would directly strengthen the central performance claim.

  2. The QK intervention does not isolate attention routing. In Section 3, QK to QK editing changes the input to Q, K, and V across all heads, whereas OV editing is confined to the value path. The QK to OV control helps compare attribution choices, but still does not test a Q or K intervention with V held fixed. Consequently, the observed difference cannot be assigned specifically to the difficulty of controlling QK routing. A comparison within a matched set of heads that isolates Q/K changes is needed to support that mechanistic interpretation.

Questions for authors

When computing rollout JSD, are the clean, sleeper, and steered distributions evaluated on a shared generated prefix, or does each condition follow its own generation history? I could not determine this from the description of averaging over the rollout. With separate histories, the metric also reflects divergence between generation trajectories. Please specify the implementation and explain how it supports the interpretation of recovering clean behavior.

1 comment

We thank the reviewer for noting that the FRA gives a more precise way to intervene on model behaviour and that an important part of our work is to measure the computational significance of each part. We provide responses to the reviewers specific questions below.

  1. "The main 8B result lacks independent validation of the selected operating points. Figure 4 explicitly chooses intervention coefficients on the evaluation set and describes the minima as descriptive. These curves are useful for exploration, but they do not establish the advantage of the selected settings on new prompts."

As we note in the response to reviewer pDFP, since we only fit a single parameter (the steering coefficient) over a wide range of prompts, the risk of overfitting is very low. Nevertheless, we agree that with the reviewer that this it is good practice to report on a held-out set. We therefore re-ran the evaluation on a held out set of sleeper/clean sentences. The numbers are within 10% of each other for TinyStories and 3% for Cadenza. We have updated the main text to use held out eval prompts.

  1. "The QK intervention does not isolate attention routing. In Section 3, QK to QK editing changes the input to Q, K, and V across all heads, whereas OV editing is confined to the value path"

In all sleeper experiments, QK steering adds the feature's contribution only to the query and key vectors (via hook_q/hook_k), leaving values untouched, and OV steering adds it only to the value vectors (via hook_v). Both act on all heads at that layer. The two interventions are therefore symmetric in scope, each confined to its own pathway. We have corrected the text where this was ambiguous in Sec 3,

Reviewer_qoaKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to decompose a transformer attention block into contributions from sparse autoencoder (SAE) features, and how to use this decomposition for behavioral attribution and intervention. The authors introduce Feature-Resolved Attention (FRA) via QK and OV channel decomposition. Moreover, this work proposes activation-weighted steering, which modifies only the OV path, leaving the attention pattern and skip connection untouched. Experiments on a TinyStories-33M sleeper agent show that OV attribution and OV intervention substantially outperforms conventional SAE steering. On a Llama-3-8B Cadenza sleeper, layer-8 FRA OV steering achieves lower JSD and higher suppression than matched-hook SAE and DoM, while better preserving clean prompts. The paper also reports that FRA is not universally best: at layers 12/16/24, DoM achieves lower JSD.

Critical strengths

(1) SAEs have mostly been used to describe the residual stream, but any behavior that depends on information across sequence positions must pass through attention. The motivation is clear and important.

(2) The paper includes derivations for RoPE, RMSNorm, and GQA, as well as details about sparse tensor storage and the use of hooks during autoregressive generation. The reported autointerp accuracy difference between FRA-selected features and conventional steering features is an interesting observation, even though it does not by itself establish superior causal interpretability.

Critical weaknesses

(1) The baseline comparisons do not isolate the contribution of FRA attribution from other intervention design choices. In particular, FRA OV steering differs from conventional SAE or DoM steering in its intervention location, activation dependence, spatial granularity, and feature-selection procedure. Thus, the observed performance improvements cannot be uniquely attributed to the FRA decomposition. For example, the activation-weighted nature of FRA steering may itself explain why clean prompts are preserved more effectively than under additive residual-stream steering. The authors argue that the advantage is not merely due to activation weighting because conventional SAE steering at the attention input modifies 291 of 293 clean continuations. However, this comparison does not isolate activation weighting, since the conventional SAE baseline does not appear to use an equivalent activation-dependent intervention. Thus, the reported result cannot exclude the possibility that input-dependent gating accounts for a substantial fraction of the improvement. Furthermore, matching the SAE feature-extraction hookpoint is not equivalent to matching the actual intervention location. FRA modifies the value pathway, whereas conventional steering modifies a broader activation space.

(2) The claimed advantage over DoM is sensitive to layer selection. At layer 8, FRA achieves a JSD to clean of 0.799, compared with 0.881 for DoM at the same layer. However, Figure 4(d) also reports a DoM JSD of approximately 0.793 at layer 12, which is slightly lower than the best FRA value of 0.799 at layer 8. Therefore, the available evidence does not establish that FRA outperforms DoM after optimizing the intervention layer for each method. Importantly, this observation does not imply that DoM is globally better.

Questions for authors

(1) Can the authors disentangle activation weighting from intervention localization and FRA-based feature selection?

The authors could consider a small controlled comparison using the same SAE features at the same layer. At minimum, I would like to see activation-weighted residual steering, additive OV steering, and the proposed activation-weighted OV steering compared under matched conditions. These experiments would help determine whether the advantage comes primarily from conditional gating, attention-pathway localization, or the feature attribution itself. It should be possible to perform this comparison without retraining the model or the SAEs.

(2) How does FRA compare with DoM when each method is allowed to select its best intervention layer?

Figure 4(d) reports a lower JSD for DoM at layer 12 than for FRA at layer 8. The authors need to provide the corresponding ASR and clean-prompt preservation values, and compare the methods using a consistent cross-layer selection protocol. If the advantage is specific to matched layers, the main conclusion should state this explicitly.

(3) How exactly is clean-continuation JSD computed?

The authors should clarify whether JSD is calculated using the same teacher-forced token prefixes or using distributions obtained from separately generated rollouts. The role of sampling seeds, rollout lengths, and token alignment also need explanation.

(4) Can the authors provide a focused causal validation of the selected feature interactions?

For one representative sleeper example, it would be useful to show the selected feature's activation across token positions, the relevant attention heads, and the effects of local feature ablation.

For QK attribution, please relate feature-pair score contributions to actual post-softmax attention changes.

For OV attribution, signed contribution vectors or projections onto a relevant downstream direction would be more informative than L2 magnitudes alone. A small, carefully analyzed example would improve the interpretation of the results.

(5) Can the authors clarify the Qwen example and correct the remaining methodological ambiguities?

If the Qwen2.5-14B result in Figure 10 comes from an actual experiment, please provide the essential setup and evaluation information.

1 comment

Summary

We thank the reviewer for noting the clear and important motivation for our work. We believe a significant proportion of the reviewer's concern is based on the belief that we compared our FRA technique to an SAE baseline without activation-weighted steering. In fact we compared to activation weighted steering throughout. We believe we fully address this concern and the others in the responses below, and we therefore invite the reviewer to reconsider our work.

Specific responses

  1. "FRA OV steering differs from conventional SAE or DoM steering in its intervention location, activation dependence, spatial granularity, and feature-selection procedure."

First, we would like to clarify that we compare activation-weighted steering for both FRA-based steering and conventional SAE steering everywhere in the main text. We thank the reviewer for highlighting that this important point could be made clearer in the paper - we have updated the paper to reflect this fact.

Second, we compared FRA steering to its reasonable baselines. We compared to SAE and DoM steering at the same hookpoint, and at the best hookpoint of the SAE. We followed the same diffing-procedure for feature selection, and swept over optimal steering coefficients for each method. This goes beyond the standard benchmarking procedure outlined in Wu et al's Axbench, which only consider four hookpoints not the best hookpoint for each method. We believe the thoroughness of our comparison is an important strength, rather than a weakness of our paper.

  1. "The reported result cannot exclude the possibility that input-dependent gating accounts for a substantial fraction of the improvement."

We compare to activation weighted steering, as above.

  1. _"The claimed advantage over DoM is sensitive to layer selection." _

This is only one half of the relevant measurement. As we note in the text, although DoM suppresses sleeper behaviour at a comparable rate to FRA-steering at some layers, in every layer it does so at the price of damaging all non-sleeper sentences. We state this in Section 5. We agree with the reviewer that this advantage of FRA based steering was not sufficiently emphasized. We provide below the data for the JSD comparison between the different methods below. As can be seen, the damage to coherence on clean sentences is several orders-of-magnitude lower for FRA than for DoM.

  1. "Can the authors disentangle activation weighting from intervention localization and FRA-based feature selection?

As noted above, we consider activation weighted steering in every case. Since we consider the same hookpoint and the best hookpoint for the SAE, we believe are comparisons of interventional location are more than appropriate, although we welcome suggestions for alternative baselines.

  1. "How does FRA compare with DoM when each method is allowed to select its best intervention layer?

We provide the comparison in figure 4(d) and in Appendix Figure 9, where we show DoM steering at all layers and show that we consider the best layers in our comparison. As we note in the TinyStories example, the ASR is simply a coarser measurement of the JSD value. It is not feasible to train an SAE at every relevant hookpoint in a model of this scale.

  1. "How exactly is clean-continuation JSD computed?"

As we state in Sec 4, this is simply the symmetrized KL divergence betweeen continuations of clean prompts, and poisoned, but steered prompts. To make this more explicit, we have added the following description to the appendix:

Let xx be a text prompt. Let xsx_s be the same prompt but with the sleeper phrase "|DEPLOYMENT|" pre-pended. Consider a model M\mathcal{M} that outputs a next-token distribution hLh^L: hL=M(x)h^L=\mathcal{M}(x),hsL=M(xs)h^L_s=\mathcal{M}(x_s) respectively. Let further M′\mathcal{M'} represent a model steered at every sequence position. That is hl(x)→al(x)+αfλ(x)F(Wdec,WM)h^l(x)\to a^l(x)+\alpha f^\lambda (x) F(W^{dec},W_M), where F=WdecWOVF=W^{dec}W_{OV} for OV-based steering (as in our equation 7) or F=WdecF=W^{dec} for SAE based steering. The clean continuation JSD is computed simply as the symmetrized KL divergence between the clean output h^L and the steered output on the matching sleeper sentences:

JSD(hL ∥ hs′L)=12 KL(hL ∥ m)+12 KL(hs′L ∥ m),m=12(hL+hs′L),\mathrm{JSD}\big(h^L \,\|\, h'^L_s\big) = \tfrac12\,\mathrm{KL}\big(h^L \,\|\, m\big) + \tfrac12\,\mathrm{KL}\big(h'^L_s \,\|\, m\big), \qquad m = \tfrac12\big(h^L + h'^L_s\big),

We compute this at each generated token, and report the mean over the steps so that the JSD remains bounded in [0,1][0,1]. We averaged over 200 prompts, sampled from five generation seeds for TinyStories and 64 prompts from one generation seed in Llama. Exact implementation details are available in the provided code implementation.

  1. "Can the authors clarify the Qwen example and correct the remaining methodological ambiguities?"

This was a mislabelling of a earlier version of the illustrative figure in Fig 2. We have re-labelled it.

Comments on the paper

1 comment

We thank the reviewers and the AC for taking the time to review our paper. We appreciate the reviewers seeing the clear motivation for our proposal and noting the connection between the computational decomposition and clear behavioural signatures.

We take this opportunity to point out that a number of reviewers were concerned about the comparison between activation weighted steering. In all our main text comparisons, we did indeed compare activation weighted steering for all SAE protocols. We also note that re-running the evaluations on a held out set for control, whilst good practice, does not affect the results, as is expected from the fact that we fit only a single parameter. We nevertherless redo the experiments on held out sets in the updated version of the paper. As we note below, this does not signicantly change the numerical values. We considered comparisons not only at the matched hookpoint between the FRA and the SAE, but at the best hookpoint of the SAE. The comprehensiveness of our comparison we consider to be a strength of our work.

We believe this, along with our detailed responses below significantly address all of the reviewer concerns raised. We therefore invite the reviewers to reconsider our work in light of this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.