- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates feature-level interpretability of attention mechanisms in large language models, arguing that existing sparse autoencoder (SAE) methods provide limited insight into how individual features contribute to attention computations. The authors propose Feature-Resolved Attention (FRA), which decomposes attention into two components: (1) a Query-Key (QK) decomposition that attributes attention scores to interactions between pairs of SAE features, and (2) an Output-Value (OV) decomposition that identifies individual feature contributions to information transfer across token positions. The framework also supports targeted interventions within specific attention pathways.
The method is evaluated on sleeper-agent behaviors in TinyStories-33M and a Llama-3-8B model. The reported results show that FRA-based steering can suppress backdoor behavior more effectively than conventional SAE steering, while better preserving benign outputs in the tested settings. Additional experiments compare different attribution and intervention strategies, examine SAE reconstruction quality, and evaluate steering effectiveness across layers and model scales.
Critical strengths
- The decomposition of attention into QK and OV feature contributions is useful for understanding how information moves between tokens. In particular, the OV decomposition provides a way to identify which SAE features contribute to the attention output, rather than only examining their activations in the residual stream.
- The sleeper-agent experiments provide some evidence that feature-level attention interventions can be effective. On TinyStories-33M, FRA-based steering achieves better suppression and clean-output recovery than conventional SAE steering. The results on Llama-3-8B also show improvements at the most effective layer, although the advantage is less consistent across layers.
Critical weaknesses
- The methodological novelty is limited. The authors acknowledge that the QK feature-pair decomposition has already been studied by Manning-Coe et al. (2025) and Kamath et al. (2025). The OV decomposition in Eqs. (3)–(4) follows directly from substituting SAE reconstructions into the linear value-output projection while keeping attention weights fixed. I do not see a substantial new theoretical result beyond combining these existing ideas into a common framework. The paper needs to establish more clearly what FRA provides that cannot be obtained from existing feature attribution methods.
- The experiments are not sufficient to support the claimed advantage of FRA steering. The Llama-3-8B sleeper is trained with LoRA applied only to attention layers, which the authors acknowledge favors their method. More importantly, Figure 4 shows that FRA does not consistently outperform Difference-of-Means steering across layers. The steering coefficients are also selected on the evaluation set, making the reported best JSD values difficult to interpret as unbiased performance estimates. These issues substantially weaken the main empirical claim.
- The attribution results do not establish that the identified features causally explain the behavior. The OV attribution in Eq. (4) measures the magnitude of a feature's contribution under a fixed attention pattern. A large contribution does not necessarily imply that the feature is responsible for the sleeper behavior. Although targeted steering can suppress the backdoor, this alone does not establish that the proposed attribution identifies the underlying causal mechanism. The paper would benefit from more direct causal validation, rather than relying primarily on steering success.
- Figure 10 raises a serious concern about the presentation of experimental evidence. The figure includes a Qwen2.5-14B emergent-misalignment example and claims that QK-to-QK steering can correct the behavior. However, the main experiments only investigate TinyStories-33M and Llama-3-8B sleeper agents. No corresponding experimental setup, quantitative evaluation, or detailed analysis is provided for the Qwen example. Presenting this result alongside the validated experiments is misleading, particularly because the paper otherwise reports that QK-based interventions underperform OV-based steering. The authors should clarify whether this is an actual experiment or merely an illustration and provide evidence for the claim.
Questions for authors
- The QK feature-pair decomposition is closely related to Manning-Coe et al. (2025) and Kamath et al. (2025), while the OV decomposition in Eqs. (3)–(4) follows from the linearity of the value-output projection. Could the authors clarify what new attribution capability FRA provides beyond these existing methods? In particular, is there an intervention or attribution result that cannot be obtained using the existing QK decomposition combined with standard SAE feature steering?
- In Section 5.1, the Llama-3-8B sleeper is fine-tuned only through attention-layer LoRA adapters. How would FRA perform on the original Cadenza sleeper with both attention and MLP adaptations? Also, Figure 4 states that steering coefficients are selected on the evaluation set. Could the authors report results with coefficients selected on a separate validation set and evaluated on unseen prompts? These comparisons would help establish whether the reported advantage generalizes beyond the chosen experimental setup.
- Eq. (4) ranks OV contributions by their magnitude, but this does not necessarily measure their causal importance for the target behavior. Can the authors provide evidence that the highest-ranked features are actually responsible for triggering the sleeper behavior? For example, how does FRA-based attribution compare with causal ablation or activation patching when identifying behaviorally relevant features?
- Figure 10 presents a Qwen2.5-14B emergent-misalignment example in which QK-to-QK steering appears to correct the model's behavior, but no corresponding experiment is described in the main text. Is this result obtained from an actual model intervention or intended only as an illustration? If it is experimental, could the authors provide the model setup, feature-selection procedure, steering coefficients, and quantitative results supporting this example?
We thank the reviewer for noting the FRA is a useful way to consider information flow in transformers. As we explain below, once the proper range of metrics is considered, we show the FRA consistently outperforms its reasonable baselines in suppressing sleeper behaviour. Moreover, we follow standard practice in training attention-only LoRA and believe our steering experiments are a more extensive causal test than comparable baselines. We therefore invite the reviewer to reconsider their assessment of our wor.
We provide specific replie below.
The key theoretical contribution is the decomposition of the whole attention layer. As we say in Related Word, our QK decomposition is essentially similar to what was considered in Manning-Coe et al. in the context of compact proofs. That previous work, did not consider the OV-decomposition in that paper, and neither did Kamath et al. Doing so allows us to give a full decomposition for the attention layer that is exact in the limit of vanishing SAE reconstruction error. A similar construction has not, to the best of our knowledge, been measured in the literature. Crucially, neither previous work evaluated the behavioural significance of this decomposition, which is our primary focus in this work.
This only considers one part of the relevant evaluation metric. As we explain in our responses to reviewer qoak, the JSD on clean sentences for DoM is several orders-of-magnitude higher. We have updated the main text to reflect this.
We agree that this favours FRA relative to residual stream steering. However,applying LoRAs at the attention layers is standard practice in PEFT, as in Hu et al [2022] and Mangrulkar [2023]. We hence do not consider this a significant limitation.
We thank the reviewer for this observation. Since we only fit a single parameter (the steering coefficient) over a wide range of prompts, the risk of overfitting is very low. Nevertheless, we agree that with the reviewer that this it is good practice to report on a held-out set. We therefore re-ran the evaluation on a held out set of sleeper/clean sentences. The numbers are within 10% of each other for TinyStories and 3% for Cadenza. We have updated the main text to include the held out examples.
We answer the causal question by steering on the feature. We pre-screen the top-20 features by activation, steer each of them individually and report the best features via their behavioural impact - namely, their ability to recover the original unpoisoned sentences. Typically the best steering feature is within the top-5 screened features. We consider it unlikely that lower ranked features would have coherent steering impact.
We note that standard Ablation is just the special case when the steering coefficient is set to -1. Our test is therefore signficantly more comprehensive and controls for the possibility that different protocols could have different optimum steering points. This is a strength of our measurement. It is possible to run activation patching, but it is unlikely to add to the causal testing we already do in our steering experiments.
This is a mislabelling of a supplementary illustrative figure based on an earlier draft, we have relabelled the figure.