- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how language models combine the three in-context retrieval mechanisms: positional, lexical, and reflexive. Using the 20-fact boxes task on Gemma 2-2B, the authors first identify L18H6 as the primary retrieval head via direct logit attribution and causal restoration, showing that its effect is almost entirely explained by where it attends among candidates. Then they estimate low-rank subspaces of the final-token residual stream at layer 18 for each mechanism using counterfactual pair differences, and show that these subspaces jointly reproduce the effect of a full counterfactual patch and can be edited individually to redirect the answer. The authors decompose L18H6's attention logits into contributions from a task-general component and each mechanism subspace, showing that the mechanisms cast additive "votes" resolved by the softmax. Finally, they characterize the subspaces, finding that the positional subspace encodes fact order while the lexical and reflexive subspaces share a semantically compressed entity channel.
Critical strengths
- The central claims are supported by several complementary methods: DLA, whole-head versus attention-only causal restoration, sufficiency tests with a reverse control, targeted single-subspace edits, and a non-interventional attention decomposition on unpatched prompts.
- Appendix E convincingly shows that the uncentered contrastive covariance estimator of Lee et al. (2026) absorbs the task-general query component into the order-ID subspace, which motivates separating out g in the residual stream.
- The authors present replication on a second model family and released code.
Critical weaknesses
W1. Additivity holds by construction
- Mechanisms come from Gur-Arieh et al. (2025); QK-logit decomposition follows Lee et al. (2026).
- Since the query is linear in the residual stream, additive logit contributions are guaranteed by construction. The real contribution is that each component targets a mechanism-specific candidate and is causally separable, and the framing should say so.
W2. Narrow scope
- Results rest on a single synthetic task (20 facts, two entity types) and two models.
- Llama already shows a less concentrated circuit (Appendix A), so the "single primary head" picture may not generalize.
W3. Separability vs. shared content channel
- Section 4.1 claims independent manipulability, but Section 6.2 reports strong lexical/reflexive overlap (cosine 0.83). I think this needs a bit of a explanation.
W4. Unexplained portions of the effect
- The "Other / mixed" outcomes in Fig. 9 (~23–31%) are not analyzed.
- Restoring L18H6 recovers only ~37–58% of the counterfactual margin.
- The 99.7% variance explained may be dominated by the format component (g). Variance among same-type candidates should be reported.
Questions for authors
- In a setting where both lexical and reflexive mechanisms contribute substantially, does editing one content subspace also move the other mechanism's vote in L18H6?
- What do the "Other / mixed" outcomes in Fig. 9 look like? Does the model still output a candidate token, and which components dominate L18H6's attention logits in these cases?
- What fraction of attention-logit variance is explained among same-type candidates only?
- Can the word-order effect be explained by whether the answer token's key can encode the identity of its paired entity?
2 comments
Dear Authors,
Thank you for your detailed experiments. I do think these results improve the paper. Since I think the initial score is high enough for the paper, I will maintain my score.
Good Luck!
We thank the reviewer for their detailed review and precise questions, and we’re glad they found our central claims supported by complementary methods.
W1:
Additivity holds by construction. We agree that additivity follows from construction and have reframed the contribution in the abstract and the introduction to be of the form: "We show that the same attention heads read all three subspaces, each directing attention to its mechanism's target." We also updated Section 5’s title to be “Retrieval Head Attention Decomposition” rather than “Additive Combination in QK Space.” Per-subspace decomposition of the attention logit is now just described as the method we use to show that the subspaces are directing attention, not the finding.
W2:
Single task with only two entity types
We have now added the music task (most of the other Gur Arieh et al. tasks are variations of this same format). The top heads are the same and the swaps work similarly. Because both the lexical and reflexive mechanisms are active, we were also able to examine that case. Added as Appendix K.
Circuit concentration
We’ve now extended our analysis to L19H1, L22H4 and L23H2 for Gemma and L24H24, L22H14, L27H7, and L25H7 for Llama and found that the pattern is consistent with L18H6. While this does imply a more distributed computation than a single primary retrieval head, we have softened that language in the manuscript, now stating that even if distributed beyond a single head, the core computation is the same: retrieval heads reading all three subspaces and softmax deciding over the contributions.
W3:
Explaining separability of lexical and reflexive mechanism despite the shared channel. We have now addressed this with the addition of the music task where both lexical and reflexive mechanisms are active. We show that they still can be separately manipulated even while having strong overlap.
These findings are compatible because our swap method subtracts one entity representation within the subspace and adds in another, it does not overwrite the entire subspace. The lexical mechanism uses the queried entities and the reflexive mechanism uses the answer entity, so the two mechanisms hold different entities in any one prompt. Swapping out one queried entity for another as a targeted intervention therefore does not disrupt the answer entity's representation even if it is occupying the same subspace. We have added this to Appendix K.
W4:
Unexplained portions:
Q1:
No, we are able to separately edit the two content mechanisms. In the newly added music task, at L18H6, swapping one content subspace moves its contribution from the original target to the swap target nearly entirely, for example: reflexive 1.90 → -0.09 at the original target and 0.06 → 2.06 at the swap target, while the other subspaces’ contributions change by at most 0.05. We showcase this in the newly added Appendix K and elaborate more in the reply to W3.
Q2:
For the "Other / mixed" outcomes in Fig. 9, does the model still output a candidate token? Yes, in 99.1%, see W4.
Which components dominate L18H6's attention logits in these cases?
In these cases, the token the model outputs generally has attention logit contributions from multiple mechanisms (positional 1.2-1.8, content 1.9-2.0). The positional signal within the head is diffuse and spread across nearby tokens (Figure 5a) and the content signal also leaks some attention onto semantic neighbors (Figure 5b).
Q3:
The fraction of attention-logit variance explained among same-type candidates only is 99.7%, the same as overall, now noted in Section 5.1. The task-general component g contributes 17-22%.
Q4:
Our results are consistent with this. Under causal attention: a token’s key can only carry the identity of entities that come before it in its fact. When the answer comes after the queried entity, the answer token’s key can carry the queried entity’s identity and the lexical subspace matches on it. When the answer comes before the queried entity in context, its key only carries its own identity and only the reflexive subspace can match on it. We have added this explanation to Appendix B.