acceptodds
Under review as a conference paper at ICLR 2027

The Role of Attention Layers and Context Tokens in Entity Copying

Abstract

Large language models (LLMs) perform reliably on the task of entity copying, in which the model copies the tokens referring to an entity, or entity tokens, from the prompt into its outputs to answer a question. Although entity copying is straightforward for most LLMs, existing research does not provide a systematic account of which layers specialize in this fundamental task or how other tokens in the same sequence, termed context tokens, influence the model's ability to copy the entity tokens. To address these questions, we conduct experiments on Qwen3-8B using two novel methodologies: genie-in-a-bottle, which controls exactly which layers can participate in an entity copying task, and attention lobotomy, which cuts off specific tokens' attention to entity tokens without affecting the remaining attention distribution. We find that layers in the second half of the model are both necessary and sufficient for entity copying, whereas those in the first half play little to no role. Moreover, context tokens' joint attention to entity tokens plays a decisive role in successful entity copying, even though these context tokens do not store entity information in their intermediate states unless they satisfy particular semantic properties. Our findings establish the critical role of late layers in entity copying under the decisive guidance of context tokens, calling for future work on how entity information is propagated and consumed in models.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, large language models, transformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C3LiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the problem of entity copying in the Qwen3-8B language model. It shows that the top half of the layers are responsible for entity copying and that context tokens, both when they are semantically related or unrelated to the entity in question, affect copying through their attention weights. To this end, the paper develops two techniques, called genie-in-a-bottle and attention lobotomy, that control causal factors in the computation of attention, and test them across five prompt templates.

Critical strengths
  1. Careful causal procedures for identifying the sets of layers where entity copying takes place and the effect of attention from context tokens.

  2. Results on the Qwen3-8B model showing that entity copying occurs at the top half of the layers and that context tokens guide the model to allocate more attention to the correct entity token.

Critical weaknesses
  1. Results are restricted to just one model. As such, it is unclear whether the conclusions hold in general for LLMs or reflect peculiarities of the Qwen3-8B model.

  2. Weak conclusions for layer analysis. The analysis indicates that entity copying occurs in the top half of the layers. Such localization is neither very specific nor surprising. Can entity copying be localized more precisely? Does the copying location vary with the specific entity, context, or prompt?

  3. The examples given in the text use well-known names, such as Einstein or Sheakspear, as entities. Copying such names may be affected by their frequent appearance in training. Did you also use generic names and even made-up names to control for such bias?

Questions for authors

See comments above. Overall, the layer localization is too crude and unsurprising. The analysis of the impact of context tokens is interesting, but conclusions drawn from a single language model are not sufficiently reliable.

5 comments
AuthorsOpenReview ↗

Generalizable Results on a Different Model

We appreciate the reviewer’s request for additional models to test the generalizability of our claims. We therefore attach our findings on Gemma-4-12B, which broadly supports the same conclusions while having intriguing differences. We will update our appendix with the findings on Gemma.

In summary, results on Gemma also validate the necessity for context tokens to attend to entity tokens jointly, as the removal of their attention causes huge degradation. Differently from Qwen, however, even the entity’s identity, as is measured by the recovery of nonexact names, is hard to extract for Gemma, compared to Qwen. In addition, Gemma also bolsters the role of description tokens in transferring entity information, as appending the correct, incorrect, and “a human” description show the same trend of improved results. Differently from Qwen, however, incorrect descriptions do not show a large discrepancy from correct descriptions, and can also meaningful recover entity identity in some cases.

SetupQwenGemma
No disabling, recovers identity500/500 (100%)494/500 (98.8%)
Context disabled: recovers exact names227/500 (45.4%)24/500 (4.8%)
Context disabled: recovers only identity219/500 (43.8%)50/500 (10.0%)
SetupQwenGemma
Name only9/500 (1.8%)80/500 (16.0%)
+ Correct description89/500 (17.8%)173/500 (34.6%)
+ Incorrect description13/500 (2.6%)108/500 (21.6%)
+ “a human”13/500 (2.6%)136/500 (27.2%)
AuthorsOpenReview ↗

Results regarding Layer Contributions are Genuinely Novel

While we understand the role of deeper layers in entity copying is not surprising for the reviewer, we would like to point out that, as subjective as “surprising” is, it is actually counterintuitive that entity copying happens late in the model layers. This is because entity copying simply extracts and transfers one token’s information to another — a basic functionality of any single attention layer. The fact that early attention layers are neither necessary nor sufficient is therefore highly informative on how the model divides the labor among its layers.

Additionally, per the reviewer’s request, we have sharpened our analysis of the contributions of layers. It suggests that, as opposed to the complete second half of layers, two disjoint groups of layers in the second half jointly contribute to entity copying, separated by a group of layers with minimal contribution. This is supported by a “valley” of reconstruction success in Figure 3 and a “rise” in Figure 4. We have updated the paper’s abstract, introduction, and Section 3 with those revisions.

AuthorsOpenReview ↗

Contribution of Prompts and Contexts Have Been Explicitly Illustrated

While we share the reviewer’s interest in understanding more about the impact of specific prompt templates and contexts in entity copying, we would like to point out that it is already demonstrated in Figure 3 and 4. In both figures, experiments on different templates support the same conclusions regarding the contribution of layer groups and context tokens, with only minor variations in success rates among them. Since the objective of the work is to study the contributions of layer groups and context tokens in entity copying, we believe the two figures have sufficiently illustrated what the reviewer asks for.

Reviewer_5JqMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to understand which layers and token positions are responsible for performing entity copying in Qwen3-8B. Two methods are used: a) a series of activation patching which confines information about the entity token to only certain layers (here called "genie-in-a-bottle") and b) zeroing out of context-token's attention to the entity token (here called "attention lobotomy").

The paper finds that the last third (er, half?) of Qwen3-8B's layers are 'necessary and sufficient' for entity copying.

Critical strengths

None

Critical weaknesses
  1. internal inconsistency. The biggest contradiction is key finding 2 (!) claims "We establish the necessary and sufficient role of the last third of layers" while Sections 3 and 7 say "We find that the second half [of layers] is both necessary and sufficient...".
Questions for authors
  1. [Clarification] Is "genie-in-a-bottle" meant to be a descriptive name? If so, please elaborate. (currently my understanding is that this is a series of activation patches meant to restrict access to the entity token: if so I believe a more descriptive name like "entity activation ablation" would help readers follow along better).
1 comment
Reviewer_5JqMOpenReview ↗

Authors have not responded

Just a reminder since the discussion period is nearly over: can the authors please address my concerns, notably regarding the internal inconsistency and how the "genie in a bottle" method relates to existing work? (For instance, is my understanding that it amounts to "entity activation ablation" correct?)

Reviewer_ThyyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies copying a name from a prompt with Qwen3-8B-Base. It introduces a two-boundary activation-patching procedure (“genie-in-a-bottle”) to restrict access to the entity to a window of layers and a post-softmax attention-edge ablation (“attention lobotomy”) to remove selected token-to-token value contributions. Across five prompt templates and 100 mostly well-known names, late attention windows support copying, and blocking context positions' attention to the name often changes exact-name outputs to other names for the same person. Two early-layer restorations recover much of the lost exact copying on one template. The authors also test whether the decoding position can recover the name from context positions alone and vary short descriptions appended after the names.

Critical strengths

The paper asks a precise mechanistic question and uses targeted interventions. Restricting entity access by layer and removing selected post-softmax attention contributions make the late-layer and context-position results more informative than attention visualization alone (Figs. 1, 3–7). It distinguishes literal copying from recovery of the person's identity. Reporting exact and alternate-name outputs makes a meaningful failure mode visible. Restoring context-to-entity attention in two layers provides a useful partial reversal of the ablation (Fig. 7a).

Critical weaknesses
  1. The layer and storage claims exceed what the interventions identify. The window experiment tests 8–12-layer spans in one model, after filtering out examples that already fail a context-attention ablation (Appendix A.3); it measures general success, which includes nonexact names, and does not show that the entire second half is jointly necessary and sufficient for literal copying. Figure 5 allows control-token attention outside the window at both earlier and later layers, so its improvement does not specifically identify early-layer attention as critical. In §5, failure to decode a name when direct access is blocked shows that the unmodified downstream model does not recover it from context states in this setup; it does not establish that those states contain no entity information. These distinctions directly affect the main mechanistic conclusion.
  2. The evaluation makes copying and memorized identity hard to separate. All 100 entities are recognizable public figures (Appendix A.2), and the examples of “nonexact” recovery (e.g. an alternate name) are compatible with parametric knowledge. The five templates are short and closely related; the main layer-window analysis uses four of them and only one-token names after filtering (pp. 4–5, Appendix A.3). Without unfamiliar or counterfactual names and an unfiltered accounting, it is difficult to know how broadly the reported context-token effect applies to copying literal strings.
  3. Only one model is tested.
Questions for authors
  1. Clarification that could change my assessment: Can you repeat the key context-edge and layer-window results with unfamiliar synthetic names or counterfactual familiar-name mappings, matched for tokenization, and report the unfiltered denominators and exact-copy rates? This would help distinguish copying from recovery from model priors.
  2. Clarification that could change my assessment: What evidence distinguishes “context states do not store entity information” from “the downstream model cannot read it under this ablation”? Could a matched causal patch or trained readout test recover entity identity from context states? For the profession-description result, how often does the model give the same name when the entity is replaced by another person with the same profession or hidden?
  3. Can you run more experiments on models from different vendors and with different sizes?
6 comments
AuthorsOpenReview ↗

Generalizable Results on a Different Model

We appreciate the reviewer’s request for additional models to test the generalizability of our claims. We therefore attach our findings on Gemma-4-12B, which broadly supports the same conclusions while having intriguing differences. We will update our appendix with the findings on Gemma.

In summary, results on Gemma also validate the necessity for context tokens to attend to entity tokens jointly, as the removal of their attention causes huge degradation. As opposed to Qwen, in Gemma, even the entity’s identity, as is measured by the recovery of nonexact names, is hard to extract. In addition, Gemma also bolsters the role of description tokens in transferring entity information, as appending the correct, incorrect, and “a human” description show the same trend of improved results. Differently from Qwen, however, incorrect descriptions do not show a large discrepancy from correct descriptions, and can also meaningful recover entity identity in some cases.

Name recovery: baseline and context disabled

SetupQwenGemma
No disabling, recovers identity500/500 (100%)494/500 (98.8%)
Context disabled: recovers exact names227/500 (45.4%)24/500 (4.8%)
Context disabled: recovers only identity219/500 (43.8%)50/500 (10.0%)
SetupQwenGemma
Name only9/500 (1.8%)80/500 (16.0%)
+ Correct description89/500 (17.8%)173/500 (34.6%)
+ Incorrect description13/500 (2.6%)108/500 (21.6%)
+ “a human”13/500 (2.6%)136/500 (27.2%)
AuthorsOpenReview ↗

Copying Mechanism is Clearly Distinguished from Memorization

While underspecified, we understand the reviewer’s definition of “memorization” to mean the ability to reconstruct the entity based on knowledge of the context, such as recovering “Jordan” from “basketball player”. Under this definition, we maintain that it is mechanistically implausible for the model to use memorization.

In the basic setup, where there is no descriptive phrase like “basketball player”, all four templates are reduced to four identical sequences excluding the target entity token(s). As a result, under greedy decoding adopted in this work, unless the model actively extracts information from the entity token itself, it will only be able to produce four fixed responses. But since the model does produce a variety of answers closely correlated to the entities, this is strong evidence that it does have a copying mechanism activated.

Moreover, in the setup where there is a descriptive phrase appended, we did a further ablation (table below), where we substituted each entity token with a uniform ‘x’, making extracting entity information impossible. If the model is activating memorized facts from context cues, its performance will be relatively unaffected. However, we find that the successful recovery rate has decreased to near 0. This again bolsters the conclusion that the model has an active copying mechanism rather than relying on memory.

With actual tokenWith x
Recovered name with description70/400 (17.5%)1/400 (0.25%)
AuthorsOpenReview ↗

Update to the Findings of Layer Contributions

We agree with the reviewer that claiming the entire second half of layers is necessary and sufficient for entity copying is too strong. Instead, as is supported a “valley” of reconstruction success in Figure 3 and a “rise” in Figure 4, two disjoint groups of layers in the second half jointly contribute, separated by a group of layers with minimal contribution. We have updated the paper’s abstract, introduction, and Section 3 with those revisions.

Comments on the paper

2 comments
Area_Chair_FWaeOpenReview ↗

Reminder: Reviewer Engagement During the Discussion Period

Dear Reviewer 5JqM and Reviewer C3Li,

As the discussion period is drawing to a close, could you please review the authors' rebuttal and provide your feedback at your earliest convenience?

Active engagement in discussions with the authors is an essential part of the peer-review process and plays an important role in ensuring the success of the conference. Your participation would be greatly appreciated.

Thank you for your time and contributions.

Best regards,

AC

At this time the authors have not engaged with my review. I have posted a comment reminding them about my primary concern and primary confusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.