acceptodds
Under review as a conference paper at ICLR 2027

How Do Language Models Route Information Through Repeated Tokens?

Abstract

Repeated words are common in natural-language inputs to Language Models (LMs). We assume they serve as the anchors where the contextual information is collected and stored, and thereby detected and forwarded to the later repetition, by a set of attention heads named Duplicate Token Heads (DTHs). Therefore, we investigate such attention heads and find that: (1) Not all DTHs necessarily route information that is causally relevant to a given task since only a subset of attention heads can access the appropriate task subspace and route that information. (2) DTHs do not identify information-routing targets through simple input-token matching alone; contextual information also plays an important role. (3) DTHs become increasingly important as context length grows: ablating them causes substantially greater accuracy degradation in long contexts than in short ones, suggesting that direct local processing can serve as a competitive fallback only for short contexts, whereas long-context inference increasingly relies on anchor-based information. Our findings provide preliminary evidence that LMs utilize a cache-and-forward mechanism for information from earlier context, rather than re-traversing preceding tokens at later positions to retrieve that information.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Information Flow, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wAfqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how Duplicate Token Heads (DTHs), a set of special attention heads, route context information through repeated tokens in the input. The paper assumes that these repeated tokens serve as information anchors: DTHs collect, cache and forward contextual information to the repeated positions later in the text for later retrieval. It first uses a DTH score together with activation patching on natural text to locate DTHs that are genuinely functional, and then analyzes the query–key interactions to explain how these heads perform retrieval across matched and unmatched tokens. Experiments on six models and nine simple tasks from three families show that only a small subset of DTHs actually transfers task-relevant information: their addressing is not based on simple token matching, instead it is modulated by the prefix context through query–key interactions. The longer the context, the more severe the loss caused by ablating functional DTHs. The paper also finds that the sets of DTH sets overlap with each other across tasks, and observes weak compensatory effects in ICL.

Critical strengths
  1. The experiments cover a fairly diverse set of models and tasks, so the conclusions are reasonably robust.
  2. The procedure for finding DTHs and functional DTHs is sound, and the DTHs identified on general text such as Wikipedia generalize well. This can help guide follow-up work of the same kind.
Critical weaknesses
  1. The practical value of the paper is limited. As an interpretability paper, it does explain how DTHs work when the model performs retrieval, but it does not suggest a pratical application for this finding, and all the experiments are built on simple toy tasks.
  2. The role of DTHs is not separated from plain retrieval. Consider a retrieval setting in which no token is repeated (e.g. the question is paraphrased so that it no longer contains the identical token from the context), DTHs should then no longer be critical to retrieval, yet the paper never discusses this case.
  3. The writing has several problems: the DTH score in Sec. 2.3 does not give a formal formula; references to figures are inconsistent, as in "Fig." (Line 72) vs. "Figure" (Line 185); Fig. 2 is never referenced anywhere in the text; no Related Work section that discusses how this paper differs from prior work. Since the main text is around 8.5 pages and still has some room, the authors should fill these gaps.
Questions for authors
  1. A classic retrieval task setting is "Needle in a Haystack", where a sentence is inserted somewhere in a long context and the model must locate it precisely. It tests robustness in long contexts and has clear practical value, so the paper would be much stronger if the authors verified the DTH findings on this setting.
  2. Why do the DTHs found on static text fail to become functional DTHs in the later activation patching? Is there some parameter-level or behavioral difference that keeps only some DTHs from being genuinely functional?
  3. Some models in Sec. 2.2 have both base and instruct versions. Would the distribution of DTHs differ between them?
  4. Taking Fig. 1 as an example, do DTHs pay particular attention from a2a_2 to a1a_1? The figure only shows the attention from the answer position tt to a1a_1 and a2a_2. Before the Query is actually asked, I am curious whether DTHs already store the relevant information at the repeated token positions in advance.
4 comments

Thank you for the careful review of our paper. Your suggestions and questions have been very helpful in improving the quality of our work. Below, we respond to each of your points.

W1 Practical Value: Thank you for recognizing our explanation of how DTHs work. We agree that we do not propose a concrete practical application. However, we believe that interpretability research provides a foundation for understanding the capabilities of frontier models, and in this sense it can be viewed as a form of application in its own right. Please see our response to Reviewer kAQx (Q1) for details.

In addition, the use of simple tasks is a deliberate design choice. Activation patching and QK decomposition require counterfactual inputs that differ from the original input only minimally, and such inputs are difficult to construct in open-ended text. This is standard practice in mechanistic interpretability; for example, the IOI study (Wang et al., 2023) adopts the same approach.

W2 Retrieval Without Repeated Tokens: Thank you for this insightful suggestion. Following it, we added a retrieval setting in which” no token is repeated”, and found that the importance of DTHs does not decrease. This is an interesting result, and it may further broaden our understanding of the role functional-DTHs play in the model. Please see our response to Reviewer kAQx (Q1) for details!

W3 Presentation: Thank you for these helpful comments. We added a a formal formula of the DTH score in Sec. 2.3, unified all figure references, and now reference Fig. 2 in the main text. Regarding the Related Work section: after adding the content requested by the reviewers, the main text no longer has room for a standalone section. We will therefore clearly discuss related work in the Introduction and background, explain how our paper differs from prior work.

Q1 Needle in a Haystack: We added a “Needle in a Haystack” experiment. The results show that, across different context lengths, ablating DTHs still leads to a drop in accuracy for multiple models, although the size of this drop does not consistently increase with context length. These results indicate that DTHs continue to play an important role in long-context retrieval. The tables below report the relative accuracy change after ablation. Columns correspond to insertion positions within the background: 0% denotes the beginning and 100% the end.

Table 1: Qwen3-8B

Background length (tokens)0%25%50%75%100%
512−6.03%−18.18%−7.07%−12.50%−15.12%
1,024−8.47%−20.79%−17.31%−17.76%−11.70%
4,096−8.55%−9.91%−8.11%−9.01%−13.33%
8,192−17.95%−14.95%−15.32%−10.71%−22.09%

Table 2: Llama-3.1-8B

Background length (tokens)0%25%50%75%100%
512−12.63%−13.95%−8.24%−10.96%−1.52%
1,024−10.87%−21.43%−22.09%−15.29%+1.49%
4,096−18.75%−10.47%−11.76%−18.37%−1.35%
8,192−11.93%−18.00%−25.30%−16.49%−4.62%

Q2 DTHs and Functional-DTHs: Thank you for this insightful question. The DTH score is computed from attention patterns alone, i.e., it measures where a head attends, whereas activation patching measures what the head moves and whether this information is used downstream. A head can therefore consistently attend to repeated tokens without being functional.

Regarding parameter-level differences, we decomposed the OV matrices of these heads but did not find features that clearly separate functional DTHs from other DTHs. Regarding behavioral differences, we found that ordinary DTHs address repeated tokens based on token identity alone: they prefer any occurrence with the same identity, even when its preceding context does not match. In contrast, functional-DTHs appear to select occurrences based on both token identity and contextual information, so their preference for an occurrence with the same identity but mismatched context is substantially weaker (based on findings of section3 ).

Q3 DTH Distribution for Base and Instruct Versions: We additionally compared the base and instruction-tuned/post-trained versions of Llama-3.1-8B and Qwen3-8B. We find substantial overlap in both the DTH candidates identified on the general-text discovery dataset and the functional-DTHs identified on the entity tracking task. Notably, for both model families, the six heads with the highest recovery are identical across versions and retain the same ranking. These results suggest that post-training largely preserves existing mechanisms, which is consistent with the findings of Prakash et al. (2024).

Table: DTH Overlap Between Base and Post-trained Models

Model familyShared / per-version candidatesCandidate Jaccard overlap (%)Shared / per-version functional-DTHsFunctional Jaccard overlap (%)
Llama-3.1-8B: Base vs. Instruct49/5289.097/877.78
Qwen3-8B: Base vs. Post-trained55/5890.166/950.00

Jaccard overlap is the intersection size divided by the union size.

Q4 Mechanism Generalization: Thank you for this interesting question.

  • First question. Our mechanism predicts that functional-DTHs attend from a₂ to a₁ when their contextual information matches. We verified this using inputs such as: "The apple is in Box A, the orange is in Box B. Put the orange into Box A. Box A contains the". At the second "A", functional-DTHs indeed attend to the first "A". For Qwen3-8B, 7 of the 9 functional-DTHs place their maximum attention on the first "A". For the remaining two heads, the first "A" receives the second- or third-highest attention.

  • Second question. If we understand correctly, you are asking whether a₂ thereby already contains a₁'s information before the query. Attention from a₂ to a₁ implies that some information is written into a₂, but which information is written remains unclear. In a closely related box-tracking setting, Tang et al. (2026) found that information accumulated from earlier occurrences is hardly linearly decodable at later occurrences of the same box ID. Therefore, determining whether a₂ carries a₁'s task-relevant information in another form, and whether this information is used downstream, requires causal analysis, which we leave to future work.

Reviewer_wAfqOpenReview ↗

Kindly reminder

Thank you for the response. It seems that the authors forget to update the revised paper? I would appreciate it if the paper is updated timely.

Sorry for late upload. Revised paper has been update.

Reviewer_PWJHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper explores the existence and function of 'duplicate token heads' ('DTHs'). DTHs were first documented in the famous Wang et al. IOI paper. DTHs are attention heads that move information from one token position to a later token position with the same token identity (i.e. the same embedding). From what I can remember, the original Wang et al. paper finds that DTHs are a part of a circuit that identifies repeat tokens for the sake of down-weighting these tokens as potential outputs. From what I understand, the authors of this paper are instead studying a setting in which DTHs must resolve ambiguities that arise when the same token is used in different ways.

For example, language models can still solve entity binding tasks with context like 'thing 1 is in box A and thing 2 is in crate A', even though the 'A' token is used twice and is referring to two different objects. So the question here is how DTHs resolve this ambiguity.

To answer this question the authors look at a few different binding tasks. For each, they first locate DTHs by first identifying candidate heads using an attention based analysis, and then reducing the candidate set using a causal mediation analysis. This produces a very small handful of attention heads that appear to play a uniquely important role at duplicate token positions. The authors then introduce this concept of 'direct routing' (which I admittedly never really understood) and suggest that the attention logits of DTHs can be decomposed into parts that represent token identity and parts that represent preceding context. Finally they suggest that DTHs might be especially important in long contexts, where an alternate mechanism that simply reconstructs the contextualized meaning of a repeat token on the fly is less likely to activate.

Critical strengths
  • The authors run their analysis on multiple tasks (albeit closely related tasks) and multiple models
  • The experiments rely on a nice combination of attention analyses, causal mediation, and causal interventions
  • The overall structure of the paper makes sense, with respect to the goal of illustrating DTH's
Critical weaknesses
  • The prose gets in the way of this paper; it's often quite clunky and hard to understand. In this case I would actually recommend that the authors refer to an LLM for line edit suggestions!
  • It doesn't seem like DTH's are a new discovery, and the paper offers limited insight into DTH's beyond their existence. I think the point about DTH's needing to resolve ambiguity is interesting, but the paper affords this point limited treatment, and the explanation of how DTH's integrate prior context (and what that even means) is unsatisfying. The point about DTH's being useful in long context settings is also interesting but the presentation of results is somewhat incoherent and difficult to follow
Questions for authors
  • line 033 'both human language processing' I think you should leave this out, there is really no need or basis to refer to human language processing here
  • line 037 what is meant by 'repeated structures'?
  • line 046 the condition is more complex but I’m not sure that it is more ‘realistic’. Would prefer the framing to be that the authors are just going to look at how DTH’s resolve ambiguity.
  • line 052 'Not all DTH outputs...' this sentence is particularly clunky and difficult to parse given its importance
  • Figure 1 firstly it remained somewhat unclear to me throughout the entire paper what 'direct routing' actually is, even though it is the focus of this figure and is referred to frequently. In the figure itself, I couldn't figure out what the colons 'X : Y' meant -- are they supposed to be keys and values? Or values and keys? I also think the figure may be easier to follow if it just depicted a real prompt instead of something more abstract.
  • line 142 wondering if the authors think that there could be functional DTH’s that don’t make it past the token preference filter
  • line 190 what is the justification for this threshold parameter?
  • line 172 it's probably worth including a concrete example of the activation patching experiment. Generally the writing requires too much context from the appendix. The experiments should be easy to understand without frequently referring to the appendix, which is currently not entirely the case.
  • line 275 not sure what the 'neutral context' is
  • Figure 7 it’s almost impossible to see the green and gray lines, should jitter y pos slightly
  • Section 4 what do you make of the U-shaped curves?
  • Section 4.1 wondering what the authors think about this experiment in relation to Todd et al. function vectors work.
2 comments

Thank you for the careful review of our paper. Your suggestions and questions have been very helpful in improving the quality of our work. Below, we respond to each of your points.

Presentation (W1, Q1–Q11)

  • W1: Thank you for the suggestion. We used a language model to help polish the wording and flow of the text.

  • Q1: We agree and have removed the 'both human language processing' so that the discussion focuses on language models.

  • Q2: By “repeated structures,” we mean that the same word or token appears multiple times in the input. We have replaced this term with the more explicit phrase “repeated occurrences of the same tokens.”

  • Q3: We agree that expressions with and without a prefix are both common in natural language. We have therefore removed “more realistic” and made clear that our focus is on “how DTHs resolve ambiguity.”

  • Q4: We have rewritten this sentence to be more direct and precise: “When patching DTH outputs from one input into a counterfactual input, we find that not all DTHs exhibit strong recovery.”

  • Q5: Direct routing is our name for the following information pathway. Through functional-DTHs, the model reads contextual information that has already formed at an earlier occurrence of a repeated token. It then transfers this information to a later occurrence of the same token, where it contributes to the subsequent prediction. The colons “X:Y” in the figure does not denote a key-value pair. It means that information X is encoded in the representation at position Y. To reduce ambiguity, we have adjusted Figure 1 and added a concrete input to the figure: "The card is in red bag, the bell is in blue bag, ..., red bag contains the". For example, the “bag” position in “red bag” can form a contextualized representation that contains earlier information such as the color. Step 1 in the figure 1 shows this contextualization. Step 2 shows how functional-DTHs locate the information source and transfer the information.

  • Q6: Could some functional-DTHs fail to pass the token-preference filter? We believe this is very unlikely. We retain only the top 5% of heads by DTH score, but the DTH-score ranking in Figure 8 shows that the vast majority of DTHs fall within a small set of top-ranked heads. If a head fails the token-preference filter yet still has a large effect on the task, it would not be a DTH. It would instead be a different type of head that contributes to the task in some other way.

  • Q7: We set this threshold based on the distribution of single-head recovery experiments (Figure 3). We then kept it fixed across all models and tasks.

  • Q8: We added a concrete example of activation patching to the experimental description and restructure the relevant sections, so that readers can follow the core experiments without frequently consulting the appendix.

  • Q9: Neutral contexts are generic texts that do not contain our experimental task. They reduce the influence of task-specific context and are used later to estimate the representation direction of token identity.

  • Q10: These curves are hard to tell apart mainly because the accuracy of the random controls is very close to that of the clean condition, so the curves overlap. We have added display offsets for clear presentation.

  • Q11: In the settings that show U-shaped curves, ablating the messages of functional-DTHs causes less damage when the source information is at the beginning of the input or near the query at the end than when it is in the middle. This suggests that the model’s reliance on the direct-routing pathway depends on position, and that at some positions, the remaining computational pathways can still support correct predictions. We describe this as the possible existence of alternative mechanisms.

Novelty (W2): We agree that the existence of DTHs is not a new finding of this paper. Building on prior work, our main contributions are twofold: (1) we characterize the QK addressing mechanism of DTHs in ambiguous contexts, and (2) we characterize how much the model causally relies on this information-transfer pathway under different input conditions. Please see our response to Reviewer kAQx (W1) for a more detailed explanation.

Relation to Function Vectors (Q12): Todd et al. used causal mediation analysis to identify a set of attention heads that transport ICL task information. They called the combination of these heads’ outputs, averaged across examples, a function vector. Their attention analysis shows that these high-AIE heads mainly attend to the output tokens (labels) of the demonstrations.

Later work found that transferable task representations can also appear at format-token positions in demonstrations, such as the colon in “Answer:”. Moreover, these representations are not limited to the final query. They form at the corresponding positions as demonstrations accumulate (Li et al., 2025).

These findings led us, after completing the binding-task analysis, to explore a more specific question: can the functional-DTHs identified in the binding task use the repeated output colons to transfer task-relevant information from earlier context to the output colon of the final query?

However, our patching experiments (Table 3 in the paper) show that this pathway yields only weak recovery in the ICL settings we tested. Overall, generation tasks improve more than classification tasks, but both remain well below the strong causal effects observed in the binding task.

We hypothesize that functional-DTHs are not a primary mechanism of ICL. Todd et al. already screened attention heads one by one. If functional-DTHs were among the causal heads they identified, this distinctive attention pattern would likely have been reported in their paper. In addition, the difference between classification and generation tasks may reflect how differently the two task types rely on task recognition and on demonstration-mapping information. This is consistent with existing TR/TL analyses (Yang et al., 2026). Following Reviewer kAQx’s suggestion (W5), we will explicitly present this part as exploratory and leave further analysis to future work.

Reviewer_kAQxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper builds on prior interpretability research on duplicate token heads (DTHs), which has linked these components to the retrieval and transfer of contextual information. It examines how models distinguish repeated occurrences of the same token in different contexts and select the information source relevant to a query. Using settings such as colour–noun combinations, the paper extends the analysis from single-token identity matching to context-sensitive matching. Through value patching, ablation experiments, and query–key decomposition, the authors show that a subset of DTHs uses contextual information encoded at repeated-token positions to guide addressing and transfer task-relevant information to the query. Reliance on this pathway varies with the task, input size, and information position. The paper also extends the experiments to in-context learning, where the observed effects are weaker.

Critical strengths
  1. The paper addresses an important problem in mechanistic interpretability by extending the study of DTHs from single-token matching to context-sensitive matching of token combinations. This is a meaningful advance that brings the analysis closer to realistic language use.
Critical weaknesses
  1. The novelty of several findings in Sections 2 and 3 is difficult to assess. Prior work has established cross-task reuse of DTHs (Merullo et al., 2024) and QK-based retrieval of stored information (Prakash et al., 2026), but the manuscript does not sufficiently clarify how its findings advance these accounts. The current presentation and limited discussion of related work leave unclear which conclusions confirm known phenomena, which extend them to ambiguous repeated-token settings, and which provide new mechanistic insights. These distinctions should be made explicit when presenting the results.
  2. The role of Section 2 is unclear in the current presentation. Under the heading ''Preparation'', it combines background, experimental setup, component selection, and empirical findings. It is difficult to distinguish necessary preliminaries from new contributions, or to understand why each analysis is needed for the paper's central claims. The authors should clarify each subsection's purpose, its novelty relative to prior work, and how it supports the subsequent sections.
  3. The interpretation of Figure 7 in Section 4 is stronger than the evidence supports. The ablations show that reliance on anchor-to-target messages varies with task, position, and input size. However, residual accuracy does not identify a ''local information-collection mechanism'', nor does its decline establish that this mechanism fails in longer contexts. Increasing the number of items also changes distance and interference, leaving the cause of greater reliance unresolved. The authors should either provide targeted evidence for the proposed competing mechanism or present it as a hypothesis and narrow claims such as ''must rely'' accordingly.
  4. Section 3 leaves the scope of contextual addressing unclear. Its experiments use adjacent colour–noun phrases, where semantic association coincides with linear proximity. This design does not distinguish local prefix matching from addressing based on semantic relations. The authors should test constructions that separate the relevant attribute from the noun and include nearby, irrelevant attributes. Postnominal descriptions such as ''the bag, whose color is black'' raise a further question: under causal attention, the noun cannot encode the subsequent colour information. Where is this relation integrated, and which position supports its retrieval? Such analysis would clarify whether the proposed mechanism extends beyond local-prefix matching and how relevant relational information reaches the routing anchor.
  5. Section 4.1 provides limited additional mechanistic insight. Most improvements are small, and the absence of uncertainty estimates makes their reliability difficult to assess. Even where patching improves performance, the experiment does not distinguish task-level information from demonstration-specific content. It therefore remains unclear what substantive new conclusion this section establishes. The authors should clarify its contribution and support it with uncertainty estimates and targeted analyses of the transferred information, or present these findings as exploratory results.
Questions for authors
  1. Can this paper show a way to explain more powerful models and their powerful reasoning ability? The repeated token can explain some reasoning capacity in easier tasks but in more complex scenario, there could be no repeated tokens, even no similar or related tokens. Now the most powerful models show surprisingly ''noticing'' ability to find some pattern or relation which has never been found, like in math. Do you believe the capacity about repeated token is related to or finally growth to the powerful reasoning capacity of the most powerful models.
  2. Does the identified mechanism generalize to nonadjacent attributes, especially when a nearby distractor makes linear proximity an unreliable cue to semantic association? For descriptions such as ''the bag, whose color is black'', where is the colour–entity relation represented, given that the original noun position cannot attend to the later colour token? How is that representation subsequently retrieved?
3 comments

Thank you for the careful review of our paper. Your suggestions and questions have been very helpful in improving the quality of our work. Below, we respond to each of your points.

W1 Novelty: We acknowledge that the original paper did not sufficiently distinguish our contributions from prior findings, and we clarify this distinction below. Prior work has established the existence and cross-task reuse of DTHs(Wang et al., 2023; Merullo et al., 2024). Prakash et al. (2026) also study QK-based retrieval of stored information, but do not directly examine how DTHs selectively address relevant sources among multiple occurrences of the same token. Building on these studies, our contributions are twofold:

  1. Characterizing DTH addressing in ambiguous contexts. Prior work characterizes DTHs by their attention preference for matching tokens. This property alone, however, does not explain how they select the query-relevant source when multiple identical tokens carry different information. We show that their addressing depends not only on token identity but also on contextual information in token representations. Even when a source token in the preceding context and the target token in the query have identical token identities, a prefix mismatch lowers the attention logit directed toward that source position for most tested functional-DTHs. Thus, identical tokens do not necessarily receive equal attention: prefix information further modulates which source these heads favor.
  2. Characterizing models’ reliance on DTH-mediated information transfer. Across the tested tasks, jointly ablating the messages transmitted by functional-DTHs generally causes substantial accuracy losses, demonstrating strong reliance on this pathway. This reliance is nevertheless position-dependent. Ablation typically causes less damage when the relevant information appears at the beginning and the end of the input, particularly in the first item, than at interior positions. These findings suggest that alternative information pathways may support correct predictions under certain positional conditions; as the W3 mentioned, they do not establish the identity of those alternative mechanisms.

Following your suggestion, we revised the Introduction to distinguish these contributions more clearly from established findings.

W2 Presentation: We have reorganized Section 2 into three clearly distinguished parts: Background, Experimental Settings, and Component Selection. We also clarified the purpose of each part and how component selection supports the subsequent mechanistic and causal analyses.

W3 Interpretation of Figure 7: We agree that our description of alternative mechanisms was overly specific. Our intended observation was that, in some settings, placing the target information at the beginning of the input leads to a smaller performance decrease when information transfer through functional-DTHs is ablated, indicating that the remaining computation can still support correct predictions. However, this does not identify a specific "local information-collection mechanism," nor does it establish that such a mechanism fails in longer contexts. The purpose of this experiment is to characterize the model's reliance on the tested DTH mechanism. In the revised paper, we clarify this distinction, discuss alternative mechanisms only as possibilities, and soften statements such as "must rely."

To address the concern that increasing the number of items changes both distance and interference, we added a Needle-in-a-Haystack experiment. We insert a single queried information item into unrelated natural-language backgrounds of varying lengths and vary its insertion position. This allows us to examine the effect of background length without introducing additional competing binding items. Across several models, ablation reduces accuracy at all background lengths. The effect does not consistently increase with length and is smaller than in the original multi-item experiments.

Taken together, these results show that DTHs play an important role in long-context settings. Moreover, the larger ablation-induced drop in the multi-item experiments indicates that functional-DTHs are key to selectively transferring information among similar items.

The tables below report the relative accuracy change after ablation. Columns correspond to insertion positions within the background: 0% denotes the beginning and 100% the end.

Table 1: Qwen3-8B

Background length (tokens)0%25%50%75%100%
512−6.03%−18.18%−7.07%−12.50%−15.12%
1,024−8.47%−20.79%−17.31%−17.76%−11.70%
4,096−8.55%−9.91%−8.11%−9.01%−13.33%
8,192−17.95%−14.95%−15.32%−10.71%−22.09%

Table 2: Llama-3.1-8B

Background length (tokens)0%25%50%75%100%
512−12.63%−13.95%−8.24%−10.96%−1.52%
1,024−10.87%−21.43%−22.09%−15.29%+1.49%
4,096−18.75%−10.47%−11.76%−18.37%−1.35%
8,192−11.93%−18.00%−25.30%−16.49%−4.62%

W4 the Scope of Contextual Addressing: To address your concern that the original design does not distinguish addressing based on local prefix matching from addressing based on semantic relations, we followed your suggestion and added an experiment separating the relevant attribute from the noun. For example, the information source becomes “the apple is in black plastic bag,” while the query phrase remains “black bag.” Across six models, color mismatch still reduces the mean attention logit for 67 of the 72 heads, and ctx-ctx exhibits the largest negative channel change for 69 of the 72 heads. These results show that the observed contextual modulation does not require the color and noun to be immediately adjacent, further supporting the role of contextual information in DTH addressing.

Your example, “the bag, whose color is black,” raises a distinct question: where is a subsequently introduced attribute integrated, and how is the resulting representation retrieved? Under causal attention, the original “bag” position cannot encode the later occurrence of “black.” Consequently, even when the subsequent query contains “black bag,” reading the representation at that original “bag” position cannot provide the subsequently introduced color information. This does not contradict the mechanism we study, which transfers information from positions that already contain the relevant context. The model may still solve the task by integrating the information at later positions and reading it through other pathways, but locating those representations and circuits is beyond the scope of this paper. We would be glad to explore this in future work.

W5 exploration of ICL: We agree to present the results in Section 4.1 as exploratory. After adding uncertainty estimates, our overall conclusion remains unchanged: patching generally yields larger performance improvements on generation tasks than on classification tasks, but recovery in ICL remains modest compared with the strong causal effects observed in binding tasks. For a more detailed discussion of this section, please see our response to Reviewer PWJH’s final question (Q12).

Table 3: added uncertainty estimates

TaskAccuracy change95% CI
SST-2−0.15[−0.49, +0.20]
SST-5+0.24[−0.20, +0.68]
AGNews+0.63[+0.15, +1.12]
FP+0.05[−0.29, +0.39]
Present-past+0.39[−0.15, +0.93]
Present-ing+0.73[+0.19, +1.29]
English–French+3.66[+2.76, +4.55]
Antonym+0.39[−0.05, +0.87]
Synonym+0.83[+0.34, +1.35]

Q1 DTH mechanism for powerful models' "noticing" ability: Thank you for this exciting question. We believe our findings provide a mechanistic starting point for understanding the “surprisingly ‘noticing’ ability” you describe. Our study shows that DTHs can retrieve information from the residual stream at earlier repeated token positions, and that their addressing is modulated by both token identity and contextual information in residual stream.

In addition, to address Weakness 2 raised by Reviewer wAfq, we ran a new retrieval experiment in which “no token is repeated (e.g. the question is paraphrased so that it no longer contains the identical token from the context)”. This gave us an interesting finding: in this setting, ablating the outputs of the functional DTHs leads to a large drop in model performance.

This suggests a reasonable conjecture. functional-DTHs (and possibly other functional-DTH like attention heads) rely on contextual information in the residual stream to decide which tokens to attend to. As a result, even when the context contains no repeated tokens, or not even similar tokens, the model can still use shared properties or more abstract relations encoded in the representations to “notice” some pattern or relation which has never been found, like in math”. We thank the reviewer for this insightful question, and we plan to explore it in depth in future work.

Tabel 4 NoLiMa Retrieval Ablation

ModelQuery typeClean accuracy (%)Functional-DTH ablation (%)Random-head ablation (%) (mean)
Llama-3.1-8BDirect79.175.6771.22
Llama-3.1-8BOne-hop41.171.6734.93
Qwen3-8BDirect98.1220.3898.56
Qwen3-8BOne-hop70.7223.9771.46

Direct: query that directly contain keywords from the source text; One-hop: query shares no keywords with the source text

Q2 Mechanism Generalization: This mechanism also generalizes to nonadjacent attributes. The case of “the bag, whose color is black” involves a separate question that should be kept distinct; please see our response to W4 above for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.