acceptodds
Under review as a conference paper at ICLR 2027

Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

Abstract

Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with global attention. However, how these choices shape in-context retrieval remains unclear. To study this question, we take a mechanistic view, tracing how positional encoding (PE) choice shapes the internal mechanisms models use for in-context retrieval. Across 22 open-weight models spanning eight families, we find that standard RoPE models rely primarily on positional retrieval, while PE hybrids shift toward semantic retrieval. On a controlled pre-training ablation, we show that confining positional encoding to local layers is sufficient to produce this semantic shift, and find that it selectively degrades representations of positional information. Finally, we show that the reported long-context gains of PE hybrids mask a retrieval trade-off: while SWA NoPE improves over RoPE on multiple-target retrieval and QA, it degrades when distinguishing semantically similar keys. These behavioral differences track the mechanism shift from positional toward semantic mechanisms rather than a uniform improvement in long-context retrieval.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, in-context retrieval, positional encoding, long-context

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xw7tOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates a range of RoPE and Position Encoding hybrid over 22 open source-instruction tuned models to study mechanism allocation --- position, lexical, reflexive --- for retrieval, using the models' native PE configurations and using a broad definition of "hybrid". The lexical and reflexive mechanisms are characterized as "semantic". In an evaluation on 29 entity binding tasks, hybrid models show a lower proporation of positional --- and higher proportion of semantic mechanisms compared to RoPE-based models.

The paper then studies the pretrained checkpoints from Qiao et al (2026) involving RoPE, Sliding Window Attention (SWA) RoPE, and SWA NoPE. Again SWA NoPE tends towards semantic mechanisms compared to RoPE. This was further confirmed with experiments with ordering IDs.

Since SWA NoPE has been shown to perform better on long context than other models, does that mean that long context is generally handled better by semantic mechanisms? The paper shows that in fact SWA NoPE is not uniformly better and the benefit of its bias to semantic mechanisms is task-dependent.

Critical strengths
  • Clearly written paper with simple but potentially useful outcomes that clarify our understanding of the relative benefits of various PE options.

  • Experiments with a large number of open-source models.

Critical weaknesses
  • Experiments and ablations are limited by the available resources. The authors apparently used various models "out of the box".

  • Large difference in size between the main set of 32 models, and those of Qiao et al raises the obvious question of how sensitive all of this is to model size.

  • The results, while interesting, may seem a bit thin as a contribution.

Questions for authors

Just a clarification question: The difference between ordering IDs and positional mechanism is still not entirely clear to me. At least for the example given in Figure 2, an ordering ID of 3rd for an answer would seem to correspond to position 3 in the list.

3 comments

Thanks for the helpful feedback! We are glad the reviewer found the paper clearly written with potentially useful outcomes. One of the reviewer’s main concerns seems to be about novelty, so we would like to clarify why we believe our results constitute a meaningful contribution:

  1. Interpretability. Prior mechanistic work on in-context retrieval studied only uniform-RoPE models. We show that recent PE hybrids rely on a qualitatively different dominant mechanism, resulting in a group of models that perform retrieval fundamentally differently and should be studied accordingly.
  2. Architecture. Long-context architectures are typically compared on aggregate benchmark scores. Our mechanistic lens reveals a trade-off that these scores hide and may explain related findings on architectural design. For example, Cabannes et al. [1] find that smaller sliding windows improve long-context performance in local-global architectures, which our results suggest may reflect a stronger shift toward semantic retrieval (Section 4.3) rather than uniformly better retrieval.

Below, we also respond to the reviewer’s concerns about computational limitations (W1/W2) and how the positional mechanism uses ordering IDs (Q1).

W1/W2: Computational Limitations


Experiments and ablations are limited by the available resources.

Large difference in size between the main set of 32 models, and those of Qiao et al raises the obvious question of how sensitive all of this is to model size.

We agree that training at larger scales is beyond our resources (see Limitations), but we view our two settings as complementary. The model battery (1.7B–72B) shows that the semantic shift persists in real open-weight models and is confounded by many factors such as training data and model family. The controlled pre-training ablation removes these confounds to identify why the shift occurs, at the cost of smaller scale.

Q1: Positional Mechanism


The difference between ordering IDs and positional mechanism is still not entirely clear to me. At least for the example given in Figure 2, an ordering ID of 3rd for an answer would seem to correspond to position 3 in the list.

Thanks for this question. In this work we borrow the terminology of an ‘ordering ID’ from Prakash et al. [2] to mean this high-level list position (first item, second item, third item) that you describe. The positional mechanism is the retrieval process that uses these OIs, and the lexical mechanism uses the semantic key. To further explain the example given in Figure 2, at the items in the list the ordering IDs would be stored (1, 2, 3, 4) in both the counterfactual and original prompt, but since we are patching the pointer from the counterfactual prompt, which is a pointer to the first item, the positional mechanism corresponds to Box B.

We hope these responses clarify our contributions and address your concerns. Please let us know if there are any remaining questions or points that require further clarification, we’d be happy to elaborate.

References


  • [1] Cabannes, Loïc, et al. "Short window attention enables long-term memorization." International Conference on Learning Representations. Vol. 2026.
  • [2] Prakash, Nikhil, et al. "Language models use lookbacks to track beliefs." International Conference on Learning Representations. Vol. 2026.
Reviewer_xw7tOpenReview ↗

Response to response

Thank you for clarifying these points. Note that my third "critical weakness" about the extent of the contribution wasn't really a highly critical concern: I am satisfied this is a nice contribution. The overall scores for ICLR this year seem to be skewed downwards, so I still think 3 is an appropriate score.

We are glad you remain supportive of the paper, thank you again for taking the time to review our work!

Reviewer_27YyOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how positional encoding choices affect the mechanisms language models use for in-context retrieval. Using counterfactual activation patching across 22 instruction-tuned models from eight families, it finds that hybrid positional encoding architectures allocate relatively more retrieval to lexical and reflexive mechanisms than uniform-RoPE architectures. Experiments with controlled 665M-parameter checkpoints show that removing RoPE from global attention layers shifts mechanism allocation, whereas introducing sliding-window attention alone does not. Linear probes additionally reveal reduced decodability of entity ordering information. Finally, RULER evaluations and a controlled overlapping-key task demonstrate a retrieval trade-off: SWA NoPE improves several retrieval and QA tasks but performs worse when selecting among competing, similar keys.

Critical strengths
  1. The model battery exhibits a consistent aggregate separation between uniform-RoPE and hybrid architectures despite substantial differences in model family and scale. The controlled RoPE/SWA RoPE/SWA NoPE comparison then provides evidence that removing positional encoding from global layers, rather than sliding-window attention alone, can produce the shift. This makes the architectural finding substantially more informative than a cross-model correlation.
  2. SWA NoPE’s improvements on multiple-value and multiple-query retrieval coexist with worse discrimination among overlapping keys. The controlled key-overlap experiment, together with the error analysis in Appendix D, makes this trade-off concrete. This is consequential for understanding which retrieval workloads benefit from hybrid architectures and for evaluating architectural improvements beyond a single aggregate benchmark.
Critical weaknesses
  1. The controlled mechanism analysis uses the 20-entity Boxes task, while the behavioral conclusions concern different tasks at 16K and 32K context lengths. The paper does not directly establish that the same mechanism allocation operates in those long-context settings or that changing it mediates the observed performance differences. Likewise, reduced linear decodability at one layer supports a representational difference but does not establish impaired ordering-ID formation as the cause. These limitations matter because explaining the behavioral trade-off through the mechanism shift is a central contribution.
  2. Appendix A.5 varies the number of identical words shared by four-word keys, directly manipulating lexical overlap rather than independently measuring semantic similarity. This is particularly relevant because the paper groups lexical retrieval and reflexive retrieval together as semantic mechanisms. The results convincingly demonstrate sensitivity to overlapping identifiers, but they do not clearly distinguish semantic interference from difficulty matching partially overlapping strings. More precise terminology or a control separating lexical overlap from semantic similarity would strengthen the interpretation.
Questions for authors
  1. Do you have evidence that the mechanism allocation measured on Boxes persists in the long-context confusable-key task? A direct mechanism measurement or targeted intervention in that setting would materially strengthen my assessment of the proposed architecture–mechanism–behavior explanation.
  2. Can you clarify whether the key-overlap results should be interpreted as lexical interference or semantic interference? Do results remain consistent when comparing semantically related keys without shared words against lexically overlapping keys with little meaningful semantic relationship?
  3. How sensitive is the aggregate architectural separation to the patching-layer choice and to unknown/no-effect outcomes? Appendix B provides useful layer-selection validation, but Appendix C reports substantial unclassified or unaffected fractions for some models. Results across nearby valid pre-retrieval layers, with uncertainty estimates for the positional ratios, would help establish the robustness of the separation.
2 comments

Thanks for the thoughtful review! We are glad the reviewer appreciates our controlled comparison and the downstream utility of our work. Below, we respond to the reviewer’s main concerns with the mechanism analysis at long context (W1/Q1), our use of the word semantic (W2/Q2), and the layer selection process (Q3).

W1/Q1: Long-Context Mechanism Analysis


The controlled mechanism analysis uses the 20-entity Boxes task, while the behavioral conclusions concern different tasks at 16K and 32K context lengths.

Do you have evidence that the mechanism allocation measured on Boxes persists in the long-context confusable-key task?

Thank you for raising this point. While we cannot directly intervene on the in-context retrieval mechanisms to show they mediate performance, we ran an adapted version of the Boxes task to measure whether this mechanism shift persists in a setting closer to the long-context confusable-key task. We shrink from 20 entities to four for the Boxes task, and insert filler sentences between each entity to increase the context length to 16K/32K, matching the structure of the confusable key task. Then, we perform our same mechanistic analysis on this task to see if the semantic shift persists with a small-scale study (N=500):

CheckpointRoPESWA RoPESWA NoPE
16K (at 16K context length)4.002.341.20
32K (at 32K context length)6.023.341.08

First, we find that the absolute positional ratio increases for all models on this new task, which matches the increased positional retrieval for shorter item lists that prior work finds [1]. Second, we see at both context lengths that SWA NoPE remains the least positional architecture and this gap widens from 16K context length to 32K context length. This result suggests that this shift in mechanism allocation persists at long context.

reduced linear decodability at one layer supports a representational difference but does not establish impaired ordering-ID formation as the cause.

We agree that reduced linear decodability cannot establish impaired OI creation as the cause for the semantic shift and we agree that understanding the underlying mechanism behind this shift could be useful for better understanding the impact of PE choice on the mechanism allocation. However, we believe that the core claims of this paper are not dependent on this result, and we are pursuing this in follow-up work.

W2/Q2: Semantic Overlap


Can you clarify whether the key-overlap results should be interpreted as lexical interference or semantic interference? Do results remain consistent when comparing semantically related keys without shared words against lexically overlapping keys with little meaningful semantic relationship?

Our key-overlap results should be interpreted as content-based interference as opposed to position, and we use semantic throughout this paper to mean content-based retrieval (where lexical and semantic interference both fall under this umbrella term). We agree this classification can be confused with the in-context retrieval mechanisms (lexical, reflexive, positional), and we will describe the confusable-key task as varying key overlap rather than semantic similarity.

Q3: Layer Selection and Unclassified Outcomes


How sensitive is the aggregate architectural separation to the patching-layer choice and to unknown/no-effect outcomes?

As noted in our response to Reviewer h6Vx, we utilize the procedure introduced in Gur-Arieh et al. [1] to perform layer selection (“the last layer at which patching does not transfer an already-retrieved answer” according to the messiness-4 result). We select the last layer before retrieval specifically because earlier layers may still be computing the binding and we need a clean way of measuring retrieval across models, which is why we do not estimate the positional ratio across multiple pre-retrieval layers.

Additionally, the “unknown” and “no effect” outcomes are directly inherited from the counterfactual patching framework. Since the positional ratio is computed only over predictions attributable to a mechanism, it measures the relative allocation among identified mechanisms and is a consistent procedure applied across models.

We hope these responses clarify your questions and concerns. Please let us know if there are any remaining points that require further clarification, we’d be happy to elaborate.

References


  • [1] Gur-Arieh, Yoav, Mor Geva, and Atticus Geiger. "Mixing mechanisms: How language models retrieve bound entities in-context." International Conference on Learning Representations. Vol. 2026.

Thank you again for your review. As the discussion period is closing soon (Nov 18), we wanted to check whether our response addressed your concerns or whether anything needs further clarification.

Reviewer_h6VxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the choice of positional encoding (PE) changes the way language models retrieve information from their context. The authors use the method of Gur-Arieh et al. (2026), which separates three retrieval mechanisms: positional (find an item by its place in a list), lexical (find it by its key word), and reflexive (a direct pointer to the answer). They summarize the result with one number, r_pos = P/(L+R). A higher value means more positional retrieval.

The paper makes three claims. First, in 22 open models, every model that uses RoPE in all layers is more positional (r_pos >= 0.59) than every "PE hybrid" model (r_pos <= 0.53). Second, in controlled 665M models from Qiao et al. (2026), removing RoPE from the global layers (SWA NoPE) cuts r_pos roughly in half, while adding sliding-window attention alone (SWA RoPE) does not. Linear probes also show that the position of each item in the list is harder to decode in SWA NoPE, while the semantic information is unchanged, and the authors suggest this weaker "ordering ID" is why the model moves toward semantic retrieval. Third, on RULER, SWA NoPE improves only some tasks, and on a new task with similar-looking keys it does worse than RoPE, with a gap that grows as the keys become more similar. The authors say this is a trade-off caused by the shift to semantic retrieval.

Critical strengths
  1. The paper turns an interpretability idea into a testable prediction. It links a mechanism inside the model to an architecture choice, and then to behavior. The confusable-key task was designed from this idea. It shows that SWA NoPE's better average RULER score hides a setting where it is worse than RoPE. This is useful, because it warns against reading RULER averages as a general improvement.

  2. The controlled checkpoints are a good choice. The models from Qiao et al. (2026) differ only in architecture, so they allow a fairer comparison than models from different labs. The error analysis in Table 9 is also careful: it shows that SWA NoPE fails by picking a competing key, while SWA RoPE fails in a different way (repeated or broken outputs).

Critical weaknesses
  1. The separation between RoPE models and PE hybrids is confounded with model family and is less robust than it looks.

Seven of the nine "PE hybrids" are Gemma-3 or Gemma-4. Each family belongs to only one group, so the clean separation is really based on about 8 families, not 22 independent models. The two hybrids outside Gemma (Command R7B at 0.53 and SmolLM3 at 0.49) are very close to the lowest RoPE model (Llama-3.1 8B at 0.59). Inside the RoPE group, Gemma-2 2B and 9B (0.65 and 0.67) are already near the bottom. This looks like a family effect that continues into Gemma-3 and Gemma-4, which could come from shared training data or recipes rather than from the PE design.

The method is also not the same for all models. Each model uses a different layer and different token positions for patching, and the share of "unknown" and "no effect" samples changes a lot (for example, 40% no effect for Gemma-4 12B and 43% unknown for Qwen3 1.7B). r_pos ignores these samples. This makes it hard to know whether small differences in r_pos between models, especially near the boundary, are meaningful.

  1. The explanation in Section 4.3 does not explain the shift in the battery.

The paper argues that the shift happens because RoPE layers only see a local window, and predicts a weaker effect when the window covers the context. The 20-entity prompts should fit inside the windows in Table 1 (512-4096 tokens). Consistent with this, Command R7B, the only battery model with the SWA NoPE design, shows almost no shift (r_pos 0.53, lexical 23.8 vs. positional 21.8). But this means the ranking inside the hybrids is the opposite of what the explanation predicts: the two models that remove PE from some layers (Command R7B and SmolLM3) are the least semantic, while Gemma-3 and Gemma-4, which keep RoPE or p-RoPE in their global layers, are the most semantic. Gemma-3 uses RoPE in every layer, which matches the SWA RoPE condition that did not shift in Section 4.2. The RoPE base does not explain it either, since Qwen2.5 also uses a 1M base and is the most positional model. The battery therefore gives no evidence for the proposed mechanism.

Questions for authors
  1. How long, in tokens, are the binding prompts (including the chat template) for each model in the battery? If they fit inside the sliding windows, the "local view" explanation in Section 4.3 cannot explain the strong shift in Gemma-3 and Gemma-4. How do you reconcile this with the fact that Command R7B, the only model with the SWA NoPE design, shows almost no shift? (Could change my assessment.)
  2. What explains why Gemma-3 and Gemma-4 are the most semantic models in the battery, given that they keep RoPE (or p-RoPE) in their global layers? Could shared training data or distillation within the Gemma family explain this, rather than the PE design? (Could change my assessment.)
  3. Can you show the battery results at the family level, or with a test that accounts for models from the same family, such as a mixed model or a permutation test over families?
4 comments

Thanks for the detailed review! We are glad the reviewer appreciates the actionable aspects of our work and recognizes the benefits of our controlled study isolating the impact of PE choice. Below, we respond to the reviewer’s main concerns about the semantic shift in the model battery (W2/Q1/Q2). We also respond to concerns about the impact of model family (W1/Q3) and methodological consistency (W1).

W2/Q1/Q2: Model Battery Semantic Shift


The explanation in Section 4.3 does not explain the shift in the battery.

We would like to first clarify the scope of our claims. As we note in Section 4.1, the battery has many potential confounders that cannot be disentangled using open-weight models (model family, parameter count, context length, training data). For that reason, we are unable to provide a satisfactory answer for each model’s positional ratio, and instead take this high-level view between RoPE and PE hybrid architectures. The model battery presents two core results: (1) that there exist deployed open-weight models whose dominant retrieval mechanism is semantic and (2) that moving from RoPE toward PE hybrids is strongly correlated with a shift toward semantic retrieval. To isolate the impact of PE choice and remove these confounders, we use the controlled checkpoints in Section 4.2.

How long, in tokens, are the binding prompts (including the chat template) for each model in the battery? If they fit inside the sliding windows, the "local view" explanation in Section 4.3 cannot explain the strong shift in Gemma-3 and Gemma-4.

The binding prompts are 150-400 tokens, so they fit within the sliding windows. However, Section 4.3 does not make any claims about sliding window size during inference. Confining positional encoding to local layers shapes the retrieval mechanisms a model learns when training sequences exceed the sliding window. At inference, the binding prompts are built to measure these already learned mechanisms.

What explains why Gemma-3 and Gemma-4 are the most semantic models in the battery, given that they keep RoPE (or p-RoPE) in their global layers?

As mentioned above, it is difficult to explain the exact position of models given the many potential confounders in the model battery. One explanation is the training data for Gemma-3/Gemma-4 which might promote semantic retrieval. Sliding window size could also be a factor since Command R7B (4096) has a larger sliding window than Gemma-3 (1024) and Gemma-4 (512/1024). Also, Gemma-3 and Gemma-4 both vary the RoPE base across local vs. global layers, while Qwen and Gemma-2 keep the same RoPE base in all layers. Finally, Gemma-4 uses partial-RoPE in global layers. Without knowing the exact long-context procedure for each model, it is difficult to make concrete claims about these relative rankings. We use the controlled checkpoints in Section 4.2 since these factors cannot be disentangled with open-weight models.

W1/Q3: Model Family


The separation between RoPE models and PE hybrids is confounded with model family and is less robust than it looks.

Inside the RoPE group, Gemma-2 2B and 9B (0.65 and 0.67) are already near the bottom. This looks like a family effect that continues into Gemma-3 and Gemma-4, which could come from shared training data or recipes rather than from the PE design.

We agree that the separation shown in Figure 3 is confounded by many factors as discussed above. However, we do note that Gemma-2 27B is the fifth most positional model (0.98) and in the same Gemma family as the examples given. Treating families as the unit of analysis (Q3), a permutation test over the eight families gives a probability of 1/70 (p ≈ 0.014) of observing complete separation by chance, so the separation is not an artifact of counting multiple models per family.

W1: Layer Selection


Each model uses a different layer and different token positions for patching, and the share of "unknown" and "no effect" samples changes a lot.

We utilize the procedure introduced in Gur-Arieh et al. [1] to perform layer selection (“the last layer at which patching does not transfer an already-retrieved answer” according to the messiness-4 result) and patched token selection (“the ‘?’ and ‘:’ of the query, plus the final two chat-scaffold positions”). While the layer and token positions vary across models, the selection procedure is standardized.

Additionally, the “unknown” and “no effect” outcomes are directly inherited from the counterfactual patching framework. Since the positional ratio is computed only over predictions attributable to a mechanism, it measures the relative allocation among identified mechanisms and is a consistent procedure applied across models. Finally, our main claim from the model battery is that this semantic shift occurs, not the exact separation, and this claim is robust to small differences in the positional ratio between models.

We hope these responses address your concerns. Please let us know if there are any remaining questions or points that require further clarification, we’d be happy to elaborate.

References


  • [1] Gur-Arieh, Yoav, Mor Geva, and Atticus Geiger. "Mixing mechanisms: How language models retrieve bound entities in-context." International Conference on Learning Representations. Vol. 2026.

Thank you again for your review. As the discussion period is closing soon (Nov 18), we wanted to check whether our response addressed your concerns or whether anything needs further clarification.

Reviewer_h6VxOpenReview ↗

Reply

Thank you for the response. It addresses my concerns on layer selection and Section 4.3, and I understand the limits of open-weight models. However, Gemma-3 and SmolLM3 shift without the design described in Section 4.3, so I suggest framing the battery as a correlation and reporting the permutation test. I will take your response into account in my final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.