- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper evaluates a range of RoPE and Position Encoding hybrid over 22 open source-instruction tuned models to study mechanism allocation --- position, lexical, reflexive --- for retrieval, using the models' native PE configurations and using a broad definition of "hybrid". The lexical and reflexive mechanisms are characterized as "semantic". In an evaluation on 29 entity binding tasks, hybrid models show a lower proporation of positional --- and higher proportion of semantic mechanisms compared to RoPE-based models.
The paper then studies the pretrained checkpoints from Qiao et al (2026) involving RoPE, Sliding Window Attention (SWA) RoPE, and SWA NoPE. Again SWA NoPE tends towards semantic mechanisms compared to RoPE. This was further confirmed with experiments with ordering IDs.
Since SWA NoPE has been shown to perform better on long context than other models, does that mean that long context is generally handled better by semantic mechanisms? The paper shows that in fact SWA NoPE is not uniformly better and the benefit of its bias to semantic mechanisms is task-dependent.
Critical strengths
-
Clearly written paper with simple but potentially useful outcomes that clarify our understanding of the relative benefits of various PE options.
-
Experiments with a large number of open-source models.
Critical weaknesses
-
Experiments and ablations are limited by the available resources. The authors apparently used various models "out of the box".
-
Large difference in size between the main set of 32 models, and those of Qiao et al raises the obvious question of how sensitive all of this is to model size.
-
The results, while interesting, may seem a bit thin as a contribution.
Questions for authors
Just a clarification question: The difference between ordering IDs and positional mechanism is still not entirely clear to me. At least for the example given in Figure 2, an ordering ID of 3rd for an answer would seem to correspond to position 3 in the list.
3 comments
Response to response
Thank you for clarifying these points. Note that my third "critical weakness" about the extent of the contribution wasn't really a highly critical concern: I am satisfied this is a nice contribution. The overall scores for ICLR this year seem to be skewed downwards, so I still think 3 is an appropriate score.
We are glad you remain supportive of the paper, thank you again for taking the time to review our work!
Thanks for the helpful feedback! We are glad the reviewer found the paper clearly written with potentially useful outcomes. One of the reviewer’s main concerns seems to be about novelty, so we would like to clarify why we believe our results constitute a meaningful contribution:
Below, we also respond to the reviewer’s concerns about computational limitations (W1/W2) and how the positional mechanism uses ordering IDs (Q1).
W1/W2: Computational Limitations
We agree that training at larger scales is beyond our resources (see Limitations), but we view our two settings as complementary. The model battery (1.7B–72B) shows that the semantic shift persists in real open-weight models and is confounded by many factors such as training data and model family. The controlled pre-training ablation removes these confounds to identify why the shift occurs, at the cost of smaller scale.
Q1: Positional Mechanism
Thanks for this question. In this work we borrow the terminology of an ‘ordering ID’ from Prakash et al. [2] to mean this high-level list position (first item, second item, third item) that you describe. The positional mechanism is the retrieval process that uses these OIs, and the lexical mechanism uses the semantic key. To further explain the example given in Figure 2, at the items in the list the ordering IDs would be stored (1, 2, 3, 4) in both the counterfactual and original prompt, but since we are patching the pointer from the counterfactual prompt, which is a pointer to the first item, the positional mechanism corresponds to Box B.
We hope these responses clarify our contributions and address your concerns. Please let us know if there are any remaining questions or points that require further clarification, we’d be happy to elaborate.
References