acceptodds
Under review as a conference paper at ICLR 2027

RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings

14 views

Abstract

Transformers process tokens without any inherent notion of order, making positional encoding a fundamental requirement rather than an architectural refinement. Rotary Position Embedding (RoPE) has become the default positional encoding in modern language models, yet it is heavily biased toward nearby tokens. Existing alternatives have been evaluated under different settings, leaving the literature fragmented and without a clear replacement. We bring structure to this landscape by examining a specific weakness of RoPE: its slow frequency bands, whose wavelengths exceed the training context and expose models to unseen angles during extrapolation. We therefore introduce DaRoPE, which preserves standard RoPE on the fast bands but replaces absolute position on the slow bands with bounded coordinates learned from contextual representations. Therefore, the slow-band geometry depends on the data rather than only on positional distance. We compare representative encodings under matched conditions across synthetic tasks, symbolic music, genomics, neural signals, and language models spanning 124M to 50B parameters. Across these experiments, DaRoPE leads on non-text benchmarks, mitigates recency bias, while remaining best or on par in language modeling and length extrapolation. Moreover, the learned coordinates also make the mechanism interpretable, revealing how attention layers leverage contextual information beyond token distance. Together, these results support DaRoPE as the best overall default among the evaluated methods, when there is no domain-specific reasons to prefer another.

Primary area:
foundation or frontier models, including LLMs
Keywords:
positional encoding, rotary position embeddings, long-context modeling, length extrapolation, llm, extrapolation, sequence modeling, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kLK4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  • DaRoPE preserves RoPE's fast bands and replaces token positions in slow bands with learned coordinates that are context- and position-dependent.
  • The most competitive prior work, HoPE, removes rotation in slow bands. DaRoPE builds on this idea and further exploits the slow bands with an efficient content-dependent scheme.
  • Experiments cover language modeling, retrieval probes, synthetic tasks, and non-text tasks. The language-related experiments include dense model from 124M to 1B parameters and a 50B-A2.57B MoE model.
  • DaRoPE preserves model quality and HoPE-like extrapolation while improving performance on retrieval, non-text tasks, and some long-context tasks.
Critical strengths
  • DaRoPE approach for slow-bands is content-dependent but cheap (adds only a few parameters and linear with context length). Table 1 shows the results.

  • DaRoPE preserves in-domain performance, shows comparable extrapolation performance to HoPE, and demonstrates substantial retrieval improvements (Figure 2) within the training context. This shows DaRoPE ability to recovers distant content without being overwritten by what with smallest distance.

Critical weaknesses
  • I think the evidence is insufficient to support the claim that DaRoPE is the best overall default. First, language-modeling quality and extrapolation are largely tied with HoPE for both dense and large MoE models (Figure 1 and Table 2), although DaRoPE improves on some long-context benchmarks (Table 3). Second, the controlled retrieval tasks, where DaRoPE shows substantial gains, are evaluated only on 1B models; whether these gains persist at the 50B MoE scale remains an open question. Third, the improvements over HoPE on non-text tasks are small, and comparable tuning opportunities are not documented, as discussed below.

  • The tuning comparison between DaRoPE and the baselines needs clarification. First, Appendix C reports an additional validation sweep for DaRoPE on non-text tasks, including changing MAESTRO's slow-band count from 17 to 16, without documenting a comparable search for HoPE. Given the small non-text improvements over HoPE, this makes it difficult to separate the benefit of the learned coordinate from configuration selection. Second, both DaRoPE and HoPE use the same base within each experiment, their optimal bases (hyper parameter) need not to be the same. The paper does not report whether the advantage persists when each method's base is selected independently on validation data. With comparable tuning opportunities, baselines such as HoPE might achieve better results and narrow the reported gap.

Questions for authors
  1. Could the authors clarify the validation search spaces and tuning budgets for DaRoPE and HoPE? How does DaRoPE perform on MAESTRO with the canonical 17 slow bands, and how does HoPE perform under the same search budgets? Does HoPE benefit from the hyperparameter sweep such as base?

  2. Figure 2 reports retrieval results only for the 1B models. Could the authors provide results for 50B MoE models to show whether the retrieval advantage over HoPE persists at that scale?

  3. Related Work (Section 6) cites CARoPE, which also uses head-wise, input-dependent rotary. Could the authors clarify the expected advantages of DaRoPE over CARoPE? Were any direct comparisons performed?

  4. Minor typos and grammatical issues:

  • Line 69 "motivates a"
  • Line 153 "role,"
  • Line 264 "More details can be ..."
2 comments

Thank you for the positive review. We have fixed the typos and toned down the claims of the abstract and conclusion, to better reflect that DaRoPE is tied with HoPE for language modelling. Here are the answers to your questions:

1. Tuning. We agree that this is a needed clarification. The revised PDF reports the searches (Appendix C). All selections use validation loss only.

  • Search spaces and budgets. The shared architecture and optimizer (depth, width, batch, learning rate, dropout, weight decay) were selected with RoPE-10k (48 runs) and used unchanged for every method. DaRoPE then received 21 runs of its own (αh\alpha_h initialization, slow/fast frontier, whw_h scale). For PoPE, we fixed a data-loader bug of the official code (finite loader, partial evaluation); with this fix PoPE improves on its published results (JSB 0.482 vs. 0.489; MAESTRO 1.440 vs. 1.486).
  • 17 slow bands on MAESTRO. In the paper, DaRoPE (16 bands) reaches 1.4180 vs. 1.4289 for HoPE. With 17 bands, DaRoPE reaches 1.4195, still ahead of HoPE. The split rule θj<2π/L\theta_j < 2\pi/L needs no tuning, so the revised PDF now uses it on every dataset. We thank the reviewer for pointing out this inconsistency.
  • Base for HoPE. The base is HoPE's only hyperparameter. At its best base (100k, among 10k, 100k, 500k and 2M), HoPE reaches 1.4226 on MAESTRO, still behind DaRoPE at the shared base (1.4195). On JSB and Sleep-EDF, validation selects the shared base for both methods; on HRG, all bases give the same test NLL within 0.001 for both methods.

2. Retrieval at 50B. We thank the reviewer for this question. We ran both probes on the 50B models (same 800 instances per setting as at 1B). Both models are much stronger than at 1B, and the comparison is more balanced: DaRoPE leads on return-from-digression up to 512 tokens (+5 to +18 points), while HoPE leads beyond 1k tokens and on key–value recall (−12 points on average). Scale itself reduces the recency bias that these probes measure (HoPE goes from 16.8% to 71.5% after a 2,048-token digression); DaRoPE already provides this benefit at 1B (88.1%).

This is consistent with the paper's 50B synthetic long-context benchmarks (LongBench synthetic, RULER, BABILong), where neither method dominates. On language, DaRoPE is on par with HoPE in domain (54.7 vs. 54.3), and its edge appears on natural long-context tasks: LongBench multi-document QA (21.1 vs. 14.6) and RepoBench (+5.0 edit similarity, +3.1 exact match). The revised PDF now states that the probe advantage is clear at 1B and smaller at 50B (Appendix B).

3. CARoPE. We trained CARoPE in our 124M and 1B setups. It matches DaRoPE within the training length but fails beyond it (perplexity at 12–16k: 228 vs. 57 for DaRoPE), and at 1B it falls behind on retrieval (key–value recall with 256 pairs: at most 2.5% vs. 30.5%). CARoPE's angles sum content-dependent steps over all previous tokens and leave the trained range beyond the training length; DaRoPE's coordinate is bounded and computed per token. Since CARoPE cannot extrapolate, it cannot match DaRoPE where extrapolation matters. Details are in our answer to Reviewer 2zuq.

Thanks for the clarification and the additional experiments. I will keep my positive assessment and maintain my current score.

I think the content-dependent design of RoPE's slow bands is worth exploring, and could be useful at larger scale, but the current evidence is not enough for me to raise my score. The 50B retrieval results indeed show that the advantage over HoPE depends on the task and model scale (HoPE wins at every tested key-value recall setting at 50B). Independently tuning HoPE’s base also substantially narrows the gap on MAESTRO. These results support my original concern about how broadly DaRoPE improves over HoPE.

Reviewer_XVpyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method that replaces absolute positional information in the low-frequency components of RoPE with data-dependent positional representations. The authors introduce a new positional encoding method called Data-aware RoPE (DaRoPE). Conventional RoPE assigns different rotation angles to tokens based on their positions. However, in real sequences, positional proximity does not necessarily correspond to semantic or contextual relevance. DaRoPE handles the frequency components of RoPE as follows:

  • High-frequency components (Fast bands): Preserve the standard RoPE mechanism to encode positional relationships between tokens.
  • Low-frequency components (Slow bands): Replace absolute positional indices with learned coordinates derived from the contextual representations of individual tokens.

As a result, semantically or contextually related tokens can be assigned similar rotation angles in the low-frequency components, even when they are far apart in the sequence.

Critical strengths

DaRoPE may be effective for long-context extrapolation, retrieval tasks, and sequence modeling across various domains, including music, genomics, and EEG signals.

Critical weaknesses

Despite its provocative title, I find the proposed idea to have relatively limited novelty and impact, and I am not fully convinced by the experimental results. I am skeptical that DaRoPE will be adopted as widely as the authors suggest. In particular, I believe the claims in the abstract and conclusion should be substantially toned down. The current experimental evidence does not sufficiently support the authors' strong claims regarding the general applicability and superiority of DaRoPE. Given the limited novelty and concerns about the experimental evaluation, I lean toward rejection.

1. The authors evaluate DaRoPE using a 50B-parameter model. However, the reported performance appears substantially lower than what might be expected from modern language models of a similar total parameter scale. For example, the model achieves only 29.9% accuracy on GSM8K. In contrast, several existing models with substantially fewer parameters have reported much higher performance on this benchmark. Even Llama-3-8B has been reported to achieve substantially better results under certain evaluation settings. Although the proposed model has 50B total parameters, its overall benchmark performance appears relatively weak. This raises concerns about whether the model is sufficiently capable to provide convincing evidence for the effectiveness of the proposed positional encoding method. I acknowledge that the model is a mixture-of-experts architecture with only 2.57B active parameters per token, and that benchmark performance depends on training data, instruction tuning, and evaluation protocols. Nevertheless, the authors should explain why the reported performance is relatively low and provide stronger evidence that their conclusions generalize to more capable language models.

2. Several closely related ideas have already been explored in the literature:

  • CoPE: Introduces content-dependent positional representations.
  • CARoPE and Selective RoPE: Introduce input-dependent modifications to rotary positional representations.
  • DAPE: Explores data-adaptive positional encoding to improve length extrapolation.
  • HoPE: Removes problematic low-frequency components of RoPE to improve long-context modeling.

Given these existing studies, the main novelty of DaRoPE appears to lie in combining previously explored ideas: applying data-dependent positional representations specifically to the low-frequency components of RoPE while maintaining computational efficiency. While this combination may offer practical benefits, I do not find it to be a sufficiently substantial conceptual advance to justify the strong claims made throughout the paper. The authors repeatedly position DaRoPE as a strong general-purpose replacement for existing positional encoding methods. However, the proposed mechanism appears to be a relatively straightforward combination of existing approaches, rather than a fundamentally new positional encoding paradigm.

3. Although the authors cite several relevant studies, the experimental comparisons are limited. Among the closely related methods mentioned above, only HoPE and CoPE are included in the experiments, and CoPE is evaluated only on synthetic tasks. In particular, CARoPE and Selective RoPE are highly relevant because they also investigate input-dependent rotary positional representations. However, neither is included in the experimental comparisons. Without comparisons against these closely related approaches, it is difficult to determine whether DaRoPE provides meaningful advantages beyond existing data-dependent positional encoding methods.

Questions for authors

See the Weaknesses.

2 comments

Thank you for your review. We agree that the claims of the abstract and conclusion should be toned down, and we have done so in the revised PDF (point 3). We address the main points below.

1. Benchmark performance at 50B. We thank the reviewer for raising this point, which deserves a clearer explanation in the paper. The model has 2.57B active parameters and was trained on 617B tokens (9.5×10219.5\times10^{21} FLOPs), without instruction tuning. Frequently quoted GSM8K scores, such as the 79.6% of Llama-3-8B-Instruct, come from instruction-tuned models with much larger budgets. Evaluated as base models in the same harness, our model compares well with open models trained with more compute:

Base model (same harness)Compute (× ours)GSM8KHumanEvalMBPP
DaRoPE MoE (ours)1×29.937.850.6
OLMoE-1B-7B~4×12.714.018.8
DCLM-7B~11×44.522.032.2
Llama 3 8B~75×56.337.848.6

The comparison between encodings is matched (same architecture, data and budget): DaRoPE keeps in-domain quality (54.7 vs. 54.3) and improves the long-context tasks that combine evidence across documents (Table 3). We have added this comparison to Appendix B.

2. Related methods. We thank the reviewer for this list. All five methods are discussed in our Related Work; HoPE is our main baseline throughout, and CoPE is evaluated where its cost allows. We have now added the other three experimentally.

CoPE and DAPE compute on the full L×LL\times L attention matrix, which limits their scale (1B, one H100, compiled implementations, ktokens/s):

4k16k32kMemory at 8k (GB)
DaRoPE146.195.364.54.1
CoPE35.710.7out of memory13.0
DAPE24.1out of memoryout of memory29.9

CARoPE and Selective RoPE accumulate content-dependent angles over all previous tokens. Beyond the training length, these angles leave the trained range and perplexity explodes, as with RoPE (124M, PG-19):

0–4k4–8k8–12k12–16k
RoPE-10k442,2595,17814,172
DaRoPE44424757
CARoPE4545111228
Selective RoPE469449,6257,327

Selective RoPE's authors note that "incorporating RoPE is notoriously detrimental to the length-extrapolation capabilities of sequence models [...] we consider it to be out of the scope of our research." Neither CARoPE nor Selective RoPE therefore extrapolates beyond the training length, a key property of DaRoPE.

What DaRoPE adds is a single property: the phase between two tokens depends only on these two tokens and stays bounded at any length. This is what lets it keep RoPE's cost and HoPE's extrapolation. It also counters recency bias in retrieval: at 1B, DaRoPE picks the distant answer over a recent decoy 88.1% of the time after a 2,048-token digression (HoPE 16.8%), and reaches 30.5% key–value recall with 256 pairs (at most 7.25% for every other encoding). We see the breadth of the matched comparison (six encodings, five domains, up to a 50B mixture of experts) as a contribution of the paper in itself.

3. Claims. The abstract and conclusion now state that DaRoPE matches HoPE on language modelling and extrapolation, counters recency bias in retrieval, and has the best mean non-text rank, at near-RoPE cost.

We hope these results address the reviewer's concerns, and we remain open to further clarifications or additional comparisons if the reviewer deems them necessary.

Dear Reviewer XVpy, As the discussion closes today, we would value your feedback. We addressed your three points in the revised PDF: claims toned down, CARoPE, Selective RoPE and DAPE added as baselines (none both scales and extrapolates), and a base-model comparison showing that our 50B model outperforms OLMoE with 4× less compute and matches Llama 3 8B on code with 75× less. As the other two reviewers now lean toward acceptance, we would be grateful if you could tell us whether these results address your concerns and, if so, reconsider your score.

Reviewer_2zuqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces Data-aware Rotary Position Embedding (DaRoPE), a positional encoding that retains standard Rotary Position Embedding (RoPE) on high-frequency (fast) bands with wavelength λj≤L\lambda_j \le L while replacing the token index mm on low-frequency (slow) bands with a bounded, per-head coordinate ch(xm)=L⋅σ(wh⊤xm+αhβm)∈[0,L]c_h(x_m) = L \cdot \sigma(w_h^\top x_m + \alpha_h \beta_m) \in [0, L] predicted from the token representation xmx_m. Because this slow-band modification changes only per-token rotation angles, DaRoPE preserves O(L)O(L) complexity in sequence length LL and FlashAttention compatibility. The authors evaluate DaRoPE against RoPE, NoPE, HoPE, YaRN, CoPE, and PoPE across synthetic tasks, symbolic music, genomics, electroencephalography (EEG), and language models from 124M parameters to a 50B-parameter Mixture-of-Experts (MoE) model.

Critical strengths
  1. Repurposing RoPE's slow bands with a per-head logistic-sigmoid coordinate ch(xm)∈[0,L]c_h(x_m) \in [0, L] is simple and adds dmodel+1d_{\mathrm{model}} + 1 scalars per head, which is fewer than 0.3% of model parameters. Furthermore, the per-token rotary formulation maintains FlashAttention compatibility and achieves 59.8 ktokens/s prefill throughput at 1B parameters, compared with 61.7 ktokens/s for RoPE-10k and 4.3 ktokens/s for CoPE (Table 1).
  2. The controlled evaluation spans five domains and multiple model scales up to a 50B MoE model (2.57B active parameters). At the 1B scale, DaRoPE reduces recency bias and retrieval interference relative to RoPE and HoPE (Figure 2) while matching HoPE on length extrapolation from L=4096L = 4096 to 16,38416,384 tokens on PG-19 (Figure 1).
Critical weaknesses
  1. [Resolved in revision] The initial submission cited Context-Aware Rotary Position Embedding (CARoPE) in Related Work without experimental comparison. In the revised manuscript, Appendix A.4 (Tables 7 and 8) compares DaRoPE against CARoPE and Selective RoPE under matched 124M and 1B setups. Because CARoPE accumulates content-dependent phase steps over all preceding tokens, its angles leave the trained range beyond the training length L=4096L = 4096, causing 12k to 16k PG-19 perplexity to rise to 228 (or 6,400 with a 10 times lower projection learning rate) versus 57 for DaRoPE (Table 7). At 1B parameters, CARoPE also reaches at most 2.5% accuracy on K=256K = 256 key–value recall versus 30.5% for DaRoPE (Table 8).
  2. [Resolved in revision] The initial formulation ablations in Appendix Table 5 did not isolate the benefit of the content term wh⊤xmw_h^\top x_m from that of bounding the slow-band coordinate in [0,L][0, L] with the learnable positional prior αhβm\alpha_h \beta_m. In the revision, Appendix D.1 (Tables 14 and 15) evaluates DaRoPE trained without wh⊤xmw_h^\top x_m, namely ch(m)=L⋅σ(αhβm)c_h(m) = L \cdot \sigma(\alpha_h \beta_m). Removing the content term degrades 124M extrapolation perplexity at 8k to 16k tokens from 52 to 1,060 and reduces 1B key–value recall at K=256K = 256 pairs from 30.5% to 5.9% (Table 14). Moreover, Table 15 shows that when the content term wh⊤xmw_h^\top x_m is present, training drives the median prior weight αh\alpha_h from 0.5 down to 0.003 at 124M and 0.009 at 1B, leaving 99.6% to 99.8% of the slow-band coordinate variance to the content term.
  3. On the 50B MoE retrieval probes added in Appendix B.1 (Table 12), the advantage over HoPE is smaller than at 1B: DaRoPE leads on return-from-digression up to D=512D = 512 tokens (79.3% versus 67.8%), whereas HoPE leads at D=2048D = 2048 tokens (71.5% versus 60.6%) and on K=256K = 256 key–value recall (52.6% versus 22.8%). However, the authors have appropriately toned down the broad claims in the revised Abstract and Conclusion.
Questions for authors
  1. Can the authors provide a matched experimental comparison with CARoPE at 124M and 1B parameters? [Answered in Appendix A.4, Tables 7 and 8.]
  2. How does DaRoPE perform when the content term wh⊤xmw_h^\top x_m is removed while retaining the logistic-sigmoid bound in [0,L][0, L] and the learnable positional prior αhβm\alpha_h \beta_m? [Answered in Appendix D.1, Tables 14 and 15.]
1 comment

Thank you for your review and for the two precise questions. In brief, CARoPE fails beyond its training length while DaRoPE needs its content term to extrapolate and to retrieve under interference. Both experiments are included in the revised PDF.

1. CARoPE. We thank the reviewer for pointing out this baseline we had overlooked. We trained CARoPE in our 124M setup with the recipe shared by all encodings (peak learning rate 3×10−33\times10^{-3}, context 4096; the CARoPE paper used 6×10−46\times10^{-4} and context 512). In its published form, training is unstable near the learning-rate peak, then recovers; with a 10× lower learning rate on CARoPE's own projection, it trains as smoothly as RoPE.

Both versions fail beyond the training length (perplexity):

0–4k4–8k8–12k12–16k
DaRoPE44424757
CARoPE, shared recipe4545111228
CARoPE, lower learning rate452914,1466,400

CARoPE's angle at position m is a sum of content-dependent steps over all previous tokens, so it keeps growing and, beyond the training length, reaches values never seen in training. CARoPE's own evaluation stops at twice its training length. DaRoPE's slow-band coordinate is bounded and computed per token. At 1B, CARoPE also falls behind on retrieval (key–value recall with 256 pairs: at most 2.5% vs. 30.5%). We have added this extra baseline to our context extrapolation experiments.

2. Content term. We trained DaRoPE without the content term (bound and positional prior kept). The content term matters beyond extrapolation: at 1B, it also drives retrieval under interference.

124M, perplexity 0–4k124M, perplexity 8–16k1B, key–value recall (256 pairs)1B, return after 2,048 tokens
DaRoPE445230.5%88.1%
Trained without content441,0605.9%80.5%

Without content, the model uses its slow bands as a compressed position clock. Beyond 4k this clock stops, so every distant token gets the same slow-band angle, a configuration never seen in training. With content, training turns the positional prior off at every scale (αh\alpha_h weights the prior, whw_h the content; all heads start at αh\alpha_h = 0.5):

124M350M1B124M, no content
Median αh\alpha_h0.0030.0070.0090.55
Heads with ∣αh∣<0.05\lvert\alpha_h\rvert < 0.0584%80%77%4%
Coordinate variance from content (median head)99.8%99.6%99.6%0%

Editing the trained model confirms it: setting αh\alpha_h to 0 changes nothing (perplexity 44 / 52 at 0–4k / 8–16k), while setting whw_h to 0 breaks it (61 / 227).

Comments on the paper

1 comment

We thank the reviewers for their comments. Overall, they find the method simple and practical at near-RoPE cost, the retrieval gains compelling, and the evaluation broad across domains and scales.

The open questions were about additional comparisons, which we ran:

  • CARoPE and Selective RoPE, trained in our 124M and 1B setups (all reviewers);

  • the cost of CoPE and DAPE at 1B (XVpy);

  • DaRoPE without its content term (2zuq);

  • the full tuning protocol, with HoPE's base and slow-band count tuned (kLK4);

  • both retrieval probes at 50B (kLK4).

    DaRoPE remains ahead of all requested additional baselines. These results are included in the revised PDF (changes in blue).

To our knowledge, the paper is the broadest matched comparison of positional encodings to date: six encodings under identical conditions, 15 synthetic tasks, language models from 124M to 1B and a 50B mixture of experts, two retrieval probes, five long-context suites, and four non-text datasets (music, genomics, EEG). In comparison, CoPE, CARoPE and DAPE were each evaluated on text or synthetic tasks only, with models of at most 124M (CoPE, CARoPE) or 350M parameters (DAPE). Following the reviewers, we have toned down the claims of the abstract and conclusion. While we do not claim DaRoPE is the single best PE method, we believe it is a useful addition because it repurposes RoPE's slow bands instead of discarding them: each token gets a bounded, content-driven coordinate that keeps RoPE's cost and extrapolation while countering recency bias.

We are ready to run further analyses during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.