- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- DaRoPE preserves RoPE's fast bands and replaces token positions in slow bands with learned coordinates that are context- and position-dependent.
- The most competitive prior work, HoPE, removes rotation in slow bands. DaRoPE builds on this idea and further exploits the slow bands with an efficient content-dependent scheme.
- Experiments cover language modeling, retrieval probes, synthetic tasks, and non-text tasks. The language-related experiments include dense model from 124M to 1B parameters and a 50B-A2.57B MoE model.
- DaRoPE preserves model quality and HoPE-like extrapolation while improving performance on retrieval, non-text tasks, and some long-context tasks.
Critical strengths
-
DaRoPE approach for slow-bands is content-dependent but cheap (adds only a few parameters and linear with context length). Table 1 shows the results.
-
DaRoPE preserves in-domain performance, shows comparable extrapolation performance to HoPE, and demonstrates substantial retrieval improvements (Figure 2) within the training context. This shows DaRoPE ability to recovers distant content without being overwritten by what with smallest distance.
Critical weaknesses
-
I think the evidence is insufficient to support the claim that DaRoPE is the best overall default. First, language-modeling quality and extrapolation are largely tied with HoPE for both dense and large MoE models (Figure 1 and Table 2), although DaRoPE improves on some long-context benchmarks (Table 3). Second, the controlled retrieval tasks, where DaRoPE shows substantial gains, are evaluated only on 1B models; whether these gains persist at the 50B MoE scale remains an open question. Third, the improvements over HoPE on non-text tasks are small, and comparable tuning opportunities are not documented, as discussed below.
-
The tuning comparison between DaRoPE and the baselines needs clarification. First, Appendix C reports an additional validation sweep for DaRoPE on non-text tasks, including changing MAESTRO's slow-band count from 17 to 16, without documenting a comparable search for HoPE. Given the small non-text improvements over HoPE, this makes it difficult to separate the benefit of the learned coordinate from configuration selection. Second, both DaRoPE and HoPE use the same base within each experiment, their optimal bases (hyper parameter) need not to be the same. The paper does not report whether the advantage persists when each method's base is selected independently on validation data. With comparable tuning opportunities, baselines such as HoPE might achieve better results and narrow the reported gap.
Questions for authors
-
Could the authors clarify the validation search spaces and tuning budgets for DaRoPE and HoPE? How does DaRoPE perform on MAESTRO with the canonical 17 slow bands, and how does HoPE perform under the same search budgets? Does HoPE benefit from the hyperparameter sweep such as base?
-
Figure 2 reports retrieval results only for the 1B models. Could the authors provide results for 50B MoE models to show whether the retrieval advantage over HoPE persists at that scale?
-
Related Work (Section 6) cites CARoPE, which also uses head-wise, input-dependent rotary. Could the authors clarify the expected advantages of DaRoPE over CARoPE? Were any direct comparisons performed?
-
Minor typos and grammatical issues:
- Line 69 "motivates a"
- Line 153 "role,"
- Line 264 "More details can be ..."
2 comments
Thanks for the clarification and the additional experiments. I will keep my positive assessment and maintain my current score.
I think the content-dependent design of RoPE's slow bands is worth exploring, and could be useful at larger scale, but the current evidence is not enough for me to raise my score. The 50B retrieval results indeed show that the advantage over HoPE depends on the task and model scale (HoPE wins at every tested key-value recall setting at 50B). Independently tuning HoPE’s base also substantially narrows the gap on MAESTRO. These results support my original concern about how broadly DaRoPE improves over HoPE.
Thank you for the positive review. We have fixed the typos and toned down the claims of the abstract and conclusion, to better reflect that DaRoPE is tied with HoPE for language modelling. Here are the answers to your questions:
1. Tuning. We agree that this is a needed clarification. The revised PDF reports the searches (Appendix C). All selections use validation loss only.
2. Retrieval at 50B. We thank the reviewer for this question. We ran both probes on the 50B models (same 800 instances per setting as at 1B). Both models are much stronger than at 1B, and the comparison is more balanced: DaRoPE leads on return-from-digression up to 512 tokens (+5 to +18 points), while HoPE leads beyond 1k tokens and on key–value recall (−12 points on average). Scale itself reduces the recency bias that these probes measure (HoPE goes from 16.8% to 71.5% after a 2,048-token digression); DaRoPE already provides this benefit at 1B (88.1%).
This is consistent with the paper's 50B synthetic long-context benchmarks (LongBench synthetic, RULER, BABILong), where neither method dominates. On language, DaRoPE is on par with HoPE in domain (54.7 vs. 54.3), and its edge appears on natural long-context tasks: LongBench multi-document QA (21.1 vs. 14.6) and RepoBench (+5.0 edit similarity, +3.1 exact match). The revised PDF now states that the probe advantage is clear at 1B and smaller at 50B (Appendix B).
3. CARoPE. We trained CARoPE in our 124M and 1B setups. It matches DaRoPE within the training length but fails beyond it (perplexity at 12–16k: 228 vs. 57 for DaRoPE), and at 1B it falls behind on retrieval (key–value recall with 256 pairs: at most 2.5% vs. 30.5%). CARoPE's angles sum content-dependent steps over all previous tokens and leave the trained range beyond the training length; DaRoPE's coordinate is bounded and computed per token. Since CARoPE cannot extrapolate, it cannot match DaRoPE where extrapolation matters. Details are in our answer to Reviewer 2zuq.