- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper extends RoPE by adding a learnable scaling parameter per frequency band. The core motivation is to avoid undesirable gradient dynamics near frequencies of zeros, thereby allowing better conditioning compared to vanilla RoPE and p-RoPE. The experimental setup is quite robust, ensuring each tested model scale meets the Chinchilla scaling laws, and tests the efficacy of this learnable RoPE at 52M to 2.5B parameter scales. The experiments show that the proposed method LeRoPE consistently shows the lowest perplexity across scales compared to vanilla RoPE and p-ROPE, contributing better token-efficiency during pre-training. Also, it conducts an analysis to examine what the proposed LeRoPE learns.
Critical strengths
This is a good, well-written paper on extending RoPE to mitigate its drawback on undesirable gradient dynamics. The proposed method is quite simple and must be easy to use, adding a negligible number of parameters. Specifically, it adds a one additional scalar parameter per frequency band in RoPE, re-scaling each frequency dynamically. Yet, it contributes to better token-efficiency during pre-training, requiring 3.4% less compute compared to vanilla RoPE.
Critical weaknesses
- While I acknowledge the computational resource constraints, experiments presented in this paper are mostly single-run, except for 217M models. Given the difference between the proposed method (LeRoPE) and other methods like p-RoPE is quite small in terms of both validation PPL in Tables 1 & 2 and in-distribution loss gap in Figure 1 (b), I am a bit concerned about the statistical significance of the proposed method. If possible, the paper should run at least two more runs each for each method (e.g., RoPE, p-RoPE, and the proposed method) at the largest tested scales: 2.5B using different random seeds.
- The evaluation domains remain foundational, limiting to language modeling and needle-in-a-haystack. It remains unclear whether the benefit observed in the paper translates into general capabilities measured through general benchmarks (e.g., ARC, HellaSwag, WinoGrande, OpenBookQA etc.).
Questions for authors
I don’t understand why the paper explicitly mentions that “the main focus of our work is on in-distribution performance” in L409-412. As this is the language modeling work, I think it is really important to investigate how any token-efficiencies observed on validation data translate into performance on general benchmarks. This is a common approach taken in previous pre-training work:
- Instruction Pre-Training: Language Models are Supervised Multitask Learners (Cheng et al., EMNLP 2024)
- Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (Yamaguchi et al., ACL 2026)
While in-distribution performance is also important, I think this is more like a prerequisite in the evaluation of the proposed method.
3 comments
Thanks for conducting additional experiments!
LeRoPE reaches a lower loss than the baselines on all 15 paired runs and at a given model size every LeRoPE run is better than the best RoPE-10k run.
This is promising indeed.
The standard benchmarks at this scale such as HellaSwag, ARC, PIQA etc. mostly target knowledge acquisition so positional encodings affect these less so than other architectural changes. This is especially true in our case, in which we use the same data ordering for all models of the same size.
I understand. Nonetheless, if the submission includes this discussion in the paper and some of the results (including task-breakdowns), that would be great.
To sum, while the response has faithfully addressed my concerns, it remains unclear how the efficacy of the proposed method translates into general capabilities (especially at scale in terms of data and parameter size). Therefore, I keep the current overall rating as is, while raising the soundness score to 3. Nonetheless, I'd like to put a note here that my stance on this paper is not a "weak accept"; rather I am more on "accept" (but I can't say it's "clear" accept.)
We thank the reviewer for raising the soundness score. We will add the zero-shot and 5-shot per-task results to the appendix.
general capabilities
We agree that how LeRoPE’s gains carry over to larger models is an important question. We include LongPPL (Fang et al., ICLR 2025) evaluations for the 2.5B models.
LongPPL uses an evaluator model to identify key tokens in a given long context by contrasting their likelihood with a short local context window with the full preceding context. It selects tokens which have a delta from the short and full context above a given threshold and that can be accurately predicted given the full context. To evaluate a given model it calculates the perplexity/loss of just these key tokens. The authors found that this metric has a much stronger correlation than perplexity with downstream long context performance on LongBench, LongEval and RULER.
The original LongPPL score is calculated on documents from GovReport and uses a sliding window context of 4k to contrast against the full document. In this setting we set the full context length to be 16k. We include both the original LongPPL setup on GovReport as well as a shorter version with 512 window with 2k total context on C4 validation documents. We use Llama-3.1-8B to select the key tokens, which the authors find is a sufficient evaluator.
We report the standard perplexity as well as the perplexity just over the key tokens for both settings below. We evaluating on the GovReport documents at 16k we apply YaRN. We observe that while the raw perplexity difference may be small, LeRoPE clearly outperforms the other methods on key tokens. This holds true both for sequences at the training length and for sequences at 8x the training length.
| 2.52B | C4, 2K: key-token PPL (512 window) | C4, 2K: PPL | GovReport, 16K: LongPPL | GovReport, 16K: PPL |
|---|---|---|---|---|
| RoPE | 4.60 | 9.74 | 13.26 | 7.60 |
| p-RoPE | 4.39 | 9.73 | 13.00 | 7.51 |
| LeRoPE | 3.91 | 9.69 | 7.11 | 7.57 |
We thank the reviewer for their constructive comments on our submission. We address your concerns below.
| Seed Noise
We conduct 2 additional seed runs for the 1.3B models for RoPE-10k and LeRoPE as well as additional seed runs for p-RoPE at 217M and 608M. LeRoPE reaches a lower loss than the baselines on all 15 paired runs and at a given model size every LeRoPE run is better than the best RoPE-10k run. We present the results below:
We continue to observe that the within-method standard deviation across seed runs as well as the standard deviation of the paired difference between methods continues to decrease with scale.
While we are not able to compute mutli-seed 2.5B runs due to compute restrictions, we believe this supports the statistical significance of our method’s gains. If we assume the same seed-to-seed loss variance as we observed in our 1.3B models and extend it to the 2.5B models we get a 95% CI of 0.0015–0.0051 for the loss gap. This equates to 1.5 to 5.3% CI for the compute multipliers. We note that this is a conservative estimate, as we observed seed variance to be slightly decreasing with scale while this estimate uses the variance from a smaller model. Even in this case a compute multiplier of 0 is not included in our interval.
| In-distribution loss
We apologize for the confusion this may have caused. Here, when we say in-distribution performance we refer to sequence length, not domain. Our primary focus was on developing a positional encoding that works better for sequences of lengths seen during training rather than focusing on extrapolation. Our philosophy behind this is models are rarely served at context lengths much past those seen during training (frequently after context extension rounds).
While we agree on the importance of task-level generalization out-of-distribution, it is difficult to measure this on models of our scale and token counts. The standard benchmarks at this scale such as HellaSwag, ARC, PIQA etc. mostly target knowledge acquisition so positional encodings affect these less so than other architectural changes. This is especially true in our case, in which we use the same data ordering for all models of the same size.
Nevertheless, we include zero-shot and 5-shot evaluations for our 2.5B models on ARC-E, HellaSwag, PIQA, WinoGrande and SciQ following the default settings provided in lm-eval-harness. We present the averaged results below:
We additionally conduct these evals on our 3 seed 1.3B models for RoPE and LeRoPE to estimate seed noise. The within-method standard deviation of the five-task average is 0.19 for zero-shot and 0.65 for 5-shot. If it holds at 2.52B, two single runs must differ by more than 0.76 and 2.54 points, respectively, to be distinguishable at 95% CIs. These results are within noise across all three positional encodings for both the 0-shot and 5-shot setting.