acceptodds
Under review as a conference paper at ICLR 2027

LeRoPE: Learnable RoPE Frequencies Improve Language Modeling

Abstract

Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in RoPE typically follow a geometric sequence specified by a fixed base-frequency hyperparameter. Prior work has improved performance by either increasing this parameter to slow rotation or by applying RoPE to only a subset of QK dimensions. In this work we modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters. We validate Learned RoPE by training a ladder of language models from scratch, ranging from 52M to 2.5B parameters. We observe and analyze the emergence of a high-norm, positional LeRoPE band. LeRoPE consistently outperforms RoPE and partial RoPE across all scales, with RoPE requiring 3.4% more compute (FLOPs) to match LeRoPE at the largest scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
positional encodings, rope, language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7xPQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper extends RoPE by adding a learnable scaling parameter per frequency band. The core motivation is to avoid undesirable gradient dynamics near frequencies of zeros, thereby allowing better conditioning compared to vanilla RoPE and p-RoPE. The experimental setup is quite robust, ensuring each tested model scale meets the Chinchilla scaling laws, and tests the efficacy of this learnable RoPE at 52M to 2.5B parameter scales. The experiments show that the proposed method LeRoPE consistently shows the lowest perplexity across scales compared to vanilla RoPE and p-ROPE, contributing better token-efficiency during pre-training. Also, it conducts an analysis to examine what the proposed LeRoPE learns.

Critical strengths

This is a good, well-written paper on extending RoPE to mitigate its drawback on undesirable gradient dynamics. The proposed method is quite simple and must be easy to use, adding a negligible number of parameters. Specifically, it adds a one additional scalar parameter per frequency band in RoPE, re-scaling each frequency dynamically. Yet, it contributes to better token-efficiency during pre-training, requiring 3.4% less compute compared to vanilla RoPE.

Critical weaknesses
  1. While I acknowledge the computational resource constraints, experiments presented in this paper are mostly single-run, except for 217M models. Given the difference between the proposed method (LeRoPE) and other methods like p-RoPE is quite small in terms of both validation PPL in Tables 1 & 2 and in-distribution loss gap in Figure 1 (b), I am a bit concerned about the statistical significance of the proposed method. If possible, the paper should run at least two more runs each for each method (e.g., RoPE, p-RoPE, and the proposed method) at the largest tested scales: 2.5B using different random seeds.
  2. The evaluation domains remain foundational, limiting to language modeling and needle-in-a-haystack. It remains unclear whether the benefit observed in the paper translates into general capabilities measured through general benchmarks (e.g., ARC, HellaSwag, WinoGrande, OpenBookQA etc.).
Questions for authors

I don’t understand why the paper explicitly mentions that “the main focus of our work is on in-distribution performance” in L409-412. As this is the language modeling work, I think it is really important to investigate how any token-efficiencies observed on validation data translate into performance on general benchmarks. This is a common approach taken in previous pre-training work:

  • Instruction Pre-Training: Language Models are Supervised Multitask Learners (Cheng et al., EMNLP 2024)
  • Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (Yamaguchi et al., ACL 2026)

While in-distribution performance is also important, I think this is more like a prerequisite in the evaluation of the proposed method.

3 comments

We thank the reviewer for their constructive comments on our submission. We address your concerns below.

| Seed Noise

We conduct 2 additional seed runs for the 1.3B models for RoPE-10k and LeRoPE as well as additional seed runs for p-RoPE at 217M and 608M. LeRoPE reaches a lower loss than the baselines on all 15 paired runs and at a given model size every LeRoPE run is better than the best RoPE-10k run. We present the results below:

ScaleRoPE, mean ± SDp-RoPE, mean ± SDLeRoPE, mean ± SDMargin, 3 seedsMean margin ± SD (95% CI)
217M2.9046 ± 0.00102.9036 ± 0.00142.8981 ± 0.0019RoPE − LeRoPE: 0.0057 / 0.0087 / 0.00520.0065 ± 0.0019 (0.0018–0.0112)
p-RoPE − LeRoPE: 0.0051 / 0.0049 / 0.00380.0046 ± 0.0007 (0.0029–0.0063)
608M2.6294 ± 0.00092.6284 ± 0.00092.6253 ± 0.0004RoPE − LeRoPE: 0.0051 / 0.0040 / 0.00310.0041 ± 0.0010 (0.0016–0.0066)
p-RoPE − LeRoPE: 0.0042 / 0.0031 / 0.00200.0031 ± 0.0011 (0.0004–0.0058)
1.34B2.4490 ± 0.00052.44812.4456 ± 0.0002RoPE − LeRoPE: 0.0031 / 0.0039 / 0.00330.0034 ± 0.0004 (0.0024–0.0045)
p-RoPE − LeRoPE: 0.0024 / — / ——

We continue to observe that the within-method standard deviation across seed runs as well as the standard deviation of the paired difference between methods continues to decrease with scale.

While we are not able to compute mutli-seed 2.5B runs due to compute restrictions, we believe this supports the statistical significance of our method’s gains. If we assume the same seed-to-seed loss variance as we observed in our 1.3B models and extend it to the 2.5B models we get a 95% CI of 0.0015–0.0051 for the loss gap. This equates to 1.5 to 5.3% CI for the compute multipliers. We note that this is a conservative estimate, as we observed seed variance to be slightly decreasing with scale while this estimate uses the variance from a smaller model. Even in this case a compute multiplier of 0 is not included in our interval.

| In-distribution loss

We apologize for the confusion this may have caused. Here, when we say in-distribution performance we refer to sequence length, not domain. Our primary focus was on developing a positional encoding that works better for sequences of lengths seen during training rather than focusing on extrapolation. Our philosophy behind this is models are rarely served at context lengths much past those seen during training (frequently after context extension rounds).

While we agree on the importance of task-level generalization out-of-distribution, it is difficult to measure this on models of our scale and token counts. The standard benchmarks at this scale such as HellaSwag, ARC, PIQA etc. mostly target knowledge acquisition so positional encodings affect these less so than other architectural changes. This is especially true in our case, in which we use the same data ordering for all models of the same size.

Nevertheless, we include zero-shot and 5-shot evaluations for our 2.5B models on ARC-E, HellaSwag, PIQA, WinoGrande and SciQ following the default settings provided in lm-eval-harness. We present the averaged results below:

2.52Bzero-shot Average5-shot Average
RoPE68.671.5
p-RoPE69.271.4
LeRoPE68.771.5

We additionally conduct these evals on our 3 seed 1.3B models for RoPE and LeRoPE to estimate seed noise. The within-method standard deviation of the five-task average is 0.19 for zero-shot and 0.65 for 5-shot. If it holds at 2.52B, two single runs must differ by more than 0.76 and 2.54 points, respectively, to be distinguishable at 95% CIs. These results are within noise across all three positional encodings for both the 0-shot and 5-shot setting.

Thanks for conducting additional experiments!

LeRoPE reaches a lower loss than the baselines on all 15 paired runs and at a given model size every LeRoPE run is better than the best RoPE-10k run.

This is promising indeed.

The standard benchmarks at this scale such as HellaSwag, ARC, PIQA etc. mostly target knowledge acquisition so positional encodings affect these less so than other architectural changes. This is especially true in our case, in which we use the same data ordering for all models of the same size.

I understand. Nonetheless, if the submission includes this discussion in the paper and some of the results (including task-breakdowns), that would be great.

To sum, while the response has faithfully addressed my concerns, it remains unclear how the efficacy of the proposed method translates into general capabilities (especially at scale in terms of data and parameter size). Therefore, I keep the current overall rating as is, while raising the soundness score to 3. Nonetheless, I'd like to put a note here that my stance on this paper is not a "weak accept"; rather I am more on "accept" (but I can't say it's "clear" accept.)

We thank the reviewer for raising the soundness score. We will add the zero-shot and 5-shot per-task results to the appendix.

general capabilities

We agree that how LeRoPE’s gains carry over to larger models is an important question. We include LongPPL (Fang et al., ICLR 2025) evaluations for the 2.5B models.

LongPPL uses an evaluator model to identify key tokens in a given long context by contrasting their likelihood with a short local context window with the full preceding context. It selects tokens which have a delta from the short and full context above a given threshold and that can be accurately predicted given the full context. To evaluate a given model it calculates the perplexity/loss of just these key tokens. The authors found that this metric has a much stronger correlation than perplexity with downstream long context performance on LongBench, LongEval and RULER.

The original LongPPL score is calculated on documents from GovReport and uses a sliding window context of 4k to contrast against the full document. In this setting we set the full context length to be 16k. We include both the original LongPPL setup on GovReport as well as a shorter version with 512 window with 2k total context on C4 validation documents. We use Llama-3.1-8B to select the key tokens, which the authors find is a sufficient evaluator.

We report the standard perplexity as well as the perplexity just over the key tokens for both settings below. We evaluating on the GovReport documents at 16k we apply YaRN. We observe that while the raw perplexity difference may be small, LeRoPE clearly outperforms the other methods on key tokens. This holds true both for sequences at the training length and for sequences at 8x the training length.

2.52BC4, 2K: key-token PPL (512 window)C4, 2K: PPLGovReport, 16K: LongPPLGovReport, 16K: PPL
RoPE4.609.7413.267.60
p-RoPE4.399.7313.007.51
LeRoPE3.919.697.117.57
Reviewer_vGoXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies learning RoPE frequencies during language-model pretraining. LeRoPE introduces one log-scale parameter per frequency band,$ \widehat{\theta}_m=e^{\alpha_m}\theta_m,$ shared across attention heads and layers. The authors train models from 52M to 2.52B parameters, compare validation perplexity against standard and partial RoPE, and estimate compute multipliers from the resulting loss–compute curves. They also analyze the learned frequency profiles, identify a dominant positional band, and evaluate retrieval and length extrapolation with positional rescaling. The appendices include stronger fixed-frequency baselines, parameterization ablations, repeated-seed experiments, and code pretraining.

Critical strengths
  1. The method is simple and adds only 32 parameters in the evaluated architectures. Table 1 reports lower validation perplexity at every tested scale.
  2. The experiments cover several useful controls. In particular, the stronger-base comparisons, matched-seed runs at 217M and 608M, and code-pretraining results strengthen the evidence beyond the main scaling ladder.
  3. The dominant-band analysis is interesting. The authors report the deterioration under naive extrapolation and investigate it through dominant-band interpolation, which helps explain a limitation of the learned spectrum.
Critical weaknesses
  1. The efficiency improvement should be stated more clearly relative to the baseline and estimation procedure. On p. 6, §5.1, the authors describe the gain as: “RoPE requiring 3.4% more compute to match it at the largest scale.” The number is consistent with the estimator in §4.2. However, it is an extrapolated compute multiplier against the default-base RoPE model. As Appendix A.2 states on p. 16:“Estimating the multiplier requires extrapolating the RoPE curve” At 2.52B, the validation-loss difference implied by Table 1 is
\approx 0.00328\ \text{nats}.$$ The repeated-seed experiments at smaller scales support the direction of the improvement, but they do not directly establish the uncertainty of the 3.4% estimate at the largest scale. The alternative fit in Appendix A.2 is helpful; I would also like to see how much the estimate changes under plausible variation in the losses and the local loss–compute slope. The stronger fixed-frequency baseline is relevant here. Appendix D, p. 26, reports that increasing the RoPE base to 500,000 captures roughly 60% of LeRoPE’s gain at 217M. Against this baseline, the estimated multipliers are approximately 1.015× at 217M and 1.02× at 608M. These still favor LeRoPE, but the remaining gain is smaller. Could the authors include this stronger baseline at one of the larger scales and provide an uncertainty or sensitivity estimate for the multiplier? The abstract and conclusion should make clear that 3.4% is an estimated FLOP multiplier relative to the default-base baseline. 2. Several statements about seed variability need correction. The caption of Table 9 on p. 27 states: “Seed effects are strongly correlated across methods trained on the same seed, so paired margins vary substantially less than the marginal deviations suggest.” The displayed 608M results do not support this explanation. Using the rounded values in the table, the sample standard deviations are approximately $$s_{\mathrm{RoPE}}=0.000872,\qquad s_{\mathrm{LeRoPE}}=0.000351,$$ while the standard deviation of the paired differences is $$s_{\mathrm{RoPE}-\mathrm{LeRoPE}}=0.001002.$$ Thus, the paired differences vary more than either method individually in this comparison. At 217M, their standard deviation is also close to LeRoPE’s marginal standard deviation, rather than substantially smaller. Appendix F contains another statement that does not agree with the table. On p. 30, the authors write: “LeRoPE is better under every seed and its worst run is better than RoPE’s best.” The first part agrees with Table 13, but the second does not. Across the four reported runs, $$\max L_{\mathrm{LeRoPE}}=1.2407 \min L_{\mathrm{RoPE}}=1.2333.$$ The ranges therefore overlap. LeRoPE still improves on RoPE within every matched seed, which is the more relevant result and should be retained. Please correct these descriptions and explain why Table 13 reports mean and standard deviation over runs 1–3 but the mean paired margin over all four runs. These are concerns about the statistical reporting; they do not negate the consistent direction of the paired improvements. 3. The aggregation of long-context results needs clarification. On p. 9, §7, the authors state: “For extrapolation evaluations we filter the validation set of C4 for documents longer than 4096 in length.” Table 11 on p. 29 reports in-distribution, extrapolated, and full-window perplexities over positions 0–4095. The RoPE values are 10.138, 8.861, and 10.042, respectively. If the two positional ranges contain approximately equal numbers of evaluated tokens from the same documents, and perplexity is computed from pooled token loss, the full-window value would be approximately $$\exp\left( \frac{\log 10.138+\log 8.861}{2} \right) \approx 9.478,$$ rather than 10.042. Different sample sets or averaging conventions could explain this, so the discrepancy alone does not establish an evaluation error. Please specify the document and token counts and the aggregation formula for each column. The RULER aggregate also needs a qualification. Table 14, p. 31, notes:“VT rows are not bolded because neither protocol measures variable tracking faithfully.” Yet VT is included in both “All 11” averages. Appendix E.3 explains that the scores are affected by copying the demonstration or failing to follow the output template. It is therefore unclear how much this task contributes evidence about variable tracking. Please identify which VT protocol is used in Figure 5 and report the aggregate without VT. The eight-task retrieval averages already favor LeRoPE, so this would clarify the result without relying on a task the authors themselves regard as unreliable.
Questions for authors

Same as above

4 comments

We thank the reviewer for their particularly thorough and constructive comments.

Per position perplexity

We apologize for the confusion this caused. The documents used to report loss over positions were filtered incorrectly which led to different token counts for the in-distribution and extrapolation set.

We have rerun evaluation on the filtered set and will update the values in the paper accordingly. This set includes only documents that are at least 4096 tokens from the validation set. We report perplexity on the first 4096 tokens, the in-distribution column corresponds to the average perplexity on tokens 0-2047 and the Extrap column refers to the average perplexity for tokens from position 2048-4095. We use YaRN here for extrapolation. The updated set of positional perplexities retain the original ordering with LeRoPE outperforming all other methods:

MethodIn-distExtrapFull
RoPE9.8739.3579.611
p-RoPE9.8609.3309.591
RoPE, b = 500,0009.8249.3119.564
LeRoPE, dominant band only9.8149.3449.576
LeRoPE9.8149.2729.539

RULER evals without VT

We included the zero-shot VT version in our aggregation for Figure 5. Below are the average performance for tasks not including variable tracking:

Method2K4K8K16K32K
RoPE.419.380.290.208.133
p-RoPE.443.401.321.192.118
LeRoPE.499.465.414.318.178

We will update our figure in the final version to include the 10 tasks as well as an additional figure visualizing the 8 retrieval task performance as well.

Seed Noise

First, we thank the reviewer for their careful corrections on the seed reporting in the appendix. We will fix these in the camera-ready version. We will aggregate the results over all runs 1-4 in Table 13, there was no reason they shouldn't have been.

During the rebuttal period we conducted 2 additional seed runs for the 1.3B models for RoPE-10k and LeRoPE as well as additional seed runs for p-RoPE at 217M and 608M. LeRoPE reaches a lower loss than the baselines on all 16 paired runs and at a given model size every LeRoPE run is better than the best RoPE-10k run for our models trained on c4. We present the results below:

ScaleRoPE, mean ± SDp-RoPE, mean ± SDLeRoPE, mean ± SDMargin, 3 seedsMean margin ± SD (95% CI)
217M2.9046 ± 0.00102.9036 ± 0.00142.8981 ± 0.0019RoPE − LeRoPE: 0.0057 / 0.0087 / 0.00520.0065 ± 0.0019 (0.0018–0.0112)
p-RoPE − LeRoPE: 0.0051 / 0.0065 / 0.00510.0056 ± 0.0008 (0.0036–0.0076)
608M2.6294 ± 0.00092.6284 ± 0.00092.6253 ± 0.0004RoPE − LeRoPE: 0.0051 / 0.0040 / 0.00310.0041 ± 0.0010 (0.0016–0.0066)
p-RoPE − LeRoPE: 0.0042 / 0.0031 / 0.00200.0031 ± 0.0011 (0.0004–0.0058)
1.34B2.4490 ± 0.00052.44812.4456 ± 0.0002RoPE − LeRoPE: 0.0031 / 0.0039 / 0.00330.0034 ± 0.0004 (0.0024–0.0045)
p-RoPE − LeRoPE: 0.0024 / — / ——

We continue to observe that the within-method standard deviation across seed runs as well as the standard deviation of the paired difference between RoPE and LeRoPE continue to decrease with scale.

While we are not able to compute multi-seed 2.5b runs due to compute restrictions, we believe this supports the statistical significance of our method’s gains. If we assume the same pairwise loss difference SD as we observed in our 1.3B models and extend it to the 2.5B models we get a 95% CI of 0.0015–0.0051 for the loss gap. This equates to 1.5 to 5.2% CI for the compute multipliers. If we take the SD of the pairwise variance across all three scales this gives us a CI of 0.0002–0.0064 for the loss gap equating to CI of 0.2% to 6.6% for compute efficiency. In both cases the intervals do not include 0.

Stronger Baselines

We were also able to train a 1.3B and 2.5B model with RoPE 500k during the rebuttal period. We observed similar performance compared to our smaller models with RoPE-500k recovering 55% of LeRoPE’s gain over RoPE at 1.3B and 61% for 2.5B. Additionally we conduct evaluations on RULER's 8 retrieval tasks for the 2.5B model, where we find that its performance lags LeRoPE across context lengths. Accuracy reported below:

Model2K4K8K16K32K
RoPE.509.457.351.245.152
p-RoPE.548.498.398.238.146
RoPE, b = 500,000.543.493.420.286.149
LeRoPE.608.573.511.390.214

We would like to emphasize while RoPE 500k improves over our base RoPE in terms of loss this does not translate to equally strong retrieval gains.

Below we present the loss gaps for RoPE 500k and LeRoPE’s compute multipliers w.r.t it. We either estimated CI’s using the standard deviation of the pairwise loss gap between RoPE and LeRoPE from the same scale above with 1.3B’s SD for 2.5B or used the standard deviation of the pairwise RoPE 500k-LeRoPE 217M run. We believe each of these estimates are on the wider side. We are either using the SD from a 217M run which we observed to decrease with scale for RoPE-10k or we used the RoPE 10k-LeRoPE gap which we observed to be smaller when using RoPE 500k instead. We provide the numbers below:

ScaleRoPE 500k − LeRoPE gapShare of LeRoPE's gain recovered by 500kLeRoPE CM w.r.t. 500kCM 95% CI, 217M 500k − LeRoPE paired SD (0.00083)CM 95% CI, same-scale RoPE − LeRoPE paired SD
217M (3 seed avg)+0.002758%1.0231.005–1.0400.984–1.063
608M+0.002453%1.0220.990–1.0560.984–1.063
1.34B+0.001455%1.0140.980–1.0520.997–1.033
2.52B+0.001361%1.0130.977–1.0500.995–1.031

LeRoPE maintains a small, consistent loss advantage over RoPE 500k at every scale while achieving stronger retrieval performance. We will include the additional RoPE 500k runs as well as this CI discussion in the final version

| VT RULER concerns

Given the concerns on whether our VT results accurately demonstrate long context utilization or not we include LongPPL to accompany the RULER evaluations.

LongPPL uses an evaluator model to identify key tokens in a given long context by contrasting their likelihood with a short local context window with the full preceding context. It selects tokens which have a delta from the short and full context above a given threshold and that can be accurately predicted given the full context. To evaluate a given model it calculates the perplexity/loss of just these key tokens. The authors found that this metric has a much stronger correlation than perplexity with downstream long context performance on LongBench and LongEval.

The original LongPPL score is calculated on documents from GovReport and uses a sliding window context of 4k to contrast against the full document. In this setting we set the full context length to be 16k. We include both the original LongPPL setup on GovReport as well as a shorter version with 512 window with 2k total context on C4 validation documents. We use Llama-3.1-8B to select the key tokens, which the authors find is a sufficient evaluator.

We report the standard perplexity as well as the perplexity just over the key tokens for both settings below. We evaluating on the GovReport documents at 16k we apply YaRN. We observe that while the raw perplexity difference may be small, LeRoPE outperforms the other methods on key tokens. This holds true both for sequences at the training length and for sequences at 8x the training length. We will include this along with the extended RULER score breakdown in the appendix.

2.52BC4, 2K: key-token PPL (512 window)C4, 2K: PPLGovReport, 16K: LongPPLGovReport, 16K: PPL
RoPE4.609.7413.267.60
p-RoPE4.399.7313.007.51
RoPE 500k4.019.709.807.62
LeRoPE3.919.697.117.57

Thank the authors for providing this additional information. It has addressed my concerns. I'm happy to accept this paper.

Reviewer_XkV9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LeRoPE makes RoPE frequencies trainable through one multiplicative scale per frequency band, parameterized in log space and shared across all heads and layers. This adds 32 parameters in the studied models. The paper trains models from 52M to 2.52B parameters on C4, with additional experiments on Python code, and reports consistent improvements in validation loss. The analysis identifies a dominant learned band whose wavelength is approximately twice the training context length in the C4 experiments. Frozen-frequency transfer and band-specific interventions are used to study its role. The paper also evaluates retrieval and context extension with YaRN. At 2.52B, it estimates that default RoPE requires 3.4% more training compute to match LeRoPE’s loss, while acknowledging that unmodified LeRoPE extrapolates poorly.

Critical strengths
  1. The improvement is small but supported by repeated evidence. LeRoPE improves C4 perplexity at all five scales, and the matched-seed experiments at 217M and 608M consistently favor it.
  2. The comparisons include useful controls. Learning rates are swept separately at the smaller scales, and the appendices test larger RoPE bases, different partial-RoPE fractions, alternative parameterizations, and frequency-sharing choices.
  3. The analysis goes beyond reporting a lower loss. Transferring and freezing learned frequencies tests whether the final spectrum is useful, while dominant-band interventions connect the observed frequency structure to model behavior.
  4. The gains extend to code pretraining and retrieval. Under the same YaRN recipe, the 16K RULER retrieval mean increases from 24.5% with RoPE to 39.0% with LeRoPE, providing a concrete downstream benefit.
Critical weaknesses
  1. The methodological contribution needs a narrower framing. Learning RoPE frequencies in log space is already part of AdaRoPE, acknowledged in Appendix B. The contribution here lies mainly in global sharing and the empirical analysis. However, Table 8’s shared-versus-per-head difference is only 0.0018 nats, within the reported seed variability. This does not establish the claim that sharing is consistently better.
  2. The headline efficiency gain depends on the baseline and estimator. The 3.4% figure is extrapolated from the default-base RoPE scaling curve. Appendix D shows that base 500,000 recovers roughly half or more of the gain at smaller scales, but this stronger baseline is not evaluated at 2.52B. The residual advantage at the headline scale and the uncertainty of the estimate remain unclear.
  3. The dominant-band interpretation is not fully established. Figure 4 shows contributions averaged by distance, which can hide content-dependent variation. It therefore does not directly establish that the band is nearly purely positional. Appendix E.1 also needs to specify whether “zeroing a band” removes its frequency, Q/K activations, or score contribution; these interventions test different claims.
  4. Table 11’s aggregation needs clarification. If evaluation uses documents longer than 4096 tokens, the two 2048-token regions should have approximately equal weight. The RoPE perplexities of 10.138 and 8.861 would then imply a full-window perplexity of about 9.478, rather than 10.042. Different subsets or weighting could explain this, but the current description does not.
Questions for authors
  1. Table 8 reports losses of 2.8978 for shared frequencies and 2.8996 for per-layer/per-head frequencies, with the difference within seed variability. What supports “consistently best” in Section 3.1? Does the ordering hold across matched seeds?
  2. How much of the 2.52B advantage remains against RoPE with base 500,000? Can the extrapolated compute multiplier also be reported with uncertainty, rather than only the 3.4% point estimate?
  3. In Appendix E.1, what exactly is set to zero? To test the “nearly purely positional” interpretation, does replacing the dominant band’s contribution with a per-layer, per-head function of distance estimated on calibration text preserve held-out loss?
  4. For Table 11, what document filter, token counts, and averaging rule produce the reported full-window perplexity? In particular, why do 10.138 and 8.861 aggregate to 10.042, and is the 4096-length filter measured in tokens or characters?
  5. The gap between Fixed LeRoPE and full LeRoPE is only 0.0021 nats in Table 8. Does this gap survive matched-seed repeats? How stable is the reported 63.6% recovery before interpreting the remaining gain as evidence for joint training dynamics?
3 comments

We thank the reviewer for their constructive review and helpful questions that allowed us to further support our interpretation of our method.

Seed Noise

During the rebuttal period we conducted 2 additional seed runs for the 1.3B models for RoPE-10k and LeRoPE as well as additional seed runs for p-RoPE at 217M and 608M. LeRoPE reaches a lower loss than the baselines on all 16 paired runs and at a given model size every LeRoPE run is better than the best RoPE-10k run for our models trained on c4. We present the results below:

ScaleRoPE, mean ± SDp-RoPE, mean ± SDLeRoPE, mean ± SDMargin, 3 seedsMean margin ± SD (95% CI)
217M2.9046 ± 0.00102.9036 ± 0.00142.8981 ± 0.0019RoPE − LeRoPE: 0.0057 / 0.0087 / 0.00520.0065 ± 0.0019 (0.0018–0.0112)
p-RoPE − LeRoPE: 0.0051 / 0.0065 / 0.00510.0056 ± 0.0008 (0.0036–0.0076)
608M2.6294 ± 0.00092.6284 ± 0.00092.6253 ± 0.0004RoPE − LeRoPE: 0.0051 / 0.0040 / 0.00310.0041 ± 0.0010 (0.0016–0.0066)
p-RoPE − LeRoPE: 0.0042 / 0.0031 / 0.00200.0031 ± 0.0011 (0.0004–0.0058)
1.34B2.4490 ± 0.00052.44812.4456 ± 0.0002RoPE − LeRoPE: 0.0031 / 0.0039 / 0.00330.0034 ± 0.0004 (0.0024–0.0045)
p-RoPE − LeRoPE: 0.0024 / — / ——

We continue to observe that the within-method standard deviation across seed runs as well as the standard deviation of the paired difference between RoPE and LeRoPE continue to decrease with scale.

While we are not able to compute multi-seed 2.5b runs due to compute restrictions, we believe this supports the statistical significance of our method’s gains. If we assume the same pairwise loss difference SD as we observed in our 1.3B models and extend it to the 2.5B models we get a 95% CI of 0.0015–0.0051 for the loss gap. This equates to 1.5 to 5.2% CI for the compute multipliers. If we take the SD of the pairwise variance across all three scales this gives us a CI of 0.0002–0.0064 for the loss gap equating to CI of 0.2% to 6.6% for compute efficiency. In both cases the intervals do not include 0.

Stronger Baselines

We were also able to train a 1.3B and 2.5B model with RoPE 500k during the rebuttal period. We observed similar performance compared to our smaller models with RoPE-500k recovering 55% of LeRoPE’s gain over RoPE at 1.3B and 61% for 2.5B. Additionally we conduct evaluations on RULER's 8 retrieval tasks for the 2.5B model, where we find that its performance lags LeRoPE across context lengths. Accuracy reported below:

Model2K4K8K16K32K
RoPE.509.457.351.245.152
p-RoPE.548.498.398.238.146
RoPE, b = 500,000.543.493.420.286.149
LeRoPE.608.573.511.390.214

We would like to emphasize while RoPE 500k improves over our base RoPE in terms of loss this does not translate to equally strong retrieval gains.

Below we present the loss gaps for RoPE 500k and LeRoPE’s compute multipliers w.r.t it. We either estimated CI’s using the standard deviation of the pairwise loss gap between RoPE and LeRoPE from the same scale above with 1.3B’s SD for 2.5B or used the standard deviation of the pairwise RoPE 500k-LeRoPE 217M run. We believe each of these estimates are on the wider side. We are either using the SD from a 217M run which we observed to decrease with scale for RoPE-10k or we used the RoPE 10k-LeRoPE gap which we observed to be smaller when using RoPE 500k instead. We provide the numbers below:

ScaleRoPE 500k − LeRoPE gapShare of LeRoPE's gain recovered by 500kLeRoPE CM w.r.t. 500kCM 95% CI, 217M 500k − LeRoPE paired SD (0.00083)CM 95% CI, same-scale RoPE − LeRoPE paired SD
217M (3 seed avg)+0.002758%1.0231.005–1.0400.984–1.063
608M+0.002453%1.0220.990–1.0560.984–1.063
1.34B+0.001455%1.0140.980–1.0520.997–1.033
2.52B+0.001361%1.0130.977–1.0500.995–1.031

LeRoPE maintains a small, consistent loss advantage over RoPE 500k at every scale while achieving stronger retrieval performance. We will include the additional RoPE 500k runs as well as this CI discussion in the final version.

Per position perplexity

We apologize for the confusion this caused. The documents used to report loss over positions were filtered incorrectly which led to different token counts for the in-distribution and extrapolation set.

We have rerun evaluation on the filtered set and will update the values in the paper accordingly. This set includes only documents that are at least 4096 tokens from the validation set. We report perplexity on the first 4096 tokens, the in-distribution column corresponds to the average perplexity on tokens 0-2047 and the Extrap column refers to the average perplexity for tokens from position 2048-4095. We use YaRN here for extrapolation. The updated set of positional perplexities retain the original ordering with LeRoPE outperforming all other methods:

MethodIn-distExtrapFull
RoPE9.8739.3579.611
p-RoPE9.8609.3309.591
RoPE, b = 500,0009.8249.3119.564
LeRoPE, dominant band only9.8149.3449.576
LeRoPE9.8149.2729.539

Multi-seed Fixed LeRoPE and Per Head/Layer Parametrizations

We conduct multi-seed evaluations of fixed LeRoPE and alternative parameterizations (per head/layer and per layer) at 217M bringing the total of models trained per method up to 5 (the 3 seeds from above + an additional 2). For fixed LeRoPE we find that on average 85.02% of LeRoPE’s gains are recovered against the seed’s corresponding RoPE baseline run. These results are a bit noisy with a standard deviation of 14.55 with one seed recovering 103.6% of the corresponding seed’s LeRoPE gains. These results weaken the joint training dynamic interpretation however we did observe different dominant frequency patterns depending on the training corpus. These results suggest determining a useful set of frequencies from smaller runs for a given pre-training corpus before a large pre-training run may be a promising method.

For the alternative parameterizations we found the results to mostly be within seed noise with 95% confidence intervals including 100% of the gains. The per-head per-layer parameterization was the most susceptible to seed noise which is undesirable for large scale training while fully sharing the frequencies had the lowest. For simplicity and parameter saving, sharing frequencies across layers is a reasonable design choice. We include the complete numbers in the table below and will include this discussion in Appendix D.

MethodRun 1Run 2Run 3Run 4Run 5MeanSD95% CIraw loss SD (lerope = 0.0014)
Fixed82.4%63.2%88.9%87.0%103.6%85.0%14.6%66.9% to 103.1%0.0021
Per-layer97.3%103.1%125.0%93.2%102.9%104.3%12.3%89.0% to 119.6%0.0016
Per-layer-and-head68.4%70.4%119.8%85.7%105.1%89.9%22.3%62.2% to 117.5%0.0022

Dominant band positional information

We apologize for the underspecified description. Here we set the total band contribution to 0.

We test 5 different functions of position to replace the dominant band in our 2.5B model. This gives us a logit of qsTkt=f(s,t)+∑m=0m≠domdh/2−1(qs(m))Tρ(eαmθm)(t−s)kt(m).q_s^T k_t = f(s,t) + \sum_{\substack{m = 0 \\ m \neq \mathrm{dom}}}^{d_h/2 - 1} (q_s^{(m)})^T \rho(e^{\alpha_m} \theta_m)^{(t-s)} k_t^{(m)}.

For f(s,t)f(s,t) we try:

  • A lookup table of relative position, f(s−t)f(s - t)
  • A rotation: Acos(eαdomθdom(s−t))+Bsin(eαdomθdom(s−t))A cos (e^{\alpha_{dom}} \theta_{dom} (s-t)) + B sin (e^{\alpha_{dom}} \theta_{dom} (s-t)) where we estimate A and B.
  • A lookup table of relative distance, f(s−t)f(s -t) for 1≤t≤s1 \le t \le s, and a separate lookup table for the sink g(s)g(s).
  • Acos(eαdomθdom(s−t))+Bsin(eαdomθdom(s−t))A cos (e^{\alpha_{dom}} \theta_{dom} (s-t)) + B sin (e^{\alpha_{dom}} \theta_{dom} (s-t)) for 1≤t≤s1 \le t \le s and Asinkcos(eαdomθdom(s−t))+Bsinksin(eαdomθdom(s−t))A_{sink} cos (e^{\alpha_{dom}} \theta_{dom} (s-t))+ B_{sink} sin (e^{\alpha_{dom}} \theta_{dom} (s-t)) for the sink.
  • A lookup table of absolute position, f(s,t)f(s, t)

We estimate the parameters for these functions per head per layer using 20k documents from the validation set and compute the loss on a different 8k. Additionally we include a no rotation ablation where the band contribution is qsdom⋅ktdomq^{dom}_s \cdot k^{dom}_t. We report these 5 functions as well as the no rotation ablation from Appendix E.1 below:

Methodloss increase
0 contribution+0.762
qsm⋅ktmq^m_s \cdot k^m_t (rotation off)+0.722
f(s−t)f(s-t)+0.224
Acos+BsinA cos + B sin+0.231
f(s−t)+g(s)f(s-t) + g(s)+0.0211
Acos+Bsin+Asinkcos+BsinksinA cos + B sin + A_{sink} cos + B_{sink} sin+0.0211
f(s,t)f(s, t)+0.0206

These results show that provided you properly account for the sink token, the dominant band can be replaced with functions of just distance while maintaining performance. The best performing functions recover 97% of the loss gap from not including the band completely. This further suggests the dominant band is predominantly a positional signal. We repeat this experiment for every band and find that the dominant band is the only one that can be reconstructed with these distance functions. The next most recoverable bands are the fastest few bands but at best (with the sink-aware ones) only recover ~80% of the loss gap from zeroing out their contributions. We will include these additional ablations and corresponding visualizations in Appendix E.1 in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.