acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Base: Frequency Allocation in RoPE

Abstract

In rotary position embeddings (RoPE), reallocating frequencies within a fixed range can substantially change how effectively a model uses context. Controlled interventions show that performance gains persist even when total log-frequency displacement is also matched. Analysis of complete sine–cosine pairs characterizes positional overlap; reassigning the same frequencies across coordinates changes task performance, demonstrating the role of learned frequency use. Viewing allocation as channel-wise distance scaling motivates TailSpline, a closed-form transition from native frequencies to the extended low-frequency tail. Derived from a discrete tail-connection problem, the construction uses only public RoPE parameters and retains standard rotary computation without weight updates or calibration. At fourfold extension to 32K, TailSpline improves Llama-3-8B's full thirteen-task RULER mean by over 11 percentage points against both YaRN and MrRoPE-Pro. It achieves the highest mean in every main frozen-extension RULER comparison across five model families up to 70B parameters, alongside improvements in language modeling and natural question answering. Allocation also improves native-window language modeling with frozen weights and strengthens extrapolation during training and adaptation. These findings establish internal frequency allocation as a distinct design dimension of RoPE that shapes how models use context within and beyond the native window.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rotary position embeddings, long-context language models, context length extrapolation, positional encoding, spectral geometry, weight-table co-adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7LXQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates frequency allocation in positional interpolation methods for context window extension. YaRN, a well-known positional interpolation method, modifies the RoPE frequencies by changing some frequencies while keeping others unchanged during fine-tuning for context extension. The main contribution of this paper is essentially limited to replacing part of YaRN's frequency allocation scheme with a new mathematical formulation. Furthermore, the authors do not perform fine-tuning in their context extension experiments, despite fine-tuning typically being employed for this purpose. In addition, some baseline results are missing. For example, Table 2 does not report the LongBench v2 results for Llama-3-8B with YaRN, even though YaRN is one of the main baselines. Overall, I find the experimental setup questionable, and the omission of certain baseline results further undermines the validity of the evaluation. Therefore, I recommend rejecting this paper.

Critical strengths

The authors are addressing the important problem of frequency allocation for context expansion.

Critical weaknesses

Novelty

The main contribution of the proposed method is essentially limited to replacing part of YaRN's frequency allocation scheme with a new mathematical formulation.

Experimental Setup

Typically, studies on context extension using YaRN report results after fine-tuning. However, the authors do not perform any fine-tuning in their main experiments (Tables 1 and 2). While the performance obtained without fine-tuning, i.e., under length extrapolation, may be interesting, I believe such results should be presented as supplementary evidence rather than as the primary evaluation. For a paper focusing on context extension, I find this experimental setup highly questionable. Furthermore, the reported RULER scores are considerably lower than those reported in some previous studies, making it difficult for me to assess whether the experimental setup and results are reliable. For example, a previous study [1] reported a RULER score of 68.69% for YaRN after fine-tuning Llama-3-8B with a context length of up to 32K. Although the experimental conditions are not identical, this highlights the importance of evaluating the proposed method under standard fine-tuning-based context extension settings. In addition, several baseline results are missing. For example, Table 2 does not report the LongBench v2 results for Llama-3-8B with YaRN, despite YaRN being one of the main baselines. Presentation The authors appear to have conducted a thorough literature review. However, the table captions are unnecessarily lengthy while omitting important information. It is difficult to identify which benchmark is being evaluated at a glance. In particular, the caption of the table reporting RULER results does not clearly identify the benchmark as RULER. Instead, the authors use the term "Full-13," which appears to be their own terminology and is not used in the original RULER paper. This makes the results unnecessarily difficult to interpret.

Overall, I find the experimental setup inappropriate for adequately evaluating the proposed method as a context extension technique. In particular, the lack of fine-tuning-based comparisons and the omission of several important baseline results substantially weaken the experimental evidence. Combined with the limited methodological novelty, these concerns lead me to recommend rejection of this paper.

[1] A Controlled Study on Long Context Extension and Generalization in LLMs. COLM 2025.

Questions for authors

See the Weaknesses.

Reviewer_qys8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates RoPE’s frequency range from the allocation of frequencies within that range. It uses controlled interventions to study allocation and proposes TailSpline, a closed-form rule for training-free context extension. The proposed method achieves full-suite RULER performance across several model families. Additional experiments study native-window reallocation and allocation changes during training.

Critical strengths
  • The frozen-model comparisons are carefully controlled and show substantial gains. The results in Table 1 show promising performance compared to other training-free approaches. The matched-displacement controls are particularly useful for showing that the distribution of frequency changes matters beyond their total magnitude.

  • TailSpline is simple to deploy: it requires no training or calibration and retains standard rotary computation. Testing the same construction across five model families, including partial RoPE and a 70B model, strengthens its practical value. I also appreciate the reported uncertainty and native-window trade-offs.

Critical weaknesses
  • Transfer to natural tasks is less convincing than the RULER results. In Table 2, many tasks do not improve compared with MrRope-Pro (confidence interval covering zero). On tasks that do significantly improve, the performance of YaRN is much better than MrRope-Pro, so the overall advantage is small. These results suggest possible benefits, but the practical improvement in natural long-context use remains uncertain.

  • The results show that frequency allocation matters, but I’m less convinced by the choice of TailSpline’s specific shape. Equation (7) favors a smooth transition at the low-frequency end, yet the simple step profile does better on OLMo and is competitive on Llama (Table 9). It would help to test this simpler option on the other models to see what TailSpline adds.

Questions for authors
  • What makes the tasks in Table 2 different from each other? Why MrRoPE-pro sometimes leads to worse performance than YaRN?
  • How does the simple step profile compare with TailSpline on the other models?
  • Why smooth only the low-frequency end? What happens if both ends are smoothed?
Reviewer_5698OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper divides the RoPE frequency table into two parts: range which is defined by the endpoints and allocation which is defined by the normalized positions inside the range. It argues that allocation should be considered as an independent design dimension. The paper proposes TaliSpline which is a training-free method with a closed-form transition profile that applies more stretching in the earlier part of the transition band. It also introduces NCP for frequency reallocation with the native window and Cosh for non-geometric allocation during training. Experiments cover five model families with model sizes up to 70B. TailSpline achieves the highest average RULER score in all main panels and outperforms YaRN and MrRoPE-pro by more than 11 points on Llama-3-8B at 32k. The paper further controls the range, total displacement, and coordinate assignment to study the contribution of allocation itself.

Critical strengths
  1. The paper formulation is clear, and the control experiments are well designed. The paper separates range and allocation and uses three levels of control: the same endpoints, the same total displacement, and permuted coordinates. These experiments help examine alternative explanations step by step. This methodology is also useful for future work.
  2. The experimental results are strong and reasonably convincing. TailSpline achieves the highest average RULER score in all main panels across five model families from 1B-70B. On Llama-3-8B at 32k, it outperforms both baselines by more than 11 points. The same frequency table also outperforms MrRoPE-Pro from 8K to 32K. The paper reports repaired intervals throughout and includes results where the method does not perform well.
  3. The method is practical and easy to use. TailSpline has a closed-form expression and O(K) complexity. It only uses public parameters and does not require search, calibration, or weight updates. It can directly replace an existing frequency table.
Critical weaknesses
  1. Limited novelty. MrRoPE already studies per-dimension scaling and transition profiles, while YaRN's paper formula already applies front-loaded stretching.
  2. Unclear contribution of the shape. The equal-displacement control explains most of the gain, the TailSpline only improves over it on Llama at 32K. The bending cost is assumed rather than derived.
  3. Insufficient validation. Reproduced baselines are far below MrRoPE's reported numbers. and some experiments use few prompts. Attention gain lacks an ablation, NCP is tested only one model, and Cosh shows no clear advantage over exponential allocation.
Questions for authors
  1. Why does TaliSpline outperform the equal-displacement control only on Llama at 32K? How much of the improvement comes from the specific comes from the specific shape, and how much comes from applying stretching earlier and with a larger total amount?
  2. What causes the large gap between the reproduced baseline scores and the results reported by MrRoPE?
  3. Does the YaRN curve in Fig 1 follow the official implementation or the paper formula? The increments of these two versions change in opposite directions.
  4. If the attention gain is reduced or removed, can the performance drop at 8K be avoided while keeping the improvement at 32K?
  5. Can the one-sided bending cost be derived from the analysis in Sec 3, instead of being introduced as an assumption?
  6. Since an exponential allocation with matched total movement achieves the same performance, is the cosh form necessary?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.