acceptodds
Withdrawn from ICLR 2027

How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose a data-centered explanation: RoPE frequencies are selected to match the relative-distance structure of the training data. Viewing each frequency as a positional lens, we formalize a field-resolution tradeoff and show that, for a data-induced dependency profile of width , the optimal frequency scales as . This frequency-matching principle explains controlled observations on synthetic and text-based data, and suggests that the mid-low frequency bands observed in language models arise from the multi-scale dependency structure of natural language. We further connect frequency selection to position-interpolation-based length generalization: scaling frequencies down expands the effective field while reducing resolution. This helps when longer-context dependencies are approximate dilations of those seen during training, but can fail when relevant dependencies do not scale with context length. Empirically, we show that natural language exhibits approximate self-similarity across positional scales, explaining why test-time frequency scaling can support long-context generalization. Overall, our results identify a data-driven mechanism behind emergent RoPE frequency usage and show that long-context generalization depends on two forms of scale matching: between learned frequencies and training-time dependencies, and between frequency scaling and how those dependencies extend to longer contexts.

Primary area:
learning theory
Keywords:
RoPE, positional encoding, data, length generalization, deep learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dAPROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how the dependency structure of training data influences the use of different RoPE frequencies in Transformers. The authors introduce a “frequency energy” metric based on the amplitudes of Q/K subspace contributions and propose that broader data dependencies favor lower frequencies. Under their definitions of positional contrast and field coverage, they derive an inverse relationship between the optimal frequency and dependency width. They further show that when the test-time dependency distribution is a uniform dilation of the training distribution, the corresponding position interpolation preserves the proposed positional contrast utility.

The experiments include frequency-spectrum analyses of pretrained models, synthetic retrieval tasks with varying block lengths, iGSM datasets with different reasoning lengths, and evaluations of position interpolation on synthetic retrieval and arithmetic tasks. The authors also compare mutual-information profiles across tokenization granularities to support approximate self-similarity in natural-language dependencies.

Critical strengths
  1. Intervening on training data provides a useful approach to studying frequency usage. The synthetic experiment fixes sequence length and retrieval offset while varying block length, then measures the learned frequency distribution. This offers evidence about how data statistics influence internal representations and is the paper’s most valuable empirical contribution.

  2. The discussion of when position interpolation should help offers useful insight. The paper distinguishes settings where dependency distances expand with context length from those where the relevant distances remain fixed. The synthetic retrieval example, arithmetic evaluation, and additional NTK-aware and YaRN comparisons give this distinction some practical relevance.

Critical weaknesses
  1. The frequency metric and theoretical optimum are insufficiently connected to the mechanisms through which the model performs the task.

The “frequency energy” in Section 3 is mathematically well defined, but the quantity being averaged simplifies to:

\left\lVert k_j^{(m)} \right\rVert_2^{2}.$$ Thus, the metric measures the expected product of squared Q/K subvector norms. It can characterize the amplitude of a frequency component, but it does not capture that component’s phase-dependent contribution at the relative distances actually encountered, nor does it directly establish its importance for positional retrieval or final predictions. The paper should clarify what information this metric adds beyond existing norm-based analyses and validate whether higher energy corresponds to greater functional importance. Similarly, Theorem 1 obtains the inverse relationship between optimal frequency and dependency width by exploiting the monotonicity of positional contrast within a prescribed half-period coverage constraint. The result holds under these definitions, but the paper does not establish a connection between this utility and the training loss, learning dynamics, or allocation of frequency energy. In particular, the theory optimizes a single continuous frequency, whereas the experiments measure a geometric mean over multiple discrete frequencies weighted by energy. Why these quantities should follow the same quantitative law remains unclear. This gap limits the paper’s claim to provide a predictive theory of frequency selection. 2. The key experiments do not sufficiently validate the proposed dependency-scale mechanism, and the length-generalization analysis contains comparability issues. In the block-structured experiment, the authors identify dependency width with the support of the input autocorrelation, which equals the block length. However, the task always requires retrieving the value exactly 16 positions earlier. Changing block length changes statistical correlations among inputs, while the direct positional relation required for exact retrieval remains fixed. The paper does not sufficiently justify why the former represents the task-relevant dependency width in the theory. It also does not report retrieval errors or whether the model recovers the target’s independent noise component. Consequently, the results do not establish whether the model performs accurate retrieval or primarily makes approximate predictions using the shared block signal. These behaviors would imply different explanations for the observed frequency shift. For natural language, similar mutual-information profiles across BPE vocabulary sizes do not directly establish that, under a fixed tokenizer, the dependency distribution in longer contexts is a uniform dilation of the distribution in shorter contexts. There remains a substantial gap between Figure 4B and the assumptions of Theorem 2. Furthermore, Table 2 in Appendix F.2 cites natural-language results from Chen et al., but those results come from fine-tuned PI models, whereas the arithmetic experiment does not report a corresponding fine-tuning procedure. The original paper’s columns denote evaluation context-window sizes; the submission relabels the same values as disjoint position intervals. This requires correction and prevents the current comparison from attributing performance differences primarily to differences in self-similarity between natural language and arithmetic.
Questions for authors

The following questions seek clarification or validation of the current central claims, and their answers could change my assessment.

  1. Interpretation of the metric: Given that frequency energy simplifies to the expected product of squared Q/K subvector norms, what evidence establishes its ability to measure functional importance in positional modeling? Do frequency perturbation or ablation results support this interpretation?

  2. Connection between theory and measurement: Why should the optimum of the single-frequency utility in Theorem 1 predict the geometric mean of frequencies weighted by energy measured in the experiments? Does this connection require additional assumptions about learning or optimization?

  3. Task performance in the synthetic experiment: What are the training and test errors for the different block lengths in Figure 2? Does the model recover the target position’s independent noise component? How do these results justify using input autocorrelation width as the dependency width of the fixed-offset retrieval task?

  4. Evidence for length generalization: How does similarity across tokenization granularities establish the dilation of dependency distributions required by Definition 4? Please also clarify and correct the differences in fine-tuning protocols and column definitions in Appendix F.2, and explain which evidence still supports the comparison between natural language and arithmetic after these corrections.

Reviewer_QC5aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why pretrained LLMs tend to predominantly utilize certain RoPE frequencies. The authors hypothesize that the frequencies utilized by a model are determined by the positional dependency width of the training data. To test this hypothesis, they construct synthetic datasets with different positional dependency widths and pretrain models on these datasets to examine how RoPE frequency usage changes. Their experimental results show that changing the dependency width of the training data leads to corresponding changes in the RoPE frequencies utilized by the models. Overall, I find this paper easy to follow, and I think the research question is interesting and well-motivated.

Critical strengths

I find the idea of investigating RoPE frequency usage by manipulating the dependency width of training datasets interesting. The most closely related work is probably the frequency-band analysis presented at ICLR 2026 [1]. In [1], the authors use conventional datasets and conduct experiments under settings similar to standard LLM pretraining, investigating RoPE frequency usage from both theoretical and empirical perspectives. In contrast, the authors of this paper explicitly define and control the dependency width of their datasets and investigate whether changes in this width lead to changes in RoPE frequency usage. This approach provides novel insights that have not been explored in previous studies. The authors also provide theoretical analyses to support their hypothesis, and their results suggest that the hypothesis is likely to hold. Furthermore, these findings are potentially applicable not only to standard LLM pretraining but also to context extension using positional interpolation.

[1] Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation–Extrapolation Trade-off.

Critical weaknesses

1. Limited Novelty

I believe the novelty of this paper is somewhat limited. The authors' main finding—that changing the dependency width of the training data leads to changes in RoPE frequency usage—appears closely related to the findings of Oka et al. [1].

Oka et al. pretrained models on WikiText-103 under settings similar to standard LLM pretraining. They varied the maximum training sequence length and demonstrated that RoPE frequency usage changes accordingly. They also provided theoretical analyses suggesting that the most heavily utilized RoPE frequency can be predicted from the maximum training sequence length. In contrast, the present paper introduces the dependency width (W) of the training data and investigates how RoPE frequency usage changes when (W) is varied while keeping the maximum sequence length fixed. I acknowledge that the maximum training sequence length and the dependency width (W) are not necessarily equivalent. The former determines the maximum observable positional distance, whereas the latter characterizes the range of task-relevant dependencies in the data. Therefore, explicitly controlling the dependency width provides a more fine-grained experimental analysis than simply varying the maximum training sequence length.

However, I find it unclear to what extent reformulating the findings of Oka et al. in terms of data dependency width constitutes a substantial conceptual advance. Both studies arrive at closely related conclusions: the positional scale of the training data influences which RoPE frequencies are utilized by the model. I have a similar concern regarding the theoretical contribution. Oka et al. established a relationship between the maximum training sequence length and the frequency bands utilized by the model. The present paper derives a relationship between the dependency width (W) and the preferred RoPE frequency, expressed as (\theta^\star \asymp 1/W). Although the assumptions and theoretical formulations differ, the resulting conclusions appear conceptually similar.

In my view, the authors should more clearly explain what fundamentally new understanding is gained by introducing dependency width (W), beyond the findings already established by Oka et al. In particular, it would be valuable to demonstrate cases where the dependency-width-based theory makes predictions that cannot be explained by the maximum-sequence-length-based analysis of Oka et al. Without such evidence, I find it difficult to assess whether the proposed framework represents a substantial theoretical advance or primarily a refinement and reinterpretation of existing findings.

2. Presentation Several figures contain multiple panels labeled A, B, and C within a single figure. Although this presentation style is not uncommon, I suggest using a more structured subfigure format, with explicit subfigure labels and corresponding descriptions in the captions. This would improve the clarity and readability of the figures.

[1] Frequency Bands in RoPE: Base Frequency and Context Length Shape the Interpolation–Extrapolation Trade-off, ICLR 2026.

Questions for authors

Q1. What new insights or contributions can be gained by defining the dependency width of the training dataset rather than using the maximum training sequence length?

Q2. Furthermore, I am not fully convinced that the experimental results sufficiently support the conclusion that RoPE frequency selection is determined by the dependency structure of the training data rather than by the maximum sequence length.

These are my two main concerns, and I would be willing to increase my score if the authors can provide convincing explanations and evidence addressing both points.

Reviewer_srVkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper claims that frequency usage in models trained with RoPE is determined by underlying characteristics in the pre-training data. They treat each frequency as a “positional lens” with a field-resolution tradeoff. Then for a given dependency profile width W for some data distribution there is an optimal admissible frequency that follows \theta^* = \pi / W. They then connect the concept of a frequency’s field/resolution to positional interpolation and claim that interpolating works only when data dependencies scale with context length. They include synthetic experiments with toy data/models, larger scale training of gpt-2 size models and analysis of existing pretrained models to support their claims.

Critical strengths
  • The paper poses an interesting problem. Most existing RoPE frequency analysis papers do not consider the underlying data distribution and mostly focus on context length or architectural choices.
  • The block-drift experiment does properly control for context length and demonstrates different frequency utilization based on just data which hasn’t been shown before.
Critical weaknesses
  • The author’s energy metric can not distinguish between positional and semantic utilization. EmE_m is a product of q and k norms and independent of both relative distance and frequency. High energy allocation to low frequencies is consistent with p-RoPE’s interpretation that channels that do not complete a rotation and appear relatively stable over training lengths are used for semantic information. The paper contains little discussion of this and does not provide any information to refute this reading of their results.
  • Theorem 1’s optimal solution is determined by its admissible set, not any of the data characteristics. θ∗=π/W\theta^* = \pi / W follows from Definition 3 alone.
Questions for authors
  • In the block-drift experiments what was the training loss? This task seems to have an optimal solution of just copying an offset of 16 before, the models were only trained for 200 steps and it's unclear if they reached this solution or not? Do the attention heads assign the majority of attention mass to an offset 16 tokens before?
  • What fraction of energy in figures 1, 3b, and 6 is there for frequencies below π/L\pi / L? As B or the number of operations in iGSM grow which bands gain more energy?
  • You use W = B for the block-drift experiments. In definition 1 you define the kernel in terms of task-relevancy. In this problem since δ\delta is fixed across B why isn't W=16?
  • Could you clarify why you used a RoPE base of 100,000100,000 for the iGSM experiments vs 10,00010,000 for the block-drift ones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.