acceptodds
Under review as a conference paper at ICLR 2027

Beats and Aliases Can Break Rotary Length Extrapolation

Abstract

Rotary position encodings degrade beyond their training length, and a common account blames planes whose period exceeds the training window. We show that a spectrum whose every period is below 13 tokens can still collapse: what is missing is not another frequency but a difference of frequencies. For fixed content a rotary logit is a trigonometric polynomial of the offset, and in that fixed-kernel problem classical super-resolution bounds say that gaps above about let the training offsets determine it stably, while a gap with can raise the worst-case error near the half-beat offset by a factor . In a 0.46M-parameter character-level model, twelve pairs detuned by cycles per token make training on 512 tokens collapse after position 1,024; across a sweep of eight beat periods the onset tracks the beat, and wherever a model has failed, stretching only the gaps at inference removes 52–86% of the damage while an equal shift that keeps them removes at most 9%. Both recur at 3.3M parameters, after ten times longer training and on a second corpus, although in the first two the collapse begins earlier than the beat predicts. In spectra of harmonics of coprime periods, 59% of the strongest out-of-window attention weights fall where the Chinese remainder theorem predicts (base rate 4%), and masking them helps at least three times more than random masking. For RoPE's own geometric spectrum gaps and periods rank the planes alike, our experiments do not separate them, and a beat-specific prediction fails. Spectral arithmetic is an additional diagnostic for rotary extrapolation, computable before training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rotary position embedding, RoPE, length extrapolation, long context, positional encoding, transformers, super-resolution, spectral analysis, Chinese remainder theorem, language modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_yvX1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that RoPE models can fail beyond their training length even when every rotation plane has a short period. The proposed mechanism is that each pair of nearly equal frequencies forms a slow “beat,” and if the training window covers only part of the beat’s cycle, the beat’s envelope can reach values never seen in training. They show that frequency gaps larger than about 1/L1/L let training pin down the attention pattern at all distances, while a smaller gap makes this unstable. They also show that frequencies with whole-number periods make attention repeat ("alias") at distances predicted by the Chinese remainder theorem.

They test their theory on small character-level models (0.46M and 3.3M parameters). A model with twelve nearly equal frequency pairs, trained on 512 tokens, fails after position 1,024. Changing only the frequency gaps at test time removes 52–86% of the damage, while an equal change that keeps the gaps removes at most 9%.

Critical strengths
  1. This paper presents a new explanation for RoPE length failure, with clear causal evidence. The paper shows that two nearly equal frequencies can cause failure even when every period is short. To my knowledge this view is new. On constructed frequency sets, it predicts failures that the period-based explanation cannot. In the "twin" experiments, the failure point moves with the beat length. Changing only the gaps at test time removes 52–86% of the damage, while shifting both frequencies by the same amount, which keeps the gaps, removes at most 9%.
Critical weaknesses
  1. It has limited practical contribution. The gap-based explanation only works on frequency sets built to show the effect, such as twelve pairs sharing one gap. For standard RoPE, the pairs with small gaps are the same slow pairs that the period-based explanation already identifies, so the experiments cannot separate the two, and the one prediction unique to the gap-based view failed.
Questions for authors

(Minor clarification) Resonance RoPE (Wang et al., 2024) already notes that integer wavelengths make the pattern repeat after the least common multiple of the wavelengths. This is similar to Proposition 2(ii) and is worth mentioning.

2 comments
AuthorsOpenReview ↗

A published modification where the period criterion flags no change: integer wavelengths raise far loss in 10 of 10 paired seeds as far attention moves to the predicted aliases (YaRN hides it); gap mechanism tested directly; Resonance RoPE credited

Thank you for the clear summary and for recognizing the causal evidence. We agree with the premise of your weakness, and we answer it with what the revision adds. New and rewritten text is in blue in the revised PDF.

Geometric RoPE: agreed, and deployed rescalings behave the same way. For RoPE the gaps scale with the frequencies, so the two accounts rank the planes alike, and the one beat-specific prediction failed; the paper keeps both. We also audited all 12 LongRoPE factor lists of six released Phi models (new Table 18): the factor never decreases from one plane to the next, so the rescaling can only widen relative gaps, and the two orderings agree (Kendall τ≥0.989\tau\ge0.989). For the gap half of the account, then, no deployed spectrum we examined separates it from periods.

Where the accounts separate on a published modification: integer wavelengths. Rounding the wavelengths of period-clipped RoPE to integers, the step Resonance RoPE adds, moves no period by more than half a token, so no plane changes status under the period criterion. The arithmetic instead gives alias lattices from least common multiples, which we listed before training. With 10 seeds per arm, paired by seed (same initialization and data order):

  • far loss on [4L,8L)[4L,8L) rises in 10 of 10 pairs, by 0.072 bpc on average (95% CI 0.024 to 0.120; Wilcoxon p=0.002p=0.002), while in-window loss is unchanged;
  • the predicted alias offsets receive 18% of the second layer's far attention, against 7% without rounding (base rate 8%);
  • masking them in both models removes the increment (0.068 to -0.004 bpc), and at matched attention the alias mask leaves 0.032 bpc of it against 0.044 for a mask of other far keys, less in 10 of 10 pairs;
  • with YaRN at inference, which Resonance RoPE pairs rounding with, the increment is no longer detectable (-0.008 bpc): the cost belongs to the rounding step, and YaRN hides it at our scale.

So on a published modification the arithmetic listed before training where far attention would go, and rounding then raised the far loss in every pair, where the period criterion flags nothing. This is the alias half of the account, and the practical claim is a diagnostic of the whole spectral arithmetic, not a better encoding.

The gap mechanism is now tested directly. For every query and key beyond the window we cap each twin pair's beat envelope at its largest value over the trained offsets for that pair of tokens; no logit inside the window changes. In the base models this removes all of the excess loss over the carriers-only control at the 2,048-token beat (100%, 83 to 117% per seed, 5 seeds) and at the slower 3,072 and 4,096 beats (the capped twins end below the control), and 31 to 79% at the faster 768 to 1,536 beats; raising the envelope to its trained minimum instead removes none at any beat, and at 2,048 a uniform reduction of far twin-pair logits of the same total size removes 62% (new Table 2).

Why constructed spectra at all. They are the cleanest test: across the twin sweep every period changes by at most 3.2% while the beat period ranges from 384 to 32,768 tokens. The practical claim is narrower and we state it as such: the arithmetic is a diagnostic computable from the frequency list; it coincides with periods on geometric spectra; and it adds information when a method breaks geometric spacing, as rounding does, or moves the clipping threshold, as clipping by period does, which leaves 28 plane pairs of a Llama-2-sized spectrum beating into the partially resolved band at a period count of zero, at a cost we have not measured.

Resonance RoPE and Proposition 2(ii). You are right, and thank you. We now credit Wang et al. (2024) next to Proposition 2: the periodicity in (ii) is their observation in our notation. What we add is (iii), which partial agreements a finite window never contains, and the measurements above showing that trained heads attend there and that masking those offsets removes the rounding increment.

Thank you for the detailed response and the new experiments. My concern about practical usefulness remains. However, the mechanism in this paper is clear shown, so I raised my score to 3.

Reviewer_jKh3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates why length extrapolation in RoPE cannot be explained solely by the periods of individual rotary planes. The authors express the attention logit for fixed content as a trigonometric polynomial of the offset and use classical super-resolution results to analyze frequency-node separation: even when all individual periods are short, nearby frequencies can produce a slow beat envelope insufficiently covered by the training window; commensurate spectra can redirect attention to predictable CRT aliases. When the spectrum contains an unrotated content node, the period criterion becomes a special case of the node-separation criterion.

Experiments with synthetic twin and residue spectra provide evidence and interventions supporting these mechanisms. However, the conclusions primarily concern small character-level models. For standard geometric RoPE, frequency gaps and periods yield the same ordering of planes, and no additional predictive or corrective benefit has been demonstrated.

Critical strengths
  1. A controlled counterexample to explanations based on individual periods. Every plane in the twin spectra has a period shorter than 13 tokens, and each upper-frequency plane’s period is at most 3.2% shorter than its carrier’s. Adjusting the nearby-frequency gaps substantially changes out-of-window degradation while individual periods remain nearly unchanged, showing that individual periods alone do not fully characterize extrapolation risks.

  2. Convincing mechanistic experiments. The beat-period sweep, distant attention peaks, and inference-time stretch interventions targeting twin gaps provide mutually supporting evidence. In evaluation ranges classified as failed in the base twin experiments, stretch removes 52%–86% of the damage, whereas gap-preserving shifts remove at most 9%. Failure and repair also recur on another corpus, in a larger model, and after longer training. The limited benefit of attention temperature for twins further suggests that this failure differs from ordinary attention dilution.

  3. Testable CRT predictions. The strongest distant attention weights in residue models are substantially enriched at predicted aliases (59% on average, versus a baseline of approximately 4%), supported by alternative-modulus controls and randomization tests. Alias masking removes approximately 26%–42% of the distant damage, providing roughly 3–5 times the benefit of masking an equal number of random offsets. For rounded RoPE, applying the same alias mask to both models also reduces the average additional damage introduced by rounding from approximately 0.087 bpc to 0.001 bpc, supporting the involvement of aliases in this additional damage.

  4. Clear reporting of scope and limitations. The authors distinguish fixed-kernel theory from trained-model behavior and explicitly report the negative results for standard RoPE, failures of onset predictions, and limitations of experimental scale.

Critical weaknesses
  1. External validity remains unclear. Experiments are limited to character-level models with 0.46M/3.3M parameters, with approximately 488, 1,464, and 4,882 training steps in the base, larger-model, and longer-training settings, respectively. The models use little context beyond a few dozen characters, so the metrics primarily measure interference from distant tokens with local prediction rather than whether models can use distant information. Evaluations on pretrained models and retrieval or delayed-dependency tasks are missing; the larger-model and longer-training checks also each use only one seed.

  2. The connection between fixed-kernel theory and network damage remains insufficient. The two theorems characterize stable recovery or worst-case ill-conditioning under noisy observations, rather than directly bounding trained-model damage. Theorem 1 provides no stability guarantee for standard RoPE or the main failing twin configurations, and the training-length sweep shows no clear resolution threshold. Onset predictions further depend on assumptions about envelope-magnitude readout, damage caused by magnitude overshoot, and particular phases. Earlier failures in the larger model and after longer training show that these assumptions are insufficient.

  3. CRT-specific causality requires stricter controls. The so-called attention-matched control matches only the number of masked offsets, not the attention mass actually removed, so differences in benefit may still reflect differences in masking strength. The additional damage from rounding also varies substantially across seeds. Although alias masking largely eliminates the average increment, its specificity still requires a mass-matched control. Moreover, the experiment tests only rounding, without the interpolation used alongside it in Resonance RoPE, and therefore cannot assess the complete method.

  4. Predictive utility and incremental contribution remain limited. For standard RoPE, beat-based clipping does not outperform period-based clipping, and the beat-specific repair prediction fails. The practical cost of nearby-frequency risks remaining after period clipping has not been established. The correlation between risk counts and damage is based on only 10 spectra, including deliberately constructed twins. Conceptually, the perspective primarily extends the period criterion to all node pairs, using classical theoretical tools. At present, it is better viewed as a risk-screening approach than a reliable performance-prediction criterion.

Questions for authors
  1. Can the authors demonstrate the complete chain of “computed nearby-frequency risk → actual use of the beat by the model → damage repaired by a targeted intervention” in a sufficiently trained pretrained model or a nonstandard spectrum used in practice? Do such examples exist in spectra produced by dimension-wise rescaling methods such as LongRoPE?

  2. Can stretch improve passkey retrieval, needle, or delayed-copy tasks while preserving short-context performance? This would help distinguish reducing distant interference from restoring long-range capabilities.

  3. Why do phases concentrate near +π/2+\pi/2? Would changing data locality, adding explicit distance decay, or intervening on twin-pair phases while preserving their gaps correspondingly change distant attention peaks and failure onsets?

  4. When the training window covers more than half a beat period, the stated half-cycle interval need not coincide with the set that strictly exceeds the maximum training-window magnitude. For example, with T=768T=768, L=512L=512, and ϕ=π/2\phi=\pi/2, the integer offsets exhibiting strict overshoot are ⋃n≥0512+768n,…,640+768n\bigcup_{n\ge0}{512+768n,\ldots,640+768n}, which excludes the measured onsets at 704, 1536, and 2240. Given the sensitivity to phase, can the authors recheck this using measured per-head/per-pair phases? Please also explain the mapping between overshoot sets defined over offsets and 64-token onset bins indexed by query position.

  5. Can the authors add masking controls matched by removed attention mass and distinguish the additional damage introduced by rounding from overall out-of-window damage? Do alias enrichment and intervention effects persist when combined with YaRN and fine-tuning?

2 comments
AuthorsOpenReview ↗

Both corrections made: phase-correct overshoot sets with a one-sided onset bound; an envelope cap that leaves the window untouched (all excess removed at slow beats, part at fast ones); mass-matched masks; ten paired rounding seeds

Thank you for an unusually precise review. You were right about the overshoot set and the masking control; both are corrected (blue in the revised PDF).

Q4. Overshoot set and onsets. Your set is exact for ϕ=π/2\phi=\pi/2, which our text assumed for every beat. Measured per pair, the learned phase depends on the beat (0.89 to 1.55 rad; Table 10), and in 80 to 96% of combinations it puts the envelope's trained maximum at Δ=0\Delta=0, so the strict set is (u/ε−ϕ/(πε), u/ε)(u/\varepsilon-\phi/(\pi\varepsilon),\ u/\varepsilon); for the 768 beat its first element is 551, not 512. You are also right about the mapping: a query at mm sees every offset up to mm, so the proposition only bounds the onset from below. Every onset of the base sweep comes after the first overshoot offset at the mean phase (the closest by 46 positions) under all nine onset definitions, and after the 10th percentile of the per-pair first overshoot. Interval claim withdrawn (Sec. 4, Fig. 2c, App. E).

W2/Q3. Theory to trained model. We now test the envelope hypothesis directly (new Table 2): beyond the window we cap each twin pair's ∣A∣|A|, per query and key, at its maximum over the trained offsets; no logit below offset LL changes. Of the excess over the carriers-only control on [4L,8L)[4L,8L) it removes 31, 58 and 79% at the 768, 1,024 and 1,536 beats, 100% at 2,048 (83 to 117% per seed, 5 seeds), and all of it at 3,072 and 4,096 (the capped twins end 0.08 and 0.16 bpc below the control). Raising the undershoot instead (floor) removes none at any beat, and at 2,048 a shrink of the same total size over all far twin-pair logits removes 62%. So in the base models the overshoot carries all of the failure where the window covers at most a quarter of the beat cycle, and part of it elsewhere. As the overshoot is also where ∣A∣|A| is largest, the cap alone cannot separate leaving the trained range from being large.

On earlier failures at scale: both checks now have three seeds (Tables 14, 15). The cap removes 96 and 95% of the 3.3M model's excess on [2L,4L)[2L,4L) and [4L,8L)[4L,8L), and 97, 69 and 56% of the 10x model's on [L,2L)[L,2L), [2L,4L)[2L,4L), [4L,8L)[4L,8L): the overshoot carries the 3.3M failure and, after longer training, most of the early excess but only part of the far one. Both still collapse before the bound at their mean phase (3.3M: 1,024 against 1,130, seed 0 earliest; 10x: 704 against 930), and we keep that as a failed prediction; after longer training more pairs overshoot early (20% by offset 704, against 2 to 15% in the five base seeds), a candidate cause we have not tested. Theorem 1 is the resolved regime and covers neither RoPE nor the twins; we agree the LL-sweep is not a threshold test. Why near π/2\pi/2 we can describe, not derive (App. E); rotating the twin phases at all offsets is not clean (in-window loss 2.91 to 4.48 bpc), and we did not rotate only beyond LL, vary locality or add distance decay.

W3/Q5. Masking and rounding. Agreed: the old control matched the count, not the mass. The new one removes the same attention from aliases and from other far keys per layer, query, head and 256-offset bin (Table 17). In the residue models the two help equally (0.063 vs 0.064 bpc), so we withdrew the "beyond equally prominent offsets" claim; what survives is location: aliases, 11% of far offsets, receive 40% of layer-2 far attention (10% in the stratified model). Rounding, 10 seeds per arm paired by seed: far loss rises in 10 of 10 pairs, by 0.072 bpc (95% CI 0.024 to 0.120; Wilcoxon p=0.002p=0.002; per pair 0.004 to 0.185), in-window unchanged. On the masking windows the increment is 0.068 bpc, and -0.004 with all aliases masked; at matched attention the alias mask leaves 0.032 and a mask of other far keys 0.044, less in 10 of 10 pairs. (Within the rounded model alone 0.062 vs 0.045, but 0.026 vs 0.022 without rounding, hence the increment.) With YaRN at inference the increment is not detectable (-0.008 bpc). We did not fine-tune.

W1/Q2. External validity. A delayed-copy probe: an exact copy of the next 48 characters, 64 to 448 positions back, lowers the loss by at most 0.08 bpc in all 7 models probed (including 10x training and planted-repeat training). These models cannot retrieve, so restoring retrieval cannot be tested at our scale; we now say the metric measures interference.

Q1. LongRoPE; complete chain. All 12 LongRoPE factor lists of six released Phi models never decrease from plane to plane, so rescaling only widens relative gaps (Kendall τ≥0.989\tau\ge0.989; Table 18). For the beat mechanism we found no deployed example. For the alias mechanism the chain holds for a published modification, integer wavelengths, in our small models: lattice listed before training, attention moves onto it (18% vs 7% of layer-2 far attention), and masking it removes the rounding increment.

W4. Agreed: risk screening, not loss prediction, and we have not shown what the close pairs left after period clipping cost; the paper says so.

Thank you for the thorough rebuttal. The new envelope-cap experiment and the mass-matched masking control resolve my concerns about the overshoot set and the masking control. However, the main limitations remain: external validity, and the absence of a deployed spectrum where either hazard has been shown to matter. I will therefore keep my score.

Reviewer_ZXTYOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper assesses the detailed behavior of RoPE, which has been commonly used in major LLMs. While RoPE is a reasonable way of capturing multiple auto-correlation lengths over tokens in the transformer algorithm, it is known that catastrophic error can happen if the input context length is greater than the design length L. This paper investigated this issue and identified that interference between dimensions having close wavelengths is a major factor of it. Intuitively, it can be viewed as beats with a certain long wavelength.

Critical strengths
  • Identified the main mechanism of RoPE failure to be a beat phenomenon for the first time.

  • Provided a framework predicting the occurrence of beats and hence where RoPE failure begins.

  • Provided a quick remedy to increase the gap between the wavelengths to recover more than half of the performance degradation.

Critical weaknesses
  • While the identified mechanism is highly valuable, a fundamental solution to eliminate the failure is yet to be proposed.

  • Empirical evaluations were made on a relatively small scale, which is significantly smaller than commercial LLMs.

  • The argument is largely anecdotal, and the main conclusion relies on an unverified assumption of the envelope hypothesis. There is no clear organic connection between the presented theorems and the speculated mechanism.

Questions for authors

Please comment on the second and third points in the weakness section.

2 comments
AuthorsOpenReview ↗

The envelope hypothesis tested directly: capping each beat envelope at its trained maximum, with no in-window logit changed, removes all excess at the 2,048-token and slower beats and part of it at faster ones; scale limits stated

Thank you for the positive assessment. We answer your second and third points with new evidence; new and rewritten text is in blue in the revised PDF.

W3. The envelope hypothesis and the link to the theorems. The chain is: Theorem 2 shows which spectra contain kernels the training window cannot pin down (a node pair with εL≤1/2\varepsilon L\le1/2); Proposition 1 shows how such a kernel leaves its trained range (the envelope ∣A∣|A| of the pair takes values beyond those seen on the window); and the hypothesis says a trained head reads ∣A∣|A|, so that an envelope larger than any trained value is what hurts. The submitted paper tested this only through its predictions. We now test it directly (new Table 2). For every query and key beyond the window we cap each twin pair's ∣A∣|A| at its largest value over the trained offsets for that pair of tokens, computed exactly from the measured query and key planes. No logit at an offset below LL changes, so the in-window loss is identical. Share of the excess over the carriers-only control on [4L,8L)[4L,8L) removed:

beat (tokens)seedsexcess, bpcafter the capremoved by the capremoved by the floor (control)
76830.300.2131%0%
1,02430.670.2958%0%
1,53630.940.2079%-2%
2,04850.670.01100% (83 to 117% per seed)-6%
3,07230.40-0.08121%-20%
4,09630.10-0.16297%-159%

The floor raises the undershoot to the trained minimum; a share above 100% means the capped twins end below the control. At 2,048 tokens a shrink that removes the same total logit magnitude from all far twin-pair logits removes only 62%. So in the base models the overshoot carries all of the failure where the window covers at most a quarter of the beat cycle, and part of it where it covers more. This measured link, not a theorem, is what connects the fixed-kernel analysis to the trained model. One caveat, now in the paper: the overshoot is also where ∣A∣|A| is largest, so the cap alone does not separate leaving the trained range from being large.

We also corrected where the overshoot begins: the learned phase differs between beats (0.89 to 1.55 rad), and since a query sees every offset up to its own position, the proposition bounds the onset only from below. Every onset of the base-model sweep respects this bound (Fig. 2c, Table 10); in the 3.3M-parameter and 10x-longer-trained models the collapse starts before it, which the paper lists as a failed prediction.

On the argument looking anecdotal: the cap now covers six beats with three to five seeds each, the integer-wavelength result below has ten paired seeds, and the scaled and longer-trained checks three seeds each.

W2. Scale. We agree, and three things changed. (i) Both scale checks now have three seeds. In the 3.3M-parameter model the cap removes 96 and 95% of the excess on [2L,4L)[2L,4L) and [4L,8L)[4L,8L), so the overshoot carries its failure; after ten times longer training it removes 97, 69 and 56% on [L,2L)[L,2L), [2L,4L)[2L,4L) and [4L,8L)[4L,8L): most of the early excess, but only part of the far one. (ii) A delayed-copy probe shows what our metric measures: an exact copy of the next 48 characters, 64 to 448 positions back, lowers the loss by at most 0.08 bpc in every model we probed, so these models do not retrieve, and we claim that the hazard exists and can be intervened on, not that it dominates in pretrained LLMs. (iii) For standard geometric RoPE the gaps are proportional to the frequencies, so on a pretrained RoPE model the two accounts rank the planes identically and differ only in the threshold, where our one test went against the beat account; we do not show that beats cause RoPE's failure. The accounts separate on spectra that break geometric spacing. In one published modification, integer wavelengths (Resonance RoPE's rounding step), rounding raises the far loss in 10 of 10 paired seeds (by 0.072 bpc; Wilcoxon p=0.002p=0.002) although no period moves by half a token, far attention moves to the predicted alias offsets, and masking them removes the increment; with YaRN at inference it is no longer detectable.

W1. A fundamental solution. We do not claim one, and the paper says so. For the twins, changing only the gaps (stretch) repairs the failure, and capping the envelope repairs it fully at slow beats and partly at fast ones in the base models; for RoPE, clipping by period and YaRN already do what our account would recommend. What the account offers is a diagnostic, computable from the frequency list before training, that sees hazards periods cannot, such as where integer wavelengths send far attention.

Thank you again for your support.

Reviewer_ZXTYOpenReview ↗

Thanks

Thank you for providing the additional details. I like your work and maintain the current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.