- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper shows that RoPE models can fail beyond their training length even when every rotation plane has a short period. The proposed mechanism is that each pair of nearly equal frequencies forms a slow “beat,” and if the training window covers only part of the beat’s cycle, the beat’s envelope can reach values never seen in training. They show that frequency gaps larger than about let training pin down the attention pattern at all distances, while a smaller gap makes this unstable. They also show that frequencies with whole-number periods make attention repeat ("alias") at distances predicted by the Chinese remainder theorem.
They test their theory on small character-level models (0.46M and 3.3M parameters). A model with twelve nearly equal frequency pairs, trained on 512 tokens, fails after position 1,024. Changing only the frequency gaps at test time removes 52–86% of the damage, while an equal change that keeps the gaps removes at most 9%.
Critical strengths
- This paper presents a new explanation for RoPE length failure, with clear causal evidence. The paper shows that two nearly equal frequencies can cause failure even when every period is short. To my knowledge this view is new. On constructed frequency sets, it predicts failures that the period-based explanation cannot. In the "twin" experiments, the failure point moves with the beat length. Changing only the gaps at test time removes 52–86% of the damage, while shifting both frequencies by the same amount, which keeps the gaps, removes at most 9%.
Critical weaknesses
- It has limited practical contribution. The gap-based explanation only works on frequency sets built to show the effect, such as twelve pairs sharing one gap. For standard RoPE, the pairs with small gaps are the same slow pairs that the period-based explanation already identifies, so the experiments cannot separate the two, and the one prediction unique to the gap-based view failed.
Questions for authors
(Minor clarification) Resonance RoPE (Wang et al., 2024) already notes that integer wavelengths make the pattern repeat after the least common multiple of the wavelengths. This is similar to Proposition 2(ii) and is worth mentioning.
2 comments
Thank you for the detailed response and the new experiments. My concern about practical usefulness remains. However, the mechanism in this paper is clear shown, so I raised my score to 3.
A published modification where the period criterion flags no change: integer wavelengths raise far loss in 10 of 10 paired seeds as far attention moves to the predicted aliases (YaRN hides it); gap mechanism tested directly; Resonance RoPE credited
Thank you for the clear summary and for recognizing the causal evidence. We agree with the premise of your weakness, and we answer it with what the revision adds. New and rewritten text is in blue in the revised PDF.
Geometric RoPE: agreed, and deployed rescalings behave the same way. For RoPE the gaps scale with the frequencies, so the two accounts rank the planes alike, and the one beat-specific prediction failed; the paper keeps both. We also audited all 12 LongRoPE factor lists of six released Phi models (new Table 18): the factor never decreases from one plane to the next, so the rescaling can only widen relative gaps, and the two orderings agree (Kendall τ≥0.989). For the gap half of the account, then, no deployed spectrum we examined separates it from periods.
Where the accounts separate on a published modification: integer wavelengths. Rounding the wavelengths of period-clipped RoPE to integers, the step Resonance RoPE adds, moves no period by more than half a token, so no plane changes status under the period criterion. The arithmetic instead gives alias lattices from least common multiples, which we listed before training. With 10 seeds per arm, paired by seed (same initialization and data order):
So on a published modification the arithmetic listed before training where far attention would go, and rounding then raised the far loss in every pair, where the period criterion flags nothing. This is the alias half of the account, and the practical claim is a diagnostic of the whole spectral arithmetic, not a better encoding.
The gap mechanism is now tested directly. For every query and key beyond the window we cap each twin pair's beat envelope at its largest value over the trained offsets for that pair of tokens; no logit inside the window changes. In the base models this removes all of the excess loss over the carriers-only control at the 2,048-token beat (100%, 83 to 117% per seed, 5 seeds) and at the slower 3,072 and 4,096 beats (the capped twins end below the control), and 31 to 79% at the faster 768 to 1,536 beats; raising the envelope to its trained minimum instead removes none at any beat, and at 2,048 a uniform reduction of far twin-pair logits of the same total size removes 62% (new Table 2).
Why constructed spectra at all. They are the cleanest test: across the twin sweep every period changes by at most 3.2% while the beat period ranges from 384 to 32,768 tokens. The practical claim is narrower and we state it as such: the arithmetic is a diagnostic computable from the frequency list; it coincides with periods on geometric spectra; and it adds information when a method breaks geometric spacing, as rounding does, or moves the clipping threshold, as clipping by period does, which leaves 28 plane pairs of a Llama-2-sized spectrum beating into the partially resolved band at a period count of zero, at a cost we have not measured.
Resonance RoPE and Proposition 2(ii). You are right, and thank you. We now credit Wang et al. (2024) next to Proposition 2: the periodicity in (ii) is their observation in our notation. What we add is (iii), which partial agreements a finite window never contains, and the measurements above showing that trained heads attend there and that masking those offsets removes the rounding increment.