We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis is *only* related to the length of the context instead of its specific content. We prove that as context length increases, RoPE-based attention becomes unpredictable and loses two properties that are central to its effectiveness. First, it loses its locality bias: RoPE is no more likely to favor nearer positions than substantially farther ones. Second, it loses consistency in token relevance: a key vector that receives a higher attention score than an alternative at one position may receive a lower score at another. With a longer context, both failure modes approach a probability of 0.5, no better than random guessing. We further prove that the attention score can remain unchanged when a key token is moved to a different position, or even replaced by a different token, indicating a failure to distinguish positions or tokens. Increasing the RoPE base hyperparameter, a common practice in today’s long-context models, helps distinguish different tokens, but inevitably sacrifices the ability to distinguish positions, as modifying the RoPE base merely trades off the two objectives but cannot preserve both at the same time. Our empirical analysis shows that multi-head, multi-layer architectures are insufficient to overcome these limitations. Our findings suggest that fundamentally new mechanisms for encoding position and token order may be needed in future Transformer long-context language models.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.Reviewer_RTiVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary
This paper studies the weaknesses of RoPE from both empirical and theoretical perspectives. The authors prove that RoPE has some undesired properties when modeling positional bias, i.e., remote tokens should receive lower attention scores, and extracting the desired tokens. The theoretically proven properties are validated by the empirical results.
Critical strengths
The authors study a very important problem: the weaknesses of RoPE, which is widely adopted in LLMs.
The weaknesses identified by the authors are verified via both theoretical and empirical results.
Critical weaknesses
The authors adopt the Gaussian approximation for the attention logit distribution. However, such an approximation is not justified either theoretically or empirically. For example, theoretically, the authors should establish the central limit theorem or concentration for it. For empirical verification, the authors should conduct hypothesis testing.
The authors implicitly assume that M is overlarge compared to B in Appendix B. In other words, the authors just consider for example, the lowest freq be 1/106 but the value of M can be 109, which is an unrealistic assumption.
When examining the appendix, I find that the authors have many implicitly stated assumptions. This makes the correctness and plausibility of the results hard to evaluate. Please explicitly state all the assumptions.
The position bias of long-context ability in Fig 4 is well known. Please cite the related papers there.
The logic of the paper is not very smooth. For example, the authors state that “A good positional encoding is designed for position identification, but should not negatively affect token identification.” The attention score always consists of two parts: the position bias and the semantic information. The positional embedding can only contribute to the first part. Thus, this is a fundamental trade-off. All existing positional embeddings, except NoPE (No positional embedding), negatively affect token identification, e.g., Alibi, variants of RoPE, sinusoidal, due to the explicit bias injected into the tokens.
Questions for authors
Please explicitly state all assumptions for clear evaluations.
TL;DR. The Gaussian approximation is now justified by the Salem-Zygmund central limit theorem for lacunary sums and verified empirically (four probabilities predicted within 0.01). Our scope is M≤cB, and the revised Fig. 3b stops at M=B. The assumptions are collected in a box at the start of Appendix C. A new proposition shows that additive encodings such as ALiBi and T5 bias cannot show token inversion or aliasing, and a from-scratch comparison of RoPE, ALiBi and NoPE on the indexing task shows RoPE performing best.
Thank you for your review. The proof corrections raised by the other reviewers are in the common response.
W1. Gaussian approximation. We agree the CLT was not justified correctly.
The RoPE product S(m)=∑nancos(mθn+ϕn) is a lacunary trigonometric sum (Hadamard gap ratio q=B1/h>1), and regular amplitudes give the Lindeberg condition. The Salem-Zygmund CLT for non-integer frequencies (Proposition B.2) then applies without independence among summands; the limit is in the number of summands h, not in M.
Empirical check. On synthetic data the Kolmogorov distance between the RoPE product over distances and the normal surrogate is 0.01 to 0.04 for random phases and 0.05 to 0.13 for aligned phases. On 6400 query-key pairs from real text over all 32 heads of Llama 3.1-8B layer 0 at 4K, 32K and 128K, the median distance is 0.04, and the surrogate predicts the exhaustive probability of each of the four events within 0.01.
The limit of Theorem 2 (pigeonhole, Theorem B.7) and the short-context bounds of Theorems 3 and 4 (Cantelli, Proposition B.8) need no distributional assumption.
Head dimension. For aligned phases, once h exceeds about 3lnB to 5lnB (distance 0.014 at h=64, 0.073 at h=512) the single normal degrades and the mixture of Gaussians (Definition B.4) takes over. For generic phases and typical h, B the error stays small.
Hypothesis tests: with thousands of distances per pair, any normality test rejects for distances above about 0.01, so we report the distance and the induced probability error instead.
W2. M overlarge compared to B. Our scope is the opposite: footnote 3 assumes M≤cB, and the high-frequency components only need Mθn≫1. The original Fig. 3b was confusing. The curves are truncated at M=B and the scope is stated in assumption A2.
W3 and Q1. Implicit assumptions. We agree. The revision adds an assumption box at the start of Appendix C, and each theorem states the assumptions it uses:
0. normal surrogate with measured error (W1);
regular amplitudes (closed forms only; common response);
M≤cB (W2);
distance m uniform over the interval (for probabilities);
non-degenerate inputs (amax>0; ah−1>0 for RoPE decay; k=k′ for token inversion);
regular difference amplitudes for token inversion and aliasing (closed forms only);
precision model ε=c⋅2−fmax(d,∑nan) (aliasing);
removed: linear independence of frequencies over the rationals.
The assumptions in Appendix C.4 that two RoPE products are independent and that an is the same for all pairs are removed.
W4. Position bias in Fig. 4. The revision discusses position bias and cites the lost-in-the-middle, RULER and BABILong studies in Appendix D.2.3. Fig. 4 concerns position inversion, where a random farther location scores higher than a nearer one, whereas position-bias studies concern certain locations favored by a trained head. A new index-position sweep in Appendix D shows that indexing accuracy drops to random with the index and never recovers, unlike the U-shaped retrieval curves of position-bias studies.
W5. Is the trade-off fundamental? Every encoding other than NoPE adds position-dependent terms, but this need not harm token identification. For any additive bias ⟨q,k⟩+b(m) (ALiBi, T5 bias), the score difference of two keys is ⟨q,k1−k2⟩, independent of distance, so token inversion and aliasing cannot occur. The negative effect on token identification arises from multiplicative distance terms as in RoPE, not from additive ones as in ALiBi. Since RoPE is the de facto standard, we study this trade-off as its property and limitation. The revision adds a proposition in Appendix C.5 and a from-scratch comparison of 4-layer transformers differing only in positional encoding (RoPE bases 104, 5×105; ALiBi; NoPE; three seeds) on the indexing task: RoPE beats ALiBi, which beats NoPE, and at 2048 elements RoPE scores 0.93 to 1.00 while ALiBi and NoPE are near random.
Presentation. The revision adds the assumption box, states assumptions in each theorem, unifies the token-inversion definition in Section 4 and Appendix C (Def. C.3), cuts Fig. 3b at M=B, and rewords the trade-off discussion.
This paper investigates potential limitations of RoPE in distinguishing positions and tokens in long contexts. The authors express the attention score for a fixed query–key pair as a sum of rotary components and use a Gaussian approximation to analyze its distribution over randomly sampled relative distances. They examine four phenomena: position inversion, position aliasing, token inversion, and token aliasing. The paper further argues that increasing the RoPE base creates a trade-off between positional and token discrimination.
The experiments include an analysis of a single attention head in Llama-3.1-8B, score-collision statistics under different numerical precisions, and a list-indexing task evaluated on six complete models. Based on these results, the authors argue that RoPE has intrinsic limitations that cannot be resolved simply by increasing context length or adjusting its parameters.
Critical strengths
Analyzing positional and token-related phenomena within a unified framework is informative. The four-way categorization helps distinguish changes in attention-score ordering from numerical collisions and provides concrete quantities for studying the effects of the RoPE base.
The observations concerning numerical precision and complete-model behavior are useful. The BF16–FP16 comparison illustrates how finite precision affects score collisions. The list-indexing results across six models also demonstrate that a long advertised context window does not guarantee reliable positional access.
Critical weaknesses
The proofs of the central probability claims contain substantive gaps. Theorem 6 combines collision events sharing the same reference position as though they were independent, although they jointly depend on the reference score. The stated exponential convergence lower bound is therefore not established. The variance expression in Theorem 7 is also inconsistent with the preceding amplitude transformation. These issues directly concern the paper’s principal probability claims and require correction and reassessment of the associated conclusions.
The single-head phenomena do not sufficiently support the strong conclusions about complete models. Equal scalar attention scores do not imply that inputs are indistinguishable through value vectors, other heads, or subsequent layers. Non-monotonic attention scores likewise do not establish a failure of positional encoding. The indexing experiment lacks matched positional-encoding controls to rule out factors such as counting ability or training distribution. Although the limitations acknowledge the absence of direct causal evidence, the abstract’s claim that multi-head, multi-layer architectures cannot overcome these limitations remains too strong.
Questions for authors
Dependence in Theorem 6: Collision events comparing multiple candidate positions with the same reference position are not independent. Can the probability of having at least one collision be rederived by conditioning on the reference score? Does the claimed exponential convergence lower bound remain valid after this correction?
Variance in Theorem 7: Applying the approximation in Appendix B to the difference amplitudes derived in Appendix C.3 appears to give:
σ2≈2∑n<λ(M)an2sin2(2ϕn).
This differs from the expression in Theorem 7. For example, when all phases equal 180 degrees, the stated variance becomes zero even though the corresponding difference still varies with position. Is this a typographical error, and how would its correction affect the probability estimates and associated conclusions?
Applicability of the Gaussian approximation: At finite context lengths and practical head dimensions, how do dependencies among frequency components affect the approximation error? Which probability claims remain valid when a small number of components have substantially larger amplitudes, and what theoretical or empirical evidence supports their applicability?
Attribution of complete-model failures: Are there experiments that vary the positional-encoding mechanism while controlling other conditions and directly connect indexing failures to the four phenomena defined in the paper? How does the existing evidence rule out explanations such as counting limitations or insufficient task training and support general conclusions about multi-head, multi-layer models?
TL;DR. We confirm both proof errors. Theorem 6 is rederived by conditioning on the reference score (polynomial instead of exponential rate), and Theorem 7 is restated with the exact mean and variance, so its limit at M=Θ(B) is below 1/2 while the trends in M and B and the Section 4 trade-off are unchanged.
Thank you for the questions on proof errors and soundness. Both errors are fixed (common response; proofs in Appendix C).
Q1 and W1 (part 2). Event dependence in Thm 6.
Rederive by conditioning on the reference score?Yes. Conditioning on Sm1 makes the other scores independent, so p(M)=1−E[(1−Pr(∣D∣<ε))M−1] with D=Sm1−Sm2. The revision gives a lower bound valid for every threshold t>0, and p(M)→1 follows.
Exponential convergence still valid?No. The rate is polynomial: 1−p(M)=M−1+o(1). In practice the probability still grows fast: on all 32 heads of Llama 3.1-8B layer 0 the exhaustive value is 0.9995 at M=4096 (Table 5).
Q2 and W1 (part 3). Variance in Thm 7.
Typographical error?Yes, and your expression is correct. Thm 7 is restated, and the mean is corrected as well.
Effect on results.The probability changes. With random amplitudes and phases (B=105, h=64, M=65536) the original expression gives 0.382, exhaustive counting 0.336 and the revised formula 0.322 (Table 16). The limit changes. At M=Θ(B), p(M)=Φ(−Θ(h/lnB))<1/2; a random model with B=104, h=64 gives 0.38 to 0.41 (Table 17). The trends do not change. For M≤B, on average over keys, the probability increases with M and decreases with B, so the Section 4 trade-off holds.
With random amplitudes and phases (h=64; B=104 to 106), p(M) reaches 0.3 at 51% to 71% of B, and the window inconsistency Q1024 (Definition B.17) reaches 1/2 at 7% to 10% of B (Tables 10, 17), so token inversion significantly reduces the usable context length.
Q3 and W1 (part 1). Gaussian approximation. We agree the original proof did not establish the CLT. The RoPE product S(m)=∑nancos(mθn+ϕn) is a lacunary trigonometric sum (Hadamard gap ratio q=B1/h>1), and regular amplitudes give the Lindeberg condition. The high-frequency part then satisfies the Salem-Zygmund CLT (Proposition B.2), which needs no independence among summands; integer-distance accuracy is measured in Appendix B.3. Over short sub-intervals the low-frequency part is nearly constant and the high-frequency part is pseudorandom (Definition B.12), so the law is a mixture of Gaussians (Definition B.4); for readability we replace the low-frequency part by its mean and use one Gaussian.
Dependencies among components. The CLT needs no independence, and rational dependence among frequencies does not change the Gaussian limit for a cosine sum (remark after Proposition B.2). The error comes from the low-frequency components (worst case: aligned phases, Fig. 2(c)).
A few large amplitudes. All quantitative claims remain valid; the assumption is used only in closed forms, where h is replaced by heff=(∑nan)2/∑nan2 (Proposition B.9). Inversion probabilities increase and aliasing probabilities decrease when fewer components are non-negligible (common response).
Evidence. On 6400 real pairs over all 32 heads of Llama 3.1-8B layer 0 at 4K to 128K, the median distance is 0.04 and the four event probabilities are within 0.01 of exhaustive counting.
Q4 and W2. Attribution of full-model failures.
Controlled experiments. We train 4-layer transformers differing only in positional encoding (RoPE bases 104, 5×105; ALiBi; NoPE; three seeds) from scratch on the indexing task. RoPE beats ALiBi, which beats NoPE; at 2048 elements RoPE scores 0.93 to 1.00 and ALiBi and NoPE are near random.
Direct connection? No. We do not intervene on trained models. The causal claims in the abstract and Sections 5 and 6 are removed, and the results are presented as consistent with the single-head analysis.
Counting limitations. A labeled-list control keeps large models at 0.73 to 1.00 at 4096 elements versus chance unlabeled. The failure is locating a position by its index.
Insufficient training. No evidence rules it out; training on the task extends effective context (0.68 at 64 elements after 20k steps, 0.93 to 1.00 at 2048 after 150k).
General conclusions on full models? No. The generalization is now a hypothesis, and aliasing statements refer to one head with the other keys and values fixed. For a single head, though, different value vectors do not make aliased positions distinguishable: swapping the two keys leaves the head output unchanged (Section 3, Fig. 1).
The paper studies how the unnormalized attention score changes with the distance between a fixed query and key. A normal approximation to the sum of rotary components is used to estimate the probabilities of four events: inversions of positional preference and key ranking, and numerical logit collisions across positions and keys. These events are interpreted as an unavoidable trade-off between distinguishing positions and distinguishing tokens. Validation includes an analysis of one attention head in Llama 3.1-8B and a direct list-indexing task evaluated on six off-the-shelf models.
Critical strengths
A substantive attempt to build a unified theory of long-context RoPE. The paper connects rotary frequency structure, context length, and numerical precision within a common probabilistic framework for four explicitly defined events. This is an ambitious and valuable direction: a quantitative account could help explain when positional and content information remain usable as context grows. The formulation makes the proposed mechanisms explicit and testable, and its attention to numerical precision adds practical relevance to the geometric analysis. These are worthwhile contributions even though the guarantees require further justification.
Concrete empirical evidence of limitations in the evaluated settings. The curves and collision maps demonstrate specific effects in the selected Llama head, and the indexing experiment reveals substantial errors across six models under the specified response protocol. These observations motivate further investigation, although they do not establish the proposed causal explanation. The extensive appendices and explicit acknowledgment of this limitation help delimit what the experiments show.
Critical weaknesses
The inference from individual score events to limitations of the full model is not justified. Even correct probabilities for the four inversion/aliasing events would not, by themselves, establish a loss of the model's ability to distinguish positions and tokens. The theory treats preference for nearer keys and preservation of the initial score ranking as reliability criteria, without establishing their necessity for the intended computation: a nonmonotonic response can select a desired offset, and relevance may depend on position (§§3–4). Equal scores for different keys do not guarantee equal outputs when values differ. The joint positional collisions in Fig. 1 establish invariance of one head's output with other representations fixed, not a limitation of the full model. Likewise, Fig. 4 shows rising inversion frequency where the score itself starts to increase with distance; a frequency near 50% does not establish almost arbitrary positional preference.
The correct answers in the indexing experiment are specified by construction and the errors are real, but their causal connection to these events is unestablished. Without a comparable control, such as a NoPE model trained under the same conditions, the accuracy decline remains compatible with a substantial benefit from RoPE relative to its absence. That comparison would assess relative benefit, not prove the causal role of the four events, and is unnecessary for a narrowly scoped score theorem. Appendix F, p. 34, acknowledges the lack of causal evidence. Nevertheless, the abstract states that multiple heads and layers are insufficient to overcome the limitations, and §§5–6, p. 9, interpret the indexing decline as the cost of optimizing token identification and as evidence that the same bottleneck persists. These stronger conclusions are central to the paper's significance but remain insufficiently supported.
The quantitative probability guarantees are not established by the proofs provided. These guarantees are potentially the main novel contribution: frequency separation and instability of semantic channels have already been examined, notably in Round and Round We Go!, §§3–6. The concerns below affect both the common normal approximation and the subsequent event-specific arguments.
Normal approximation (B, pp. 22–24). For rationally dependent frequencies, small pairwise covariance does not justify invoking the standard central limit theorem and Berry–Esseen bound without conditions accommodating the dependence. For rationally independent frequencies, asymptotic phase uniformity may justify an independent model, but its finite-interval error is not bounded. The claimed O(1/λ(M)) rate does not control these preliminary approximations; increasing length at fixed h also does not make the number of summands unbounded. Appendix F's caveats about asymmetry and the frequency threshold leave room for heuristic value, but do not supply quantitative error control.
Position inversion (C.1, pp. 25–26). Remark 2.1 permits interpreting SM as a surrogate for the farther half of the interval, but equations (3)–(4) do not account for its starting point. Applying full-prefix parameters to that half requires a correction even if Gaussianity is accepted. At the paper's Fig. 2 parameters, h=64, B=105, M=32768, an=1, ϕn=0, exhaustive counting gives inversion probability 0.17056, compared with 0.30310 on the right-hand side of (5). This checks the combined application of B and C.1 to the original RoPE sum; it neither refutes Theorem 5 within its specified surrogate nor disproves the asymptotic limit 1/2. It does show that the published quantitative bound cannot be transferred to the original sum without correction.
Position aliasing (C.2, Theorem 6, p. 28). Comparisons against one reference position share its logit. Even assuming independent normal scores X1,…,XM, with r(x)=Pr(∣X−x∣<ε), the probability that the reference has at least one match is 1−E[(1−r(X1))M−1]. Jensen's inequality makes this no greater than 1−(1−Er(X1))M−1, whereas the proof uses the product to derive a lower bound. This invalidates the proof step, though not necessarily the final bound with the smaller constant E.
Token inversion (C.3, pp. 28–30). The score S is compared with S′ for a constructed prime key with identical amplitudes and zero phases. The difference D=S−S′ has sinusoidal coefficients An=−2ansin(ϕn/2). Uniform original amplitudes do not ensure the absence of dominant components required by B: if an=1 and the only nonzero phase is ϕj=π, then D(m)=−2cos(mB−j/h) is a single harmonic. Expressing the difference as a RoPE sum therefore does not justify a central limit theorem. There is also a correctable variance error: the decomposition and (4) give 2∑n<λ(M)an2sin2(ϕn/2), not Theorem 7's ∑n<λ(M)an2sin2ϕn. Correcting this seems to leave probability 1/2 for a centered, nondegenerate normal surrogate, but does not justify that surrogate or extend the prime construction to an ordinary pair of keys.
Token aliasing (C.4, pp. 30–31). Adding score variances is correct under the explicitly stated independence assumption, but that assumption is not justified for fixed keys evaluated at the same random distance. In general, Var(S1−S2)=Var(S1)+Var(S2)−2Cov(S1,S2). Distinct keys can be arbitrarily close while preserving their two-dimensional block lengths and hence equal amplitudes; their difference variance can then approach zero while individual variances remain nonzero. Separately, the claimed μD/σD=Θ((h−λ)/λ) is inferred from only an upper bound. Here μD,σD describe the score difference and λ=λ(M) counts high-frequency components. For λ<h, a lower bound is missing: the means can coincide even with uniform amplitudes. Independence does not fix this step. These issues leave the general quantitative claim unestablished; they do not by themselves refute the limiting formula at λ=h within the independent normal model.
Questions for authors
How well does the normal approximation describe scores in trained models? Can you assess it on a substantial sample of query–key pairs from real inputs and representative heads/layers, using each theorem's distance sampling for fixed pairs? Please report statistical normality tests alongside quantitative fit diagnostics and predicted versus observed inversion/aliasing rates at realistic context lengths and RoPE bases. Include amplitude concentration for the original sums and relevant score differences, so departures from the stated assumptions can be distinguished from approximation error under them. The key issue is the accuracy of the paper's probability estimates, not exact Gaussianity or a test p-value; empirical fit would not itself establish a central limit theorem.
What evidence links the four proposed events to task errors in trained models? Can you provide a matched RoPE/NoPE comparison to assess RoPE's contribution to task performance, and/or a controlled intervention that changes the incidence of the proposed events and measures the effect on accuracy? These address different claims: the NoPE comparison assesses relative benefit or harm, while attributing errors to the four events requires evidence that isolates their effect from changes in content and task relevance. A focused experiment in a tractable trained model would already be informative. Given Appendix F's acknowledgment of the causal gap, what model-level conclusions remain supported if such evidence is unavailable?
Does the position-inversion result survive correction of B and C.1? After addressing the approximation error in B and the farther-half distribution in C.1, does the position-inversion probability still converge to 1/2, and under what asymptotic regime for context length, RoPE base, and head dimension? Which conclusions can be established for the original RoPE sums, beyond the normal surrogate, and what finite-context lower bound remains justified?
Can the position-aliasing lower bound in C.2 be established without the disputed factorization? What argument replaces the product over comparison events that share a reference logit, and does it still yield the claimed lower bound after accounting for this dependence?
Under what conditions do the token-level conclusions in C.3–C.4 hold? What restrictions on phases and key pairs justify the normal approximation to score differences and the applicability of the independence assumption in C.4? After correcting the variance in C.3, does the token-inversion limit of 1/2 hold for the original scores? In C.4, what supports the lower bound needed for the claimed Θ scaling? Do the resulting claims cover arbitrary key pairs, or only the prime construction and other restricted cases?
TL;DR. We accept both weaknesses: the abstract and Sections 5 and 6 no longer attribute full-model failures to single-head failure modes, and the proofs in Appendices B and C are corrected, with every corrected probability within 0.01 of exhaustive counting on all 32 heads of Llama 3.1-8B layer 0.
Thank you for the detailed investigation of Appendices B and C.
W1. Single head to full model. Accepted: the submission gave no causal connection and the claims were too strong.
Equal scores do not guarantee equal outputs when values differ. For a single head they do: swapping two keys that alias at the same position pair leaves the head output unchanged even with different values (Section 3, Fig. 1). The revision states the invariance for one head only.
Whether a non-monotone response selects a desired offset is tested by the window inconsistency Qw (Definition B.17): a profile that selects an offset has Qw=O(w/M), while on real data the median Q1024 over 6400 pairs is 0.39 at 32K and 0.44 at 128K (Table 11).
Fig. 4(a) is a real head; the appendix now reports all 6400 pairs. At 128K the locality bias is reversed for most pairs and heads, and the part of the sign set by the pseudorandom high-frequency components (Definition B.12) is unpredictable at any resolution.
NoPE comparison added (Q2).
W2. Errors in proofs. Confirmed and corrected.
Prior work on frequency split and semantic channels is credited in Section 2; the wrong first author is fixed.
Normal approximation. The correct tool is the Salem-Zygmund CLT for lacunary sums (Proposition B.2), which needs no independence among summands or rational independence among frequencies. We measure the error: Kolmogorov distance below 0.052 for random phases; the aligned-phase case of Fig. 2(c) is the worst (0.09 to 0.13), where the mixture of Gaussians (Definition B.4) stays below 0.051.
Position inversion. Exact half-interval moments; constant 2ln2 instead of ln2 (Q3).
Token aliasing. Exact covariance instead of independence.
Q1. Empirical validation. For each of the 32 heads of Llama 3.1-8B layer 0 we drew 200 query-key pairs from text and computed exact RoPE scores at 4K, 32K and 128K. Median position inversion at 4K/32K/128K: new formula 0.198/0.090/0.667, exhaustive 0.196/0.096/0.674, old formula 0.389/0.346/0.516; Kolmogorov distance to the normal 0.04 (old 0.2 to 0.4).
Every probability is within 0.01.
Yes for layer 0; no for deeper layers, where q, k, v depend on context through previous layers.
3, 4. Regular amplitudes do not hold on real pairs (median heff=27 of 64), yet the estimates are accurate because the probabilities use exact amplitudes (common response).
Agreed; the CLT covers the high-frequency part and the low-frequency error is measured (Definition B.4 gives the precise mixture).
Q2. Link to task errors. We train 4-layer transformers differing only in positional encoding (RoPE bases 104, 5×105; ALiBi; NoPE; three seeds) from scratch on the indexing task. RoPE beats ALiBi, which beats NoPE; at 2048 elements RoPE scores 0.93 to 1.00 while ALiBi and NoPE are near random. A labeled-list control keeps large models at 0.73 to 1.00 at 4096 elements versus chance unlabeled, so the failure is locating a position by its index. The revision no longer presents Fig. 9 as a cost of RoPE.
Q3. Position inversion. Mostly. Thm 5 now reads Pr≈Φ(−2ln2h/(lnB(lnM−1.51)+0.97h)). The limit 1/2 needs (lnB)2/h→∞ under M≤cB, and convergence is slower (real heads: 0.2, 0.1, 0.67 at 4K, 32K, 128K).
Q4. Aliasing without factorization. Yes. Conditioning on the reference score gives p(M)=1−E[(1−Pr(∣D∣<ε))M−1]; the original inequality had the wrong direction. p(M)→1 still holds, with 1−p(M)=M−1+o(1), polynomial instead of exponential; on real heads p(M)=0.9995 at M=4096 (Table 5).
Q5. Token identification. Thms 7 and 8 now treat the difference sum D(m) of any two keys with its exact mean and variance.
No restriction for the probability formula; regular difference amplitudes (A6) only for closed forms, where h becomes heff (Appendix B.6).
The independence assumption is dropped; the exact variance is used (Proposition B.1).
No. At M=Θ(B), p(M)=Φ(−Θ(h/lnB))<1/2; a random model with B=104, h=64 gives 0.38 to 0.41 (Table 17). The trends in M and B and the Section 4 trade-off are unchanged.
The lower bound on ∣μD∣ is dropped. Thm 8 gives two-sided bounds needing only an upper bound on ∣μD∣; Fig. 6(b), Thm 4 and "up to 5 percent of positions" are reworded as lower bounds.
Yes; the prime key and the opposite-phase pair are special cases.
Thank you for the substantial revision. The corrections in C.2–C.4 and the added NoPE/ALiBi comparison address several specific concerns from my review. I especially appreciate that the authors tested their theoretical predictions against exhaustive counts on real attention heads.
I used AI assistance to check the revised derivations in Appendix C. These checks found no further errors in C.2–C.4. I am not pursuing the remaining asymptotic question in C.1, as it does not materially affect my assessment. I have not independently verified the full revised theory, including the expanded Appendix B. My remaining concerns are primarily about the interpretation and scope of the results.
The title overstates what the four single-head events establish about the ability to distinguish positions and tokens.
The revised theory quantifies when a single head reverses a preference or assigns the same score to two alternatives. The gap lies in interpreting these events as failures of discrimination. A position inversion actually requires different scores at the two distances. Calling it an error requires a reason why the nearer occurrence should rank higher for the task. For token aliasing, the abstract interprets an unchanged score after a token replacement as “a failure to distinguish positions or tokens” (lines 021–023), even though the replacement can change the value vector and hence the head output. The abstract acknowledges the causal limitation of the indexing experiment (lines 027–031). Yet the Takeaways box (p. 9) infers a “maximum effective context length” for each head without showing at what event rate the head’s ability to perform a task-relevant computation deteriorates.
The labeled-list control is a valuable addition because it shows how the same models respond when the index is explicit. On lists of 4,096 elements, Kimi-K2.5 and deepseek-flash score near chance without index labels (0.27 and 0.28 at 12.3K tokens) and almost perfectly with them (1.00 and 0.99 at 27.7K, Table 18). Labels restore retrieval even as the prompt more than doubles in length. Attributing the unlabeled failure to RoPE requires evidence that the encoding limits recovery of the index from token order. With task-specific training, the RoPE models outperform NoPE and ALiBi on unlabeled indexing within the training range (Appendix D.3). The main claims should consistently distinguish demonstrated score-level phenomena, observed task-specific limitations, and a proposed but unverified connection between them. This would preserve the contribution while making its practical implications more precise.
Position aliasing gives a quantitative example. Under the paper’s BF16 resolution criterion, Theorem B.7 guarantees that, for any fixed query and key in any RoPE head, at least 87.45% of 2,048 distances have another distance with a score within the specified tolerance. Table 5 reports a median fraction of 99.95% at 4K in Llama’s first layer. Appendix D.3 does not measure aliasing during indexing, but the theorem applies to the fixed-pair score profiles of those models as well. On lists of 2,048 single-token elements, the two RoPE models reach mean indexing accuracies of 99% and 96% (Table 21). A distance can have an alias somewhere in a head’s score profile even when the model reliably retrieves the requested element. Interpreting this frequency as a limit on usable context requires showing when such an alias interferes with retrieval of the queried position.
A practically consequential trade-off has not yet been established.
The correction to the token-aliasing bound matters. The lower bound decreases with the base, yet at a context length of 10,000 the exhaustive rate for keys with independent random phases rises from 5.3% to 6.4% as the base goes from 100,000 to 1,000,000 (Appendix C.4, p. 51). Even if two score diagnostics move in opposite directions, the range of bases that allow good performance on both tasks at a given length could be broad, narrow, or empty.
The abstract says that increasing the base “inevitably sacrifices the ability to distinguish positions” (lines 023–027). That claim requires evidence that a gain in token identification comes at a cost in positional accuracy under comparable conditions.
Fig. 6b leaves unclear when token aliasing becomes a long-context limitation.
The informal Theorem 4 states that token aliasing becomes more likely as context length increases. In Fig. 6b, the solid curves for keys with independent random phases already reach roughly 5% below 100 tokens. Appendix C.4 reports exhaustive rates of 5–7% across the tested settings. The pronounced rise in the dashed curves concerns a lower bound attained for keys with opposite low-frequency phases. The statement about growth with length should specify which key pairs it describes and whether it concerns the bound or the observed rate.
On real first-layer heads, Table 5 does show an increase in median token aliasing, from 2.1% at 4K to 5.4% at 128K. The practical question is what that increase costs the model. At a rate of 5%, the paper counts about 1,600 aliasing positions in a 32K context for a given query and pair of keys. To explain declining task accuracy, one would need to connect that count to the chance of a collision in a comparison that matters for the answer. Fig. 6b and Table 5 leave that connection open.
Single-head inversion rates do not determine how much protection other heads and layers provide.
Appendix E estimates that eight active retrieval heads will all exhibit position inversion “once every 250 tokens” if each head has an inversion probability of 0.5. The figure of 1 in 256 follows if the eight heads invert independently. If they invert on the same comparisons, the joint probability is instead 0.5, with exactly the same individual rates. Table 5 reports inversion frequencies for individual heads and token pairs. Estimating the joint rate requires knowing how often the active heads invert on the same task-relevant comparison. Translating that rate into errors per token also requires showing when a joint inversion changes the answer.
For layers, Appendix E argues that residual connections carry earlier errors forward. A residual connection also carries information that later layers can use to recover a distinction that one head missed. The quantitative validation concerns layer 0, and Appendix D.3 measures indexing accuracy without tracing the four events through the trained models. The authors appropriately describe Appendix E as an intuitive discussion. How much these events persist, are corrected, or affect the output therefore remains open.
The revised theory quantifies how often four events occur in individual RoPE heads. The practical question is which of them already limit models at lengths where failures are observed, and which might matter only at much greater lengths. Even an accurate estimate of when an event becomes frequent leaves open when it begins to impair a task. The indexing experiments do not isolate RoPE’s contribution to the observed failure, and the results do not show that better token identification must cost positional accuracy. I would frame the title, abstract, and Takeaways around the single-head results and present their implications for usable context length as an open question.
We thank the reviewers for reading the submission closely. The three reviews agree that (1) the proofs in Appendices B and C contain errors, and (2) the abstract and Sections 5 and 6 generalize the single-head analysis to full models with claims that are too strong. We accept both and reflect them in the revision.
Math. We corrected the math and checked it against exhaustive counting over all distances.
Normal approximation (Appendix B). The approximation now rests on the central limit theorem for lacunary trigonometric sums (Proposition B.2), which is stated for a continuous position variable and needs no independence among summands. Since we apply it to integer positions, we measure its accuracy. Over any short sub-interval of the context, the low-frequency terms are nearly constant and the high-frequency terms are pseudorandom (Definition B.12), so the exact law is a mixture of Gaussians, one per sub-interval (Definition B.4). For simplicity we replace the low-frequency part by its mean and use a single Gaussian (Appendix B.2). This loses the skew visible in Fig. 2(c), but the Kolmogorov distance stays within 0.052 and the four probability estimates are within 0.01 of exhaustive counting on real heads.
The submitted Eqs. 3 and 4 are replaced by exact moments.
Thm 5 (position inversion) uses exact moments of the second half of the context and approaches the limit 1/2 more slowly.
Thm 6 (position aliasing) conditions on the reference score; the rate is polynomial instead of exponential.
Thm 7 (token inversion) covers arbitrary key pairs instead of the prime key, corrects the variance, and its limit at M=Θ(B) is below 1/2.
Thm 8 (token aliasing) uses the exact covariance and gives a two-sided bound.
Abstract. Claims on multi-head, multi-layer architectures are weakened to observations. "Prove" is changed to "under the normal surrogate justified theoretically and empirically".
Visuals. Figures 3b, 3c, 6b and the submitted Tables 3 and 4 (Tables 14 and 15 of the revision) are recomputed with the updated formulae.
Full-model observations are described as consistent with the single-head analysis, and the causal implications are removed.
Assumption box is added at the start of Appendix C.
Regular amplitude assumption. All reviewers raised this. All quantitative claims in the revision (probabilities, inversion limits, bounds at specific context lengths) hold without it, since they are computed from the surrogate's exact moments. The assumption is used only in the order-of-magnitude closed forms in the number of summands h. If a few components dominate, Appendix B.6 replaces h by the effective count heff=(∑nan)2/∑nan2∈[1,h]; when the dominant components have random indices, the head follows the closed forms of a regular head with heff components (Proposition B.9). On 6400 Llama token pairs the median heff is 27 (h=64). Inversion probabilities increase when fewer components are non-negligible: position inversion at B=105, M=32768, ϕn=0 has median 0.17 at heff=64 and 0.38 at heff=11. Aliasing probabilities decrease instead: positional pair aliasing in BF16 at the same B, M is 0.056 at heff=64 and 0.013 at heff=4. This replaces the heuristic "as if reducing the total number of dimensions" in the original Limitations.
Positional encodings. A proposition in Appendix C.5 shows that token inversion and aliasing do not occur under additive positional encodings such as ALiBi and T5 bias.
Control experiments. Two are added for the indexing task.
Labeled lists. All models perform better on the labeled control than on the unlabeled task, and large models stay high most of the time. This is consistent with the single-head analysis (models do well on token identification and fail on position identification), without a causal conclusion.
4-layer transformers trained from scratch that differ only in positional encoding (RoPE with bases 104 and 5×105, ALiBi, NoPE; three seeds each) on the indexing task with identical data (lists of 4 to 2048 elements) and training settings. RoPE performs better than ALiBi, which performs better than NoPE. After 150k steps, at 2048 elements, both RoPE bases score 0.93 to 1.00 while ALiBi and NoPE fall to near random.
Full references for all works cited in the replies are in the revised bibliography.
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.
Reply to RTiV
TL;DR. The Gaussian approximation is now justified by the Salem-Zygmund central limit theorem for lacunary sums and verified empirically (four probabilities predicted within 0.01). Our scope is M≤cB, and the revised Fig. 3b stops at M=B. The assumptions are collected in a box at the start of Appendix C. A new proposition shows that additive encodings such as ALiBi and T5 bias cannot show token inversion or aliasing, and a from-scratch comparison of RoPE, ALiBi and NoPE on the indexing task shows RoPE performing best.
Thank you for your review. The proof corrections raised by the other reviewers are in the common response.
W1. Gaussian approximation. We agree the CLT was not justified correctly.
W2. M overlarge compared to B. Our scope is the opposite: footnote 3 assumes M≤cB, and the high-frequency components only need Mθn≫1. The original Fig. 3b was confusing. The curves are truncated at M=B and the scope is stated in assumption A2.
W3 and Q1. Implicit assumptions. We agree. The revision adds an assumption box at the start of Appendix C, and each theorem states the assumptions it uses: 0. normal surrogate with measured error (W1);
The assumptions in Appendix C.4 that two RoPE products are independent and that an is the same for all pairs are removed.
W4. Position bias in Fig. 4. The revision discusses position bias and cites the lost-in-the-middle, RULER and BABILong studies in Appendix D.2.3. Fig. 4 concerns position inversion, where a random farther location scores higher than a nearer one, whereas position-bias studies concern certain locations favored by a trained head. A new index-position sweep in Appendix D shows that indexing accuracy drops to random with the index and never recovers, unlike the U-shaped retrieval curves of position-bias studies.
W5. Is the trade-off fundamental? Every encoding other than NoPE adds position-dependent terms, but this need not harm token identification. For any additive bias ⟨q,k⟩+b(m) (ALiBi, T5 bias), the score difference of two keys is ⟨q,k1−k2⟩, independent of distance, so token inversion and aliasing cannot occur. The negative effect on token identification arises from multiplicative distance terms as in RoPE, not from additive ones as in ALiBi. Since RoPE is the de facto standard, we study this trade-off as its property and limitation. The revision adds a proposition in Appendix C.5 and a from-scratch comparison of 4-layer transformers differing only in positional encoding (RoPE bases 104, 5×105; ALiBi; NoPE; three seeds) on the indexing task: RoPE beats ALiBi, which beats NoPE, and at 2048 elements RoPE scores 0.93 to 1.00 while ALiBi and NoPE are near random.
Presentation. The revision adds the assumption box, states assumptions in each theorem, unifies the token-inversion definition in Section 4 and Appendix C (Def. C.3), cuts Fig. 3b at M=B, and rewords the trade-off discussion.