acceptodds
Under review as a conference paper at ICLR 2027

The Fates’ Spindle: Rotation Representation Emerges for Relative Position in Transformer

Abstract

Large language models rely on relative positional information across diverse tasks, while the mechanisms underlying relative position reasoning have yet to be elucidated. Using a controlled synthetic task that isolates relative position retrieval, we systematically study Transformers across positional encoding schemes and uncover a unified geometric mechanism. Across NoPE, ALiBi, T5 relative bias, KERPLE, and learned absolute embeddings, models converge to nearly identical geometry: keys and queries collapse into an approximately two dimensional subspace and trace a uniform rotation completing one cycle over the training context. This structure is equivalent to generalized RoPE, with all active channels sharing a frequency of approximately per position, where is the training context length. We further show that this single frequency representation is theoretical inferior to suitable multifrequency alternatives, explaining the advantage of richer frequency components in RoPE and related representations. Guided by this analysis, we evaluate a new frequency allocation scheme, providing support for the framework and its practical utility. Our findings provide a understanding of how Transformers learn relative position representations, establish synthetic tasks and geometric analysis as a principled approach to investigating positional capabilities, and inform future positional encoding design.

Primary area:
learning theory
Keywords:
transformer, positional encoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MGpHOpenReview ↗
Rating
2(−1)
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies positional addressing using XoM, a synthetic relative-position retrieval task, and observes approximately two-dimensional rotating trajectories in centered, position-averaged queries and keys across five non-RoPE encodings. It develops worst-case addressing-margin benchmarks and a conditional spectral account of low-frequency concentration during optimization. It proposes Geometric RoPE, a deterministic frequency allocation based on Bose–Chowla constructions and Katz bounds, with margin guarantees and conditional asymptotic optimality. Experiments report improved retrieval margins and convergence on synthetic tasks, alongside preliminary transfer results from an approximately 60M-parameter language model.

Critical strengths
  1. Substantive cross-encoding observation: The shared rotational geometry is supported across multiple seeds, sequence lengths, and initializations, with negative results also disclosed.
  2. The theoretical analysis provides coherent addressing bounds and a nontrivial deterministic frequency allocation under explicit assumptions, without adding trainable modules. Target-value replacement produces a 98.325% replacement-answer rate for the identified RoPE head, supporting its role in the copying pathway.
Critical weaknesses
  1. The geometry concerns centered positional means, while interventions modify values or head outputs rather than rotational Q/K components, leaving residuals and biases as possible determinants of retrieval rankings. The optimization account also depends on unmeasured assumptions concerning competition, key geometry, path energy, spectral separation, optimizer residuals, and subspace drift, so it remains a conditional hypothesis.
  2. Comparisons primarily against standard RoPE cannot distinguish bandwidth expansion from benefits attributable to the Bose–Chowla structure. Dimension- and frequency-range-matched random, uniform, and globally scaled RoPE controls are needed.
  3. Language-model results use one matched-seed pair, leaving uncertainty in the 0.46-percentage-point average language-task gain and MQAR improvement unquantified. Missing MQAR construction, adaptation, and scoring details, downstream prompting configurations, and an ablation of the layer-selection choice limit reproducibility and interpretation. Supplementing these details or experimental results would be highly beneficial.
Questions for authors

See weaknesses.

1 comment

W1: Thank you for your suggestion. Our additional experiments confirm that the centered two dimensional trajectory described in the paper plays a decisive role in task performance. Through a causal intervention, we showed that the marker token’s attention to individual keys follows the centered trajectory. We decomposed each key into a component in the plane of the centered, token averaged trajectory and a residual. We then swapped the centered components at the target position and another position while keeping the residuals fixed. For most heads, peak attention shifted to the swap position in nearly all test sequences. These results show that the centered trajectory determines which position the marker attends to.

The detailed experimental results are as follows: ALiBi (99.99% / 99.99% / 0.32%); NoPE (99.99% / 99.99% / 0.39%); APE (99.95% / 99.95% / 0.43%); T5 (100.00% / 100.00% / 1.33%); KERPLE (100.00% / 100.00% / 0.31%). For each method, the three values correspond sequentially to the original test accuracy, test accuracy after residual ablation, and test accuracy after low-rank circle ablation.

To clarify the roles of the residual and the centered component, we measured test accuracy after removing each one separately. Removing the residual had virtually no effect on accuracy. Removing the 2d centered component, however, reduced accuracy to 1.33% or less for every model.

Test accuracy (original / residual removed / circular component removed) ALiBi (99.99% / 99.99% / 0.32%), NoPE (99.99% / 99.99% / 0.39%), APE (99.95% / 99.95% / 0.43%), T5 (100.00% / 100.00% / 1.33%), and KERPLE (100.00% / 100.00% / 0.31%).

Together, these experiments show that the centered trajectory is the decisive factor in the models’ ability to perform the task. We agree with your observation that the reasoning presented in the paper is an explanatory hypothesis. We now have a deeper understanding of how the model converges to this two dimensional circular solution and can fully explain the dynamics of that process. A clear account of the mechanism requires substantial space and extensive experimental evidence. We therefore plan to present it in detail in a separate paper. We hope for your understanding.

W2: Your assessment is correct. Although Geometric RoPE uses the Bose–Chowla structure, its gains come from distributing frequencies more broadly. In this sense, Geometric RoPE is a form of scaled RoPE. Other approaches, such as scaling RoPE frequencies to maximize the ability to distinguish relative positions, should yield variants that match or even outperform Geometric RoPE. Our main contribution is to demonstrate the feasibility of this direction. For that reason, we did not compare Geometric RoPE with other methods.

We proposed Geometric RoPE for three reasons.

  1. Maximizing the ability to distinguish relative positions gives us an explicit, quantitative criterion for designing and evaluating RoPE frequency allocations. The same criterion also lets us compare how well different positional encodings distinguish relative positions.
  2. Geometric RoPE provides a deterministic way to allocate frequencies. Given the model parameters, we can calculate both which frequencies to use and how well the resulting encoding can theoretically distinguish relative positions. This reduces the need for computationally expensive experiments such as hyperparameter tuning.
  3. We wanted to connect our observations in small models more directly to the development of large models in practice. Geometric RoPE offers a natural, readily adoptable way to study the properties of these solutions from a geometric perspective and to seek better, deterministic solutions for real world settings.

W3: We agree with you. Our language modeling experiments are too small in scale to demonstrate the practical value of our approach and we regret this limitation. Our current computational resources prevent us from conducting the large language model experiments we had hoped to run. As soon as sufficient resources become available, we will address this limitation by running additional experiments at the largest scale we can support.

For the MQAR evaluation, we used each model’s final training checkpoint with all parameters frozen. We used no MQAR specific training or fine tuning and no natural language prompts. We evaluated eight configurations, each specified as (sequence length, number of key value pairs). For both models, we used the same data generation code, a random seed of 6198, and a batch size of 8. We first filled each sequence with random background tokens sampled from the token ID range [2, 50280). We then wrote the key value pairs consecutively at the start of the sequence, with each key immediately followed by its value. In the second half of each sequence, we placed the keys at predetermined, evenly spaced positions as queries, preserving their original order without shuffling. Each query’s label was its corresponding value token.

Reviewer_qdF8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a thorough and controlled investigation on how Transformers learn relative positional representations across different positional encoding schemes. They use a synthetic retrieval task (coined XoM), with which the authors show that models without RoPE nevertheless develop an approximately two-dimensional rotational representation, effectively learning a single-frequency version of RoPE (proportional to the context length). They then provide a geometric explanation for this behavior and introduce a new frequency allocation scheme designed to improve positional discrimination. I find the convergence toward similar geometries across architectures interesting on its own, even if quite “localized” in terms of its generalizability.

Critical strengths
  1. The observation that NoPE, ALiBi, T5, KERPLE, and learned absolute embeddings converge toward similar rotational structures is compelling. It suggests that relative positional retrieval might impose strong constraints on the geometry of learned representations, independently of how positional information is initially provided.

  2. I appreciate the attempt to connect the observed low-dimensional geometry to an explicit optimization objective. Interpreting positional retrieval through geometric separation provides a useful mathematical perspective on why some positional codes might be more effective than others.

  3. The authors use their mechanistic observations to motivate a different RoPE frequency allocation. This makes what the authors propose more interesting than a purely descriptive study, even if the practical improvements still need stronger validation.

Critical weaknesses
  1. Observing circular trajectories in averaged keys and queries does not necessarily establish that these representations are causally responsible for retrieval. It would be more compelling to see interventions that selectively disrupt the rotational subspace, or modify its frequency, and measure the resulting effect on performance. This would make the mechanistic claim considerably stronger.

  2. The optimization argument does not fully explain why the representation emerges. The authors show that a circular configuration is optimal under certain dimensional constraints, but this is different from demonstrating why gradient descent converges toward that configuration. In particular, the low-rank collapse is not convincingly derived from the actual training dynamics. The theoretical analysis also assumes a retrieval setting that differs from the causal XoM task.

  3. The paper overlooks the positional–symbolic trade-off. Relative positional retrieval is only one of the operations attention heads must implement. Previous work shows that different frequency ranges support different positional and symbolic behaviors (see [1]), with important consequences for generalization. Maximizing positional discrimination may therefore come at the expense of symbolic operations. The proposed Geometric RoPE is evaluated on relatively limited language-model settings, without sufficiently addressing this potential trade-off. More generally, I believe the paper lacks empirical validation at a larger scale, and in natural language settings. The argument that studying a relative positioning task requires a controlled setting induces a strong bias towards observing simpler solutions that may actually not emerge in natural language settings. This is for me the main weakness of this work. It would be interesting to observe some form of prediction emerging from this work that could be extrapolated and then potentially observed at scale, as was proposed in [2].

[1] Urrutia, Felipe, et al. "Decoupling Positional and Symbolic Attention in Transformers." International Conference on Learning Representations. Vol. 2026. 2026. [2] Urrutia, Felipe, et al. "Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization." arXiv preprint arXiv:2605.31558 (2026).

Questions for authors
  1. What happens when the model must simultaneously implement positional and symbolic operations? For instance, would the same low-dimensional rotational structure emerge in tasks combining relative indexing with content-based retrieval?

  2. The emergent rotation completes approximately one cycle over the training context. Does this imply that the learned mechanism is intrinsically tied to that context length? What happens when sequences exceed the training length?

  3. Could the authors try to find some approximation to this task in natural language, and observe similar geometric patterns with distinct positional encoding schemes?

2 comments

W1: Thank you for your advice. Please see our response to Reviewer iasD (W1) for details.

W2: Our paper focuses on how models with different positional encodings solve the relative position matching problem, what form the learned solution takes, and what it enables. We show that five models, each using a different positional encoding, spontaneously learn the same two dimensional, single frequency rotational solution. This solution maximizes the minimum separation between representations of distinct relative positions in two dimensions. Understanding this shared solution gives us a unified geometric perspective on how these models process relative positions and what capabilities this supports.

We believe that characterizing this phenomenon and explaining its geometric significance is a meaningful contribution in its own right. Much as Prevalence of neural collapse during the terminal phase of deep learning training characterized neural collapse, we have sought to describe the five models’ convergence to this shared solution as clearly as possible. Why this solution emerges and how it develops during training are questions beyond the main focus of this paper. As we state in the paper, why the model converges to this particular solution remains an important and difficult question.

W3: Thank you for your suggestions. We agree that there is a trade off between positional and semantic processing within a model. As the two papers you cited show, this trade off can lead individual attention heads to specialize exclusively in either positional or semantic processing. In earlier experiments, we trained models jointly on relative position, absolute position, and semantic tasks and examined how they performed each task. Our findings corroborate the results reported for RoPE in the cited papers. Some heads performed only semantic matching, while others performed only relative position matching. In models where we did not explicitly introduce rotation, the latter heads developed the two dimensional circular representations of position shown in our paper. We therefore share your view of this trade off and understand it, at the mechanistic level, as a division of labor among attention heads.

This division of labor motivated our question. Can we strengthen positional heads to improve a model’s ability to represent relative position? Our paper shows that Geometric RoPE performs better on relative position matching. Using it only in heads dedicated to positional processing would therefore be likely to improve the model’s relative position capability while preserving its semantic capability.

Because Geometric RoPE is better at relative position matching, gradient descent may favor using heads with Geometric RoPE for relative position and heads with standard RoPE for semantic tasks. The model can improve its relative position capability while making little use of Geometric RoPE for semantic tasks. Our language experiments showed performance improved on position related tasks and remained largely unchanged on semantic tasks. Accounting for the trade off between position and semantics was central to this design. We will cite both papers in the Methods section and explain how they provide theoretical and empirical support for our experimental design, and how our earlier experiments support their conclusions.

Following your suggestion, we will build on what we have learned about relative position to study its interaction with semantics. By coincidence, the simple relative position task we designed closely resembles the task in the first paper you cited. We will cite that paper in the Related Work section and discuss the similarity in detail.

We believe the mechanism we observed can generalize to larger models on more complex tasks. Our earlier mixed task experiments showed that some heads specialize exclusively in relative position matching, consistent with the findings for RoPE models in the papers you cited. We have used all the computing resources available to our group to test these findings, including through experiments on language tasks.

Q1: In our experiments combining position and semantic matching, we found that some attention heads were dedicated to semantic matching and others to relative position matching. And we also observed the same circular representations shown in the paper.

Q2: The period of the circular is close to the training sequence length. When the key and query norms are held fixed, the optimal two dimensional solution is a circle whose period exactly matches the training sequence length. We tested how well the five models generalize to longer sequences. APE failed to generalize at all, and the other four generalized poorly.

Q3: The direct analogue is a natural language version of the XoM task. Testing whether the geometry emerges would require pretraining a large language model on the task or applying SFT to it. And we have not yet been able to run it.

I thank the authors for their rebutall, and acknowledge their answer to weakness 1. However, my other concerns remain and I would have hoped to see more concrete experimental results to answer them. More generally, I fail to see the added value over previous work, above and beyond showing how position emerges in models without RoPE. This alone, imo, does not warrant an acceptance in ICLR. I would encourage the authors to push their work further, and understand why models that develop these schemes fail to generalize, and what components of the architecture can be tweaked (from a theoretical perspective) to improve their performance. As such, I will maintain my score.

Reviewer_iasDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how Transformers retrieve tokens by relative position, using a synthetic task (XoM) in which a marker token specifies the offset of the token to be copied. Across five non-RoPE encodings (NoPE, ALiBi, T5 bias, KERPLE and learned APE), the authors find that the position-averaged keys and marker queries at the retrieval layer lie close to a two-dimensional circle traversed once over the training context, with each marker's query phase aligned to its target position, which they interpret as an emergent single-frequency RoPE. They then cast retrieval as maximizing a normalized query-key margin, derive circle and simplex benchmarks under explicit assumptions, and sketch an optimization argument for why low-rank representations might be favored. Building on this, they propose Geometric RoPE, a frequency allocation based on the Bose-Chowla construction with a provable coherence bound, and evaluate it on XoM as well as in a 60M language model where it replaces standard RoPE in two layers.

Critical strengths
  1. The central observation is consistent and important. Five distinct non-RoPE encodings converge to nearly the same rotational Q/K geometry. PCA, singular value and Fourier spectra, multiple seeds, initialization sweeps, longer contexts and value-substitution experiments all support this, so it rests on more than visualization alone.

  2. Substantive theory and frequency design. The convex-hull characterization, the circle and simplex benchmarks, and the finite-field allocation give concrete tools for reasoning about positional discrimination. The explicit coherence certificate is especially useful because it ties frequency choice directly to a worst-case retrieval margin.

Critical weaknesses
  1. The mechanistic claims go further than the evidence. The low-rank circle shows up in position-averaged Q/K, which doesn't prove it's what drives retrieval for individual samples. Value substitution shows the head carries the answer, not that the rotation itself is necessary. Directly shifting the query phase and checking where retrieval moves would be far more convincing.

  2. The frequency comparison is missing obvious baselines. Geometric RoPE is mostly compared against default RoPE, whose bound comes from capping frequencies at one radian per step. A rescaled RoPE with a wider frequency range could recover much of the gain. Without rescaled or random-frequency controls, it's unclear what the finite-field construction itself adds.

  3. The language-model results are thin. Only two of eight layers are changed, there's a single seed, and the average gain is under half a point with no error bars or perplexity. The MQAR jump looks impressive, but the evaluation setup isn't described well enough to interpret it.

Questions for authors
  1. Does the geometry actually cause retrieval? If you replace the positional Q/K components with their fitted 2D circle, does retrieval still work, and how much breaks when you remove them? Most convincing would be to shift a marker’s query phase and check whether retrieval moves to the position the geometry predicts. This needs a random-subspace control and should account for the additive biases in ALiBi, T5 and KERPLE.

  2. Do the conditions of your optimization argument hold in real training? Have you measured the spectral gaps, optimizer residuals and subspace drift during AdamW training? If not, would you be willing to present this as a possible mechanism rather than an explanation?

  3. Is the frequency gain more than a wider frequency range? Does Geometric RoPE still beat rescaled RoPE and random frequencies under matched settings? For the language model, please also explain how layers 3 and 6 were chosen, report validation perplexity, and describe the MQAR setup.

1 comment

W1: We have not overclaimed. Taken together, our initial and follow up experiments clearly demonstrate that the low rank circle is the most important factor in enabling matching. We decomposed each key into the component on the low rank circle and a residual. We then swapped only the circular components of the keys at the target and another prefix position, leaving each residual in place. If the circular component guides retrieval, the marker’s attention should follow that component to the swap position.

The detailed experimental results are as follows: ALiBi (99.99% / 99.99% / 0.32%); NoPE (99.99% / 99.99% / 0.39%); APE (99.95% / 99.95% / 0.43%); T5 (100.00% / 100.00% / 1.33%); KERPLE (100.00% / 100.00% / 0.31%). For each method, the three values correspond sequentially to the original test accuracy, test accuracy after residual ablation, and test accuracy after low-rank circle ablation.

We also measured test accuracy after removing either the residual or the circular component from queries and keys.

Test accuracy (original / residual removed / circular component removed) ALiBi (99.99% / 99.99% / 0.32%), NoPE (99.99% / 99.99% / 0.39%), APE (99.95% / 99.95% / 0.43%), T5 (100.00% / 100.00% / 1.33%), and KERPLE (100.00% / 100.00% / 0.31%).

Removing the residual preserved accuracy to the precision reported for all five models. Removing the circular component reduced accuracy to 0.31–1.33%. These interventions identify what directs attention to the target, while the target value substitution experiment in the appendix shows that the final layer determines which token is retrieved. Together, they support a causal role for the low rank circle in relative position retrieval on this task.

W2: Your observation is correct. Geometric RoPE is a new frequency allocation scheme for RoPE, and rescaling RoPE can reproduce these frequencies exactly. Thus geometric RoPE is itself a form of rescaled RoPE. Since its gains come entirely from the wider frequency range, comparing the two at matched frequencies would add little. This is why we did not include a separate comparison with rescaled RoPE. The advantage of the finite field construction is that it provides a deterministic rule for choosing frequencies across model configurations. It also lets us calculate directly how well the chosen frequencies distinguish relative positions, without extensive parameter searches or tuning based on experience. An extensive grid search could, of course, identify effective frequencies for rescaled RoPE for each model configuration. Tabulating those results across configurations, as we do for Geometric RoPE in the appendix, could offer much the same practical value.

W3: We agree with the limitation you identified. However, it may be too much to ask university research groups to conduct larger scale experiments. Our main contribution is to explain how models with different positional encodings represent relative positions. These representations trace single-frequency circles in two-dimensional planes. We also show that, in two dimensions, this configuration maximizes the minimum separation between representations of distinct relative positions. This unified geometric perspective explains why non rotary positional encodings and RoPE differ in their ability to represent relative positions. Building on this insight, we propose Geometric RoPE to connect our findings to practical applications. However, Limited computing resources prevented us from running many experiments on larger models, so we can only extrapolate from our small model results. These results suggest that Geometric RoPE may be effective for language model training, but they are insufficient to establish its effectiveness in practice. We modified only two layers because our previous experiments suggest that real world language tasks involve more than relative position. We therefore chose to emphasize the ability to distinguish relative positions only in selected layers.

RoPE and Geometric RoPE had similar perplexities at context lengths of 128 (40.0242 / 40.1033), 256 (34.7484 / 34.8481), and 512 (31.7154 / 31.8183), with RoPE listed first in each pair.

We evaluated both frozen final checkpoints directly on tokens without MQAR training, fine-tuning, or prompts. Using the same code, seed 6198, and batch size 8, we generated 256 sequences per configuration at lengths 64, 128, 256, and 512.

Each sequence began with key–value pairs against a random token background. Keys, values and background tokens were sampled tfrom [2, 25140),. Keys reappeared in order at evenly spaced positions in the second half.

Q1: See W1

Q2: We agree that the optimization argument describes a possible mechanism. The manuscript presents the derivation as an interpretation, and we will make its conditional status clearer. The derivation alone does not establish that the assumptions hold during AdamW training.

Q3: See W2 & W3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.