- Rating
- 2(−1)
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies positional addressing using XoM, a synthetic relative-position retrieval task, and observes approximately two-dimensional rotating trajectories in centered, position-averaged queries and keys across five non-RoPE encodings. It develops worst-case addressing-margin benchmarks and a conditional spectral account of low-frequency concentration during optimization. It proposes Geometric RoPE, a deterministic frequency allocation based on Bose–Chowla constructions and Katz bounds, with margin guarantees and conditional asymptotic optimality. Experiments report improved retrieval margins and convergence on synthetic tasks, alongside preliminary transfer results from an approximately 60M-parameter language model.
Critical strengths
- Substantive cross-encoding observation: The shared rotational geometry is supported across multiple seeds, sequence lengths, and initializations, with negative results also disclosed.
- The theoretical analysis provides coherent addressing bounds and a nontrivial deterministic frequency allocation under explicit assumptions, without adding trainable modules. Target-value replacement produces a 98.325% replacement-answer rate for the identified RoPE head, supporting its role in the copying pathway.
Critical weaknesses
- The geometry concerns centered positional means, while interventions modify values or head outputs rather than rotational Q/K components, leaving residuals and biases as possible determinants of retrieval rankings. The optimization account also depends on unmeasured assumptions concerning competition, key geometry, path energy, spectral separation, optimizer residuals, and subspace drift, so it remains a conditional hypothesis.
- Comparisons primarily against standard RoPE cannot distinguish bandwidth expansion from benefits attributable to the Bose–Chowla structure. Dimension- and frequency-range-matched random, uniform, and globally scaled RoPE controls are needed.
- Language-model results use one matched-seed pair, leaving uncertainty in the 0.46-percentage-point average language-task gain and MQAR improvement unquantified. Missing MQAR construction, adaptation, and scoring details, downstream prompting configurations, and an ablation of the layer-selection choice limit reproducibility and interpretation. Supplementing these details or experimental results would be highly beneficial.
Questions for authors
See weaknesses.
W1: Thank you for your suggestion. Our additional experiments confirm that the centered two dimensional trajectory described in the paper plays a decisive role in task performance. Through a causal intervention, we showed that the marker token’s attention to individual keys follows the centered trajectory. We decomposed each key into a component in the plane of the centered, token averaged trajectory and a residual. We then swapped the centered components at the target position and another position while keeping the residuals fixed. For most heads, peak attention shifted to the swap position in nearly all test sequences. These results show that the centered trajectory determines which position the marker attends to.
The detailed experimental results are as follows: ALiBi (99.99% / 99.99% / 0.32%); NoPE (99.99% / 99.99% / 0.39%); APE (99.95% / 99.95% / 0.43%); T5 (100.00% / 100.00% / 1.33%); KERPLE (100.00% / 100.00% / 0.31%). For each method, the three values correspond sequentially to the original test accuracy, test accuracy after residual ablation, and test accuracy after low-rank circle ablation.
To clarify the roles of the residual and the centered component, we measured test accuracy after removing each one separately. Removing the residual had virtually no effect on accuracy. Removing the 2d centered component, however, reduced accuracy to 1.33% or less for every model.
Test accuracy (original / residual removed / circular component removed) ALiBi (99.99% / 99.99% / 0.32%), NoPE (99.99% / 99.99% / 0.39%), APE (99.95% / 99.95% / 0.43%), T5 (100.00% / 100.00% / 1.33%), and KERPLE (100.00% / 100.00% / 0.31%).
Together, these experiments show that the centered trajectory is the decisive factor in the models’ ability to perform the task. We agree with your observation that the reasoning presented in the paper is an explanatory hypothesis. We now have a deeper understanding of how the model converges to this two dimensional circular solution and can fully explain the dynamics of that process. A clear account of the mechanism requires substantial space and extensive experimental evidence. We therefore plan to present it in detail in a separate paper. We hope for your understanding.
W2: Your assessment is correct. Although Geometric RoPE uses the Bose–Chowla structure, its gains come from distributing frequencies more broadly. In this sense, Geometric RoPE is a form of scaled RoPE. Other approaches, such as scaling RoPE frequencies to maximize the ability to distinguish relative positions, should yield variants that match or even outperform Geometric RoPE. Our main contribution is to demonstrate the feasibility of this direction. For that reason, we did not compare Geometric RoPE with other methods.
We proposed Geometric RoPE for three reasons.
W3: We agree with you. Our language modeling experiments are too small in scale to demonstrate the practical value of our approach and we regret this limitation. Our current computational resources prevent us from conducting the large language model experiments we had hoped to run. As soon as sufficient resources become available, we will address this limitation by running additional experiments at the largest scale we can support.
For the MQAR evaluation, we used each model’s final training checkpoint with all parameters frozen. We used no MQAR specific training or fine tuning and no natural language prompts. We evaluated eight configurations, each specified as (sequence length, number of key value pairs). For both models, we used the same data generation code, a random seed of 6198, and a batch size of 8. We first filled each sequence with random background tokens sampled from the token ID range [2, 50280). We then wrote the key value pairs consecutively at the start of the sequence, with each key immediately followed by its value. In the second half of each sequence, we placed the keys at predetermined, evenly spaced positions as queries, preserving their original order without shuffling. Each query’s label was its corresponding value token.