acceptodds
Under review as a conference paper at ICLR 2027

Rotary Orientation Contraction in RoPE

Abstract

Rotary Position Embeddings (RoPE) encode relative position by rotating pairs of query and key coordinates according to token position. However, these pairs already have content-dependent orientations before the positional rotation is applied, so content-dependent orientation and positional phase interact within the same angular term of the attention score. We introduce Rotary Orientation Contraction (ROC), a simple modification to RoPE that learns how strongly to contract these pre-existing orientations while preserving the standard RoPE frequencies and positional rotations. Across language-modeling experiments, ROC improves perplexity over matched RoPE baselines. These results identify pre-RoPE content geometry as a useful design dimension for rotary attention and show that it can be modified efficiently, with essentially no measured forward-pass overhead, while preserving the standard paired RoPE structure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rotary position embeddings, positional encoding, Transformer attention, language modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aZZsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ROC (Rotary Orientation Contraction) which enables transformer models using RoPE to control how much pre-RoPE query-key vector orientations interact with the rotations applied by RoPE itself. ROC applies a linear interpolation between a vector and its rotation onto the first coordinate axis, and the degree to which the interpolation is applied is learned by the model during training. Language modeling performance is evaluated through held-out perplexity on FineWeb-Edu at two model sizes, the smaller of which is trained on three seeds. ROC is compared against RoPE and PoPE. The efficiency of ROC is also tested against RoPE. Models using ROC are shown to achieve lower perplexity than baselines with QK normalization.

Critical strengths
  • The idea of allowing the model to learn how much pre-RoPE orientation to keep is interesting and sound.
  • ROC is presented in a way that is intuitive to understand for readers who have the background.
  • The efficiency study and the principal-angle contraction ablations are insightful.
Critical weaknesses
  • The abstract correctly claims improvements in "perplexity", but there are several places in the rest of the paper that claim improved "language modeling performance" (intro, section 3.2) which is too broad to be supported by better perplexity without downstream evals, since perplexity gains may not translate to capability [1].

  • Section 3.3 claims that some of the variations provide "marginal additional gains", but all variations are single seed runs and fall well within the seed variance reported by the previous section for the ROC model. Similarly, section 3.5 claims that ROC "provides no benefit without normalization" over RoPE, but single seed runs are not sufficient to support this claim. Relating to this, the claim that ROC improves perplexity "across language-modeling experiments" in the abstract should be made more specific, as it already is in the intro which states "when combined with QK normalization".

  • Only the 124M run in tables 1 and 3 seems to have three seeds, the 355M and tables 2, 4, and 7 are single seed runs. The paper also doesn't mention what the seeds vary (just initialization, or also data/data order?). If it's just initialization, there's a chance that the SD is understated. Moreover, with only three seeds, the SD estimates themselves are likely unreliable.

  • Based on the reported numbers and sample size, the empirical results for the main claims except that of ROC vs RoPE are unlikely to be statistically significant. I asked an LLM to write a script to run a Welch t-test on the results and compute the p-values with Holm-Bonferroni corrections (and computed some of the values manually myself to confirm), here's the results: | Claim | Δ perplexity | p raw | p Holm | |------------------------------------------|--------------|----------|----------| | C1: ROC vs RoPE | -0.23 | 1.47e-04 | 7.37e-04 | | C2: PoPE vs RoPE | -0.10 | 0.0316 | 0.0633 | | C3: ROC vs PoPE | -0.13 | 0.0158 | 0.0633 | | C4: Principal-angle vs RoPE | -0.13 | 0.0158 | 0.0633 | | C5: ROC vs Principal-angle | -0.10 | 0.0316 | 0.0633 |

  • Because SDs are reported to two decimals, the Holm-corrected values could range roughly between 2e-4 and 3e-3 for C1, and between 0.03 and 0.13 for C2-C5. This means that the p-values may not even clear the commonly used α=0.05\alpha=0.05 bar for C2-C5. In addition, for ML results, I agree with [3] that we should be skeptical about anything greater than 0.0010.001 since the number of possible hypotheses is large (also citing [4], which treats p-values between 0.01 and 0.05 as a marker of fragile and less replicable findings). Again, these are only based on the reported numbers and may not be exact, the authors are welcome to re-compute these values and run their own statistical tests using the exact data.

  • Since a single training budget is used across methods at each model size and no loss curve data is provided, the "performance" gains could just be ROC training faster initially where RoPE could catch up or surpass it later if the models were to be trained longer.

  • It seems that one shared hyperparameter setting is used for all the methods and both sizes. The difference in perplexity could be a result of untuned optimizer parameters.

  • From the setting used for training mentioned in the paper (section 3.1 and appx A.1), the PoPE baseline was modified (QK-norm, gating, using Muon for hidden matrices instead of AdamW throughout). This makes sense for a fair comparison, but according to table 4, ROC is either worse than or on par with RoPE without QK-norm, whereas the original PoPE paper (which does not mention QK-norm or gating) reports gains over RoPE. This might suggest that ROC only performs better than or on par with PoPE under a setting that favors it, or that PoPE is more robust to other variations in the architecture.

  • The scaling trend is not clear from the provided data. While the gap between the perplexity achieved by the models seems to shrink from 124M to 355M, it cannot be said whether there's no difference in the perplexity due to the lack of multiple seeds, or whether ROC scales worse due to the lack of results for at least one more model size.

  • For a positional encoding paper, I believe it is important to test length-extrapolation and long context performance. It is even more relevant in this case because the baseline (PoPE) explicitly tests length extrapolation. [2] shows the effects of training on short contexts and evaluating zero-shot on longer ones across many PE methods.


[1] Zeyuan Allen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.

[2] Ben Anson, Xi Wang, and Laurence Aitchison. Scale-invariant attention. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.

[3] Neel Nanda. Highly Opinionated Advice on How to Write ML Papers. AI Alignment Forum, 2025.

[4] Paul C. Bogdan. One Decade Into the Replication Crisis, How Have Psychological Results Changed? Advances in Methods and Practices in Psychological Science, 8(2), 2025.

Questions for authors
  • What do the seeds vary? If the seeds only vary initialization, what effect does also varying data (chosen from different 4B/10B subsets of FineWeb-Edu) and/or data order have on the resulting SD of the perplexity values?
  • Can the authors share per-seed values and loss curves?
  • Can the authors provide a comparison with PoPE under the training and architecture settings reported by its authors, and/or add PoPE to table 4?

Suggestions:

Overall, stronger experiments and empirical results would significantly benefit the paper. The main ones are tuning hyperparameters (at least learning rate) per model per size for the main comparisons, long-context extrapolation results, and running with more seeds for statistically reliable results. The authors could determine the approximate number of seeds required through power analysis using the existing data or similar.

1 comment

We thank the reviewer for their detailed assessment and constructive suggestions. We have expanded the empirical evaluation and clarified several claims in response to these concerns.

Statistical reliability and experimental variation.

We agree that the limited number of seeds warrants caution, particularly when interpreting differences between secondary ablations. However, we wish to distinguish these exploratory comparisons from our primary claim that ROC improves perplexity over matched RoPE baselines. Using the corrected individual-seed results, we independently repeated the reviewer's two-sided Welch's t-test and obtained p = 0.000706. Even applying a conservative Bonferroni correction across five comparisons yields p = 0.00353, confirming that the primary improvement remains statistically significant. We have also revised the interpretation of secondary ablations, particularly contraction-coefficient granularity, to avoid drawing strong conclusions from small single-run differences.

The three primary 124M runs vary model initialization, while holding the training data and its order fixed. Individual-seed perplexities are provided in Appendix A.2, and we have additionally included validation-loss curves in the revised manuscript.

To provide further evidence across training configurations, we conducted three additional runs per method using a substantially different 49.4M-parameter architecture (512 hidden dimensions, 12 layers, 8192-token vocabulary), trained for 5B tokens using a shuffled-data training pipeline. ROC again improves validation cross-entropy (2.5864 ± 0.0006 vs. 2.6077 ± 0.0021 for RoPE). Although this does not isolate data-order variance, it provides complementary evidence that ROC's improvement persists under a different training configuration.

Hyperparameter sensitivity and convergence.

We agree that shared hyperparameters do not guarantee independently optimal performance for each method. ROC is a relatively non-invasive modification to RoPE, and our experiments were designed to evaluate its effect under matched optimization conditions rather than claim independently optimized performance for every method. We have explicitly acknowledged this limitation in the revised manuscript. Nevertheless, the consistent ROC–RoPE improvement across matched seeds provides evidence that the benefit is not merely an isolated training outcome.

Regarding convergence, we have included validation-loss curves to better characterize training dynamics. Furthermore, the additional 49.4M experiments retain ROC's advantage after 5B training tokens, corresponding to over 100 tokens per parameter. While this does not eliminate the possibility of different behavior at still larger training budgets, it provides evidence that ROC's advantage is not restricted to short training runs.

PoPE comparison and QK normalization.

We agree that PoPE's more substantial reparameterization may respond differently to optimization settings. Our primary experiments compare methods under a shared training configuration, rather than claiming independently optimal performance for PoPE.

We additionally evaluated PoPE without QK normalization or gated attention under our training recipe. In this setting, PoPE achieves a validation perplexity of 19.50 compared with 19.73 for RoPE, confirming that PoPE can improve performance without these components. We have also clarified ROC's dependence on QK normalization in the abstract.

Downstream performance and length extrapolation.

To address the concern that perplexity improvements may not translate to broader capabilities, we have added zero-shot evaluations on six downstream benchmarks, following the selection used in PoPE. ROC achieves the highest average accuracy at both 124M (56.71% vs. 56.44% for RoPE) and 355M (63.31% vs. 62.79%), outperforming RoPE on four and five of the six tasks, respectively.

We have also evaluated zero-shot length extrapolation at 4096, 8192, and 16384 tokens, beyond the 2048-token training context. At 124M, ROC outperforms RoPE at 4096 and 16384 tokens but underperforms at 8192 tokens. At 355M, ROC performs comparably at 4096 tokens but worse at 8192 and 16384 tokens. These results indicate no consistent extrapolation advantage, which we acknowledge in the limitations and document in the appendix.

Reviewer_VvbQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Rotary Orientation Contraction (ROC), a simple modification to RoPE that learns how strongly to contract the pre-existing content-dependent orientations while preserving the standard RoPE frequencies and positional rotations. In 124M and 355M model sizes, ROC improves perplexity over RoPE and PoPE.

Critical strengths

The paper has a clear presentation, with complete motivation and corresponding validation.

Critical weaknesses
  1. The paper has only 7 pages of main text, and its contribution is somewhat limited. Given that PoPE, the most closely related work in the same field, still enables extrapolation, it is worth investigating whether ROC offers a similar capability.
  2. The model scales in this paper are too small, and the number of baselines included for comparison is quite limited. Could the authors include transfer experiments, such as fine-tuning a 1B+-parameter RoPE-based LLM, to verify that the proposed ROC method remains effective at larger scales?
Questions for authors

Although RoPE is an important technique in the development of LLMs, it currently faces significant challenges within frontier open-source LLMs. In hybrid models, combining sliding-window attention or linear attention with NoPE full attention can also encode positional information effectively, sometimes even better. Therefore, we need to further clarify both the necessity and the efficacy of proposed improvements to RoPE.

Specifically, it is worth examining why such modifications are worthwhile. Besides the formal risks, does the content–position interaction in RoPE limit its performance in downstream tasks? Furthermore, it remains to be determined whether such modifications remain relevant and applicable within current frontier model architectures.

1 comment

We thank the reviewer for their feedback. We address the concerns regarding scaling, practical relevance, and applicability to modern architectures below.

Scaling and practical relevance.

While our corrected experiments show comparable relative perplexity reductions at 124M (1.14%) and 355M (1.06%), we agree with the reviewer that the lack of evaluation at larger scales remains a limitation, as the benefits could plausibly diminish with scale. However, we emphasize that ROC's simplicity and negligible measured forward-pass overhead make even modest improvements potentially valuable. Thus, even if relative gains diminish at larger scales, this would not necessarily undermine ROC's practical utility, given its minimal additional inference cost.

Downstream performance.

To evaluate whether ROC's improvements extend beyond in-distribution perplexity, we have added zero-shot evaluations on six downstream benchmarks, following the selection used in PoPE. ROC achieves the highest average accuracy at both 124M (56.71% vs. 56.44% for RoPE) and 355M (63.31% vs. 62.79%), outperforming RoPE on four and five of the six tasks, respectively. These results provide evidence that modifying pre-RoPE content geometry can also benefit downstream performance.

Length extrapolation.

We have additionally evaluated zero-shot length extrapolation beyond the 2048-token training context. At 124M, ROC outperforms RoPE at 4096 and 16384 tokens but underperforms at 8192 tokens; at 355M, ROC performs comparably at 4096 tokens but worse at 8192 and 16384 tokens. These results do not establish a consistent extrapolation advantage. ROC is not intended as a dedicated context-extension method, and its compatibility with established context-extension techniques remains an open question.

Relevance to modern hybrid architectures.

We thank the reviewer for raising this insightful point about the role of RoPE in modern hybrid architectures. We agree that recent developments suggest full-dimensional explicit positional encoding may be less necessary, or even less desirable, in full-attention layers when other sequence mixers already provide positional information. Nevertheless, rotary positional mechanisms continue to play an important role, increasingly through more selective application. Notably, recent architectures such as Qwen3.8 combine Gated DeltaNet with attention layers that retain partial RoPE, while Gemma 4 similarly employs partial RoPE in its global attention layers alongside sliding-window attention.

We believe ROC could complement this direction: by contracting content-dependent orientation, it may enable more effective positional modeling with fewer rotary dimensions, improving utilization of the remaining rotary subspace. Thus, while we agree that full-dimensional RoPE may become less prevalent in global attention layers, we see ROC as potentially useful within this architectural evolution, rather than dependent on the widespread use of full-dimensional RoPE.

Another interesting observation is that Mamba-3 introduces data-dependent RoPE into its recurrence despite already incorporating positional information through its decay dynamics, suggesting that rotary mechanisms may provide complementary benefits even when positional information is already encoded through other means.

Reviewer_9LShOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper notes that in RoPE each two-dimensional rotary pair of a query or key already has a content-dependent orientation ϕ\phi before the positional rotation, so the content phase ϕq−ϕk\phi_q-\phi_k and the positional phase (m−n)ωi(m-n)\omega_i enter the same cosine term of the attention logit. It proposes Rotary Orientation Contraction (ROC): before RoPE, each pair xx is replaced by (1−α)x+\alphaxe1(1-\alpha)x+\alphaxe_1, a Cartesian interpolation toward a same-magnitude vector on the first axis. The coefficient α∈[0,1]\alpha\in[0,1] is learned through a sigmoid (initialized at 0.5), shared by queries and keys, and by default set per layer, head and rotary pair; α=0\alpha=0 recovers RoPE and α=1\alpha=1 removes the pre-RoPE orientation. RoPE's dimensionality, frequencies and rotations are unchanged; ROC is applied before QK normalization and RoPE.

Validation uses decoder-only LMs pretrained on FineWeb-Edu at two scales labeled 124M and 355M, with parameter counts excluding the input embedding matrix, trained on 4B and 10B tokens respectively. The main experiments use QK normalization and gated attention. ROC lowers validation perplexity from 19.30 to 19.07 at 124M (three matched seeds) and from 14.13 to 14.00 at 355M, ahead of an adapted PoPE baseline (19.20 / 14.05). Ablations show that one coefficient per layer captures nearly all of the gain; magnitude-preserving principal-angle contraction also helps, but less (19.17 / 14.06); ROC gives no gain without QK normalization (+0.03 PPL); and higher-frequency pairs generally learn larger coefficients. In a compiled single-Transformer-block benchmark on an H200, forward throughput is essentially unchanged, while forward+backward throughput is 0.946×–0.986× of RoPE.

Critical strengths

S1. A lightweight modification with a consistent gain over a matched baseline. ROC is a per-pair 2D map with only 4,608 contraction parameters at the 124M scale, recovers RoPE exactly at α=0\alpha=0, and preserves standard RoPE's dimensionality and rotations. The 124M improvement (19.30±0.0219.30\pm0.02 to 19.07±0.0219.07\pm0.02 over three matched seeds) is large relative to the reported seed variation, and an improvement is also observed at 355M. These gains are measured on top of QK normalization and gated attention. Forward throughput is essentially unchanged in the compiled single-block benchmark. Retaining RoPE's paired representation is a practical structural advantage over PoPE's doubled number of rotary components, although an end-to-end runtime advantage over PoPE is not established.

S2. Ablations that make the result informative. The magnitude-preserving principal-angle variant also beats RoPE (19.17 / 14.06), so learned control over the pre-RoPE orientation helps on its own; a single coefficient per layer already captures nearly all of the gain (19.090 vs. 19.080); and the learned coefficients are systematically larger for high-frequency pairs, consistent with Barbero et al. (2025), who find that high frequencies carry positional patterns. The null result without QK normalization is reported rather than hidden. Together these make the paper a useful, reproducible data point on the RoPE design space, even though the mechanism remains open (W2).

Critical weaknesses

W1. Thin evidence of significance: a small gain, at small scale, on a single in-distribution metric. The improvement is about 1.2% in perplexity (≈0.012 nats/token) at 124M and 0.9% (≈0.009 nats) at 355M, where apparently a single run is reported and the gain appears to shrink with scale. The only metric is validation perplexity on FineWeb-Edu at the 2048-token training length: there is no downstream evaluation, no evaluation beyond the training context and no test with context-extension methods, although length generalization and long-context behaviour are central criteria for positional encodings (and the stated goal of CoCA, the closest prior work constraining query–key orientation). In addition, learning rates appear to be shared across methods rather than tuned for each; the baselines omit simple alternatives that also separate content from position, such as p-RoPE (Barbero et al., 2025, already cited) or CoCA (Zhu et al., 2024); and forward+backward throughput is 0.946× at width 1024 (not reported for the 768-wide 124M model), so it is unclear whether the gain survives a wall-clock-matched comparison. For an effect of this size, these gaps decide whether ROC matters in practice; this is the main reason for my rating.

W2. The causal explanation of the gain remains insufficiently isolated. The experiments support the usefulness of modifying pre-RoPE geometry in the evaluated setting, but do not establish that the gain is specifically caused by reducing content–position interaction. The dependence on QK normalization is clearly reported and discussed, including for the magnitude-preserving principal-angle variant in Appendix A.2, but its source remains unresolved. The abstract should qualify the scope of the improvement to the QK-normalized configurations. The principal-angle variant recovers part, but not all, of ROC's improvement in Table 3. However, this comparison changes the angular map and its branch-cut behavior as well as preserving magnitude, so it cannot isolate the contribution of magnitude modulation. The authors already hypothesize that ROC's continuous handling of the antipodal region contributes to its advantage; a control matching ROC's output direction while restoring the original pair magnitude would help distinguish these effects. For the reported per-layer coefficients below 1/21/2, ROC is an invertible geometric warp before QK normalization, rather than an information-erasing map. This does not negate its usefulness, but motivates clarifying what aspect of contraction drives the improvement. Expanding the transformation also yields cross terms and a cosine term without explicit original content phase; after head-level normalization, however, its coefficient can still depend on content orientation. These observations leave several plausible explanations—angle warping, pair-magnitude modulation, and their interaction with QK normalization—which the current experiments do not fully separate.

Questions for authors

Q1. Mechanism control. At the 124M scale, could you compare ROC with a variant that preserves exactly ROC's output direction but restores the original pair magnitude? The principal-angle baseline changes both the angular map and branch-cut behavior, so it does not isolate magnitude modulation. Please specify the numerical treatment of zero or near-zero transformed pairs.

Q2. Beyond in-distribution perplexity. Using the existing checkpoints: perplexity at lengths beyond 2048, behaviour under a training-free context-extension method (e.g., NTK-aware scaling), and a few zero-shot benchmarks at 355M. Because ROC adds a content-phase-free positional term (W2d), I would expect its length-generalization behaviour to differ from RoPE's, in either direction.

1 comment

We thank the reviewer for their thoughtful assessment and constructive suggestions. We address the concerns regarding practical significance and mechanistic interpretation below.

W1/Q2 — Practical significance and additional evaluation.

We have added zero-shot evaluations on six downstream benchmarks, following the selection used in PoPE. ROC achieves the highest average accuracy at both 124M (56.71% vs. 56.44% for RoPE) and 355M (63.31% vs. 62.79%), improving over RoPE on four and five of the six tasks, respectively. These results show that the improvements are not restricted to in-distribution perplexity.

Regarding scaling, our corrected experiments show comparable relative perplexity reductions at 124M (1.14%) and 355M (1.06%). While larger-scale validation remains important, we emphasize that ROC's negligible measured forward-pass overhead makes even modest improvements potentially valuable. Thus, even if relative gains diminish at larger scales, this would not necessarily undermine ROC's practical utility, given its minimal additional inference cost.

We have also evaluated zero-shot length extrapolation at 4096, 8192, and 16384 tokens, beyond the 2048-token training context. At 124M, ROC performs better than RoPE at 4096 and 16384 tokens but worse at 8192 tokens; at 355M, it performs comparably at 4096 tokens but worse at 8192 and 16384 tokens. These results indicate no consistent extrapolation advantage. We do not position ROC as a context-extension method, and acknowledge that its compatibility with dedicated extension techniques remains to be evaluated.

W2/Q1 — Mechanistic interpretation and magnitude-restoration control.

We agree that the experiments do not fully isolate the mechanism behind ROC's advantage. However, we wish to clarify the scope of our claims. The principal-angle contraction experiment was intended to compare two natural realizations of orientation contraction, rather than establish a complete causal explanation. Likewise, the proposed role of continuous behavior near the principal-angle branch cut was explicitly presented as a hypothesis, with the manuscript acknowledging that magnitude variation may also contribute.

We agree with the reviewer that a magnitude-restored variant of ROC provides a useful additional control. We therefore evaluated a variant that retains the direction produced by Cartesian interpolation while approximately restoring the original pair magnitude, using a relative regularization term (ϵ=0.01\epsilon=0.01) in the normalization denominator for numerical stability near zero.

At 124M, magnitude-restored ROC achieves perplexities of 19.18 and 19.19 across two runs, compared with 19.16 ± 0.02 for principal-angle contraction and 19.09 ± 0.01 for standard ROC (three runs each). Magnitude-restored ROC thus performs worse than both alternatives. Since it retains ROC's angular mapping while largely removing its magnitude modulation, these results suggest that the magnitude changes induced by Cartesian interpolation contribute to its advantage. Its underperformance relative to principal-angle contraction further indicates that the Cartesian-derived angular mapping alone does not account for ROC's superior performance in this setting.

We thank the reviewer for suggesting this additional control, which helps clarify the role of contraction geometry without requiring a complete mechanistic decomposition.

Comments on the paper

1 comment

Following submission, we identified an unintended positional interpolation factor of 2 in our original experiments. This setting was applied consistently across methods, so the comparisons remained controlled. We have corrected the setting and updated the results, with only minor numerical changes and no change to our main conclusions.

Method124M355M
RoPE19.31 ± 0.0214.16
PoPE19.20 ± 0.0414.04
ROC19.09 ± 0.0114.01

All values report FineWeb-Edu validation perplexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.