- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ROC (Rotary Orientation Contraction) which enables transformer models using RoPE to control how much pre-RoPE query-key vector orientations interact with the rotations applied by RoPE itself. ROC applies a linear interpolation between a vector and its rotation onto the first coordinate axis, and the degree to which the interpolation is applied is learned by the model during training. Language modeling performance is evaluated through held-out perplexity on FineWeb-Edu at two model sizes, the smaller of which is trained on three seeds. ROC is compared against RoPE and PoPE. The efficiency of ROC is also tested against RoPE. Models using ROC are shown to achieve lower perplexity than baselines with QK normalization.
Critical strengths
- The idea of allowing the model to learn how much pre-RoPE orientation to keep is interesting and sound.
- ROC is presented in a way that is intuitive to understand for readers who have the background.
- The efficiency study and the principal-angle contraction ablations are insightful.
Critical weaknesses
-
The abstract correctly claims improvements in "perplexity", but there are several places in the rest of the paper that claim improved "language modeling performance" (intro, section 3.2) which is too broad to be supported by better perplexity without downstream evals, since perplexity gains may not translate to capability [1].
-
Section 3.3 claims that some of the variations provide "marginal additional gains", but all variations are single seed runs and fall well within the seed variance reported by the previous section for the ROC model. Similarly, section 3.5 claims that ROC "provides no benefit without normalization" over RoPE, but single seed runs are not sufficient to support this claim. Relating to this, the claim that ROC improves perplexity "across language-modeling experiments" in the abstract should be made more specific, as it already is in the intro which states "when combined with QK normalization".
-
Only the 124M run in tables 1 and 3 seems to have three seeds, the 355M and tables 2, 4, and 7 are single seed runs. The paper also doesn't mention what the seeds vary (just initialization, or also data/data order?). If it's just initialization, there's a chance that the SD is understated. Moreover, with only three seeds, the SD estimates themselves are likely unreliable.
-
Based on the reported numbers and sample size, the empirical results for the main claims except that of ROC vs RoPE are unlikely to be statistically significant. I asked an LLM to write a script to run a Welch t-test on the results and compute the p-values with Holm-Bonferroni corrections (and computed some of the values manually myself to confirm), here's the results: | Claim | Δ perplexity | p raw | p Holm | |------------------------------------------|--------------|----------|----------| | C1: ROC vs RoPE | -0.23 | 1.47e-04 | 7.37e-04 | | C2: PoPE vs RoPE | -0.10 | 0.0316 | 0.0633 | | C3: ROC vs PoPE | -0.13 | 0.0158 | 0.0633 | | C4: Principal-angle vs RoPE | -0.13 | 0.0158 | 0.0633 | | C5: ROC vs Principal-angle | -0.10 | 0.0316 | 0.0633 |
-
Because SDs are reported to two decimals, the Holm-corrected values could range roughly between 2e-4 and 3e-3 for C1, and between 0.03 and 0.13 for C2-C5. This means that the p-values may not even clear the commonly used bar for C2-C5. In addition, for ML results, I agree with [3] that we should be skeptical about anything greater than since the number of possible hypotheses is large (also citing [4], which treats p-values between 0.01 and 0.05 as a marker of fragile and less replicable findings). Again, these are only based on the reported numbers and may not be exact, the authors are welcome to re-compute these values and run their own statistical tests using the exact data.
-
Since a single training budget is used across methods at each model size and no loss curve data is provided, the "performance" gains could just be ROC training faster initially where RoPE could catch up or surpass it later if the models were to be trained longer.
-
It seems that one shared hyperparameter setting is used for all the methods and both sizes. The difference in perplexity could be a result of untuned optimizer parameters.
-
From the setting used for training mentioned in the paper (section 3.1 and appx A.1), the PoPE baseline was modified (QK-norm, gating, using Muon for hidden matrices instead of AdamW throughout). This makes sense for a fair comparison, but according to table 4, ROC is either worse than or on par with RoPE without QK-norm, whereas the original PoPE paper (which does not mention QK-norm or gating) reports gains over RoPE. This might suggest that ROC only performs better than or on par with PoPE under a setting that favors it, or that PoPE is more robust to other variations in the architecture.
-
The scaling trend is not clear from the provided data. While the gap between the perplexity achieved by the models seems to shrink from 124M to 355M, it cannot be said whether there's no difference in the perplexity due to the lack of multiple seeds, or whether ROC scales worse due to the lack of results for at least one more model size.
-
For a positional encoding paper, I believe it is important to test length-extrapolation and long context performance. It is even more relevant in this case because the baseline (PoPE) explicitly tests length extrapolation. [2] shows the effects of training on short contexts and evaluating zero-shot on longer ones across many PE methods.
[1] Zeyuan Allen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[2] Ben Anson, Xi Wang, and Laurence Aitchison. Scale-invariant attention. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[3] Neel Nanda. Highly Opinionated Advice on How to Write ML Papers. AI Alignment Forum, 2025.
[4] Paul C. Bogdan. One Decade Into the Replication Crisis, How Have Psychological Results Changed? Advances in Methods and Practices in Psychological Science, 8(2), 2025.
Questions for authors
- What do the seeds vary? If the seeds only vary initialization, what effect does also varying data (chosen from different 4B/10B subsets of FineWeb-Edu) and/or data order have on the resulting SD of the perplexity values?
- Can the authors share per-seed values and loss curves?
- Can the authors provide a comparison with PoPE under the training and architecture settings reported by its authors, and/or add PoPE to table 4?
Suggestions:
Overall, stronger experiments and empirical results would significantly benefit the paper. The main ones are tuning hyperparameters (at least learning rate) per model per size for the main comparisons, long-context extrapolation results, and running with more seeds for statistically reliable results. The authors could determine the approximate number of seeds required through power analysis using the existing data or similar.
We thank the reviewer for their detailed assessment and constructive suggestions. We have expanded the empirical evaluation and clarified several claims in response to these concerns.
Statistical reliability and experimental variation.
We agree that the limited number of seeds warrants caution, particularly when interpreting differences between secondary ablations. However, we wish to distinguish these exploratory comparisons from our primary claim that ROC improves perplexity over matched RoPE baselines. Using the corrected individual-seed results, we independently repeated the reviewer's two-sided Welch's t-test and obtained p = 0.000706. Even applying a conservative Bonferroni correction across five comparisons yields p = 0.00353, confirming that the primary improvement remains statistically significant. We have also revised the interpretation of secondary ablations, particularly contraction-coefficient granularity, to avoid drawing strong conclusions from small single-run differences.
The three primary 124M runs vary model initialization, while holding the training data and its order fixed. Individual-seed perplexities are provided in Appendix A.2, and we have additionally included validation-loss curves in the revised manuscript.
To provide further evidence across training configurations, we conducted three additional runs per method using a substantially different 49.4M-parameter architecture (512 hidden dimensions, 12 layers, 8192-token vocabulary), trained for 5B tokens using a shuffled-data training pipeline. ROC again improves validation cross-entropy (2.5864 ± 0.0006 vs. 2.6077 ± 0.0021 for RoPE). Although this does not isolate data-order variance, it provides complementary evidence that ROC's improvement persists under a different training configuration.
Hyperparameter sensitivity and convergence.
We agree that shared hyperparameters do not guarantee independently optimal performance for each method. ROC is a relatively non-invasive modification to RoPE, and our experiments were designed to evaluate its effect under matched optimization conditions rather than claim independently optimized performance for every method. We have explicitly acknowledged this limitation in the revised manuscript. Nevertheless, the consistent ROC–RoPE improvement across matched seeds provides evidence that the benefit is not merely an isolated training outcome.
Regarding convergence, we have included validation-loss curves to better characterize training dynamics. Furthermore, the additional 49.4M experiments retain ROC's advantage after 5B training tokens, corresponding to over 100 tokens per parameter. While this does not eliminate the possibility of different behavior at still larger training budgets, it provides evidence that ROC's advantage is not restricted to short training runs.
PoPE comparison and QK normalization.
We agree that PoPE's more substantial reparameterization may respond differently to optimization settings. Our primary experiments compare methods under a shared training configuration, rather than claiming independently optimal performance for PoPE.
We additionally evaluated PoPE without QK normalization or gated attention under our training recipe. In this setting, PoPE achieves a validation perplexity of 19.50 compared with 19.73 for RoPE, confirming that PoPE can improve performance without these components. We have also clarified ROC's dependence on QK normalization in the abstract.
Downstream performance and length extrapolation.
To address the concern that perplexity improvements may not translate to broader capabilities, we have added zero-shot evaluations on six downstream benchmarks, following the selection used in PoPE. ROC achieves the highest average accuracy at both 124M (56.71% vs. 56.44% for RoPE) and 355M (63.31% vs. 62.79%), outperforming RoPE on four and five of the six tasks, respectively.
We have also evaluated zero-shot length extrapolation at 4096, 8192, and 16384 tokens, beyond the 2048-token training context. At 124M, ROC outperforms RoPE at 4096 and 16384 tokens but underperforms at 8192 tokens. At 355M, ROC performs comparably at 4096 tokens but worse at 8192 and 16384 tokens. These results indicate no consistent extrapolation advantage, which we acknowledge in the limitations and document in the appendix.