- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The optimizers ignore RoPE's different frequency assignment to keys/queries. The paper identifies the sensitivity of each key/query pair and proposes a position-aware preconditioning, which keeps the length-changing part and rescales the turning part. Their preconditioning-aware mechanism can be cheaply added to the optimizers.
Critical strengths
- The paper proposes a cheap and lightweight wrapper with no extra cost to the optimizer states.
- They build a bridge between RoPE and preconditioning for optimizers.
- The paper has good ablations.
Critical weaknesses
Minor issues:
- The second and third paragraphs of the introduction need more citations.
- What is R in R(h\theta_i)?
- Mention what this sign, "⪰", means.
- It may be good to add the training/validation loss to Table 1 as well.
- What is your GPU setup?
Serious issues: 6. Unfortunately, the paper is not clear and is not very well-written. 6. The baselines are limited, and the models aren't big. 6. The paper uses some heavy maths and formalisms, which makes the paper hard to understand. 7. Some numbers don't match, e.g., Table 7’s AdamW number 3.420 is not the same as AdamW in Table 3: 3.441. The setup behind Table 7 needs to be explained.
Questions for authors
- I wonder how you handle the muon's orthogonality? You didn't address it!
- I also didn't understand why the sequence length is small. (sequence length of 256)
6 comments
2. Experimental Scale and Sequence Length (regarding issue 6(2), 7, and Q2)
We evaluate PA through paired comparisons with AdamW and Muon, which construct updates in substantially different ways. Both optimizers are tested on the 130M and 350M LLaMA-style models, and we additionally evaluate the Muon v.s. PA-Muon pair on Qwen2.5-0.5B. Figures 1–2 show fewer updates to reach the corresponding baseline training loss across these comparisons, while Table 1 reports modest per-step overhead. We select learning rates separately for each method following the protocol in Appendix E.3. These comparisons test whether the RoPE-aware correction provides additional benefit across different base optimizers and model settings. Our model sizes reflect the resources available for comparisons with adequate training budgets, and validation at substantially larger scales remains valuable future work.
To examine a longer sequence length, we additionally train the 130M model at sequence length 1024, where both PA variants achieve lower final validation loss than their corresponding base optimizers. These runs use 5,000 updates and a global batch of 256 sequences, giving approximately 1.31B tokens and matching the token budget of our original 130M experiments. All four runs use 500 warmup updates, and training/evaluation seed 42. Both PA variants use . The method-specific learning rates and results are:
| Method | Main learning rate | Fallback AdamW learning rate | Final validation loss |
|---|---|---|---|
| AdamW | — | 3.237305 | |
| PA-AdamW | — | 3.198975 | |
| Muon | 3.101807 | ||
| PA-Muon | 3.063965 |
These results show that the validation-loss improvements also appear at sequence length 1024 in the tested 130M setting.
For the main experiments, we choose sequence length 256 to balance training cost with maintaining long training trajectories and completing tuning and ablations within our available compute budget. With a global batch of 256 sequences, the 130M, 350M, and Qwen2.5-0.5B runs use 20K, 60K, and 80K updates, corresponding to approximately 1.31B, 3.93B, and 5.24B tokens, respectively (Appendix E.1). Within each baseline–PA comparison, we match the sequence length, global batch size, and training budget. This sequence-length choice also has been commonly used in optimizer research. For example, GaLore [1] and LDAdam [4] use a maximum sequence length of 256 for its LLaMA/C4 pretraining experiments.
Regarding the AdamW values in Tables 3 and 7. These values correspond to different experimental configurations and are therefore not conflicting results for the same experiment. Table 3 reports the 130M AdamW configuration selected from the learning-rate sweep, with learning rate , evaluation seed 42, and validation loss 3.4413 (Appendix E.3). Table 7 reports the fixed-rate ablation configuration, with learning rate , evaluation seed 12,345, and validation loss 3.42035 (Appendix E.5). Both use training seed 42 and 10,000 updates. Appendix E.5 explicitly notes that the ablation protocol differs from the sweep protocol in both learning rate and evaluation seed.
3. Clarifications on Notation, Experimental Details, and References (regarding the minor issues)
Regarding notation (Minors 2 and 3), the original submission defines inline in Sec. 2 (Lines 110–113) as the planar rotation matrix with angle . Thus, represents rotation by , where is the relative token displacement and is the RoPE frequency of pair . To make this definition easier to locate, we now display it separately as Eq. (1). We use in its standard matrix-order sense: means that is positive semidefinite, equivalently for every . For clarity, we now state this convention explicitly in Sec. 3.3; in particular, means that is positive semidefinite.
For experimental reporting (Minors 4 and 5), we have added the plotted endpoint training and validation losses directly to Sec. 5.1. The (training, validation) loss pairs change from baseline to standard PA as follows: at 130M, for AdamW and for Muon; at 350M, for AdamW and for Muon. For Qwen2.5-0.5B with Muon, the corresponding pair changes from to . We have also specified the GPU models at the beginning of Sec. 5. For the main experiments, the 130M and 350M LLaMA-style runs use 2 and 4 NVIDIA L40S GPUs, respectively, with 48 GB memory per GPU; Qwen uses 4 NVIDIA A800 GPUs with 80 GB memory per GPU. The GPU counts are reported in Appendix E.1.
Regarding references (Minor 1), the second and third paragraphs of our introduction primarily develop our own research motivation and the reasoning leading to PA, so not every step corresponds to a result drawn from prior work. To make the relevant background statements more explicit, we have added a citation to Martens [2] in the second paragraph, supporting the distinction between parameter changes and changes in model behavior, and to Bottou et al. [3] in the third paragraph, supporting the connection between local geometry and admissible step sizes. These additions provide references for the background underlying our discussion while retaining the presentation of our own RoPE-specific reasoning.
We thank the reviewer for the constructive feedback and would appreciate a reassessment in light of the revisions and additional experimental results. We would be very grateful if you could consider raising your score.
References
- Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., & Tian, Y. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. Proceedings of the 41st International Conference on Machine Learning, 235, 61121–61143.
- Martens, J. (2020). New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research, 21(146), 1–76.
- Bottou, L., Curtis, F. E., and Nocedal, J. (2018). Optimization Methods for Large-Scale Machine Learning. SIAM Review, 60(2), 223–311.
- Robert, T., Safaryan, M., Modoranu, I. V., & Alistarh, D. (2025, May). Ldadam: Adaptive optimization from low-dimensional gradient statistics. In International Conference on Learning Representations (Vol. 2025, pp. 4778-4814).
Response to Reviewer D5pd
We thank the reviewer for the comments and suggestions. We provide a detailed clarification to address your concerns below.
1. Presentation and Clarification of Our Motivation and Method (regarding issues 6(1), 6(3), and Q1)
Thank you for pointing out the presentation issues. We have revised Sections 2–4 by bringing key formulas and lemma statements from the Appendix into the main text and making the connections between our positional analysis, the attention response, and the PA update more explicit. Below, we provide a brief overview of our motivation and method, with references to the revised manuscript.
Motivation Overview. We begin with the two-dimensional parameter cells associated with each RoPE frequency and separate an update’s local scale and turn components. After normalizing out the cell magnitude, we find that the same local turn can produce different positional responses across rotary pairs, which we quantify through the sensitivity Si (Eq. (3); Proposition 2.1).
We then trace these turns into actual attention scores and find that the same rotary factor appears in the RoPE-specific response and its loss-weighted local GGN bound. This connects our positional measurement to the training objective while keeping this isolated component distinct from the full loss curvature (Lemma 3.1; Proposition 3.3).
Based on this connection, we preserve the base optimizer’s local scale component and adjust its turn using RoPE information, giving the PA update in Eq. (22). We then evaluate this correction in pretraining with both AdamW and Muon, where our results show fewer updates to reach matched training loss with modest per-step overhead.
Regarding Muon. PA acts on the query/key cells after Muon has completed its usual momentum, Newton–Schulz orthogonalization, and shape scaling, so the final corrected update generally does not preserve matrix orthogonality. PA preserves the base proposal’s local scale component and adjusts its turn. We now state this ordering and its effect on orthogonality explicitly in Section 4.1. We derive this correction from RoPE’s positional structure and use AdamW and Muon as two different base optimizers that supply the initial update. Our goal is to incorporate this additional model-specific information into the update and evaluate whether the correction improves pretraining efficiency. The paired Muon comparisons in Figures 1–2 show that PA-Muon reaches the baseline training loss in fewer updates, with modest per-step overhead reported in Table 1.