- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper observes that coordinate-wise second-moment scaling in Adam-style optimizers changes not only the magnitude of an update but also its direction, and proposes a way to quantify and directly control this rotation. For a vector scaled by positive per-coordinate factors , the angle between and is exactly given by , where . The rotation is therefore the energy-weighted anisotropy of the scale on the coordinates that carry the update. Building on this identity, the authors propose rotation-budgeted preconditioning. At every step, it selects the preconditioning exponent so that the applied update stays within a target angle of the momentum direction, using the strongest preconditioning that satisfies this constraint.
Critical strengths
This paper shows, both analytically and empirically, that adaptive optimizers such as Adam influence the direction of an update and not only its magnitude. Analytically, the authors derive that the rotation induced by coordinate-wise scaling is exactly , which reduces the change in direction to a single interpretable quantity. Empirically, they measure that second-moment scaling produces rotations of roughly or more on both Qwen2.5-7B and Mistral-7B, and that this rotation persists throughout training. However, this strength amounts to a quantification of the fact that the optimizer changes the update direction. It does not establish that this rotation is harmful to learning, nor that the proposed control is practically beneficial, so it is insufficient to offset the weaknesses described below.
Critical weaknesses
- The manuscript is too poorly written and incomplete to allow its claims and contributions to be assessed.
The paper is very hard to read and its context is difficult to follow. Key concepts (RObE, pva244, construction, and the P1 margin) are used without being defined in the main text, and even the meaning of the held-out P1 margin, the only performance metric, cannot be determined. The figures are not tied together or explained within the narrative, and some are hard to reconcile with the reported numbers (e.g., Figure 3(d) versus Table 5). The appendix retains draft text such as REQUIRED FOR SUBMISSION'' and artifact requirement,'' so the manuscript does not read as a submission-ready paper. The paper also does not clearly state what its contribution is (the identity, the controller, or the performance gain) or how it differs from prior work. Without the evaluation metric and objective, the experimental results cannot be verified. This is therefore not a matter of presentation alone, but one that prevents any assessment of the validity of the results.
- There is no in-depth analysis of how the rotation affects learning, and the motivation for the proposed method is not established.
The paper measures that second-moment scaling rotates the momentum direction, but it does not analyze how this rotation affects learning. It does not justify why the momentum direction should serve as the reference, why a budget of is appropriate, or why rotation should be regarded as harmful in the first place. The central theoretical result, , is an algebraic identity that follows directly from the definition of the cosine, so its numerical validation carries little information. In the mechanism controls (Table~10), the ordering of outcomes does not follow the rotation angle, and this is left unexplained. The performance evidence rests on a single undefined metric, three seeds, and a comparison without learning-rate tuning, and the proposed dynamic budget improves on fixed by only . The central claim that controlling rotation is beneficial is therefore not supported.
Questions for authors
Questions:
Q1. Metrics, objectives, and implementation details.
Please define held-out P1 margin, construction, RObE, and pva244 in the main text. In particular, what is the P1 margin a difference of, what are its range and interpretation, and does it correlate with standard metrics such as pass@? Also, appears to follow directly from the definition of the cosine. What exactly is novel, and how does it differ from partially adaptive methods (e.g., Padam), the Kantorovich inequality, and trust-ratio methods (LARS/LAMB)? Finally, for , which algorithm is used to find ? Were there training steps where was non-monotone in ? What value of is used? Please also resolve the inconsistency between Figure3(d) and Table5, and the differing canonical rotation angles (, , and ).
Q2. Reference direction, budget, and causality of rotation.
Why should rotation be considered harmful, and why is the momentum direction the right reference? On Mistral, the raw gradient and momentum already differ by . Do the conclusions hold if the reference is changed (e.g., to the raw gradient or the natural gradient)? How was chosen, and does the effect persist at other values of ? In Table~10, the ordering of outcomes does not follow the rotation angle. How do you explain this, and is there an experiment that directly tests the hypothesis that rotation itself drives performance? Could the roughly rotation arise from coordinates near the floor or from scale differences between LoRA and matrices? Please provide per-module and per-matrix breakdowns.
Q3. Fair performance comparison.
Does AdamW remain the lowest when the learning rate is tuned separately for each arm? The gap in seed~2 () dominates the mean. What happens with more seeds? Does the difference between ROT BUDGET and a tuned fixed exceed seed-to-seed variation?
Suggestions for Future Work (do not affect my assessment)
-
Reporting downstream performance on Mistral (e.g., held-out loss or pass@) would allow the practical value of the control to be judged.
-
Providing per-checkpoint training curves would reveal learning dynamics that endpoint values alone cannot show.