- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Rotated Manifold Optimization (RoM), an optimizer for LoRA that operates on the rank-r Riemannian manifold. RoM applies row and column normalization in a coordinate frame defined by the left singular vectors of the tangent momentum, followed by tangent-space projection and retraction. By exploiting the low-rank structure of tangent vectors, the method avoids materializing full weight-sized matrices. Experiments cover supervised finetuning and reinforcement learning, demonstrating consistently lower validation loss and more pronounced downstream improvements in the RL setting.
Critical strengths
- The paper evaluates RoM on mathematical reasoning and code generation under SFT, as well as mathematical reasoning under GRPO. The SFT experiments cover multiple model families and sizes, compare several LoRA optimizers.
- The factorized implementations of gradient computation, projection of the previous momentum onto the current tangent space, rotation, and retraction preserve the memory efficiency of LoRA. Reducing the required SVDs to small matrices makes the proposed method has practical utility.
- RoM achieves lower validation loss across the experiments.
Critical weaknesses
- Section 3.2 cites prior work on rotated optimization and multi-norm normalization, and explains how the low-rank structure makes the left singular frame efficient to compute. However, the paper provides limited justification for preferring this frame over alternative orthogonal bases or a right-sided formulation, and for adding column normalization beyond row normalization.
- The relationship between the normalization objective and its implementation requires clarification. Equation 16 constrains both the maximum row norm and the maximum column norm to equal one, whereas Algorithm 3 alternately normalizes every row and every column. For a reduced matrix with , ending with unit-norm columns gives , while a maximum row norm of one would imply . Thus, the returned matrix cannot satisfy both stated constraints in this setting. The paper should clarify the dimension-dependent scaling or the precise sense in which Algorithm 3 approximates Equation 16.
- The RL comparison includes only Adam and uses a single model and training recipe.
Questions for authors
Section 3.2 motivates the design through prior work on rotated optimization and multi-norm normalization. However, the specific advantages of using the left singular basis and adding column normalization remain insufficiently explained. Could the authors clarify these choices and provide controlled comparisons with alternative bases and a row-only normalization variant?
6 comments
W2. The relationship between the normalization objective and its implementation requires clarification.
We acknowledge that the previous normalization objective Eq (16) was not accurate. We missed the constants in the norm constraints. The corrected normalization objective is:
To achieve this, Algorithm 3 operates directly on the rotated representation , but is equivalent to the classical Sinkhorn algorithm [1] applied to the element-wise square of , and converges to a fixed point whose rows have the same norm and columns have the same norm [2]. In each iteration:
In our implementation of Algorithm 3 (and the initial version of the submission), we didn't multiply by the two constants because:
- Column normalization is invariant to scalars, so has no effect
- depends only on the rank and is the same for all LoRA layers, so we absorbed it into the learning rate, and the learning rate was swept in all experiments
Therefore, whether or not the two constants are explicitly multiplied in Algorithm 3 won't affect our results or findings
Updates in revised pdf (marked in red): We have corrected the constants to make the normalization objective and algorithm consistent. We have moved Algorithm 3 to the main body (now it's Algorithm 1), and have added sentences explaining its connection to the Sinkhorn algorithm and why explicitly multiplying by these constants is not necessary in practice.
[1] Concerning nonnegative matrices and doubly stochastic matrices. Sinkhorn & Knopp, 1967
[2] Gradient Multi-Normalization for Efficient LLM Training. Scetbon et al. NeurIPS 2025
W3. The RL comparison includes only Adam and uses a single model and training recipe.
We have added the RL results for Riemannion to Figure 2 and Table 4 (numbers are in our first comment). We tuned its learning rate and used the same experimental setup. RoM improved performance faster than Riemannion and also achieved better final performance
We hope our response and revision can address the reviewer’s concerns. We would be grateful if the reviewer could consider revising their score. We are happy to answer any further questions during the discussion period.
Kind Reminder
Dear Reviewer McSh,
Thank you once again for your time and feedback! As the discussion period nears its end, we would greatly appreciate it if you could take a moment to check our rebuttal and revision. If you have any further questions or concerns, we would be more than happy to address them!
Thanks
We thank Reviewer McSh for their time and valuable feedback. We are glad that they appreciated the breadth of our evaluation across SFT, RL, and model families, and that they found our factorized implementation practical. We respond to the raised concerns as follows.
Rotation basis: The use of left singular vectors is actually a natural choice motivated by existing matrix optimizers. Consider Muon as an example, for a gradient G with its SVD decomposition: G=USV⊤, Muon normalizes G as Muon(G)=UV⊤. Under the rotation-normalization framework, Muon can be exactly recovered if we choose U as the rotation basis and row-normalization as the normalization function applied to the rotated representation U⊤G. Specifically, given the orthogonality of U and V, we have: Urow-normalize(U⊤G)=Urow-normalize(SV⊤)=UV⊤=Muon(G)
Earlier matrix optimizers like SOAP/Shampoo consider eigenvectors of GG⊤ for preconditioners, which are exactly the left singular vectors of G. Therefore, choosing the singular vectors as the rotation basis is a natural choice extending the design of existing matrix optimizers.
Which side to rotate: In the paper, we apply rotation to the left side of the adapter W (thus using left singular vectors) for ease of illustration of the idea. In implementation, we rotate the side with the smaller dimension for computational efficiency. If W is square, we rotate the side that faces the input (exactly the left side in this paper's notation)
As an ablation study of this design choice, we experimented with rotating the output side of each square adapter (using singular vectors of the output side) and compared the final held-out loss (×10−3) with our original design. Experiments were based on math SFT, using Llama 3B and 8B with rank 16. All hyperparameters were the same as those for our main results.
The difference is on the same order as seed variance and negligible compared with difference between optimizers
Row-only normalization variant: From the above Muon example, we can see that Muon is equivalent to row-normalization in the singular-vector frame. Plugging the row normalization as the function h into RoM's update rule (Eq 17), we can get that the row-only normalization variant of RoM is equivalent to applying Muon first and then projecting the result onto the tangent space of the fixed-rank manifold, which is exactly Riemannion. Therefore, RoM and Riemannion have already formed a controlled comparison regarding the additional column normalization, and RoM is consistently better than Riemannion.
We have added the RL results for Riemannion, and the results further validate the advantage of RoM's normalization