acceptodds
Under review as a conference paper at ICLR 2027

Demystifying Manifold Constraints in LLM Pre-training

Abstract

The recent success of matrix optimizers (e.g., Muon) suggests that specific normalization of momentum—such as orthogonalization and row-wise normalization—benefits both the stability and acceleration of LLM training. Consequently, several recent studies have suggested that weights should also be normalized, leading to a Riemannian optimization problem. While such constrained training frameworks demonstrate superior performance, the effects of explicitly constraining weights, and their interaction with existing stabilization mechanisms, remain less understood. To bridge this gap, we study manifold constrained training dynamics through activation scales, rotational dynamics, and the update-to-weight ratio. We propose a Riemannian spectral steepest descent optimizer called MACRO, alongside a radius selection principle to serve as our testbed. Our analysis and numerical experiments reveal that RMSNorm and manifold constraints serve overlapping roles, and that weight decay can be completely eliminated when manifold constraints are applied. By controlling the update-to-weight ratio, constrained training significantly alleviates update cancellation, empirically demonstrating that MACRO is robust to low-precision computation and competitive with existing algorithms for standard LLM pre-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM pre-training, manifold-constrained optimization, training dynamics, normalization, weight decay, low-precision training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A8WgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how explicit constraints on weight matrices affect LLM pre-training. It proposes MACRO, an optimizer that combines tangent-space projection, matrix-sign updates, and weight normalization. The authors also derive rules for choosing constraint radii and investigate how weight constraints interact with RMSNorm, weight decay, and BF16 training.

Experiments on Qwen3-like models ranging from 240M to 1.2B parameters show that MACRO improves validation perplexity over the original Muon baseline, with a relatively small computational overhead for the Frobenius variant. The paper also reports substantially smaller performance degradation under BF16 parameter training and provides theoretical analyses of weight rotation, relative update sizes, and convergence.

Critical strengths
  1. A useful analysis of weight constraints from multiple perspectives. The paper examines weight constraints through activation scales, weight rotation, and relative update sizes rather than relying only on final validation loss. This provides a useful perspective on how weight normalization may affect training dynamics, especially under low-precision computation.

  2. A practical optimizer with encouraging empirical results. MACRO uses a single-loop update and avoids the expensive inner optimization required by methods such as SSO. The Frobenius variant achieves competitive validation performance with only a small increase in per-step cost over Muon. The BF16 results are also encouraging and motivate further investigation.

Critical weaknesses

1.The claimed Riemannian steepest-descent interpretation and convergence guarantees are not fully justified for the implemented algorithm.

Section 3.1 describes MACRO as a Riemannian spectral steepest-descent method. However, Algorithm 1 first projects the momentum onto the tangent space and then applies the matrix-sign operation. The latter does not generally preserve tangency: even if ⟨W,Φ⟩=0\langle W,\Phi\rangle=0, it does not follow that ⟨W,msign⁡(Φ)⟩=0\langle W,\operatorname{msign}(\Phi)\rangle=0. Therefore, the resulting direction is not necessarily the solution to the steepest-descent problem restricted to feasible tangent directions. Figure 12 also reports nonzero tangent-space violations.

The convergence analysis in Appendix C.1 explicitly covers Frobenius and oblique constraints. For the spectral constraint, the authors acknowledge that the spectral sphere is not smooth everywhere, while the implementation uses an approximate projection rather than the exact one. The argument based on almost-sure uniqueness of the largest singular value does not clearly establish the conditions needed to extend the convergence guarantee to the implemented spectral variant.

2.The experiments do not sufficiently distinguish the benefits of weight constraints from those of MACRO's specific update rule, and the proposed mechanisms are not fully established.

The results support the usefulness of controlling weight norms, but they provide weaker evidence that MACRO's specific update rule is necessary. MuonH already enforces weight constraints and achieves performance comparable to, or sometimes better than, MACRO (Table 4). Moreover, the separately tuned Muon baseline in Table 11 nearly matches MACRO-fro in that comparison.

This distinction is particularly important for the BF16 results, which are among the paper's strongest findings. The main precision comparison includes Muon but not the norm-constrained MuonH baseline. In addition, switching to BF16 changes the storage precision of parameters, gradients, and optimizer states together. The proposed update-cancellation explanation relies mainly on a simplified Gaussian model, without directly establishing how much of the observed performance gap is caused by canceled parameter updates. As a result, the experiments do not clearly separate the effects of controlling weight norms, the specific MACRO update, and other low-precision effects.

Questions for authors
  1. What exactly is the constrained optimization problem solved by MACRO's update? Since the matrix-sign operation does not necessarily preserve tangency, can the authors clarify in what sense the resulting direction is a Riemannian spectral steepest-descent direction? In particular, what convergence guarantee applies to the practical spectral implementation using approximate projection? A precise distinction between proven results and heuristic approximations would materially affect my assessment.

  2. Can the authors isolate the source of the BF16 improvement? A key comparison would be MACRO-fro versus MuonH-fro under the same BF16 setting, since both control weight norms. It would also be useful to measure the actual fraction or magnitude of parameter updates lost to BF16 rounding, rather than relying only on a Gaussian estimate. These results could establish whether the improvement is due primarily to norm control, MACRO's specific update, or the proposed update-cancellation mechanism.

8 comments

Rebuttal Paper Revision summary:

We have uploaded a revised manuscript to address the reviewers' comments. The major updates include:

  1. Abstract and Introduction: We have removed all mentions of "Riemannian steepest-descent" and instead describe it as a "modified Riemannian steepest-descent" algorithm. We changed the introduction of MACRO in Section 3.1 accordingly.

  2. New Convergence Theorem: We limit the scope of Theorem 1 to the Frobenius and Oblique Manifolds. Moreover, we provide a new convergence result for the MACRO-spec in Appendix C.2. This new theorem closes the gap between theory and the algorithm used in practice.

  3. Distance between MACRO and SSO/FSO Updates: We monitor and report the distance between the MACRO update and the true solution of the constrained LMO subproblem in Figure 15.

  4. Real Update Cancellation under BF16: We report the empirical update cancellation ratio during actual training in Figure 6 (Section 4.3).

All changes are marked in blue.

Weakness part:

weakness 1: Therefore, the resulting direction is not necessarily the solution to the steepest-descent problem restricted to feasible tangent directions. Figure 12 also reports nonzero tangent-space violations.

Thank you for raising the fact that MACRO is not a "Riemannian steepest-descent" algorithm. It would have been better to have described it as a "modified Riemannian steepest-descent" algorithm. Consequently, we have removed all mention of "steepest-descent" in our updated manuscript. Moreover, we want to further address this concern from two perspectives:

  1. First, the convergence guarantee still holds even when we use this approximate update(please check the details in the answer of question 1).

  2. We also test the relative distance between MACRO and the true solution during training as your reference. We use the SSO and FSO updates as the true solution, and we compute the relative error between the MACRO update and the SSO/FSO update as follows:

∥UMACRO−USSO/FSO∥F/∥UMACRO∥F\Vert \text{U}_{\text{MACRO}} - \text{U}_{\text{SSO/FSO}} \Vert_F / \Vert \text{U}_{\text{MACRO}} \Vert_F

We summarize the mean values during training in the table below. Figure 15 (Appendix D.4, page 48) shows the relative error curve during training.

MatrixMACRO-frob MeanMACRO-spec Mean
FFN Up4.94%5.14%
FFN Down3.48%3.33%
FFN Gate4.45%4.96%
Attention Value7.96%6.32%
Attention Query9.26%7.01%
Attention Key9.98%7.73%

The table shows that the relative difference between the MACRO update direction and the SSO update direction is less than 10% for both variants. We hope that this supplementary materials may address your concern about the gap between MACRO and "true" solution of the steepest-descent problem restricted to feasible tangent directions.

weakness 2. MuonH already enforces weight constraints and achieves performance comparable to, or sometimes better than, MACRO (Table 4). Moreover, the separately tuned Muon baseline in Table 11 nearly matches MACRO-fro in that comparison.

First, we want to further explain Table 12 in the paper. We acknowledge that after joint tuning of the learning rate and weight decay, the Muon baseline nearly matches MACRO-frob. However, this comparison is not completely fair because MACRO only requires learning rate tuning. For example, in Table 12, Muon requires 12 tuning runs to match the performance of MACRO with only 3 runs. Furthermore, Table 12 shows that the optimal weight decay for Muon depends on the selected learning rate. This confirms that two-dimensional grid search is necessary for Muon. These results verify the benefits discussed in Section 4.2: adding manifold constraints removes the need for weight decay, making hyperparameter tuning much easier in practical training.

Questions part:

1.a. What exactly is the constrained optimization problem solved by MACRO's update?

The local constrained direction problem is the spectral steepest-descent subproblem used by SSO: min⁡O{−⟨Mt,O⟩:∥O∥2≤1, O∈TWtM}\min_{O}\lbrace-\langle M_t,O\rangle:\Vert O\Vert_2\le1,\ O\in T_{W_t}\mathcal{M}\rbrace, where MtM_t is the momentum, TWtMT_{W_t}\mathcal{M} is the tangent space, and OO is subtracted in the parameter update. MACRO approximates this direction construction through one alternating pass of tangent projection and the spectral oracle: it first projects the momentum onto the tangent space and then applies the matrix-sign operation, without an iterative inner solve to convergence.

  1. b. ...can the authors clarify in what sense the resulting direction is a Riemannian spectral steepest-descent direction? ..what convergence guarantee applies to ... using approximate projection?

The intermediate matrix-sign output need not remain tangent; we therefore describe MACRO as an inexact realization of Riemannian spectral steepest descent, with convergence established for the complete update after feasibility restoration. This interpretation can be quantified for the spectral constraint: under a uniform relative gap σ1(Wt)−σ2(Wt)≥κR>0\sigma_1(W_t)-\sigma_2(W_t)\ge\kappa R>0 (which will be empirically verified later), the first-order tangent direction induced by radial normalization, rescaled into the unit spectral ball, achieves at least a κ/2\kappa/2 fraction of the optimal linear improvement in the constrained subproblem. Thus, an exact tangent-subproblem solution is unnecessary for the convergence guarantee. Specifically, the new Theorem 2 in Appendix C.2 directly analyzes the radial normalization QR(Z)=RZ/∥Z∥2Q_R(Z)=RZ/\Vert Z\Vert_2 used by MACRO-spec and establishes an O(T−1/4)\mathcal{O}(T^{-1/4}) expected first-order stationarity rate, with gap-dependent constants, under the stated smoothness, stochastic-gradient, stepsize, and momentum assumptions. This closes the outstanding discrepancy between theory and the method really used in the experiments. We empirically verify our new assumption (σ1(Wt)−σ2(Wt)≥κR>0\sigma_1(W_t)-\sigma_2(W_t)\ge\kappa R>0) during real training. Figure 7 (Section C.3, page 26) shows plots of σ1−σ2\sigma_1 - \sigma_2 and (σ1−σ2)/σ1(\sigma_1 - \sigma_2)/\sigma_1 across different layer types during training. None of the layer types reach a spectral gap of zero, which verifies our assumption in practice.

1.c. A precise distinction between proven results and heuristic approximations would materially affect my assessment.

These results closes the theoretical gap associated with replacing nearest-point projection by the radial normalization used in our experiments. We hope that the clarified inexact-direction interpretation and the new convergence result resolve the concern that this implementation choice is supported only heuristically.

2.a. A key comparison would be MACRO-fro versus MuonH-fro under the same BF16 setting, since both control weight norms.

Thank you for pointing this out. We firstly offer the MuonH results under the same BF16 setting:

Averaged Train loss on last 5% stepsValidation PPL
MuonH-fro3.043520.77
MACRO-fro3.037020.75

Under the same setting, MuonH nearly matches the performance of MACRO. This result aligns with the conclusion from Table 4: under the standard setting, moving from MuonH to SSO yields no significant improvement. These results show that computing the exact projection on the tangent space does not boost training performance.

Similarly, beyond empirical observations, interactions with existing training components require only the weight constraint rather than a specific algorithm. We emphasize that this paper focuses on understanding training dynamics under manifold constraints, not on proposing a SOTA optimizer.

MACRO serves as a testbed to explain the underlying mechanisms of manifold constraints. We designed MACRO to obtain a single-loop optimizer with theoretical convergence guarantees.

2.b. ...measure the actual fraction or magnitude of parameter updates lost to BF16 rounding...

We agree that adding the actual cancellation fraction monitor strengthen our paper. We use MACRO-frob and MuonH-frob as background optimizers under the BF16 master weight setting. Every 20 steps, we convert the weights and updates back to full FP32 precision. We then compare the FP32-updated weights with the BF16-updated weights to identify elements canceled under low precision. Finally, we compute the cancellation ratio and plot the results in Figure 6 on Page 9. From the figue one can show that MuonH nearly matches the cancellation ratio of MACRO, although MACRO achieves both slightly lower cancellation ratio validation PPL. This indicates that cancellation control mainly stems from the weight constraint rather than a specific algorithm. Moreover, the empirical cancellation ratio exhibits the same tendency as pdrop(ρ)p_{\text{drop}}(\rho).

Summery:

In our response, we present three new experiments:

  1. The distance between the MACRO update and the true double-loop update direction (SSO/FSO) (Figure 15, page 48).
  2. The performance comparison between MACRO-frob and MuonH-frob under the BF16 master weight setting.
  3. The empirical update cancellation ratio of MuonH and MACRO during BF16 master weight training (Figure 6, page 9).

From these three experiments, we conclude:

  1. The relative difference between the MACRO update direction and the SSO/FSO update direction is small.MuonH nearly matches the performance of MACRO under BF16 master weight training, which isolates the source of performance gains under BF16.
  2. The Gaussian model aligns well with the empirical cancellation ratio.

In addition, we provide a new theorem (Theorem 2, page 23) along with empirical verification of its spectral gap assumption. These results show that:

  1. MACRO approximates the update direction using a one-step alternating projection to solve the constrained spectral steepest-descent problem.
  2. The new theorem close the gap between theoretical result and the empirical algorithm used in the experiments.

We believe these additional results and theoretical contributions successfully address your concerns.

Reviewer_ydspOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of manifold constraints in LLM pre-training, focusing on their effects on activation scales, weight rotation dynamics, and the update-to-weight ratio. The authors propose MACRO, a single-loop Riemannian optimizer that incorporates tangent-space projection and matrix-sign updates, along with a radius selection strategy and theoretical convergence guarantees. The paper further analyzes the interactions between manifold constraints and existing training mechanisms, including RMSNorm, weight decay, and low-precision computation. Experiments on Qwen3-style models ranging from 240M to 1.2B parameters demonstrate that MACRO achieves competitive performance compared with existing optimizers while improving training stability and robustness under low-precision settings.

Critical strengths
  1. The paper investigates an interesting research problem from a meaningful perspective.

  2. The paper provides a relatively solid theoretical analysis.

Critical weaknesses

1. The proposed method offers limited algorithmic novelty compared with existing approaches, and its motivation is not sufficiently justified.

MACRO is closely related to the existing MuonH optimizer, with the main difference being the introduction of tangent-space projection before the Matrix Sign operation. However, the paper does not sufficiently identify the specific limitations of MuonH in practical optimization or explain why tangent-space projection is necessary to address these limitations.

Although the authors emphasize the lack of rigorous convergence guarantees for MuonH, I believe that the absence of theoretical analysis alone does not provide sufficient motivation for developing a new optimization algorithm. The authors should further clarify the concrete limitations of existing methods and explain how tangent-space projection effectively addresses these limitations in practice.

Questions for authors

1. Contributions of different mechanisms to performance improvements.

The paper analyzes the effects of manifold constraints on LLM training from multiple perspectives. However, it remains unclear to what extent the observed improvements can be attributed to manifold constraints themselves, the specific optimizer design, or other training stabilization mechanisms. Could the authors provide more comprehensive ablation studies and analyses to clarify the individual contributions of these components to training stability and final model performance?

2. Necessity of explicit manifold constraints.

The paper controls weight norms by explicitly constraining model parameters to Frobenius or spectral spheres. However, could similar benefits be achieved through simpler norm-based regularization methods, such as L2 regularization or other soft constraints? It would be valuable for the authors to further explain the necessity of explicit manifold constraints and provide comparisons with conventional regularization approaches to demonstrate their distinctive advantages.

5 comments

Rebuttal Revision summary:

We have uploaded a revised manuscript to address the reviewers' comments. The major updates include:

  1. Abstract and Introduction: We have removed all mentions of "Riemannian steepest-descent" and instead describe it as a "modified Riemannian steepest-descent" algorithm. We changed the introduction of MACRO in Section 3.1 accordingly.

  2. New Convergence Theorem: We limit the scope of Theorem 1 to the Frobenius and Oblique Manifolds. Moreover, we provide a new convergence result for the MACRO-spec in Appendix C.2. This new theorem closes the gap between theory and the algorithm used in practice.

  3. Distance between MACRO and SSO/FSO Updates: We monitor and report the distance between the MACRO update and the true solution of the constrained LMO subproblem in Figure 15.

  4. Real Update Cancellation under BF16: We report the empirical update cancellation ratio during actual training in Figure 6 (Section 4.3).

All changes are marked in blue.

Question part:

Could the authors provide more comprehensive ablation studies and analyses to clarify the individual contributions of these components to training stability and final model performance?

Thank you for your question. We clarify that the stabilization mechanism is an empirical result obtained using MACRO as a testbed, not our original design goal.

Take the interaction with weight decay as an example. First, we explain how weight constraints affect rotational dynamics, which relates to the mechanism of weight decay. Next, we use MACRO as a testbed to evaluate weight decay sensitivity under manifold constraints. Replacing weight decay with manifold constraints is our empirical conclusion, not a design principle of MACRO.

Comparing the empirical results in Table 5, the path from MuonH to SSO/FSO represents a more rigorous solution to the constrained LMO subproblem. However, neither SSO/FSO nor MACRO provides significant gains over MuonH. This indicates that exact tangent projections and better LMO subproblem solvers do not yield practical benefits.

Instead, adding the weight constraint (the transition from Muon to MuonH) provides the largest performance gain. This observation aligns with our analysis, as all key interactions depend on the weight constraint itself rather than a specific algorithm.

  1. However, could similar benefits be achieved through simpler norm-based regularization methods, such as L2 regularization or other soft constraints? It would be valuable for the authors to further explain the necessity of explicit manifold constraints and provide comparisons with conventional regularization approaches to demonstrate their distinctive advantages.

We agree that adding more discussion on soft penalties strengthens our paper.Weight decay was originally introduced as ℓ2\ell_2 regularization. However, [1] showed that decoupling weight decay from the gradient update (as in AdamW) performs better than classical ℓ2\ell_2 penalties. Other soft regularizations, such as ℓ1\ell_1 penalties, focus on sparsity rather than weight control and are rarely used in deep learning.Moreover, [2] proposed soft-control methods in the spectral space, such as Spectral Hammer and Spectral Cap. However, that work lacks evaluation on large-scale language model training. We will add this discussion in the camera-ready version.

[1] Loshchilov et al. Decoupled Weight Decay Regularization, ICLR, 2019

[2] Newhouse et al. Training Transformers with Enforced Lipschitz Constants, arxiv, 2025

Weaknees part:

  1. MACRO is closely related to the existing MuonH optimizer...the paper does not sufficiently identify the specific limitations of MuonH in practical optimization... The authors should further clarify the concrete limitations of existing methods and explain how tangent-space projection effectively addresses these limitations in practice.

Response: Thank you for your comment. We emphasize that we do not claim to propose a state-of-the-art optimizer. Our primary goal is to understand how manifold constraints affect training dynamics. In this context, MACRO serves as a testbed to explain the underlying mechanisms of manifold constraints. We designed MACRO as a single-loop optimizer with theoretical convergence guarantees.

The main contribution of our paper is analyzing the interaction of manifold constraints with trainable RMSNorm, weight decay, and low-precision training. These discussions are novel and do not appear in prior work on manifold optimizers.

At the same time, we acknowledge that MuonH nearly matches the performance of MACRO. This indicates that the primary empirical benefits stem from the weight constraints themselves. We will further discuss this point in our response to Question 1. Analyzing the effects of weight constraints remains our core contribution.

Dear Reviewer ydsp,

Thank you again for your constructive feedback. We have provided detailed responses to your questions and concerns, along with additional clarifications and revisions.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether any concerns remain. We would be happy to provide further clarification.

Thank you for your time and consideration!

Thank you again for your constructive feedback. Since the discussion period closes in less than 24 hours, we wanted to gently follow up to see if our responses and new results have addressed your concerns, or if there is anything else we can clarify before the deadline.

Best,

Authors

Reviewer_Q7uUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies LLM pre-training in which every weight matrix is constrained to a Frobenius sphere, a spectral sphere or an oblique manifold, and asks how the constraint interacts with RMSNorm, weight decay and BF16 master weights. The authors propose MACRO, a single-loop Riemannian optimizer built on Muon. At each step MACRO keeps an EMA of the stochastic gradients, projects it onto the tangent space of the constraint set at the current weights, orthogonalizes the projection with msign\mathrm{msign} and rescales the result to ROt/∥Ot∥MRO_t/\Vert O_t\Vert_{\mathcal M}, where RR is the radius of the constraint. After the step the weights are projected back onto the constraint set, so the ratio of the update to the weight is controlled through the learning rate. For the spectral sphere the experiments use the scaling RW/∥W∥2RW/\Vert W\Vert_2 instead of the exact projection. Theorem 1 gives an O(T−1/4)O(T^{-1/4}) rate for the nuclear norm of the Riemannian gradient, and Proposition 1 chooses the radii rDoutr\sqrt{D_{out}} and rDout/Dinr\sqrt{D_{out}/D_{in}} from a normalized Gaussian initialization.

Critical strengths

∙\bullet I have not seen this kind of study of manifold constraints before. Instead of comparing final losses only, the paper looks at how the constraint interacts with the other parts of training through activation scales, rotation angles and the update-to-weight ratio. I think this is a useful way to look at the recent manifold-constrained variants of Muon.

∙\bullet The paper covers three model sizes up to 1.2B with five seeds in the main table, and the single-loop method is indeed much cheaper than SSO and FSO. The authors also report results that contradict their main claim, such as the joint Muon sweep in Table 11.

∙\bullet I checked Proposition 1 with Theorem 2 and Lemma 2, and the derivation of Proposition 2, and found them correct. The tangent projection also has a real role that the paper could show. For W∝QW\propto Q with QQ orthogonal and ∇L=aW+εQK\nabla L=aW+\varepsilon QK with KK symmetric, traceless and a>ε∥K∥2a>\varepsilon\Vert K\Vert_2, one has msign(∇L)=Q\mathrm{msign}(\nabla L)=Q. MuonH orthogonalizes the momentum without projecting it, so its step is purely radial and the retraction returns WW unchanged, although the Riemannian gradient εQK\varepsilon QK is not zero and WW is not a stationary point. MACRO first projects onto the tangent space, which removes the radial part aWaW, so its direction msign(εQK)\mathrm{msign}(\varepsilon QK) is not zero and the iterate moves. An example of this kind in the paper would show where the projection matters.

Critical weaknesses

W1. The analysis is close to prior work, and several of its facts are fairly direct

A similar analysis, from a different point of view, is given in [1]. That paper studies how weight decay drives the norm and the angular update of each neuron to an equilibrium and introduces rotational variants of standard optimizers, which keep the weight norm fixed, set the angular update directly through the learning rate and use no weight decay. They have no transient phase by design, match the baselines with weight decay and need less learning-rate warmup. These are close to the claims of Section 4.2 and to the warmup result in Table 12. The paper cites [1] only for the equilibrium itself.

Several facts in Sections 4.1 and 4.2 also follow directly from the construction. Every retraction in the paper is invariant to positive scaling, so PM((1−ηλ)Wt−η∇~t)=PM(Wt−η1−ηλ∇~t)P_{\mathcal M}((1-\eta\lambda)W_t-\eta\tilde\nabla_t)=P_{\mathcal M}(W_t-\tfrac{\eta}{1-\eta\lambda}\tilde\nabla_t), and weight decay only rescales the learning rate by at most 1.0031.003 in Table 10. On the Frobenius sphere, θt≈ηt\theta_t\approx\eta_t is the geometry of a step of size ηt∥Wt∥F\eta_t\Vert W_t\Vert_F that is nearly orthogonal to WtW_t, and it holds for MuonH and FSO as well. The inverse relation between RR and ∥γ∥\Vert\gamma\Vert in Figure 1 is the scale symmetry (cW,γ/c)(cW,\gamma/c) of a gain followed by a linear layer.

I think the paper would gain a lot from additional ablations and more detailed experiments, which it currently lacks. They would separate what is specific to MACRO from what any normalized update would do. I list the ones I consider most important in Q1.

W2. Theorem 1 contains inaccuracies and does not cover the spectral sphere

The statement and the proof of Theorem 1 contain several inaccuracies that need to be fixed. For example, inequality (16) is written with the Euclidean smoothness constant LL, but it does not hold with this constant. Since the manifold is compact, the gradient is bounded on it and the correct constant is finite, but it is not LL.

Also, the theorem does not cover the spectral sphere, because the spectral sphere does not satisfy Assumption 1. It is not smooth where the largest singular value is multiple. The restricted set with σ1≥σ2+ϵ\sigma_1\ge\sigma_2+\epsilon proposed in Appendix C.1 has a boundary, and nothing keeps the iterates from crossing it. If I understand correctly, the main text does not state this explicitly. Theorem 1 is stated for Algorithm 1 in general, and Section 5 says that MACRO "retains the convergence guarantee established in Section 3". Only Appendix C.1 restricts the proof to the Frobenius and oblique cases.

W3. Lack of discussions about msign and tangent space

The direction Ot=msign(ΠTMt)O_t=\mathrm{msign}(\Pi_T M_t), where ΠT\Pi_T is the projection onto the tangent space, is in general not tangent, so PMP_{\mathcal M} is applied to a non-tangent vector and the step is not the steepest descent within the tangent space that Section 3.1 describes. The natural problem is the LMO with an extra constraint that keeps the direction in the tangent space, max⁡∥Φ∥2≤1,Φ∈TWtM⟨Mt,Φ⟩\max_{\Vert\Phi\Vert_2\le1,\Phi\in T_{W_t}\mathcal M}\langle M_t,\Phi\rangle. The paper should state this problem explicitly. It would be interesting to see, for each manifold, how far the direction of MACRO is from its true solution, as is done for the fixed-rank manifold in [2].

[1] A. Kosson, B. Messmer and M. Jaggi. Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks. ICML 2024.

[2] V. Bogachev, V. Aletov, A. Molozhavenko, D. Bobkov, V. Soboleva, A. Alanov and M. Rakhuba. LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters. ICLR 2026.

Questions for authors

Q1. Ablations for the comparison with Muon. To support the claim that MACRO outperforms Muon, I would like to see the following.

(1) While looking through the code, I noticed that some hyperparameters differ between the methods. For example, MACRO orthogonalizes with 10 PolarExpress iterations, while Muon uses only 5 Newton-Schulz iterations. Could the authors describe the experimental protocol in more detail? If the comparisons were run with the configurations from the code, could the authors provide experiments in which all methods share the same hyperparameters?

(2) An AdamW baseline and a comparison at equal wall-clock time, since MACRO-spec takes 12.9% longer per step than Muon.

Q2. Does Theorem 1 apply to MACRO-spec as it is run? If so, under which assumption on the gap σ1−σ2\sigma_1-\sigma_2 along the trajectory, and with the scaling RW/∥W∥2RW/\Vert W\Vert_2 instead of PMP_{\mathcal M}? Could the authors plot this gap during training?

Q3. Based on these results, how should one choose the constraint and its radius in practice? Appendix D.5 and Table 4 favor the Frobenius sphere at the tested scales, while in the architecture without learnable RMSNorm MACRO-spec is the best (Table 2). Do the authors expect the Frobenius sphere to remain the better choice at larger scale or in other architectures, and are there settings, such as low precision or the absence of normalization layers, in which they would recommend the spectral or oblique constraint?

If all of my main concerns are addressed, I will be glad to give the paper a positive score.

8 comments

Rebuttal Revision summary:

We have uploaded a revised manuscript to address the reviewers' comments. The major updates include:

  1. Abstract and Introduction: We have removed all mentions of "Riemannian steepest-descent" and instead describe it as a "modified Riemannian steepest-descent" algorithm. We changed the introduction of MACRO in Section 3.1 accordingly.

  2. New Convergence Theorem: We limit the scope of Theorem 1 to the Frobenius and Oblique Manifolds. Moreover, we provide a new convergence result for the MACRO-spec in Appendix C.2. This new theorem closes the gap between theory and the algorithm used in practice.

  3. Distance between MACRO and SSO/FSO Updates: We monitor and report the distance between the MACRO update and the true solution of the constrained LMO subproblem in Figure 15.

  4. Real Update Cancellation under BF16: We report the empirical update cancellation ratio during actual training in Figure 6 (Section 4.3).

All changes are marked in blue.

Weakness part:

Weaknees 3: The paper should state this problem explicitly. It would be interesting to see, for each manifold, how far the direction of MACRO is from its true solution, as is done for the fixed-rank manifold in [2].

[2] consider the Remannian optimization framework for applying semi-orthogonal constraints in fine-tuning. We considered the different constraint in pretraining. In our setting, the true solution to the LMO with an extra constraint can be computed as the update direction in SSO/FSO. As you noted in Weakness 3, our MACRO algorithm approximates the this true solusion. We address this concern from two perspectives:

  1. First, the convergence guarantee(Theorem 2, Page 23) close the gap between theoretical result and the empirical algorithm used in the experiments.

  2. We also test the relative distance between MACRO and the true solution during training. We use the SSO and FSO updates as the true solution, and we compute the relative error between the MACRO update and the SSO/FSO update as follows:

∥UMACRO−USSO/FSO∥F/∥UMACRO∥F\Vert \text{U}_{\text{MACRO}} - \text{U}_{\text{SSO/FSO}} \Vert_F / \Vert \text{U}_{\text{MACRO}} \Vert_F

We summarize the mean values during training in the table below. Figure 15 (Appendix D.4, page 48) shows the relative error curve during training.

MatrixMACRO-frob MeanMACRO-spec Mean
FFN Up4.94%5.14%
FFN Down3.48%3.33%
FFN Gate4.45%4.96%
Attention Value7.96%6.32%
Attention Query9.26%7.01%
Attention Key9.98%7.73%

The table shows that the relative difference between the MACRO update direction and the SSO update direction is less than 10% for both variants. This result confirms that MACRO provides a good approximation of the true solution for the constrained LMO problem.

Question part:

1.a. Could the authors describe the experimental protocol in more detail? ..., could the authors provide experiments in which all methods share the same hyperparameters?

Response:

Thank you for pointing out this setting difference. We use the PyTorch implementation of Muon which uses the 5-step Newton-Schultz method as the default method for the msign operator. For a fair comparison, we add Muon with the 10-step PolarExpress method for the 240M model with 1 seed.

Muon (PE 10 steps)Muon (NS 5 steps)MACRO-froMACRO-spec
Val PPL20.6120.6420.4520.53

While Muon with 10-step PE performs better than Muon with 5-step NS, a performance gap remains between Muon and the MACRO variants which shows that the empirical benefits do not entirely come from the msign matrix algorithm.

1.b. An AdamW baseline and a comparison at equal wall-clock time...

Response: We agree that adding the AdamW baseline strengthens our paper. We train a 240M Qwen3 model with AdamW using 3 random seeds. All experimental settings are the same as in Table 4. The learning rate grid is {0.002, 0.004, 0.008, 0.016, 0.032}. The result table is shown below:

AdamW (lr*=8e-3)MuonMACRO-froMACRO-spec
Val PPL21.0926 ± 0.00820.72 ± 0.01020.50 ± 0.01120.58 ± 0.011

The table shows that AdamW performs worse than the Muon and MACRO variants under the same token budget. We use token efficiency to compare different optimizers. This is important for LLM training because the amount of training data can become a bottleneck which is commonly used in the optimizer comparison[1][2].

[1] Muon is Scalable for LLM Training, 2025

[2] modded-nanogpt, https://github.com/KellerJordan/modded-nanogpt.git

  1. Does Theorem 1 apply to MACRO-spec as it is run? ... Could the authors plot this gap during training?

Response:

The original proof of Theorem 1 addresses nearest-point projection and did not directly justify the radial normalization RW/∥W∥2RW/\|W\|_2 used in MACRO-spec. We have clarified this scope and added Theorem 2 in Appendix C.2, Page 23 with the blue text, which analyzes the radial update itself. This closes the outstanding discrepancy between theory and the method really used in the experiments. The additional geometric assumption is that there exists a deterministic constant κ>0\kappa>0, independent of the training horizon, such that σ1(Wt)−σ2(Wt)≥κR\sigma_1(W_t)-\sigma_2(W_t)\ge\kappa R at every iteration almost surely. Pointwise nonzero gaps alone do not provide uniform convergence constants. Specifically, with Ot=msign⁡(PTWtMt)O_t=\operatorname{msign}(P_{T_{W_t}}M_t), the new theorem covers Wt+1=R(Wt−hOt)/∥Wt−hOt∥2W_{t+1}=R(W_t-hO_t)/\Vert W_t-hO_t\Vert_2, where h=ηR/(1+ϵ)h=\eta R/(1+\epsilon). Under the original loss-smoothness and stochastic-gradient assumptions, interpreted conditionally on the training history, choosing the proper hyperparameters yields min⁡0≤t<TE∥PTWt∇L(Wt)∥∗=O(T−1/4)\min_{0\le t<T}\mathbb{E}\Vert P_{T_{W_t}}\nabla\mathcal{L}(W_t)\Vert_*=\mathcal{O}(T^{-1/4}). This result analyzes the mathematical radial update with exact primitives and requires neither nearest-point projection nor tangency of OtO_t. The revised draft contains the explicit theorem and full proof. We empirically verify our new assumption during real training. Figure 7 (Section C.3, page 26) shows plots of spectral gap across different layer types during training. None of the layer types reach a spectral gap of zero, which verifies our assumption in practice.

  1. how should one choose the constraint and its radius in practice?... in which they would recommend the spectral or oblique constraint?

Thank you for bringing up this question. First, Table 2 and Table 4 are not directly comparable because they use different settings. The experiments in Table 2 train models on OpenWebText to ablate the interaction between trainable RMSNorm and manifold constraints. We recommend using Table 4 as the standard experimental setting.

Under this standard setting, the Spectral Sphere does not outperform the Frobenius Sphere, and it adds higher computational cost. Therefore, we recommend using the Frobenius Sphere for standard setups.

For low-precision training scenarios, we also recommend the Frobenius Sphere. It provides more direct control over the update-to-weight ratio, which makes update cancellation predictable.

For settings without trainable RMSNorm, we recommend the Spectral Sphere. It provides better output scale control than the Frobenius Sphere, which stabilizes training. Table 2 confirms that the Spectral Sphere achieves lower validation loss in this setup.

Finally, Figure 14( Page 47) compares the Frobenius Sphere and the Oblique Manifold. The results show that the Frobenius Sphere achieves lower validation loss while maintaining similar training dynamics. So in our paper, oblique manifold is not a better constraint comparing with Frobenius Sphere.

Summary

In our response, we present three new experiments:

  1. The result of train 240M model by Muon with 10-step PolarExpress.
  2. The result of train 240M model using AdamW with 3 random seeds.
  3. The distance between the MACRO update and the true double-loop update direction (SSO/FSO) (Figure 15, page 48).

From these three experiments, we conclude:

  1. The relative difference between the MACRO update direction and the SSO/FSO update direction is small.MuonH nearly matches the performance of MACRO under BF16 master weight training, which isolates the source of performance gains under BF16.
  2. Neither using 10-step PolarExpress in Muon nor AdamW will close the gap between Muon and MACRO.

In addition, we provide a new theorem (Theorem 2, page 23) along with empirical verification of its spectral gap assumption(Fig 7, page 26). These results show that:

  1. MACRO approximates the update direction using a one-step alternating projection to solve the constrained spectral steepest-descent problem.
  2. The new theorem close the gap between theoretical result and the empirical algorithm used in the experiments.

We believe these additional results and theoretical contributions successfully address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks and Summary of the Discussion Period

Dear SAC, AC, and Reviewers,

Firstly, we sincerely appreciate the constructive feedback from the reviewers as well as the time and effort you have invested in our submission. As the discussion phase draws to a close, we would like to provide a summary of our rebuttal and discussion. In the following summary, we use S for Strength, W for Weakness, and Q for Question:

Strengths:

We are very pleased that the reviewers (Q7uU, ydsp, A8Wg) acknowledged that our paper:

  • Provides a novel and useful way to investigate manifold constraints in LLM pre-training (Q7uU S.1, ydsp S.1, A8Wg S.1).
  • Presents comprehensive empirical results demonstrating the capability and potential of our approach in LLM pre-training (Q7uU S.2, A8Wg S.2).
  • Establishes solid theoretical results (Q7uU S.3, ydsp S.2).

During the rebuttal phase, we provided further empirical and theoretical results by:

  • Adding a new Convergence Theorem for MACRO-spec, which entirely closes the gap between theory and the algorithm used in practice, along with an empirical verification of the σ\sigma-gap. We believe this new theorem addresses the concerns in Q7uU W.2, Q.2; A8Wg W.1, Q.1.

  • Monitoring the error between the MACRO update direction and the exact solution of the constrained LMO subproblem. We believe this empirical analysis addresses the concerns in Q7uU W.3, Q.3; A8Wg W.1.

  • Providing additional empirical results on the cancellation ratio of MuonH and MACRO during BF16 master-weight training, which addresses the concerns in A8Wg W.2, Q.2; ydsp W.1.

  • Adding further ablation studies on performance and wall-clock time comparisons against PE-10, Muon, and AdamW to address Q7uU W.1, Q.1.

  • Expanding the discussion on how to select constraints (Q7uU Q.3) and on other types of soft constraints (ydsp Q.2).

Our primary goal is to understand how manifold constraints interact with existing components of LLM pre-training rather than merely developing a SOTA optimizer to outperform MuonH. We believe this clarification directly addresses ydsp W.1, Q.1.

During the discussion phase, we were glad to receive positive recognition from Reviewers Q7uU and ydsp. Although Reviewer A8Wg has not yet posted a follow-up response, we believe our additional results and clarifications thoroughly address Reviewer A8Wg's concerns.

We sincerely appreciate the time and effort you have invested in our submission and have full confidence in a fair and rigorous decision process.

Sincerely,

Authors of submission 26362

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.