- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how explicit constraints on weight matrices affect LLM pre-training. It proposes MACRO, an optimizer that combines tangent-space projection, matrix-sign updates, and weight normalization. The authors also derive rules for choosing constraint radii and investigate how weight constraints interact with RMSNorm, weight decay, and BF16 training.
Experiments on Qwen3-like models ranging from 240M to 1.2B parameters show that MACRO improves validation perplexity over the original Muon baseline, with a relatively small computational overhead for the Frobenius variant. The paper also reports substantially smaller performance degradation under BF16 parameter training and provides theoretical analyses of weight rotation, relative update sizes, and convergence.
Critical strengths
-
A useful analysis of weight constraints from multiple perspectives. The paper examines weight constraints through activation scales, weight rotation, and relative update sizes rather than relying only on final validation loss. This provides a useful perspective on how weight normalization may affect training dynamics, especially under low-precision computation.
-
A practical optimizer with encouraging empirical results. MACRO uses a single-loop update and avoids the expensive inner optimization required by methods such as SSO. The Frobenius variant achieves competitive validation performance with only a small increase in per-step cost over Muon. The BF16 results are also encouraging and motivate further investigation.
Critical weaknesses
1.The claimed Riemannian steepest-descent interpretation and convergence guarantees are not fully justified for the implemented algorithm.
Section 3.1 describes MACRO as a Riemannian spectral steepest-descent method. However, Algorithm 1 first projects the momentum onto the tangent space and then applies the matrix-sign operation. The latter does not generally preserve tangency: even if , it does not follow that . Therefore, the resulting direction is not necessarily the solution to the steepest-descent problem restricted to feasible tangent directions. Figure 12 also reports nonzero tangent-space violations.
The convergence analysis in Appendix C.1 explicitly covers Frobenius and oblique constraints. For the spectral constraint, the authors acknowledge that the spectral sphere is not smooth everywhere, while the implementation uses an approximate projection rather than the exact one. The argument based on almost-sure uniqueness of the largest singular value does not clearly establish the conditions needed to extend the convergence guarantee to the implemented spectral variant.
2.The experiments do not sufficiently distinguish the benefits of weight constraints from those of MACRO's specific update rule, and the proposed mechanisms are not fully established.
The results support the usefulness of controlling weight norms, but they provide weaker evidence that MACRO's specific update rule is necessary. MuonH already enforces weight constraints and achieves performance comparable to, or sometimes better than, MACRO (Table 4). Moreover, the separately tuned Muon baseline in Table 11 nearly matches MACRO-fro in that comparison.
This distinction is particularly important for the BF16 results, which are among the paper's strongest findings. The main precision comparison includes Muon but not the norm-constrained MuonH baseline. In addition, switching to BF16 changes the storage precision of parameters, gradients, and optimizer states together. The proposed update-cancellation explanation relies mainly on a simplified Gaussian model, without directly establishing how much of the observed performance gap is caused by canceled parameter updates. As a result, the experiments do not clearly separate the effects of controlling weight norms, the specific MACRO update, and other low-precision effects.
Questions for authors
-
What exactly is the constrained optimization problem solved by MACRO's update? Since the matrix-sign operation does not necessarily preserve tangency, can the authors clarify in what sense the resulting direction is a Riemannian spectral steepest-descent direction? In particular, what convergence guarantee applies to the practical spectral implementation using approximate projection? A precise distinction between proven results and heuristic approximations would materially affect my assessment.
-
Can the authors isolate the source of the BF16 improvement? A key comparison would be MACRO-fro versus MuonH-fro under the same BF16 setting, since both control weight norms. It would also be useful to measure the actual fraction or magnitude of parameter updates lost to BF16 rounding, rather than relying only on a Gaussian estimate. These results could establish whether the improvement is due primarily to norm control, MACRO's specific update, or the proposed update-cancellation mechanism.
8 comments
Questions part:
1.a. What exactly is the constrained optimization problem solved by MACRO's update?
The local constrained direction problem is the spectral steepest-descent subproblem used by SSO: , where is the momentum, is the tangent space, and is subtracted in the parameter update. MACRO approximates this direction construction through one alternating pass of tangent projection and the spectral oracle: it first projects the momentum onto the tangent space and then applies the matrix-sign operation, without an iterative inner solve to convergence.
- b. ...can the authors clarify in what sense the resulting direction is a Riemannian spectral steepest-descent direction? ..what convergence guarantee applies to ... using approximate projection?
The intermediate matrix-sign output need not remain tangent; we therefore describe MACRO as an inexact realization of Riemannian spectral steepest descent, with convergence established for the complete update after feasibility restoration. This interpretation can be quantified for the spectral constraint: under a uniform relative gap (which will be empirically verified later), the first-order tangent direction induced by radial normalization, rescaled into the unit spectral ball, achieves at least a fraction of the optimal linear improvement in the constrained subproblem. Thus, an exact tangent-subproblem solution is unnecessary for the convergence guarantee. Specifically, the new Theorem 2 in Appendix C.2 directly analyzes the radial normalization used by MACRO-spec and establishes an expected first-order stationarity rate, with gap-dependent constants, under the stated smoothness, stochastic-gradient, stepsize, and momentum assumptions. This closes the outstanding discrepancy between theory and the method really used in the experiments. We empirically verify our new assumption () during real training. Figure 7 (Section C.3, page 26) shows plots of and across different layer types during training. None of the layer types reach a spectral gap of zero, which verifies our assumption in practice.
1.c. A precise distinction between proven results and heuristic approximations would materially affect my assessment.
These results closes the theoretical gap associated with replacing nearest-point projection by the radial normalization used in our experiments. We hope that the clarified inexact-direction interpretation and the new convergence result resolve the concern that this implementation choice is supported only heuristically.
2.a. A key comparison would be MACRO-fro versus MuonH-fro under the same BF16 setting, since both control weight norms.
Thank you for pointing this out. We firstly offer the MuonH results under the same BF16 setting:
| Averaged Train loss on last 5% steps | Validation PPL | |
|---|---|---|
| MuonH-fro | 3.0435 | 20.77 |
| MACRO-fro | 3.0370 | 20.75 |
Under the same setting, MuonH nearly matches the performance of MACRO. This result aligns with the conclusion from Table 4: under the standard setting, moving from MuonH to SSO yields no significant improvement. These results show that computing the exact projection on the tangent space does not boost training performance.
Similarly, beyond empirical observations, interactions with existing training components require only the weight constraint rather than a specific algorithm. We emphasize that this paper focuses on understanding training dynamics under manifold constraints, not on proposing a SOTA optimizer.
MACRO serves as a testbed to explain the underlying mechanisms of manifold constraints. We designed MACRO to obtain a single-loop optimizer with theoretical convergence guarantees.
2.b. ...measure the actual fraction or magnitude of parameter updates lost to BF16 rounding...
We agree that adding the actual cancellation fraction monitor strengthen our paper. We use MACRO-frob and MuonH-frob as background optimizers under the BF16 master weight setting. Every 20 steps, we convert the weights and updates back to full FP32 precision. We then compare the FP32-updated weights with the BF16-updated weights to identify elements canceled under low precision. Finally, we compute the cancellation ratio and plot the results in Figure 6 on Page 9. From the figue one can show that MuonH nearly matches the cancellation ratio of MACRO, although MACRO achieves both slightly lower cancellation ratio validation PPL. This indicates that cancellation control mainly stems from the weight constraint rather than a specific algorithm. Moreover, the empirical cancellation ratio exhibits the same tendency as .
Summery:
In our response, we present three new experiments:
- The distance between the MACRO update and the true double-loop update direction (SSO/FSO) (Figure 15, page 48).
- The performance comparison between MACRO-frob and MuonH-frob under the BF16 master weight setting.
- The empirical update cancellation ratio of MuonH and MACRO during BF16 master weight training (Figure 6, page 9).
From these three experiments, we conclude:
- The relative difference between the MACRO update direction and the SSO/FSO update direction is small.MuonH nearly matches the performance of MACRO under BF16 master weight training, which isolates the source of performance gains under BF16.
- The Gaussian model aligns well with the empirical cancellation ratio.
In addition, we provide a new theorem (Theorem 2, page 23) along with empirical verification of its spectral gap assumption. These results show that:
- MACRO approximates the update direction using a one-step alternating projection to solve the constrained spectral steepest-descent problem.
- The new theorem close the gap between theoretical result and the empirical algorithm used in the experiments.
We believe these additional results and theoretical contributions successfully address your concerns.
Rebuttal Paper Revision summary:
We have uploaded a revised manuscript to address the reviewers' comments. The major updates include:
Abstract and Introduction: We have removed all mentions of "Riemannian steepest-descent" and instead describe it as a "modified Riemannian steepest-descent" algorithm. We changed the introduction of MACRO in Section 3.1 accordingly.
New Convergence Theorem: We limit the scope of Theorem 1 to the Frobenius and Oblique Manifolds. Moreover, we provide a new convergence result for the MACRO-spec in Appendix C.2. This new theorem closes the gap between theory and the algorithm used in practice.
Distance between MACRO and SSO/FSO Updates: We monitor and report the distance between the MACRO update and the true solution of the constrained LMO subproblem in Figure 15.
Real Update Cancellation under BF16: We report the empirical update cancellation ratio during actual training in Figure 6 (Section 4.3).
All changes are marked in blue.
Weakness part:
Thank you for raising the fact that MACRO is not a "Riemannian steepest-descent" algorithm. It would have been better to have described it as a "modified Riemannian steepest-descent" algorithm. Consequently, we have removed all mention of "steepest-descent" in our updated manuscript. Moreover, we want to further address this concern from two perspectives:
First, the convergence guarantee still holds even when we use this approximate update(please check the details in the answer of question 1).
We also test the relative distance between MACRO and the true solution during training as your reference. We use the SSO and FSO updates as the true solution, and we compute the relative error between the MACRO update and the SSO/FSO update as follows:
∥UMACRO−USSO/FSO∥F/∥UMACRO∥F
We summarize the mean values during training in the table below. Figure 15 (Appendix D.4, page 48) shows the relative error curve during training.
The table shows that the relative difference between the MACRO update direction and the SSO update direction is less than 10% for both variants. We hope that this supplementary materials may address your concern about the gap between MACRO and "true" solution of the steepest-descent problem restricted to feasible tangent directions.
First, we want to further explain Table 12 in the paper. We acknowledge that after joint tuning of the learning rate and weight decay, the Muon baseline nearly matches MACRO-frob. However, this comparison is not completely fair because MACRO only requires learning rate tuning. For example, in Table 12, Muon requires 12 tuning runs to match the performance of MACRO with only 3 runs. Furthermore, Table 12 shows that the optimal weight decay for Muon depends on the selected learning rate. This confirms that two-dimensional grid search is necessary for Muon. These results verify the benefits discussed in Section 4.2: adding manifold constraints removes the need for weight decay, making hyperparameter tuning much easier in practical training.