- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a Normalize-Then-Precondition framework that combines marginal-scale normalization with directional spectral preconditioning for matrix optimizers. Building on this framework, the authors introduce NormPre, which includes NormPre-G for global spectral preconditioning and NormPre-L for localized spectral refinement. Empirically, NormPre consistently outperforms AdamW, Muon, and MANO across various LLM pretraining settings. The authors also establish convergence guarantees for both variants.
Critical strengths
-
The method is lightweight and easy to implement, requiring only minor modifications to the original Muon.
-
The experiments covers multiple model families such as GPT-2, LLaMA and Qwen3, and multiple datasets including C4 and Pile.
Critical weaknesses
-
Motivation seems not sufficiently clear. This work primarily addresses directional interactions and scale imbalance, but the value of solving this problem is not clearly established. In particular, the paper does not explain, from a design-motivation perspective, why addressing this issue is beneficial for training. The row/column perspective is essentially formal and is not directly linked to training efficiency. Moreover, scale can be oriented in arbitrary directions and is not limited to rows or columns. The paper does not justify why considering row- or column-wise scaling is necessary. For example, Muon can be viewed as scale normalization within the eigenspace and already achieves this goal. Furthermore, in the extreme case where row (or column) norms are highly imbalanced, although the leading Gram eigenvector approaches the coordinate axis of the largest row, this may not prevent the optimizer from capturing correlations among the remaining rows.
-
The experimental evaluation is insufficient.
- All main experiments use 10,000 steps, corresponding to only about 5B tokens. For the larger models in experiments, the data is too little to reflect realistic training scenarios. It is suggested to train for at least 1$\times$ Chinchilla-optimal data (roughly 20$\times$ the model parameters). For LLaMA-1.3B, this is about 25B tokens, and about 30B tokens for Qwen3-1.7B. The current budget is only about 0.2× or less of this amount.
- Model scale is limited to 1.7B, with no validation on larger models.
- The set of baselines is also relatively small. It is recommended to compare with recent Muon variants using similar row/col scaling such as NorMuon [1].
- MoE models are not evaluated.
- Based on the experimental description, the baseline learning rates do not appear to have been sufficiently searched or tuned, especially for the larger models. Thus the observed advantages may not reflect the true performance of the baselines under fair and well-tuned settings.
[1] NorMuon: Making Muon more efficient and scalable. ICML 2026.
Questions for authors
-
The gradient-norm bound in Theorem 1 is standard for non-convex optimization and no stronger than SGD. My question concerns the significance and value of this theoretical result. What does it actually establish, and what does it add beyond a standard convergence guarantee? I acknowledge the technical effort involved in the derivation. However, the rate alone cannot explain why NormPre outperforms Muon, which makes it weak support for the paper’s central claims. Although end-to-end analysis of LLM training is difficult, it is encouraged to consider a more realistic model (even a toy model) to analyze the advantages brought by NormPre. Such intuitive theoretical understanding would help readers understand the role of the proposed algorithm more than a single non-convex rate.
-
On line 73, the authors present an illustrative comparison between 1e4 and 1e-3. Would such a large disparity occur in realistic LLM training scenarios?
-
What is the motivation behind momentum projection? Why is the relaxed tangent projection appropriate, and better than both no projection and strict tangent projection? Operations such as msign may undermine this calibration of radial scale information.
-
Table 6b shows that row normalization only performs better than alternating row/column for both NormPre-G and NormPre-L. This appears to suggest that the alternating row/column design is redundant and even harmful. In addition, what would happen if only columns are normalized?
5 comments
- We agree that Muon can still capture correlations among the remaining rows even when the leading Gram eigenvector approaches the largest row’s coordinate axis. This is consistent with our example in lines 74–78, which explicitly considers strongly correlated remaining rows and unchanged directional interactions. The point of the example is that marginal scales can strongly shape the principal mode of the full Gram matrix even when directional relationships remain unchanged. This provides supplementary intuition for exploring an alternative organization: first construct a normalized reference that removes row-magnitude weighting, then refine its directional interactions through a spectral transformation. The practical benefit of this construction is assessed through the ablations below.
3. Existing and newly added ablations connect the design to training performance.
The following validation losses are obtained on GPT-2 Small/OpenWebText with seed 1337 and a 10k-step training budget. We present the component comparisons for NormPre-G and NormPre-L separately. The NormPre-G variants use NS5, while the NormPre-L variants use localized spectral preconditioning with randomized sketching. Muon is included in both tables as a baseline for reference.
NormPre-G
| Configuration | Relaxed tangent | Marginal normalization | Gram matrix | Validation loss |
|---|---|---|---|---|
| w/o tangent processing | ✗ | ✓ | 3.0945 | |
| w/o marginal normalization | ✓ | ✗ | 3.0893 | |
| Geometry from | ✓ | ✓ | 3.0900 | |
| NormPre-G | ✓ | ✓ | 3.0711 | |
| Muon | ✗ | ✗ | 3.1137 |
NormPre-L
| Configuration | Relaxed tangent | Marginal normalization | Gram matrix | Validation loss |
|---|---|---|---|---|
| w/o tangent processing | ✗ | ✓ | 3.1139 | |
| w/o marginal normalization | ✓ | ✗ | 3.1565 | |
| Geometry from | ✓ | ✓ | 3.1305 | |
| NormPre-L | ✓ | ✓ | 3.0864 | |
| Muon | ✗ | ✗ | 3.1137 |
Here, denotes the momentum signal, the update before marginal normalization in each configuration, and its normalized counterpart; when tangent processing is omitted. The notation is written in the active row orientation. The Gram matrix column identifies the geometry used to construct the spectral preconditioner. The “Geometry from ” variants retain as the base update but construct their preconditioners from .
-
Rows 1 and 3 in each table: ablations already reported in the original manuscript. Tables 6(a) and 7 in Appendix A.4 report the momentum-processing and interaction-geometry-source ablations, respectively, together with the results of 3.0711 for NormPre-G and 3.0864 for NormPre-L. Removing relaxed tangent processing increases validation loss for both variants. Retaining the normalized base update but constructing its preconditioner from also increases loss, supporting the use of normalized interaction geometry for spectral refinement.
-
Row 2 in each table: new ablations added during the rebuttal period. These controls bypass input marginal normalization while retaining relaxed tangent processing, the respective spectral routines and their parameters, the learning-rate schedules, and the original final RMS reference/scaling rules. For NormPre-G, the same five-step Newton–Schulz routine is applied to instead of , with its internal numerical normalization retained. For NormPre-L, the same sketch configuration (, , ) and spectral threshold are retained. These controls provide empirical support for the contribution of input marginal normalization in both variants under the tested setting.
-
Row 5 in each table: a result from an existing Muon run. For completeness, we report its validation loss of 3.1137 with seed 1337.
For NormPre-L, Table 8 and Figure 4(c) additionally connect the spectral budget to the measured trade-off between validation loss and optimizer latency, providing a practical link between localized spectral refinement and computational efficiency.
In summary, we emphasize that our starting point is the full-Gram and diagonal-Gram representations of orthogonalization and normalization, together with the observation that normalization already provides a useful update. These observations motivate an insight into organizing geometric information hierarchically, followed by a central question: how much spectral modification should an already useful normalized update undergo? Together, this insight and question guide our systematic framework and its two spectral constructions. The example in lines 74–78 provides supplementary geometric intuition about how marginal scales might affect the full-Gram geometry, while the ablations above provide empirical support for the resulting design choices. We have revised the Introduction to make this progression clearer.
Q2: The experimental evaluation is insufficient.
A: Thank you for these suggestions. Our current evaluation covers multiple architectures, model sizes, and datasets: GPT-2 Small on OpenWebText, LLaMA-{130M, 350M, 1.3B} on C4, and Qwen3-{0.6B, 1.7B} on Pile. We focus on dense language models and compare against AdamW, Muon, and MANO under matched training budgets within each setting.
Regarding training duration, the reported token budgets for GPT-2 Small and LLaMA-130M already exceed the commonly used reference. We have also supplemented the evaluation with learning-rate sweeps on GPT-2 Small, as detailed below.
1. Learning-rate choices in the main experiments. Different optimizers may have different optimal learning rates. Our main experiments use predefined learning-rate schedules together with established update-scaling conventions, including RMS matching to bring matrix-update magnitudes into a range comparable to AdamW. The optimizer-specific configurations are detailed in Appendix A.1. This provides a practical basis for comparing optimization behavior without separate optimizer-specific learning-rate tuning. We further complement this comparison with the additional search below, allowing each optimizer to select its best learning rate within the tested grid.
2. Additional learning-rate searches confirm that the advantages persist within the tested range. We evaluated all five optimizers on GPT-2 Small/OpenWebText at times their original learning-rate configurations, using seed 1337 and a 10k-step training budget. All other training settings remain unchanged. Among these tested learning rates, we select the one yielding the lowest final validation loss for each optimizer. The corresponding learning rates and losses are reported below.
| Optimizer | Selected base peak LR | Final validation loss |
|---|---|---|
| AdamW | 3.1001 | |
| Muon | 3.1076 | |
| MANO | 3.1057 | |
| NormPre-G | 3.0482 | |
| NormPre-L | 3.0720 |
After each optimizer selects its best tested learning rate, the strongest baseline is AdamW at 3.1001. NormPre-G and NormPre-L reduce validation loss by approximately 0.0518 and 0.0281, respectively. Thus, the performance advantages persist after giving every optimizer the same number of learning-rate candidates and allowing each to select its own best configuration within the tested grid.
Given the available time and computational resources, extending larger-model training to Chinchilla-style token budgets, evaluating MoE architectures, expanding comparisons with recent optimizers, and conducting more extensive learning-rate searches on larger models remain directions for future work. We appreciate these suggestions and will make every effort to add further experiments as time and computational resources allow, while clarifying the scope of our evaluation and the remaining directions for future work in the revised manuscript.
Thank you for your suggestions. We appreciate your recognition of the lightweight implementation and the evaluation across multiple model architectures and datasets. In response, we have revised the Introduction to clarify our hierarchical insight and central design question, clarified the roles of the optimization formulations and convergence analysis, and added some ablation experiments. Below, we also explain the rationale and supporting evidence for relaxed tangent processing and alternating normalization, and discuss opportunities to broaden the evaluation. We address each concern in turn.
A: Thank you for these questions. We would like to clarify the two central questions behind our framework, how the construction differs from Muon's full-Gram transformation, and the experimental evidence supporting our design choices.
1. Two core questions in the Introduction with distinct roles.
Muon's full-Gram representation reveals its joint treatment of marginal scales and cross-row interactions, while normalization draws solely on the Gram diagonal to equalize row norms without altering pairwise angles. Together with the observation that normalization alone already provides a useful update, this suggests organizing geometric information hierarchically: first use marginal scales to construct a normalized update, then use its remaining directional correlations for refinement. This motivates our first question: “Can we construct a normalized update from scales and refine it via directional interactions?” Muon already processes both types of information through its full-Gram transformation; our aim is to explore an alternative hierarchical organization of these components. This still leaves open how the second-stage spectral transformation should refine the normalized update.
The second question is “Should the normalized update undergo a full-spectrum transformation, or serve as a reference for more targeted spectral modifications?” This question guides the derivation of concrete optimization methods within the framework. Since normalization already provides a useful update, we consider both global transformation through spectral-norm steepest descent and localized refinement that favors preserving this normalized reference. For the latter, regularized spectral steepest descent introduces a quadratic penalty on the Frobenius norm of the update, and the resulting problem is equivalent to finding the closest update to the normalized reference in Frobenius distance within the unit spectral-norm ball. We then derive localized spectral preconditioning through an optimal rank-constrained approximation to the resulting correction. These constructions lead to NormPre-G and NormPre-L, respectively.
We have revised the Introduction (Section 1) to make these two questions and their progression explicit.
2. Row/column normalization constructs the reference, and spectral preconditioning processes its directional interactions.
For nonzero rows, the full Gram matrix can be written as XX⊤=D(X)ΓD(X),Ψ=D(X)−1X,Γ=ΨΨ⊤. Here, D(X) contains row magnitudes, while Γ records pairwise row cosine similarities. This separates marginal-scale weighting from directional correlations. The row/column operation constructs a reference update in the layer's input/output coordinates. The subsequent spectral transformation acts along eigenvectors of Γ, which generally involve combinations of multiple coordinates. Thus, the framework explicitly processes interactions beyond individual rows or columns. The two stages have different roles: marginal normalization constructs a useful reference, and spectral preconditioning determines how to refine it.
Muon indeed uses the full singular spectrum and cross-row information, as explained in Section 2.1 of the original manuscript. The distinction concerns the geometry supplied to the spectral transformation. Even for a fixed input signal X, marginal normalization can change its singular vectors, i.e., in general, msign(D(X)−1X)=msign(X). Consequently, the distinction persists even with exact orthogonalization.