- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper provide a theoretical explanation of why adding row norm in existing methods can obtain better accuracy, namely the row-normalized method could facilitate population loss decay as well as training loss decay. The authors adopt the linear model and Gaussian-cloud data, and assume high-dimensional scaling to prove that the row-norm methods achieve better accuracy under multiple training settings, including full batch, random-reshuffling. They also provide a discussion on when row-norm methods may loss its advantage, together with how to restore it. Empirical validations of the theoretical settings and results are also provided.
Critical strengths
- The authors provide a theoretical explanation of a technique used in practical algorithms that obtain good performance.
- The whole theory is rather complete: it includes discussion on both the advantages of row-norm methods and the downside of them. Anisotropic data is considered instead of only overly simplified settings; especially, they consider power-law spectra which connects closely with the practice. Experimental validation is also provided.
Critical weaknesses
- It is still unclear when row-norm methods are better beyond the proposed theoretical settings. See Questions
- The proof techniques do not seem new.
Questions for authors
- The authors already addressed in the limitations that the empirical settings might be different. I wonder whether the authors could provide additional discussion on what specifically leads to the success of Adam and Muon from the perspective of why row-norm methods may fail; maybe beyond listing the limitations of the theoretical settings like linear model and restricted data etc.
- To echo the above questions, I can understand why Adam and Muon perform better in the coordinate-aligned settings. However, if the rotation would break their advantages, I would assume the row-norm methods could succeed in more complicated practical training tasks. I wonder whether the authors could provide some empirical or theoretical evidence and discussion for this (beyond listing the empirical results in the introduction), especially what type of tasks are suitable for adding row-norm.
5 comments
Response to Reviewer LKVY (2/3)
Q2/W1. Tasks Favoring Row Normalization
Based on our theoretical results, we suggest two rules of thumb for when row normalization may be beneficial. First, row normalization is particularly well motivated for classification heads, which are the focus of our theoretical analysis. Second, it may be especially beneficial with sufficiently long training, since our results concern asymptotic classifier quality rather than convergence speed, although this does not imply that row-normalized methods perform worse under shorter training budgets.
Beyond our theoretical scope, we also have an intuition regarding class imbalance. Since row normalization normalizes the gradients associated with each class separately, it may mitigate the slow learning of low-frequency classes under unnormalized GD or SGD. Thus, class-imbalanced tasks may be particularly worth considering, although this intuition does not imply an advantage over Adam or Muon. Glentis et al. (2026) also discuss this mechanism in the context of LM-head training.
Glentis et al. (2026) highlight the memory efficiency of the row-normalized optimizer, which maintains only one optimizer state per parameter, compared with two in Adam. This makes the row-normalized optimizer particularly attractive compared with Adam in memory-constrained settings.
We provide three concrete examples supporting the use of row normalization for classification heads.
-
Language-model training. Both our last-layer experiments and Lau and Su (2026) support applying row-normalized optimizers to LM heads. Additional evidence comes from full-model pretraining with row-normalized optimizers, including SCALE, MOGA, and RMNP (Glentis et al., 2026; Xu et al., 2026; Deng et al., 2026). Specifically, SCALE achieves lower perplexity than Adam and Muon on LLaMA 350M and 1B, while RMNP achieves lower perplexity than AdamW and Muon on GPT-2 125M, 355M, and 770M. These methods also apply row normalization to other layers, so their results do not isolate the contribution of the last layer, but demonstrate that row normalization can be effective in practical full-model training that includes the last layer.
-
Intent classification with pretrained representations. We added a new experiment in Appendix I.7 on CLINC150 (Larson et al., 2019), which classifies sentences into 150 intent classes. We freeze a pretrained sentence-transformer encoder and train only a zero-initialized, bias-free linear classifier on its 768-dimensional representations. Across all five label budgets (1, 2, 5, 10, and 20 examples per class), RowGDM achieves higher mean test accuracy than both Adam and Muon over five seeds. The improvements range from 0.147 to 0.964 percentage points over Adam and from 0.542 to 2.587 points over Muon, with learning rates and checkpoints selected using validation data. In paired seed-level comparisons, RowGDM outperforms Muon in all 25 runs and Adam in 22 of 25 runs. Although the gains over Adam are modest, these results provide a practical example where row normalization achieves favorable performance in classification-head training on learned representations.
Response to Reviewer LKVY (3/3)
Q2/W1. Tasks Favoring Row Normalization (Continued)
- Image classification with fixed kernel features. We added a new experiment in Appendix I.8 on EMNIST (Cohen et al., 2017), which classifies handwritten images into 47 character classes. We use a fixed 512-dimensional random Fourier feature map and train only a zero-initialized, bias-free linear classifier. Across all three label budgets (1, 5, and 20 examples per class), RowGDM achieves higher mean test accuracy than both Adam and Muon over five seeds. The improvements range from 0.440 to 1.069 percentage points over Adam and from 0.435 to 1.852 percentage points over Muon, with learning rates and checkpoints selected using validation data. In paired seed-level comparisons, RowGDM outperforms Muon in all 15 runs and Adam in 14 of 15 runs. These results provide a non-language example where row normalization achieves favorable performance in classification-head training on fixed nonlinear representations.
- Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. Memory-efficient LLM pretraining via minimalist optimizer design. In International Conference on Machine Learning, pp. 35197–35226. PMLR, 2026.
- Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers. arXiv preprint arXiv:2605.18106, 2026.
- Ruihan Xu, Jiajin Li, and Yiping Lu. On the width scaling of neural optimizers under matrix operator norms I: Row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952, 2026.
- Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. RMNP: Row-momentum normalized preconditioning for scalable matrix-based optimization. In International Conference on Machine Learning, pp. 24191–24239. PMLR, 2026.
- Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, et al. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1311–1316, 2019.
- Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926. IEEE, 2017.
Response to Reviewer LKVY (1/3)
We thank the reviewer for the thoughtful comments and helpful suggestions on the practical relevance of our results.
W2. About the proof technique
While we use established implicit-bias results and concentration tools, our main technical contribution is the derivation of concrete classifier structures under our Gaussian-cloud data model, whereas prior implicit-bias results characterize the selected classifiers as solutions to norm-specific max-margin problems without explicitly determining the structure of these solutions. This requires new arguments beyond the existing implicit-bias proof techniques. Specifically, we derive a common form of lower bounds on the norms of classifiers satisfying the training margin constraints, then construct classifiers that provide matching upper bounds for each of the three norms associated with the row-normalized optimizer, Muon, and Adam. By matching these bounds, we establish geometric properties shared by every norm-minimizing classifier, rather than merely analyzing the constructed examples. This allows us to go beyond identifying the implicit bias and rigorously compare the population accuracies of the resulting classifiers. We have added a proof overview in Appendix B to make these techniques and their contributions clearer.
Q1/W1. When Adam and Muon Can Outperform Row Normalization
We would like to highlight two concrete mechanisms that can favor Adam or Muon over row normalization.
First, associative-memory learning, which models factual associations in transformer hidden layers, provides a setting where Muon's spectral normalization may be advantageous over row normalization (Kim et al., 2026). Under power-law item frequencies, frequent associations dominate the gradient, while rare associations contribute weaker signals. Muon's spectral normalization amplifies smaller singular components, enabling substantially greater one-step recovery than SGD. Row-normalized optimizers normalize individual rows but do not address the global singular-value imbalance. Although Kim et al. do not directly compare Muon with row-normalized optimizers, their results suggest why Muon can be advantageous in this setting.
This does not contradict our results, which focus on the final linear classifier, where Muon's singular-value equalization distorts class-mean directions and degrades population accuracy. Together, the two results complement the symmetry-based perspective of Lau and Su (2026), which motivates spectral normalization as a natural choice for hidden layers and row normalization as a natural choice for the last layer. While symmetry considerations alone do not establish which optimizer performs better, Kim et al.'s results and ours provide complementary performance-based evidence supporting the broader picture that spectral normalization can be advantageous for hidden-layer associative learning, whereas row normalization can be preferable for last-layer classification.
Second, our Proposition 4.5 identifies a concrete setting where Adam outperforms RowGDM. When coordinates carrying strong class signals also have high test-noise variance, RowGDM's tendency to preserve class-mean directions can amplify noise and hurt population accuracy. In contrast, Adam's tendency to clip large coordinates reduces the influence of the noisy coordinates, acting as beneficial implicit regularization.
In practice, unusually large normalization weights have been observed in LLMs (Qwen Team, 2025). If such large scaling factors occur in the RMSNorm layer immediately preceding the LM head(last layer), they could amplify both class signals and within-class noise along the same coordinates, potentially creating conditions favorable to Adam.