acceptodds
Under review as a conference paper at ICLR 2027

From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon

Abstract

Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear. We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification. Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions. Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances. It holds for power-law spectra with class-mean exponent below one, even under heavily anisotropic test noise. When both covariances are diagonal and sufficiently close, the advantage over Adam can reverse, while applying the same random rotation to both restores it by changing only their alignment with Adam's coordinate axes. Synthetic and last-layer language-model experiments support the predicted advantage.

Primary area:
optimization
Keywords:
Row normalization, Adam, Muon, implicit bias, population accuracy, multiclass classification, normalized steepest descent, random reshuffling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LKVYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper provide a theoretical explanation of why adding row norm in existing methods can obtain better accuracy, namely the row-normalized method could facilitate population loss decay as well as training loss decay. The authors adopt the linear model and Gaussian-cloud data, and assume high-dimensional scaling to prove that the row-norm methods achieve better accuracy under multiple training settings, including full batch, random-reshuffling. They also provide a discussion on when row-norm methods may loss its advantage, together with how to restore it. Empirical validations of the theoretical settings and results are also provided.

Critical strengths
  • The authors provide a theoretical explanation of a technique used in practical algorithms that obtain good performance.
  • The whole theory is rather complete: it includes discussion on both the advantages of row-norm methods and the downside of them. Anisotropic data is considered instead of only overly simplified settings; especially, they consider power-law spectra which connects closely with the practice. Experimental validation is also provided.
Critical weaknesses
  • It is still unclear when row-norm methods are better beyond the proposed theoretical settings. See Questions
  • The proof techniques do not seem new.
Questions for authors
  • The authors already addressed in the limitations that the empirical settings might be different. I wonder whether the authors could provide additional discussion on what specifically leads to the success of Adam and Muon from the perspective of why row-norm methods may fail; maybe beyond listing the limitations of the theoretical settings like linear model and restricted data etc.
  • To echo the above questions, I can understand why Adam and Muon perform better in the coordinate-aligned settings. However, if the rotation would break their advantages, I would assume the row-norm methods could succeed in more complicated practical training tasks. I wonder whether the authors could provide some empirical or theoretical evidence and discussion for this (beyond listing the empirical results in the introduction), especially what type of tasks are suitable for adding row-norm.
5 comments
AuthorsOpenReview ↗

Response to Reviewer LKVY (1/3)

We thank the reviewer for the thoughtful comments and helpful suggestions on the practical relevance of our results.

W2. About the proof technique

While we use established implicit-bias results and concentration tools, our main technical contribution is the derivation of concrete classifier structures under our Gaussian-cloud data model, whereas prior implicit-bias results characterize the selected classifiers as solutions to norm-specific max-margin problems without explicitly determining the structure of these solutions. This requires new arguments beyond the existing implicit-bias proof techniques. Specifically, we derive a common form of lower bounds on the norms of classifiers satisfying the training margin constraints, then construct classifiers that provide matching upper bounds for each of the three norms associated with the row-normalized optimizer, Muon, and Adam. By matching these bounds, we establish geometric properties shared by every norm-minimizing classifier, rather than merely analyzing the constructed examples. This allows us to go beyond identifying the implicit bias and rigorously compare the population accuracies of the resulting classifiers. We have added a proof overview in Appendix B to make these techniques and their contributions clearer.

Q1/W1. When Adam and Muon Can Outperform Row Normalization

We would like to highlight two concrete mechanisms that can favor Adam or Muon over row normalization.

First, associative-memory learning, which models factual associations in transformer hidden layers, provides a setting where Muon's spectral normalization may be advantageous over row normalization (Kim et al., 2026). Under power-law item frequencies, frequent associations dominate the gradient, while rare associations contribute weaker signals. Muon's spectral normalization amplifies smaller singular components, enabling substantially greater one-step recovery than SGD. Row-normalized optimizers normalize individual rows but do not address the global singular-value imbalance. Although Kim et al. do not directly compare Muon with row-normalized optimizers, their results suggest why Muon can be advantageous in this setting.

This does not contradict our results, which focus on the final linear classifier, where Muon's singular-value equalization distorts class-mean directions and degrades population accuracy. Together, the two results complement the symmetry-based perspective of Lau and Su (2026), which motivates spectral normalization as a natural choice for hidden layers and row normalization as a natural choice for the last layer. While symmetry considerations alone do not establish which optimizer performs better, Kim et al.'s results and ours provide complementary performance-based evidence supporting the broader picture that spectral normalization can be advantageous for hidden-layer associative learning, whereas row normalization can be preferable for last-layer classification.

Second, our Proposition 4.5 identifies a concrete setting where Adam outperforms RowGDM. When coordinates carrying strong class signals also have high test-noise variance, RowGDM's tendency to preserve class-mean directions can amplify noise and hurt population accuracy. In contrast, Adam's tendency to clip large coordinates reduces the influence of the noisy coordinates, acting as beneficial implicit regularization.

In practice, unusually large normalization weights have been observed in LLMs (Qwen Team, 2025). If such large scaling factors occur in the RMSNorm layer immediately preceding the LM head(last layer), they could amplify both class signals and within-class noise along the same coordinates, potentially creating conditions favorable to Adam.

  • Juno Kim, Eshaan Nichani, Denny Wu, Alberto Bietti, and Jason D. Lee. Sharp capacity scaling of spectral optimizers in learning associative memory. arXiv preprint arXiv:2603.26554, 2026.
  • Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers. arXiv preprint arXiv:2605.18106, 2026.
  • Qwen Team. Qwen3-Next: Towards ultimate training & inference efficiency. Qwen Blog, 2025. https://qwen.ai/blog?id=e34c4305036ce60d55a0791b170337c2b70ae51d.
AuthorsOpenReview ↗

Response to Reviewer LKVY (2/3)

Q2/W1. Tasks Favoring Row Normalization

Based on our theoretical results, we suggest two rules of thumb for when row normalization may be beneficial. First, row normalization is particularly well motivated for classification heads, which are the focus of our theoretical analysis. Second, it may be especially beneficial with sufficiently long training, since our results concern asymptotic classifier quality rather than convergence speed, although this does not imply that row-normalized methods perform worse under shorter training budgets.

Beyond our theoretical scope, we also have an intuition regarding class imbalance. Since row normalization normalizes the gradients associated with each class separately, it may mitigate the slow learning of low-frequency classes under unnormalized GD or SGD. Thus, class-imbalanced tasks may be particularly worth considering, although this intuition does not imply an advantage over Adam or Muon. Glentis et al. (2026) also discuss this mechanism in the context of LM-head training.

Glentis et al. (2026) highlight the memory efficiency of the row-normalized optimizer, which maintains only one optimizer state per parameter, compared with two in Adam. This makes the row-normalized optimizer particularly attractive compared with Adam in memory-constrained settings.

We provide three concrete examples supporting the use of row normalization for classification heads.

  1. Language-model training. Both our last-layer experiments and Lau and Su (2026) support applying row-normalized optimizers to LM heads. Additional evidence comes from full-model pretraining with row-normalized optimizers, including SCALE, MOGA, and RMNP (Glentis et al., 2026; Xu et al., 2026; Deng et al., 2026). Specifically, SCALE achieves lower perplexity than Adam and Muon on LLaMA 350M and 1B, while RMNP achieves lower perplexity than AdamW and Muon on GPT-2 125M, 355M, and 770M. These methods also apply row normalization to other layers, so their results do not isolate the contribution of the last layer, but demonstrate that row normalization can be effective in practical full-model training that includes the last layer.

  2. Intent classification with pretrained representations. We added a new experiment in Appendix I.7 on CLINC150 (Larson et al., 2019), which classifies sentences into 150 intent classes. We freeze a pretrained sentence-transformer encoder and train only a zero-initialized, bias-free linear classifier on its 768-dimensional representations. Across all five label budgets (1, 2, 5, 10, and 20 examples per class), RowGDM achieves higher mean test accuracy than both Adam and Muon over five seeds. The improvements range from 0.147 to 0.964 percentage points over Adam and from 0.542 to 2.587 points over Muon, with learning rates and checkpoints selected using validation data. In paired seed-level comparisons, RowGDM outperforms Muon in all 25 runs and Adam in 22 of 25 runs. Although the gains over Adam are modest, these results provide a practical example where row normalization achieves favorable performance in classification-head training on learned representations.

AuthorsOpenReview ↗

Response to Reviewer LKVY (3/3)

Q2/W1. Tasks Favoring Row Normalization (Continued)

  1. Image classification with fixed kernel features. We added a new experiment in Appendix I.8 on EMNIST (Cohen et al., 2017), which classifies handwritten images into 47 character classes. We use a fixed 512-dimensional random Fourier feature map and train only a zero-initialized, bias-free linear classifier. Across all three label budgets (1, 5, and 20 examples per class), RowGDM achieves higher mean test accuracy than both Adam and Muon over five seeds. The improvements range from 0.440 to 1.069 percentage points over Adam and from 0.435 to 1.852 percentage points over Muon, with learning rates and checkpoints selected using validation data. In paired seed-level comparisons, RowGDM outperforms Muon in all 15 runs and Adam in 14 of 15 runs. These results provide a non-language example where row normalization achieves favorable performance in classification-head training on fixed nonlinear representations.
  • Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. Memory-efficient LLM pretraining via minimalist optimizer design. In International Conference on Machine Learning, pp. 35197–35226. PMLR, 2026.
  • Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers. arXiv preprint arXiv:2605.18106, 2026.
  • Ruihan Xu, Jiajin Li, and Yiping Lu. On the width scaling of neural optimizers under matrix operator norms I: Row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952, 2026.
  • Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. RMNP: Row-momentum normalized preconditioning for scalable matrix-based optimization. In International Conference on Machine Learning, pp. 24191–24239. PMLR, 2026.
  • Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, et al. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1311–1316, 2019.
  • Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926. IEEE, 2017.
Reviewer_akMxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper connects optimizer-induced maximum-margin geometry to population performance in high-dimensional Gaussian multiclass classification. It proves advantages for row normalization over spectral and coordinate-wise alternatives, extends restricted stochastic comparisons through AdamProxy, and develops anisotropic persistence and coordinate-dependent reversal results. Synthetic and frozen-language-model-head experiments support the qualitative ordering. The theoretical contribution is substantial, although its practical interpretation is narrower than the title might suggest.

Critical strengths

Theorem 3.1 and Corollary 3.3 establish a population-accuracy ordering and exponential error-ratio separation, going beyond characterizing the norms associated with optimizer limits. This addresses a substantive missing connection between optimization geometry and generalization. Theorem C.2 establishes geometric separation uniformly over complete margin-solution sets. Pairwise Gaussian error bounds and continuity then transfer this separation to trajectories without assuming directional convergence in the isotropic setting. This avoids relying on a conveniently selected optimum. Proposition 3.2 identifies class-mean direction preservation for row normalization, coordinate-magnitude equalization for Adam/sign methods, and singular-value equalization for spectral methods. Figure 1 and the row-rigidity analysis make the explanation accessible while retaining mathematical substance

Critical weaknesses

Proposition 3.5 approximates one normalized epoch displacement at fixed dimension as both momentum parameters approach one; it does not transfer long-horizon behavior or accuracy ordering to actual RR-Adam. The bounded mini-batch-count and high-momentum assumptions further restrict the stochastic regime. Five-seed frozen-head gains are consistent but small: approximately 0.26% perplexity over Adam at 60M and 0.16% at 130M.

Questions for authors
  1. Do the predicted population-error exponents and boundary distortions emerge in controlled dimension and sample-size scaling experiments?
  2. How closely do RR-Adam and AdamProxy boundary trajectories agree at practical momentum values and mini-batch sizes?
  3. Does a common orthogonal rotation of frozen language-model features change the optimizer ordering in the way predicted by the coordinate-dependence explanation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer akMx (1/2)

We thank the reviewer for the thorough assessment and insightful questions.

W1/Q2. AdamProxy Approximation and Empirical Agreement with RR-Adam

Our analysis builds on the AdamProxy framework of Baek et al. (2026), which is based on a one-epoch approximation in the high-β2\beta_2 regime. Our result extends this framework from incremental Adam (i.e., deterministic fixed order) in binary classification to random reshuffling Adam in multiclass classification, which requires more complicated analysis. At fixed data and dimension, Proposition 3.5 approximates a normalized late-epoch RR-Adam displacement using the proxy direction at the current RR-Adam iterate, as both momentum parameters approach one. Proposition 3.6 establishes the population-accuracy ordering against AdamProxy.

Our theoretical result does not establish long-horizon agreement; however, as a response to the reviewer’s question and to assess agreement beyond the theoretical guarantee, we added a direct comparison of RR-Adam and AdamProxy in Appendix I.4, using two practical momentum settings, (β1,β2)∈{(0.9,0.95),(0.9,0.999)}(\beta_1,\beta_2) \in \lbrace(0.9,0.95),(0.9,0.999)\rbrace, and mini-batch sizes b∈{10,30,120}b \in \lbrace10,30,120\rbrace, on a synthetic dataset with d=20d=20, k=60k=60, and N=480N=480.

To compare decision boundaries, we normalize each class-weight difference wi−wjw_i-w_j to unit length and average the Euclidean distance between the resulting normals over all class pairs. We also report the relative parameter error ∥WRR−Adam−WAP∥F/∥WAP∥F\lVert W^{\mathrm{RR-Adam}}-W^{\mathrm{AP}}\rVert_{\mathrm F}/\lVert W^{\mathrm{AP}}\rVert_{\mathrm F}, where WRR−AdamW^{\mathrm{RR-Adam}} and WAPW^{\mathrm{AP}} denote the parameters of RR-Adam and AdamProxy, respectively. At approximately 10610^6 updates, we obtain:

Batch size(β1,β2)(\beta_1,\beta_2)Boundary disagreementRelative parameter error
10(0.9,0.95)0.019944.83%
10(0.9,0.999)0.008553.71%
30(0.9,0.95)0.005100.90%
30(0.9,0.999)0.007002.91%
120(0.9,0.95)0.002100.37%
120(0.9,0.999)0.007391.73%

These results indicate reasonable agreement at the approximately 10610^6 updates. Both discrepancies decreased across the four reported checkpoints at β2=0.999\beta_2 = 0.999. At β2=0.95\beta_2 = 0.95, both ended above their first reported values for each batch size. The largest final discrepancies occurred at b=10b=10 and (β1,β2)=(0.9,0.95)(\beta_1, \beta_2) = (0.9, 0.95). Appendix I.4 also includes qualitative comparisons of the trajectories and quantitative results at smaller training steps.

  • Beomhan Baek, Minhak Song, and Chulhee Yun. Implicit bias of per-sample Adam on separable data: Departure from the full-batch regime. In International Conference on Learning Representations, pp. 120180–120229, 2026.

W2. Assumptions for Stochastic Optimizers

We agree that these assumptions restrict the stochastic regime. To be specific, Assumption 2.3 requires each mini-batch to contain at least a 1/m01/m_0 fraction of the training set. The sufficient momentum threshold used in Proposition 3.4 moves closer to one as m0m_0 increases. This controls the margin deficit relative to the full-batch geometry for RR-RowSGDM, RR-Muon, and RR-Signum.

However, similar restrictions arise in prior implicit-bias analyses. Li et al. (2026, Theorem 4.4) obtain a margin bound that becomes loose as the number of mini-batches increases or momentum decreases. Similarly, Baek et al. (2026, Theorem 5.1) require sufficiently high momentum for mini-batch Signum to retain its full-batch implicit bias.

Moreover, Li et al. (2026, Section 4.4) show that, without momentum, batch-size-one stochastic steepest descent can exhibit a fundamentally different, data-dependent implicit bias, whose characterization is particularly challenging for general datasets. Thus, our assumptions focus on a regime where full-batch-like implicit bias can be preserved. Extending our analysis to smaller batches and lower momentum, where the implicit bias may fundamentally change, is an important direction for future work.

  • Jichu Li, Xuan Tang, and Difan Zou. The implicit bias of steepest descent with mini-batch stochastic gradient. In International Conference on Machine Learning, pp. 70353–70413. PMLR, 2026.
  • Beomhan Baek, Minhak Song, and Chulhee Yun. Implicit bias of per-sample Adam on separable data: Departure from the full-batch regime. In International Conference on Learning Representations, pp. 120180–120229, 2026.
AuthorsOpenReview ↗

Response to Reviewer akMx (2/2)

W3. Scope of Our Paper and Purpose of the Empirical Experiments

We agree that the perplexity improvements are modest. However, the primary goal of our paper is not to develop a new optimizer that achieves substantial empirical gains, but to theoretically establish why row normalization can yield better population performance than competing optimizer geometries. Our last-layer training experiments are intended to provide empirical evidence in a more realistic setting, although the improvements are modest. This may partly reflect the gap between our asymptotic analysis under a Gaussian data model and finite-time training on more complex language-model representations.

Q1. Empirical Validation of Asymptotic Predictions

We added controlled dimension and sample-size scaling experiments in Appendix I.5, comparing trained RowGDM, Adam, and exact-SVD Muon classifiers. Under joint scaling of the dimension and samples per class, the estimated exponent gaps and boundary-normal distortion angles show trends consistent with the theory.

From d=16d=16 to d=2048d=2048, the normalized log-error ratios relative to RowGDM, which measure the population-error exponent gaps characterized in Corollary 3.3, increase from 0.02136 to 0.08575 for Adam, toward its predicted limit 0.09085, and from 0.00889 to 0.02584 for Muon, toward its asymptotic lower-bound reference 0.03247.

Over the same range, the mean angles between learned decision-boundary normals and the corresponding population-mean differences change from 21.62° to 4.50° for RowGDM, 29.79° to 36.93° for Adam, and 28.28° to 21.55° for Muon, trending toward the respective predictions 0°, 37.07°, and 21.12°.

Q3. Effect of Orthogonal Feature Rotation

The optimizer ordering was unchanged under a common orthogonal rotation, but rotation worsens Adam's performance relative to RowSGDM as the feature covariances move farther from diagonal structure, qualitatively consistent with the coordinate-dependence mechanism in our theory. We added this result in Appendix I.6 and summarize the result in this response.

We apply the same fixed Haar-random orthogonal rotation to all frozen LLaMA-60M features and retrain the output head, keeping each optimizer's learning rate, training stream, and schedule fixed. The degradation in perplexity is higher in Adam, which makes the performance gap between RowSGDM and Adam larger.

OptimizerOriginal test PPLRotated test PPLChange
RowSGDM28.6472228.65858+0.01136
Muon28.7023928.72106+0.01868
Adam28.7197128.82601+0.10630

In our theoretical construction, applying a common rotation to training and test samples makes a previously diagonal class-mean covariance non-diagonal. To examine whether a similar change occurs in language-model representations, we measure how far their class-mean covariance is from diagonal before and after rotation. The fraction of squared covariance entries outside the diagonal increases from 57.97% to 98.62%, indicating that the rotation moves the class-mean covariance farther from the diagonal structure.

Unlike the theoretical reversal-and-recovery construction, the original language-model covariances are already non-diagonal, and RowSGDM already outperforms Adam before rotation. Thus, an ordering reversal is not observed. Nevertheless, the rotation moves the class-mean covariance farther from diagonal structure and degrades Adam's performance more than RowSGDM's, qualitatively resembling the coordinate dependence identified by our theory.

Reviewer_QiKhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that row-normalized optimizers can outperform Adam and Muon in high-dimensional classification tasks. The authors link optimizer geometry to preservation of class-mean decision-boundary directions in a Gaussian-cloud data model. Empirical experiments are conducted on synthetic and last-layer language-model experiments. The results are reported to support the main claims.

Critical strengths
  • The idea of connecting optimizer geometry and population accuracy is quite interesting.
  • The theoretical section is well framed, and empirical evidence supports the main claim.
Critical weaknesses
  • In the synthetic experiment, learning rates are selected and evaluated on the same sample. This may make the reported impact unreliable.
  • In the real data experiment, only the output head is trained. It's unclear how the method performs on full model training.
Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer QiKh

We thank the reviewer for the careful reading and constructive feedback on our experiments.

W1. About the synthetic experiment learning rate selection

We agree that reusing the same sample for learning-rate selection and evaluation can introduce selection bias. We therefore evaluated the selected classifiers on independent Monte Carlo samples of 120,000 fresh examples per seed without retuning the learning rates. The qualitative optimizer comparisons about when row-normalized methods perform better or worse than other optimizers remain unchanged. Furthermore, across these settings and optimizers, the largest absolute change in mean accuracy over 30 runs was 0.087 percentage points. We have updated Figure 2 (left), the synthetic experiment figures in Appendix I.1, and the corresponding description.

W2. Focus of Our Paper and Full-Model Training Results from Prior Work

Row-normalized optimizers have already been applied to full-model training in prior work. Specifically, prior full-model pretraining studies have evaluated optimizers that incorporate row normalization, including SCALE, MOGA, and RMNP (Glentis et al., 2026; Xu et al., 2026; Deng et al., 2026). To summarize prior work, SCALE achieves lower perplexity than Adam and Muon on LLaMA 350M and 1B, while RMNP achieves lower perplexity than AdamW and Muon on GPT-2 125M, 355M, and 770M. Our work builds on these prior full-model training studies while theoretically investigating the properties of row-normalized optimizers in the last layer.

Our paper establishes theoretical results that aim to elucidate when and why row-normalized optimizers can perform well on the last layer. We intentionally freeze the pretrained backbone and train only the last layer in our language-model experiments to test our theory in a setting closer to our theoretical setup.

  • Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. Memory-efficient LLM pretraining via minimalist optimizer design. In International Conference on Machine Learning, pp. 35197–35226. PMLR, 2026.
  • Shenyang Deng, Zhuoli Ouyang, Tianyu Pang, Zihang Liu, Ruochen Jin, Shuhua Yu, and Yaoqing Yang. RMNP: Row-momentum normalized preconditioning for scalable matrix-based optimization. In International Conference on Machine Learning, pp. 24191–24239. PMLR, 2026.
  • Ruihan Xu, Jiajin Li, and Yiping Lu. On the width scaling of neural optimizers under matrix operator norms I: Row/column normalization and hyperparameter transfer. arXiv preprint arXiv:2603.09952, 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision of the Paper

We thank the reviewers and the area chair for their time, careful reading, and thoughtful feedback. Your comments have been very helpful for improving our paper. For the reviewers' convenience, we highlighted our major modifications in blue.

Below, we provide a brief summary of the major changes in the revised manuscript.

  • We added a proof overview in Appendix B, explaining how our norm-based arguments characterize the classifier geometries selected by different optimizers.
  • We revised the synthetic experiments to use independent samples for learning-rate selection and evaluation, and updated the corresponding figures (Figure 2 and Appendix I.1).
  • We added experiments comparing RR-Adam and AdamProxy trajectories across practical momentum settings and mini-batch sizes in Appendix I.4.
  • We added dimension and sample-size scaling experiments to examine the predicted population-error exponents and decision-boundary distortions in Appendix I.5.
  • We added an experiment on orthogonal rotations of frozen language-model features in Appendix I.6, showing that rotation degrades Adam's performance more than RowSGDM's.
  • We added an intent classification experiment on CLINC150 using pretrained sentence representations in Appendix I.7, where RowGDM outperforms Adam and Muon.
  • We added an image classification experiment on EMNIST using fixed random Fourier features in Appendix I.8, where RowGDM outperforms Adam and Muon.
Area_Chair_AgKYOpenReview ↗

Please read the responses and engage with authors

Dear Reviewers,

The discussion period is coming to an end soon. I encourage you to read the authors' responses and enter any further questions and comments you might have. Thank you very much for your thoughtful reviews and engagement throughout the process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.