acceptodds
Under review as a conference paper at ICLR 2027

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training

Abstract

Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through an anonymous GitHub repository at https://anonymous.4open.science/r/NormPre.

Primary area:
optimization
Keywords:
large language model training, matrix optimizers, spectral preconditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FFCNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Normalize-Then-Precondition framework that combines marginal-scale normalization with directional spectral preconditioning for matrix optimizers. Building on this framework, the authors introduce NormPre, which includes NormPre-G for global spectral preconditioning and NormPre-L for localized spectral refinement. Empirically, NormPre consistently outperforms AdamW, Muon, and MANO across various LLM pretraining settings. The authors also establish convergence guarantees for both variants.

Critical strengths
  1. The method is lightweight and easy to implement, requiring only minor modifications to the original Muon.

  2. The experiments covers multiple model families such as GPT-2, LLaMA and Qwen3, and multiple datasets including C4 and Pile.

Critical weaknesses
  1. Motivation seems not sufficiently clear. This work primarily addresses directional interactions and scale imbalance, but the value of solving this problem is not clearly established. In particular, the paper does not explain, from a design-motivation perspective, why addressing this issue is beneficial for training. The row/column perspective is essentially formal and is not directly linked to training efficiency. Moreover, scale can be oriented in arbitrary directions and is not limited to rows or columns. The paper does not justify why considering row- or column-wise scaling is necessary. For example, Muon can be viewed as scale normalization within the eigenspace and already achieves this goal. Furthermore, in the extreme case where row (or column) norms are highly imbalanced, although the leading Gram eigenvector approaches the coordinate axis of the largest row, this may not prevent the optimizer from capturing correlations among the remaining rows.

  2. The experimental evaluation is insufficient.

    • All main experiments use 10,000 steps, corresponding to only about 5B tokens. For the larger models in experiments, the data is too little to reflect realistic training scenarios. It is suggested to train for at least 1$\times$ Chinchilla-optimal data (roughly 20$\times$ the model parameters). For LLaMA-1.3B, this is about 25B tokens, and about 30B tokens for Qwen3-1.7B. The current budget is only about 0.2× or less of this amount.
    • Model scale is limited to 1.7B, with no validation on larger models.
    • The set of baselines is also relatively small. It is recommended to compare with recent Muon variants using similar row/col scaling such as NorMuon [1].
    • MoE models are not evaluated.
    • Based on the experimental description, the baseline learning rates do not appear to have been sufficiently searched or tuned, especially for the larger models. Thus the observed advantages may not reflect the true performance of the baselines under fair and well-tuned settings.

[1] NorMuon: Making Muon more efficient and scalable. ICML 2026.

Questions for authors
  1. The T−0.5T^{-0.5} gradient-norm bound in Theorem 1 is standard for non-convex optimization and no stronger than SGD. My question concerns the significance and value of this theoretical result. What does it actually establish, and what does it add beyond a standard convergence guarantee? I acknowledge the technical effort involved in the derivation. However, the T−0.5T^{-0.5} rate alone cannot explain why NormPre outperforms Muon, which makes it weak support for the paper’s central claims. Although end-to-end analysis of LLM training is difficult, it is encouraged to consider a more realistic model (even a toy model) to analyze the advantages brought by NormPre. Such intuitive theoretical understanding would help readers understand the role of the proposed algorithm more than a single non-convex rate.

  2. On line 73, the authors present an illustrative comparison between 1e4 and 1e-3. Would such a large disparity occur in realistic LLM training scenarios?

  3. What is the motivation behind momentum projection? Why is the relaxed tangent projection appropriate, and better than both no projection and strict tangent projection? Operations such as msign may undermine this calibration of radial scale information.

  4. Table 6b shows that row normalization only performs better than alternating row/column for both NormPre-G and NormPre-L. This appears to suggest that the alternating row/column design is redundant and even harmful. In addition, what would happen if only columns are normalized?

5 comments

Thank you for your suggestions. We appreciate your recognition of the lightweight implementation and the evaluation across multiple model architectures and datasets. In response, we have revised the Introduction to clarify our hierarchical insight and central design question, clarified the roles of the optimization formulations and convergence analysis, and added some ablation experiments. Below, we also explain the rationale and supporting evidence for relaxed tangent processing and alternating normalization, and discuss opportunities to broaden the evaluation. We address each concern in turn.

Q1: Motivation seems not sufficiently clear. Why separate marginal scales and directional interactions? Muon already rescales the eigenspectrum, and a dominant leading Gram eigenvector does not imply that it cannot capture other correlations.

A: Thank you for these questions. We would like to clarify the two central questions behind our framework, how the construction differs from Muon's full-Gram transformation, and the experimental evidence supporting our design choices.

1. Two core questions in the Introduction with distinct roles.

  • Muon's full-Gram representation reveals its joint treatment of marginal scales and cross-row interactions, while normalization draws solely on the Gram diagonal to equalize row norms without altering pairwise angles. Together with the observation that normalization alone already provides a useful update, this suggests organizing geometric information hierarchically: first use marginal scales to construct a normalized update, then use its remaining directional correlations for refinement. This motivates our first question: “Can we construct a normalized update from scales and refine it via directional interactions?” Muon already processes both types of information through its full-Gram transformation; our aim is to explore an alternative hierarchical organization of these components. This still leaves open how the second-stage spectral transformation should refine the normalized update.

  • The second question is “Should the normalized update undergo a full-spectrum transformation, or serve as a reference for more targeted spectral modifications?” This question guides the derivation of concrete optimization methods within the framework. Since normalization already provides a useful update, we consider both global transformation through spectral-norm steepest descent and localized refinement that favors preserving this normalized reference. For the latter, regularized spectral steepest descent introduces a quadratic penalty on the Frobenius norm of the update, and the resulting problem is equivalent to finding the closest update to the normalized reference in Frobenius distance within the unit spectral-norm ball. We then derive localized spectral preconditioning through an optimal rank-constrained approximation to the resulting correction. These constructions lead to NormPre-G and NormPre-L, respectively.

We have revised the Introduction (Section 1) to make these two questions and their progression explicit.

2. Row/column normalization constructs the reference, and spectral preconditioning processes its directional interactions.

  • For nonzero rows, the full Gram matrix can be written as XX⊤=D(X)ΓD(X),Ψ=D(X)−1X,Γ=ΨΨ⊤.XX^\top=D(X)\Gamma D(X), \Psi=D(X)^{-1}X, \Gamma=\Psi\Psi^\top. Here, D(X)D(X) contains row magnitudes, while Γ\Gamma records pairwise row cosine similarities. This separates marginal-scale weighting from directional correlations. The row/column operation constructs a reference update in the layer's input/output coordinates. The subsequent spectral transformation acts along eigenvectors of Γ\Gamma, which generally involve combinations of multiple coordinates. Thus, the framework explicitly processes interactions beyond individual rows or columns. The two stages have different roles: marginal normalization constructs a useful reference, and spectral preconditioning determines how to refine it.

  • Muon indeed uses the full singular spectrum and cross-row information, as explained in Section 2.1 of the original manuscript. The distinction concerns the geometry supplied to the spectral transformation. Even for a fixed input signal XX, marginal normalization can change its singular vectors, i.e., in general, msign⁡(D(X)−1X)≠msign⁡(X).\operatorname{msign}\left(D(X)^{-1}X\right)\neq\operatorname{msign}(X). Consequently, the distinction persists even with exact orthogonalization.

  • We agree that Muon can still capture correlations among the remaining rows even when the leading Gram eigenvector approaches the largest row’s coordinate axis. This is consistent with our example in lines 74–78, which explicitly considers strongly correlated remaining rows and unchanged directional interactions. The point of the example is that marginal scales can strongly shape the principal mode of the full Gram matrix even when directional relationships remain unchanged. This provides supplementary intuition for exploring an alternative organization: first construct a normalized reference that removes row-magnitude weighting, then refine its directional interactions through a spectral transformation. The practical benefit of this construction is assessed through the ablations below.

3. Existing and newly added ablations connect the design to training performance.

The following validation losses are obtained on GPT-2 Small/OpenWebText with seed 1337 and a 10k-step training budget. We present the component comparisons for NormPre-G and NormPre-L separately. The NormPre-G variants use NS5, while the NormPre-L variants use localized spectral preconditioning with randomized sketching. Muon is included in both tables as a baseline for reference.

NormPre-G

ConfigurationRelaxed tangentMarginal normalizationGram matrixValidation loss ↓\downarrow
w/o tangent processing✗✓ΨΨ⊤\Psi\Psi^\top3.0945
w/o marginal normalization✓✗XX⊤XX^\top3.0893
Geometry from XX✓✓XX⊤XX^\top3.0900
NormPre-G✓✓ΨΨ⊤\Psi\Psi^\top3.0711
Muon✗✗MM⊤MM^\top3.1137

NormPre-L

ConfigurationRelaxed tangentMarginal normalizationGram matrixValidation loss ↓\downarrow
w/o tangent processing✗✓ΨΨ⊤\Psi\Psi^\top3.1139
w/o marginal normalization✓✗XX⊤XX^\top3.1565
Geometry from XX✓✓XX⊤XX^\top3.1305
NormPre-L✓✓ΨΨ⊤\Psi\Psi^\top3.0864
Muon✗✗MM⊤MM^\top3.1137

Here, MM denotes the momentum signal, XX the update before marginal normalization in each configuration, and Ψ=D(X)−1X\Psi=D(X)^{-1}X its normalized counterpart; X=MX=M when tangent processing is omitted. The notation is written in the active row orientation. The Gram matrix column identifies the geometry used to construct the spectral preconditioner. The “Geometry from XX” variants retain Ψ\Psi as the base update but construct their preconditioners from XX⊤XX^\top.

  • Rows 1 and 3 in each table: ablations already reported in the original manuscript. Tables 6(a) and 7 in Appendix A.4 report the momentum-processing and interaction-geometry-source ablations, respectively, together with the results of 3.0711 for NormPre-G and 3.0864 for NormPre-L. Removing relaxed tangent processing increases validation loss for both variants. Retaining the normalized base update but constructing its preconditioner from XX also increases loss, supporting the use of normalized interaction geometry for spectral refinement.

  • Row 2 in each table: new ablations added during the rebuttal period. These controls bypass input marginal normalization while retaining relaxed tangent processing, the respective spectral routines and their parameters, the learning-rate schedules, and the original final RMS reference/scaling rules. For NormPre-G, the same five-step Newton–Schulz routine is applied to XX instead of Ψ\Psi, with its internal numerical normalization retained. For NormPre-L, the same sketch configuration (r=32r=32, o=8o=8, p=1p=1) and spectral threshold are retained. These controls provide empirical support for the contribution of input marginal normalization in both variants under the tested setting.

  • Row 5 in each table: a result from an existing Muon run. For completeness, we report its validation loss of 3.1137 with seed 1337.

For NormPre-L, Table 8 and Figure 4(c) additionally connect the spectral budget to the measured trade-off between validation loss and optimizer latency, providing a practical link between localized spectral refinement and computational efficiency.

In summary, we emphasize that our starting point is the full-Gram and diagonal-Gram representations of orthogonalization and normalization, together with the observation that normalization already provides a useful update. These observations motivate an insight into organizing geometric information hierarchically, followed by a central question: how much spectral modification should an already useful normalized update undergo? Together, this insight and question guide our systematic framework and its two spectral constructions. The example in lines 74–78 provides supplementary geometric intuition about how marginal scales might affect the full-Gram geometry, while the ablations above provide empirical support for the resulting design choices. We have revised the Introduction to make this progression clearer.

Q2: The experimental evaluation is insufficient.

A: Thank you for these suggestions. Our current evaluation covers multiple architectures, model sizes, and datasets: GPT-2 Small on OpenWebText, LLaMA-{130M, 350M, 1.3B} on C4, and Qwen3-{0.6B, 1.7B} on Pile. We focus on dense language models and compare against AdamW, Muon, and MANO under matched training budgets within each setting.

Regarding training duration, the reported token budgets for GPT-2 Small and LLaMA-130M already exceed the commonly used 20N20N reference. We have also supplemented the evaluation with learning-rate sweeps on GPT-2 Small, as detailed below.

1. Learning-rate choices in the main experiments. Different optimizers may have different optimal learning rates. Our main experiments use predefined learning-rate schedules together with established update-scaling conventions, including RMS matching to bring matrix-update magnitudes into a range comparable to AdamW. The optimizer-specific configurations are detailed in Appendix A.1. This provides a practical basis for comparing optimization behavior without separate optimizer-specific learning-rate tuning. We further complement this comparison with the additional search below, allowing each optimizer to select its best learning rate within the tested grid.

2. Additional learning-rate searches confirm that the advantages persist within the tested range. We evaluated all five optimizers on GPT-2 Small/OpenWebText at {0.5, 0.75, 1, 1.5, 2}\{0.5,\,0.75,\,1,\,1.5,\,2\} times their original learning-rate configurations, using seed 1337 and a 10k-step training budget. All other training settings remain unchanged. Among these tested learning rates, we select the one yielding the lowest final validation loss for each optimizer. The corresponding learning rates and losses are reported below.

OptimizerSelected base peak LRFinal validation loss ↓\downarrow
AdamW1.2×10−31.2\times10^{-3}3.1001
Muon4.5×10−44.5\times10^{-4}3.1076
MANO9×10−49\times10^{-4}3.1057
NormPre-G1.2×10−31.2\times10^{-3}3.0482
NormPre-L9×10−49\times10^{-4}3.0720

After each optimizer selects its best tested learning rate, the strongest baseline is AdamW at 3.1001. NormPre-G and NormPre-L reduce validation loss by approximately 0.0518 and 0.0281, respectively. Thus, the performance advantages persist after giving every optimizer the same number of learning-rate candidates and allowing each to select its own best configuration within the tested grid.

Given the available time and computational resources, extending larger-model training to Chinchilla-style token budgets, evaluating MoE architectures, expanding comparisons with recent optimizers, and conducting more extensive learning-rate searches on larger models remain directions for future work. We appreciate these suggestions and will make every effort to add further experiments as time and computational resources allow, while clarifying the scope of our evaluation and the remaining directions for future work in the revised manuscript.

Reviewer_NcciOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

We're doign LLM pretraining. There is a lot of interest right now on preconditioned matrix optimizers. This paper proposes normalising the update before applying preconditioning, then introduces two spectral preconditinoing methods and tests the optimizers on small scale models.

Critical strengths

The paper suggests that before applying preconditioning, one should normalize the update. At a high level, the motivation is to make LLM trainging faster.

I really like the local/global spectral noramliation section, which results in norm-G/L. I think this might be the most interesting part of the paper.

Critical weaknesses

lines 74 - 78 appear to contain the driving motivation for your work, but I'm not able to follow your intuition. Diagonal Gramm processing (normalization) prevents a single direction dominating the Gramm matrix? So we should normalize and then condition. This appear to be the argument, but as the main basis for the paper I feel that you should expound on this - I have to read it several times and I'm still not sure I understand the motivation fully.

The main thrust of the paper seems to be that normalizing and then preconditioning should help. And that Muon, which only preconditions, can be improved on. But I don't see a clean ablation of "just precondition", which would validate the paper's main idea more strongly. Muon is not a clean ablation because momentum details etc are changed.

Questions for authors

Can you explain the main intuition as to why normalizing before preconditioning should work? I failed to understand the paper's explanation.

Why did you not run the obvious ablation - just precondition without normalization?

Fig 2. Comic-sans? for real? Is that a way to check for AI-generated reviews?

4 comments

Thank you for your careful review and constructive feedback. We especially appreciate your positive assessment of the global and localized spectral constructions that lead to NormPre-G and NormPre-L. In response to your comments, we have revised the Introduction to clarify the two central questions and their connection to our framework (Q1), added direct ablations of input marginal normalization for both variants (Q1–Q2), and updated the font in Figure 2 (Q3). We address each of your concerns below.

Q1: The motivation in lines 74–78 is difficult to follow. Why should normalizing before preconditioning help?

A: Thank you for giving us the opportunity to clarify our motivation. We would like to emphasize that our motivation combines an insight into organizing geometric information hierarchically with a central question: how much spectral modification should an already useful normalized update undergo? Our optimization formulations give two concrete answers to this question, and the ablations provide empirical support for the resulting design choices, including marginal normalization and spectral preconditioning. We clarify on these points below.

1. Two core questions in the Introduction with distinct roles.

We first clarify the two core questions in the Introduction, as they are central to understanding our motivation!

  • Muon's full-Gram representation reveals its joint treatment of marginal scales and cross-row interactions, while normalization draws solely on the Gram diagonal to equalize row norms without altering pairwise angles. Together with the observation that normalization alone already provides a useful update, this suggests that whether we can organize geometric information hierarchically: first use marginal scales to construct a normalized update, then use its remaining directional correlations for refinement. This motivates our first question: “Can we construct a normalized update from scales and refine it via directional interactions?”

  • The second question is “Should the normalized update undergo a full-spectrum transformation, or serve as a reference for more targeted spectral modifications?” This question guides the derivation of concrete optimization methods within the framework. Since normalization already provides a useful update, we consider both global transformation through spectral-norm steepest descent and localized refinement that favors preserving this normalized reference. For the latter, regularized spectral steepest descent introduces a quadratic penalty on the Frobenius norm of the update, and the resulting problem is equivalent to finding the closest update to the normalized reference in Frobenius distance within the unit spectral-norm ball. We then derive localized spectral preconditioning through an optimal rank-constrained approximation to the resulting correction. These constructions lead to NormPre-G and NormPre-L, respectively.

  • We have revised the Introduction (Section 1) to make these two questions and their progression explicit.

2. The role of the illustrative example in lines 74–78.

We apologize for the confusion caused by this example. Our starting point is the full-Gram and diagonal-Gram representations of orthogonalization and normalization, together with the observation that normalization already provides a useful update. This leads to the two central questions above, which guide our systematic framework. The example in lines 74–78 provides supplementary geometric intuition by illustrating how marginal scales affect the full-Gram geometry. We explain it in more detail below. Its role is illustrative, while the practical benefit of normalizing before spectral preconditioning is assessed through the ablations summarized in the next point.

  • The example holds row directions fixed while varying row magnitudes: one row has norm 10410^4, while the others have norm 10−310^{-3}. As this scale disparity increases, the leading Gram eigenvector approaches the largest row's coordinate axis, even when the other rows are strongly correlated. The example illustrates that marginal scales can strongly affect the spectral geometry of the full Gram matrix even when pairwise directional relationships remain unchanged.

  • For nonzero rows, this distinction can be expressed as XX⊤=D(X)ΓD(X)XX^\top=D(X)\Gamma D(X), where D(X)D(X) contains the row norms, Ψ=D(X)−1X\Psi=D(X)^{-1}X, and Γ=ΨΨ⊤\Gamma=\Psi\Psi^\top. Marginal normalization preserves row directions and pairwise angles while removing their magnitude weighting; Γ\Gamma has unit diagonal and row cosine similarities off the diagonal. This suggests a way to organize the information hierarchically: first construct an update from marginal scales, then process the directional correlations retained in that update. These understandings can be found in Section 2 of the original manuscript.

3. Existing and newly added ablations support the design empirically. We bring together the following validation losses on GPT-2 Small/OpenWebText, using seed 1337 and a 10k-step training budget. All five configurations use five Newton–Schulz iterations (NS5).

ConfigurationRelaxed tangentMarginal normalizationGram matrixValidation loss ↓\downarrow
w/o tangent processing✗✓ΨΨ⊤\Psi\Psi^\top3.0945
w/o marginal normalization✓✗XX⊤XX^\top3.0893
Geometry from XX✓✓XX⊤XX^\top3.0900
NormPre-G✓✓ΨΨ⊤\Psi\Psi^\top3.0711
Muon✗✗MM⊤MM^\top3.1137

Here, MM denotes the momentum signal, XX the update before marginal normalization in each configuration, and Ψ=D(X)−1X\Psi=D(X)^{-1}X its normalized counterpart; X=MX=M when tangent processing is omitted. The notation is written in the active row orientation. The Gram matrix column identifies the geometry used to construct the spectral preconditioner. The “Geometry from XX” variant retains Ψ\Psi as the base update but constructs its preconditioner from XX⊤XX^\top.

  • Rows 1 and 3: ablations already reported in the original manuscript. Tables 6(a) and 7 in Appendix A.4 report the momentum-processing and interaction-geometry-source ablations, respectively, together with the original NormPre-G result of 3.0711. Removing relaxed tangent processing increases validation loss to 3.0945. Retaining the normalized base update Ψ\Psi but constructing its preconditioner from XX increases loss to 3.0900. The latter comparison directly supports constructing the preconditioner from the normalized interaction geometry.

  • Row 2: a new ablation added during the rebuttal period. This variant bypasses input marginal normalization while retaining relaxed tangent processing, five Newton–Schulz iterations, and the original final update-scaling rule. Its validation loss is 3.0893, approximately 0.0182 higher than the NormPre-G reference. This supports the contribution of the normalized input with the NS routine retained. We provide the corresponding NormPre-L control and further experimental details in our response to Q2 below.

  • Row 5: a result from an existing Muon run. For completeness, we report its validation loss of 3.1137 with seed 1337.

Together, these comparisons support the contributions of relaxed tangent processing, marginal normalization, and constructing the preconditioner from normalized interaction geometry under the tested setting.

We sincerely hope these clarifications address your concern!

Q2: The main thrust of the paper seems to be that normalizing and then preconditioning should help. And that Muon, which only preconditions, can be improved on. But I don't see a clean ablation of "just precondition", which would validate the paper's main idea more strongly. Muon is not a clean ablation because momentum details etc are changed.

A: Thank you for pointing this out. We apologize for overlooking this direct control in the original submission. We have now added it for both NormPre-G and NormPre-L. The NormPre-G comparison is summarized in our response to Q1 above; here we provide the corresponding component comparison for NormPre-L.

The following results are validation losses on GPT-2 Small/OpenWebText, using seed 1337 and a 10k-step training budget. The NormPre-L variants use localized spectral preconditioning, and Muon is included as a baseline for reference.

ConfigurationRelaxed tangentMarginal normalizationGram matrixValidation loss ↓\downarrow
w/o tangent processing✗✓ΨΨ⊤\Psi\Psi^\top3.1139
w/o marginal normalization✓✗XX⊤XX^\top3.1565
Geometry from XX✓✓XX⊤XX^\top3.1305
NormPre-L✓✓ΨΨ⊤\Psi\Psi^\top3.0864
Muon✗✗MM⊤MM^\top3.1137

The notation follows Q1: MM is the momentum signal, XX is the update before marginal normalization, and Ψ=D(X)−1X\Psi=D(X)^{-1}X. The Gram matrix column identifies the geometry used to construct the spectral preconditioner. The “Geometry from XX” variant retains Ψ\Psi as the base update while constructing its preconditioner from XX⊤XX^\top.

  • Rows 1 and 3: ablations already reported in the original manuscript. Tables 6(a) and 7 in Appendix A.4 report these results together with the original NormPre-L result of 3.0864. Removing relaxed tangent processing increases validation loss to 3.1139. Retaining the normalized base update but constructing its preconditioner from XX increases loss to 3.1305. These comparisons support the roles of relaxed tangent processing and normalized interaction geometry in localized refinement.
  • Row 2: a new ablation added during the rebuttal period. This variant bypasses input marginal normalization while retaining relaxed tangent processing, the same sketch configuration (r=32r=32, o=8o=8, p=1p=1) and spectral threshold, the learning-rate schedule, and the original final RMS reference/scaling rule. Only the input to localized spectral preconditioning changes from Ψ\Psi to XX. Its validation loss is 3.1565, approximately 0.0701 higher than the NormPre-L result reported in the original manuscript (3.0864). This directly supports the contribution of input marginal normalization with the localized spectral configuration held fixed.

  • Row 5: a result from an existing Muon run. As in Q1, we report its validation loss of 3.1137 with seed 1337 for completeness.

Together with the NormPre-G control above, these results provide direct empirical support for input marginal normalization in both variants under the tested setting. For NormPre-L, the component and geometry-source ablations support using normalization both to construct the reference update and to define the interaction geometry for subsequent spectral refinement.

Q3: “Fig 2. Comic-sans? for real? Is that a way to check for AI-generated reviews?”

A: Thank you for pointing this out, and we have replaced it with a conventional font!

Once again, thank you for your thoughtful comments and helpful suggestions! They have helped us explain our motivation more clearly, strengthen the experimental support for our design choices, and improve the presentation of the paper. We sincerely hope that these clarifications, additional experiments, and revisions have satisfactorily addressed your concerns!

AuthorsOpenReview ↗

Additional Responses

Dear Reviewer Ncci,

Thank you for your responses and for recognizing the promise of the additional results! We appreciate your feedback on the clarity of the exposition. We have now updated the revised manuscript, clarifying the motivation and method descriptions, incorporating the additional experiments, and making the supporting ablations easier to find from the main text.

Summary of changes in the revised manuscript.

  • Clearer motivation in Section 1: We make the two central questions and their progression explicit: how to construct a normalized update from marginal scales and refine it through directional interactions, and how much spectral modification to apply to this already useful update. We also clarify “interaction information” and the role of the normalized reference in NormPre-L.

  • Clearer implementation details in Sections 3.1–3.2: We add citations for tangent projection and explicitly state that all main NormPre-L experiments use the sketch-based implementation.

  • More precise discussion in Section 4.1: We distinguish NormPre-L’s modest improvement on LLaMA-130M from the larger gains in other settings and explicitly direct readers to the component ablations.

  • Validation perplexities in Appendix A.2: We report the validation perplexity results.

  • Expanded ablations in Appendix A.4: We present detailed ablations on momentum processing, marginal normalization, the source of interaction geometry, normalization orientation, and the spectral budget.

  • Additional learning-rate search in Appendix A.5: We compare all five optimizers using five learning-rate candidates each. Both NormPre variants retain lower final validation losses than all three baselines when each optimizer selects its best tested learning rate.

We would be very grateful if you could take another look at the revised manuscript and let us know whether the presentation is now clearer. In particular, we would warmly welcome suggestions on any passages or explanations that remain difficult to follow. Thank you again for helping us improve the paper, and we hope the revised version makes the framework easier to understand, evaluate, and apply!

Reviewer_X6T9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a new optimizer (mainly second-order) built on the Muon optimizer. They developed the NormPre optimizer, which works with spectral preconditioning, including localized and global spectral preconditioning. In the global case, their algorithm uses the Newton-Schulz iteration adopted by Muon and approximates the full-spectrum preconditioning. In the localized setting, they used exact top-r and sketched-based eigenvalue decompositions.

Critical strengths

Some strengths came to my attention during reading:

  1. First of all, they use topics in randomized linear algebra, which are very ubiquitous in theoretical computer science but are not well-established in LLMs despite their popularity.
  2. The paper has some theoretical foundations, such as a convergence theorem.
  3. Their proposal seems novel.
Critical weaknesses
  1. Unfortunately, the paper is not very well-written; it took me some time to follow and understand. Some parts need clarification, e.g., what does it mean by "via interaction information"?
  2. Lack of citations throughout the manuscript, e.g., tangent momentum, RMS scaling.
  3. More importantly, I guess the results are under-trained. For example, in Figure 1, we can't understand if your method is better than Muon for 10000 Training Steps. It may get worse in future steps. Particularly for bigger models like Llamma 1.3B, it needs 20000 training steps to verify your superior claim.
  4. Also, how did you tune the hyperparameters? E.g., the learning rates for all baselines and yours.
  5. Your superior performance claim is not clear to me is not clear as in Figure 1-b, the result of NormPre-L is on par with Muon.
Questions for authors

As well as the above questions, I have the following concerns as well.

  1. I think I didn't understand how your NormPre-G works better than Muon, as it approximates the full spectrum via Newton-Schulz iterations.
  2. I didn't understand whether NormPre-L in the main paper's experiments section is the exact top-r or the sketch-based counterpart.
10 comments

Thank you for your careful review and constructive suggestions. We appreciate your recognition of the novelty of our proposal and its theoretical foundations. In response, we have revised the Introduction to clarify the motivation and terminology (Q1), supplemented relevant citations (Q2), and added learning-rate searches and component ablations to strengthen the experimental evidence (Q4 and Q6). We also discuss the training budgets and performance differences across settings (Q3 and Q5), explain the distinction between NormPre-G and Muon (Q6), and further clarify the use of the sketch-based implementation for NormPre-L (Q7). We address each concern in detail below.

Q1: The presentation is difficult to follow in places. What does “via interaction information” mean?

A: Thank you for pointing this out! We would like to further clarify the terminology and motivation.

1. Meaning of “via interaction information”.

“Interaction information” refers to pairwise relationships between different rows or columns of a matrix update.

  • The utilization of interaction information is already apparent in Muon's full-Gram representation through the XX⊤XX^{\top} term. Let xi⊤x_i^{\top} denote the ii-th row of XX. The Gram entries satisfy (XX⊤)ii=∥xi∥22(XX^{\top})_{ii}=\lVert x_i\rVert_2^2 and (XX⊤)ij=⟨xi,xj⟩=∥xi∥2∥xj∥2cos⁡θij(XX^{\top})_{ij}=\langle x_i,x_j\rangle=\lVert x_i\rVert_2\lVert x_j\rVert_2\cos\theta_{ij} for i≠ji\ne j, where θij\theta_{ij} is the angle between rows ii and jj. The diagonal entries describe individual row scales, while the off-diagonal entries describe their directional relationships weighted by row magnitudes. Muon's transformation depends on the full Gram matrix and therefore already uses both marginal-scale and cross-row interaction information.

  • Marginal normalization constructs Ψ=D(X)−1X\Psi=D(X)^{-1}X by equalizing row norms while preserving pairwise angles. Consequently, Γ=ΨΨ⊤\Gamma=\Psi\Psi^\top has unit diagonal and Γij=cos⁡θij\Gamma_{ij}=\cos\theta_{ij} off the diagonal: the directional relationships remain, with the row-magnitude factors removed. Our framework first constructs this normalized update and then uses the spectrum and eigenvectors of Γ\Gamma to refine it. This is what we mean by refinement “via interaction information.”

These interpretations were given in the original manuscript in Section 2.1, following Equation (2) for the full Gram matrix, and Section 2.2, following Equation (3) for the normalized interaction matrix. We agree that they should be explained earlier. In the revised manuscript, after the paired full-Gram and diagonal-Gram representations in the Introduction, we have clarified that “cross-row interactions” refer to pairwise relationships between rows and added a reference to Section 2 for a more detailed explanation.

2. Two questions in the Introduction with distinct roles.

  • Muon's full-Gram representation reveals its joint treatment of marginal scales and cross-row interactions, while normalization draws solely on the Gram diagonal to equalize row norms without altering pairwise angles. Together with the observation that normalization alone already provides a useful update, this suggests that whether we can organize geometric information hierarchically: first use marginal scales to construct a normalized update, then use its remaining directional correlations for refinement. This motivates our first question: “Can we construct a normalized update from scales and refine it via directional interactions?”

  • The second question is “Should the normalized update undergo a full-spectrum transformation, or serve as a reference for more targeted spectral modifications?” This question guides the derivation of concrete optimization methods within the framework. Since normalization already provides a useful update, we consider both global transformation through spectral-norm steepest descent and localized refinement that favors preserving this normalized reference. For the latter, regularized spectral steepest descent introduces a quadratic penalty on the Frobenius norm of the update, and the resulting problem is equivalent to finding the closest update to the normalized reference in Frobenius distance within the unit spectral-norm ball. We then derive localized spectral preconditioning through an optimal rank-constrained approximation to the resulting correction. These constructions lead to NormPre-G and NormPre-L, respectively.

  • We have revised the Introduction (Section 1) to make these two questions and their progression explicit.

Q2: Citations are missing, for example for tangent momentum and RMS scaling.

A: Thank you for suggesting more citations. In the revised manuscript, we have added citations to Heo et al. (2021) and Gu and Xie (2026) in Section 3.1 to acknowledge prior work on tangent projection of momentum-based updates. We also clarify that our relaxed formulation recovers strict tangent projection when the corresponding weight vector has unit norm. In addition, we have supplemented the references for the Pile-related data Biderman et al. (2023) and preprocessing descriptions in the appendix. Regarding RMS scaling, Section 3.1 of the original manuscript already cites Liu et al. (2025) and explicitly attributes the target update RMS of 0.2 to the Muon convention.

Q3: The models may be undertrained, and the advantage may not persist. In particular, LLaMA-1.3B should be trained for 20,000 steps.

A: Thank you for raising this concern. We are making every effort to supplement the evaluation with 20k-step training experiments on LLaMA-1.3B. However, given the limited time available for the rebuttal, completed results are not yet available. Our existing experiments already provide empirical evidence for NormPre’s effectiveness across multiple architectures, model sizes, and datasets, including GPT-2 Small on OpenWebText, LLaMA on C4, and Qwen3 on Pile. In particular, the reported training budgets for GPT-2 Small and LLaMA-130M exceed the commonly used 20N20N reference. We emphasize that both NormPre variants achieve lower validation losses than the evaluated baselines under the reported training budgets. The additional learning-rate searches and component ablations presented in our responses to Q4 and Q6 further support these findings. These results support the effectiveness of our approach within the evaluated settings. Longer training on larger models will provide further evidence on whether the advantages persist as training progresses, and we are working to extend the evaluation in this direction.

Q4: How were the hyperparameters tuned, especially the learning rates of all baselines and the proposed methods?

A: Thanks for your question. We clarify the learning-rate protocol used in our main experiments and provide additional search results below.

1. Learning-rate choices in the main experiments. Different optimizers may have different optimal learning rates. Our main experiments use predefined learning-rate schedules together with established update-scaling conventions, including RMS matching to bring matrix-update magnitudes into a range comparable to AdamW. The optimizer-specific configurations are detailed in Appendix A.1. This provides a practical basis for comparing optimization behavior without separate optimizer-specific learning-rate tuning. We further complement this comparison with the additional search below, allowing each optimizer to select its best learning rate within the tested grid.

2. Additional learning-rate searches confirm that the advantages persist within the tested range. We evaluated all five optimizers on GPT-2 Small/OpenWebText at {0.5, 0.75, 1, 1.5, 2}\{0.5,\,0.75,\,1,\,1.5,\,2\} times their original learning-rate configurations, using seed 1337 and a 10k-step training budget. All other training settings remain unchanged. Among these tested learning rates, we select the one yielding the lowest final validation loss for each optimizer. The corresponding learning rates and losses are reported below.

OptimizerSelected base peak LRFinal validation loss ↓\downarrow
AdamW1.2×10−31.2\times10^{-3}3.1001
Muon4.5×10−44.5\times10^{-4}3.1076
MANO9×10−49\times10^{-4}3.1057
NormPre-G1.2×10−31.2\times10^{-3}3.0482
NormPre-L9×10−49\times10^{-4}3.0720

After each optimizer selects its best tested learning rate, the strongest baseline is AdamW at 3.1001. NormPre-G and NormPre-L reduce validation loss by approximately 0.0518 and 0.0281, respectively. Thus, the performance advantages persist after giving every optimizer the same number of learning-rate candidates and allowing each to select its own best configuration within the tested grid.

Q5: Your superior performance claim is not clear to me is not clear as in Figure 1-b, the result of NormPre-L is on par with Muon.

A: Thank you for pointing this out. We agree that NormPre-L's improvement over Muon in Figure 1(b) is modest. We have explicitly clarified this distinction in Section 4.1 of the revised manuscript.

1. LLaMA-130M/C4: different improvement magnitudes for the two variants. Table 1 reports validation losses of 3.1019 for Muon, 3.0737 for NormPre-G, and 3.0930 for NormPre-L. Both variants achieve lower validation loss, with reductions of 0.0282 for NormPre-G and 0.0089 for NormPre-L relative to Muon. Accordingly, in the revised manuscript, we will describe NormPre-L in this setting as comparable to Muon, with a small numerical improvement, while distinguishing the larger improvement achieved by NormPre-G in the discussion of Figure 1(b) in Section 4.1.

2. Larger improvements in other evaluated settings. Table 1 shows more pronounced improvements for both variants in other settings. The following examples compare against the strongest baseline in each setting, which is MANO in both cases:

SettingStrongest baseline: validation lossNormPre-G: validation loss (reduction)NormPre-L: validation loss (reduction)
LLaMA-1.3B/C4MANO: 2.89632.8571 (0.0392)2.8662 (0.0301)
Qwen3-1.7B/PileMANO: 2.62052.5868 (0.0337)2.5897 (0.0308)

These results support the performance benefits of both variants under the evaluated training budgets, while showing that the magnitude of the improvement varies across settings.

Q6: I think I didn't understand how your NormPre-G works better than Muon, as it approximates the full spectrum via Newton-Schulz iterations.

A: Thanks for your question! We clarify the algorithmic distinction and bring together the relevant existing and newly added experiments below.

1. The same Newton–Schulz routine can produce different update directions. Muon applies Newton–Schulz iterations to its momentum matrix, while NormPre-G applies them to the marginally normalized update obtained after relaxed tangent processing. Even for a fixed input signal XX, row normalization can change its singular vectors, i.e., in general, msign⁡(D(X)−1X)≠msign⁡(X).\operatorname{msign}\left(D(X)^{-1}X\right)\neq\operatorname{msign}(X). Here, D(X)D(X) contains the row norms of XX. Mapping singular values to one still leaves the resulting update dependent on the singular vectors of the input. Thus, the distinction persists even with exact orthogonalization. This is consistent with our hierarchical construction insight in the Introduction: we first remove the marginal-scale weighting and then perform spectral refinement of the normalized update.

2. Existing and new ablations support the contributions of these design choices. We bring together five configurations below to make their differences and results easier to compare. The following results are validation losses on GPT-2 Small/OpenWebText, with seed 1337 and a 10k-step training budget, and these configurations use five Newton–Schulz iterations (NS5).

ConfigurationRelaxed tangentMarginal normalizationGram matrixValidation loss ↓\downarrow
w/o tangent processing✗✓ΨΨ⊤\Psi\Psi^\top3.0945
w/o marginal normalization✓✗XX⊤XX^\top3.0893
Geometry from XX✓✓XX⊤XX^\top3.0900
NormPre-G✓✓ΨΨ⊤\Psi\Psi^\top3.0711
Muon✗✗MM⊤MM^\top3.1137

Here, MM denotes the momentum signal, XX the update before marginal normalization in each configuration, and Ψ=D(X)−1X\Psi=D(X)^{-1}X its normalized counterpart; X=MX=M when tangent processing is omitted. The Gram matrix column identifies the geometry used to construct the spectral preconditioner. The “Geometry from XX” variant retains Ψ\Psi as the base update but constructs its preconditioner from XX⊤XX^\top.

  • Rows 1 and 3: ablations already reported in the original manuscript. Tables 6(a) and 7 in Appendix A.4 report the momentum-processing and interaction-geometry-source ablations, respectively. Relative to the original NormPre-G result of 3.0711, removing relaxed tangent processing increases validation loss to 3.0945, while retaining the normalized base update Ψ\Psi but constructing its preconditioner from XX increases loss to 3.0900. These results support the contributions of relaxed tangent processing and constructing the preconditioner from normalized interaction geometry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.