acceptodds
Under review as a conference paper at ICLR 2027

Normalize your normalized layers

Abstract

Normalization, both in the forward pass and in the optimizer, is a central component of modern deep learning. But where and how should we normalize? We study this question from a purely optimization-based perspective and find that the answer prescribes a placement of normalization layers consistent with state-of-the-art architectures. Exploiting the scale invariance induced by normalization layers, we develop NorSODA, a weight-normalized method that achieves noise-robust accelerated rates by carefully incorporating weight decay. Our key technical result is an online-to-batch conversion that wraps any no-regret algorithm while only making assumptions on the sphere. Interestingly, the result extends to non-Euclidean geometries, yielding a close cousin of MuonH that removes the need to normalize the update direction.

Primary area:
optimization
Keywords:
Deep learning, optimization, scale-invariance, weight normalization, non-Euclidean

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MSwKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when scaling the weights keeps the same hidden neurons in different substructures with Normalizations, such as MLPs and Attentions. Under this condition, weight decay can be ignored to simplify the training settings. The authors also discussed the convergence rates under this setting with detailed mathematical analyses. Experiments are also conducted to identify the optimization methods.

Critical strengths

The mathematical analyses are detailed and meaningful.

The article is well-organized and easy to follow.

Critical weaknesses

Confusion on the title. I did not get what the title tells us. In the main context, I mainly see the discussion on the authors conduct researches on weight decay. I did not get what is normalized from the main context related to the title.

Limited experiments. In experiments, this paper shows the acceleration using NorSODA. However, the improvement is limited, and the final performance is not recorded. It is better if the authors test their methods on additional tasks like image-classifications and so on.

Questions for authors
  1. Can the authors further explain what the meaning of the title is, and its relationship to the research topic of this paper?
  2. What is the final performance of Figure 2? Are there other indexes which can test the performance except the validation loss? Note that the lower loss does not always obtain the better performance.
  3. Can the method applied on the ViTs for image-classification tasks? What are the training/validation accuracy? Will we get the similar conclusions as the current experiments?
2 comments

We thank the reviewer for the positive assessment. Following the comments, we have clarified the title and added a new image-classification experiment on CIFAR-10.

Confusion on the title

The title is meant as a play on words, conveying that fact that normalization of weight can safely be used by the optimizer for weights that are followed by normalization layer. We have change the title to "Normalize your normalized layers: acceleration under scale-invariance" to specify more precisely the main theoretical contribution.

Limited experiments

We confirm with new experiments that the improvements of NorSODA carry over to image classification on CIFAR-10 without any retuning. Specifically, we train a convolutional network with batch normalization on CIFAR-10 (Appendix E.3), which differs from the nanoGPT experiments in almost every respect, as it uses:

  • images instead of text
  • convolutions instead of attention and MLPs
  • batch normalization instead of RMSNorm
  • and multiple epochs instead of a single pass.

The results are in Figure 6, with an accompanying table that reports the final test accuracy at every horizon.

Questions

Can the authors further explain what the meaning of the title is, and its relationship to the research topic of this paper?

See the response above regarding the title.

What is the final performance of Figure 2? Are there other indexes which can test the performance except the validation loss? Note that the lower loss does not always obtain the better performance.

The final performance of the two methods is stated in the plot (see the zoomed-in panel). For pre-training of LLMs we are not sure of a better measure. However, for the new CIFAR-10 experiments we are reporting the test accuracy (Figure 6).

Can the method applied on the ViTs for image-classification tasks? What are the training/validation accuracy? Will we get the similar conclusions as the current experiments?

Yes, we obtain similar results on CIFAR-10 without any retuning as mentioned above.

Thanks for your detailed reply. All my concerns are addressed. I will maintain my score.

Reviewer_HKE6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when weight normalization can be combined with acceleration without breaking convergence.

The main contributions are:

  • Identifying that appropriately placed normalization makes certain weight groups scale-invariant. Post-sublayer normalization and QK-normalization make attention weights and MLP weights with homogeneous activations scale-invariant. They also identify exceptions, e.g., output layers and SwiGLU gate projection.
  • Showing that weight-normalized steepest descent converges without weight decay under the stated assumptions.
  • NorSODA: instantiating the online-to-batch conversion with optimistic dual averaging yields noise-robust accelerated rates in finite-p geometries.

Validation supports theoretical proofs using a 12-layer, width-768 transformer trained on FineWeb10B. The experiments show that dropping MuonH's update normalization preserves performance, and a practical NorSODA variant achieves lower validation loss than MuonH across training horizons.

Critical strengths
  • Theoretical contribution for combining weight normalization with acceleration. The normalized online-to-batch conversion yields noise-robust accelerated rates while requiring assumptions only at normalized iterates. This addresses the difficulty of analyzing accelerated normalized methods and could support the analysis of methods beyond NorSODA.

  • Clear connection betweent their theory and practical optimizer design. The experiments show that removing MuonH's update-direction normalization preserves performance. A practical NorSODA variant achieves modest validation loss across training horizons.

Critical weaknesses
  • The theory doesn't apply directly to the optimizer in experiments. The accelerated guarantee requires certain update rules, parameter schedules, and objective assumptions. However, the practical NorSODA variant adopt different choices and the paper doesn't show that the required assumptions hold during training. Therefore, the theory doesn't prove acceleration for the tested method.

  • Limited generality and lack of controlled ablations. The experiment uses a single transformer configuration and dataset. The paper does not show robustness across model sizes, context lengths, batch sizes, architectures, or datasets. Also, NorSODA changes several parts of MuonH, such as primal averaging, learning rate scheduling, and output anchor. These limitations make it hard to determine which changes produced the gains and whether the proposed framework yields a robust advantage.

Questions for authors
  • Please clarify the theoretical guarantees. Which guarantees apply to the practical NorSODA variant in the experiments? Please explain how the required assumptions are justified in this setting.

  • Do you have results beyond the single model and dataset that demonstrate that the gains persist across different training settings?

1 comment

We thank the reviewer for the helpful comments. They prompted two additional experimental results that we believe address the concerns, namely an empirical verification of the assumptions and a test of the robustness of the results in a different experimental setting.

The paper doesn't show that the required assumptions hold during training.

We now measure both assumptions along the NorSODA nanoGPT trajectory (Appendix E.2). Both settle after a short warmup of roughly the first 20% of training. From then on:

  • the star-convexity ratio ζk\zeta_k stays above 1.
  • the co-coercivity assumption holds with a constant L^1k\hat L_1^k that stays within a factor of 2 of its median.
  • the Bregman gap between consecutive iterates is positive at every sampled pair, including during warmup.

This observation is also consistent with the practical design of NorSODA, whose offset Ω\Omega in effect warms up the averaging schedule λk\lambda_k.

Limited generality and lack of controlled ablations.

The existing nanoGPT experiments already partially ablate the changes relative to MuonH:

  • dropping the update normalization from MuonH does not change performance (Figure 2a).
  • retuning MuonH's learning rate for longer horizons with the Marin scaling rule only partially closes the gap with NorSODA (Figure 4a).

Thus, the main driver of the improvement in NorSODA is the scheduling of λk\lambda_k.

Regardingly generality, we confirm with new experiments that the improvements of NorSODA carry over to CIFAR-10 without any retuning. Specifically, we train a convolutional network with batch normalization on CIFAR-10(Appendix E.3), which differs from the nanoGPT experiments in almost every respect in that the setup uses:

  • images instead of text
  • convolutions instead of attention and MLPs
  • batch normalization instead of RMSNorm
  • and multiple epochs instead of a single pass.

In order to put the ablation in one place, we have added Figure 6, which includes a table that adds one change at a time, going from MuonH to NorSODA.

Questions

Which guarantees apply to the practical NorSODA variant, and how are the assumptions justified?

As mentioned above we now provide empirical support of the assumptions after a warmup phase (Appendix E.2), which also justify the warmup of the λk\lambda_k schedule in the practical NorSODA variant.

Results beyond the single model and dataset?

Yes, see the new CIFAR-10 experiments mentioned above (Appendix E.3).

Reviewer_e9VFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies optimization of scale-invariant parameter groups induced by architectural normalization. It develops a normalized online-to-batch conversion, instantiates it with optimistic dual averaging as NorSODA, and connects its large-p limit to MuonH. Experiments on a sandwich-normalized language model report modest, consistent improvements over MuonH. The conversion is elegant, but the main accelerated theorem has an uncontrolled initialization term, and the practical optimizer substantially departs from the analyzed algorithm.

Critical strengths

This paper presents a theoretical basis for the combination of Nesterov acceleration and weight normalization in MuonH. Elegant normalized online-to-batch conversion. Lemma 5.3 and Appendix D.2 use radial gradient orthogonality to replace a normalized average by its unnormalized precursor inside an inner product. This preserves the telescoping structure while querying gradients only at normalized iterates.

Critical weaknesses

Main accelerated theorem lacks initialization control. Improvements are not attributed through component ablations. Exact invariance is idealized relative to implemented normalization. Appendix A.1 explicitly ignores stabilizing ε, whereas the main architectural motivation treats scale invariance as exact.

Questions for authors

Which practical component accounts for the improvement over static-direction-scaling MuonH when the schedule, primal averaging, and head anchor are separately controlled? How stable are the 2× and 3× improvements across seeds and matched horizon-specific tuning, and what is the measured runtime benefit?

1 comment

We thank the reviewer for the careful reading and for appreciating the normalized online-to-batch conversion. The comments led us to fix the initialization in the accelerated theorem and to add a new CIFAR-10 experiment that ablates the changes from MuonH one at a time.

Main accelerated theorem lacks initialization control.

Thank you for catching this. The theorem now requires the initialization to lie on the sphere, z0∈Sω,ρz^0\in\mathcal S_{\omega,\rho}, so that xˉ0=z0\bar x^0=z^0. We also noticed that the Hölder-smoothness condition must hold from k=0k=0, which we fixed. Together, these bound the initial gradient by ∥∇f(xˉ0)∥∗≲LνRpν\lVert\nabla f(\bar x^0)\rVert_\ast\lesssim L_\nu R_p^\nu. As a result, the term from the optimistic step at k=0k=0 is at most a constant times the leading term D/ηD/\eta, and the rate is unchanged.

Improvements are not attributed through component ablations.

The existing nanoGPT experiments already partially ablate the changes relative to MuonH:

  • dropping the update normalization from MuonH does not change performance (Figure 2a)
  • retuning MuonH's learning rate for longer horizons with the Marin scaling rule only partially closes the gap with NorSODA (Figure 4a)

Thus, the main driver of the improvement in NorSODA is the scheduling of λk\lambda_k.

This is further supported by the new CIFAR-10 experiment (Appendix E.3, Figure 6) where we change MuonH into NorSODA one modification at a time. Dropping the update normalization leaves accuracy unchanged. Subsequently adding the primal averaging λk=2/(k+Ω)\lambda_k=2/(k+\Omega) is then what leads to the final improvement in accuracy across horizons.

Exact invariance is idealized relative to implemented normalization. Appendix A.1 explicitly ignores stabilizing ε\varepsilon

In practice, ε\varepsilon is tiny, so the statements still holds approximately. Specifically, with ε\varepsilon, rescaling the weights by aa multiplies the RMSNorm output by 1+O((ε/σ2)∣1−a−2∣)1+O\big((\varepsilon/\sigma^2)|1-a^{-2}|\big). As a result, the radial derivative ⟨∇Wf,W⟩\langle\nabla_W f,W\rangle is damped by ε/(σ2+ε)\varepsilon/(\sigma^2+\varepsilon), and f(xˉ)=f(x)f(\bar x)=f(x) holds up to the same relative factor. Measuring σ2\sigma^2 at initialization, we find ε/σ2≤2.4⋅10−3\varepsilon/\sigma^2\le 2.4\cdot10^{-3} at every normalization acting on hidden matrices (with ε=10−6\varepsilon=10^{-6}). Both identities therefore hold up to a negligible error.

Questions

Which practical component accounts for the improvement over static-direction-scaling MuonH when the schedule, primal averaging, and head anchor are separately controlled?

See the points about ablation above.

How stable are the 2× and 3× improvements across seeds and matched horizon-specific tuning?

  • Seeds: at 1×1\times, NorSODA improves on MuonH on every one of the three seeds (table in Figure 3). We have not repeated the 2×2\times and 3×3\times runs over seeds, but the gap there is even larger. The new CIFAR-10 results average over 10 seeds at every horizon and shows that the gap remains (Figure 6).
  • Horizon-specific tuning: please see Figure 4a which shows that retuning MuonH's learning rate for 2×2\times and 3×3\times with the Marin scaling rule only partially closes the gap.

What is the measured runtime benefit?

The saving from dropping the update direction normalization is one Frobenius norm per hidden matrix per step, which is negligible next to the Newton–Schulz iterations. Our point is instead about simplicity. We believe it is valuable to reduce implementation complexity since the operation can be dropped without sacrificing anything (as show by Figure 2a and Figure 6).

Comments on the paper

1 comment

We have added the following experiments (all marked in orange in the manuscript):

  • More seeds at 2x and 3x for nanoGPT in Figure 3, as asked by Reviewer e9VF (NorSODA is still consistently better).
  • More measurements to verify the assumptions, as originally asked by Reviewer HKE6, which continue to support that the assumptions hold after a short warmup period (now in the main text, Figure 4).
  • In addition to the new CNN experiments on CIFAR-10 we have added ResNet-18 on both CIFAR-10 and CIFAR-100 (Figure 7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.