acceptodds
Under review as a conference paper at ICLR 2027

Breaking Balance for Speed: Rescaling Normalization at Initialization

Abstract

Gradient flow in homogeneous neural networks is known to preserve a conservation law between adjacent weight matrices, which follows from a known rescaling symmetry that has strong implications for the theoretical analysis of deep learning dynamics. Normalization breaks this symmetry, yet, as we show, its affine parameters introduce another conservation law with the downstream weights that determines how quickly the two parameter groups learn relative to one another. Contemporary initialization schemes bring them out of balance, which typically induces significant benefits for generalization. Leveraging this insight, we deliberately rescale the initialization to move normalization and weight parameters further out of balance and control their relative learning speeds. Across convolutional networks and Transformers, we find that this can accelerate optimization and yield consistent performance gains over standard and balanced initializations.

Primary area:
optimization
Keywords:
normalization layers, conservation laws, training dynamics, layer balance, feature learning, initialization, implicit bias, LayerNorm, BatchNorm

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_533jOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers conservation laws induced by symmetry in networks. In prior work, it has been observed that the scale-symmetry of linear-ReLU stacks leads to a conservation of additive gaps between layer norms through gradient flow, which seems to be a direct consequence of applying a Noether-like argument to gradient flow on the objective function (I am not an expert in this area, but this sounds plausible). The submission now asks the question whether such an analysis would still applicable when normalization layers were added, which break the ReLU-scale-symmetry (positive homogeneity). What the authors find is a symmetry between the rescaling parameter and the weight norm, which might look rather inconsequential at first glance. However, a more careful analysis shows that the sum of the squared norms of weights and the two parameters is strictly conserved under gradient flow, while, semantically, the product of the two scaling parameters (overall weight norm and learned scaling parameter) determine the output. This leads to a a drift in scaling when an initial imbalance is present at initialization, which affects the numerical behavior. The authors continue to show analytically and experimentally that introducing a certain amount of imbalance can help in accelerating convergence as this controls the preconditioning of the different layers during gradient decent (I must admit that I was not fully able to follow the argument, but it seems obvious that this must have an effect on training dynamics). Overall, the paper adds to the understanding of the training dynamics of normalized networks, this time with focus on the learnable parameters, which, to my knowledge before reading the paper, have received limited attention and looked more like an "innocent" add-on. For this reason, I would believe that the paper could form the foundation of some important piece of progress in further elucidating training dynamics of DNNs.

Critical strengths

I think that this is a strong paper and, if there are no problems with related work or technical details (which I have limited ability to check), that it would be a strong candidate for inclusion in ICLR. Particular strengths include:

  • A very simple and elegant formulation / derivation from a connection of conserved quantities to symmetries of potentials.
  • A clear case of conservation and rescaling following different invariant equations, which strongly suggests emergent dynamical phenomena in (small time step) gradient descent.
  • Experimental evidence that show impact on convergence in practice, validating the theoretical predictions.
Critical weaknesses

I did not find any substantial weaknesses, but there is a bit of room for improvement:

  • The paper is well-written and not difficult to read as such, but it is quite technically-leaning and assumes that the reader is familiar with advanced concepts such as generalized applications of Noether's theorem (beyond Lagrangian actions in classical physics). I would suggest two aspects to consider in a revision:

    • Add a bit of explanation/recap of the technical concepts and terms (Symmetry and conservation laws, the notion of balance and imbalance, etc.), maybe in a short appendix to make the paper more accessible to a broader audience.
    • While technical rigor is commendable, the paper formulates its results mostly in technical language only. It would be good to give some short intuition / application examples to explain how the mechanisms work in practice and what the intuitive ideas behind them are. For example, what kind of loss functions meet the Polyak-Łojasiewicz inequality? For someone seeing this notion for the first time, this is not immediately obvious. Similarly, it would be good to qualitatively/intuitively describe what balance/imbalance means (Appendix B is again very technical in its formulation) and how it affects learning dynamics, in broad strokes, in the context of the symmetries discussed in this paper.
  • On a more technical level, using complex transformer architectures in the empirical studies makes it difficult to see how realistic the formal model is for a complex architecture; even for the ResNet, there might be a few non-trivial aspects (such as the residual addition potentially affecting the scale-symmetry, depending on where it is placed wrt. the normalization). A short discussion might help. Maybe even some experiments with a simplistic 5-layer MLP on CIFAR-10 or the similar might be useful to validate the ideas more directly (in addition to ResNet and Transformers). Nonetheless, showing the effect in a realistic setting is of course also valuable and as such, a strength.

Questions for authors

I do not have any major question, but it would be nice if you could restate in simple, non-technical terms how the convergence behavior is/can be influenced by tweaking initialization balance.

3 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your suggestions/questions in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them. We provide a more intuitive explanation of the symmetries, balance and convergence results in our revised manuscript in the appendix and in the main text (highlighted in blue). We have also added an experiment with a 5-layer MLP on CIFAR-10 to better showcase the highlighted mechanisms in a simple setting in Appendix F pages 26-27.

Add a bit of explanation/recap of the technical concepts and terms (Symmetry and conservation laws, the notion of balance and imbalance, etc.), maybe in a short appendix to make the paper more accessible to a broader audience. … For example, what kind of loss functions meet the Polyak-Łojasiewicz inequality? For someone seeing this notion for the first time, this is not immediately obvious.

In our revised manuscript, we provide a more intuitive description highlighting the background of technical concepts such as symmetries and their connection to conserved quantities. As follows, we discuss imbalance and how it interacts with convergence speed, which is formally described using the PL inequality.

Symmetry and Balance. In neural networks, a symmetry is a parameter transformation that leaves the represented function unchanged. For example, in a ReLU network, multiplying the incoming weights of a hidden unit by λ>0\lambda > 0 and dividing its outgoing weights by λ\lambda leaves the network function unchanged. Because the rescaling leaves the loss unchanged, gradient flow does not push the parameters along that rescaling direction. As a result, the relative scale between the two parameter groups remains constant over time, yielding a conserved quantity. This is the basic intuition behind the connection to Noether's theorem. In the classical setting, the conserved quantity measures the difference between the squared scales of the two parameter groups. Our main observation is that normalization relocates this structure to the Norm--Weight interface, where the downstream weight and normalization affine parameters share a conserved quantity CjC_j. We call the interface j∈[d]j \in [d], balanced when Cj=0C_j = 0 and imbalanced when Cj≠0C_j \neq 0; ∣Cj∣|C_j| measures the degree of imbalance, while its sign indicates which side carries more scale. Although these different parameterizations can represent the same function, they induce different effective optimization dynamics. Thus, changing the balance at initialization provides a function preserving way to control these dynamics, and in the setting analyzed in Section 4.2 and in Appendix C.1, greater imbalance can lead to faster convergence.

PL Inequality and Convergence. To make this convergence statement precise, we use the Polyak--Łojasiewicz (PL) inequality. Intuitively, PL requires that when the loss is far from its minimum, its gradient cannot be arbitrarily small. Strongly convex objectives satisfy PL, as do least-squares problems under suitable conditions, while PL is weaker than strong convexity and can also hold for some non-convex objectives. In our setting, the imbalance controls the effective update scale, and the PL condition converts this larger update scale into a faster guaranteed decrease of the loss. We therefore use PL as a sufficient condition for formalizing the connection between initialization imbalance and convergence speed.

On a more technical level, using complex transformer architectures in the empirical studies makes it difficult to see how realistic the formal model is for a complex architecture; even for the ResNet, there might be a few non-trivial aspects (such as the residual addition potentially affecting the scale-symmetry, depending on where it is placed wrt. the normalization). A short discussion might help.

Thank you for highlighting this point. We agree that, in a full ResNet or Transformer, architectural components such as residual connections can make the connection to the full model less immediate. Our analysis therefore focuses on the local Norm--Weight interfaces, where a normalization affine parameter is followed by a positively homogeneous downstream map. These interfaces occur repeatedly in both architectures and are the locations where the rescaling symmetry applies directly.

For ResNets, for example, we focus on interfaces such as BN1→conv2\texttt{BN}_1\rightarrow\texttt{conv}_2 in ResNet-20, where the Norm--Weight structure is directly present and the function-preserving rescaling applies exactly. More generally, our analysis applies to the normalization-to-weight interfaces (illustrated in Figure 2 on page 4) throughout the network that have this local structure, allowing us to study the same symmetry and effective dynamics within the full architecture.

Maybe even some experiments with a simplistic 5-layer MLP on CIFAR-10 or the similar might be useful to validate the ideas more directly (in addition to ResNet and Transformers).

Thank you for the great suggestion. We have included an experiment with a 5-layer MLP on CIFAR-10 in Appendix F, observing the same effects in this simpler architecture. λ\lambda-rescaling changes the optimization behaviour across the four Norm--Weight interfaces without changing the overall function. The full sweep shows that the choice of initialization balance has a clear effect on both training loss and test accuracy, providing a controlled setting where the influence of imbalance can be studied without the additional architectural complexity of ResNets or Transformers.

I do not have any major question, but it would be nice if you could restate in simple, non-technical terms how the convergence behavior is/can be influenced by tweaking initialization balance.

By changing the initialization balance redistributes the scale between the normalization parameters and the downstream weights without changing the function represented by the network. This changes how quickly the two parameter groups respond during optimization and therefore changes the effective update of their shared parameterization. As analyzed in Section 4.2, a more imbalanced initialization can produce a larger effective update scale and hence faster convergence. We have addressed this in more detail in Appendix G.

Dear authors,

thanks a lot for the detailed feedback and explanation. This confirms my rough intuitive understanding, and thanks to your explanation, I now finally understand how the change of balance can be used to tune per-layer learning rates; this is really nice. In my experience, divergences in effective (after accounting for symmetry) learning rates do indeed have the potential of greatly affecting how normalization layers affect training dynamics and, in extreme settings (e.g., very deep networks), even training success as such. As far as I know, the influence of the additional learned scaling/shifting parameters has not yet been understood explicitly.

This paper shows very clearly how tempering with the additional parameters leads to adjustments of per-layer effective learning rates. While this might be less relevant for training schemes that use weight- or layer-adpative learning rates (such as 2nd order methods or ADAM-variants), which might compensate for such effects to some degree (unlike plain SGD), the insight is still very valuable in my opinion.

Update to my review: I would thus keep my rating and positive recommendation - I like the paper. Adding the base-line experiment and improving the writing to make the analysis more accessible is of course highly appreciated.

Reviewer_eCdSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how the conservation law changes in networks with normalization layers and derives the balance equation for this setup. The authors show that an adapted conservation law shifts to the interface between the normalization layer and the model weights. They show that the magnitude imbalance between the normalization layer and model weights controls relative and overall learning speed. They show that imbalance is already present in close to standard setups (DeiT-tiny, ResNet20 and ResNet50, GPT2-small) trained with SGD + momentum or AdamW and that artificially increasing the imbalance via parameter rescaling at initialization helps training speed as well as generalization. For weight decay they show that it reduces the imbalance exponentially towards 0, with rate depending on decay strength.

Critical strengths
  • The paper derives novel insights into the training dynamics of neural networks, close to setups used in practice. Gaining these insights helps the community better improve training dynamics from first principles and hence can be very impactful. Particularly, since the authors derive a simple intervention that improves overall learning speed and generalization in their experiments
  • The paper is well written, and the experimental setups are carefully designed and close to setups in practice. Additionally the authors validate the theory in practice and run experiments on 3 seeds including standard deviation for generalization results.
Critical weaknesses
  • The improvements in the ResNet50/ImageNet experiment seems small and might reflect run to run variance.
  • For the Figure 5 and Appendix Figures 7, 9, 10, 11, it is not clear if the plot is based on a single run or the average across runs and blocks. Also since the plot 5b shows averages across all blocks, standard deviation would be interesting to see as well.
  • The additional results in Appendix D would benefit from a more in-depth discussion and maybe also adding a summary sentence in the main body about the ResNet50 results on ImageNet.
Questions for authors
  • What is the difference between the experimental setup of Figure 6b and Figure 12?
1 comment

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your suggestions/questions in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

The improvements in the ResNet50/ImageNet experiment seems small and might reflect run to run variance

We have added more seeds for the ResNet/ImageNet experiment and now have over three random seeds and report the mean and standard deviation in Table 10 on Page 29. The rescaled initialization with λ=1.75\lambda=1.75 reaches 76.04±0.05%76.04\pm0.05\% test accuracy, compared with 75.87±0.04%75.87\pm0.04\% for the standard initialization (λ=1\lambda = 1). Accordingly, the improvement is significant.

For the Figure 5 and Appendix Figures 7, 9, 10, 11, it is not clear if the plot is based on a single run or the average across runs and blocks. Also since the plot 5b shows averages across all blocks, standard deviation would be interesting to see as well.

Thank you for pointing this out. We have changed the corresponding figures in the main text on page 9 for clarification. The plots are based on a single representative run. Within each run, each plotted curve is averaged over all corresponding Norm--Weight interfaces across the network (Transformer blocks for DeiT-Tiny and the corresponding layers/blocks for ResNet). We have also updated Figure 5b for DeiT-Tiny on CIFAR-100. At the LN1→qkv\mathrm{LN}_1\rightarrow\texttt{qkv} interface, the relative parameter speed is averaged over all 12 Transformer blocks and the shaded region now reports the standard deviations.

The additional results in Appendix D would benefit from a more in-depth discussion and maybe also adding a summary sentence in the main body about the ResNet50 results on ImageNet.

Thank you for this suggestion. We have expanded the discussion in Appendix D to more clearly interpret the conserved quantity trajectories and relative parameter speed plots for both ResNet-20 and ResNet-50. In particular, for ResNet-20 we now explain that the λ=5\lambda = 5 initialization produces a substantially more positive imbalance than the standard initialization, and that this is accompanied by a reversal in the relative parameter speeds: the BatchNorm gain γ\gamma moves faster relative to its scale than the downstream convolutional weight WW, whereas the standard initialization shows the opposite ordering. We then connect this behaviour to the effective dynamics in Section 4.2, where changing the Norm--Weight scale allocation changes the two components of the effective preconditioner. We also note that ResNet-50 shows the same behaviour across its BN$\rightarrow$Conv interfaces under λ=1.75\lambda = 1.75, while giving an improvement in ImageNet performance. A brief summary of the ResNet-50/ImageNet result has now also been added to the main text on page 8.

What is the difference between the experimental setup of Figure 6b and Figure 12?

The experimental setup is the same in Figures 6b & 12. Both use GPT-2 Small trained on SlimPajama under the same λ\lambda-initialization rescaling. The difference is only in the reported metric and purpose of the figure. Figure 6b reports validation perplexity to evaluate held-out performance, whereas Figure 12 reports the corresponding training loss to make the optimization effect more explicit. Figure 12 therefore complements Figure 6b by showing that the imbalanced initialization also leads to faster convergence during training, which connects to our Theorem 4.6. For clarification, we have added Table 7 (page 28), which reports the full λ\lambda sweep together with final training loss, validation loss, and validation perplexity. Among the tested settings, λ=0.5\lambda = 0.5 achieves the best final validation perplexity and train loss loss and is highlighted in Figures 6b & 12.

Reviewer_VqyQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the affine parameters of normalization layers interact with downstream weights during training. It shows that normalization replaces the classical balance law between adjacent weights with a feature-wise conserved quantity linking each downstream weight column to its normalization gain and shift, and that this quantity controls the interface's preconditioning and convergence speed. Based on this, the authors propose a simple function-preserving initialization that multiplies downstream weights by λ\lambda and divides the normalization parameters by λ\lambda. Experiments on ResNet-20/50, DeiT-Tiny, and GPT-2 Small report faster optimization and improved accuracy or perplexity over standard and λ\lambda-balanced initializations.

Critical strengths

The paper shows that normalization does not simply destroy the classical δ\delta-balance; it relocates it to a feature-wise conserved quantity Cj=∥W⋅j∥2−γj2−βj2C_j=\|W_{\cdot j}\|^2-\gamma_j^2-\beta_j^2 at each Norm–Weight interface. This is a simple, checkable, and reusable lens: it connects two largely separate literatures and yields a quantity that is measurable during training and controllable at initialization. This is the main reason the paper is interesting despite my concerns.

The proposed initialization (W→λWW\to\lambda W, (γ,β)→(γ,β)/λ(\gamma,\beta)\to(\gamma,\beta)/\lambda) changes the optimization dynamics without changing the initial function, architecture, optimizer, or runtime cost, and the preconditioner analysis explains why the split between γ\gamma and WW matters. The experiments show consistently faster training across convnets and Transformers, suggesting the idea could be practically useful once the causal claims are properly isolated.

Critical weaknesses

The rescaling leaves Π=γW\Pi=\gamma W unchanged but changes the preconditioner in Π˙=−M∇ΠL\dot\Pi=-M\nabla_\Pi L, with M=γ2I+WW⊤M=\gamma^2 I+WW^\top; when p=1p=1 this reduces to the scalar factor γ2+W2\gamma^2+W^2, and for p>1p>1 it is the preconditioner/effective step size. For fixed Π\Pi, MM is minimized at C=0C=0 and increases with imbalance, so the intervention is, to first order, a per-interface rescaling of the effective step size. The paper holds the nominal learning rate fixed across initializations and does not compare with per-group learning rates, a retuned baseline learning rate, or AdamW at matched effective step size. Consequently, the gains cannot be attributed to “imbalance” rather than to a larger effective learning rate; indeed, the accuracy ordering of the three initializations matches their MM ordering. This is consequential because it undermines the paper’s main claim that breaking balance is a new, independent lever.

The final λ\lambda is selected according to test performance (Appendices D/E), which is test-set tuning; the reported test gains are therefore optimistically biased. The results also rely on only 2 seeds for ResNet-20/DeiT and 3 for GPT-2 (ResNet-50 unspecified), provide no results table or confidence intervals, compare at equal epochs rather than equal compute or after convergence, and use SGD rather than AdamW for Transformers. Moreover, the theory assumes β=0\beta=0 and p=1p=1 (or C<0C<0), while the experiments train β\beta and use p>1p>1; for ResNet λ=5\lambda=5, C>0C>0, so the guarantee does not cover the main experiments. These issues make the reported improvements difficult to trust or generalize, which is decisive for a paper whose main contribution is empirical validation of the proposed lever.

Questions for authors

The rescaling changes the preconditioner M=γ2I+WW⊤M=\gamma^2 I+WW^\top (scalar γ2+W2\gamma^2+W^2 for p=1p=1) while leaving Π=γW\Pi=\gamma W fixed. How do the gains compare with per-group learning rates, a retuned baseline learning rate, or AdamW at matched effective step size? Does any advantage remain?

Appendices D/E state that the final λ\lambda is selected according to test performance. Was any validation-based selection used? Please report the full λ\lambda sweep with seeds and variance.

Why does CjC_j remain nearly flat in Fig. 5a but drift substantially (and in some layers change sign) in Fig. 7c? Please reconcile this with Theorem 4.1 and Proposition 4.3.

Lemma 4.5/Theorem 4.6 assume β=0\beta=0 and p=1p=1 (or C<0C<0), while the experiments train β\beta and use p>1p>1; for ResNet λ=5\lambda=5, C>0C>0. Can the theory be extended to these settings, or should the claims be restricted accordingly?

2 comments

We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your suggestions/questions in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them.

The rescaling changes the preconditioner … How do the gain compare with per-group learning rates, a retuned baseline learning rate

Thank you for suggesting this connection. We have now made the relationship between our λ\lambda-rescaling at initialization (from Section 5) and per-group learning rates explicit in the revised Appendix C.1, Proposition C.10 (pages 22-23). In particular, we show that when the downstream weights and normalization affine parameters are assigned learning rates ηW\eta_{W} and ηγ,β\eta_{\gamma, \beta}, respectively, the Norm--Weight symmetry induces a corresponding learning-rate-weighted balance law as described in Proposition C.10. We further derive the resulting effective preconditioner and show that choosing ηW=ηλ−2\eta_{W} = \eta\lambda^{-2} and ηγ,β=ηλ2\eta_{\gamma, \beta} = \eta\lambda^{2} produces, up to the common factor η\eta, exactly the same effective preconditioning from our function preserving λ\lambda-rescaling (as derived in the main text in Section 4.2), which enables training with a common learning rate. The proof follows directly from Lemma 4.5 by evaluating the preconditioner at the rescaled parameters.

Thus, under gradient flow, our λ\lambda-rescaling at initialization and the corresponding per-group learning rates are mathematically equivalent ways of controlling the relative optimization speeds of the normalization parameters and the downstream weights. This equivalence holds for arbitrary downstream dimension pp, where W⋅j∈RpW_{\cdot j}\in\mathbb{R}^{p} and is therefore not restricted to the scalar case p=1p=1; the normalization shift β=0\beta = 0 assumption is used only to keep the effective parameter notation simple and intuitive. Full derivations are provided in the revised Appendix C.1 (pages 22-24).

or AdamW at matched effective step size

We agree with the reviewer that Transformers are typically trained with AdamW.

Our conservation law derived in Theorem 4.1 applies to Euclidean gradient flow and would not remain invariant under adaptive optimization such as AdamW. More generally, Marcotte et al. 2025 showed that conservation laws can be broken when moving beyond Euclidean gradient flow, including under momentum and simplified Adam dynamics. However, our proposed initial rescaling initialization principle can still be applied. Our λ\lambda-rescaling is function persevering and directly controls the initial Norm–Weight balance and the gradient. We demonstrate this with an additional AdamW experiment and show that this initialization intervention can still be effective in this setting.

Accordingly, on DeiT-Tiny trained on CIFAR-100, we keep the AdamW training recipe fixed and vary only the λ\lambda-initialization rescaling across all LayerNorm$\to$Linear interfaces. The best result is obtained at E[Cj(0)]=20\mathbb{E}[C_j(0)]=20 with the corresponding λ=8.624\lambda = 8.624. We explain the details in the revised Appendix F, Table 6 (pages 26 and 28) and provide the final performance metrics in the table below.

λ\lambdaCˉ(0)\bar{C}(0)Test acc. (%)Train loss (final)
0.158−40-4019.94±1.4519.94 \pm 1.454.006±0.0284.006 \pm 0.028
0.183−30-3026.13±3.5326.13 \pm 3.533.861±0.0643.861 \pm 0.064
0.224−20-2028.75±1.7428.75 \pm 1.743.796±0.0443.796 \pm 0.044
0.316−10-1050.69±0.2250.69 \pm 0.222.995±0.0482.995 \pm 0.048
0.353−8-857.53±1.3257.53 \pm 1.322.516±0.0732.516 \pm 0.073
0.445−5-562.64±0.5462.64 \pm 0.542.136±0.0832.136 \pm 0.083
0.686−2-268.82±0.3668.82 \pm 0.361.957±0.0331.957 \pm 0.033
0.905−1-169.35±0.2769.35 \pm 0.271.959±0.0311.959 \pm 0.031
1.000−0.73-0.7369.26±0.4269.26 \pm 0.421.974±0.0321.974 \pm 0.032
1.3880069.05±0.3069.05 \pm 0.302.013±0.0322.013 \pm 0.032
2.1301168.93±0.1168.93 \pm 0.112.061±0.0312.061 \pm 0.031
2.8112268.65±0.4068.65 \pm 0.402.098±0.0312.098 \pm 0.031
4.3335568.96±0.3568.96 \pm 0.352.141±0.0332.141 \pm 0.033
6.104101068.91±0.5368.91 \pm 0.532.176±0.0132.176 \pm 0.013
6.684121269.00±0.4969.00 \pm 0.492.170±0.0122.170 \pm 0.012
7.471151569.51±0.1269.51 \pm 0.122.179±0.0302.179 \pm 0.030
8.624202069.53±0.16\mathbf{69.53 \pm 0.16}2.161±0.0382.161 \pm 0.038
10.560303069.04±0.6469.04 \pm 0.642.157±0.0252.157 \pm 0.025
12.193404069.41±0.2969.41 \pm 0.292.160±0.0302.160 \pm 0.030

the results also rely on only 2 seeds for ResNet-20/DeiT and 3 for GPT-2 (ResNet-50 unspecified), provide no results table or confidence intervals

We ran more seeds for the following settings: DeiT-Tiny/CIFAR-100, ResNet-20/CIFAR-100, and ResNet-50/ImageNet so that each setting now reports results over three random seeds. We have also added detailed result tables for the full λ\lambda sweeps, including the corresponding CC targets, which make the systematic trend across rescaling values visible numerically. For all multi-seed settings, we report the mean and standard deviation across the three runs. The complete results are provided in the revised Appendix F, Tables 5-10 (pages 27-29).

Appendices D/E state that the final λ\lambda is selected according to test performance. Was any validation-based selection used? Please report the full λ\lambda sweep with seeds and variance.

We did not use a separate validation split to select λ\lambda. Instead, for each experiment we evaluate a broad range of λ\lambda values, train every setting to completion under the same training hyperparameters, and observe the behaviour across the full sweep rather than relying on a single isolated choice.

Figure 13 in Appendix F on page 35 in our revised manuscript visualizes the λ\lambda sweep performance throughout training for the DeiT-Tiny on CIFAR-100 with both SGD and AdamW.

In Figure 13.a for DeiT-Tiny/CIFAR-100 with SGD the λ=0.1\lambda = 0.1 setting is unstable and performs substantially worse than the others. By gradually increasing λ\lambda, the performance improves sharply and then plateaus across the remaining settings. λ=0.3\lambda = 0.3 is the optimal one. In Figure 13.b with AdamW, we observe that moving from a strongly negative balance toward positive targets Cˉ(0)>0\bar C(0) > 0 progressively improves performance before saturating, with the best result in this sweep (over 3 seeds) is obtained at λ≈8.6\lambda \approx 8.6.

Why does CjC_j remain nearly flat in Fig. 5a but drift substantially (and in some layers change sign) in Fig. 7c? Please reconcile this with Theorem 4.1 and Proposition 4.3.

First, note the difference in the y axis scale. The lines in Figure 5a approach zero similarly as in Figure 7c as now in Figure 5a the initial imbalance is ≈−11\approx -11. The differences between the trajectories of the CjC_j in the two figures arise due to the different scale of initial imbalance and weight decay. As we have shown in Proposition 4.3, weight decay acts multiplicatively on the balance and therefore moves the imbalance to zero proportionally to the imbalance size. Moreover, in both settings of Figure 5a and Figure 7c, small weight decay is used in combination with a decaying learning rate schedule. This explains the general drifting and that zero is not directly reached at the end of training. Figure 1b explains this, as it plots the imbalance for several weight decay values. It illustrates that a larger weight decay strength is required to approach zero closely. We explain this in the main paper in Section 5 under the “Weight decay gradually restores balance” paragraph on Page 9.

Lemma 4.5/Theorem 4.6 assume β=0\beta =0 and p=1p=1 (or C<0C<0), while the experiments train β\beta and use p>1p>1; for ResNet λ=5\lambda = 5, C>0C>0. Can the theory be extended to these settings, or should the claims be restricted accordingly?

We would like to highlight that the derived invariance holds in the general case and thus covers Theorem 4.1 and the representation cost argument in Proposition 4.4. Moreover, as noted in Remark C.9, the restriction (p=1p=1) in Theorem 4.6 is only needed for an arbitrary-sign guarantee of the invariant. For general output dimension, the same PL convergence argument applies when (Cj<0C_j<0) for every feature. We have moved the remark to the main text to emphasize the possible extension.

We have additionally extended this analysis to separate learning rates. The corresponding condition is negative weighted imbalance (Cj(η)=∥W⋅j∥22/ηW−γj2/ηγ<0C_j^{(\eta)}= \lVert W_{\cdot j} \rVert_2^2/\eta_W - \gamma_j^2/\eta_\gamma < 0), which gives

Mj(η)⪰ηWγj2Ip⪰ηWηγ∣Cj(η)∣Ip.M_j^{(\eta)} \succeq\eta_W\gamma_j^2I_p \succeq\eta_W\eta_\gamma|C_j^{(\eta)}|I_p.

The first inequality follows because W⋅jW⋅j⊤W_{\cdot j}W_{\cdot j}^{\top} is positive semidefinite. To provide some more intuition for the sign condition, negative weighted imbalance prevents the normalization gain from vanishing, ensuring a positive effective learning rate in every direction, including those orthogonal to the weight column. This regime is consistent with the gain-dominated initialization observed in our DeiT experiments and is strengthened by the (λ<1\lambda<1) rescaling used in our Transformer experiments. Full derivations are provided in the Appendix C.1 (pages 22-24).

Comments on the paper

1 comment

Dear AC,

Our paper identifies a new feature-wise conservation law between normalization affine parameters and downstream weights. We show that the resulting initialization imbalance controls their relative learning speeds and effective preconditioning. This motivates a simple function-preserving rescaling that demonstrates improved generalization and faster convergence across vision and language tasks.

The initial reviews highlighted the simple and reusable conservation-law which motivates our practical initialization intervention (VqyQ), the novel optimization insights and carefully designed experiments (eCdS), and the interesting theoretical formulation supported by experiments across realistic architectures (533j).

During the rebuttal, we addressed the reviewers questions and concerns, as summarized below:

  • Theory: For clarity, the main text presents the scalar-output case for intuitive purposes, while the revised appendix now proves that the analysis extends to the general output dimension (p>1p >1). In particular, we show that the feature-wise weighted balance remains conserved and that negative weighted imbalance provides a uniform positive lower bound on the effective preconditioner throughout training. We also formalized the equivalence between initialization rescaling and per-group learning rates under gradient flow.
  • Optimizers and statistical significance: We added a three-seed AdamW experiment on DeiT-Tiny/CIFAR-100, showing that the initialization intervention remains effective beyond the setting where the conservation law holds exactly. We also strengthened the ResNet-50/ImageNet evaluation with additional seeds and observed a statistical improvement over standard initialization (i.e. the baseline) as well as provided additional explanations in the main text and complete tables for all experiments in the appendix.
  • Intuition and controlled experiment: We added general intuition for the connection between symmetry, balance, and convergence in both the revised main text and appendix. We also clarified how the theory applies locally within larger architectures and added a controlled five-layer MLP experiment on CIFAR-10 to demonstrate the mechanism without the additional complexity of ResNets or Transformers.

Reviewer 533j maintained their clear-accept score of 4 and appreciated the additional explanations and intuition, which made the paper’s main ideas more accessible. Reviewers eCdS and VqyQ did not follow up after our rebuttal and manuscript revisions and therefore kept their initial scores of 3 and 2 unchanged.

In summary, our paper provides new insight into how initialization imbalance affects training dynamics in modern day architectures, combining practical effectiveness, simplicity, and theoretical insight. It introduces a simple initialization intervention that consistently improves convergence speed and final performance, with supporting theoretical analysis and empirical validation across multiple settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.