- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers conservation laws induced by symmetry in networks. In prior work, it has been observed that the scale-symmetry of linear-ReLU stacks leads to a conservation of additive gaps between layer norms through gradient flow, which seems to be a direct consequence of applying a Noether-like argument to gradient flow on the objective function (I am not an expert in this area, but this sounds plausible). The submission now asks the question whether such an analysis would still applicable when normalization layers were added, which break the ReLU-scale-symmetry (positive homogeneity). What the authors find is a symmetry between the rescaling parameter and the weight norm, which might look rather inconsequential at first glance. However, a more careful analysis shows that the sum of the squared norms of weights and the two parameters is strictly conserved under gradient flow, while, semantically, the product of the two scaling parameters (overall weight norm and learned scaling parameter) determine the output. This leads to a a drift in scaling when an initial imbalance is present at initialization, which affects the numerical behavior. The authors continue to show analytically and experimentally that introducing a certain amount of imbalance can help in accelerating convergence as this controls the preconditioning of the different layers during gradient decent (I must admit that I was not fully able to follow the argument, but it seems obvious that this must have an effect on training dynamics). Overall, the paper adds to the understanding of the training dynamics of normalized networks, this time with focus on the learnable parameters, which, to my knowledge before reading the paper, have received limited attention and looked more like an "innocent" add-on. For this reason, I would believe that the paper could form the foundation of some important piece of progress in further elucidating training dynamics of DNNs.
Critical strengths
I think that this is a strong paper and, if there are no problems with related work or technical details (which I have limited ability to check), that it would be a strong candidate for inclusion in ICLR. Particular strengths include:
- A very simple and elegant formulation / derivation from a connection of conserved quantities to symmetries of potentials.
- A clear case of conservation and rescaling following different invariant equations, which strongly suggests emergent dynamical phenomena in (small time step) gradient descent.
- Experimental evidence that show impact on convergence in practice, validating the theoretical predictions.
Critical weaknesses
I did not find any substantial weaknesses, but there is a bit of room for improvement:
-
The paper is well-written and not difficult to read as such, but it is quite technically-leaning and assumes that the reader is familiar with advanced concepts such as generalized applications of Noether's theorem (beyond Lagrangian actions in classical physics). I would suggest two aspects to consider in a revision:
- Add a bit of explanation/recap of the technical concepts and terms (Symmetry and conservation laws, the notion of balance and imbalance, etc.), maybe in a short appendix to make the paper more accessible to a broader audience.
- While technical rigor is commendable, the paper formulates its results mostly in technical language only. It would be good to give some short intuition / application examples to explain how the mechanisms work in practice and what the intuitive ideas behind them are. For example, what kind of loss functions meet the Polyak-Łojasiewicz inequality? For someone seeing this notion for the first time, this is not immediately obvious. Similarly, it would be good to qualitatively/intuitively describe what balance/imbalance means (Appendix B is again very technical in its formulation) and how it affects learning dynamics, in broad strokes, in the context of the symmetries discussed in this paper.
-
On a more technical level, using complex transformer architectures in the empirical studies makes it difficult to see how realistic the formal model is for a complex architecture; even for the ResNet, there might be a few non-trivial aspects (such as the residual addition potentially affecting the scale-symmetry, depending on where it is placed wrt. the normalization). A short discussion might help. Maybe even some experiments with a simplistic 5-layer MLP on CIFAR-10 or the similar might be useful to validate the ideas more directly (in addition to ResNet and Transformers). Nonetheless, showing the effect in a realistic setting is of course also valuable and as such, a strength.
Questions for authors
I do not have any major question, but it would be nice if you could restate in simple, non-technical terms how the convergence behavior is/can be influenced by tweaking initialization balance.
3 comments
Maybe even some experiments with a simplistic 5-layer MLP on CIFAR-10 or the similar might be useful to validate the ideas more directly (in addition to ResNet and Transformers).
Thank you for the great suggestion. We have included an experiment with a 5-layer MLP on CIFAR-10 in Appendix F, observing the same effects in this simpler architecture. -rescaling changes the optimization behaviour across the four Norm--Weight interfaces without changing the overall function. The full sweep shows that the choice of initialization balance has a clear effect on both training loss and test accuracy, providing a controlled setting where the influence of imbalance can be studied without the additional architectural complexity of ResNets or Transformers.
I do not have any major question, but it would be nice if you could restate in simple, non-technical terms how the convergence behavior is/can be influenced by tweaking initialization balance.
By changing the initialization balance redistributes the scale between the normalization parameters and the downstream weights without changing the function represented by the network. This changes how quickly the two parameter groups respond during optimization and therefore changes the effective update of their shared parameterization. As analyzed in Section 4.2, a more imbalanced initialization can produce a larger effective update scale and hence faster convergence. We have addressed this in more detail in Appendix G.
Dear authors,
thanks a lot for the detailed feedback and explanation. This confirms my rough intuitive understanding, and thanks to your explanation, I now finally understand how the change of balance can be used to tune per-layer learning rates; this is really nice. In my experience, divergences in effective (after accounting for symmetry) learning rates do indeed have the potential of greatly affecting how normalization layers affect training dynamics and, in extreme settings (e.g., very deep networks), even training success as such. As far as I know, the influence of the additional learned scaling/shifting parameters has not yet been understood explicitly.
This paper shows very clearly how tempering with the additional parameters leads to adjustments of per-layer effective learning rates. While this might be less relevant for training schemes that use weight- or layer-adpative learning rates (such as 2nd order methods or ADAM-variants), which might compensate for such effects to some degree (unlike plain SGD), the insight is still very valuable in my opinion.
Update to my review: I would thus keep my rating and positive recommendation - I like the paper. Adding the base-line experiment and improving the writing to make the analysis more accessible is of course highly appreciated.
We would like to express our gratitude for your time and efforts in providing valuable comments on our manuscript. Below, we elaborate on your suggestions/questions in a detailed point-by-point response. In case of any open questions, we would be happy to discuss them. We provide a more intuitive explanation of the symmetries, balance and convergence results in our revised manuscript in the appendix and in the main text (highlighted in blue). We have also added an experiment with a 5-layer MLP on CIFAR-10 to better showcase the highlighted mechanisms in a simple setting in Appendix F pages 26-27.
In our revised manuscript, we provide a more intuitive description highlighting the background of technical concepts such as symmetries and their connection to conserved quantities. As follows, we discuss imbalance and how it interacts with convergence speed, which is formally described using the PL inequality.
Symmetry and Balance. In neural networks, a symmetry is a parameter transformation that leaves the represented function unchanged. For example, in a ReLU network, multiplying the incoming weights of a hidden unit by λ>0 and dividing its outgoing weights by λ leaves the network function unchanged. Because the rescaling leaves the loss unchanged, gradient flow does not push the parameters along that rescaling direction. As a result, the relative scale between the two parameter groups remains constant over time, yielding a conserved quantity. This is the basic intuition behind the connection to Noether's theorem. In the classical setting, the conserved quantity measures the difference between the squared scales of the two parameter groups. Our main observation is that normalization relocates this structure to the Norm--Weight interface, where the downstream weight and normalization affine parameters share a conserved quantity Cj. We call the interface j∈[d], balanced when Cj=0 and imbalanced when Cj=0; ∣Cj∣ measures the degree of imbalance, while its sign indicates which side carries more scale. Although these different parameterizations can represent the same function, they induce different effective optimization dynamics. Thus, changing the balance at initialization provides a function preserving way to control these dynamics, and in the setting analyzed in Section 4.2 and in Appendix C.1, greater imbalance can lead to faster convergence.
PL Inequality and Convergence. To make this convergence statement precise, we use the Polyak--Łojasiewicz (PL) inequality. Intuitively, PL requires that when the loss is far from its minimum, its gradient cannot be arbitrarily small. Strongly convex objectives satisfy PL, as do least-squares problems under suitable conditions, while PL is weaker than strong convexity and can also hold for some non-convex objectives. In our setting, the imbalance controls the effective update scale, and the PL condition converts this larger update scale into a faster guaranteed decrease of the loss. We therefore use PL as a sufficient condition for formalizing the connection between initialization imbalance and convergence speed.
Thank you for highlighting this point. We agree that, in a full ResNet or Transformer, architectural components such as residual connections can make the connection to the full model less immediate. Our analysis therefore focuses on the local Norm--Weight interfaces, where a normalization affine parameter is followed by a positively homogeneous downstream map. These interfaces occur repeatedly in both architectures and are the locations where the rescaling symmetry applies directly.
For ResNets, for example, we focus on interfaces such as BN1→conv2 in ResNet-20, where the Norm--Weight structure is directly present and the function-preserving rescaling applies exactly. More generally, our analysis applies to the normalization-to-weight interfaces (illustrated in Figure 2 on page 4) throughout the network that have this local structure, allowing us to study the same symmetry and effective dynamics within the full architecture.