- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 2(−1)
- Soundness
- 3
Read review
Paper summary
This work studies deep neural networks with equivariant layers and demonstrate that the equivariance enables coherent signal propagation across depth. Theoretically, the authors analyze the information propagation of an U(1)-equivariant network, give the conditions of spontaneous symmetry breaking, and show that the spontaneous symmetry breaking helps to preserve the information. The authors also empirically show that adding equivariance can improve the performance of deep RNNs and GRUs.
Critical strengths
The paper provides an interesting perspective on information propagation in deep neural networks. The idea that equivariance can protect certain directions of information propagation is novel and complements the classical edge-of-chaos framework. The connection to spontaneous symmetry breaking and Goldstone modes offers an interesting interpretation. The experimental results suggest that equivariance can improve trainability and performance of deep networks.
Critical weaknesses
- My main concern is the practical relevance of the proposed mechanism. Most widely used neural network architectures do not have the equivariance considered in this paper. Although equivariant neural networks are important in certain applications, it is unclear how broadly the proposed mechanism applies to modern architectures.
Moreover, the theoretical analysis focuses on (U(1))-equivariant networks, which are not commonly used in practical applications. It would strengthen the paper if the authors could extend the theoretical analysis to more practically relevant settings, such as networks equivariant under spatial rotations.
- Relationship between the proposed mechanism and the edge of chaos.
The paper emphasizes that equivariance enables information propagation without fine-tuning to the edge of chaos. However, the phase transition at () appears the same as the critical initialization studied in the classical edge-of-chaos literature.
Furthermore, Section 2.3 shows that the magnitude of the two-input correlation still decays exponentially with depth, even in the symmetry-broken phase. This raises a question about what information is actually preserved. Is the main advantage of equivariance that correlations decay more slowly, resulting in a longer but still finite information propagation length? The authors should clarify this distinction and, ideally, provide a quantitative comparison of information propagation lengths between equivariant and non-equivariant networks.
- Performance of non-equivariant networks near critical initialization.
Figure 3 shows that the non-equivariant networks perform poorly even around (), where the equivariant networks undergo their phase transition. This observation deserves further explanation. Classical edge-of-chaos theory suggests that information propagation can be improved near critical initialization, so one might expect the non-equivariant networks to exhibit better trainability when appropriately initialized.
Questions for authors
-
Can the theoretical framework be extended to more practically relevant symmetries?
-
How does the symmetry-breaking transition at () relate to the classical edge-of-chaos transition? Would the performance of the baselines in Figure 3 improve with appropriately tuned initialization?
-
Since the two-input correlation magnitude decays exponentially in the symmetry-broken phase, in what sense does the Goldstone-like mode enable long-range information propagation?
8 comments
(continued) response to Reviewer 1zHb
Question 2, Weakness 3: How does the symmetry-breaking transition at (σ_W = 1) relate to the classical edge-of-chaos transition? Would the performance of the baselines in Figure 3 improve with appropriately tuned initialization?
In an equivariant model, the phase that is conventionally called the chaotic phase can also be called a spontaneously broken phase, as we briefly explain on p4. The idea is that once there is a nonzero value (on average) for the activations (as measured) by the order parameter c^{l} defined in equation (6), then the radial norm of the activations is nonzero and a symmetry acting on the input necessarily acts coherently on the outputs as well. As we explain in detail in the paper, this results in concrete trainability benefits. One can imagine that the equivariance creates a non-chaotic channel within the otherwise chaotic phase, which is the channel created by the action of the symmetry.
Regarding the baselines, we thank the referee for raising this interesting and important point. The point of Figure 3 is to validate our theoretical results by changing only one parameter (i.e. the presence/absence of the weight constraints associated with equivariance). Following the referee’s suggestion, we have however now performed a careful study of the “conventional” edge of chaos and related this to our work in a new Appendix E. This required tuning of the baselines. The results can be summarized in the new Figure 12 – note that the left-hand side (generic models) shows trainability in a small window around the transition, whereas the right-hand side (equivariant models) show trainability in a large range of parameters, provided we are in the spontaneously broken phase, consistent with the main philosophy of the paper.
This study also showed that the equivariant models are robust under changing the learning rate (see Figure 13), unlike the generic ones, showing another advantage of our construction. Finally, the referee may be interested in the new Figure 14, which presents a preliminary study comparing our method to other standard mitigations (e.g. LayerNorm, residual connections), and demonstrating that it is broadly comparable, and in some regimes better.
Question 3 Since the two-input correlation magnitude decays exponentially in the symmetry-broken phase, in what sense does the Goldstone-like mode enable long-range information propagation?
We believe that the most concrete sense in which the Goldstone mode enables information propagation is through the conditioning of the Jacobian seen in Eq (7) – note that this equation shows that a certain component of the Jacobian remains order 1 provided the order parameter d^{(L)} is nonzero. This follows directly from the presence of the symmetry *and* the non-vanishing of the order parameter in the SSB phase, as the derivation in Appendix C.3 makes clear. (We note also that this derivation essentially parallels that of (Hofman and Iqbal 2018) in the physics Goldstone context). The result is nontrivial as generally Jacobian components either explode or vanish exponentially in $L$ without any further mitigations. We argue around Figures 5ab that this contributes to training dynamics.
As the referee points out (and we agree), the impact of the two-input correlation is somewhat trickier to cleanly understand. It’s important to note that the relative phase of the two-input correlation \phi_l doesn’t decay at all due to the equivariance (see Eq (11)), but the magnitude still appears to decay exponentially. It is of course impossible to measure the relative phase if the magnitude is zero, so it is not clear to us how these two effects interplay in practice. We leave this interesting theoretical question – as well as the construction of networks for which we have a non-trivial fixed point for , which does not seem impossible – for future study.
I thank the authors for their detailed response and the additional experiments, which have addressed most of my concerns. My main remaining concern is the generality of the proposed mechanism. If the authors could extend their results to a more practically relevant symmetry, such as spatial translation or rotation equivariance, I would be happy to raise my score.
Response to Reviewer 1zHb
We thank the referee for their careful reading and for their favorable impression of the novelty and interest of our work. The insightful questions they asked improved our paper, a revised version of which is now posted. Below we present our replies to the points that they raise.
Weakness 1: My main concern is the practical relevance of the proposed mechanism. Most widely used neural network architectures do not have the equivariance considered in this paper.
We thank the reader for bringing up this important point. We want to stress that one of the points of our paper is that this is a new application of equivariance. Until now equivariance has generally been aligned to the task at hand, e.g. for a problem involving molecules one might use spatial rotations, etc. However in this work we demonstrate that equivariance of internal layers – completely unrelated to the task! – can still help in training very deep networks due to their ability to propagate information coherently through depth. The idea for this came from physics, where an analogue of this mechanism allows for long-range propagation of waves such as sound and light.
Thus one is now in principle motivated to apply this framework to any task, not necessarily those where the task has a symmetry. So in our opinion the fact that commonly used networks don’t have an equivariance (yet) is not detracting from our method – rather we encourage the reader to add equivariance in a simple manner to their favorite model and observe benefits in training and information propagation, as we show in toy MLPs in Section 2 and in RNN benchmarks in Section 3.
Weakness 1, continued …It would strengthen the paper if the authors could extend the theoretical analysis to more practically relevant settings, such as networks equivariant under spatial rotations.
And Question 1: Can the theoretical framework be extended to more practically relevant symmetries?
Note that as the choice of symmetry is unrelated to the task, the choice of U(1) vs O(k) etc. is basically on the same footing as an architectural design choice, so we feel that it is not necessarily true that U(1) is less practically relevant for information propagation than the spatial rotation group O(3).
Nevertheless, we believe that the theoretical framework could indeed be extended to O(k); this is largely a straightforward extension using the tools of Appendix C, and we expect all central conclusions to follow through, including the phase structure. One key difference is that we expect the previous Abelian two-replica U(1) Goldstone phase \phi_{l} to now become an element of the group O(k) itself, and thus more degrees of freedom can be carried in it, resulting in the superior performance at depth of O(k) vs U(1) observed empirically.
We are currently performing the careful theoretical analysis needed for this and will update the draft as soon as it is finished, but we did not want to delay submission of the other responses. We will add an update to this response as soon as this is complete.