acceptodds
Under review as a conference paper at ICLR 2027

Spontaneous symmetry breaking and Goldstone modes for deep information propagation

Abstract

In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes, which allow coherent information propagation over long distances and times. In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry and may therefore support analogous Goldstone-like degrees of freedom. We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations, providing a mechanism for stable information flow without relying on architectural stabilizers such as residual connections or normalization. In feedforward networks, this results in improved trainability and representational diversity across layers. In recurrent settings, we demonstrate the same mechanism is valuable for long-term memory by propagating information over recurrent iterations, thereby improving performance of RNNs and GRUs on long-sequence modeling tasks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
deep information propagation, recurrent networks, Goldstone modes, symmetry breaking, symmetries, equivariance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1zHbOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
2(−1)
Soundness
3
Read review
Paper summary

This work studies deep neural networks with equivariant layers and demonstrate that the equivariance enables coherent signal propagation across depth. Theoretically, the authors analyze the information propagation of an U(1)-equivariant network, give the conditions of spontaneous symmetry breaking, and show that the spontaneous symmetry breaking helps to preserve the information. The authors also empirically show that adding equivariance can improve the performance of deep RNNs and GRUs.

Critical strengths

The paper provides an interesting perspective on information propagation in deep neural networks. The idea that equivariance can protect certain directions of information propagation is novel and complements the classical edge-of-chaos framework. The connection to spontaneous symmetry breaking and Goldstone modes offers an interesting interpretation. The experimental results suggest that equivariance can improve trainability and performance of deep networks.

Critical weaknesses
  1. My main concern is the practical relevance of the proposed mechanism. Most widely used neural network architectures do not have the equivariance considered in this paper. Although equivariant neural networks are important in certain applications, it is unclear how broadly the proposed mechanism applies to modern architectures.

Moreover, the theoretical analysis focuses on (U(1))-equivariant networks, which are not commonly used in practical applications. It would strengthen the paper if the authors could extend the theoretical analysis to more practically relevant settings, such as networks equivariant under spatial rotations.

  1. Relationship between the proposed mechanism and the edge of chaos.

The paper emphasizes that equivariance enables information propagation without fine-tuning to the edge of chaos. However, the phase transition at (σW=1\sigma_W=1) appears the same as the critical initialization studied in the classical edge-of-chaos literature.

Furthermore, Section 2.3 shows that the magnitude of the two-input correlation still decays exponentially with depth, even in the symmetry-broken phase. This raises a question about what information is actually preserved. Is the main advantage of equivariance that correlations decay more slowly, resulting in a longer but still finite information propagation length? The authors should clarify this distinction and, ideally, provide a quantitative comparison of information propagation lengths between equivariant and non-equivariant networks.

  1. Performance of non-equivariant networks near critical initialization.

Figure 3 shows that the non-equivariant networks perform poorly even around (σW=1\sigma_W=1), where the equivariant networks undergo their phase transition. This observation deserves further explanation. Classical edge-of-chaos theory suggests that information propagation can be improved near critical initialization, so one might expect the non-equivariant networks to exhibit better trainability when appropriately initialized.

Questions for authors
  1. Can the theoretical framework be extended to more practically relevant symmetries?

  2. How does the symmetry-breaking transition at (σW=1\sigma_W=1) relate to the classical edge-of-chaos transition? Would the performance of the baselines in Figure 3 improve with appropriately tuned initialization?

  3. Since the two-input correlation magnitude decays exponentially in the symmetry-broken phase, in what sense does the Goldstone-like mode enable long-range information propagation?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 1zHb

We thank the referee for their careful reading and for their favorable impression of the novelty and interest of our work. The insightful questions they asked improved our paper, a revised version of which is now posted. Below we present our replies to the points that they raise.

Weakness 1: My main concern is the practical relevance of the proposed mechanism. Most widely used neural network architectures do not have the equivariance considered in this paper.

We thank the reader for bringing up this important point. We want to stress that one of the points of our paper is that this is a new application of equivariance. Until now equivariance has generally been aligned to the task at hand, e.g. for a problem involving molecules one might use spatial rotations, etc. However in this work we demonstrate that equivariance of internal layers – completely unrelated to the task! – can still help in training very deep networks due to their ability to propagate information coherently through depth. The idea for this came from physics, where an analogue of this mechanism allows for long-range propagation of waves such as sound and light.

Thus one is now in principle motivated to apply this framework to any task, not necessarily those where the task has a symmetry. So in our opinion the fact that commonly used networks don’t have an equivariance (yet) is not detracting from our method – rather we encourage the reader to add equivariance in a simple manner to their favorite model and observe benefits in training and information propagation, as we show in toy MLPs in Section 2 and in RNN benchmarks in Section 3.

Weakness 1, continued …It would strengthen the paper if the authors could extend the theoretical analysis to more practically relevant settings, such as networks equivariant under spatial rotations.
And Question 1: Can the theoretical framework be extended to more practically relevant symmetries?

Note that as the choice of symmetry is unrelated to the task, the choice of U(1) vs O(k) etc. is basically on the same footing as an architectural design choice, so we feel that it is not necessarily true that U(1) is less practically relevant for information propagation than the spatial rotation group O(3).

Nevertheless, we believe that the theoretical framework could indeed be extended to O(k); this is largely a straightforward extension using the tools of Appendix C, and we expect all central conclusions to follow through, including the phase structure. One key difference is that we expect the previous Abelian two-replica U(1) Goldstone phase \phi_{l} to now become an element of the group O(k) itself, and thus more degrees of freedom can be carried in it, resulting in the superior performance at depth of O(k) vs U(1) observed empirically.

We are currently performing the careful theoretical analysis needed for this and will update the draft as soon as it is finished, but we did not want to delay submission of the other responses. We will add an update to this response as soon as this is complete.

AuthorsOpenReview ↗

(continued) response to Reviewer 1zHb

Question 2, Weakness 3: How does the symmetry-breaking transition at (σ_W = 1) relate to the classical edge-of-chaos transition? Would the performance of the baselines in Figure 3 improve with appropriately tuned initialization?

In an equivariant model, the phase that is conventionally called the chaotic phase can also be called a spontaneously broken phase, as we briefly explain on p4. The idea is that once there is a nonzero value (on average) for the activations (as measured) by the order parameter c^{l} defined in equation (6), then the radial norm of the activations is nonzero and a symmetry acting on the input necessarily acts coherently on the outputs as well. As we explain in detail in the paper, this results in concrete trainability benefits. One can imagine that the equivariance creates a non-chaotic channel within the otherwise chaotic phase, which is the channel created by the action of the symmetry.

Regarding the baselines, we thank the referee for raising this interesting and important point. The point of Figure 3 is to validate our theoretical results by changing only one parameter (i.e. the presence/absence of the weight constraints associated with equivariance). Following the referee’s suggestion, we have however now performed a careful study of the “conventional” edge of chaos and related this to our work in a new Appendix E. This required tuning of the baselines. The results can be summarized in the new Figure 12 – note that the left-hand side (generic models) shows trainability in a small window around the transition, whereas the right-hand side (equivariant models) show trainability in a large range of parameters, provided we are in the spontaneously broken phase, consistent with the main philosophy of the paper.

This study also showed that the equivariant models are robust under changing the learning rate (see Figure 13), unlike the generic ones, showing another advantage of our construction. Finally, the referee may be interested in the new Figure 14, which presents a preliminary study comparing our method to other standard mitigations (e.g. LayerNorm, residual connections), and demonstrating that it is broadly comparable, and in some regimes better.

Question 3 Since the two-input correlation magnitude decays exponentially in the symmetry-broken phase, in what sense does the Goldstone-like mode enable long-range information propagation?

We believe that the most concrete sense in which the Goldstone mode enables information propagation is through the conditioning of the Jacobian seen in Eq (7) – note that this equation shows that a certain component of the Jacobian remains order 1 provided the order parameter d^{(L)} is nonzero. This follows directly from the presence of the symmetry *and* the non-vanishing of the order parameter in the SSB phase, as the derivation in Appendix C.3 makes clear. (We note also that this derivation essentially parallels that of (Hofman and Iqbal 2018) in the physics Goldstone context). The result is nontrivial as generally Jacobian components either explode or vanish exponentially in $L$ without any further mitigations. We argue around Figures 5ab that this contributes to training dynamics.

As the referee points out (and we agree), the impact of the two-input correlation is somewhat trickier to cleanly understand. It’s important to note that the relative phase of the two-input correlation \phi_l doesn’t decay at all due to the equivariance (see Eq (11)), but the magnitude Δ\Delta still appears to decay exponentially. It is of course impossible to measure the relative phase if the magnitude is zero, so it is not clear to us how these two effects interplay in practice. We leave this interesting theoretical question – as well as the construction of networks for which we have a non-trivial fixed point for Δ\Delta, which does not seem impossible – for future study.

I thank the authors for their detailed response and the additional experiments, which have addressed most of my concerns. My main remaining concern is the generality of the proposed mechanism. If the authors could extend their results to a more practically relevant symmetry, such as spatial translation or rotation equivariance, I would be happy to raise my score.

Reviewer_zZ7KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates symmetry-equivariant hidden layers with a particular focus on their initialization regimes, offering two distinctive phases of unbroken symmetry and spontaneously broken symmetry. Placing the layers on-initialisation within the spontaneously-broken regime, the gradients seem to circumvent classical vanishing gradient issues. On experiments on toy-datasets and a 5-epoch training, the effect seems to hold. On the UAE time series classification task, their equivariant layers help standard RNN/GRU without any additional skip-connections or normalisations.

Critical strengths
  • studying symmetry equivariance within hidden layers for information dynamics is very relevant to the community
  • relying on physics concepts for better information propagation rather than technical tricks is timely for rigorous advances in the field
  • theoretical analysis on the gradients through symmetry propagation in RNNs offering guarantees on the rank and non-vanishing properties
Critical weaknesses
  • No closed-form definition of what it means to for the layers to have the goldstone mode
  • No explicit definition of the case of breaking the symmetry within symmetry-equivariant layers
  • The reliance on complex-valued representation seems to give unfair capacity to the models
  • Experimental results for T-max=200 where only U(1)-GRU is reported to solve the delay-copy task is not provided, nor is the performance of baseline methods on the sequential MNIST task reported
Questions for authors
  • Please could you provide a formal definition of what it means for equivariant layers to propagate the goldstone mode?
  • What role does the capsule concept play in your construction?
  • How did you make sure for a fair comparison of number of parameters when you are comparing complex-valued models, typically doubling the parameters?
  • What role does the last layer have on the symmetry breaking? And why do you deduce from a non-zero representation that the symmetry must have been spontaneously broken?
7 comments
AuthorsOpenReview ↗

Response to reviewer zZ7K

We thank the referee for their careful reading, and we completely agree with the philosophy that importing ideas from physics for information propagation may lead to important advances, which is indeed the primary motivation for this work. Below we respond directly to the points raised, both in the weaknesses section and the explicit questions.

Question 1 (and Weakness 2, 3): Please could you provide a formal definition of what it means for equivariant layers to propagate the goldstone mode?

We thank the reviewer for their interest in this. A formal definition for spontaneous symmetry breaking always requires a limit (e.g. in conventional physics it is defined in the infinite volume limit). For our purposes we do not have a concept of spatial volume (as we discuss in Appendix C.4) and thus we must use a different concept.

One useful definition in the deep learning context is the following. Denoting the function defined by a stack of LL equivariant layers by f(L)(x)f^{(L)}(x), they propagate the Goldstone mode on an input $x$ if:

∥lim⁡L→∞(f(L)(gx)−f(L)(x))∥ k\|\lim_{L \to \infty} (f^{(L)}(g x) - f^{(L)}(x))\| \> k

where kk is a nonzero constant which is independent of LL and gg is a symmetry transformation. Informally, this means that the transformation done on the input can be detected at the output, and can then be used for information propagation purposes.

If we are in the unbroken phase then all activations decay exponentially in L, and this relation is not satisfied as the left hand side is exponentially small in L. In the spontaneously broken phase activations tend to a constant norm at the output and this relation is satisfied. In fact if the symmetry g acts linearly (as is the case in all examples studied in this paper) the non-zero norm measured by the order parameter in (6) is sufficient for the formal definition above to be satisfied.

Though we did not frame our arguments in terms of precisely the formal definition given above, it is implicit in quantitative arguments that are directly related to the training benefits we see. For example, Eq (7) relates a certain component of an input/output Jacobian to an order parameter $d^{(L)}$, which measures the norm of pre-activations at depth $L$. When $d^{(L)}$ is nonzero then the Jacobian has a constant component even at infinite-depth. This is possible because information is propagated through by the Goldstone mode, as the derivation in Appendix C.3 makes clear.

Question 2: What role does the capsule concept play in your construction?

Answer: The capsule idea simply summarizes the fact that the representation space now can be broken into many sub-representations, each of which transforms the same way under the symmetry group G. One may call each of them a capsule. Formally speaking the internal state becomes a direct sum over many representations of the group G (in this case each representation is just the usual complex representation for U(1), or the fundamental representation for O(k)), and each element in the direct sum can be thought of as a capsule. In the notation of Eq (18) α\alpha, β\beta run over capsules, and a, b run over the degrees of freedom within each capsule.

Question 3: How did you make sure for a fair comparison of number of parameters when you are comparing complex-valued models, typically doubling the parameters?

We thank the reviewer for highlighting this important point on fairness which we have indeed taken into consideration and now emphasized in our updated PDF. As reported at the end of the Model Training and Evaluation section: “Parameter counts are reported in real trainable parameters, where complex weights count as two real parameters”. This is true for all reported values (Figures 6, 7, & Tables 1-5). This point is reiterated in the appendix section D.2.1 under the paragraph titled Parameter counts. In our updated PDF, we have now included explicit equations for how the parameter counts are computed for each RNN model on the copy task to exemplify this.

AuthorsOpenReview ↗

(continued) Response to reviewer zZ7K

Question 4: What role does the last layer have on the symmetry breaking? And why do you deduce from a non-zero representation that the symmetry must have been spontaneously broken?

The last (and the first) layers are not equivariant under the symmetry; they cannot be, as they involve the action of the input or output data, which do not transform under the symmetry, in contrast to the usual situation in equivariant neural networks and geometric deep learning. For us, the equivariance is only a tool used to propagate the information through the layers. For example, in the U(1) case there is an overall phase which propagates all the way through, and the final layer can then read information off of this phase to perform the final calculation.

As described above in the response to question 1, if the activations x have non-zero norm and the symmetry acts linearly (as is always the case in the examples in this paper), then acting by g x on the input will have a non-trivial action on the output f(x). In the context of this paper, this is spontaneous symmetry breaking – the NN dynamics constrains the norm of the activations, but the action of the symmetry orbit is not determined by the NN dynamics, and is instead determined by the action of the symmetry on the input.

Weakness 1: No connection to a formal analysis for vanishing gradients is given.

We would like to stress that Eq (7) (which is carefully derived in Appendix C.3) is a theoretical derivation which shows that a certain component of the Jacobian of the input/output map cannot vanish if we are in the spontaneously broken phase – it is instead directly related to the value of the order parameter, which is nonzero in that phase. This means that the gradients that contribute to this Jacobian component cannot vanish, and we believe that this is one of the key factors contributing to the training benefits that we see (as argued around Figure 5). Thus we feel that this is a formal analysis showing which gradients do not vanish.

However we are happy to revisit or extend the theoretical analysis if you would like to see something else explained in more detail – could you be specific about what kind of formal analysis you are looking for precisely?

Weakness 4: The reliance on complex-valued representation seems to give unfair capacity to the models.

As described in our above answer to the related question, we have accounted for the increased capacity of complex-valued representations in two complementary ways throughout the manuscript. First, we have counted all complex parameters as two real-valued parameters on all datasets for all reported values. This is clearly seen by comparing the ‘# Params’ in Figure 2: the U(1) RNN with 32 hidden units has 3,338 parameters, while the Generic RNN with 32 hidden units has 1,738 parameters.

Secondly, to account for the fact that the complex representations may be seen as encoding the equivalent of two real valued activations (through the real and imaginary components) we have consistently reported real-valued baselines with twice the number of hidden units as the comparable complex models. This is the reason for the (N=64) baselines in Figure 6, while the equivariant model has N=32. As reported in the main text on the copy task: “the equivariant model significantly outperforms both baselines across 11 random seeds, including the 64-unit baseline with comparable real hidden dimensionality”. This is additionally true of the feedforward experiments on MNIST and Fashion MNIST where all models have 64 real activation coordinates (32 complex units, and 16 O(4) blocks). In the updated PDF, we have added a clarifying paragraph to appendix section D.1 to emphasize this.

To further reinforce the conclusion that the hidden state capacity of equivariant models is not confounding our results, we have included an additional set of ‘matched state size’ results for the UEA suite of tasks in Table 6 of the updated PDF. We see that, again, in this extended suite of tasks, the equivariant models generally outperform the non-equivariant baselines even in this stricter setting with significantly fewer trainable parameters.

AuthorsOpenReview ↗

(continued) Response to reviewer zZ7K

Weakness 5: Experimental results for T-max=200 where only U(1)-GRU is reported to solve the delay-copy task is not provided

We now report the full results for 11 random seeds on the T_max=200 task the Appendix section D.2.3. We see that these results match the prior performance distribution: none of the 11 baseline GRU models were able to solve the task, while at least one of the U(1) GRU models reaches significantly lower ‘solution’ level accuracy (test CE <0.005). Therefore, while the length of this task is difficult for all models, the equivariant model maintains a clear advantage in both mean performance and best performing seeds.

Weakness 5, continued: the performance of baseline methods on the sequential MNIST task [is not] reported.

We have now updated Figure 7 in the revised PDF to include a number of published baselines which were previously mentioned in the adjacent text to provide context. We have additionally updated our Table 4 in the appendix to include a wider range of baselines that would otherwise crowd the figure, and referenced it in the figure caption.

While we agree with the reviewer that it is important to include these data points to situate our models’ performance in the related literature, we caution against making a direct comparison between diverse models for the purpose of this paper. The primary contribution of our paper is the theoretical understanding of a novel Goldstone mechanism which is applicable to a broad range of model architectures, rather than to compete directly with state of the art. Therefore, the most meaningful comparisons are directly between baseline models and their equivariant counterparts (e.g. RNN vs. O(k) RNN; GRU vs. U(1) GRU); and we have intentionally started with simple architectures to provide universally agreeable baselines. Understanding the relevance of our proposed mechanism to more exotic previously published architectures will require a new set of controlled experiments. We are currently investigating these new equivariant architectures, and will include them in a later response if time permits.

Reviewer_Da4TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Hidden layers are equivariant under U(1) or O(k); only the input and output maps break the symmetry. A large-width analysis of the U(1)-invariant activation magnitude has a transition at weight scale sigma_W = 1. Above it, activations sit at a nonzero radius and a global phase is copied from layer to layer. The authors treat that phase as a Goldstone mode that carries signal without LayerNorm or skip connections. Equivariant RNNs and GRUs then improve long-delay copy, permuted MNIST, and the UEA archive.

Critical strengths

The critical scale sigma_W = 1 is derived, not fit, and training accuracy on 100-layer Fashion-MNIST turns on exactly there for U(1) and O(4). The control uses the same nonlinearity with a generic linear map, so the plot isolates equivariance. During training the protected Jacobian stays finite while the generic Jacobian collapses, and the rank floor is visible across depth. On sequences the gains repeat: U(1) models beat parameter-matched RNNs and GRUs on copy and permuted MNIST; on UEA, O(4)-RNN moves average accuracy from 62.55% to 66.82% and wins significantly on 14 of 30 datasets, including EigenWorms (length greater than 17,000, more than 10 points). Complex weights are counted as two real parameters.

Critical weaknesses

phi_{l+1} = phi_l is an identity of the equivariant layer. It holds in either phase; in the unbroken phase the phase is undefined because the state has collapsed to 0. What spontaneous breaking supplies is a nonzero radius, so this built-in channel stays observable. That is thinner than a Goldstone theorem. The two-point magnitude still decays exponentially, and the paper finds no stable Delta_star > 0. The introduction frames the method as a replacement for LayerNorm and residual connections, but every control is an unstabilized MLP, RNN, or GRU. Permuted-MNIST accuracy is reported as comparable to IndRNN, coRNN, and LEM (about 96-97%). The T_max = 200 copy result is one seed. Part of the O(k) gain is the rank floor itself, which should be separated from the phase story.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to reviewer Da4T

We thank the referee for their positive assessment and careful reading of our work. We are especially grateful to the reviewer for noting that we carefully controlled our experiments for parameter count by counting complex weights as two real parameters. Below is a response to the specific weaknesses raised:

Weakness: phi_{l+1} = phi_l is an identity of the equivariant layer. It holds in either phase; in the unbroken phase the phase is undefined because the state has collapsed to 0. What spontaneous breaking supplies is a nonzero radius, so this built-in channel stays observable. That is thinner than a Goldstone theorem.

Response: We actually feel that it is not really thinner, and is in fact quite similar to the standard Goldstone theorem, with important modifications arising from the fact that there is no "space". Let's briefly review this in the physics context. Consider a complex scalar field ϕ(x,t)\phi(x,t) with dynamics given by a U(1)-preserving potential V(ϕ†ϕ)V(\phi^{\dagger}\phi). Note that one can always decompose the field into a modulus and a phase ϕ(x,t)=ρ(x,t)eiθ(x,t)\phi(x,t) = \rho(x,t) e^{i\theta(x,t)} and then obtain an equation of motion for the phase which is ∂μ(ρ2∂μθ)=0\partial_{\mu}(\rho^2 \partial^{\mu} \theta) = 0. This equation holds independent of whether the symmetry is spontaneously broken or not. It is analogous to stating that in our deep learning setup that ϕl+1=ϕl\phi_{l+1} = \phi_{l} is an identity of the equivariant layer.

Nevertheless generally this does not result in a stable massless excitation in θ\theta: indeed it is only when the symmetry is spontaneously broken (i.e. that V(ϕ†ϕ)V(\phi^{\dagger}\phi) has a stationary point away from zero) that ρ\rho is stabilized at a nonzero radius, and the phase degree of freedom \theta can then be interpreted as a Goldstone mode. This is thus very similar to our own deep learning setup, where the spontaneous breaking also provides a nonzero radius indeed makes it possible for the network to extract information from this built-in channel.

This results in concrete results, e.g. in the proof of our Jacobian result in Eq (7) (indicating that a certain component of the Jacobian remains well-conditioned at all L) was heavily inspired by a result from the Goldstone physics literature (Hofman & Iqbal 2018), whose proof we essentially took over in the deep learning setting (as shown in Appendix C.3 and discussed further in C.4).

Weakness: The two-point magnitude still decays exponentially, and the paper finds no stable Delta_star > 0.

We thank the referee for drawing attention to this important point. We are not certain as to the interpretation of this two-replica fact: clearly there is still a clean benefit arising from the equivariance, as shown empirically. Possibly relevant is the fact that for any non-zero \Delta, \phi_{l} is exactly conserved, and it seems that this conveys useful information even though \Delta itself may decay with layers. There are also multiple benefits to equivariance: for example the Jacobian result shown in Eq. (7) arises solely from the single-replica analysis (and thus knows nothing about \Delta), and it seems this is sufficient for a strong training benefit.

We leave to future work the development of a model which has Delta_star > 0 (which seems possible with further adjustment of the architecture – it depends on details of the activation functions etc. – and would perhaps have performance that is even better than ours).

Weakness: The introduction frames the method as a replacement for LayerNorm and residual connections, but every control is an unstabilized MLP, RNN, or GRU.

We are happy to perform controls with LayerNorm and residual connections; we have now performed a preliminary investigation in the case of the MLP, which we show in the new Figure 13 in the new Appendix E. In this case our Goldstone method is broadly comparable, and indeed performs better than existing mitigations over a range of parameters.

In general, however we stress that the motivation of the Goldstone method is not necessarily that it is always better than existing methods; rather it is a different and physically motivated way of arising at a similar result, and thus an alternative tool in the deep learning practitioner’s toolkit. We feel that in future work researchers may find yet more regimes where the benefits of our Goldstone method are complementary to those of LayerNorm and/or residual connections, and therefore may be used in combination with these to yield optimal performance.

AuthorsOpenReview ↗

(Continued) response to reviewer Da4T

Weakness: Part of the O(k) gain is the rank floor itself, which should be separated from the phase story.

We believe the “rank floor” arises directly from the equivariance and is thus intimately related to the “phase story”. To explain this further: in the spontaneously broken phase, the radius of the activations is non-zero, and thus the O(k) symmetry must act faithfully on the activations. The minimum possible dimension of a non-trivial O(k) representation is k. Thus the activations cannot collapse to an effective dimensionality which is lower than k (i.e. the “rank-floor”), but this arises entirely from the restriction of equivariance. A simple way to see this is to note that in Figure 5 (b) the non-equivariant model collapses to a 1-dimensional subspace, but the minimum possible dimension of the other models is indeed bounded by the dimensionality predicted by equivariance.

Weakness: Permuted-MNIST accuracy is reported as comparable to IndRNN, coRNN, and LEM (about 96-97%).

We highlight that comparable performance to these high-performing baselines is presented as a strength of our model rather than a limitation. Specifically, our base models are simple RNNs and GRUs which otherwise have not been competitive on this benchmark, but can be made competitive with the addition of our construction. In our updated PDF, we have included the performance of additional baselines on the psMNIST task where our model can be seen to outperform some published variants such as the uRNN of Helfrich et al. 2017, and the LSTM reported by Rusch et al. 2022 – please see the updated Figure 7 and Table 4.

Weakness: The T_max = 200 copy result is one seed.

To clarify, we had originally trained 4 seeds for each model and only one of the U(1) GRU seeds succeeded in solving the task; none of the baseline GRU seeds were able to solve it. Given this level of variability, we did not report the raw results initially. We have since run a full 11 seeds for each model on this task, comparable to the other copy lengths, and now report the full distribution of their performance on the task in the Appendix section D.2.3. We see that these results match the prior performance distribution: the mean performance of the U(1) GRU is significantly better than that of both baselines, while the U(1) model is also the only model that has any seeds which are able to solve the task to a high degree of accuracy.

I thank the authors for their comments and maintain my rating of weak accept. I also included the shorthand notes used to guide aspects of AI reviewing.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Updated draft and summary

Dear referees,

We would like to thank you all for your careful reading and feedback on our paper. We feel that incorporating responses to your questions has made our paper stronger. We have submitted a revised version with changes in red. Important additions include:

  • A new Appendix E with further investigations of MLPs suggested by the referees, including a careful study of the edge of chaos, comparison with standard mitigations of LayerNorm and residual connections, and a study of training while adjusting the learning rates.
  • Further runs on the T = 200 copy task, with results in Appendix D, Table 3, and Figure 11.
  • A new table 8 on the UEA dataset with matched hidden-state dimensionality (complementing Table 7, at matched parameter-count).
  • Clarifications on how the parameter-count is measured and the inclusion of more baselines from the literature in Figure 7.
  • Small clarifications in wording throughout (indicated in red).

We look forward to continuing the discussion.

Dear reviewers,

The authors have now responded to your comments/critiques. As we are nearing the end of the discussion period, please read through the authors' rebuttal and respond if there are points that remain unclear (or to let them know they have addressed your concerns).

Sincerely,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.