acceptodds
Under review as a conference paper at ICLR 2027

Layerwise and Exotic Equivariance Constraints in Shallow Neural Networks

Abstract

We prove the existence and uniqueness of a canonical form for shallow neural networks with activations that are uncoupling, meaning that the activations take pre-activations to linearly independent functions. Prior work shows that ReLU, Tanh and several other activations are uncoupling. We generalize this to also include modern activations such as GELU, SiLU and ELU. Using the canonical form, we characterize all possible equivariant shallow networks with uncoupling activations in terms of equivariance constraints on the weights and biases. Some of the constraints define classical layerwise equivariant neural networks, but some are exotic variants that are not layerwise equivariant, instead having inter-layer constraints. Our results generalize prior work by Agrawal and Ostrowski (2022), who characterized single output invariant shallow ReLU networks. Furthermore, we show that layerwise networks and exotic networks can have qualitatively different optimization behaviour and test the practical impact of different choices of equivariant structures in the MLP blocks of equivariant vision transformers.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Geometric deep learning, Identifiability, Equivariance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FzraOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper characterizes equivariant shallow neural networks with uncoupling activations, including ReLU, GELU, and SiLU. It studies intrinsic activation symmetries: affine transformations of pre-activations whose effects can be compensated by output scaling and affine corrections. These identities generate parameter transformations that preserve the network’s computed function. The authors prove that every shallow network can be reduced to a canonical form, which is unique for uncoupling activations once a normalization convention is fixed. They use this result to derive weight constraints characterizing equivariant networks, including both layerwise constructions and exotic constructions with dependencies across layers. They also study their optimization behavior and compare several constructions in the MLP blocks of equivariant vision transformers. Among the tested designs, one layerwise construction achieved the highest reported accuracy, while the exotic constructions underperformed both layerwise alternatives and the unconstrained baseline.

Critical strengths
  • A unified framework covering modern activations. One framework that covers modern activations such as ReLU, GELU, SiLU. It also accommodates vector-valued outputs and general group equivariance, extending the scope beyond scalar invariant ReLU networks.

  • Explicit constraints that capture both layerwise and exotic equivariance. The framework translates functional equivariance into equations on network parameters, including dependencies across layers that standard layerwise constructions miss. This provides a systematic basis for constructing and comparing equivariant architectures.

  • A precise account of parameter redundancy. The canonical-form results establish output-preserving transformations and for uncoupling activations it is unique once normalization is fixed. Consequently, equivalent almost canonical networks are related by a unique gauge transformation, giving a rigorous characterization of their remaining parameter ambiguity.

Critical weaknesses

The authors acknowledge the paper’s main limitations.

  • Clarity: The paper would benefit from a preliminary section introducing the group-theoretic concepts needed to follow the analysis. The main text could also explain the homomorphism Ψ\Psi and its role more explicitly.
  • Experimental evaluation: A broader evaluation across additional datasets and architectures would strengthen the submission and provide a clearer understanding of the relative benefits and limitations of the different constructions.
Questions for authors

No Questions.

3 comments
AuthorsOpenReview ↗

Response to reviewer Fzra

We thank the reviewer Fzra for their positive review and charitable characterization of our results.

Weaknesses

  • (Clarity) We thank the reviewer for suggesting to make the role of the homomorphism Ψ\Psi clearer in the main paper, as we believe this to be a good idea. We have now moved the technical wreath product result from the appendix to Proposition 3.7 and illustrated it by applying it to a width 44 GELU-network. This example is accompanied by a new figure as well. Please see the blue text in Section 3 of the updated PDF for this new content.
  • (Experimental evaluation) A thorough empirical investigation of the impact of the choice of Ψ\Psi on performance is unfortunately beyond the scope of this paper, but we regard it as important future work. The intention of the experiment in our paper was simply to verify that there is a problem to be solved, i.e., the unconstrained network does not find the optimal configuration of Ψi\Psi_i for the different layers on its own and the different Ψi\Psi_i perform differently.

We hope that this resolves the concerns.

Thank you for the detailed response and clarifications. The authors have addressed my main concerns, and I will maintain my current score.

Thank you again for the positive review and for engaging with our rebuttal. Thank you especially for the suggestion to clarify the role of the homomorphism Ψ\Psi, which helped improve the paper.

Reviewer_cDcuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is devoted to the characterization of equivariant shallow networks with uncoupling activations (which take pre-activations to linearly independent functions; in particular, ReLU, Tanh, GELU, SiLU, ELU) in terms of equivariance constraints on the weights and biases. The paper shows that ordinary equivariant and exotic neural networks may have different optimization dynamics, and, as a proof-of-concept, verifies this on ViT-L.

Critical strengths

I like the writing style of the paper (besides some template violations, see weaknesses). In general, the text is very clear for a broad audience, and enough effort is applied to ensure this: examples throughout the paper, the formulation of the main results in simple words, the analogues that are considered in the literature, etc. The work is technically dense, but it seems accurate to me from the mathematical point of view (but I did not check all the details).

Critical weaknesses

The developed theory looks nice and curious in some sense; however, I have doubts regarding its practicality and applicability from both practical and theoretical points of view. I mean, the theory looks curious, but what we should do with it is not so straightforward (for me). If we treat the paper as a practical one, then the provided ViT-L experiment looks more like a proof-of-concept (and the authors write about it) with a particular choice of activation and equivariance group. I somewhat expected more from the experimental part after such a dense theory. At the same time, if we qualify this work more as theoretical (and do not expect too much from experiments), then the question arises why do we need this theory apart from curiosity.

Few notes on the presentation of content:

  • It is not so obvious what precisely “Qualitative plot of the functions” means mathematically in Figures 1–2.
  • I would like to note that the paper violates the template a bit. For example, there is no title of the first section (‘Introduction’?) and that is why the numbering of the environments started from 0. I have probably also noticed some spacing violations, in particular, on the first page (after the abstract). But I do not think that these violations are significant.
Questions for authors

Since I have some doubts about the significance of the paper, I would like to clarify the following points:

What is the pipeline for applying the obtained results (practically or for theoretical investigation)? What is the main motivation for studying this? What is the main obstruction to extending the classification to deep networks?

4 comments
AuthorsOpenReview ↗

Response to reviewer cDcu (Weaknesses)

We thank reviewer cDcu for their helpful review and the kind remarks on the writing style and ease-of-read. We especially put effort into fitting the examples into the paper, so it is good that they were appreciated.

Weaknesses

  1. As the reviewer points out, the experiment in Section 4 is a proof-of-concept experiment. A serious empirical investigation of which choice of Ψ\Psi yields the best results in practice is out-of-scope of this paper. This work should instead be considered mainly a theoretical contribution. To explain the theory's relevance, consider designing an inherently equivariant neural network to tackle a learning task which has some known symmetry (ρin(\rho_\mathrm{in}, ρout)\rho_\mathrm{out}). The first step to designing such a network is knowing what the search space of equivariant architectures is. Our Theorem 3.1 together with the Corollaries 3.3 (new in the updated version) and 3.4 gives a characterization of all equivariant almost canonical shallow networks. Thus, given GG, σ\sigma, and hidden width nn, we know to apply the result of Kovács (previously Proposition B.11, now 3.7 in our paper) to find all non-conjugate Ψ:G→Γσ≀Sn\Psi:G\to\Gamma_\sigma \wr S_n and then solve Equation (17) for almost canonical networks. This then gives the space of possible equivariant architectures to try. We have added an illustrative example in Section 3, see the blue text in the updated PDF. Note that our results do not say which architectures are the best, only which are possible, but this is an important first step. Does this answer the concerns, or is there something we could add to the paper or this discussion that would further clarify the situation?
  2. On formatting issues, we were simply not aware that the header for the introduction section was a necessity. We have fixed this in the updated version of the paper. We are not aware of any other spacing violations present in the paper, unless what is being referenced is precisely the lack of a header in the first section. If the reviewer could point them out, we will make sure to fix them. On Figures 1—2, what is meant by qualitative is that the images are not trying to capture any quantitative aspects of the functions / spaces. In the updated version of the paper the figures are numbered in the reverse order. For Figure 2 (formerly Figure 1), we simply want to convey the non-linearity of the exotic parameter space and the fact the spaces intersect in 00. However, no numerical information can be read off from the visualization. Similarly, for Figure 1 (formerly Figure 2), we want to capture the general shape of the functions produced by this particular example (e.g., the double mode of the exotic parameterization when b>a>0b>a>0, which never appears for the layerwise parameterization) as an example of the theory producing networks defining different functions. However, the functions' values depend on the specific values of the parameters and are not of particular interest. Does this resolve the question about the paper's presentation?
AuthorsOpenReview ↗

Response to reviewer cDcu (Questions)

Questions

These are all good questions, we will attempt to answer them here and to clarify them further in the final version of the paper.

  • (Pipeline) As we wrote in the reply 1. to the weaknesses, one way to answer this question is to consider the design of an equivariant neural network. It is not clear a priori what the best choice of equivariant architecture is. Indeed, not even among the layerwise variants is it clear which choice of intermediate representations yields the best results. We further demonstrate that there are exotic variants that also need to be considered. The pipeline would then be to find the non-conjugate Ψ\Psi and then solve Equation (17) for almost canonical networks. This gives the space which we need to search to find the optimal architecture. The new Example 3.8 contains a small example of the procedure of finding all equivariant architecture, and Appendix A contains a more practically relevant version of architecture design. How to best conduct the search over architectures in general is an interesting question for future work. However, the present work lays the basis for this task.
  • (Motivation) The motivation behind this paper is precisely the question of optimal design of equivariant neural networks (i.e., in the context of our paper, which is the optimal Ψ\Psi). The results of our paper therefore provide a springboard for further research into the topic. There is also the broader motivation of trying to understand the different equivariant architectures theoretically.
  • (Extension to deep networks) Our theoretical framework relies on being able to find a canonical form of the neural networks, which essentially resolves the identifiability problem, since the parameters of a canonical network are uniquely determined by the function defined by the network. Thus, the main obstruction to extending the results to deep networks is related to the problem of identifiability of deep networks, which is an open problem in general, and in particular open for many of the activations we consider. In shallow networks, the effects of the symmetries of σ\sigma, such as σ(−x)=σ(x)−x\sigma(-x) = \sigma(x)-x, are controlled by the fact that we never need to compose σ\sigma with itself. Deeper networks can have more complicated symmetries induced by the composition of layers.

Does this answer all of the questions and concerns?

I thank the authors for their detailed response. I believe the answers and the changes to the manuscript have made the outcomes clearer. I find the results potentially interesting to the ICLR community and useful for future research. Therefore, I am raising my score to 3.

Thank you once more for your helpful review and for engaging with the rebuttal. We agree that the changes have made the manuscript clearer, and the questions and concerns you raised about how our results could be applied certainly helped in that regard.

Reviewer_otuZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how neural networks can respect input symmetries, such as horizontal image reflections. Usually, this is achieved by making each layer respect the symmetry separately. The authors characterize exotic constructions in which relationships between layers ensure that the whole network respects the symmetry. Their theory covers networks with one hidden layer and an affine skip connection, using common activations including GELU and SiLU. They also show that training with transformed examples can break an initially symmetric network’s behavior. Experiments with reflection-equivariant vision transformers illustrate that different symmetry constraints affect accuracy.

Critical strengths
  1. The paper is well-written and easy to follow.

  2. The paper proposes a meaningful alternative to standard symmetry-based design. They characterize how different layers can jointly enforce symmetry and extends earlier results to widely used activations.

Critical weaknesses
  1. The experimental conclusions seem overstated. Averaged over five seeds, the baseline achieves 75.9% top-1 accuracy versus 75.1% for Ψ2\Psi_2. The exotic Ψ3\Psi_3 also outperforms layerwise Ψ1\Psi_1 on average, hence the results do not establish a general ordering between the approaches. Exotic models additionally use projection each 25 optimizer steps, which prevents attributing their performance solely to their symmetry constraints. While getting similar performance with half as many parameters is promising, it seems that the claim about unconstrained training fails to find an optimal equivariant structure is not well established.

  2. I think the claimed functional distinction between experimental designs needs a separate justification. As Ψ1\Psi_1 and Ψ4\Psi_4 are conjugate, Corollary 2.3 cannot establish that they represent different functions. Gauge transformations do not need to preserve C=0C=0, so this also does not prove their no-skip function classes coincide. The acknowledged gap limits how directly the classification supports the experimental design claims, without invalidating the general theorem.

Questions for authors
  1. Could the authors provide the parameter counts and training costs (including the projection overhead) for each of the four constructions? Since the strength of equivariant network is its efficiency, I think this is an important comparision to discuss.

  2. Did the authors record how far the model departs from AW=0AW=0 during the 25 optimizer steps (before projection)? Also, have you tried different intervals when doing the projection? I wonder if the performance varies (and by how much).

  3. What practical tasks or settings would the authors expect exotic equivariant constructions to achieve an advantage, and how could we decide when to use them? I am trying to understand the practical application of the findings.

3 comments
AuthorsOpenReview ↗

Response to reviewer otuZ

We thank reviewer otuZ for highlighting of the strengths in the developed theory and for the helpful review. Several of the raised weaknesses and questions were very helpful to us in strengthening the paper, see our answers below.

Weaknesses:

  1. We have aimed not to overstate the experimental conclusions, since the experiment should mainly be regarded as a proof-of-concept of how to apply our theoretical results. Nevertheless, we believe that the main conclusions are interesting, namely that different Ψ\Psi yield different performance and that unconstrained training does not find an optimal equivariant structure. Since unconstrained networks are outperformed by some networks with a particular equivariant structure (Ψ2\Psi_2), they do not find the optimal structure. Our judgement is that the average accuracy is not required for our conclusions, in particular we are not claiming that training with Ψ2\Psi_2-constraints in our setting is better on average than training unconstrained. We welcome further input about specific statements that could be refined or softened.
  2. We agree that the comparison of Ψ1\Psi_1 and Ψ4\Psi_4 was not done in sufficient detail and thank the reviewer for pointing this out. However, it is indeed possible to use our theory to show that with C=0C=0, Ψ1\Psi_1 and Ψ4\Psi_4 do parametrize different families of functions. In short, M:=C+AW/2M:=C+AW/2 is preserved under gauge transforms for all GELU-networks and Ψ4\Psi_4 requires AW=0AW=0, which means that M=0M=0. But there are Ψ1\Psi_1-networks with C=0C=0 and M≠0M\neq0, which are therefore not expressible by any Ψ4\Psi_4-network with C=0C=0. See the blue text just before Section A.1 in the updated PDF for more details.

Questions:

  1. The number of parameters is 304M for the non-equivariant baseline and 152M for all the equivariant versions. In terms of training costs, we did not put an emphasis on optimizing the training throughput. In fact, as reported in Section A.2, all our training runs were equally fast. As the reviewer correctly points out, the implementations in (Bökman et al, 2025) should however allow equivariant networks to train faster than the non-equivariant baseline. Upon further investigation we found that we had a data loading bottleneck in our training runs, which led to the waiting for new data hiding the speed of the different architectures. With this data bottleneck sorted out we find comparable comptational benefits as the ones by Bökman et al. For more details, please see the added Section A.3 in the updated PDF.
  2. We did not log ∥AW∥F\lVert AW\rVert_F before projection, but we logged $\lVert\Delta A\rVert / \lVert A\rVert $, where ΔA=AQQ⊤\Delta A = AQQ^\top and QQ is an orthonormal basis of the column space of WW, i.e., ΔA\Delta A is the quantity subtracted from AA in the projection step. We have ∥ΔA∥/∥A∥=∥AQQ⊤∥/∥A∥=∥AQQ⊤∥∥W∥/(∥A∥∥W∥)≥∥AQQ⊤W∥/(∥A∥∥W∥)=∥AW∥/(∥A∥∥W∥)\lVert \Delta A\rVert /\lVert A\rVert = \lVert AQQ^\top\rVert /\lVert A\rVert = \lVert AQQ^\top\rVert \lVert W \rVert / (\lVert A \rVert \lVert W \rVert) \geq \lVert AQQ^\top W\rVert /(\lVert A\rVert \lVert W\rVert ) = \lVert AW\rVert / (\lVert A\rVert \lVert W\rVert ), so this is an upper bound for the relative deviation of AWAW from 00. The mean of $\lVert \Delta A\rVert / \lVert A\rVert $ over an epoch was at most 0.0860.086 for Ψ3\Psi_3 and 0.0930.093 for Ψ4\Psi_4, with the value going down over the course of training as the learning rate decays. On request from the reviewer we have trained Ψ3\Psi_3-variants (we picked Ψ3\Psi_3 because its performance was stable over seeds) with projection intervals of 1010 and 5050 to compare with the interval 2525 that we used for the main experiments. We find that the performance remains comparable between the different intervals, and that the deviation from AW=0AW=0 goes to zero for all of them. So it might even be possible to increase the projection interval further. For more details, see the blue text in Section A.2 of the updated PDF.
  3. The main practical application of the findings in this paper is the classification of all possible ways that shallow MLPs can be made equivariant and the fact that different choices of weight-tying Ψ\Psi lead to different functions. Our contribution therefore includes both layerwise and exotic constructions. Practically, this gives a precise picture of the design space of equivariant MLPs, which was not known before. We regard it as difficult but important future work to determine what tasks/problems require which Ψ\Psi. This seems to us to be a mainly experimental question, which requires large scale experiments and therefore is out of scope of this paper.

Do these answers resolve the concerns and questions?

I thank the authors for their response, which has addressed my major concerns. I appreciate the new results and examples in the manuscript. I therefore will raise my score to 3.

Thank you again for your helpful review and for engaging with the rebuttal. In particular, thank you for pointing out the missing argument for why Ψ1\Psi_1 and Ψ4\Psi_4 yield different functions. Thank you also for your interested questions, which helped us further flesh out the experimental section.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment about the updated version

Updated version

All of the changes to the paper have been marked blue. With the exception of added figures.

Added corollary

We added a corollary of the main result (Corollary 3.3 to Theorem 3.1 in the updated version), which was previously left implicit. This corollary hopefully clarifies the usefulness of the main result of the paper.

The corollary states that for any equivariant function which is defined by a shallow almost canonical network, that by Theorem 3.1 satisfies equation (17) for some Ψ\Psi, there is another shallow almost canonical network which defines the same function and which satisfies equation (17) for the homomorphism Ψ′\Psi' that is the representative of the conjugacy class of Ψ\Psi.

Added example

We have illustrated the above corollary with the new Example 3.8, that shows how to apply the theory to a small GELU-network. This example is accompanied by the new Figure 3, illustrating the weight sharing structures possible for this example.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.