Layerwise and Exotic Equivariance Constraints in Shallow Neural Networks

We prove the existence and uniqueness of a canonical form for shallow neural networks with activations that are uncoupling, meaning that the activations take pre-activations to linearly independent functions. Prior work shows that ReLU, Tanh and several other activations are uncoupling. We generalize this to also include modern activations such as GELU, SiLU and ELU. Using the canonical form, we characterize all possible equivariant shallow networks with uncoupling activations in terms of equivariance constraints on the weights and biases. Some of the constraints define classical layerwise equivariant neural networks, but some are exotic variants that are not layerwise equivariant, instead having inter-layer constraints. Our results generalize prior work by Agrawal and Ostrowski (2022), who characterized single output invariant shallow ReLU networks. Furthermore, we show that layerwise networks and exotic networks can have qualitatively different optimization behaviour and test the practical impact of different choices of equivariant structures in the MLP blocks of equivariant vision transformers.