- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper characterizes equivariant shallow neural networks with uncoupling activations, including ReLU, GELU, and SiLU. It studies intrinsic activation symmetries: affine transformations of pre-activations whose effects can be compensated by output scaling and affine corrections. These identities generate parameter transformations that preserve the network’s computed function. The authors prove that every shallow network can be reduced to a canonical form, which is unique for uncoupling activations once a normalization convention is fixed. They use this result to derive weight constraints characterizing equivariant networks, including both layerwise constructions and exotic constructions with dependencies across layers. They also study their optimization behavior and compare several constructions in the MLP blocks of equivariant vision transformers. Among the tested designs, one layerwise construction achieved the highest reported accuracy, while the exotic constructions underperformed both layerwise alternatives and the unconstrained baseline.
Critical strengths
-
A unified framework covering modern activations. One framework that covers modern activations such as ReLU, GELU, SiLU. It also accommodates vector-valued outputs and general group equivariance, extending the scope beyond scalar invariant ReLU networks.
-
Explicit constraints that capture both layerwise and exotic equivariance. The framework translates functional equivariance into equations on network parameters, including dependencies across layers that standard layerwise constructions miss. This provides a systematic basis for constructing and comparing equivariant architectures.
-
A precise account of parameter redundancy. The canonical-form results establish output-preserving transformations and for uncoupling activations it is unique once normalization is fixed. Consequently, equivalent almost canonical networks are related by a unique gauge transformation, giving a rigorous characterization of their remaining parameter ambiguity.
Critical weaknesses
The authors acknowledge the paper’s main limitations.
- Clarity: The paper would benefit from a preliminary section introducing the group-theoretic concepts needed to follow the analysis. The main text could also explain the homomorphism and its role more explicitly.
- Experimental evaluation: A broader evaluation across additional datasets and architectures would strengthen the submission and provide a clearer understanding of the relative benefits and limitations of the different constructions.
Questions for authors
No Questions.
3 comments
Thank you for the detailed response and clarifications. The authors have addressed my main concerns, and I will maintain my current score.
Thank you
Thank you again for the positive review and for engaging with our rebuttal. Thank you especially for the suggestion to clarify the role of the homomorphism , which helped improve the paper.
Response to reviewer Fzra
We thank the reviewer Fzra for their positive review and charitable characterization of our results.
Weaknesses
We hope that this resolves the concerns.