- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a complete characterisation, under some assumptions, of when multi-head self-attention is equivariant to a symmetry group. The authors prove that for finite groups G, any polynomially parametrised equivariant attention architecture covers only one of many possible equivariant configurations, so in general it loses expressivity when compared with an unconstrained layer of the same size. They give a lower bound on how many such architectures are needed to cover all of them, and show that one model with the size of G times as many heads suffices. Proofs of the results are given in the appendix. The theoretical results are illustrated with several small experiments.
Critical strengths
A complete characterisation of equivariant multi-head self-attention (Theorem 1). The main theorem is an “if and only if” result, so it gives a full description of when multi-head self-attention respects a symmetry. The authors demonstrated that this theorem brings seemingly different designs under one picture: making every head equivariant (Example 3) and the Platonic Transformer (Example 7) turn out to be two extremes of the same structure. The authors also point out that architectural designs in between haven’t been explored in previous work. I think this is the paper’s main contribution, since it shows what is possible when building equivariant attention and suggests that there are new designs to be explored in the future.
An interesting trade-off between equivariance and expressivity (Theorem 2). The second main result showed that building the symmetry into an attention layer of fixed size seems to add assumptions beyond the symmetry itself. A single equivariant architecture can only cover part of what an unconstrained layer of the same size can express. The authors also give a way around this by using |G| times more heads, so the result is not necessarily negative. I found this result very interesting, and it should be relevant to anyone choosing between equivariant versus unconstrained models. As far as I know, this cost has not been shown for attention before, although, as the authors state, a similar effect was shown for linear networks by Kohn et al. (2025).
Critical weaknesses
Practical impact is not yet clear. As I understand it, the trade-off in Theorem 2 comes from counting possibilities rather than from measuring a drop in performance, so I’m not sure how much it matters in practice. The paper also suggests that there are new equivariant designs to try, but I didn’t see any of them tested on a real-world task. The experiments are fairly small and synthetic. They seem to illustrate the theory well, but it would help to see whether the choice of design makes a difference on a more natural task.
Some assumptions and presentation issues. Theorem 2 only applies to “polynomially parameterised” architectures, and I wasn’t sure how restrictive this is, or whether other kinds of designs could avoid the trade-off. Some assumptions in Theorem 1 are also not discussed much, such as no biases or positional encodings, and whether at least three tokens is really needed. More generally, the paper is quite hard to follow without a background in representation theory and algebraic geometry, especially the appendix. There also seems to be a typo in the statement of Theorem 1: R_h is written as W_vᵀW_o, which doesn’t match the definition R_h = W_o W_v in Section 2 (line 119).
Questions for authors
- The teacher–student experiment (Section 4/Fig 4) implements some of the new equivariant designs on a small synthetic task. Have you tried comparing these designs on a larger or real-world task? This would show whether the trade-off in Theorem 2 matters in practice, and whether any of the new designs perform well.
- Is the assumption of at least three tokens in Theorem 1 really needed, or is it only needed for the proof?
4 comments
Reply to Reviewer Cyri - Experiments (W1,Q1)
W1 Practical impact is not clear. [...] there are new equivariant designs [...] experiments are farily small and synthetic. [...] but it would help to see whether the choice of design makes a difference on a more natural task.
Q1 The teacher–student experiment (Section 4/Fig 4) implements some of the new equivariant designs on a small synthetic task. Have you tried comparing these designs on a larger or real-world task? This would show whether the trade-off in Theorem 2 matters in practice, and whether any of the new designs perform well.
The experiments we had in the submitted version mainly answered whether Theorem 1's structure can be found by training. To demonstrate practical relevance, we have included two new experiments (see Section 5 and Appendix F):
- tetrahedral-group-equivariant transformer on QM9
- -equivariant vision transformer on Outex (texture classification)
Our empirical conclusion is that the choice of equivariant attention type does matter, and among the orbit types we have probed, the regular orbit seems to be best performing in terms of accuracy.
We have also written up a heuristic explanation of the observed ordering in Appendix G.
Thank you very much for your responses. I think this work will find a lot of interest at ICLR, and I think that readers will appreciate the new Appendix B containing the background mathematics. Given that you have answered my questions well, I will raise my score to a 4, and my confidence to a 3.
Many thanks again for the feedback and for taking the time to engage in the discussion.
Response to Reviewer Cyri: theory (W2, Q2)
Thank you for the review. We agree that the suggested experiments (W1, Q1) would be informative and have started running them. We will post the results if we manage to finish the experiments before the end of the discussion period. We address the theoretical points here:
L≥3 is also necessary. The theorem would not be true for L≤2 tokens. For L=1 attention is just a linear map x↦∑hRhx that depends only on the OV matrices, so the equivalent characterization of equivariance is just ∑hRh being equivariant. We now give a counterexample for L=2 in Appendix C.5 in the revised version, with a remark after Theorem 1 on the necessity of L≥3.