acceptodds
Under review as a conference paper at ICLR 2027

Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention

13 views

Abstract

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group , then can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For acting on copies of the regular representation as the token feature space, we show that there are components for eight attention heads. We illustrate this with experiments on artificial tasks and show that the predicted head-cluster structures can arise naturally when the attention layer is pressured to become equivariant.

Primary area:
learning theory
Keywords:
equivariance, equivariant neural networks, self-attention, expressivity, representation theory, algebraic geometry, identifiability, interpretability

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_CyriOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a complete characterisation, under some assumptions, of when multi-head self-attention is equivariant to a symmetry group. The authors prove that for finite groups G, any polynomially parametrised equivariant attention architecture covers only one of many possible equivariant configurations, so in general it loses expressivity when compared with an unconstrained layer of the same size. They give a lower bound on how many such architectures are needed to cover all of them, and show that one model with the size of G times as many heads suffices. Proofs of the results are given in the appendix. The theoretical results are illustrated with several small experiments.

Critical strengths

A complete characterisation of equivariant multi-head self-attention (Theorem 1). The main theorem is an “if and only if” result, so it gives a full description of when multi-head self-attention respects a symmetry. The authors demonstrated that this theorem brings seemingly different designs under one picture: making every head equivariant (Example 3) and the Platonic Transformer (Example 7) turn out to be two extremes of the same structure. The authors also point out that architectural designs in between haven’t been explored in previous work. I think this is the paper’s main contribution, since it shows what is possible when building equivariant attention and suggests that there are new designs to be explored in the future.

An interesting trade-off between equivariance and expressivity (Theorem 2). The second main result showed that building the symmetry into an attention layer of fixed size seems to add assumptions beyond the symmetry itself. A single equivariant architecture can only cover part of what an unconstrained layer of the same size can express. The authors also give a way around this by using |G| times more heads, so the result is not necessarily negative. I found this result very interesting, and it should be relevant to anyone choosing between equivariant versus unconstrained models. As far as I know, this cost has not been shown for attention before, although, as the authors state, a similar effect was shown for linear networks by Kohn et al. (2025).

Critical weaknesses

Practical impact is not yet clear. As I understand it, the trade-off in Theorem 2 comes from counting possibilities rather than from measuring a drop in performance, so I’m not sure how much it matters in practice. The paper also suggests that there are new equivariant designs to try, but I didn’t see any of them tested on a real-world task. The experiments are fairly small and synthetic. They seem to illustrate the theory well, but it would help to see whether the choice of design makes a difference on a more natural task.

Some assumptions and presentation issues. Theorem 2 only applies to “polynomially parameterised” architectures, and I wasn’t sure how restrictive this is, or whether other kinds of designs could avoid the trade-off. Some assumptions in Theorem 1 are also not discussed much, such as no biases or positional encodings, and whether at least three tokens is really needed. More generally, the paper is quite hard to follow without a background in representation theory and algebraic geometry, especially the appendix. There also seems to be a typo in the statement of Theorem 1: R_h is written as W_vᵀW_o, which doesn’t match the definition R_h = W_o W_v in Section 2 (line 119).

Questions for authors
  1. The teacher–student experiment (Section 4/Fig 4) implements some of the new equivariant designs on a small synthetic task. Have you tried comparing these designs on a larger or real-world task? This would show whether the trade-off in Theorem 2 matters in practice, and whether any of the new designs perform well.
  2. Is the assumption of at least three tokens in Theorem 1 really needed, or is it only needed for the proof?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Cyri: theory (W2, Q2)

Thank you for the review. We agree that the suggested experiments (W1, Q1) would be informative and have started running them. We will post the results if we manage to finish the experiments before the end of the discussion period. We address the theoretical points here:

W2 Some assumptions and presentation issues. Theorem 2 only applies to “polynomially parameterised” architectures, and I wasn’t sure how restrictive this is, or whether other kinds of designs could avoid the trade-off. Some assumptions in Theorem 1 are also not discussed much, such as no biases or positional encodings, and whether at least three tokens is really needed.

[...] the paper is quite hard to follow [...] especially the appendix.

There also seems to be a typo [...]: R_h is written as W_vᵀW_o, [...]

  • Most equivariant attention architectures we know of in the literature actually have linear β\beta. For example, specifying a GG-representation on Rd\mathbb{R}^d and constraining WqW_q and WkW_k to be separately equivariant result in linear constraints on WqW_q and WkW_k. We stated the theorem for polynomial β\beta because the generalization was free: the only place the property of this map enters in the proof is the algebro-geometric fact that a polynomial image of an irreducible set is still irreducible.
    • A more careful thought shows that the proof also works for analytic β\beta: Even if β\beta is only analytic and not polynomial, the image β(RP)\beta(\mathbb{R}^P) is still irreducible. So, in the context of Theorem 2, β(RP)\beta(\mathbb{R}^P) is still trapped in one component. In the revised version, Theorem 2 is now stated for analytic β\beta.
    • So anything that bypasses Theorem 2 must involve non-analytic β\beta (or a parameter space that's not Euclidean). This boils down to questions like "how can you parameterize {x=0 or y=0}⊂R2\{x = 0 \text{ or } y=0\}\subset \mathbb{R}^2 nicely with one parameter space?". Even for piecewise analytic maps like ReLU, the number of reachable components is bounded by the number of analytic regions.
  • biases: the theorem generalizes nicely with biases. Key biases are redundant, and the query biases are subject to the same type of constraints as the QK matrices, while the OV biases only need to satisfy a global invariance constraint. We have decided to mention this as a remark after Theorem 1, and include an appendix section (C.4) to elaborate on this. We have not generalized this to the generalized softmax (see Reviewer KqqD's Q1): we believe the situation there is genuinely more complicated, as the key bias is no longer redundant.
  • positional encodings: if the group acts on positions (possibly trivially) and the positional encodings themselves are compatible with the group action, i.e., pg⋅i=ρin(g)pip_{g\cdot i} = \rho_{\text{in}}(g) p_{i}, then Theorem 1 still applies. The more difficult question is how the composition of adding (unconstrained) positional encodings followed by attention can be equivariant. Our proof of Theorem 1 does not extend to this case without significant changes, because the proof uses the permutation equivariance of (vanilla) attention, and adding positional encodings breaks that symmetry. RoPE makes the attention bilinear forms MhM_h position-dependent, so it is also outside the reach of the proof. We will explore this in future work.
  • See Q2 below for the L≥3L\ge 3 assumption.
  • We agree that the presentation of the mathematical parts of the appendix could be improved. To this end, we have added a new section (the new Appendix B) that recalls the relevant mathematics in the revised version.
  • Yes, RhR_h should be WoWvW_oW_v and not Wv⊤WoW_v^\top W_o. Thanks for catching the typo; corrected in the revision.

Q2 Is the assumption of at least three tokens in Theorem 1 really needed, or is it only needed for the proof?

L≥3L\ge 3 is also necessary. The theorem would not be true for L≤2L\le 2 tokens. For L=1L=1 attention is just a linear map x↦∑hRhxx\mapsto \sum_h R_h x that depends only on the OV matrices, so the equivalent characterization of equivariance is just ∑hRh\sum_h R_h being equivariant. We now give a counterexample for L=2L=2 in Appendix C.5 in the revised version, with a remark after Theorem 1 on the necessity of L≥3L\ge 3.

AuthorsOpenReview ↗

Reply to Reviewer Cyri - Experiments (W1,Q1)

W1 Practical impact is not clear. [...] there are new equivariant designs [...] experiments are farily small and synthetic. [...] but it would help to see whether the choice of design makes a difference on a more natural task.

Q1 The teacher–student experiment (Section 4/Fig 4) implements some of the new equivariant designs on a small synthetic task. Have you tried comparing these designs on a larger or real-world task? This would show whether the trade-off in Theorem 2 matters in practice, and whether any of the new designs perform well.

The experiments we had in the submitted version mainly answered whether Theorem 1's structure can be found by training. To demonstrate practical relevance, we have included two new experiments (see Section 5 and Appendix F):

  • tetrahedral-group-equivariant transformer on QM9
  • D4D_4-equivariant vision transformer on Outex (texture classification)

Our empirical conclusion is that the choice of equivariant attention type does matter, and among the orbit types we have probed, the regular orbit seems to be best performing in terms of accuracy.

We have also written up a heuristic explanation of the observed ordering in Appendix G.

Thank you very much for your responses. I think this work will find a lot of interest at ICLR, and I think that readers will appreciate the new Appendix B containing the background mathematics. Given that you have answered my questions well, I will raise my score to a 4, and my confidence to a 3.

Many thanks again for the feedback and for taking the time to engage in the discussion.

Reviewer_KqqDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The submitted paper

  1. provides identification results for multi-head self-attention (MHSA),
  2. shows a full characterization of the equivariant counterparts of these models when the symmetry group acts token-wise on the input,
  3. highlights that, in general, the full class of equivariant MHSA cannot be realized by a single polynomial parameterization of an equivariant architecture of the same size.
Critical strengths
  • Soundness: the paper provides a complete and sound theoreitcal analysis of equivariant MHSA. The mathematics and notation appear consistent although, due to the short time allowed by the review phase, I have not been able to check the proofs in detail.
  • Impact: The paper represent a timely and relevant contribution since equivariant transformers are relevant architectures in the current GDL practice, nevertheless they are relatively understudied from the theoretical viewpoint.
Critical weaknesses

Clarity and readality: the topic addressed by the authors merges different notions coming from machine learning, representation theory and algebraic geometry in unconvetional ways therefore a certain level of complexity and difficulty in accessibility is unavoidable. However, some additional measures to mitigate this issue can be employed. I suggest to better distinguish between parameterization spaces and functional spaces, adding notation clearly referencing these different spaces. For example, the use of the term model can make the exposition difficult to follow on a first reading, as it is not immediately clear whether it refers to a class of functions, a parameterized family, or an individual function realized by a given architecture. While the notation and terminology are formally unambiguous, the distinctions between these objects require some effort to become familiar with, making the paper less accessible than necessary.

I would also better discuss Definition 1, explaining why it is necessary to introduce the polynomial parameterization β\beta and give some additional simple examples from the current literature.

Questions for authors

How would the statements of Theorem 1 and Theorem 2 adapt extending the definition to other forms of self-attention (e.g. substituting expexp in the definition of MHSA with another positive real function)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KqqD (W1, W2, Q1)

Thank you for the review. The presentation tips are helpful, and your question led to a genuine generalization of our Theorems 1 and 2. We respond to each below.

W1 Clarity and readability: the topic [...] merges different notions [...] in unconvetional ways therefore a certain level of complexity and difficulty in accessibility is unavoidable. [...] I suggest to better distinguish between parameterization spaces and functional spaces, adding notation clearly referencing these different spaces. For example, the use of the term model can make the exposition difficult to follow on a first reading, [...]

We agree that the nature of the topic and of the way we posed and solved the problems has made presentation challenging. We tried to mitigate this by using a more concrete language in the main text in the submitted version.

In the revised paper, we have included an additional appendix section (B) that reviews the parts of representation theory and algebraic geometry that are needed in the paper. We hope this will help the interested reader better understand the rest of the appendix.

We have supplied Definition 1 as well as the paragraph above it with more details that clarify the distinction between the different objects involved in Sec. 4. We replaced "model" -> "architecture", as the latter is less likely to be thought of as the realized map. We have also made it clear that "architecture" really just means a fixed parameterized map in Sec. 4. We have introduced the notation F\mathcal{F} for the set of functions realized by unconstrained attention, and FG\mathcal{F}^G for the GG-equivariant ones. The GG-completeness definition is now repeated in terms of these. We hope these changes make Definition 1 and Theorem 2 clearer.

W2 I would also better discuss Definition 1, explaining why it is necessary to introduce the polynomial parameterization and give some additional simple examples from the current literature.

We agree that Definition 1 should have been better motivated. Considering polynomial β\beta is actually more general than needed for most equivariant attention implementations, as they mostly involve linear β\beta. We had stated Theorem 2 for polynomial β\beta merely because the same proof carries over.

Theorem 2's assumption could be further relaxed: the conclusion still holds if β\beta is only analytic.

In the revised paper, we have included a remark after Definition 1 to clarify how existing architectures fit in our picture.

Q1 How would the statements of Theorem 1 and Theorem 2 adapt extending the definition to other forms of self-attention (e.g. substituting in the definition of MHSA with another positive real function)?

We interpret the substitution as follows: instead of exp⁡\exp, one takes an arbitrary S:R→R>0S: \mathbb{R}\rightarrow \mathbb{R}_{>0} for the softmax.

Theorem 1 and Theorem 2 are still true if the function σ:x↦S(x)S(x)+(L−1)S(0)\sigma: x\mapsto \frac{S(x)}{S(x) + (L-1)S(0)} is at least 2H−12H-1 times differentiable at 00 (where HH is the number of heads and LL the number of tokens) and σ(n)(0)≠0\sigma^{(n)}(0)\neq 0 for all n∈{0,1,…,2H−1}n \in \{0, 1, \dotsc, 2H-1\}. The proof idea is largely the same, but the previous proof for S=exp⁡S=\exp cannot be carried over word by word. So, we have decided to add a new section (Appendix C.3) for the proof of the general setting.

I thank the authors for their responses and the additions to the manuscript. These have been clarifying, addressed my concerns, and led to substantial improvements to the manuscript. I also believe that this paper will constitute a relevant contribution to the field. For these reasons, I recommend this paper for clear acceptance and update my score accordingly.

Thank you again for the feedback and for engaging with our response.

Reviewer_VJ8xOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the structure and expressivity of group-equivariant multi-head self-attention (MHSA). The first main theoretical contribution gives a necessary and sufficient characterization of equivariance in terms of the QK and OV matrices. Specifically, equivariance requires the non-dead head clusters to admit a compatible group action, with appropriate constraints on the QK matrices and cluster-summed OV matrices.

The second contribution establishes a lower bound on the number of polynomially parameterized equivariant architectures needed to represent all equivariant functions realizable by an unconstrained MHSA of the same width. The authors show that this number can grow rapidly with the feature dimension, even for a model with just eight attention heads. The theory is accompanied by teacher-student experiments and visualizations of head structures emerging during training.

Critical strengths
  1. The paper addresses an interesting and fundamental question: whether imposing a correct symmetry can restrict expressivity beyond the intended equivariance constraint.
  2. The theoretical analysis provides a necessary and sufficient characterization of equivariant MHSA. The treatment of repeated QK matrices and dead head clusters is particularly interesting, as it also accounts for nongeneric parameter configurations.
  3. The use of Zariski-irreducible components and equivariant vector bundles to study the space of rank-constrained attention maps is interesting. The proof strategy builds on existing algebraic-geometric techniques while adapting them to the nonlinear softmax attention setting.
  4. The experiments provide interesting empirical support for the theoretical characterization, particularly through visualizations of emerging head-cluster structures and group orbits during training.
Critical weaknesses
  1. The expressivity lower bound applies only to exact equivariance under polynomial parameterizations. It is unclear how broadly this extends to other architectures and whether similar limitations hold for approximate equivariance.
  2. The theoretical analysis focuses on a single MHSA layer and does not account for residual connections, normalization, positional encodings, or deeper Transformer architectures.
  3. The lower bound establishes a separation in exact expressivity but does not quantify approximation error. It is unclear whether different equivariant architectures exhibit meaningful performance differences in practice.
  4. Most experiments involve synthetic tasks, and the vision experiment uses an already equivariant backbone, although its final attention layer is unconstrained. The experiments primarily illustrate the theoretical characterization rather than directly measuring the practical consequences of the expressivity trade-off.
  5. The group-averaging construction preserves expressivity by increasing the number of attention heads, but the corresponding computational trade-offs are not evaluated empirically. Comparisons of runtime, memory, and downstream performance would strengthen the practical motivation.
Questions for authors

Do the structural characterization and expressivity limitations extend to standard Transformer blocks containing residual connections, normalization, and MLPs?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VJ8x: theory (W1, W2, Q1)

Thank you for your review. We agree that the suggested experiments would be interesting and have started working on them. We hope to be able to address the raised experimental concerns (W3, W4, W5) later during the discussion phase. The theoretical concerns are addressed below:

W1 The expressivity lower bound applies only to exact equivariance under polynomial parameterizations. It is unclear how broadly this extends to other architectures and whether similar limitations hold for approximate equivariance.

  • Only polynomial parameterizations: current equivariant attention layers mostly use linear parameterizations and we have included a remark after Definition 1 to clarify this. We had considered polynomial parameterization maps because the generalization is immediate. In fact, the statement still holds even if the parameterization is only analytic (see Reply to Cyri W2). However, once one allows arbitrary parameterizations, we have no control over the image β(RP)\beta(\mathbb{R}^P) anymore. It could be arbitrarily ugly and complicated, but intuitively this means β\beta should also be complicated to implement on a computer in this case.

  • Other architectures: one possible generalization is the following: for an architecture that is the sum of HH identical units, i.e. fθ(x)=∑h=1Hϕθh(x)f_\theta(x) = \sum_{h=1}^H \phi_{\theta_h}(x), the conclusion "fθf_\theta is GG-equivariant iff GG permutes units with the θh\theta_h tied equivariantly according to the GG-action" is possible if (1) there is adjunction: GG acts on θ\theta so that g−1fθ(gx)=fg−1θ(x)g^{-1}f_{\theta}(gx) = f_{g^{-1}\theta}(x), and (2) the parameterization map is injective up to permuting the units. For MHSA and shallow MLP (1 hidden layer), (1) holds and (2) is true away from a special zero-measure subset Θ0\Theta_0 (e.g. where QK matrices coincide or where first-layer weights coincide) of the single-unit parameter space Θ\Theta.

  • Approximate equivariance: our experiments already suggest that approximate equivariance should lead to structures close to Theorem 1's prediction, as the models are only approximately equivariant after training. We agree that a rigorous approximate version of Theorem 1 would be desirable. This would be related to the stability of the realization map of attention. Ideally, we would like to prove something like "if the equivariance error at parameter θ\theta is ϵ\epsilon, then a1(ϵ)≤inf⁡θ0∈equiv. locusdist(θ,θ0)≤a2(ϵ)a_1(\epsilon) \le \inf_{\theta_0\in \text{equiv. locus}}\mathrm{dist}(\theta, \theta_0) \le a_2(\epsilon)", with some useful notion of distance. We have not been able to prove such a statement.

W2 The theoretical analysis focuses on a single MHSA layer and does not account for residual connections, normalization, positional encodings, or deeper Transformer architectures.

Q1 Do the structural characterization and expressivity limitations extend to standard Transformer blocks containing residual connections, normalization, and MLPs?

We only have a partial answer. For one transformer block with residual connections and an MLP (but without norms), assuming that both the MLP block, including the residual stream, and Id+∑hRh\mathrm{Id} + \sum_h R_h are injective, equivariance of the whole block is equivalent to the attention and MLP layers being individually equivariant. The characterization of the equivariance locus of MLP will depend on the symmetries of the activation (see W1). We don't think this result is satisfactory because of the injectivity assumptions and omission of norms, so we decided not to include it in the paper. But this is a future direction.

  • Normalization: Layer norm depends on a choice of basis on the token feature space VinV_{\text{in}}, and most choices will simply break the symmetry (except for permutation representations, then layer norm is itself equivariant but the proof of Theorem 1 will need to be adapted to inputs on the sphere). The question of whether and how attention's parameters can be chosen to compensate for that doesn't share the same structure as Theorem 1.
  • Positional encodings: if the positional encodings are compatible with the group action (pg⋅i=ρin(g)pip_{g\cdot i} = \rho_{\text{in}}(g) p_{i}) with the group acting on positions (possibly trivially), then Theorem 1 still applies. Currently we do not have the answer to the more difficult question of how adding positional encodings followed by attention can be equivariant jointly: Theorem 1's proof relies on the permutation symmetry of vanilla attention and adding positional encodings breaks that symmetry. RoPE similarly makes the attention bilinear forms MhM_h position-dependent, so it is also outside the reach of the proof.
AuthorsOpenReview ↗

Addendum to W1: generalization to other architectures

For a finer characterization, which also produces the "GG permutes clusters" picture, one can consider architectures with the following additional properties: (1) the single-unit parameter space is a product, Θ=P×Q\Theta = P\times Q, (2) the parameterization factorizes through (P×Q)H→P⊗F[Q],(r,m)↦∑hrh⊗δmh(P\times Q)^H \rightarrow P \otimes \mathcal{F}[Q], (r, m) \mapsto \sum_h r_h\otimes \delta_{m_h}, where F[Q]\mathcal{F}[Q] is the free vector space over QQ and δm\delta_m is the point mass at mm, and (3) the parameterization by P⊗F[Q]P\otimes \mathcal{F}[Q] is injective into function space. (1) and (2) are true for MHSA and shallow MLP. (3) is true for MHSA if L≥3L\ge 3 and for MLP if the activation doesn't have "accidental symmetries" like ReLU.

Thank you for the clarifications.

AuthorsOpenReview ↗

Reply to Reviewer VJ8x - Experiments (W3, W4, W5)

Reply to Reviewer VJ8x - Experiments

W3 The lower bound establishes a separation in exact expressivity but does not quantify approximation error. It is unclear whether different equivariant architectures exhibit meaningful performance differences in practice.

W4 Most experiments involve synthetic tasks, and the vision experiment uses an already equivariant backbone, although its final attention layer is unconstrained. The experiments primarily illustrate the theoretical characterization rather than directly measuring the practical consequences of the expressivity trade-off.

We have included two new experiments in the revised paper (see Sec. 5 and Appendix F):

  • tetrahedral-group-equivariant transformer on QM9
  • D4D_4-equivariant vision transformer on Outex (texture classification)

Our empirical conclusion is that the choice of equivariant attention type does matter, and among the orbit types we have probed, the regular orbit seems to be best performing in terms of accuracy.

We have also written up a heuristic explanation of the observed ordering in Appendix G.

W5 The group-averaging construction preserves expressivity by increasing the number of attention heads, but the corresponding computational trade-offs are not evaluated empirically. Comparisons of runtime, memory, and downstream performance would strengthen the practical motivation.

This is important and but it will depend on the implementation of the equivariant attention layers, and there are several possibilities (e.g. in spatial domain or in Fourier domain, and in the Fourier domain one can choose how the irrep tensors are stored). Our current implementation is not optimized, so we have decided to leave this for future work.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revised paper uploaded: summary of changes

We have uploaded a revised version of the paper. Changes are marked in orange.

Theorems 1 strengthened: Reviewer KqqD asked whether Theorems 1&2 apply to more general forms of attention. The answer is yes, interpreting more general as replacing exp⁡\exp by an arbitrary positive function SS in the softmax. With clean conditions on SS, the statement of Theorem 1, and hence Theorem 2, is still true. We have added a remark after Theorem 1 as well as a new section in the appendix for the proof. A version of Theorem 1 with biases is also stated and proved in the appendix. Theorem 1 is now stated without the orthogonality assumption. This involved changing the GG-action on MhM_h to ρin(g−1)⊤Mhρin(g−1)\rho_{\text{in}}(g^{-1})^\top M_h \rho_{\text{in}}(g^{-1}) instead of conjugation.

Theorem 2 strengthened: All reviewers commented on the polynomial assumption of Theorem 2, either requesting more motivation or raising concerns about the applicability in practice. We believe this is not a strong assumption in practice: most variants of equivariant attention in the literature fall in the linear case. In the submitted version we stated Theorem 2 for polynomial parameterizations because the proof carries over verbatim.

On the other hand, while engaging with Reviewer Cyri's question, it became clear to us that Theorem 2 holds even for analytic parameterizations after a small modification of the proof. We have thus changed all occurrences of "polynomial(-ly)" to "analytic(-ally)" in the abstract, Sec. 4, and Appendix D.

New math appendix: Reviewers KqqD and Cyri pointed out that the math-heavy parts of the paper, especially the appendix, can be hard to follow for some readers. To alleviate this, we have added a new section (Appendix B) that summarizes the relevant material from algebraic geometry and representation theory used in our paper. The section is mostly self-contained, with references to standard textbooks in case the reader wants to dive deeper.

Additional changes:

  • fixed several typos and added some minor clarifications
  • References:
    • added Tran et al. (ICLR 2025) and Wang & Wang (NeurReps 2025): Tran et al.'s Theorem 3.2 and Wang & Wang's Theorem 2 are equivalent, up to slight differences in the assumptions, to Henry's Theorem 2.1, which we already cite in the submitted version. Both predate Henry, and we choose to cite all three in the revision.
    • more existing equivariant attention layers
AuthorsOpenReview ↗

Paper updated with experiments

We have added two new experiments to the paper in order to address the comments of Reviewers VJ8x and Cyri.

This involved extending Section 5 with brief summaries of the results, with details given in Appendix F. We also included a new Appendix G, where we give a heuristic explanation of the observed dependence of performance on orbit types.

  • Fixed a mix-up between two Kundu & Kondor papers: arXiv:2510.18813 -> arXiv:2405.15932
  • Fixed a typo in Appendix B.

No other changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.