acceptodds
Under review as a conference paper at ICLR 2027

Symmetry Slippage in Equivariant ML: Creation, Computation, and Corrections

Abstract

Symmetry-based inductive biases are prevalent throughout machine learning for science and beyond. Often, the symmetries realized by an architecture differ from those prescribed. This mismatch can occur in two ways: when prescribed symmetries are not realized (*under-equivariance*) and when *non*-prescribed symmetries *are* realized; we call this *symmetry slippage*. Although famous examples of symmetry slippage abound, a systematic study of its origin, estimation, and corrections has yet to exist. This work aims to initiate such a study. We formulate symmetry slippage in terms of subarchitectural decompositions, and use this framing to identify three common mechanisms by which contemporary techniques can unintentionally create it. These creation mechanisms recover the familiar examples, theoretically ground popular heuristics, and identify symmetry slippage across nine active research areas. While we find that guaranteeing an absence of symmetry slippage is hard in general, we prove conditions under which our decomposition framing is exact. We use this result to exactly compute, for the first time, the realized symmetries of a classical family subsuming e.g. convolutional architectures. Finally, we study *grippage encodings* (GEs), simple symmetry-correcting features that restore expressivity. We find that GEs improve empirical performance across our nine case studies.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
equivariant learning, geometric deep learning, AI4Science, symmetry slippage

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qMrHOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work discusses symmetry slippage and the ways to identify and correct it. Symmetry slippage occurs when a model has more symmetry constraints than exist in the data. This can worsen performance as the model isn't able to express exactly what needs to occur. They present grippage encodings, a way to fix this over equivariance by restoring the lost expressivity. The test across 9 test cases and find that it improves performance across all of them.

Critical strengths

Not only does this work properly identify a key area in equivariant machine learning, but it applies an effective resolve across a large variety of test cases. They show that their method not only thoroughly improves performance, but also works across a variety of the failure cases. I believe this is a very strong way to show that their method works and should be applied in scenarios of unknown symmetry.

Critical weaknesses

The biggest weakness of this work is that they don't compare their method against a non-equivariant model that has learned through data augmentation as has been shown to outperform equivariant networks that are not properly structured to match the real symmetry of the data. Showing that they outperform these data augmentation models (like AlphaFold3 for example) would further show the benefit of their method.

Questions for authors
  1. How does your method compare to a method that combines equivariant and non-equivariant functions to produce symmetry breaking? What advantages does your method have over symmetry breaking methods, does it ensure retained equivariance?
  2. How minimal is the retraining needed for grippage encodings?
3 comments
AuthorsOpenReview ↗

1 - Response to Reviewer qMrH

We thank the reviewer for their thoughtful review and positive assessment of the manuscript. We appreciate their recognition of the importance of symmetry slippage and the effectiveness and broad applicability of our framework for identifying and correcting it. Below, we address the reviewer’s concerns.


W1 The biggest weakness of this work is that they don't compare their method against a non-equivariant model that has learned through data augmentation as has been shown to outperform equivariant networks that are not properly structured to match the real symmetry of the data. Showing that they outperform these data augmentation models (like AlphaFold3 for example) would further show the benefit of their method.

We thank the reviewer for their insight. We have conducted two additional experiments. The first compares a symmetry-slipped MLP, its GE-corrected counterpart, and a non-equivariant version trained with data augmentation on the Lorentz-invariant learning task of Case Study 1. As the reviewer foresaw, the augmented architecture decisively outperformed the symmetry-slipped architecture. After considerably more optimization, it still underperformed the symmetry-corrected architecture (i.e., +GE) in-distribution, and substantially underperformed out-of-distribution. This illustrates that data augmentation can provide an effective alternative to architectures exhibiting symmetry slippage, but that explicitly correcting the unintended symmetry can offer significant advantages in both training efficiency and generalization under the prescribed transformations.

ArchitectureID NMSE ↓OOD NMSE ↓Epochs
Symmetry-slipped MLP0.4904 ± 0.00020.4588 ± 0.0001313
Non-equivariant MLP + augmentation0.0610 ± 0.00160.3655 ± 0.02031,563
Symmetry-corrected MLP (+GE)0.0314 ± 0.00020.0274 ± 0.0002188

We also conducted an experiment comparing GEs against three commensurably simple alternatives: canonicalization, data augmentation, and a hybrid approach described in Q1.1 below. Specifically, we considered an SO⁡(2)\operatorname{SO}(2)-invariant learning task with a parameter λ\lambda controlling its sensitivity to reflections R∈O⁡(2)−SO⁡(2)R\in\operatorname{O}(2)-\operatorname{SO}(2). As noted in our Discussion, GEs guarantee that the hypothesis class contains functions that correct the unintended symmetry, but do not guarantee that training will find such functions. To investigate whether transformation behavior in practice exploits this restored expressivity, we varied λ\lambda from 0.1 (little reflection sensitivity) to 0.9 (strong reflection sensitivity), measuring predictive performance, learned transformation behavior, and training efficiency. The details are in Appendix B.4, page 28 of the revised manuscript.

λ\lambdaArchitectureMSE (×103\times 10^3) ↓Reflection error (×103\times 10^3) ↓Epochs ↓
0.1Slipped MLP1.71481.60—
MLP + GE0.0071.9342
Canonicalization0.0172.06102
MLP + augmentation0.0566.33517
Hybrid0.0237.58150
0.5Slipped MLP41.929408.01—
MLP + GE0.0153.8735
Canonicalization0.0335.1896
MLP + augmentation0.63318.02490
Hybrid0.06511.86125
0.9Slipped MLP135.087734.41—
MLP + GE0.0266.6871
Canonicalization0.0499.19104
MLP + augmentation0.10312.11704
Hybrid0.15420.48388

We found that the GE-corrected architecture learned approximately reflection-invariant functions when λ=0.1\lambda=0.1, but increasingly reflection-sensitive functions as λ\lambda increased, closely matching the target's transformation behavior throughout. Direct canonicalization achieved only slightly worse predictive performance, but required greater training effort; these were exacerbated for the augmentation model. These results suggest that GEs can yield transformation behavior making restored expressivity accessible to optimization in practice with lower adaptation cost.

AuthorsOpenReview ↗

2 - Response to Reviewer qMrH

Q1.1 How does your method compare to a method that combines equivariant and non-equivariant functions to produce symmetry breaking?

We thank the reviewer for their question. It is worth mentioning that grippage encodings are a method that combines equivariant and non-equivariant functions to correct symmetry slippage; in the special case that symmetry slippage is over-equivariance, this correction is via symmetry-breaking. Specifically, GEs append features that respect the prescribed transformation law but need not respect the additional, unintended symmetries of the original architecture. The distinction from simply combining an equivariant architecture with an unrestricted non-equivariant component is that GEs are designed to break unprescribed symmetries without sacrificing prescribed equivariance. To empirically demonstrate the distinction, we have included this comparison in the latter table above (Table 2 in Appendix B.4 of the revised PDF).

Q1.2 What advantages does your method have over symmetry breaking methods?

In the special case where symmetry slippage is over-equivariance, grippage encodings may be viewed as a symmetry-breaking method, rather than an alternative. Notably, this is a somewhat different sense of “symmetry breaking” from that often studied in the literature. In this setting of over-equivariance, GEs concern symmetry breaking at the level of an architecture's hypothesis class, in the sense that an unintended symmetry is considered broken once the architecture can realize some function violating it. In contrast, much of the symmetry-breaking literature studies individual functions and, more specifically, the possibility of producing outputs with smaller stabilizers than their inputs. One recent paper which utilizes symmetry-breaking in a similar “architecture-level” sense as us is [1].

However, symmetry slippage is more general than over-equivariance; indeed, the manuscript describes instances of over-quotienting that is not over-equivariance (e.g. WL test for GNNs, Case Study 4), and extension-mistyping (Case Studies 7-9) is never over-equivariance except in trivial cases. It is for this reason that we tend to refer to GEs as ‘symmetry-correcting features’ throughout the paper rather than ‘symmetry-breaking’ features: the notion is more general than symmetry-breaking for us.

Q1.3 Does it ensure retained equivariance?

Yes: the grippage encodings used in all of our experiments retain the prescribed equivariance. In principle, they do not ensure by construction that the new architecture’s transformation law is exactly the prescribed law, however; that is more subtle, and is where our theoretical results in Section 4 come into play.

Q2 How minimal is the retraining needed for grippage encodings?

We thank the reviewer for the question. The retraining required to exploit a GE depends on where it is inserted and the architecture's downstream components. Theoretically, our GE results concern the expressivity of the augmented architecture and do not require retraining the original parameters; however, optimizing the newly available expressivity generally requires training downstream components. We demonstrate in Case Study 5, where we retrain only the affine readout of an ECG foundation model while keeping the backbone frozen. A systematic investigation of the minimum retraining required across architectures remains an interesting direction for future work.


[1] Abhinav Goel, Derek Lim, Hannah Lawrence, Stefanie Jegelka, and Ningyuan Huang. Any-Subgroup Equivariant Networks via Symmetry Breaking. arXiv:2603.19486, 2026.

Thank you for your thorough response and answers. I greatly appreciate you taking the time to answer my questions so thoroughly.

Reviewer_f49NOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The submitted paper present a thorough analysis of the instantnces in which a machine learning model is designed to perform a certain equivariant task but unintendedly introduces additional symmetries or remove necessary ones, calling the former phenomenon symmetry slippage while addressing the latter as under-equivariance. Furthermore, they identify three different ways to create symmetry slippage: group-closure, over-quotienting and extension-mistyping. Finally, to amend for symmetry slippage, the authors introduce grippage encodings and validate their effectiveness empirically.

Critical strengths
  • The authors introduce symmetry slippage generalizing in a meaningful way the concept of over-equivariance already present in literature.
  • They present an in depth analysis and taxonomy of these phenomena, presenting ways for constructing models with these properties, and showing approaches to solve these issues.
  • They present a wide number of relevant examples fitting, and hence validating, the taxonomy they introduced
Critical weaknesses
  1. Reproducibility: Authors do not provide supplementary material, therefore the code for reproducing experiments is unavailable.
  2. Clarity and readability: the paper is particularly dense and difficult to access. Instead of working with nine different examples at the end of the main body of the paper, I would suggest to chose a few of them and use them as working examples since the beginning. This could better explain and motivate each choice in the language adopted by the authors which, in my opinion, looks obscure and difficult to parse. Moreover, even if the paper is directed to researchers in geoemtric deep learning, the authors assume advanced mathematical knowledge which I do not find appropriate to give for granted. For example, I doubt that Example 2 effectively serves its intended purpose of helping readers understand the notion of group-closure. The example relies on concepts from algebraic geometry, such as the Zariski topology, that may fall outside the background of some readers. Although the authors refer readers to Cox's textbook, I suspect that even explicitly providing a definition of the Zariski topology would not be sufficient to make the example and the underlying theory more accessible.

I am currently proposing a negative score mainly due to the readability issues. However, I am open to discussion and, in case the authors could remedy these issues, I am inclined to reassess my score.

Questions for authors

The authors cite the work on Weisfeiler-Leman test as a constraint for MPNNs which cannot be captured by the notion of equivariance, can the presented theory be generalized to work with equivalence relations between inputs instead of invariance and equivariance?

4 comments
AuthorsOpenReview ↗

1 - Response to Reviewer f49N

We thank the reviewer for their thoughtful and constructive review. We appreciate their recognition of the significance of symmetry slippage, the depth of our theoretical analysis, and the breadth of our empirical validation. Below, we address the stated concerns.


W1 Reproducibility: Authors do not provide supplementary material, therefore the code for reproducing experiments is unavailable.

We thank the reviewer for pointing this out. We have now uploaded an anonymized repository as supplementary material, including the code necessary for reproducing experiments. If accepted, we commit to making public the repository alongside the camera-ready version.

W2 Clarity and readability: the paper is particularly dense and difficult to access. Instead of working with nine different examples at the end of the main body of the paper, I would suggest to chose a few of them and use them as working examples since the beginning. This could better explain and motivate each choice in the language adopted by the authors which, in my opinion, looks obscure and difficult to parse.

We appreciate the reviewer’s suggestion. Following the reviewer’s advice, we have reoriented the exposition so as to foreground (1) simple motivational examples and (2) important examples from our nine case studies. We hope these help motivate and justify the language used. In particular,

  1. We now motivate symmetry slippage in Section 2 with a toy example involving SO⁡(2)⊂O⁡(2)\operatorname{SO}(2) \subset \operatorname{O}(2). We then revisit this example throughout Section 3.2, once in a new leading paragraph conceptually motivating our creation mechanisms, and again within each creation mechanism’s subsection to demonstrate how the formalism we introduce generalizes a now-familiar, concrete scenario. We have also added Figures 1 and 2 to visually aid in these motivating discussions.
  2. We now foreshadow the core takeaways from our case studies in Sections 2 and 3. For instance, in Section 2 we contrast our toy SO(2) example, for which simply appending det⁡[u,,v]\det [u \\, \\, v] repairs the symmetry slippage, with a simple modification where the dot product’s addition is replaced by subtraction. This yields the indefinite case SO+(1,1)SO^+(1,1), for which we allude to the potentially surprising fact (made precise in Section 3 and featuring in Case Studies 1 and 2) that no analytic polynomial features can repair the latter. We hope that emphasizing examples such as this illustrates early on that symmetry slippage can be subtle, and motivates thus the subsequent formalism.

To further improve clarity and readability, we have made the following additional revisions:

  • We have rewritten Section 2 so that the notion of transformation law we employ is motivated conceptually and with examples, before introducing the ambient transformation groups J,KJ, K (we now call them JXJ_X and JYJ_Y) and ambient group action Γ↷X\Gamma \curvearrowright X needed for a precise definition.

  • We have rewritten both the exposition surrounding, and precise definitions of, our creation mechanisms and their properties. We have rewritten the creation mechanism Definitions 2, 5, 7 and Example 2 to be less technical while (largely) maintaining rigor. We also deferred what was Prop. 9 to Appendix B, as its main use is as a technical lemma for the appendix proofs of extension-mistyping symmetry slippage in Case Studies 7-9. We hope these adjustments have made the conceptual simplicity of what are now Props. 3, 6, and 8 more transparent and have made the theoretical contributions of Section 4 easier to understand.

W3 Moreover, even if the paper is directed to researchers in geoemtric deep learning, the authors assume advanced mathematical knowledge which I do not find appropriate to give for granted. For example, I doubt that Example 2 effectively serves its intended purpose of helping readers understand the notion of group-closure. The example relies on concepts from algebraic geometry, such as the Zariski topology, that may fall outside the background of some readers. Although the authors refer readers to Cox's textbook, I suspect that even explicitly providing a definition of the Zariski topology would not be sufficient to make the example and the underlying theory more accessible.

We thank the reviewer for their comment. The revised manuscript rewrites Example 2 in a manner that is slightly informal but avoids the algebraic geometry altogether, presenting only the consequences used by our case studies. Of course, some algebraic geometry is still needed to derive these consequences, and so we added a new appendix section (Appendix B.1) providing a self-contained exposition of Example 2, introducing the relevant algebro-geometric concepts as needed without assuming prior knowledge.

AuthorsOpenReview ↗

2 - Response to Reviewer f49N

Q1 The authors cite the work on Weisfeiler-Leman test as a constraint for MPNNs which cannot be captured by the notion of equivariance, can the presented theory be generalized to work with equivalence relations between inputs instead of invariance and equivariance?

This is a good question! We understand the question as follows: The transformation-law formalism is sufficiently general to encode arbitrary input equivalence relations, but our treatment takes prescribed laws to arise from explicitly specified group actions. More general equivalence relations enter only when analyzing realized transformation laws, particularly through over-quotienting, as in the WL example and Case Study 4. Can the theory be generalized to allow prescribed laws that encode invariance to equivalence relations beyond those induced by group actions?

While over-quotienting can accommodate such extensions, in general our current theory does rely on the structure of group actions. One could technically contrive JXJ_X and Γ\Gamma to encode arbitrary equivalence relations as group orbit relations, but this would not necessarily yield meaningful instances of group-closure or extension-mistyping, and we are not aware of settings where doing so would be natural. We are likewise unsure how deeper theoretical results such as (what is now) Theorem 11 would generalize. This would all make for exciting future work.

I thank the authors for their answers.

However, I believe that they have not properly addressed the main issue I raised in my review. In particular, the paper, its language, and its examples are not adequate to connect the presented theory with the geometric deep learning community.

Although the proposed examples are useful and highlight mathematical issues, they fail to illustrate how these issues could arise in models relevant (or potentially relevant) to the geometric deep learning community.

The language also remains more oriented toward mathematicians and, hence, difficult to access for the broader audience of machine learning conferences.

Finally, the figures added during the rebuttal are difficult to interpret and do not seem to help readers better understand the concepts presented in the paper.

For these reasons, I do not consider the manuscript, as presented, suitable for publication at a machine learning conference, and I therefore recommend rejection, although the topic itself could be interesting.

AuthorsOpenReview ↗

Response to Reviewer f49N

However, I believe that they have not properly addressed the main issue I raised in my review. In particular, the paper, its language, and its examples are not adequate to connect the presented theory with the geometric deep learning community. Although the proposed examples are useful and highlight mathematical issues, they fail to illustrate how these issues could arise in models relevant (or potentially relevant) to the geometric deep learning community.

We thank the reviewer for their continued engagement, and for clarifying their concerns regarding the connection between our theoretical framework and its relevance to GDL. We would like to respectfully note that, apart from the expository SO(2)⊂O(2)SO(2) \subset O(2) example, our highlighted examples connect our theory to current problems in the geometric deep learning community. This includes both the examples found within the main exposition and the nine case studies in Section 5. For instance:

  • Example 1, Subexample 2 describes how the well-studied family of A-MLPs (e.g. G-CNNs [1]) arise as an example of group-closure. Independently, Theorem 11 provides conditions under which an architecture’s transformation law can be computed from subarchitectures. Then, Props. 12-13 apply these conditions to answer an open GDL question from 2017, namely, computing the exact symmetries of A-MLPs [2].

  • Example 1 also appears in Case Study 3, where it is shown that current Adjoint-equivariant architectures exhibit symmetry slippage.

  • Example 2, and the SO+(1,1)SO^+(1,1) example that foreshadow it, appear in Case Studies 1-2, where it is discovered that current Lorentz-equivariant architectures exhibit symmetry slippage and current units-equivariant architectures exhibit symmetry slippage.

  • The remaining Case Studies 4-9 locate distinct occurrences of symmetry slippage in active research, in each instance using Section 3 to identify the architectural component that created it. These case studies are quite diverse, ranging from symmetry slippage in equivariant neural operators to equivariant architectures for magnetic materials.

We hope these connections help demonstrate the direct relevance of our theoretical framework to contemporary geometric deep learning research, and that the reviewer might reconsider this central aspect of their assessment.


[1] Taco Cohen and Max Welling. Group Equivariant Convolutional Networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 48, pp. 2990–2999, 2016.

[2] Siamak Ravanbakhsh, Jeff Schneider, and Barnabás Póczos. Equivariance Through Parameter-Sharing. In Proceedings of the 34th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 70, pp. 2892–2901, 2017.

Reviewer_d1s8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces symmetry slippage, a term for cases where a neural network architecture has symmetries beyond those intended by its designers. The authors describe three mechanisms that can lead to this phenomenon: group-closure, over-quotienting, and extension-mistyping.

They then develop several theoretical results concerning the composition and computation of architectural symmetries, including an exact characterization for a particular family of parameter-sharing MLPs. Finally, they propose Grippage Encodings (GEs), which add features to break unwanted symmetries, and evaluate them in nine case studies.

The paper tries to bring several known problems in equivariant learning into one theoretical framework.

Critical strengths

The problem itself is worth consideration. There is a meaningful difference between designing a network to respect a certain symmetry and knowing exactly which symmetries the resulting architecture enforces.

Critical weaknesses

Poor writing and unnecessarily complicated presentation. I found this paper frustrating to read. Here I provide some examples:

  • Section 2 (lines 110–120) introduces JJ, KK, Γ\Gamma, HH, several group actions, and transformation laws almost all at once, without a simple example explaining why these objects are needed. This happens repeatedly throughout the paper. Definitions pile up, followed by propositions whose conclusions often seem obvious once the assumptions are understood.
  • Proposition 9 even uses the phrase "with certain properties" without stating the assumptions explicitly. I have no problem with technical mathematics, but the notation here often makes relatively simple ideas unnecessarily difficult to follow.

Limited scientific significance and unclear practical value. After working through the formalism, I still struggle to see what I have gained.

  • Proposition 1 and several results in Section 3 largely follow from their assumptions. The more interesting exact symmetry results are demonstrated only for a restricted family of parameter-sharing MLPs (Theorem 12 and Propositions 13–14).
  • The nine case studies do not convincingly show that the framework helps discover problems that would otherwise be difficult to identify. GE is also unconvincing as a practical method: direct retyping achieves better mean performance in Tables 9–11, while the improvement on the broader ECG task in Table 7 is very small. I expected a much clearer demonstration of why this framework is useful beyond describing known architectural limitations in a different language.
Questions for authors

How much of the formalism is actually necessary to understand and use this framework?

Could the authors rebuild the background around one explicit neural architecture and one unintended transformation, introducing JJ, KK, Γ\Gamma, HH, the paired-action notation, and the sub architecture decomposition only when they become necessary?

More importantly, which conclusions genuinely depend on distinguishing all these ambient groups and actions? I would like to understand what would be lost if the formulation were simplified. Right now, it is difficult to tell which abstractions are mathematically essential and which merely add another layer of notation.

What is the most important new conclusion that follows from this framework? Beyond the exact result for the restricted (A)-MLP family, I would like to see a concrete example where the proposed theory reveals something that would not follow as directly from existing equivariance or expressivity analyses.

What is the actual advantage of grippage encodings (GE) over direct corrections?

Given the retyping results in Tables 9–11 and the limited improvements on the native ECG benchmark, I would like to understand when GE is preferable to simpler alternatives. A controlled comparison of predictive performance, transformation behavior, and adaptation cost would make the methodological contribution much more convincing.

6 comments
AuthorsOpenReview ↗

1 - Response to Reviewer d1s8

We thank the reviewer for their thoughtful and detailed review, including their recognition of symmetry slippage as a problem worth studying. We address the reviewer’s concerns below.


W1.1: Section 2 (lines 110–120) introduces J, K, Gamma, H, several group actions, and transformation laws almost all at once, without a simple example explaining why these objects are needed. This happens repeatedly throughout the paper.

Following the reviewer’s advice, we have reoriented the exposition so as to foreground (1) simple motivational examples and (2) important examples that justify the abstraction. In particular,

  • (1) We now motivate symmetry slippage in Section 2 with a toy example involving SO⁡(2)⊂O⁡(2)\operatorname{SO}(2) \subset \operatorname{O}(2). We then revisit this example throughout Section 3.2, once in a new leading paragraph conceptually motivating our creation mechanisms, and again within each creation mechanism’s subsection to demonstrate how the formalism we introduce generalizes a now-familiar, concrete scenario. We have also added Figures 1 and 2 to visually aid in these motivating discussions.

  • (2) We now foreshadow the core takeaways from our case studies in Sections 2 and 3. For instance, in Section 2 we contrast our toy SO(2) example, for which simply appending det⁡[u,,v]\operatorname{det}[u \\, \\, v] repairs the symmetry slippage, with a simple modification where the dot product’s addition is replaced by subtraction. This yields the indefinite case SO+(1,1)SO^+(1,1), for which we allude to the potentially surprising fact (made precise in Section 3 and featuring in Case Studies 1 and 2) that no analytic polynomial features can repair the latter. We hope that emphasizing examples such as this illustrates early on that symmetry slippage can be subtle, and motivates thus the subsequent formalism.

We have also rewritten Section 2 so that the notion of transformation law we employ is motivated conceptually and with examples, before introducing the ambient transformation groups JJ, KK (we now call them JXJ_X and JYJ_Y) and ambient group action Γ↷X\Gamma \curvearrowright X that allow for a precise definition.

W1.2: Definitions pile up, followed by propositions whose conclusions often seem obvious once the assumptions are understood. Proposition 9 even uses the phrase "with certain properties" without stating the assumptions explicitly. I have no problem with technical mathematics, but the notation here often makes relatively simple ideas unnecessarily difficult to follow.

We thank the reviewer for their comment and have better integrated the definitions and properties in Sections 2-3 with their consumers in Sections 4-5. In particular, we have rewritten both the exposition surrounding, and precise definitions of, our creation mechanisms and their properties. In addition to adding visuals and bookending these passages with conceptual remarks and examples as discussed in our response to W1.1, we have rewritten the creation mechanism Definitions 2, 5, 7 and Example 2 to be less technical while (largely) maintaining rigor. We also deferred Prop. 9, a technical lemma used to verify extension-mistyping in our case studies which we agree was stated too informally in the main text, to Appendix B. In addition to benefiting the overall exposition, we hope these adjustments have clarified the statements of what are now Props. 3, 6, and 8, which, in contrast to the theoretical contributions of Section 4, are indeed intended to be straightforward once the relevant definitions are understood.

AuthorsOpenReview ↗

2 - Response to Reviewer d1s8

W2.1: Proposition 1 and several results in Section 3 largely follow from their assumptions. The more interesting exact symmetry results are demonstrated only for a restricted family of parameter-sharing MLPs (Theorem 12 and Propositions 13–14).

We thank the reviewer for their comment.

  1. We would like to first clarify that Theorem 12 (now Theorem 11) applies to any architecture satisfying the stated injectivity conditions, not only parameter-sharing MLPs. Our premier application of this result, Prop. 14 (now 13), is indeed to a class of parameter-sharing MLPs, with Prop. 13 (now 12) being the layer-wise lemma that Theorem 12 (now 11) upgrades to a global result.

  2. We would also like to remark that our choice to focus on A-MLPs (e.g. G-CNNs) arose not because this class of architectures is easier to analyze, but because we believe they are surprisingly difficult to analyze despite being a well-developed area of the literature. Indeed, the ReLU single-layer case was already an open question as of 2017 [1], which to our knowledge has not been answered in the interim decade. Proposition 13 already answers that question, and Prop. 14 nontrivially strengthens it to resolve the global A-MLP case. We hope this helps illustrate the significance of obtaining exact symmetry characterizations even for familiar architecture classes.

W2.2: The nine case studies do not convincingly show that the framework helps discover problems that would otherwise be difficult to identify.

We thank the reviewer for the feedback. Each case study applies the framework to identify what is, to our knowledge, previously-unrecognized symmetry slippage in active and visible research. Moreover, we believe that the 3:1 ratio of case studies to creation mechanisms supports our claim that the framework helps discover these problems: each creation mechanism is specific enough to identify symmetry slippage in a given architecture, while being general enough to apply to three problems in three different subfields. Lastly, we believe many of these problems would otherwise be difficult to identify. As one example, the group-closure symmetry slippage of Case Study 1 appears to have gone unrecognized across several widely cited architectures (exceeding 400 citations), which is perhaps unsurprising given that identifying this symmetry slippage requires certain algebro-geometric ideas that see little equivariant ML discussion in the first place. We appreciate, however, that this is ultimately a subjective, counterfactual judgment.

W2.3: GE is also unconvincing as a practical method: direct retyping achieves better mean performance in Tables 9–11, while the improvement on the broader ECG task in Table 7 is very small.

We thank the reviewer for their comment. As discussed in the paper, direct retyping can indeed be preferable should it be an available technique. In many situations it is not available, however. Most pertinently, direct retyping only applies to symmetry slippage caused by extension-mistyping; it does not directly apply to symmetry slippage created by group-closure or over-quotienting. In contrast, one advantage of GEs is that they provide a simple and uniformly applicable formalism.

The ECG task in Table 7 investigated a potential second advantage of GEs, namely, whether they can be applied to modify the transformation law obeyed by a model for which full retraining is undesirable and/or the prescribed transformation law is task- or target-dependent. Specifically, the original model was trained on a multilabel task; for 2 of these labels symmetry slippage via over-quotienting occurs, but for many other labels the quotienting is in fact desirable. The goal was to find out whether exposing GEs to the model, which ostensibly help with the 2 labels but could hurt with other labels, improved or impaired performance. Table 7 shows that they improved performance. The modest nature of the improvement is consistent with the fact that only a small subset of labels are known to be affected by this symmetry slippage. Indeed, when those labels were isolated, the improvement was more pronounced (Table 6).

W2.4 I expected a much clearer demonstration of why this framework is useful beyond describing known architectural limitations in a different language.

We would like to clarify that, while our framework does recover known architectural limitations, such as WL-invariance in GNNs or over-equivariance as it is discussed e.g. in molecular learning (chirality), the framework does describe new architectural limitations, cf. our response to W2.2 above. We appreciate the opportunity to better articulate this distinction and hope that the revised manuscript makes the framework's contributions to identifying previously unrecognized architectural limitations clearer.

AuthorsOpenReview ↗

3 - Response to Reviewer d1s8

Q1 How much of the formalism is actually necessary to understand and use this framework?

We thank the reviewer for the question. Although we believe that our framework exists at an appropriate level of abstraction (e.g. it is general enough to apply across our nine case studies, but specific enough to identify problems in each of them), we also believe that the main text did not do a good enough job motivating, describing, or exemplifying this formalism. As detailed in our responses to W1.1 and W1.2, we have improved all three in the revised manuscript.

Q2 Could the authors rebuild the background around one explicit neural architecture and one unintended transformation, introducing , , , , the paired-action notation, and the sub architecture decomposition only when they become necessary?

We thank the reviewer for the suggestion and have done exactly this, cf. W1.1 and W1.2.

Q3 More importantly, which conclusions genuinely depend on distinguishing all these ambient groups and actions? I would like to understand what would be lost if the formulation were simplified. Right now, it is difficult to tell which abstractions are mathematically essential and which merely add another layer of notation.

This is a very good question. Conceptually, it is prescribed symmetries that correspond to subgroups HH of the ambient acting group Γ\Gamma, and it is realized symmetries that live in the ambient transformation groups JJ and KK (renamed JXJ_X and JYJ_Y in the revised PDF). In particular, the realized law need not consist of transformations induced by Γ\Gamma. Distinguishing these objects is important e.g. to the following conclusions:

  • Over-quotienting beyond over-equivariance: The WL example (Section 3.2.2, Appendix A) and Case Study 4 identify slipped transformations outside of the ambient group Γ=Sn\Gamma = S_n. Identifying JXJ_X with Γ\Gamma would prevent these transformations from being identified.

  • Extension-mistyping: The prescribed and realized actions of the same group G≤ΓG\leq \Gamma can differ, even when they agree upon restriction to H≤GH \leq G. In particular, the realized GG-action need not take values in the image of the prescribed Γ\Gamma-action, so identifying JXJ_{X} with that image excludes the transformations responsible for extension-mistyping in such a case.

We hope that our revised Section 2 (cf. W1.1) better motivates the need for distinction.

Q4 What is the most important new conclusion that follows from this framework? Beyond the exact result for the restricted (A)-MLP family, I would like to see a concrete example where the proposed theory reveals something that would not follow as directly from existing equivariance or expressivity analyses.

For one concrete example beyond our core theoretical contributions (the hardness result Thm. 9, composition result Thm. 11, and A-MLP results Props. 13-14) where the proposed theory reveals something that would not follow as directly from existing equivariance or expressivity analyses, we offer the phenomenon of Zariski group-closure (Example 2, Appendix B.1). It has the consequence that for certain groups GG, analytic features equivariant under the identity component G0G_0 are already equivariant under all of GG. We are not aware of the algebraic foundations of this phenomenon having previously been utilized in ML, much less the phenomenon itself. In spite of this, Case Studies 1 and 2 show it causes symmetry slippage across several architectures and also used it to give (to our knowledge) the first theoretical grounding of a common heuristic in weight-space learning.

Q5 What is the actual advantage of grippage encodings (GE) over direct corrections?

We thank the reviewer for their question. Advantages of GEs include

  • Generality: GEs apply across different mechanisms of symmetry slippage, whereas corrections such as direct retyping often apply only in special cases (e.g. extension-mistyping).

  • Architectural simplicity: GEs require only the addition of features to an existing architecture.

  • Theoretical simplicity: The sufficient conditions of Proposition 15 are straightforward to verify in practice.

  • Minimal retraining: GEs can modify an architecture's transformation law without requiring extensive retraining (Remark 1, Case Study 5).

  • Task-dependent adaptation: GEs can adapt an architecture's transformation law to the target or task at hand (Remark 1).

See also our responses to W2.3 above and Q6 below.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful evaluation of our manuscript and constructive suggestions. We especially appreciate their recognition of symmetry slippage as an important problem, and their positive assessments of the empirical breadth and theoretical depth of our investigation.

Here we summarize the main concerns, and our responses. We have also updated the PDF to reflect all changes.


Concern 1: Clarity of exposition

The reviewers highlighted several ways in which the mathematical formalism could be made easier to follow. We have substantially reorganized Sections 2-3 so as to (1) provide motivation via a running SO⁡(2)⊂O⁡(2)\operatorname{SO}(2) \subset \operatorname{O}(2) example and (2) allude to important examples which justify the level of abstraction used.

We have also added two explanatory figures and simplified multiple definitions and examples, including rewriting Example 2 on Zariski group-closure to avoid assuming any algebro-geometric knowledge in the main text (a self-contained treatment is now in Appendix B.1).

Concern 2: The role of grippage encodings

Multiple reviewers asked about how GEs compare with alternative approaches to correct symmetry slippage.

Our nine case studies proceed by applying the creation mechanisms of Section 3 to identify previously unrecognized symmetry slippage across existing fields. To empirically demonstrate that this symmetry slippage can have adverse effects, each case study seeks to compare its slipped architectures with corrected counterparts. The main advantage of GEs in this context is that they provide a simple and systematic approach to correction, enabling a uniform empirical investigation of the practical consequences of symmetry slippage across diverse architectures.

GEs are not the only technique for correcting symmetry slippage, however, and in special cases other approaches may be available. To better understand how GEs compare with these other approaches, we have conducted two new experiments. First, we compared a GE-corrected Lorentz-equivariant architecture (cf. Case Study 1) against a non-equivariant model trained with data augmentation, finding that GEs improved training efficiency and generalized substantially better. Second, we compared GEs with canonicalization, data augmentation, and a hybrid approach suggested by Reviewer qMrH on an SO⁡(2)\operatorname{SO}(2)-invariant task (New Figure 1) with varying sensitivity to symmetry slippage, finding that GEs achieved better predictive performance, more accurate transformation behavior, and faster convergence across the tested settings. Both experiments are included in the revised manuscript (Appendix B.4).


We thank the reviewers again for their constructive feedback, which has helped us considerably improve the manuscript. We address their individual comments in detail below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.