acceptodds
Under review as a conference paper at ICLR 2027

Wider Groups, Fewer Weights: Efficient Pseudo-Orthogonal Equivariance via Geometric Algebras

Abstract

We introduce a new parameter-efficient method for constructing neural networks that are equivariant with respect to pseudo-orthogonal (or orthogonal) groups, based on geometric (or Clifford) algebras. The approach is based on finding and considering larger groups than the Clifford groups in geometric algebras, which perform the same action as the pseudo-orthogonal (or orthogonal) groups. By constructing networks equivariant to these larger groups, we automatically guarantee their equivariance to the corresponding pseudo-orthogonal (or orthogonal) group. This approach induces an expressiveness-efficiency trade-off: a larger equivariance group leads to stricter constraints on the equivariant layers and thus to fewer required optimizable parameters (weights). We propose and study specific groups for this framework, derive new equivariant mappings, and present several model configurations with varying levels of expressiveness and efficiency. Our models demonstrate strong performance on benchmark tasks including convex hull volume estimation, N-body dynamics prediction, high-energy jet tagging, and prediction of atom positions on the MD17 dataset.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
equivariant neural networks, equivariance, geometric deep learning, orthogonal group, pseudo-orthogonal group, geometric algebra, Clifford algebra, Lie group, spin group, Clifford group

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NYGvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a range of extensions that can be applied to a given group in a geometric algebra network, and explain how to construct corresponding networks that are equivariant under these extended groups. They find that these networks can have slightly increased performance, while reducing parameter count and training time. The method and experiments largely follows previous work (CGENN, GLGENN), with the only new contribution being the systematic extension to larger groups, which was however also partly done for GLGENN. I found the paper poorly structured and formatted, making it hard to extract the contributions and results. I do not see how these concerns can be overcome during the rebuttal, and therefore recommend to reject the paper.

Critical strengths
  • The authors find that networks with an extended geometric algebra equivariance group have higher performance at lower parameter count and training time, see e.g. Figure 11. However, the gains are often smaller than the uncertainties from seed variations.
Critical weaknesses
  • I think the paper adds little novelty compared to previous work. It reads like a thin wrapper around the GLGENN paper (2025), which itself is a thin wrapper around the CGENN paper (2023). These papers differ in which groups are considered to build equivariant networks using GA, and then apply small modifications to the architecture and evaluate on essentially the same benchmarks. The benchmarks are not particularly relevant: volume estimation, O(8)-regression, N-body dynamics simulation are toy examples; for top tagging the authors train only on up to 1k samples whereas the dataset has 1M samples and modern datasets contain 100M or more samples as simulation is cheap; for atom position prediction the authors do not directly compare with literature results.
  • Overall the authors do a poor job in presenting their results. The main results in Figure 3 and Table 2 are poorly structured and in my opinion not well explained in the text. The proposed methods show improvements in these plots, but the performance gain often has similar size as the uncertainties. The paper body extensively explains the theory behind the approach, but I found it hard to follow and to get an understanding for the meaning of the extended groups - in particular for the extended groups which are the main contribution of the paper. The authors formatted the paper to remove white space between paragraphs and sections, probably to make it fit into the page limit, making the paper look overcrowded. The appendices contain additional results and answer most questions that came up when reading the paper (compute cost, regularization when training on small datasets) - but the authors should do a better job in already addressing these points in the main paper.
Questions for authors
  • I find the core claim "increasing the equivariance group beyond what is realized in the problem improves performance and reduces cost" not intuitive. The optimal equivariance group should be the one realized in the problem, not more and not less. Do the authors have an explanation for what effects in the larger group can help in the learning process?
  • I do not understand how the proposed P, A, Q, B relate to actual groups. Can you explain, for the O(3) group, what operations these extensions add beyond the standard setup? It would also be good to give more intuition for what the individual extensions do, to discuss when these extensions are expected to help, and to check this in experiments.
  • Experiments focus on the low-data regime (~1k training points, ~1k steps to avoid overfitting with ~100k parameter networks). The authors argue that their architectures are more parameter efficient, but from my understanding they only compare them at fixed architecture layout, whereas they could probably also significantly reduce parameter count and still achieve strong performance. Also, it would be good to discuss issues related to overfitting in the main text, as this seems to be central to the claims in the paper.
  • A central problem of GA-based networks is the significantly increased compute cost. The authors discuss this in appendix H, but compare only across their own architectures and not with baselines. The authors cite the flash clifford package for efficient implementations, although there are several other papers in this direction. Did the authors explore how such optimized implementations can help in their approach?
  • For top tagging, several other Lorentz-equivariant architectures have been proposed since 2023 which seems to be the papers citation cutoff. In particular, https://arxiv.org/abs/2405.14806 extends the geometric algebra transformer architecture to the Lorentz group and is now widely used in high-energy physics.
6 comments
AuthorsOpenReview ↗

Response to Reviewer NYGv [1/3]

We thank Reviewer NYGv for valuable feedback and for highlighting the strengths of our work, including higher performance at a lower parameter count and shorter training time.

We are happy to address the questions:

1) Q2: Relation between the proposed groups P, A, B, Q and O(p,q)?

We thank the reviewer for this insightful question. We would like to clarify that P, A, B, and Q are not extensions of O(p,q); they are extensions of the corresponding Clifford group Γp,q\Gamma_{p,q}. The pseudo-orthogonal groups O(p,q) act on an n=p+qn=p+q-dimensional vector space, whereas our groups act on a 2n2^n-dimensional space (reflecting the dimension of the Clifford algebra). For example, in the R1,3\mathbb{R}^{1,3} setup (e.g., the O(1,3) Top Tagging experiment), the transformations induced by P, A, B, and Q do not act directly on R1,3\mathbb{R}^{1,3}. Instead, they transform a 242^{4}-dimensional space (with basis elements such as e,e1,e2,e3,e4,…,e1234e,e_1,e_2,e_3,e_4,\ldots,e_{1234}). For example, these transformations can map vectors to scalars, bivectors, trivectors, or quadvectors. Therefore, making the group wider (and getting P, A, B, Q) does not correspond to a direct extension of O(p,q), nor are we adding simple geometric operations on Rp,q\mathbb{R}^{p,q} (e.g. R3\mathbb{R}^{3} or R1,3\mathbb{R}^{1,3}). Rather, we are changing the algebraic representation space in which the network operates. Consequently, we cannot directly discuss the geometric meaning of these transformations in the original vector space. The geometric interpretation of the groups P, A, B, and Q is therefore nontrivial; developing this underlying theory is an interesting direction for future work. While we cannot yet provide a formal theorem explaining why one configuration is better than another, we emphasize that this approach works empirically: there are cases in which these wider groups improve performance.

2) Q1: Relation between wider groups and the task symmetries? Effects of wider groups on the learning process?

We agree that in an ideal infinite-data setting, the optimal equivariance group should exactly match the symmetries realized in the problem. However, in practical finite-data regimes, enforcing a wider group equivariance (and thus a more constrained network) offers two major benefits. First, enforcing wider equivariance acts as a powerful regularizer. Second, the true underlying symmetry of a problem is often not perfectly observable or easily identifiable by human inspection, especially in 2n2^n-dimensional underlying spaces. If a network equivariant to a wider group empirically outperforms a less constrained one (with a smaller group), it strongly suggests that the data possesses hidden higher-order structural constraints that the wider group successfully captures. Thus, widening the group is a way of understanding the deeper nature of the task.

AuthorsOpenReview ↗

Response to Reviewer NYGv [2/3]

3) Q3: Focus on the low-data regime?

The reviewer highlights a potential weakness regarding the focus of the experiments on the low-data regime. We would like to clarify that we do not claim our wider-group models perform best in all settings, particularly with large training sets. On the contrary, we posit that the proposed framework offers a tunable trade-off: with abundant data, more expressive models utilizing smaller groups (such as Γ) become preferable, whereas in low-data regimes, wider-group models (such as P, A, and B) are preferable. This is discussed in particular in the Configurations section (lines 348-356 of the main text). That is the reason why we concentrate on different settings with low train set size, this is the focus of our work, since we propose P, A, B, Triple. At the same time the six benchmarks that we consider (convex hull, regression, O(3)- and O(6)-N-Body, top tagging, and atomic dynamics) are standard in the equivariant literature, and many papers use the same set or a subset of experiments, in particular, CGENN, CSMPN, GLGENN, and EqMotion.

We would also like to address the reviewer's hypothesis that simply reducing the parameter count by cutting parameters in CGENN may improve performance. We conducted a new experiment on the O(6)-Convex Hull volume estimation task, where we compare the full A model (which shares weights across grades congruent to 0 or 1 modulo 4, and across grades congruent to 2 or 3 modulo 4) and Γ\Gamma (CGENN) with reduced-parameter versions of Γ\Gamma: narrow Γ\Gamma, shallow Γ\Gamma, and narrow & shallow Γ\Gamma. The depth and width of the reduced Γ\Gamma models were chosen to match the parameter count of A as closely as possible. For this task, we observe the clearest advantage of A over the reduced CGENN, especially with 256 training samples:

ModelWidthDepthParametersTest MSE, mean ± sd (256 train s.)Test MSE, mean ± sd (4096 train s.)
A3241369711.63 ± 1.164.78 ± 0.63
Γ\Gamma, full3247075311.73 ± 0.723.58 ± 0.63
Γ\Gamma, narrow1241285313.76 ± 0.437.93 ± 0.31
Γ\Gamma, shallow3211891313.65 ± 0.144.82 ± 0.28
Γ\Gamma, narrow & shallow2711393313.77 ± 0.215.13 ± 0.12

4) Q4: Efficient implementation of GA layers?

We would like to clarify that we compare our models in terms of computational cost not only with our models, but also with the two closest GA baselines, CGENN (Γ\Gamma) and GLGENN (Q). Table 3 shows that our larger-group networks are more parameter- and computationally efficient than these direct competitors in the small-data regime. We agree that GA-based methods require more computation time than non-GA methods due to their implementation. To the best of our knowledge, the main methods for accelerating GA-based networks focus primarily on accelerating the implementation of the geometric product, and thus these methods can be applied directly to our models. As an example, we mention the Flash Clifford package; if needed, Flash Kingdon (https://github.com/tBuLi/flash-kingdon) can be applied similarly. Flash Clifford allowed us to accelerate geometric product used in the P, A, B, Triple, Q, Γ\Gamma geometric-product layers and to reduce FLOPs by a factor of 2n2^n across all configurations. For example, applying Flash Clifford to the PP configuration for n=6n=6 with one linear and four geometric-product layers (with normalization and residual connections) reduces the forward-pass time from 2.875 ms to 1.721 ms and the forward-backward time from 8.267 ms to 7.377 ms.

We have added this discussion and the corresponding citations to Appendix I.

AuthorsOpenReview ↗

Response to Reviewer NYGv [3/3]

5) Q5: Other baselines for O(1,3)-Top Tagging?

We thank the reviewer for pointing out the L-GATr work, and we have now added it and several other models to the comparison. We find that we were able to achieve an accuracy of 0.945 for the Q- or A-config and 0.941 for the Γ-config when training on 1024 samples, whereas models trained on the full dataset achieve similar or lower accuracy: 0.942 CGENN, 0.936 ResNeXt, 0.930 P-CNN, 0.932 PFN, 0.94 ParticleNet, 0.922 EGNN, 0.929 LGN, 0.942 LorentzNet, including the newly added: 0.9423 L-GATr (2024), 0.9426 PELICAN (2022), 0.940 ParT (2024), 0.933 TreeNiN (2019). We have added this comparison to Section 6.

To address the reviewer's interest in the performance of our configurations on a larger training set, we conducted a new experiment and increased the train set from 1024 to 10000 jets at the same step budget. The mean changes were +0.24 in accuracy for A and +1.18 in accuracy for Γ compared with the results for 1024 train samples.

We also tested whether applying our local constraints can preserve the performance of the L-GATr model suggested by the reviewer, which was already trained on all 1.2M training jets (a certain type of mixing architecture). Projecting one hidden map to the A-config layer removed 4640 of 1077474 parameters of L-GATr. Before any updates, the validation AUC fell from 0.986363 to 0.984992 on 403k test jets. After matched fine-tuning, the projection recovered 81.3% of the initial AUC deficit on the same test jets. The selected A-config and the original L-GATr AUCs were 0.986070 and 0.986329. Thus, for large training sets, our wider configurations can be applied to fully trained models such as L-GATr to reduce parameters and, after matched fine-tuning, recover the lost performance.

6) Novelty of ideas.

We believe our work contains enough novel and non-trivial ideas to avoid being considered a thin wrapper around CGENN and GLGENN. The main novel ideas are:

(i) Using the groups P, A, B, and Triple in equivariant networks. Changing the groups was not straightforward: for each group, we identified its action, parameterized new operations equivariant with respect to it, and proved their equivariance. We also used other techniques not employed in CGENN or GLGENN. In particular, proving that equivariance with respect to P, A, B, and Triple implies O(p,q)-equivariance is non-trivial.

(ii) Showing how to mix different equivariance groups within a single architecture (in our heterogeneous configurations). To the best of our knowledge, the idea of varying the size of the equivariance group depending on the task has not been considered before, even outside the domain of GA-based networks.

Note that the same results cannot be obtained by simple regularization or by straightforwardly cutting parameters in CGENN or GLGENN.

7) Recommendations on presentation.

We have followed the reviewer's suggestions to elaborate on the computational cost and regularization effects of wider-group equivariant models in the small-data regime in the main text. We have added Section 7, devoted to these comments. For the convenience of readers not familiar with the GA formalism, we have added Appendix G, which presents the main theorems, groups, and layers in detail for the particular case of Cl1,3,0Cl_{1,3,0} of Minkowski space-time, as applied to the Top Tagging task.

Reviewer_khtSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a mathematical framework for constructing parameter-efficient equivariant neural networks. The core idea is to build larger groups than the Clifford groups in geometric algebras that perform the same action as (pseudo)-orthogonal groups. The paper discusses expressiveness efficiency tradeoffs of the proposed framework and shows that it can achieve strong performance across symmetry tasks, including N-body dynamics prediction, convex hull volume estimation, and predicting 3D dynamics of molecules.

Critical strengths
  1. The theoretical framework seems to be general and principled, covering previous works as special cases. It provides a unified perspective for constructing equivariant networks based on geometric algebras (GAs).
  2. I appreciate Section 5 where the paper discusses the trade-offs and recommendations for practical usage.
Critical weaknesses
  1. I would argue that parameter efficiency may not be the main criterion for developing equivariant networks in general. Such networks already have modest parameter count, and their main bottleneck is their large amount of compute spent on each data sample. Imposing more constraints into neural networks generally increases their compute cost, while keeping the parameter count low.
  2. The experiments are very limited, and do not signal how these networks will scale with more data. Moreover, table 2 only contains three of the eight molecules in the MD17 benchmark. From the results, I am suspecting that these networks do not scale well with data, as their performance plateaus very early.
  3. Higher parameter efficiency does not necessarily make optimization easier. The paper enforces equivariance to larger groups, resulting in less parameter count due to stricter symmetry constraints. Those additional symmetries however do not often correspond to the physical symmetries of 3D data, and therefore the optimizer may need more training steps to find good solutions, which again increases the training compute.
  4. The paper is quite dense and hard to read.
Questions for authors

[1] Could you compare the compute cost of different configurations with other equivariant networks at the same number of parameters?

5 comments
AuthorsOpenReview ↗

Response to Reviewer khtS [1/2]

We thank Reviewer khtS for valuable feedback and for highlighting the strengths of our work, including the unified perspective on constructing equivariant networks based on geometric algebras (GAs).

We are happy to address the questions:

1) Q1: Importance of parameter efficiency for equivariant networks?

We see two reasons why weight-sharing for parameter efficiency in equivariant neural networks is important:

  1. Weight sharing may be considered as a way to reduce compute cost. To answer the reviewer’s request directly, we compared Γ\Gamma (CGENN) and Triple-Γ\Gamma (ours) with other equivariant networks, GVP and VN, in O(6)-Convex Hull at matched parameter budgets: 38k and 71k (initial budgets for Triple-Γ\Gamma and Γ\Gamma respectively), and also 49k and 97k. Latency is the median of 50 repetitions on a single Tesla V100; the training set size is 4096; batch size is 128; an Adam step includes the forward pass, MSE computation, backward pass, and parameter update.
TierModel# ParamsInference median (ms)Adam step median (ms)Test MSE ↓ (mean ± std)
97kTriple-Γ97,52514.13134.5352.596 ± 0.198
97kΓ97,01521.31745.3623.203 ± 0.236
97kGVP97,7252.0069.1865.733 ± 0.189
97kVN97,4062.1368.25520.047 ± 0.610
71kTriple-Γ69,93113.39332.7522.792 ± 0.220
71kΓ70,75320.72643.8673.568 ± 0.487
71kGVP70,1692.0139.1885.600 ± 0.096
71kVN71,3452.1678.30120.885 ± 0.601
49kTriple-Γ49,21116.43840.6322.884 ± 0.137
49kΓ48,59513.99831.9793.714 ± 0.226
49kGVP49,6772.3149.1486.756 ± 0.842
49kVN49,3302.1298.27422.060 ± 0.621
38kTriple-Γ38,08115.79539.5483.029 ± 0.447
38kΓ39,05513.58631.3593.734 ± 0.178
38kGVP38,1692.0019.0856.108 ± 0.289
38kVN38,2702.4268.20522.545 ± 0.472

The results show that at every parameter count, Triple-Γ outperforms all models in terms of mean MSE, and the computational advantage over Γ emerges at larger parameter counts (71k and 97k). Triple-Γ at 38k beats Γ at 97k on all four axes: MSE 3.029 vs 3.203, inference 15.8 vs 21.3 ms, Adam step 39.5 vs 45.4 ms, while using 2.5x fewer parameters. GVP and VN are much faster than GA-based models, but their MSE is substantially worse. We note that when compute cost matters for a given application, geometric-product accelerations such as Flash Clifford or Flash Kingdon (which we mention in Appendix I) can be applied to our layers, reducing FLOPs by at least a factor of 2n2^n.

We would also like to note that we previously compared our configurations with CGENN (Г) and GLGENN (Q), with different numbers of parameters, in terms of compute cost in the O(1,3)-Top Tagging and O(6)-Convex Hull experiments, and the results are presented in Appendix I (Computational cost trade-off section). All of our models speed up CGENN and GLGENN, while many of them also achieve better task-based metrics. For the reviewer’s convenience, we repeat Table 12 below (O(1,3)-Top Tagging wall-clock time (seconds, mean ± std), 128 training samples, batch size 32, speed-up relative to the Γ-config):

Config# ParamsTrain time per step (s)Inference time (s)Accuracy (↑)Speed-up (train)
Γ321k1.0389 ± 0.1899993.13 ± 0.13370.8828 ± 0.02211.00
A253k0.3726 ± 0.0005356.77 ± 0.47290.8922 ± 0.01022.79
B253k0.3714 ± 0.0011355.70 ± 1.11870.9078 ± 0.02172.80
P251k0.5704 ± 0.5328546.33 ± 0.54310.8922 ± 0.01311.82
Q269k0.5970 ± 0.3714571.78 ± 0.45320.8992 ± 0.00501.74
Triple263k0.5846 ± 1.8581559.85 ± 1.81820.8891 ± 0.01781.78
  1. If weight-sharing improves the metrics, we gain new information about the nature of the task. We cannot always be sure which symmetry group is relevant for a particular task. Some task properties may not be apparent to human inspection. If the results are empirically better for a network equivariant with respect to a wider group, this probably means that the task or data has additional symmetry constraints. This is an interesting way to learn more about the nature of the task.

We have added the new experiment results and the corresponding discussion to the Appendix I of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer khtS [2/2]

2) Relation between parameter count and optimization efficiency.

The reviewer hypothesized that stricter symmetry constraints might require the optimizer to take more training steps to find good solutions. We agree that a lower parameter count does not always make optimization easier. However, our empirical findings show that wider-group configurations (which have fewer parameters) do not require more optimization steps and help prevent overfitting when the training set is small. As an example, we evaluate this on the O(6)-Convex Hull task and compare the loss curves for Γ (CGENN; the largest model, 71k parameters) and our A (one of the smallest models, 14k parameters) using 256 training samples. The columns indicate completed optimizer updates for the same runs, and the cells show the median MSE (↓). Our A has lower validation MSE at every reported checkpoint, despite having higher training MSE at later checkpoints.

ConfigurationMetric100200300400510
AValidation MSE12.98111.99111.65611.57811.629
ATraining MSE5.9781.7540.3820.0830.0093
ΓValidation MSE13.52012.64512.31312.26912.260
ΓTraining MSE6.3071.1320.0360.00104.52e-05

We have added these results, along with a new Figure 12 showing loss curves as a function of both optimizer steps and time (seconds since training start), to the revised manuscript (Appendix I).

3) Scaling of our models with data.

The reviewer highlights a potential weakness regarding the scaling of our models to larger datasets. We would like to respectfully clarify that we do not claim our wider-group models perform best in all settings, particularly with large training sets. On the contrary, we posit that the proposed framework offers a tunable trade-off: with abundant data, more expressive models utilizing smaller groups (such as Γ) become preferable, whereas in low-data regimes, wider-group models (such as P, A, and B) are preferable. This is discussed in particular in the Configurations section (lines 348-356 of the main text).

4) Limitations of experiments.

We would like to emphasize that our six experiments (convex hull, regression, O(3)- and O(6)-N-Body, top tagging, and atomic dynamics) are standard in the equivariant literature, and many papers use the same set or a subset of these experiments, in particular, CGENN, CSMPN, GLGENN, and EqMotion. We focused on limited training-set sizes because this is where our work contributes: we study how the framework with wider equivariant groups helps in these cases.

The submitted MD17 table covered three of the eight molecules. We have added a new experiment with a fourth molecule, ethanol, using the same 5000 training windows and 10^4 training steps to Table 2. The mean test ADE/FDE (×10^{-2}, ↓) is 4.7522/6.5801 for A (136k parameters) and 4.6983/6.4903 for Γ (369k parameters). The best mean ADE/FDE of 4.44/6.30 is obtained by CSMPN on 10^5 train steps but with 300k parameters. For now, the range of experiments (4 molecules) is the same as in the CSMPN and EqMotion papers.

5) Readability and presentation.

We agree that the paper is mathematically dense. Our goal was to keep precise formulations of the proposed layers and their equivariance properties, but we have taken several steps to improve readability without sacrificing rigor.

In particular, we have introduced all required notation and definitions in the main text, with extensive cross-references to Appendix B-F, where additional details and examples are provided. For convenience, we use notation standard in the GA-based network literature, in particular in the papers CGENN, GLGENN, CSMPN, CS-CNNs cited in our work. For readers less familiar with the GA formalism, we have added a roadmap of the notation used throughout the paper (Figure 4) and Table 4, which lists the main definitions and their first occurrence.

We have also added Appendix G with the main theorems, groups, and layers formulated in detail for the particular case of Cl1,3,0Cl_{1,3,0} of the Minkowski space-time, as applied to the Top Tagging task.

We hope these changes make the paper easier to follow.

AuthorsOpenReview ↗

We have addressed your concerns in our rebuttal

Dear Reviewer khtS,

Thank you for your time and effort in reviewing our submission. We have posted a point-by-point response and a revised manuscript with changes marked in blue, addressing the questions and concerns raised in your review. We would be grateful if you could review our response and revised manuscript and reconsider your evaluation in light of the changes. If anything remains unclear, we are happy to continue the discussion.

Reviewer_ZhFTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

This work proposes a way to achieve more parameter-efficient equivariant networks by constraining them to be equivariant to larger groups defined by Clifford algebra that contain the required group as one of their subgroup. The authors present a way to use orthogonal or pseudo-orthogonal groups as a basis to construct larger groups using geometric Clifford algebras and mapping them to constraints of equivariant neural networks. Since such constraints reduce the hypothesis space of the learned model, they require fewer parameters to be described. In addition to providing a general construction methodology, they evaluate this approach in different equivariant tasks, showcasing how their overconstrained networks are able to achieve improved scaling with the data size and number of scalable parameters.

Critical strengths
  • The idea of constraining the parameters of a network by increasing the group symmetry to which it is equivariant is quite novel and can be interesting to the community even outside the specific constructions presented in this work.
  • The presentation of the construction of such equivariant networks in larger groups defined through the geometric Clifford algebras is thorough and, although dense, it provides a lot of technical insight
  • The scaling results presented in the experiments are quite interesting and provide a good first proof of concept for the proposed aproach
Critical weaknesses
  • There is no clear discussion regarding why this type of reducing the expressivity and parameters count of the model, by introducing additional constraints, is preferable to the more classical way of controlling the expressivity by the width of the intermediate features or the depth of the learned models. Specifically, reducing the size of the hidden layers is a task-agnostic operation, while constraining the layers to a larger group introduces very specific implicit bias for the model, which is not clear how it can hurt or benefit the task.
  • The presentation of the Clifford algebras is dense, and although, as I stated in the strengths, it is quite informative, it can make the proposed solution more difficult to read to a wider audience.
Questions for authors
  • Why is overconstraining the symmetry of the network preferable to reducing the width of the latent or depth as a mechanism to achieve parameter efficiency?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZhFT

We thank Reviewer ZhFT for valuable feedback and for highlighting the strengths of our work, including the novelty of the idea of enlarging and mixing equivariance groups, which may be of interest for broad community.

We are happy to address the questions:

1) Q1 (and Weakness 1): Parameter reduction via constraints on symmetry vs. reducing the model's width/depth?

We agree with the reviewer that comparing our 'clever' weight-sharing (with respect to larger symmetry groups) against more classical approaches to parameter reduction is a natural question. In our opinion, these are fundamentally different types of restrictions: group constraints tie coefficients across subspaces of geometric algebras and product paths while retaining channels and compositions, while narrowing or reducing depth removes those channels or compositions entirely.

We conducted a new experiment on the O(6)-Convex Hull volume estimation benchmark, where we compare the full A model (which shares weights across grades congruent to 0 or 1 modulo 4, and across grades congruent to 2 or 3 modulo 4) and Γ\Gamma (CGENN) with reduced-parameter versions of Γ\Gamma: narrow Γ\Gamma, shallow Γ\Gamma, narrow & shallow Γ\Gamma. The depth and width for the reduced Γ\Gamma models were chosen to be as close as possible to matching the parameter count of A. For this task, we observe the clearest advantage of A over the reduced CGENN, especially in the case of 256 training samples:

ModelWidthDepthParametersTest MSE, mean ± sd (256 train s.)Test MSE, mean ± sd (4096 train s.)
A3241369711.63 ± 1.164.78 ± 0.63
Γ\Gamma, full3247075311.73 ± 0.723.58 ± 0.63
Γ\Gamma, narrow1241285313.76 ± 0.437.93 ± 0.31
Γ\Gamma, shallow3211891313.65 ± 0.144.82 ± 0.28
Γ\Gamma, narrow & shallow2711393313.77 ± 0.215.13 ± 0.12

Additionally, we would like to highlight that in our work, we considered a similar question regarding approaches to overfitting: comparing our approach ('clever' weight sharing) with ordinary regularization techniques, such as weight decay (Appendix J, Table 17). The results show that none of the tested weight decay values (10−210^{-2}, 10−410^{-4}, 10−510^{-5}) applied to Γ\Gamma in the N-Body task improved upon the results of P and A-Triple.

These insights suggest that ordinary parameter reduction or regularization techniques turn out to be less effective than the proposed weight-sharing. We have added Appendix K with these findings to the revised manuscript.

2) Dense presentation.

We put effort into introducing all required notation and definitions in the main text (and none of them can be omitted because they are used in the approach), giving extensive references to Appendices B-F, where we provide more details and examples. For the reader's convenience, we used the notation usually applied in the GA-based network literature (in particular, in the papers on CGENN, GLGENN, CSMPN, CS-CNNs cited in our work). For readers not yet familiar with the GA formalism, we provided a roadmap of the notation used throughout the paper (Figure 4) and Table 4, which lists the main definitions and their first occurrence.

We have also added Appendix G with the main theorems, groups, and layers formulated in detail for the particular case of Cl1,3,0Cl_{1,3,0} of the Minkowski space-time, as applied to the Top Tagging task.

AuthorsOpenReview ↗

We have addressed your concerns in our rebuttal

Dear Reviewer ZhFT,

Thank you for your time and effort in reviewing our submission. We have posted a point-by-point response and a revised manuscript with changes marked in blue, addressing the questions and concerns raised in your review. We would be grateful if you could review our response and revised manuscript and reconsider your evaluation in light of the changes. If anything remains unclear, we are happy to continue the discussion.

Reviewer_ZhFTOpenReview ↗

Official Response by Reviewer ZhFT

I thank the authors for addressing my concerns and questions regarding their submitted work. I appreciate the addition of results aimed at answering the question, positioning their proposed method against a more general approach to reducing a model's expressivity. Also, I would like to thank the authors for making the effort to present the required mathematical background for their work in a way that is more easily accessible to a broader audience. Since most of my concerns are addressed, I will increase my current recommendation for this work to accept.

Dear Reviewer ZhFT,

Thank you very much for your kind words, constructive feedback throughout the review process, and for raising your recommendation to 4. We are glad that our revisions addressed your concerns.

Just a quick note: it looks like the system might not have registered the score update yet, as it still displays 3 on our end. Could you kindly check if it saved properly when convenient? Thank you again!

Comments on the paper

1 comment

Dear reviewers, the authors responded to your reviews. Could you please go over the responses, see if you have any further questions, and if appropriate, reassess your evaluation of the paper? Note that the deadline for all reviewer-author discussions is November 18th. Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.