acceptodds
Under review as a conference paper at ICLR 2027

Gauge-Equivariant Aggregation for Spherical Attention

Abstract

Spherical learning tasks often have scalar inputs and outputs, but their spatial structure is naturally described by geometric quantities such as vector and tensor fields. In this work, we investigate if a spherical attention mechanism can benefit from using these intrinsic quantities. To formulate a spherical attention that uses geometric features consistently, we introduce a general gauge-theoretic formulation of normalized aggregation on Riemannian manifolds of arbitrary dimension. Our aggregation operator covers normalized convolution and attention, and we prove that it is equivariant under structure-preserving isometries when explicit assumptions are met. A spherical quadrature realization using exact spherical geometry and Levi-Civita transport is derived. The resulting attention mechanism is exactly gauge-equivariant and can process scalar, vector, and tensor hidden fields. We demonstrate that our implementation can evaluate its learned operators across spherical discretizations. On rotating shallow-water dynamics and Stanford 2D-3D-S omnidirectional semantic segmentation, ablations and comparisons with the Transformer and SegFormer demonstrate the benefits of geometric intermediate representations and content-dependent attention weights. Our results support geometric hidden fields as useful representations for spherical attention, although computational cost remains a limitation.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Gauge equivariance, spherical attention, isometry equivariance, geometric deep learning, normalized aggregation, numerical quadrature

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YZpWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a general framework for gauge-equivariant normalized aggregation on Riemannian manifolds, covering both convolution and attention. The authors further develop a spherical implementation using exact spherical geometry, Levi-Civita parallel transport, and quadrature weights, enabling attention over scalar, vector, and tensor feature fields. Experiments on shallow-water dynamics and spherical semantic segmentation demonstrate the benefits of geometric hidden representations.

Critical strengths
  1. The paper presents a unified framework for gauge-equivariant aggregation with explicit conditions for isometry equivariance. The theoretical formulation is clear and provides a useful perspective on geometric attention.

  2. The combination of exact spherical geometry, parallel transport, and quadrature-based aggregation is technically interesting. The resulting implementation achieves exact gauge equivariance and supports different geometric feature types.

  3. The experiments demonstrate improvements over scalar-based spherical architectures. The ablation studies provide meaningful evidence for the benefits of vector and spin-2 hidden representations, while the cross-discretization experiments demonstrate some degree of transferability.

Critical weaknesses
  1. Gauge-equivariant attention has already been investigated in GET (He et al., 2021) and EMAN (Basu et al., 2022), while quadrature-based spherical attention was introduced by Bonev et al. (2025). Although the proposed integration is valuable, the individual components are largely established. The paper could more clearly distinguish its fundamental methodological contributions from existing approaches.

  2. The isometry-equivariance theorem appears closely related to established principles in gauge-equivariant neural networks. It is unclear whether the theorem provides a substantially new theoretical insight or primarily extends existing results to normalized aggregation.

  3. Although the ablations demonstrate the benefits of geometric hidden representations and attention, the contributions of exact parallel transport, quadrature weighting, and steerable value kernels are not individually evaluated. More targeted ablations would help establish which design choices are essential to the observed improvements.

  4. The proposed models incur substantial computational overhead compared with existing spherical architectures. This raises concerns about their scalability and practical advantages, particularly when the performance improvements are relatively modest.

Questions for authors
  1. What is the precise theoretical contribution of Theorem 1 beyond existing results on gauge-equivariant convolution and attention?

  2. Could the authors clarify the key methodological differences from GET, EMAN, and the spherical attention framework of Bonev et al. (2025), particularly which capabilities cannot be achieved by straightforward extensions of these methods?

  3. Could additional ablations isolate the effects of exact parallel transport, quadrature weighting, and steerable value kernels?

  4. How do the authors justify the additional computational cost relative to the achieved performance gains? Could a comparison under matched computational budgets be provided?

1 comment

Addressing each question in order:

  1. See the general comment and our response to Reviewer gZvA.

  2. Relative to Bonev et al. (2025), our attention layer processes geometric features, particularly scalars, tangent vectors, and spin-2 fields. Our experiments provide evidence that these directional features bring value to learning scalar signals on the sphere. GET and EMAN already provide gauge-equivariant attention. Compared with GET and EMAN, our spherical realization adds quadrature weights, exact spherical geometry, and neighborhoods defined by fixed geodesic radii. Because of these additions, our learned operator can be evaluated on different spherical discretizations without retraining, as demonstrated in Table 1, although performance degrades relative to evaluation on the training grid. We do not prove that these capabilities cannot be obtained by an extension of GET or EMAN. Our contribution is the general framework addressed in question 1, a spherical realization of that framework, and experiments that study the effect of hidden field type and aggregation components on predictive performance.

  3. Thanks to your recommendation, we added ablations of quadrature weighting in attention layers and the steerable value kernel (Table 2). The quadrature ablation keeps the true Legendre-Gauss weights in the encoder, decoder, and loss. We did not add a parallel transport ablation. Parallel transport can not be meaningfully removed all together since the features leave in distinct spaces. It is also not clear to us which approximation is most informative in an exact vs approximate parallel transport experiment. For instance one could approximate surface normals, the connection, the geodesic, or the transport map. We do not, however, doubt that such experiments could be useful.

  4. Again, thanks to your recommendation, we added a matched-training-time experiment to our shallow-water comparisons (Figure 2).

Reviewer_AwzqOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper presents a framework for gauge-equivariant aggregation in an attention mechanism on spherical signals. A recent work (Bonev et al. 2025) on spherical attention works with scalar feature representations in intermediate layers. The current work suggests that higher-order feature representations (e.g., tangent vectors, tensors) could provide richer analysis of spherical signals (such as directional information). Such features must be represented in some local coordinate system (more generally, a gauge) and communication between points on the sphere must incorporate changes in coordinate system (gauge transformations). The paper extends work on Gauge-equivariant neural networks by describing a gauge-equivariant affinity mechanism, required for attention and realizes this for spherical signals. The benefit of the extended feature representations is demonstrated on two applications: Shallow-water dynamic and spherical semantic segmentation.

Critical strengths
  • Thorough mathematical treatment of gauge-equivariance in spherical attention. I am convinced by the main argument of the paper and appreciate the depth and rigor that was applied.
  • Experimental evidence supporting the main claims of the paper on two applications, demonstrating the benefit of internal geometric representations (i.e., tangent vectors, tensors).
Critical weaknesses

I didn't find critical weaknesses in the theory or contributions that would bar the paper from being accepted. I do have a few points on the writing that would be good to address.

The writing can be inaccessible and abstract in places where it could be more precise and didactic, especially on a first read. For example, the introduction states that "In this work, we ask if spherical attention can benefit further by incorporating geometry into the hidden representations themselves." This is rather vague. Being more precise about "incorporating geometry" and the expected benefits would clarify the goal. After reading the full paper, I understand that the precise difference with Bonev et al. is that they only use scalar features in intermediate representations and this work includes tangent vectors and tensors, which is the sentence that follows. Somehow, that was hard to gather from these two sentences, even though it is technically there.

The paper develops its theory with limited contextualization to related work.

  • The related work is sparse on references to gauge-equivariant neural networks, only citing two papers from the same group a wider range of works could be included (e.g., https://link.springer.com/article/10.1007/s10462-023-10502-7). It would also be interesting to clarify how the current work pushes forward the theory presented in Cohen et al. and Weiler et al. (i.e., extension of affinity from trivial affinity α=1\alpha=1 to eq. 4)
  • Among other points, the paper claims novelty over EMAN and GET by going beyond R3 (l. 076). I get that the theory in the current work is applicable to higher dimensions, but there is no experimental evidence or a realization for higher dimensions. It may be good to tone down this statement, e.g., by stating that the theoretical framework allows to realize attention for higher dimensions in the future.
  • The related work mentions that Bonev et al. 2025 is closest to the current formulation. The main difference is "ours processes scalar, vector, and tensor fields gauge-equivariantly." Since this is the crucial difference, it would be good to expand on this. How is this not supported in Bonev et al., why is that a problem, and how does the current work benefit, practically, from processing gauge-equivariantly? It becomes clearer further in the paper, but it would be helpful to be more explicit in the related work.
Questions for authors

In line 147, the paper states that "a sufficiently small neighborhood of p can be parametrized by a set of tangent vectors [...]" and this assumption (a sufficiently small neighborhood) is not picked up further in the text. I assume that this is because the focus of the paper is on the sphere where (nearly) the entire sphere can be parametrized this way and the rest of the theory holds. When does this assumption break down? This is relevant, since one of the motivations and contributions of Bonev et al. (2025) is to support global attention and not only local attention (which was supported in prior work, e.g. SphereUFormer).

Bonev et al. (2025) also validate their approach on spherical depth estimation. Did the authors also experiment on this application? If so, what was the outcome?

1 comment
  • We agree that there were places that were vague or imprecise in the original submission. We revised the wording in the areas you brought to our attention, e.g., l. 034–036, 085–087, 095–098, 153–157.

  • To address your first question: in general, if qq lies in the image of the aggregation support under exp⁡p\exp_p, then pp can “attend” to qq. This is true even if qq lies outside of the injectivity radius. To see this, notice that the coordinate-free formulation of GMGM-aggregation (l. 742–773) makes no injectivity assumption. Practically speaking, restricting the aggregation neighborhoods to lie within the injectivity radius makes the implementation easier: a unique Riemannian logarithm, unique transport along a minimizing geodesic, and each neighboring feature vector fqf_q represented only once in the pulled-back field f^p\hat f_p. Thus, global attention is possible only when exp⁡p\exp_p is surjective onto the manifold for each query point pp. As you noted, our work assumes R<πR < \pi so that the implementation supports local attention only.

  • To address your second question, we did not experiment with depth estimation, though this would be an interesting next experiment.

Reviewer_gZvAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper presents a gauge equivariant framework for aggregating geometric feature fields, like vectors and tensors. in particular, the framework yields gauge equivariant attention for scalars, vectors and tensors. They also demonstrate their results on certain spherical segmentation tasks.

Critical strengths

The framework in the paper distinguishes exact gauge equivariance from approximate rotation equivariance on a fixed grid.

Critical weaknesses
  • The conceptual advance of the paper is limited, and the paper lacks a thorough description of how it ties in with previous literature. Gauge-equivariant convolution on manifolds, with the exponential-map parametrization, geodesic transport, kernel constraint and invariant measure, was already established in (Cohen et al. 2019), (Cheng et al. 2019) and (Gerken et al. 2021), then formalized by (Weiler et al. 2023). Two of these references are omitted in the paper.

  • In particular, Section 3 of the paper presents the gauge equivariant framework without a citation for it, and the list of methods recovered by Eq. 2 omits gauge-equivariant convolution itself; the statement on p. 5 that only nonzero-frequency fields admit anisotropic kernels is a known result (Makadia et al. 2007; Esteves et al. 2018, 2020; Gerken et al. 2021, Remark 6.1) given without reference. Moreover, spin-weighted spherical CNNs (Esteves et al. 2020) and gauge-equivariant mesh CNNs (de Haan et al. 2021), which realize the same feature types and the same discretization pattern, are not cited.

Questions for authors

Could you clarify what Theorem 1 adds relative to the existing works on gauge equivariant convolutions and gauge equivariant attention?

2 comments
  • We agree that our original submission was lacking in citations and attribution for the foundational components of our work. Our revised submission now includes the two missing citations to Cheng et al. (2019) and Gerken et al. (2021) (l. 064–072, 102–103, 286–287).

  • The normalized field-dependent aggregation analogue of Definition 12.2.7 and Theorem 13.2.5 of Weiler et al. (2023) does not exist in the literature, to the authors’ knowledge, and that is what our GMGM-aggregation template operator and Theorem 1 provide. GMGM-aggregation and its equivariance under IsomGM\mathrm{Isom}_{GM}, together with the principles in Chapter 14 of Weiler et al. (2023), provide a blueprint for designing new equivariant attention-like operators on general domains with general compact structure groups. With our formulation of GMGM-aggregation, one could in theory fill out a table analogous to Table 14.1 of Weiler et al. (2023) for transformer-style architectures, of which He et al. (2021) and Basu et al. (2022) could be rows.

Thanks for the clarifications and the revision. The authors have adressed many of my concerns and I will consequently raise my score.

Comments on the paper

1 comment

We thank the reviewers for taking the time to carefully read our submission and provide thoughtful feedback. This comment summarizes the four substantive additions and revisions in our rebuttal submission. We address questions raised by individual reviewers in separate comments below their official reviews.

  1. Expanded citations and related work section: We added Cheng et al. (2019); Gerken et al. (2023); Esteves et al. (2020); and de Haan et al. (2021) as references. We clearly attribute geometric features, exponential-map neighborhoods, parallel transport, and steerable kernels to these works together with Cohen et al. (2019) and Weiler et al. (2023) (l. 064--072, 102--103, 224--225, 285--287).

  2. Clarified contribution and relevance of Theorem 1: We highlight that our aggregation operator provides a template, with Theorem 1 giving sufficient conditions for equivariance under the relevant subgroup of isometries (l. 53--56, 231--232). The template and theorem extend the convolution counterparts in Weiler et al. (2023) to field-dependent normalized aggregation. As such, our formulation covers any Riemannian manifold MM equipped with a GG-structure, for compact GG. In lines 234--239, we provide a brief example to demonstrate what the generality in GG provides, namely, that different choices of GG and GG-structure on R3\mathbb{R}^3 will produce normalized-aggregation operators that are equivariant under different symmetry subgroups. Thus, the novelty of our formulation when compared with the surface attention of He et al. (2021) and Basu et al. (2022) is the generality in both spatial domain and the symmetry subgroups under which the aggregation is equivariant.

  3. Expanded ablation experiment: We added to the ablation study quadrature weights in attention blocks and steerable value kernels (l. 425--455). To make the quadrature ablation more informative, we changed the training grid of D1 from a level-5 icosphere to a 128 x 256 Legendre-Gauss grid, where sampling is less uniform. As a result, D1's performance improved. We emphasize the architecture of D1 did not change; the training grid did.

  4. Matched compute budget comparison: In the shallow-water experiment, we extended training of the reference S2S^2 Transformer to match D1's wall-clock training time on one V100 and plotted the validation curves in Figure 2. The scheduler and learning rates were retuned for the extended run via grid search.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.