Gauge-Equivariant Aggregation for Spherical Attention

Spherical learning tasks often have scalar inputs and outputs, but their spatial structure is naturally described by geometric quantities such as vector and tensor fields. In this work, we investigate if a spherical attention mechanism can benefit from using these intrinsic quantities. To formulate a spherical attention that uses geometric features consistently, we introduce a general gauge-theoretic formulation of normalized aggregation on Riemannian manifolds of arbitrary dimension. Our aggregation operator covers normalized convolution and attention, and we prove that it is equivariant under structure-preserving isometries when explicit assumptions are met. A spherical quadrature realization using exact spherical geometry and Levi-Civita transport is derived. The resulting attention mechanism is exactly gauge-equivariant and can process scalar, vector, and tensor hidden fields. We demonstrate that our implementation can evaluate its learned operators across spherical discretizations. On rotating shallow-water dynamics and Stanford 2D-3D-S omnidirectional semantic segmentation, ablations and comparisons with the $S^2$ Transformer and $S^2$ SegFormer demonstrate the benefits of geometric intermediate representations and content-dependent attention weights. Our results support geometric hidden fields as useful representations for spherical attention, although computational cost remains a limitation.