Frequency-Aware Patch Weighting in Vision Transformers via the Funk-Hecke Theorem

In Vision Transformers (ViTs), patch features are typically used directly. Existing Fourier-based methods are global and operate on the 2D image grid, where all patches share the same frequency-band weights and cannot distinguish which patches contain more useful information. As a result, the angular-frequency components of each patch (the analogue of spatial frequency in planar Fourier analysis) and the texture and detail information they carry are often ignored. In contrast, we operate on a high-dimensional unit hypersphere and weight each patch independently. We find that the spherical harmonic Fourier features of each patch allow order-indexed weighting per patch. The Gegenbauer basis functions and the Funk–Hecke theorem provide the mathematical foundation for this operation. Based on this, we propose Funk–Hecke Adaptive Weighting (FHAW), which modulates each patch individually by an order-indexed, low-order band-limited gate, giving per-patch modulation a clear angular-frequency interpretation. This mechanism opens a new path for ViT development parallel to attention. On classification, segmentation, and zero-shot recognition tasks, FHAW consistently outperforms the corresponding baselines with only negligible additional overhead.