acceptodds
Withdrawn from ICLR 2027

Frequency-Aware Patch Weighting in Vision Transformers via the Funk-Hecke Theorem

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In Vision Transformers (ViTs), patch features are typically used directly. Existing Fourier-based methods are global and operate on the 2D image grid, where all patches share the same frequency-band weights and cannot distinguish which patches contain more useful information. As a result, the angular-frequency components of each patch (the analogue of spatial frequency in planar Fourier analysis) and the texture and detail information they carry are often ignored. In contrast, we operate on a high-dimensional unit hypersphere and weight each patch independently. We find that the spherical harmonic Fourier features of each patch allow order-indexed weighting per patch. The Gegenbauer basis functions and the Funk–Hecke theorem provide the mathematical foundation for this operation. Based on this, we propose Funk–Hecke Adaptive Weighting (FHAW), which modulates each patch individually by an order-indexed, low-order band-limited gate, giving per-patch modulation a clear angular-frequency interpretation. This mechanism opens a new path for ViT development parallel to attention. On classification, segmentation, and zero-shot recognition tasks, FHAW consistently outperforms the corresponding baselines with only negligible additional overhead.

Primary area:
learning theory
Keywords:
Vision Transformers, spherical harmonics, Funk-Hecke theorem, Gegenbauer polynomials, frequency-domain representation, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EXEGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FHAW (Funk–Hecke Adaptive Weighting), a lightweight patch gating module for visual Transformers. It projects and normalizes patch features onto a sphere, generates weights using Gegenbauer polynomials based on their similarity to learnable reference directions, and then modulates the patch features to improve performance on visual tasks.

Critical strengths
  1. Each module has approximately 25k parameters, making it easy to integrate into existing ViTs.

  2. The spherical angular frequency interpretation and orthogonal basis selection are mathematically supported.

  3. Some results have been validated with multiple seeds, and failure conditions and negative results are also reported.

Critical weaknesses
  1. Gegenbauer gating has not yet been proven to outperform conventional MLP gating within the same budget.

  2. Most improvements are small and dependent on training methods, insertion positions, and output formats.

Questions for authors
  1. With fixed projection, reference direction, and output format, and only replaced with an MLP of the same budget, can FHAW still maintain its leading position?

  2. Can the gains be retained on a strong baseline with sufficient training? Especially since Swin-B's current results are still below the official checkpoint.

Reviewer_CsTUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the Cartesian weighting disaster in Vision Transformers: vanilla point‑wise weighting in Cartesian feature space cannot cleanly manipulate angular‑frequency components of individual patch tokens, and existing frequency‑aware ViT methods operate globally on 2D feature grids and share frequency weights across all patches. The authors leverage spherical harmonic analysis, Gegenbauer polynomials and the Funk–Hecke theorem to propose Funk‑Hecke Adaptive Weighting (FHAW), a lightweight, plug‑and‑play module. FHAW projects patch features onto a low‑dimensional unit hypersphere, computes cosine similarity against learnable reference directions, builds a band‑limited gate via Gegenbauer spectral expansion and Clenshaw recursion, and applies per‑patch soft modulation with demean modulation. FHAW provides a well‑motivated spherical angular‑frequency inductive bias for per‑patch feature reweighting, while adding only ~25k parameters with O(N)O(N) complexity.

Critical strengths
  1. Novel mathematically‑grounded per‑patch modulation paradigm for ViT internal token space.
  2. Careful empirical methodology with transparent reporting of boundary conditions and failure modes.
Critical weaknesses
  1. Moderate practical performance gains with narrow applicability boundary.
  2. Risk of conceptual misinterpretation between inductive bias and exact spectral filtering.
Questions for authors
  1. You emphasize that FHAW provides an angular‑frequency inductive bias instead of exact diagonal spectral filtering. Could you quantitatively show to what degree spherical‑harmonic orders get mixed in practice for real network activations? For example, measure spectral leakage when feeding real patch features through the learned gate.
  2. Your results show gains vanish when backbone is frozen. What is the concrete mechanism why joint optimization is mandatory? Could you provide more intuition or diagnostic evidence?
  3. Why does the optimal projection dimension sit at dproj=32d_{proj}=32 (not larger/smaller)? Besides conditioning of Gegenbauer basis, are there other competing effects from feature information loss under projection?
  4. In layer‑wise diagnostics you observe best performance inserting FHAW in later layers (L8‑L12). Would you expect inserting FHAW in early layers to ever bring benefits under alternative training regimes, e.g. from‑scratch ViT pre‑training?
Reviewer_cfcbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper propose Funk–Hecke Adaptive Weighting, which enhance features based on a sphere.

Critical strengths

Strength one: The paper presents an interesting perspective: projecting feature tokens onto a sphere and using cosine similarity to determine the degree of feature enhancement. The method also appears adaptable to multiple settings.

Critical weaknesses

Weakness 1: The experiments do not sufficiently support the claim that “Deep attention also aligns token directions and dilutes high-frequency detail.” First, it is unclear why high-frequency components necessarily correspond to useful details. Second, the paper does not clearly establish how the loss of such details with increasing depth is measured. For example, texture information may be distributed across many low-level features in earlier layers but encoded more compactly as a semantic texture representation in deeper layers. A reduction in spectral bandwidth therefore does not necessarily imply a loss of useful information.

Weakness 2: The method’s generalizability appears more limited than claimed. Each new task requires retraining the FHAW projection head and the Gegenbauer spectral coefficients. These components may also depend strongly on the data distribution, potentially requiring refitting when switching datasets. Given the limited empirical gains, it is unclear whether the improvements justify the additional computational overhead.

Questions for authors

Question 1: What is the specific benefit of using Gegenbauer polynomials? Could a small neural network learn a comparable weighting function? An ablation against such an alternative would help establish whether this design choice is necessary.

Question 2: To what extent do the gains come from fine-tuning the ViT backbone rather than from the proposed method? Line 443 states that “Freezing the backbone removes the gain.” Could the authors clarify this finding?

Question 3: Why is projection onto a sphere necessary? Could the authors provide its advantage over an unconstrained feature space learned by an MLP?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.