acceptodds
Under review as a conference paper at ICLR 2027

Fourier Positional Embeddings for Vision Transformers

Abstract

Convolutional neural networks (CNNs) have long shaped progress in computer vision by incorporating strong inductive biases such as translation equivariance and locality. Vision Transformers (ViTs) now reach comparable performance across many tasks, but rely on weaker built-in spatial priors and instead depend on positional encodings to represent image structure. We introduce Fourier Positional Embedding for Vision Transformers (FViT), a positional encoding inspired by Fourier analysis. Rather than adding positional embeddings to token representations, FViT represents relative spatial structure through learned complex-valued phase factors that modulate attention, preserving the flexibility of self-attention while better aligning positional encoding with image geometry. Empirically, FViT substantially outperforms absolute, relative, and sinusoidal positional embeddings and matches Rotary Positional Embeddings (RoPE) on CIFAR-100 and ImageNet. On COCO object detection, FViT substantially improves over the standard YOLOS baseline under a matched training setup and achieves higher average recall than RoPE across all reported recall metrics, while RoPE retains an advantage in average precision. Finally, we argue that FViT and RoPE perform strongly because their phase-based positional operations reflect how translations act through Fourier characters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
artificial intelligence, deep learning, computer vision, vision transformer, Fourier transform, harmonic analysis, positional encoding, Fourier features, rotary embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uVgGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Fourier Positional Embedding for Vision Transformers (FViT), a positional encoding mechanism motivated by the Fourier shift theorem. Unlike standard RoPE, which applies position-dependent rotations to queries and keys, FViT predicts query-conditioned spatial frequencies and uses relative-position-dependent phase rotations to modulate value contributions, while retaining the standard attention logits and softmax normalization. The method is evaluated on CIFAR-100 and ImageNet classification, as well as COCO object detection. The results show performance broadly comparable to RoPE on classification and a different precision–recall trade-off on object detection, with FViT achieving higher average recall but lower average precision than RoPE. While the proposed formulation is technically plausible and the empirical results demonstrate its feasibility, the novelty over existing Fourier-based positional encodings and relative positional encoding mechanisms is insufficiently established. Moreover, the experiments do not evaluate resolution extrapolation, an important property commonly associated with RoPE and a natural motivation for studying alternative positional encoding schemes.

Critical strengths

(1) A clear and implementable formulation. FViT provides a straightforward way to incorporate relative spatial information into value aggregation through query-conditioned phase rotations. The formulation preserves the standard attention probability computation and can be integrated into existing Transformer architectures.

(2) Evaluation across multiple vision tasks. The paper studies both image classification and object detection, rather than relying on a single benchmark. The reported COCO results suggest that FViT may exhibit a different precision–recall trade-off from RoPE, particularly in average-recall metrics.

Critical weaknesses

(1) The methodological novelty is limited and insufficiently distinguished from prior work. Fourier-based positional representations have been extensively studied, including Learnable Fourier Features (NeurIPS 2021), while FourierLearner-Transformers (AISTATS 2024) explores spectral representations for relative positional encoding. Moreover, injecting relative positional information into value contributions is also an established direction. FViT's main distinction appears to be the combination of query-conditioned frequency prediction and multiplicative phase rotations of values. Although this specific parameterization may be new, the paper does not sufficiently establish why it constitutes a substantial conceptual advance over these existing approaches. The current presentation risks overstating the contribution by emphasizing its Fourier motivation without adequately isolating the genuinely novel component.

(2) The experiments do not evaluate resolution extrapolation, leaving a major motivation untested. Resolution extrapolation is a commonly cited desirable property of RoPE and is particularly relevant to vision Transformers, where models may be trained at one image resolution and evaluated at substantially larger resolutions. However, the paper reports classification and detection performance under its chosen training and evaluation configurations without systematically testing transfer from a source resolution to unseen, higher-resolution grids. There are no controlled cross-resolution evaluations, no performance-versus-resolution curves, and no comparisons of extrapolation robustness against RoPE. Consequently, the experiments do not establish whether FViT offers any advantage in the setting that would make its alternative positional encoding particularly compelling. This is a substantial gap, rather than a minor missing experiment.

(3) The empirical results provide limited evidence of an overall advantage over RoPE. On ImageNet, FViT achieves 80.81% Top-1 accuracy compared with 81.01% for RoPE, while its Top-5 accuracy is only 0.01 percentage points higher and its test loss is worse. On CIFAR-100, FViT also has lower Top-1 accuracy than RoPE, although it performs better on Top-5 accuracy and test loss. On COCO, FViT improves average recall relative to RoPE but consistently underperforms on the reported average-precision metrics. These results suggest a task- and metric-dependent trade-off rather than a clear improvement. The paper does not provide sufficient evidence that the proposed mechanism offers a compelling advantage in predictive performance, efficiency, or generalization.

(4) The key design choices are not adequately validated through ablations. The proposed method combines query-conditioned frequency prediction with value-side phase rotation, yet the experiments do not sufficiently isolate the contribution of each component. In particular, comparisons against fixed-frequency rotations, globally learned but query-independent frequencies, and alternative placements of the phase operation would be important to establish whether query conditioning is necessary and whether value-side rotation is responsible for the observed behavior. Without such comparisons, it remains unclear whether the proposed complexity is justified or whether simpler variants could achieve comparable results.

(5) The theoretical motivation is stronger than the theoretical conclusions supported by the paper. The Fourier shift theorem provides a reasonable motivation for representing spatial displacement through phase factors. However, this observation alone does not establish improved translation equivariance, resolution extrapolation, or generalization. Since the frequencies in FViT depend on the query representation, the resulting operation is not a single fixed representation of translations across different inputs. The paper acknowledges this distinction but does not establish further theoretical properties that would explain why the proposed mechanism should outperform existing alternatives. The Fourier interpretation therefore serves primarily as design motivation rather than a substantive theoretical justification for the claimed benefits.

Questions for authors

See weaknesses

Reviewer_nNDVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a new method of positional embeddings in vision transformers, motivated by Fourier analysis, which seeks to better represent spatial structure. The authors argue that absolute positional embeddings require the model to learn how geometry influences the pairwise interactions, instead of making that structure clear directly. To solve this, the authors introduce a new type of position positional embedding which is head-specific and learned, and uses a complex-valued phase between query-key interactions, with the idea that spatial displacement is then converted into phase rotation. Experiments are conducted on CIFAR-100, ImageNet, and COCO, showing that this choice of positional embedding performs comparably to methods like RoPE over long trainings.

Critical strengths

No part of this paper stands out to me as being very strong and supporting acceptance.

Critical weaknesses
  1. On the theoretical formulation and motivation, i have several concerns. First, much of the motivation is introduced by comparing against absolute positional encodings, despite RoPE (the most popular current method) not being absolute. Additionally, RoPE operates under a similar idea, using translation operations via the Fourier shift theorem (just changing whether these are applied to the Key+Query or Value, and whether they are learned or fixed).

  2. On the practical side, as far as I can tell, the approach is only compared to the standard 1-dimensional RoPE, and not existing 2D RoPE modifications (for example [1]). Additionally, even in these comparisons, the proposed Fourier PE seems to be significantly slower than RoPE for CIFAR-10 (reaching top-1 accuracy slightly lower than RoPE at a much slower pace, and requiring a very large number of epochs to reach that point, seen in Figure 2). The same seems to be true for ImageNet (Table 2 and Figure 3), and in COCO RoPE heavily outperforms on all AP metrics, while FViT does slightly better than RoPE on AR metrics. Overall, I do not see a significant or clear experimental benefit for using this choice of PE, and if anything, it requires much more training (much slower start than RoPE). Finally, unless I am missing something, despite the paper's claims about scalability to high-resolution inputs, it does not seem like this approach is ever tested on high-resolution images.

[1] Heo, Byeongho, et al. "Rotary position embedding for vision transformer." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

Questions for authors
  1. How exactly does your method compare to the 2D versions of RoPE, both theoretically and empirically?
  2. Do you have any evidence that this approach does in fact generalize well to high-resolution inputs?
  3. Why does this method have such a significant lag in performance, that requires a significant amount of training to even catch up to RoPE?
Reviewer_PoL4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fourier Positional Embedding (FViT), a positional encoding method that incorporates spatial information into Vision Transformers by applying query-dependent phase rotations to value representations. Unlike RoPE, which modifies query–key interactions, FViT preserves the standard attention weights and introduces relative positional information during value aggregation. The method draws on the relationship between spatial translations and complex-valued phase shifts in Fourier analysis, with learnable frequency parameters controlling the rotations. Experiments on CIFAR-100, ImageNet, and COCO evaluate the approach across image classification and object detection. The results show that FViT achieves classification performance comparable to RoPE and improves upon the standard YOLOS baseline in object detection, although it favors recall over precision relative to RoPE. The paper provides an alternative perspective on incorporating positional information into attention through content-dependent value modulation.

Critical strengths
  1. FViT preserves the standard softmax computation in attention while incorporating relative positional information through complex-valued phase rotations, resulting in a relatively simple design.

  2. FViT applies phase rotations to values using query-dependent frequency parameters, distinguishing it from RoPE, which rotates queries and keys, and offering an alternative approach to positional encoding.

Critical weaknesses
  1. The theoretical analysis primarily relies on basic properties of complex exponential functions to motivate the Fourier structure, but does not establish that query-dependent phase rotations lead to improved spatial modeling or translation equivariance. As a result, the theoretical analysis provides limited explanatory insight.

  2. The method offers limited performance improvements over existing approaches.

  3. The paper lacks key ablation studies and mechanistic analyses, such as disentangling the individual contributions of query-dependent frequency learning, value rotations, and the Fourier phase structure.

Questions for authors
  1. What evidence supports the necessity of query-dependent frequency learning and value rotations? Could the authors clarify their individual contributions, particularly compared with fixed-frequency rotations or simpler content-dependent value modulation?

  2. FViT does not outperform RoPE in Top-1 accuracy on either classification dataset and underperforms RoPE across all COCO AP metrics. What concrete advantages justify the additional mechanism, and are the relatively modest improvements in recall consistent across multiple random seeds?

  3. Equations (3)–(8) assume that tokens have well-defined 2D spatial coordinates. How does FViT handle class tokens and detection tokens, which do not naturally correspond to image patches? Are positional rotations applied to their attention interactions, and how does this choice affect the method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.