- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Fourier Positional Embedding for Vision Transformers (FViT), a positional encoding mechanism motivated by the Fourier shift theorem. Unlike standard RoPE, which applies position-dependent rotations to queries and keys, FViT predicts query-conditioned spatial frequencies and uses relative-position-dependent phase rotations to modulate value contributions, while retaining the standard attention logits and softmax normalization. The method is evaluated on CIFAR-100 and ImageNet classification, as well as COCO object detection. The results show performance broadly comparable to RoPE on classification and a different precision–recall trade-off on object detection, with FViT achieving higher average recall but lower average precision than RoPE. While the proposed formulation is technically plausible and the empirical results demonstrate its feasibility, the novelty over existing Fourier-based positional encodings and relative positional encoding mechanisms is insufficiently established. Moreover, the experiments do not evaluate resolution extrapolation, an important property commonly associated with RoPE and a natural motivation for studying alternative positional encoding schemes.
Critical strengths
(1) A clear and implementable formulation. FViT provides a straightforward way to incorporate relative spatial information into value aggregation through query-conditioned phase rotations. The formulation preserves the standard attention probability computation and can be integrated into existing Transformer architectures.
(2) Evaluation across multiple vision tasks. The paper studies both image classification and object detection, rather than relying on a single benchmark. The reported COCO results suggest that FViT may exhibit a different precision–recall trade-off from RoPE, particularly in average-recall metrics.
Critical weaknesses
(1) The methodological novelty is limited and insufficiently distinguished from prior work. Fourier-based positional representations have been extensively studied, including Learnable Fourier Features (NeurIPS 2021), while FourierLearner-Transformers (AISTATS 2024) explores spectral representations for relative positional encoding. Moreover, injecting relative positional information into value contributions is also an established direction. FViT's main distinction appears to be the combination of query-conditioned frequency prediction and multiplicative phase rotations of values. Although this specific parameterization may be new, the paper does not sufficiently establish why it constitutes a substantial conceptual advance over these existing approaches. The current presentation risks overstating the contribution by emphasizing its Fourier motivation without adequately isolating the genuinely novel component.
(2) The experiments do not evaluate resolution extrapolation, leaving a major motivation untested. Resolution extrapolation is a commonly cited desirable property of RoPE and is particularly relevant to vision Transformers, where models may be trained at one image resolution and evaluated at substantially larger resolutions. However, the paper reports classification and detection performance under its chosen training and evaluation configurations without systematically testing transfer from a source resolution to unseen, higher-resolution grids. There are no controlled cross-resolution evaluations, no performance-versus-resolution curves, and no comparisons of extrapolation robustness against RoPE. Consequently, the experiments do not establish whether FViT offers any advantage in the setting that would make its alternative positional encoding particularly compelling. This is a substantial gap, rather than a minor missing experiment.
(3) The empirical results provide limited evidence of an overall advantage over RoPE. On ImageNet, FViT achieves 80.81% Top-1 accuracy compared with 81.01% for RoPE, while its Top-5 accuracy is only 0.01 percentage points higher and its test loss is worse. On CIFAR-100, FViT also has lower Top-1 accuracy than RoPE, although it performs better on Top-5 accuracy and test loss. On COCO, FViT improves average recall relative to RoPE but consistently underperforms on the reported average-precision metrics. These results suggest a task- and metric-dependent trade-off rather than a clear improvement. The paper does not provide sufficient evidence that the proposed mechanism offers a compelling advantage in predictive performance, efficiency, or generalization.
(4) The key design choices are not adequately validated through ablations. The proposed method combines query-conditioned frequency prediction with value-side phase rotation, yet the experiments do not sufficiently isolate the contribution of each component. In particular, comparisons against fixed-frequency rotations, globally learned but query-independent frequencies, and alternative placements of the phase operation would be important to establish whether query conditioning is necessary and whether value-side rotation is responsible for the observed behavior. Without such comparisons, it remains unclear whether the proposed complexity is justified or whether simpler variants could achieve comparable results.
(5) The theoretical motivation is stronger than the theoretical conclusions supported by the paper. The Fourier shift theorem provides a reasonable motivation for representing spatial displacement through phase factors. However, this observation alone does not establish improved translation equivariance, resolution extrapolation, or generalization. Since the frequencies in FViT depend on the query representation, the resulting operation is not a single fixed representation of translations across different inputs. The paper acknowledges this distinction but does not establish further theoretical properties that would explain why the proposed mechanism should outperform existing alternatives. The Fourier interpretation therefore serves primarily as design motivation rather than a substantive theoretical justification for the claimed benefits.
Questions for authors
See weaknesses