acceptodds
Withdrawn from ICLR 2027

Fourier Principal Projected Transformer for Efficient and Accurate Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurate medical image segmentation requires reliable modeling of global anatomical structure in high-resolution medical images. However, existing Transformer-based methods lack explicit global structural representation and suffer from quadratic attention complexity, limiting their effectiveness and efficiency in dense prediction settings. To address these limitations, we propose the Fourier Principal Projected Transformer (FPP-Former). Specifically, FPP-Former leverages the intrinsic global correlations of spatial tokens to derive principal component representations that characterize the major variations in global anatomical appearance, enabling explicit modeling of global anatomical organization. For efficient principal component modeling, we reformulate this process in the Fourier domain, where a low-dimensional principal subspace is constructed from dominant frequency components, avoiding explicit eigen-decomposition on high-dimensional feature representations. The resulting Fourier principal components are then fed into the attention model to guide key–value interactions, where keys and values are projected into a shared principal space for structure-guided attention computation and value aggregation, thereby focusing attention on spatially continuous and structurally coherent anatomical regions. This formulation results in linear-complexity attention computation, improving efficiency for high-resolution segmentation. Extensive experiments on BraTS 2017, Synapse, and ACDC show that FPP-Former achieves average Dice scores of 83.2%, 84.84%, and 91.94%, with relative improvements of 2.34%, 0.11%, and 0.12%, respectively, while consistently improving training and inference efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Image Segmentation, Principal Component Analysis, Fourier Transform, Structure-Guided Attention, Efficient Transformer.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tJR7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents FPP-Former, a U-shaped hierarchical Transformer for medical image segmentation. Its core component is an efficient attention mechanism called Fourier Principal Projection (also called Fourier Principal Linear Attention, FPLA).

Critical strengths
  1. Replacing a data-dependent PCA basis with a Fourier basis via cyclic averaging is a neat construction. Compressing keys and values along the token axis with a structured basis that an FFT can compute is a natural extension of Linformer-style methods.
  2. The steps from the shift-averaged statistics to the Fourier diagonalization are sound. So is the optimality of picking the highest-energy frequencies for the averaged problem, and the real-valued basis construction is handled carefully.
  3. At high resolution the method sharply reduces compute, memory and latency compared with full self-attention. At 112×112 features, FLOPs drop by roughly 65 times and training memory from about 20 GB to well under 0.1 GB. It also compares favourably with other efficient attention mechanisms in memory use and inference speed.
  4. Comparing attention variants inside the same backbone is the right experiment for isolating the contribution of the attention operator.
  5. The random-subspace and random-frequency baselines and the spectral-energy analysis are useful.
  6. The method is evaluated on three datasets covering MRI and CT, and on both 3D and 2D settings.
Critical weaknesses
  1. The linear-complexity claim does not hold for the configuration used. The default number of retained modes is one eighth of the token count, so attention cost stays quadratic, just eight times smaller. Truly linear scaling would need a fixed number of modes that does not grow with resolution.
  2. The scaling results are inconsistent with the default setting. In the resolution-scaling table, the FLOPs of the proposed module stay almost constant while the token count grows 64-fold. Either a fixed number of modes was used there, or the FLOP count leaves out the projection or FFT operations.
  3. The low-rank theorem is trivial. Bounding the rank of the score matrix by the head dimension is elementary, and the approximation part restates a textbook result (Eckart–Young). Calling it a theorem overstates the theoretical contribution.
  4. The theorem contradicts the chosen subspace size. If the scores and token correlations have rank at most the head dimension (typically 32–64), it is unclear why over a thousand modes are needed at high resolution. The theorem also covers scores before the softmax, while what matters for compressing keys and values is the attention after it.
  5. The Fourier surrogate's gap to true PCA is never measured. Averaging over cyclic shifts raises the rank and can move the solution far from the real PCA subspace. The paper gives neither a theoretical nor an empirical measure of this gap.
  6. The "data-dependent principal subspace" is essentially adaptive frequency selection. After cyclic averaging, the basis vectors are always plain Fourier modes, and only the choice of frequencies depends on the data.
  7. Shift-averaging conflicts with the anatomical motivation. Averaging over shifts discards spatial localization, which contradicts the stated goal of capturing spatially coherent anatomical regions.
  8. The dominant modes are likely just the constant component and low frequencies. About 80% of the energy sits in a tiny fraction of a percent of the components. This is typical of features that are not mean-centred, so the top "principal component" may simply act as global average pooling.
  9. A fixed low-pass baseline is missing. Without comparing against simply keeping the lowest frequencies, the benefit of energy-based selection is unclear. A random-frequency baseline is a weak comparison.
  10. Closely related prior work is missing. FEDformer, Fourier Neural Operator, AFNO, GFNet, FNet and Fourier Transformer are neither discussed nor compared. Neither is Linformer with a fixed Fourier or DCT projection.
  11. It is unclear how the transform handles 2D and 3D data. The derivation assumes a 1D token sequence. If feature maps are flattened in raster order, cyclic shifts wrap across rows and slices, and the frequencies lose their spatial meaning. The paper does not say whether a 1D or a 2D/3D transform is used.
  12. Gains are small and lack statistics. The improvements on Synapse and ACDC are about one tenth of a DSC point, with no standard deviations, multiple seeds or significance tests. The Synapse test set has only 12 cases.
  13. Strong baselines are omitted or excluded. nnU-Net beats the proposed method on Synapse by about two DSC points. nnU-Net, Swin UNETR, nnFormer and UNETR++ are missing from the BraTS comparison.
  14. Baseline numbers may not match the original papers. To my recollection, the UNETR++ and nnFormer papers report higher results on Synapse, ACDC and the MSD brain-tumour set. It is unclear whether baselines were re-trained or copied from papers with different protocols.
  15. Full self-attention is more accurate on BraTS. In the same backbone, it beats the proposed attention, which contradicts the claim that removing redundant interactions improves accuracy. The real contribution appears to be efficiency at a small cost in accuracy.
  16. The subspace-size results are non-monotonic and weakly explained. Using more modes than the default lowers DSC even though the retained energy barely changes. This may simply be noise from single runs. Key efficient-attention baselines are missing. Performer, Nyströmformer, a fixed low-pass projection, and above all an exact or randomized PCA basis are absent. The PCA basis is the key check that the Fourier approximation works.
  17. The efficiency setup is underspecified. Dataset, input size, batch size and hardware are not given. The reported self-attention training memory exceeds a single 24 GB GPU.
  18. There is no whole-model efficiency comparison. Parameter counts and end-to-end compute and latency are not compared against nnU-Net, UNETR++ or SegFormer3D.
  19. Structure-awareness claims are unsupported. There are no visualizations of attention or selected modes, and no analysis of which frequencies are chosen. Boundary quality is measured on only one dataset.
  20. Architecture details are missing. Depth per stage, channel widths, number of heads, convolutional components and positional encoding are not reported. It is also unclear which features the basis is computed from.
  21. Algorithmic details are missing. The paper does not say whether the basis is computed per sample, head, layer or batch, or how batching works when samples select different frequencies. It also does not say whether the projection uses an explicit dense basis or an FFT with frequency gathering.
Questions for authors
  1. Since the default number of modes grows proportionally with the number of tokens, the attention cost remains quadratic. How do you justify the linear-complexity claim? What number of modes is used in the resolution-scaling experiment, and why do the reported FLOPs stay almost constant as resolution increases?
  2. Given that the attention scores and the token correlation statistics are both limited in rank by the head dimension, why does performance require far more modes than the head dimension? How does the method compare, in accuracy and runtime, with projecting onto an exact or randomized PCA basis?
  3. Can you quantify how closely the Fourier basis matches the true PCA subspace on real features from each stage, for example through subspace angles or the fraction of the original energy captured?
  4. Are the features mean-centred before computing spectral energy? How much of the reported energy concentration comes from the constant (zero-frequency) component alone?
  5. How does energy-based frequency selection compare with simply keeping a fixed set of the lowest frequencies? How much do the selected frequencies vary across samples and layers?
  6. Is the Fourier transform applied in 1D to the flattened token sequence, or in 2D/3D over the spatial grid? If 1D, how do you justify the wrap-around between rows and slices?
  7. Is the basis computed per sample, per head and per layer? How is batching handled when different samples select different frequencies? Does any gradient pass through the selection step?
  8. Were the baselines re-trained under your protocol or copied from the literature? Their original papers appear to report higher numbers on Synapse and ACDC. Why are nnU-Net, nnFormer, UNETR++ and Swin UNETR absent from the BraTS comparison?
  9. Can you report means and standard deviations over several seeds, or significance tests, for the main results and the subspace-size ablation? Gains of about one tenth of a DSC point are otherwise hard to interpret.
  10. Full self-attention outperforms your method on BraTS. Do you agree that the main contribution is efficiency rather than better structural modelling? If not, what evidence supports the structure-awareness claim?
  11. On which dataset, input size and hardware were the efficiency numbers measured? How was full self-attention trained given memory requirements above a single 24 GB GPU?
  12. How does the method relate to FEDformer, Fourier Neural Operator, AFNO, GFNet and Fourier Transformer? Can you compare against Linformer with a fixed Fourier or DCT projection?
  13. What are the total parameter counts and end-to-end compute costs of FPP-Former compared with nnU-Net, UNETR++ and SegFormer3D?
  14. Will the code be released?
Reviewer_s2TFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a U-shaped medical-image segmentation model whose global attention compresses keys and values into data-selected Fourier modes. It computes channel-wise Fourier energy, selects the top k modes, constructs a real Fourier basis, projects K and V into that basis, and lets each query attend to the resulting k rows. The selected modes are the leading eigenspace of a cyclically symmetrized feature covariance, not in general the principal subspace of the original token covariance or of the softmax attention operator. Experiments on BraTS 2017, Synapse, and ACDC report reduced high-resolution compute/memory and small or moderate Dice improvements over selected baselines. The authors claim that Fourier principal projection captures dominant global anatomy while reducing attention to linear complexity.

Critical strengths
  1. The Fourier transform provides an efficient eigensystem for a circulant matrix, and the submission uses this structure to construct data-dependent compressed tokens without dense eigendecomposition.
  2. The paper reports runtime and memory by resolution, revealing that the method is most useful at high resolution and can lose its efficiency advantage at low resolution. This is practically useful information if the claims are narrowed.
Critical weaknesses
  1. The theorem rank(QKT)≤dhrank(QK^{T}) \leq d_h is a standard factorization fact about pre-softmax logits. It neither proves that softmax attention is low-rank nor links the leading directions of the score matrix to the Fourier eigenspace of XXTXX^{T}. Cyclic symmetrization changes the covariance objective and imposes circular stationarity; the paper does not show that this approximation preserves the directions relevant to attention or segmentation.

  2. The advertised asymptotic linear complexity does not hold under the reported setting k/N=1/8: the appendix gives O(dhNlogN+Nkdh)O(d_h N log N + N k d_h), which is quadratic when kk grows proportionally to NN. The interaction map is likewise O(Nk)O(Nk) memory. The segmentation gains are marginal on two of the three datasets, without multi-seed variation, while the study lacks exact-PCA or strong learned-landmark controls needed to support the claimed principal-mode mechanism.

  3. The evaluation does not yet establish a reliable advantage under a traceable common protocol. Appendix B/Table 6 identifies 484 cases and a 411/73 split as BraTS 2017. However, the source and label release need reconciliation, particularly since UNETR describes 484 labeled brain-tumor cases as MSD. On Synapse, the paper's own nnU-Net baseline has Dice 86.99 versus 84.84 for the proposal, while the gain over UNETR++ is only 0.09 percentage points. The ACDC gain over nnFormer is 0.11 points, with no run-level uncertainty. Standard attention also exceeds the proposed block on the same-backbone brain-tumor ablation. Finally, the nearly constant FPLA FLOPs across the resolutions in Table 8 cannot be interpreted without the corresponding channel counts, actual k values, and operation-counting scope. These issues limit both the accuracy and scalability conclusions.

  4. The empirical improvements are mostly marginal and uncertainty is absent. Synapse and ACDC gains are 0.09 and 0.11 Dice points over the best cited methods; no multi-seed variance or significance intervals are reported. The BraTS gap is larger, but table values/protocols should be carefully reconciled. The results are not enough to support broad claims of better medical segmentation.

Questions for authors
  1. With the main experiments fixing k/N=1/8, the stated O(Nkdh)O(N k d_h) term is quadratic in N. Will the authors correct the linear-complexity claim and provide an implementation/scaling experiment that demonstrates linear asymptotics for a fixed kk or a clearly different FFT-based computation? Also, what argument links the pre-softmax rank bound to the Fourier basis used after softmax?
  2. Can the authors compare the proposed modes against exact PCA of the original covariance, a learned low-rank basis, and matched landmark attention at the same kk, with segmentation metrics and multi-seed intervals? Energy retention alone does not show that high-frequency modes can be discarded without harming rare boundaries.
  3. Please clarify the exact tokenization and attention dimensions for each dataset (especially whether Synapse and ACDC use per-slice 2D attention or a 3D volume sequence), how kk is chosen at each layer, and the validation/checkpoint-selection protocol for Synapse.
Reviewer_igivOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fourier Principal Projected Transformer (FPP-Former), an efficient architecture for medical image segmentation. The key idea is to approximate the principal subspace of token representations using dominant Fourier components and project keys and values into this compact subspace for efficient attention computation. By reducing the number of key-value tokens involved in attention, the proposed method aims to capture global anatomical structures while reducing computational costs. Experiments on BraTS 2017, Synapse, and ACDC demonstrate competitive segmentation performance and improved computational efficiency.

Critical strengths
  1. The paper explores an interesting direction by combining Fourier-domain principal component analysis with efficient self-attention. Using dominant Fourier components to construct a compact projection basis provides a potentially useful approach to reducing redundant token interactions in medical image segmentation.
  2. The paper conducts experiments on three medical image segmentation benchmarks and provides comparisons of segmentation accuracy, computational complexity, memory consumption, and inference speed. The reported efficiency improvements over standard self-attention are encouraging.
Critical weaknesses
  1. The claimed linear computational complexity is not fully justified. The paper repeatedly claims that the proposed attention mechanism reduces computational complexity from quadratic to linear. However, according to Eq. (62), the overall complexity is (O(d_h N\log N+Nkd_h)). In the experiments, the authors set (k=N/8), which means that the attention computation still has quadratic complexity with respect to (N). Therefore, the claimed linear complexity is inconsistent with the actual experimental configuration. The authors should clarify this issue and provide a more rigorous complexity analysis.
  2. The experimental comparisons are not sufficiently comprehensive, and the performance improvements are limited. Many of the compared methods in Tables 1–3 are relatively old, and stronger recent medical image segmentation approaches are missing. Moreover, FPP-Former does not consistently outperform existing methods. For example, on Synapse, its average Dice score (84.84%) is lower than that of nnU-Net (86.99%), a CNN-based framework published in 2021. On ACDC, the improvement over nnFormer is only 0.11 percentage points. Furthermore, Table 4 shows that standard self-attention achieves a higher Dice score than FPLA on BraTS 2017 (83.54% vs. 83.20%). These results raise questions about the effectiveness and generalizability of the proposed method in terms of segmentation accuracy. More recent baselines and a more balanced discussion of these results are needed.
  3. The ablation studies are insufficient to validate the key technical contributions. Although the paper compares different attention mechanisms and Fourier selection strategies, several important comparisons are missing. In particular, a comparison with direct PCA/SVD-based projection would help demonstrate the necessity of the proposed Fourier approximation. Comparisons with learnable low-rank projections and fixed low-frequency selection would also help clarify whether the improvements arise from the Fourier representation, energy-based component selection, or simply reducing the attention dimension. Furthermore, the impact of applying Fourier analysis to flattened spatial tokens rather than explicitly preserving 2D/3D spatial structure deserves further investigation.
  4. The qualitative comparisons lack sufficient explanation. Figure 2 uses different baseline methods for the three datasets, but the paper does not explain how these methods were selected. The authors should clarify the selection criteria and provide more consistent and informative comparisons. Additionally, highlighting challenging regions and providing detailed analysis of segmentation errors would help demonstrate the specific advantages of the proposed method in capturing anatomical structures and boundaries.
  5. Several presentation issues affect the clarity of the paper.

(1) In Figure 1(b), Q, K, and V appear to be jointly involved in computing the attention matrix (QK^\top), which is inconsistent with the standard attention formulation and the equations in the paper. The computational flow should be corrected or clarified.

(2) The captions of Tables 1 and 2 do not explicitly identify the corresponding datasets (BraTS 2017 and Synapse), making the results less self-contained.

(3) The relationship among FPP-Former, the FPP Block, and Fourier Principal Linear Attention (FPLA) should be explained more clearly. In particular, the transition to FPLA in the ablation section may confuse readers.

Questions for authors

Please see the "Weaknesses".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.