- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents FPP-Former, a U-shaped hierarchical Transformer for medical image segmentation. Its core component is an efficient attention mechanism called Fourier Principal Projection (also called Fourier Principal Linear Attention, FPLA).
Critical strengths
- Replacing a data-dependent PCA basis with a Fourier basis via cyclic averaging is a neat construction. Compressing keys and values along the token axis with a structured basis that an FFT can compute is a natural extension of Linformer-style methods.
- The steps from the shift-averaged statistics to the Fourier diagonalization are sound. So is the optimality of picking the highest-energy frequencies for the averaged problem, and the real-valued basis construction is handled carefully.
- At high resolution the method sharply reduces compute, memory and latency compared with full self-attention. At 112×112 features, FLOPs drop by roughly 65 times and training memory from about 20 GB to well under 0.1 GB. It also compares favourably with other efficient attention mechanisms in memory use and inference speed.
- Comparing attention variants inside the same backbone is the right experiment for isolating the contribution of the attention operator.
- The random-subspace and random-frequency baselines and the spectral-energy analysis are useful.
- The method is evaluated on three datasets covering MRI and CT, and on both 3D and 2D settings.
Critical weaknesses
- The linear-complexity claim does not hold for the configuration used. The default number of retained modes is one eighth of the token count, so attention cost stays quadratic, just eight times smaller. Truly linear scaling would need a fixed number of modes that does not grow with resolution.
- The scaling results are inconsistent with the default setting. In the resolution-scaling table, the FLOPs of the proposed module stay almost constant while the token count grows 64-fold. Either a fixed number of modes was used there, or the FLOP count leaves out the projection or FFT operations.
- The low-rank theorem is trivial. Bounding the rank of the score matrix by the head dimension is elementary, and the approximation part restates a textbook result (Eckart–Young). Calling it a theorem overstates the theoretical contribution.
- The theorem contradicts the chosen subspace size. If the scores and token correlations have rank at most the head dimension (typically 32–64), it is unclear why over a thousand modes are needed at high resolution. The theorem also covers scores before the softmax, while what matters for compressing keys and values is the attention after it.
- The Fourier surrogate's gap to true PCA is never measured. Averaging over cyclic shifts raises the rank and can move the solution far from the real PCA subspace. The paper gives neither a theoretical nor an empirical measure of this gap.
- The "data-dependent principal subspace" is essentially adaptive frequency selection. After cyclic averaging, the basis vectors are always plain Fourier modes, and only the choice of frequencies depends on the data.
- Shift-averaging conflicts with the anatomical motivation. Averaging over shifts discards spatial localization, which contradicts the stated goal of capturing spatially coherent anatomical regions.
- The dominant modes are likely just the constant component and low frequencies. About 80% of the energy sits in a tiny fraction of a percent of the components. This is typical of features that are not mean-centred, so the top "principal component" may simply act as global average pooling.
- A fixed low-pass baseline is missing. Without comparing against simply keeping the lowest frequencies, the benefit of energy-based selection is unclear. A random-frequency baseline is a weak comparison.
- Closely related prior work is missing. FEDformer, Fourier Neural Operator, AFNO, GFNet, FNet and Fourier Transformer are neither discussed nor compared. Neither is Linformer with a fixed Fourier or DCT projection.
- It is unclear how the transform handles 2D and 3D data. The derivation assumes a 1D token sequence. If feature maps are flattened in raster order, cyclic shifts wrap across rows and slices, and the frequencies lose their spatial meaning. The paper does not say whether a 1D or a 2D/3D transform is used.
- Gains are small and lack statistics. The improvements on Synapse and ACDC are about one tenth of a DSC point, with no standard deviations, multiple seeds or significance tests. The Synapse test set has only 12 cases.
- Strong baselines are omitted or excluded. nnU-Net beats the proposed method on Synapse by about two DSC points. nnU-Net, Swin UNETR, nnFormer and UNETR++ are missing from the BraTS comparison.
- Baseline numbers may not match the original papers. To my recollection, the UNETR++ and nnFormer papers report higher results on Synapse, ACDC and the MSD brain-tumour set. It is unclear whether baselines were re-trained or copied from papers with different protocols.
- Full self-attention is more accurate on BraTS. In the same backbone, it beats the proposed attention, which contradicts the claim that removing redundant interactions improves accuracy. The real contribution appears to be efficiency at a small cost in accuracy.
- The subspace-size results are non-monotonic and weakly explained. Using more modes than the default lowers DSC even though the retained energy barely changes. This may simply be noise from single runs. Key efficient-attention baselines are missing. Performer, Nyströmformer, a fixed low-pass projection, and above all an exact or randomized PCA basis are absent. The PCA basis is the key check that the Fourier approximation works.
- The efficiency setup is underspecified. Dataset, input size, batch size and hardware are not given. The reported self-attention training memory exceeds a single 24 GB GPU.
- There is no whole-model efficiency comparison. Parameter counts and end-to-end compute and latency are not compared against nnU-Net, UNETR++ or SegFormer3D.
- Structure-awareness claims are unsupported. There are no visualizations of attention or selected modes, and no analysis of which frequencies are chosen. Boundary quality is measured on only one dataset.
- Architecture details are missing. Depth per stage, channel widths, number of heads, convolutional components and positional encoding are not reported. It is also unclear which features the basis is computed from.
- Algorithmic details are missing. The paper does not say whether the basis is computed per sample, head, layer or batch, or how batching works when samples select different frequencies. It also does not say whether the projection uses an explicit dense basis or an FFT with frequency gathering.
Questions for authors
- Since the default number of modes grows proportionally with the number of tokens, the attention cost remains quadratic. How do you justify the linear-complexity claim? What number of modes is used in the resolution-scaling experiment, and why do the reported FLOPs stay almost constant as resolution increases?
- Given that the attention scores and the token correlation statistics are both limited in rank by the head dimension, why does performance require far more modes than the head dimension? How does the method compare, in accuracy and runtime, with projecting onto an exact or randomized PCA basis?
- Can you quantify how closely the Fourier basis matches the true PCA subspace on real features from each stage, for example through subspace angles or the fraction of the original energy captured?
- Are the features mean-centred before computing spectral energy? How much of the reported energy concentration comes from the constant (zero-frequency) component alone?
- How does energy-based frequency selection compare with simply keeping a fixed set of the lowest frequencies? How much do the selected frequencies vary across samples and layers?
- Is the Fourier transform applied in 1D to the flattened token sequence, or in 2D/3D over the spatial grid? If 1D, how do you justify the wrap-around between rows and slices?
- Is the basis computed per sample, per head and per layer? How is batching handled when different samples select different frequencies? Does any gradient pass through the selection step?
- Were the baselines re-trained under your protocol or copied from the literature? Their original papers appear to report higher numbers on Synapse and ACDC. Why are nnU-Net, nnFormer, UNETR++ and Swin UNETR absent from the BraTS comparison?
- Can you report means and standard deviations over several seeds, or significance tests, for the main results and the subspace-size ablation? Gains of about one tenth of a DSC point are otherwise hard to interpret.
- Full self-attention outperforms your method on BraTS. Do you agree that the main contribution is efficiency rather than better structural modelling? If not, what evidence supports the structure-awareness claim?
- On which dataset, input size and hardware were the efficiency numbers measured? How was full self-attention trained given memory requirements above a single 24 GB GPU?
- How does the method relate to FEDformer, Fourier Neural Operator, AFNO, GFNet and Fourier Transformer? Can you compare against Linformer with a fixed Fourier or DCT projection?
- What are the total parameter counts and end-to-end compute costs of FPP-Former compared with nnU-Net, UNETR++ and SegFormer3D?
- Will the code be released?