acceptodds
Under review as a conference paper at ICLR 2027

NUViT: Vision Transformers for Fourier-Space Data

Abstract

Vision Transformers (ViTs) have demonstrated strong performance in many computer vision tasks but do not naturally extend beyond the natural image domain. One important example is Fourier-space data, which is used to represent a rich and diverse set of modalities, including medical imaging and microscopy. Standard ViT tokenization partitions an image into square patches on a Cartesian grid, which poorly fit the frequency and phase structure of Fourier-space data. We introduce Non-Uniform ViT, a simple, general-purpose vision transformer that redesigns image tokenization, data representation, and positional encoding to respect this structure. Experiments on Fourier-domain image classification, 2D crystal lattice identification, MRI annotation, and cryo-EM protein heterogeneity analysis show that NUViT consistently captures the structure of Fourier data better than standard architectures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
transformers, vision transformers, fourier space, fourier transform, medical imaging, AI for science, cryo-EM, MRI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ha18OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Non-Uniform ViT (NUViT), a transformer architecture for Fourier-space data. Instead of tokenizing Fourier images with Cartesian square patches, NUViT evaluates the Fourier transform on a non-uniform polar grid via NUFFT, represents values as amplitude plus sine/cosine phase channels, uses random Fourier feature positional encoding and channel embeddings, and then feeds tokens into a standard transformer encoder.

The paper evaluates NUViT on Fourier-domain natural image classification, synthetic 2D crystal lattice classification, MRI severity classification, cryo-EM image classification, and cryo-EM 3D reconstruction by replacing the cryoDRGN encoder. NUViT often outperforms MLP and ViT baselines, with especially large gains in some cryo-EM classification settings.

Critical strengths
  1. The architectural motivation is compelling: standard Cartesian patches are a poor fit for many Fourier-domain signals, and polar/frequency-aware tokenization is a natural inductive bias.

  2. The paper tests the idea across several domains rather than only one benchmark. The cryo-EM classification and amortization results are particularly interesting and suggest that the approach may be useful beyond toy tasks.

Critical weaknesses
  1. Experimental control is not yet strong enough. The paper needs repeated runs, confidence intervals, and matched-budget baselines. NUViT has similar parameter count to ViT but higher per-epoch time and memory, and the current evidence does not fully separate the benefit of polar tokenization from extra preprocessing, representation choices, hyperparameter tuning, or compute.

  2. The baseline set is incomplete for the strength of the claims. A simple log-polar-resampled Fourier image with a standard ViT, a radial-frequency-binned ViT, stronger tuned ViT baselines, and more domain-specific Fourier/cryo-EM baselines would make the result much more convincing. FIT is omitted from cryo-EM due to cost, but the paper still makes broad Fourier-architecture claims.

Questions for authors
  1. Are all reported classification and reconstruction numbers averaged over multiple random seeds? If not, how large is run-to-run variation?

  2. How does NUViT compare to a standard ViT run on a precomputed log-polar or polar-resampled Fourier image without NUFFT inside the model?

  3. Are the baselines matched for token count, input resolution, training time, and hyperparameter tuning budget?

  4. How sensitive is performance to R, N_r, N_phi, patch size, NUFFT oversampling, amplitude versus log-amplitude, and channel positional encoding?

  5. Can the MRI claim be tested on raw k-space data rather than reconstructed images transformed back to Fourier space?

Reviewer_4KuiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Vision Transformer designed for Fourier-space data called NUViT. The key idea is to replace the standard Cartesian Fourier representation with a non-uniform polar representation obtained using NUFFT, and to represent the resulting Fourier coefficients using amplitude and phase channels. The paper evaluates the approach across natural image classification, synthetic crystal classification, MRI classification, and cryo-EM classification and reconstruction. The results show substantial improvements in considered settings, particularly on some of the synthetic cryo-EM and crystal classification benchmarks.

Critical strengths
  1. The paper studies an interesting and relatively underexplored problem: how Transformer tokenization should be adapted when the input naturally lives in Fourier space.

  2. The central idea of aligning tokenization with the geometry and statistics of Fourier data is elegant, intuitive, and potentially useful beyond the specific applications considered here.

  3. The authors evaluate the approach across several domains rather than restricting the study to a single application.

  4. Some of the reported improvements are substantial, especially for crystal classification and the synthetic cryo-EM heterogeneity tasks.

Critical weaknesses
  1. My major concern is on contribution attribution and controlled baselines: the main performance advantage appears to be closely tied to polar NUFFT resampling/tokenization rather than to a fundamentally different Transformer architecture. Indeed, the CIFAR ablation shows a very large degradation when replacing the polar grid with a Cartesian grid, whereas some of the other NUViT-specific components have considerably smaller effects. I believe a stronger set of controlled baselines is necessary to isolate the contribution of sampling geometry from representation, positional encoding, parameter count, token count, and model capacity. In particular, a standard ViT operating on exactly the same polar-resampled input would be an important baseline. Without such experiments, it is difficult to determine whether NUViT should primarily be understood as a new Transformer architecture or as an effective Fourier-domain representation/tokenization strategy.

  2. I believe there are insufficient comparison with alternative approaches. Given the broad claim of a general-purpose architecture for Fourier-space data, the baseline coverage is relatively limited. The empirical comparisons are dominated by standard ViT, MLP, Hartley representations, and FIT. Stronger frequency-aware, non-uniform, or alternative Fourier-domain approaches would make the conclusions significantly more convincing. Comparisons should ideally control for input representation, compute, and model capacity.

  3. In its current form, there are limited real-world and large-scale evaluations. Some of the most impressive gains are obtained on synthetic datasets. In contrast, the gains on the experimental cryo-EM dataset are considerably more modest. Similarly, the MRI experiment does not directly demonstrate learning from raw acquired k-space data, despite this being an important motivation for the method. I would like to see substantially broader validation on real measured Fourier-domain data and larger datasets. At present, it is difficult to assess how well the improvements demonstrated on carefully constructed/synthetic benchmarks will translate to realistic large-scale applications.

  4. Limited understanding of why the method works. The Fourier power-law motivation provides useful intuition for low-frequency-dominated signals, but it does not fully explain the complete set of empirical findings. In particular, NUViT performs extremely well on the crystal classification benchmark, where the relevant spectral structure is qualitatively different from the low-frequency-dominated setting motivating the architecture. It remains unclear whether the improvements arise from denser low-frequency sampling, radial organization, angular organization, improved localization of spectral peaks, NUFFT interpolation/oversampling, the amplitude-phase representation, or some combination of these. More targeted experiments would be valuable for understanding the actual inductive bias responsible for the results.

  5. The downstream improvements are not uniformly strong. While the supervised cryo-EM classification improvements can be very large, the improvements in cryo-EM reconstruction are substantially smaller. NUViT also does not improve over the ViT-Hartley baseline on CIFAR-100-C. These results suggest that the benefits may be more task-dependent than the overall presentation implies.

  6. NUViT introduces non-trivial computational and memory overhead relative to the ViT baseline. Given that some downstream gains are modest, a more systematic accuracy/compute analysis would help establish when the additional complexity is justified.

Questions for authors

I find the core idea elegant and interesting, and I believe that adapting tokenization to the structure of Fourier-space data is a worthwhile direction. However, in its current form, I do not think the experimental evidence is sufficient to support the broader claims made by the paper. Most importantly, the paper needs to more carefully establish what component is actually responsible for the performance gains and attribute the contribution accordingly. Stronger controlled and frequency-aware baselines, substantially broader real-world and large-scale experiments, and experiments providing a more intuitive/mechanistic understanding of why the proposed representation works would significantly strengthen the work. Below are a few suggestions to stregnthen the paper, its contributions and claims:

  1. The ablations suggest that polar NUFFT sampling is the dominant source of improvement. How does NUViT compare with a standard ViT operating on exactly the same polar-resampled input, with token count, parameters, and compute controlled? This distinction is important for understanding whether the main contribution is architectural or primarily a representation/tokenization strategy.

  2. Can the authors clarify why stronger frequency-aware or non-uniform Fourier-domain baselines are not included? Such comparisons seem important for supporting the claim that NUViT is a general-purpose architecture for Fourier-space data.

  3. What common mechanism explains NUViT's improvements across both low-frequency-dominated data and the crystal benchmark, where discriminative spectral structure is qualitatively different? Is the gain primarily due to sampling density, polar geometry, spectral localization, or another factor?

  4. The largest cryo-EM gains are observed on simulated datasets, while improvement on the experimental dataset is modest. Do the authors have additional evidence that these gains generalize to real experimental data and acquisition artifacts?

  5. Regarding MRI, should the current results be interpreted as demonstrating benefits on Fourier-transformed MRI images rather than on natively acquired k-space data? If so, I suggest clarifying the scope of this claim.

Reviewer_2P7JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes NUViT, a ViT for Fourier-space data. It uses a NUFFT to resample the spectrum onto a polar grid, which gives low frequencies more tokens than Cartesian patching does under a 1/f power-law spectrum. Each coefficient is represented as amplitude plus phase (cos/sin), and RFF positional and channel embeddings are added. The method is evaluated on Fourier-domain CIFAR, synthetic crystal lattices, OASIS MRI, CryoBench/EMPIAR cryo-EM classification, and as an encoder for cryoDRGN reconstruction. It outperforms MLP and ViT (real/Hartley) baselines, most clearly on fine-grained, high-noise cryo-EM tasks.

Critical strengths
  1. Important, under-explored problem. Fourier-space data is central to cryo-EM and MRI, yet general-purpose encoders for it have received little attention. The paper's observation, that Cartesian patching mismatches the power-law structure of Fourier spectra, is intuitive and well motivated.

  2. Simple, practical design. NUViT changes only the tokenization and keeps a standard ViT backbone. This makes it easy to adopt, and the paper shows it working as a drop-in encoder inside cryoDRGN.

  3. Strong cryo-EM results. NUViT gives large gains on hard fine-grained cryo-EM tasks, most notably on IgG-RL. It is markedly more robust to noise and generalizes better to unseen images. Quantitative latent-space analyses support these results, which matter for amortized cryo-EM reconstruction.

Critical weaknesses

W1. The gains may come from band-limiting rather than the polar grid. The per-dataset "max radius R" (A.1, A.3, A.6) is never defined in Sec. 2. If R crops the spectrum, NUViT gets a tuned low-pass filter that the baselines lack. NUViT underfitting IgG-1D (36.5% train) while every baseline memorizes is consistent with this. Please add Cartesian and real-space baselines with the same band limit and token count, plus an R sweep.

W2. Baselines and model selection. ViT-S (Hartley) fails to fit the IgG-1D training set (1.66% train, Table 5). NUViT was grid-searched, but the paper does not say whether the baselines were, and no validation split is mentioned. Table 1 lacks a real-space ViT reference. Spectral mixers (GFNet/AFNO), complex-valued networks and polar transformer networks are missing as baselines.

W3. Claims that the tables do not support. NUViT is less robust on CIFAR-C (−15.6 vs −3.9 for ViT on 10-C) and worse on 100-C (18.69 vs 20.13). MLP beats NUViT on Tomotwin-100 reconstruction (Table 6), despite L408. L305's "high-frequency" explanation contradicts the low-frequency motivation. Fig. 1B shows no clear power law for cryo-EM.

W4. No variance reported. All results are single-run. The same configuration scores 12.84% and 13.30% on IgG-1D (Table S7), and several claimed gains are within that noise (Table 6, Table 7 Tomotwin-100, EMPIAR-10076). Please report mean ± std over at least 3 seeds.

W5. MRI experiment. The moderate class has only 2 subjects (A.4), and QWK has no confidence intervals. The simulated k-space is Hermitian-symmetric, which justifies φ ∈ [0, π). Raw complex k-space is not, so the raw-MRI claim (L306) is unsupported. There is no MLP baseline.

W6. The crystal result points to a different mechanism. The energy here is at high frequencies, so the likely reason NUViT wins is that the polar grid exposes the angular symmetry of the Bragg peaks, not that it allocates more capacity to low frequencies. Noise was calibrated on ViT-S (real), and lattice orientation and augmentation are not described.

W7. Novelty and related work. Polar Fourier sampling is well established (Fourier–Mellin, polar FFT, Fourier–Bessel/steerable PCA in cryo-EM, Polar Transformer Networks), and none of it is discussed. The contribution is mainly a resampling front-end, so stronger empirical evidence is needed.

W8. Method description inconsistencies. Channels: 3 in Eq. 1 / Fig. 2 vs 2 in Eq. 4 / A.1. Positional encoding: (r, φ) in the text vs (r cos φ, r sin φ) in Fig. 2C. Non-periodic φ undermines the isotropy claim. A.6 lists 7 R values for 6 datasets. EMPIAR-10076 random chance should be 20%, not 6.25%.

W9. Minor. Typos: "traind", "Tomomtwin", "per- image", "a 3D maps". L51–53 calls the Cartesian grid "non-uniform", which is imprecise. Please check the Feathers et al. DOI.

Questions for authors

Please address the weaknesses mentioned above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.