acceptodds
Under review as a conference paper at ICLR 2027

Foveation as a Self-Supervision Signal: Blurring Beats Blanking for Learning Robust Visual Representations

Abstract

The primate retina is strikingly non-uniform, with receptor density falling off precipitously from fovea to periphery. Every saccade delivers a different retinal sampling of the visual world, yet our visual systems relate multiple glimpses and assemble a coherent sense of the scene. At some level of representation, the visual system must therefore map different retinal views of the same scene onto similar neural codes. Here we hypothesize that learning to predict fine peripheral structure from foveal feedback at the next saccade provides a natural mechanism for self-supervised learning of good visual representations. We implement this by replacing blank masks of standard masked autoencoders (MAEs) with blur masks. A vision transformer (ViT) is pre-trained to reconstruct the full-resolution image from the corrupted input, and then fine-tuned on clean images for classification. We evaluate performance on CIFAR-10 and ImageNet. Blur-MAE beats blank-MAE on the pre-training reconstruction task. Critically, blur-MAE also outperforms blank-MAE on downstream classification, showing that it learns better generalizable representations. The gap of blur-MAE superiority widens under impoverished conditions such as higher mask ratios, fewer pre-training epochs, and smaller training sets. Band-limited noise probes reveal that blank-mask models rely on high-spatial-frequency (SF) under every impoverished condition while blur-mask models do not, explaining the classification gap. A closed-form linear-ridge encoder derived from the MAE objective approximates the empirical ViT SF tuning (Pearson ) and localizes the blank-vs-blur difference in the singular-vector structure of the corrupted-input matrix.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
masked image modeling, self-supervised learning, foveated vision, spatial frequency, vision transformers, robustness, biologically inspired vision, representation learning theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XagEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a foveation-inspired masking strategy for masked image modeling that preserves a sharp foveal patch and progressively blurs the periphery. It uses SimMIM-style reconstruction to recover fine detail from coarse peripheral context rather than fill in entirely missing patches. Experiments compare this strategy with uniform blur and blank masking on CIFAR-10, ImageNet-100, and ImageNet-1k. Spatial-frequency probes and linear-model analysis examine how these corruption strategies shape the learned representations.

Critical strengths
  1. The paper provides a clear biological motivation for using foveation in masked image modeling.
  2. The author provides a broad evaluation of classification and robustness across datasets and resource-limited training settings.
Critical weaknesses
  1. The paper does not sufficiently distinguish its method from earlier foveated reconstruction work. [1] already study a sharp foveal region with progressively blurred surroundings within masked image modeling and report lower linear-probe accuracy than blank masking in their experiments. Discussing this different finding and providing controlled comparisons would help establish what the proposed retinal masking adds.
  2. It seems that retinal masking offers only small improvements over uniform blur in clean classification and sometimes performs worse. In Table 1, it improves clean accuracy by 0.5 and 0.7 percentage points on ImageNet-100 and ImageNet-1k but performs worse on both CIFAR-10 and CIFAR-10-C. Although some ImageNet robustness gains are larger, these results make a stronger case for using blur instead of blank masking than for choosing retinal masking over uniform blur.
  3. The current ImageNet-1k results are pre-trained on only 100 epochs which is substantially shorter than the 800-epoch SimMIM and 1,600-epoch MAE schedules used for their main ImageNet comparisons. Given how close the performance gap of ViT-Retina and ViT-Blur, it is very possible that, after longer pre-training, the gain will diminish. Evaluation also focuses on fine-tuned classification and corruption robustness, which makes benefits for frozen features and other downstream tasks unclear.
  4. Additionally, please consider compare to works with as close to their original setting as possible. For example, the author should consider compare to MAE and SimMIM (or other closely related baselines) instead of proposing their own baselines (e.g. Blurred, token maskin, etc). The author should also consider to use ter
  5. The ImageNet comparisons do not control for mask patch size, using 16×1616 \times 16 for retinal masking and 32×3232 \times 32 for the baselines (Table 2). This makes it difficult to determine how much of the improvement comes from retinal masking rather than the change in patch size.
  6. Masked Frequency Modeling [2] already investigates removing and reconstructing frequency components for visual pre-training and reports robustness benefits. Although its frequency-domain target differs from this paper’s pixel-space reconstruction target, the paper needs to clarify what retinal masking adds beyond frequency-based reconstruction.

[1] Robin Weiler et al. Masked Image Modeling as a Framework for Self-Supervised Learning Across Eye Movements. ICANN 2024. arXiv [2] Jiahao Xie et al. Masked Frequency Modeling for Self-Supervised Visual Pre-Training. ICLR 2023.

Questions for authors

Refer to weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XagE

W1) Relation to Weiler et al. (2024)

It is now in Related Work and the Discussion. Weiler et al. found no advantage of a foveal blur over random blank masking with the mean colour. Our study differs in that we show an advantage of biologically inspired masking over blank masking. We think two choices explain the difference from our result. (1) Gaussian blur without a cutoff is invertible. It attenuates high spatial frequencies but does not remove them, so the network can approach the target by local deconvolution rather than by predicting structure from context. We see the consequence in our own data: within the ViT-Blur (strong), the one without a frequency cutoff has the lowest reconstruction error and the lowest fine-tuned accuracy (Figures 2 and S3). (2) A fixed central fovea. Their fovea is always at the image centre; ours moves at random. Our linear theory predicts that this is not a good choice (new Figure S39): with a fixed fixation the encoder responds 3.4 times more strongly to the central disk than to the periphery and relies on higher spatial frequencies there, that is, it learns to read out the sharp centre rather than a representation that applies everywhere in the image. By the relation of Figure 3F, this higher spatial-frequency reliance is what predicts lower classification accuracy. With random fixations the two regions are treated alike. In our conditions the fixation moves from view to view and the periphery is band-limited, as in the visual system, and under those conditions retinal masking helps the network learn a more robust visual representation.

W2) Retinal masking vs uniform blur

We agree that on CIFAR-10 uniform blur performs better on clean and CIFAR-10-C accuracy. We say so in the Limitations. However, with three seeds on ImageNet-100 (table in our reply to P2HN, Q2) ViT-Retina is above ViT-Blur in every seed, by +1.3 ± 0.1 / +1.2 ± 0.4 (clean / ImageNet-100-C) at 100 epochs and +1.0 ± 0.4 / +5.5 ± 0.7 at 800, and so as in ImageNet-1k. We think CIFAR-10 differs because of resolution: on a 32-px image the blur level changes in large steps from one ring to the next, whereas at 224 px the profile changes smoothly, as in the retina. To test this we re-ran ViT-Retina FOV 5 with the blur grid at 2 px instead of 4 (token grid and model unchanged). Clean accuracy rises from 93.4 to 94.0 (single run), as much as the best uniform blur, which stays at 94.0 on the same 2-px grid.

W3a) Longer pre-training

Longer ImageNet-1k pre-training was not possible in the discussion period; we state this as a limitation. On ImageNet-100 (three seeds, reply to afhE, Q3) ViT-Blur and ViT-Retina gain 7.6 and 7.3 points from 100 to 800 epochs, the clean gap narrows only from +1.3 to +1.0 with ViT-Retina ahead in every seed, and the ImageNet-100-C gap widens from +1.2 to +5.5, so the gap does not reflect faster convergence.

W3b) Other downstream tasks

We added ADE20K segmentation with SegViT under one schedule (table in reply to afhE, Q4): ViT-Retina leads among ImageNet-1k encoders (17.4 vs 16.7 mIoU for ViT-Blank), and among ImageNet-100 encoders ViT-Blur (19.9) and ViT-Retina (17.0) are both ahead of ViT-Blank (15.6).

W4) Baselines close to their original setting

We agree. We state this as a limitation and future direction. SimMIM's own token masking is one of our baselines and labelled "SimMIM (token masking)" in the figures.

W5) Mask patch size

We re-ran the baselines with 16-px mask patches, the grid of ViT-Retina, on ImageNet-100 at mask ratio 0.65 (clean / ImageNet-100-C):

ImageNet-100, 100 ep32-px mask patches16-px mask patches
ViT-Blank78.1 / 45.677.8 / 46.7
SimMIM (token masking)77.3 / 45.177.8 / 46.1
ViT-Blur (strong, cutoff 2)78.8 / 48.479.1 / 48.9
ViT-Retina (FOV 20)—80.1 / 49.3

The patch size moves the baselines by about a point or less, in both directions, and ViT-Retina stays ahead of every baseline on both grids. At 800 epochs the 16-px ViT-Blank reaches 83.5 / 49.8, below the 32-px one (84.4 / 52.9) and far below ViT-Retina (87.3 / 62.7). The mask patch size is therefore not what gives the retinal condition its advantage. Both tables are in the new appendix.

W6) Masked Frequency Modeling

MFM is now in Related Work and the Discussion, and we ran its corruption under our recipe on CIFAR-10 (table in reply to P2HN, Q4, and the new appendix section). It performs as much as ViT-Blank or slightly better but below ViT-Retina and ViT-Blur in every condition, and our mechanistic account extends to it (new Figure S38). What retinal masking adds is where the high frequencies survive: MFM removes the same frequencies everywhere, whereas our inputs keep a sharp region inside a blurred surround, as the fovea does, which Figure S4 shows is what the encoder benefits from.

Reviewer_afhEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether biologically inspired masking strategies can improve visual representation learning compared to conventional blank masking in masked image modeling. The authors propose two approaches: ViT-Blur, which replaces blank masks with spatially blurred content, and ViT-Retina, which applies eccentricity-dependent blurring to simulate retinal foveation.

The authors evaluate these approaches across multiple image classification benchmarks, reporting improvements in reconstruction quality, downstream classification accuracy, and robustness under limited training data, reduced pre-training budgets, higher masking ratios, and test-time corruptions. The paper also investigates spatial-frequency sensitivity, suggesting that conventional blank masking encourages greater reliance on high-frequency information, which may negatively affect generalization. Furthermore, the authors derive a closed-form linear encoder to provide a theoretical explanation for the observed differences in learned representations.

Overall, the paper suggests that preserving low-frequency visual information during self-supervised pre-training can lead to more robust and transferable representations than conventional blank masking.

Critical strengths
  1. The paper introduces a biologically motivated alternative to conventional masked image modeling by replacing blank masking with frequency-selective blur inspired by the spatially varying resolution of the primate retina. The authors systematically evaluate the proposed approach under different mask ratios, pre-training budgets, and training-set sizes. The consistent improvements, particularly under constrained training conditions, provide meaningful evidence that the corruption strategy influences representation quality and robustness.

  1. The mechanistic analysis is a particularly strong contribution. Spatial-frequency-band-limited noise probes reveal that ViT-Blank becomes more dependent on high-frequency information under constrained training conditions. The negative correlation between spatial-frequency tuning centroid and classification accuracy (r = −0.73) supports this interpretation. Furthermore, the SVD-based closed-form linear ridge encoder reproduces the spatial-frequency sensitivity of the trained ViTs with a high correlation (r = 0.88). This agreement between empirical observations and theoretical analysis provides a compelling explanation for the observed performance differences.
Critical weaknesses
  1. The distinctive contribution and generalizability of frequency-selective masking remain insufficiently established. Frequency-aware inductive biases can be introduced at multiple stages of representation learning, including input preprocessing, masking or corruption strategies, latent representation regularization or distillation, and auxiliary loss functions. Without comparisons against representative alternatives, it is difficult to determine whether the improvements are specific to the proposed masking strategy or reflect a more general benefit of frequency-aware learning. Moreover, downstream evaluations are limited to image classification, leaving the transferability of the learned representations to spatially demanding tasks, such as object detection and semantic segmentation, unverified.

  1. The statistical reliability and convergence behavior of the reported ImageNet improvements remain uncertain. While the CIFAR-10 results are averaged over three seeds, the ImageNet-100 and ImageNet-1k experiments rely on single runs without measures of variability. Furthermore, ImageNet-1k pre-training is limited to 100 epochs. At a mask ratio of 0.95, ViT-Blank achieves 77.4% accuracy compared with 80.4% for ViT-Retina, but it remains unclear whether this gap would persist with longer training. The observed improvements may partly reflect faster convergence rather than superior final representation quality. These limitations weaken the evidence supporting the broader claims of the proposed method.
Questions for authors

The following questions concern issues that materially affect my assessment. Clear responses, supported by additional analyses where feasible, could address these concerns and lead me to raise my score.


[Distinguishing masking-specific benefits from general frequency-aware learning]

The paper demonstrates that frequency-selective corruption improves representation learning compared with conventional blank masking. However, similar frequency-related constraints could potentially be introduced through input preprocessing, representation-level regularization or distillation, or auxiliary training objectives.

Could the authors compare the proposed masking strategy against representative frequency-aware alternatives under matched training conditions? Have the authors considered whether these approaches are complementary and whether combining them could provide additional benefits?


[Statistical reliability of the ImageNet improvements]

The CIFAR-10 experiments report averages over three independent runs, whereas the ImageNet-100 and ImageNet-1k results appear to be based on single runs. The absence of standard deviations or confidence intervals makes it difficult to assess the consistency of the reported improvements.

Could the authors provide multi-seed evaluations for the main ImageNet comparisons, particularly between ViT-Blank, ViT-Blur, and ViT-Retina? Do the improvements remain consistent across independent runs?

This would strengthen confidence in the statistical reliability of the reported performance gains.


[Convergence behavior and the effect of pre-training duration]

The ImageNet-1k experiments use only 100 pre-training epochs. At a mask ratio of 0.95, ViT-Blank achieves 77.4% accuracy compared with 80.4% for ViT-Retina. However, this difference may partly reflect variations in convergence speed rather than differences in final representation quality.

Could the authors provide longer pre-training comparisons or learning curves showing how the performance gap evolves with additional training? Does ViT-Retina maintain its advantage after ViT-Blank approaches convergence?

This distinction is important for determining whether the proposed method improves final representation quality, training efficiency, or both.


[Generalization to spatially demanding downstream tasks]

The downstream evaluations focus exclusively on image classification. Although these results demonstrate improved recognition performance and robustness, they do not establish whether the learned representations transfer effectively to tasks requiring precise spatial localization or fine-grained visual information.

Could the authors evaluate the proposed representations on at least one representative dense prediction task, such as object detection, instance segmentation, or semantic segmentation?

This would be particularly informative because frequency-selective corruption changes the balance between low- and high-frequency information, which may affect the preservation of spatial details.


[Computational overhead and training efficiency]

The proposed corruption strategies involve additional spatial filtering and frequency-selective image processing, potentially introducing preprocessing overhead compared with conventional blank masking. However, the paper does not report wall-clock training time, throughput, or GPU memory consumption.

Could the authors provide computational comparisons between ViT-Blank, ViT-Blur, and ViT-Retina, including training time, throughput, and peak memory usage? Do the performance advantages remain consistent under matched computational budgets rather than matched epoch counts?

These results would help assess the practical efficiency and scalability of the proposed approach.

2 comments
AuthorsOpenReview ↗

Response to Reviewer afhE

Q1) Distinguishing masking-specific benefits from general frequency-aware learning

The most direct way to separate the two is a frequency-aware objective with no spatial masking at all. Masked Frequency Modeling (MFM; Xie et al., 2023) filters the whole image in the frequency domain and trains the encoder to infer the missing frequencies, so whatever it gains comes from frequency-aware learning alone. We ran its corruption under our recipe on CIFAR-10 at equal compute and data (table and details in our reply to P2HN, Q4, and in the new appendix section "Comparison with Masked Frequency Modeling"). The best MFM setting performs as much as the best ViT-Blank or slightly better, but stays below ViT-Retina and ViT-Blur in every condition, on clean and CIFAR-10-C accuracy, and the gap widens as pre-training is impoverished. Our mechanistic account extends to MFM. What MFM lacks is the fovea: Figure S4 shows that keeping no patch at full resolution, as MFM's low-pass branch does, gives the worst encoders of every family, one sharp 4 × 4 patch raises accuracy from 76–83 % to 86–90 %, and at nearly equal retained DOF the ViT-Retina beats whole-image low-pass by 8.4 points. This is what makes us think the biologically inspired masking, rather than frequency-awareness in general, is the promising direction.

Q2) Statistical reliability of the ImageNet improvements

We ran three seeds of ViT-Blank, ViT-Blur and ViT-Retina on ImageNet-100 at mask ratio 0.65, at 100 and 800 pre-training epochs (table in our reply to P2HN, Q2). The ranking ViT-Blank < ViT-Blur < ViT-Retina holds in every seed and the between-seed SEM (0.1–0.7) is well below the gaps; the smallest margin, ViT-Retina over ViT-Blur on clean accuracy at 800 epochs (1.0 point), is still more than twice the larger of the two SEMs. The means and SEMs are now in Table 1 and the single-run cells are marked. Due to the limited time of the discussion period we could not run additional seeds on ImageNet-1k; we state this as a limitation.

Q3) Convergence behaviour and the effect of pre-training duration

Due to the limited time of the discussion period we were unable to run longer pre-training on ImageNet-1k, and we state this as a limitation. On ImageNet-100, with three seeds at 100 and 800 pre-training epochs, the gap between ViT-Retina and ViT-Blank widens with longer pre-training:

ViT-Retina (FOV 20) minus ViT-Blank (mask 0.65), three-seed means100 ep800 ep
clean accuracy+2.1+3.0
ImageNet-100-C+4.0+10.1

ViT-Blank improves by 6.4 points from 100 to 800 epochs, and ViT-Retina improves by 7.3 and ViT-Blur by 7.6, so the advantage does not reflect faster convergence alone. For ViT-Retina vs ViT-Blur the clean gap narrows slightly (+1.3 → +1.0) while the robustness gap widens (ImageNet-100-C +1.2 → +5.5).

Q4) Generalisation to other tasks

We added ADE20K semantic segmentation. Following SegViT (Zhang et al., 2022), each pre-trained ViT-B/16 is fine-tuned end to end with the ATM decoder on 512² crops for 20 epochs, with the same code and schedule for every encoder (single runs):

ADE20K mIoUImageNet-1k, 100 epImageNet-100, 800 ep
ViT-Blank (mask 0.65)16.715.6
ViT-Blur (mild, cutoff 8; mask 0.65)16.1—
ViT-Blur (strong, cutoff 2; mask 0.65)—19.9
ViT-Retina (FOV 20)17.417.0

On ImageNet-1k ViT-Retina leads and ViT-Blur (mild) is slightly below ViT-Blank. On the ImageNet-100 encoders both members of the blurred family are ahead of ViT-Blank, so the advantage carries over to segmentation. Absolute mIoU is low because of the short schedule (20 epochs, about 25k iterations, against SegViT's 160k on much longer pre-trained encoders).

Q5) Computational overhead and training efficiency

All corruptions are applied on the GPU inside the forward pass and the encoder processes the same 196 tokens in every condition, so encoder FLOPs and peak memory are identical. Measured on one NVIDIA L40S (ImageNet-100, global batch 128):

pre-trainings / iterationimages / smin / epochpeak GPU memory
ViT-Blank0.1364704.58.4 GB
ViT-Blur (strong, cutoff 2)0.1394604.6 (+2 %)8.4 GB
ViT-Retina (FOV 20)0.2063106.8 (+52 %)8.4 GB

The retinal overhead is the bank of FFT low-pass filters, one per eccentricity band, instead of one; the ratio is the same on ImageNet-1k. Fine-tuning and inference carry no overhead. At matched wall-clock time, ViT-Blank pre-trained for 150 epochs (the pre-training time of ViT-Retina at 100 epochs) reaches 78.0 clean / 46.5 ImageNet-100-C (single run), against ViT-Retina's 79.9 ± 0.1 / 49.1 ± 0.3 at 100 epochs, so at equal compute ViT-Retina is still 1.9 points ahead on clean accuracy and 2.6 on ImageNet-100-C. Biologically this is not an issue, since the retina applies the fall-off at no cost.

We thank the reviewer for their constructive feedback and hope our responses have addressed their comments. We would be glad to respond to any further comments before the end of the rebuttal period!

Reviewer_P2HNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper replaces simmim's blank patch masks with uniform Gaussian/band limited blur (ViT blur) or eccentricity-dependent retinal blur around a random foveal patch (ViT Retina). On CIFAR-10, ImageNet-100 and ImageNet-1k, the blurred variants reconstruct better and fine-tune to higher clean/noisy/corruption accuracy than blank masking, especially at high mask ratio, short budgets and small data.

Critical strengths
  • I think it is a very thorough sweep. (mask ration * FOV * budget * data size * three datasets, with recon, clean, noise and -c probes, and 3 seeds on CIFAR.
  • The SF-noise probe plus Fig. S4 is a nice mechanistic story that goes beyond 'blur is easier'.
Critical weaknesses
  • The headline gains come almost entirely from impoverished regimes(mask 0.95, few epochs, small data); at the standard setting the CIFAR gain is 0.7 pp and the ImageNet runs are single-seed. The ImageNet-1k ViT-Blank baseline (78.6%) is ~4 pp below published SimMIM ViT-B at 100 epochs. So the comparison may reflect an undertuned baseline rather than the corruption type.
  • 'match information' is not really matched. Blur keeps the global low-SF layout of every patch, which is exactly the class-carrying signal, so the DOF accounting builds the conclusion into the control.
  • The linear theory is Bisulco et al. with a per-model lambda fitted to the ViT, the honest LOO number is r=0.61, and the r=-0.73 SF-centroid correlation is confounded by mask ratio driving both variables.
  • Closely related frequency-domain MIM is not discussed (ex. Masked Frequency Modeling, Xie et al. ICLR 2023), so it’s unclear what is new beyond the retinal falloff.
  • (minor) 'reproduce the the ViTs' (p.2), Fig 2 legend hides which panels are single run. Tab 1 lacks seeds/SEM
Questions for authors
  • Why is ViT-Blank on ImageNet-1k ~4 pp below the SimMIM paper at the same epochs? Can you report seeds for panels E–H?
  • At mask ratio 0.55–0.65 on ImageNet, is the blur advantage significant across seeds?
  • Does the SF-centroid/accuracy correlation survive within a fixed mask ratio?
  • How does ViT-Blur compare to MFM/low-pass-target MIM at equal compute.
2 comments
AuthorsOpenReview ↗

Response to Reviewer P2HN

W1) Where the gains come from is entirely based on impoverished conditions (Weakness 1)

We agree that on CIFAR-10 the effect is small without impoverishment conditions. On ImageNet-100, however, the difference becomes larger with longer pre-training: the Retina–Blank gap grows from +2.1 (clean) and +4.0 (ImageNet-100-C) at 100 epochs to +3.0 and +10.1 at 800 epochs.

W2) "Matched information" (Weakness 2)

We agree. We have replaced "matched information" with "matched retained degrees of freedom" throughout.

W3) The linear theory and its honest number (Weakness 3)

The revised text reports both numbers side by side: r = 0.88 with λ fitted, and r = 0.61 with λ predicted leave-one-out from tr(X_c).

W4) Masked Frequency Modeling and what is new beyond the retinal falloff (Weakness 4)

We addressed this under Q4.

W5) Presentation (Weakness 5)

Thank you for pointing these out. We updated the figure and the table.

Q1) Why ViT-Blank on ImageNet-1k is below SimMIM's numbers

SimMIM does not report a ViT-B model pre-trained for 100 epochs. Its only ViT-B result uses 800 pre-training epochs; the 100-epoch ablations use Swin-B. Our ImageNet-1k runs differ in two ways, each of which lowers the absolute number but applies identically to every condition:

  1. ℓ1 loss on the full image rather than on the masked patches only.
  2. Masking in the image (grey patches) rather than masking tokens, so that blank and blur are applied in the same domain. We also ran SimMIM's token masking and this is the SimMIM baseline at equal compute, and now named "SimMIM (token masking)".

Q2) Is the blur advantage at mask ratio 0.55–0.65 on ImageNet significant across seeds?

Yes, at 0.65. We ran three seeds of ViT-Blank, ViT-Blur and ViT-Retina on ImageNet-100 at mask ratio 0.65:

ImageNet-100, mask 0.65 — clean / ImageNet-100-C, mean ± SEM over 3 seeds100 ep800 ep
ViT-Blank77.8 ± 0.1 / 45.0 ± 0.384.2 ± 0.2 / 52.3 ± 0.3
ViT-Blur (strong, cutoff 2)78.6 ± 0.1 / 47.9 ± 0.386.2 ± 0.4 / 56.9 ± 0.7
ViT-Retina (FOV 20)79.9 ± 0.1 / 49.1 ± 0.387.2 ± 0.1 / 62.4 ± 0.1

The ranking Blank < Blur < Retina holds in every seed, at both pre-training lengths and on both measures, and the between-seed SEM (0.1–0.7) is well below the gaps. At mask ratio 0.55 (800 epochs) ViT-Blur (strong, cutoff 2) is +2.5 clean / +5.8 ImageNet-100-C above ViT-Blank (Table 3). Due to the limited time of the discussion period we could not run three seeds on ImageNet-1k. We state this as a limitation and future direction.

Q3) Does the SF-centroid correlation survive within a fixed mask ratio?

Yes, and it is stronger: within each mask ratio from 0.55 to 0.95 the correlation is r = −0.79 to −0.90 (Table 8).

Q4) ViT-Blur vs MFM / low-pass-target MIM at equal compute

MFM is now cited and discussed in the paper. MFM removes frequency content from the whole image. Our biologically inspired models keep a fovea, a patch at full resolution surrounded by lower spatial frequencies. The fovea is what makes our models more biologically grounded and what makes them perform better. We ran MFM pre-training and fine-tuning on classification with clean images on CIFAR-10 under our recipe. With probability p MFM low-passes the image and otherwise high-passes it, so p is the proportion of low-passed inputs during pre-training. We ran MFM's default p = 0.5 and its neighbours 0.3 and 0.7 (Table 7) in the full condition and at the lower end of both impoverishment conditions, and compare the best p in each condition with the best member of each of our families. Details are in the new appendix section on MFM.

CIFAR-10, single run — clean / CIFAR-10-Cfull (40k, 200 ep)reduced training epochs (50)reduced training samples (10k)
MFM, best p92.2 / 79.091.0 / 76.677.5 / 58.3
ViT-Blank, best mask ratio93.1 / 79.390.1 / 74.475.1 / 54.3
ViT-Retina, best FOV93.4 / 81.293.1 / 79.580.9 / 62.9
ViT-Blur, best member94.2 / 82.393.4 / 81.183.4 / 66.5

The best MFM setting performs as much as the best ViT-Blank in the full condition and above it when pre-training is impoverished, but stays below ViT-Retina and ViT-Blur in every condition, on clean and CIFAR-10-C accuracy, and the gap widens as pre-training is impoverished. Our mechanistic account extends to MFM: across its nine models accuracy is negatively correlated with the spatial-frequency centroid (r = −0.79, as in Figure 3F), and the linear theory reproduces the MFM tuning curves (new Figure S38, r = 0.76). The missing ingredient is the fovea. In Figure S4, with the same low-passed surround, no sharp patch, as in MFM's low-pass branch, gives the worst encoders of every family, one sharp 4 × 4 patch raises accuracy from 76–83 % to 86–90 %, and at nearly equal retained DOF ViT-Retina beats whole-image low-pass by 8.4 points.

Thank you so much for your constructive feedback. We hope our responses have addressed your comments. We would be happy to respond to any further questions before the end of the rebuttal period!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the AC: Summary of Changes

We are deeply grateful for the reviewers' detailed and constructive comments and thank the AC for their time and for overseeing our submission.

Paper Summary

We compare three pixel-space corruptions for masked image modelling under one recipe: blank masking (ViT-Blank), uniform blur with a spatial-frequency cutoff (ViT-Blur) and retinal masking, a sharp fovea at a random position with a cutoff that falls with eccentricity as in the primate retina (ViT-Retina). Each model is pre-trained to reconstruct the full-resolution image from the corrupted input and then fine-tuned on clean images for classification. On CIFAR-10, ImageNet-100 and ImageNet-1k, ViT-Blur and ViT-Retina reach higher clean accuracy and are more robust to noise, common corruptions and impoverished conditions than ViT-Blank, with ViT-Retina the most robust on ImageNet. Noise probes and a closed-form linear encoder trace the difference to the spatial frequencies the encoder comes to rely on.

Review Summary

We appreciate that the reviewers highlighted the strengths of the work: the clear biological motivation (afhE, XagE); the thorough and systematic sweep over mask ratio, field of view, pre-training budget and training-set size on three datasets (P2HN, afhE, XagE); the consistent gains under constrained training (afhE); and the mechanistic analysis, where the SF tuning centroid is negatively correlated with clean accuracy, and the closed-form linear encoder (r = 0.88) give a compelling explanation of the differences (P2HN, afhE).

Summarized below are clarifications and additional analyses in the revised paper in response to reviewers' feedback.

  • Statistical reliability. We added three seeds per condition on ImageNet-100 at 100 and 800 epochs and a further mask ratio of 0.55; the ordering Retina > Blur > Blank holds in every seed. Additional ImageNet-1k seeds were not possible in the discussion period and we state this as a limitation. (P2HN, afhE, XagE)
  • Correlation within a fixed mask ratio. We computed the SF-centroid correlation within each mask ratio; it is stronger than the pooled value. (P2HN)
  • Longer pre-training. We compared 100 and 800 epochs on ImageNet-100 with three seeds; the gaps to ViT-Blank widen and the gap to ViT-Blur does not close. Longer ImageNet-1k pre-training was not possible in the discussion period and we state this as a limitation and future direction. (P2HN, afhE, XagE)
  • Other downstream task. We added ADE20K semantic segmentation. The blur family transfers better than blank masking. (afhE, XagE)
  • Mask patch size. The reviewer noted that ViT-Retina uses a 16-px mask patch size while other models use 32 px. We re-ran the baselines at 16 px; ViT-Retina outperforms in eiter case. (XagE)
  • Compute. We added a time and memory table and an equal-wall-clock comparison; ViT-Retina stays ahead of ViT-Blank at equal compute on ImageNet-100. (afhE)
  • Masked Frequency Modeling and what our corruptions add. It is now in Related Work and the Discussion. We ran MFM's corruption at equal compute; it reaches the level of ViT-Blank or slightly above, but not that of ViT-Retina or ViT-Blur. We also discuss that keeping parts of image at its original resolution, as the fovea does, is what makes the encoder learn more robust visual representation shown in Figure S4. (P2HN, afhE, XagE)
  • Retinal masking vs uniform blur. The reviewer noted that ViT-Retina offers little benefit over ViT-Blur. That is true on CIFAR-10. However, on ImageNet-100 (three seeds) and ImageNet-1k ViT-Retina is better than ViT-Blur. (XagE)
  • Weiler et al. It is now in the Related Work and the Discussion. Weiler et al. found no advantage of a foveal blur over random blank masking with the mean colour. Our study differs in that we show an advantage of biologically inspired masking over blank masking. We explain the difference with the invertibility of Gaussian blur without a cutoff (Figure S6) and with a new linear-theory figure showing what a fixed central fovea does to the encoder. (XagE)
  • Recipe and baselines. Reviewers asked why our baselines do not reach the published SimMIM and MAE numbers. First, the loss is computed on the full image rather than on the masked region only, because the biological question is whether the whole sharp scene can be recovered from a partial view. Second, the pre-training schedule is shorter. We also clarify that our token-masking condition is exactly SimMIM's masking run under this recipe, and we have renamed it "SimMIM (token masking)" in the figures. (P2HN, XagE)

Conclusion

We thank the reviewers again and are happy to address any further questions.

Area_Chair_qadROpenReview ↗

Friendly reminder from AC

Dear Reviewers,

Thank you for your reviews. The initial ratings are 2 / 2 / 2. The authors have posted individual responses to each review, along with a general summary of changes. Thank you, Reviewer XagE, for your follow-up question on the mixed ImageNet-100 classification vs. segmentation results, and we look forward to the authors' reply.

Reviewers P2HN and afhE, we would appreciate your input as well. The response addresses several of the concerns you shared. Please let us know whether these additions change your assessment, or which concerns remain unresolved.

Thank you for your efforts.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.