- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a foveation-inspired masking strategy for masked image modeling that preserves a sharp foveal patch and progressively blurs the periphery. It uses SimMIM-style reconstruction to recover fine detail from coarse peripheral context rather than fill in entirely missing patches. Experiments compare this strategy with uniform blur and blank masking on CIFAR-10, ImageNet-100, and ImageNet-1k. Spatial-frequency probes and linear-model analysis examine how these corruption strategies shape the learned representations.
Critical strengths
- The paper provides a clear biological motivation for using foveation in masked image modeling.
- The author provides a broad evaluation of classification and robustness across datasets and resource-limited training settings.
Critical weaknesses
- The paper does not sufficiently distinguish its method from earlier foveated reconstruction work. [1] already study a sharp foveal region with progressively blurred surroundings within masked image modeling and report lower linear-probe accuracy than blank masking in their experiments. Discussing this different finding and providing controlled comparisons would help establish what the proposed retinal masking adds.
- It seems that retinal masking offers only small improvements over uniform blur in clean classification and sometimes performs worse. In Table 1, it improves clean accuracy by 0.5 and 0.7 percentage points on ImageNet-100 and ImageNet-1k but performs worse on both CIFAR-10 and CIFAR-10-C. Although some ImageNet robustness gains are larger, these results make a stronger case for using blur instead of blank masking than for choosing retinal masking over uniform blur.
- The current ImageNet-1k results are pre-trained on only 100 epochs which is substantially shorter than the 800-epoch SimMIM and 1,600-epoch MAE schedules used for their main ImageNet comparisons. Given how close the performance gap of ViT-Retina and ViT-Blur, it is very possible that, after longer pre-training, the gain will diminish. Evaluation also focuses on fine-tuned classification and corruption robustness, which makes benefits for frozen features and other downstream tasks unclear.
- Additionally, please consider compare to works with as close to their original setting as possible. For example, the author should consider compare to MAE and SimMIM (or other closely related baselines) instead of proposing their own baselines (e.g. Blurred, token maskin, etc). The author should also consider to use ter
- The ImageNet comparisons do not control for mask patch size, using for retinal masking and for the baselines (Table 2). This makes it difficult to determine how much of the improvement comes from retinal masking rather than the change in patch size.
- Masked Frequency Modeling [2] already investigates removing and reconstructing frequency components for visual pre-training and reports robustness benefits. Although its frequency-domain target differs from this paper’s pixel-space reconstruction target, the paper needs to clarify what retinal masking adds beyond frequency-based reconstruction.
[1] Robin Weiler et al. Masked Image Modeling as a Framework for Self-Supervised Learning Across Eye Movements. ICANN 2024. arXiv [2] Jiahao Xie et al. Masked Frequency Modeling for Self-Supervised Visual Pre-Training. ICLR 2023.
Questions for authors
Refer to weaknesses.
Response to Reviewer XagE
It is now in Related Work and the Discussion. Weiler et al. found no advantage of a foveal blur over random blank masking with the mean colour. Our study differs in that we show an advantage of biologically inspired masking over blank masking. We think two choices explain the difference from our result. (1) Gaussian blur without a cutoff is invertible. It attenuates high spatial frequencies but does not remove them, so the network can approach the target by local deconvolution rather than by predicting structure from context. We see the consequence in our own data: within the ViT-Blur (strong), the one without a frequency cutoff has the lowest reconstruction error and the lowest fine-tuned accuracy (Figures 2 and S3). (2) A fixed central fovea. Their fovea is always at the image centre; ours moves at random. Our linear theory predicts that this is not a good choice (new Figure S39): with a fixed fixation the encoder responds 3.4 times more strongly to the central disk than to the periphery and relies on higher spatial frequencies there, that is, it learns to read out the sharp centre rather than a representation that applies everywhere in the image. By the relation of Figure 3F, this higher spatial-frequency reliance is what predicts lower classification accuracy. With random fixations the two regions are treated alike. In our conditions the fixation moves from view to view and the periphery is band-limited, as in the visual system, and under those conditions retinal masking helps the network learn a more robust visual representation.
We agree that on CIFAR-10 uniform blur performs better on clean and CIFAR-10-C accuracy. We say so in the Limitations. However, with three seeds on ImageNet-100 (table in our reply to P2HN, Q2) ViT-Retina is above ViT-Blur in every seed, by +1.3 ± 0.1 / +1.2 ± 0.4 (clean / ImageNet-100-C) at 100 epochs and +1.0 ± 0.4 / +5.5 ± 0.7 at 800, and so as in ImageNet-1k. We think CIFAR-10 differs because of resolution: on a 32-px image the blur level changes in large steps from one ring to the next, whereas at 224 px the profile changes smoothly, as in the retina. To test this we re-ran ViT-Retina FOV 5 with the blur grid at 2 px instead of 4 (token grid and model unchanged). Clean accuracy rises from 93.4 to 94.0 (single run), as much as the best uniform blur, which stays at 94.0 on the same 2-px grid.
Longer ImageNet-1k pre-training was not possible in the discussion period; we state this as a limitation. On ImageNet-100 (three seeds, reply to afhE, Q3) ViT-Blur and ViT-Retina gain 7.6 and 7.3 points from 100 to 800 epochs, the clean gap narrows only from +1.3 to +1.0 with ViT-Retina ahead in every seed, and the ImageNet-100-C gap widens from +1.2 to +5.5, so the gap does not reflect faster convergence.
We added ADE20K segmentation with SegViT under one schedule (table in reply to afhE, Q4): ViT-Retina leads among ImageNet-1k encoders (17.4 vs 16.7 mIoU for ViT-Blank), and among ImageNet-100 encoders ViT-Blur (19.9) and ViT-Retina (17.0) are both ahead of ViT-Blank (15.6).
We agree. We state this as a limitation and future direction. SimMIM's own token masking is one of our baselines and labelled "SimMIM (token masking)" in the figures.
We re-ran the baselines with 16-px mask patches, the grid of ViT-Retina, on ImageNet-100 at mask ratio 0.65 (clean / ImageNet-100-C):
The patch size moves the baselines by about a point or less, in both directions, and ViT-Retina stays ahead of every baseline on both grids. At 800 epochs the 16-px ViT-Blank reaches 83.5 / 49.8, below the 32-px one (84.4 / 52.9) and far below ViT-Retina (87.3 / 62.7). The mask patch size is therefore not what gives the retinal condition its advantage. Both tables are in the new appendix.
MFM is now in Related Work and the Discussion, and we ran its corruption under our recipe on CIFAR-10 (table in reply to P2HN, Q4, and the new appendix section). It performs as much as ViT-Blank or slightly better but below ViT-Retina and ViT-Blur in every condition, and our mechanistic account extends to it (new Figure S38). What retinal masking adds is where the high frequencies survive: MFM removes the same frequencies everywhere, whereas our inputs keep a sharp region inside a blurred surround, as the fovea does, which Figure S4 shows is what the encoder benefits from.