- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work proposes Masked Swingers, a masked autoencoding framework that generates two independently augmented and masked views of the same image and swaps their encoded global representations (CLS tokens) before reconstructing the masked patches. This mechanism aims to encourage masked autoencoders (MAEs) to learn view-invariant image representations. The authors conduct extensive experiments involving 81 controlled pretraining runs, 10 downstream tasks spanning 67 datasets, and world-model state-probing benchmarks, demonstrating improvements in feature extraction over controlled MAE baselines.
Critical strengths
- In my view, swapping CLS tokens across views at the decoder input is an interesting-sounding idea, and it requires only a minor modification to the standard MAE architecture. Without introducing cross-sample negative pairs, batch normalization, or self-distillation teacher networks, the method incorporates multi-view invariance into pixel-reconstruction-based autoencoding while preserving the low memory requirements and stable training objective of MAE.
- The authors train 81 ViT-B/16 models across the Cartesian product of pretraining durations (100/200/800 epochs), augmentation strengths (weak/medium/strong), and color-masking strategies (none/green noise/red noise). The evaluation covers 10 frozen-feature downstream tasks spanning 67 datasets, ImageNet-1K fine-tuning, and three state-probing benchmarks. Additional systematic ablations investigate masking ratios, model scales (ViT-T/S/B/L), effective rank, and the effects of swapping alternative tokens.
Critical weaknesses
- Although all methods use matched GFLOPs and identical training settings, the controlled recipe differs substantially from official MAE training (smaller batch size, 2× Repeat Augmentation, and a 4× lower learning rate), without isolating its impact on relative performance. Moreover, Figure 2 selects the best pretraining configuration and feature type separately for each downstream task, while kNN hyperparameters are tuned directly on the evaluation split. This protocol may yield optimistic performance estimates and does not establish the transferability of a single fixed pretrained representation.
- Full-Swingers exhibits declining ImageNet kNN performance from 200 to 800 epochs, whereas Half-Swingers generally performs better under longer training and larger model scales. Although the authors attribute this behavior to view mutual information, they do not investigate the potential conflict between pixel-level reconstruction and independently augmented cross-view representations. Inconsistent geometric and photometric information may interfere with reconstruction and semantic abstraction, warranting further analysis.
- At 800 epochs, Full-Swingers achieves only 41.2% ImageNet-1K CLS kNN accuracy, compared with 53.3% for the baseline, while Half-Swingers improves to 56.1%. However, the omitted linear probing results show that even Half-Swingers underperforms the baseline (63.9% vs. 67.4%). This discrepancy raises concerns about whether CLS swapping consistently improves representation quality, particularly in terms of linear separability.
- The related work overlooks Swapping Autoencoder [1], which combines global representations from one image with spatial features from another for image generation, and Swap-VAE [2], which explicitly exchanges content representations between independently augmented views before reconstruction to encourage view-invariant learning. These mechanisms closely resemble the proposed cross-view CLS swapping strategy. A clearer distinction from these prior approaches is needed to substantiate the methodological novelty. [1] Park T, Zhu J Y, Wang O, et al. Swapping autoencoder for deep image manipulation[J]. Advances in neural information processing systems, 2020, 33: 7198-7211. [2] Liu R, Azabou M, Dabagia M, et al. Drop, swap, and generate: A self-supervised approach for generating neural activity[J]. Advances in neural information processing systems, 2021, 34: 10587-10599.
Questions for authors
- Given that the method relies on pixel-level reconstruction, why are no qualitative reconstruction results provided? Visualizing masked-pixel reconstruction after cross-view CLS swapping, together with ablations that remove or replace the swapped CLS token, would help clarify its actual contribution to reconstruction..
- Could the authors provide comparison curves across all 10 downstream tasks using a single fixed pretraining configuration (i.e., fixed augmentation strength and masking strategy) and CLS features only, rather than independently selecting the best configuration and feature type for each task?
- Why are the standard linear probing results omitted from the main text? At 800 epochs, Half-Swingers improves ImageNet-1K CLS kNN accuracy (56.1% vs. 53.3%) but underperforms the MAE baseline in linear probing (63.9% vs. 67.4%), while Full-Swingers performs substantially worse on both. How do the authors explain this discrepancy, and what does it imply about the quality of the learned global representations?
We thank the reviewers for taking the time to read, assess, and provide valuable feedback on our work. Because of the low scores (2/2/2) we will withdraw the paper but address core concerns for the public record.
Evaluations: kNN, LP, and FT
Our submission focuses on non-parametric evaluations, i.e., kNN classification and retrieval, which measure the immediate usefulness of pre-trained representations. Swingers provides the largest gains in these settings—especially on fine-grained tasks—so these results are the most interesting. On linear-probing or fine-tuning evals, Swingers mostly matches MAE/ColorMAE.
Regarding fine-tuning, Figure 3 shows that Swingers keeps MAE’s fine-tuneability on IN-1K. Additionally, after the initial submission, we fine-tuned Full-Swingers, Half-Swingers, and MAE/ColorMAE (checkpoints chosen based on IN-1K fine-tuning) on the OmniBenchmark [R1] collection of 21 diverse classification datasets. Full-Swingers achieves 43.1/44.6%, Half-Swingers achieves 42.9/45.2%, and MAE/ColorMAE achieves 41.2/44.9%—for 100/800 pre-training epochs respectively. Also after the submission, we fine-tuned on ADE20K with the EoMT [R2] segmentation approach. When selecting checkpoints based on IN-1K fine-tuning, Full-Swingers achieves 43.7/45.3%, Half-Swingers achieves 44.2/46.8%, and MAE/ColorMAE achieves 44.6/46.5%—for 100/800 pre-training epochs respectively.
Regarding linear probing, indeed, as noted by 76Qr the best MAE/ColorMAE checkpoint outperforms the best Half-Swingers checkpoint at 800 epochs (67.4% vs 63.9%). And Swingers gains over MAE/ColorMAE on IN-1K kNN are modest. Averaged over all checkpoints (which vary by pre-training length, color masking, and augmentation) Half-Swingers achieves 55.5% and MAE/ColorMAE achieves 54.4% IN-1K LP.
To summarize, after 800 epochs of pre-training our Half-Swingers models are on-par with MAE/ColorMAE pre-trained for the same length for both fine-tuning and linear-probing evals. Our Swingers primary advantage is in non-parametric evals (Fig 1), where we see gains of +6.3 pp averaged across 10 tasks, which span 67 datasets. We will make this point more clear in the introduction and abstract of the next version of our paper.
Checkpoint variability and selection
We thank the reviewers for noting their confusion, which we will improve in the next version of our paper. Averaged across all checkpoints, our Half-Swingers achieves 34.0%, Full-Swingers 33.3%, and MAE/ColorMAE 29% averaged over all 10 tasks in Fig 1. When selecting recipes based on the average 10-task performance for each method, Swingers also beats MAE/ColorMAE by +6.3 pp averaged over the tasks.
CropMAE-like ablation
During this rebuttal period, we ran an ablation that gives all encoder tokens to both decoder views (as in CropMAE; suggested by KxDs) it drops IN-1K kNN by -30 pp; doing it half of the time (like in Half-Swingers) still drops IN-1K kNN by -20 pp. We thank the reviewer for this suggestion and will put it in our ablations table. This experiment confirms the importance of an information bottleneck.
View invariance
To check if Swingers achieves the view invariance that motivates our design, we compute the InfoNCE loss over the ImageNet-1K validation set where positives are different augmentations of matching instances and negatives are the other 49,999 non-matching instances. So lower loss means a model's CLS is more similar to its positives than its negatives. We compute this loss for all 81 pre-trained checkpoints from Sec. 4.1. Full-Swingers achieves 7.8/5.8, Half-Swingers achieves 7.1/5.2, and MAE/ColorMAE achieves 10.4/9.0 InfoNCE loss for the average/best checkpoints. InfoNCE loss is the most common contrastive-learning objective, chosen to promote view invariance. To optimize for it, large batches must be held in memory simultaneously—Swingers learns it indirectly via autoencoding and does not require in-batch negatives or batch-wise statistics.
[R1] “Benchmarking Omni-Vision Representation through the Lens of Visual Realms”, Zhang et al., ECCV 2022
[R2] “Your ViT is Secretly an Image Segmentation Model”, Kerssies et al., CVPR 2025