acceptodds
Withdrawn from ICLR 2027

Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised learning (SSL) removes the need for annotations and makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches. In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms MAE by +3-5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error -64% relative to MAE on three new state-probing datasets, opening the door to world modeling. Welcome to our Swingers party.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, autoencoding, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_76QrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes Masked Swingers, a masked autoencoding framework that generates two independently augmented and masked views of the same image and swaps their encoded global representations (CLS tokens) before reconstructing the masked patches. This mechanism aims to encourage masked autoencoders (MAEs) to learn view-invariant image representations. The authors conduct extensive experiments involving 81 controlled pretraining runs, 10 downstream tasks spanning 67 datasets, and world-model state-probing benchmarks, demonstrating improvements in feature extraction over controlled MAE baselines.

Critical strengths
  1. In my view, swapping CLS tokens across views at the decoder input is an interesting-sounding idea, and it requires only a minor modification to the standard MAE architecture. Without introducing cross-sample negative pairs, batch normalization, or self-distillation teacher networks, the method incorporates multi-view invariance into pixel-reconstruction-based autoencoding while preserving the low memory requirements and stable training objective of MAE.
  2. The authors train 81 ViT-B/16 models across the Cartesian product of pretraining durations (100/200/800 epochs), augmentation strengths (weak/medium/strong), and color-masking strategies (none/green noise/red noise). The evaluation covers 10 frozen-feature downstream tasks spanning 67 datasets, ImageNet-1K fine-tuning, and three state-probing benchmarks. Additional systematic ablations investigate masking ratios, model scales (ViT-T/S/B/L), effective rank, and the effects of swapping alternative tokens.
Critical weaknesses
  1. Although all methods use matched GFLOPs and identical training settings, the controlled recipe differs substantially from official MAE training (smaller batch size, 2× Repeat Augmentation, and a 4× lower learning rate), without isolating its impact on relative performance. Moreover, Figure 2 selects the best pretraining configuration and feature type separately for each downstream task, while kNN hyperparameters are tuned directly on the evaluation split. This protocol may yield optimistic performance estimates and does not establish the transferability of a single fixed pretrained representation.
  2. Full-Swingers exhibits declining ImageNet kNN performance from 200 to 800 epochs, whereas Half-Swingers generally performs better under longer training and larger model scales. Although the authors attribute this behavior to view mutual information, they do not investigate the potential conflict between pixel-level reconstruction and independently augmented cross-view representations. Inconsistent geometric and photometric information may interfere with reconstruction and semantic abstraction, warranting further analysis.
  3. At 800 epochs, Full-Swingers achieves only 41.2% ImageNet-1K CLS kNN accuracy, compared with 53.3% for the baseline, while Half-Swingers improves to 56.1%. However, the omitted linear probing results show that even Half-Swingers underperforms the baseline (63.9% vs. 67.4%). This discrepancy raises concerns about whether CLS swapping consistently improves representation quality, particularly in terms of linear separability.
  4. The related work overlooks Swapping Autoencoder [1], which combines global representations from one image with spatial features from another for image generation, and Swap-VAE [2], which explicitly exchanges content representations between independently augmented views before reconstruction to encourage view-invariant learning. These mechanisms closely resemble the proposed cross-view CLS swapping strategy. A clearer distinction from these prior approaches is needed to substantiate the methodological novelty. [1] Park T, Zhu J Y, Wang O, et al. Swapping autoencoder for deep image manipulation[J]. Advances in neural information processing systems, 2020, 33: 7198-7211. [2] Liu R, Azabou M, Dabagia M, et al. Drop, swap, and generate: A self-supervised approach for generating neural activity[J]. Advances in neural information processing systems, 2021, 34: 10587-10599.
Questions for authors
  1. Given that the method relies on pixel-level reconstruction, why are no qualitative reconstruction results provided? Visualizing masked-pixel reconstruction after cross-view CLS swapping, together with ablations that remove or replace the swapped CLS token, would help clarify its actual contribution to reconstruction..
  2. Could the authors provide comparison curves across all 10 downstream tasks using a single fixed pretraining configuration (i.e., fixed augmentation strength and masking strategy) and CLS features only, rather than independently selecting the best configuration and feature type for each task?
  3. Why are the standard linear probing results omitted from the main text? At 800 epochs, Half-Swingers improves ImageNet-1K CLS kNN accuracy (56.1% vs. 53.3%) but underperforms the MAE baseline in linear probing (63.9% vs. 67.4%), while Full-Swingers performs substantially worse on both. How do the authors explain this discrepancy, and what does it imply about the quality of the learned global representations?
Reviewer_KxDsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Masked Swingers," a method that enhances MAE (Masked Autoencoder) self-supervised learning through the cross-view exchange of global representations. The approach involves independently augmenting and masking the same image, then using a shared encoder to extract features. Subsequently, the CLS representation from one view is combined with the local patch features of the current view and fed into the decoder to reconstruct the masked content of the current view. The paper proposes two exchange strategies—"Full-Swingers" and "Half-Swingers"—designed to facilitate a constrained flow of information across views, thereby encouraging the encoder to learn global summaries that are applicable across different views. Experiments covering tasks such as classification, retrieval, re-identification, and state prediction demonstrate that this method improves the performance of frozen representations compared to a controlled MAE baseline, while largely preserving the model's fine-tuning capability.

Critical strengths
  1. The method is concise and offers clear practical value for implementation. It retains the pixel-reconstruction objective of MAE while modifying only the mechanism for cross-view information exchange, without requiring additional teacher networks or complex new loss functions. This facilitates both implementation and integration, while also allowing for a more focused analysis of the design’s specific impact.
  2. Fundamental design factors are well-controlled. The MAE baseline employs an identical dual-view training pipeline, ensuring that gains derived from the additional view are not erroneously attributed to the exchange mechanism. Table 2 compares "no-exchange," "Half," and "Full" strategies under both shared and independent augmentation settings, effectively revealing the interplay between augmentation and exchange. Table 3 further compares the use of the CLS token, pooled patch summaries, and their combination, demonstrating that the observed gains are not limited to the CLS token alone.
  3. The experimental scope is extensive, with improvements in certain tasks holding practical significance. The paper evaluates the approach across 10 task categories and 67 datasets, while also analyzing the effects of model scale, masking ratio, and training duration. Results on tasks such as instance retrieval and fine-grained recognition suggest that the exchange of cross-view summaries offers transferable value. Furthermore, the authors report averaged results across various configurations and clearly distinguish between their controlled baselines and reference results obtained from external checkpoints.
Critical weaknesses
  1. The specific contribution of the cross-view reconstruction approach—relative to other methods—has not been sufficiently isolated. While works like CroCo and CropMAE already leverage information from an alternative view to reconstruct occluded content, this paper distinguishes itself by constraining the cross-view channel to a global summary and incorporating independent augmentations alongside masking. However, existing comparisons conflate multiple factors—including masking ratios, augmentation strategies, decoder architectures, and variations in pre-training data—making it impossible to definitively attribute performance gains solely to the single-token information bottleneck.
  2. The mechanistic explanation for the "view-agnostic semantic summary" relies primarily on indirect evidence. Since local patches from the current view are fed directly into the decoder, the reconstruction objective does not strictly necessitate that the model fully utilize the cross-view token, nor does it preclude the inclusion of view-specific information within that token. While ablation studies on independent augmentations underscore the importance of this component, they do not, by themselves, prove that the representations have achieved view-invariance. Similarly, effective rank analysis—which reflects the distribution of feature variations—is not a direct measure of semantic quality or view-invariance. The paper lacks direct interventions on the cross-view information channel, which would be necessary to disentangle whether the observed benefits stem from transferable semantics, view-dependent cues, or other forms of regularization.
  3. The representational level at which gains occur, as well as their scope of applicability, require further clarification. In Table 2, under the "independent augmentation" setting, "Half-Swingers" shows significant improvement in ImageNet kNN performance for the [CLS] token, yet no corresponding improvement is observed for average-pooled patch features. This suggests that some of the gains may be localized to the global readout mechanism, rather than reflecting a broad enhancement of all encoded features. Furthermore, the performance advantage under full fine-tuning is relatively modest. Consequently, the paper should more clearly distinguish between the claim of "improving globally frozen representations" and the broader claim of "improving visual representations in general."
  4. The evaluation methodology limits conclusions regarding the model's general transferability. In Appendix A.4, kNN hyperparameters are tuned based on the evaluation sets, and the main text involves task-specific selection of the optimal pre-training configuration and feature readout method. While applying a consistent protocol to both the controlled MAE baseline and "Swingers" ensures a degree of fairness, it does not eliminate selection bias, nor does it demonstrate that a fixed, pre-trained model would yield similarly broad advantages. While this issue does not apply to every evaluation step across all tasks, it complicates the interpretation of several key results.
Questions for authors
  1. Has there been a comparison—under consistent settings for data, augmentation, masking ratios, and training budgets—between a single-token summary and a representation derived from multiple or all cross-view tokens? If not, which existing findings independently support the advantage of a single-token bottleneck over the cross-view interaction approach used in CropMAE?
  2. Is there any analysis that directly measures view invariance—for instance, by removing, zeroing out, or shuffling the cross-view summary during the reconstruction process? Can such results distinguish between the model truly leveraging cross-view semantic information and other potential explanations?
  3. How can one explain the phenomenon in Table 2, where CLS token performance improves significantly, while the performance of averaged patch features does not show a corresponding improvement? Do the authors attribute this primarily to more effective global information aggregation, or to an overall enhancement in the quality of the encoder’s representations?
  4. Could the authors provide a summary of existing results that utilize a fixed pre-training configuration and a consistent readout method, with hyperparameters selected via an independent validation set? Which key conclusions hold true without requiring task-specific tuning of the configuration?
Reviewer_sTtpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a data augmentation scheme for Masked Autoencoders (MAE) called Masked Swingers. It first creates two views of an input by using different augmentations, and then exchanges the CLS token between both views. This helps the model to learn a representation that is invariant to different augmentations and masks. The authors provide two variants of Masked Swingers. Full Swingers, which is always exchanging the CLS token, and Half Swingers, which uses the CLS token from one view to decode both views. The models are evaluated on various datasets using ColorMAE as main baseline and kNN retrieval as the main evaluation task. In their experiments, the authors show that Swingers outperforms ColorMAE in kNN retrieval, and they provide ablation studies on the effect of different pretraining setups, such as model size, training length, and augmentation strength. The Swingers variant performing best depends on these parameters. For example, Full Swingers are best for small models and their performance degrades with longer pretraining, while Half Swingers are more robust and have a higher performance ceiling.

Critical strengths

S1. The proposed method is well-motivated via an information bottleneck, and I find it simple and elegant.

S2. The ablations are thorough, covering different pretraining setups for both the Swingers architectures as well as for the baseline. They cover augmentation strength, masking rate, model size, and training length. The authors use a range of datasets.

Critical weaknesses

W1. The paper's claims are based almost exclusively on kNN as evaluation task. The linear probing results for Swingers seem to be mixed and hidden in the huge Appendix table. The authors do not provide segmentation results, despite segmentation being a clear strong point of their ColorMAE baseline. Without these results, it is unclear whether Swingers succeed at kNN tasks at the cost of performance on other tasks.

W2. The performance of Swingers methods varies a lot depending on which variant is chosen and the pretraining parameters, such as training length, model size, and augmentation strength. The mechanisms by which the pretraining parameters affect the performance is not well understood. For example, the paper lacks a clear explanation for why Full Swingers degrade with longer pretraining length.

Minor Weaknesses

  • Abstract mixes absolute and relative gains, which is confusing.
  • It is often unclear whether the author report relative (percentage) or absolute (percentage points) gains. Examples of this are the abstract and the caption of Table 2.
  • The repeated suggestive references to swinger parties in the main text are unprofessional and add nothing to the understanding of the paper. They should be removed.
Questions for authors
  1. How do Masked Swingers perform on segmentation tasks (e.g., ADE20K)? I wonder whether Masked Swingers improve performance on kNN-like tasks at the cost of performance on other tasks, which the linear probing results suggest.
  2. Why does the performance of Full Swingers degrade with pretraining length? Why does strong red masking flip the training behavior pertaining to pretraining length? These questions are important because they can help determining in advance which Swingers configuration to pick for a given task.

Comments on the paper

1 comment

We thank the reviewers for taking the time to read, assess, and provide valuable feedback on our work. Because of the low scores (2/2/2) we will withdraw the paper but address core concerns for the public record.

Evaluations: kNN, LP, and FT

Our submission focuses on non-parametric evaluations, i.e., kNN classification and retrieval, which measure the immediate usefulness of pre-trained representations. Swingers provides the largest gains in these settings—especially on fine-grained tasks—so these results are the most interesting. On linear-probing or fine-tuning evals, Swingers mostly matches MAE/ColorMAE.

Regarding fine-tuning, Figure 3 shows that Swingers keeps MAE’s fine-tuneability on IN-1K. Additionally, after the initial submission, we fine-tuned Full-Swingers, Half-Swingers, and MAE/ColorMAE (checkpoints chosen based on IN-1K fine-tuning) on the OmniBenchmark [R1] collection of 21 diverse classification datasets. Full-Swingers achieves 43.1/44.6%, Half-Swingers achieves 42.9/45.2%, and MAE/ColorMAE achieves 41.2/44.9%—for 100/800 pre-training epochs respectively. Also after the submission, we fine-tuned on ADE20K with the EoMT [R2] segmentation approach. When selecting checkpoints based on IN-1K fine-tuning, Full-Swingers achieves 43.7/45.3%, Half-Swingers achieves 44.2/46.8%, and MAE/ColorMAE achieves 44.6/46.5%—for 100/800 pre-training epochs respectively.

Regarding linear probing, indeed, as noted by 76Qr the best MAE/ColorMAE checkpoint outperforms the best Half-Swingers checkpoint at 800 epochs (67.4% vs 63.9%). And Swingers gains over MAE/ColorMAE on IN-1K kNN are modest. Averaged over all checkpoints (which vary by pre-training length, color masking, and augmentation) Half-Swingers achieves 55.5% and MAE/ColorMAE achieves 54.4% IN-1K LP.

To summarize, after 800 epochs of pre-training our Half-Swingers models are on-par with MAE/ColorMAE pre-trained for the same length for both fine-tuning and linear-probing evals. Our Swingers primary advantage is in non-parametric evals (Fig 1), where we see gains of +6.3 pp averaged across 10 tasks, which span 67 datasets. We will make this point more clear in the introduction and abstract of the next version of our paper.

Checkpoint variability and selection

We thank the reviewers for noting their confusion, which we will improve in the next version of our paper. Averaged across all checkpoints, our Half-Swingers achieves 34.0%, Full-Swingers 33.3%, and MAE/ColorMAE 29% averaged over all 10 tasks in Fig 1. When selecting recipes based on the average 10-task performance for each method, Swingers also beats MAE/ColorMAE by +6.3 pp averaged over the tasks.

CropMAE-like ablation

During this rebuttal period, we ran an ablation that gives all encoder tokens to both decoder views (as in CropMAE; suggested by KxDs) it drops IN-1K kNN by -30 pp; doing it half of the time (like in Half-Swingers) still drops IN-1K kNN by -20 pp. We thank the reviewer for this suggestion and will put it in our ablations table. This experiment confirms the importance of an information bottleneck.

View invariance

To check if Swingers achieves the view invariance that motivates our design, we compute the InfoNCE loss over the ImageNet-1K validation set where positives are different augmentations of matching instances and negatives are the other 49,999 non-matching instances. So lower loss means a model's CLS is more similar to its positives than its negatives. We compute this loss for all 81 pre-trained checkpoints from Sec. 4.1. Full-Swingers achieves 7.8/5.8, Half-Swingers achieves 7.1/5.2, and MAE/ColorMAE achieves 10.4/9.0 InfoNCE loss for the average/best checkpoints. InfoNCE loss is the most common contrastive-learning objective, chosen to promote view invariance. To optimize for it, large batches must be held in memory simultaneously—Swingers learns it indirectly via autoencoding and does not require in-batch negatives or batch-wise statistics.

[R1] “Benchmarking Omni-Vision Representation through the Lens of Visual Realms”, Zhang et al., ECCV 2022

[R2] “Your ViT is Secretly an Image Segmentation Model”, Kerssies et al., CVPR 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.