- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates the effect of masking in representation learning, specifically distinguishing between static masks per sample, a fixed set of K randomly drawn masks per sample, and full dynamic masking that redraws masks every time a sample is drawn. The effect is mainly investigated in a setting that uses an autoencoder d -> 1 -> d trained to reconstruct hidden input values, with its learned representation evaluated on how well it recovers a shared latent signal under static, finite K and dynamic masking. They prove under specific assumptions that masked reconstruction can find useful patterns that ordinary PCA cannot, and predict using replica analysis that increased mask variety reduces the data needed to recover the hidden pattern. Synthetic experiments test the theoretical predictions, while CNNs on CIFAR-10, ViTs on ImageNet-100, and BERT on GLUE test the broader trend. They also find that cropping and flipping can partly compensate for fixed masks.
Critical strengths
S1. The paper separates masking itself from resampling masks in a simplified framework to understand the individual components.
S2. The work actually quantifies the cost of repeatedly using a limited collection of masks.
S3. As far as I can tell the quantitative analysis of finite mask reuse is novel. The rigorous result separating masked reconstruction from ordinary PCA is also a meaningful contribution.
Critical weaknesses
W1. It is unclear to me what practical guidance I can derive from the paper's conclusions. How does the insight help improve actual masked representation learning? Apart from confirming the benefit of mask diversity, which is already standard practice, the paper does not show how the theory would change or improve an existing training setup.
W2. The connection between the theory and the experiments on larger models is limited. The theory predicts changes in the sample complexity of recovering a latent feature and suggests that dynamic masking can favor less noisy coordinates. However, the ViT and BERT experiments mainly compare downstream accuracy and do not directly test either prediction. The main ViT augmentation comparison also uses only one seed.
W3. While the quantitative analysis of finite mask reuse appears novel, the contribution seems rather slim, given that the core mechanism of masking has been explored in prior work. I would have liked a clearer explanation of what new insight the theory provides beyond the expected benefit of having more prediction tasks.
W4. The paper is unnecessarily hard to read, with many sentences carrying too much information at once and making the overall story hard to follow. The text is interleaved with a lot of references to the 70 page appendix. Important assumptions and experimental details often require going to the appendix, making the main paper less self-contained.
Questions for authors
Q1. Can the authors test whether the predicted reduction in sample complexity and the noise-adaptation mechanism also occur in larger models?
Q2. Does the theory offer a useful rule for choosing K, or a masking strategy that improves on standard dynamic masking?
Q3. Does the interaction between augmentation and mask resampling in ViTs hold across multiple seeds and longer training schedules?
Q4. What happens when the data contain several shared factors, including predictable patterns that are not useful for the downstream task?
4 comments
Rebuttal 2/2
(Q3 and W2) Multiple seeds and longer training
The main ViT augmentation comparison also uses only one seed.
Does the interaction between augmentation and mask resampling in ViTs hold across multiple seeds (...)?
Table 5 repeats the experiments at with four seeds:
| Pretraining augmentation | Static top-5 accuracy | Dynamic top-5 accuracy |
|---|---|---|
| Without cropping/flipping | ||
| With cropping/flipping |
Values are means standard deviation. Across four seeds, cropping and flipping largely remove the static–dynamic gap by improving static masking.
Does the interaction (...) hold across (...) longer training schedules?
Fig. 9 extends static and dynamic pretraining to 200 epochs. Dynamic masking reaches approximately 75% accuracy, while static masking remains near 50%. Relative to the 100-epoch reference in Fig. 9, the gap between dynamic and static masking increases from 18% to 25%. Thus, extending training does not allow static masking to catch up under this protocol.
(Q4) Several shared factors
What happens when the data contain several shared factors, including predictable patterns that are not useful for the downstream task?
Our rank-one model links the downstream task to the single vector to isolate the effect of masking and mask reuse. Competition between several useful and irrelevant shared factors is an interesting, distinct question that remains open for our setting and could be at the center of future work.
However, the theory that we develop already allows us to form informed guesses on the behavior of the resulting model: the starting point is that a bottleneck autoencoder with limited representation capacity will favor the structure that best reduces masked prediction error, whether or not it is useful for the downstream task. If the additional shared structure is itself low rank, an extension of our population objective provides a concrete starting point for studying the competition between downstream-relevant and non-relevant factors.
(W4) Presentation
Important assumptions and experimental details often require going to the appendix, making the main paper less self-contained.
We appreciate the concern. The paper brings together several contributions, and explaining them within the main-text page limit required considerable compression. We prioritized the central results and their interpretation in the main text, with the complete derivations and experimental details in the appendix. This balance is challenging, and the revised manuscript gives more space to explaining the main mechanisms and making the key assumptions explicit.
We hope we have answered your questions and that our additional results resolve your concerns. If so, we would kindly ask you to reconsider the rating of our paper. We look forward to discussing any remaining questions.
References
[1] X. Y. Han, Vardan Papyan, and David L. Donoho. Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path. ICLR, 2022.
[2] Zahra Kadkhodaie, Florentin Guth, Eero P. Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. ICLR, 2024.
Thank you to the authors for the detailed responses and additional experiments, and apologies for my late response.
I appreciate the effort put into addressing my concerns. Regarding my original points:
- W1 (Practical implications): I appreciate the new dataset-size experiments demonstrating improved sample efficiency. However, my main concern remains: the work does not provide a clear, actionable takeaway for improving masked representation learning beyond the already established practice of dynamic masking.
- W2 (Theory–experiment connection): The additional experiments substantially strengthen the empirical evaluation. However, the proposed noise-adaptation mechanism is still not directly demonstrated in larger models.
- W3 (Novelty and significance): I recognize the value of the quantitative theoretical analysis and appreciate the authors' clarifications. Nevertheless, I remain unconvinced by the significance of the resulting insights. In particular, the analysis consistently favors dynamic masking, which is already the standard approach, without providing a clear direction for improving upon it.
- W4 (Presentation): I appreciate the revisions and clarifications, although I still find that the paper could be easier to follow.
- Q4 (Multiple shared factors): I agree that extending the theory beyond a single shared signal is an interesting direction for future work rather than a necessary addition to this paper.
I also find that the concerns raised by the other reviewers, particularly regarding the validity conditions of the theoretical analysis, have been sufficiently addressed.
Ultimately, while I recognize the theoretical effort and the improvements made during rebuttal, I still find the contribution too limited in terms of new, practically useful insights to warrant acceptance. I therefore maintain my Weak Reject rating. However, I recognize that theoretical understanding is valuable in its own right, and I would not oppose acceptance if the other reviewers and AC consider the contribution sufficiently significant for ICLR.
Thank you for the comment and for checking the theoretical revisions we made in response to W3Fd’s review. We are glad the concerns are now resolved.
On significance: mask resampling was already used in practice, but why it is so useful was not understood. Different views of a sample reuse the same signal and noise, so it is not obvious that resampling the masks should help at all. Our contribution is to show that mask resampling alone provides useful task variability, and to establish how large the benefits can be. Concretely, we separate the effect of masking from the effect of resampling (S1), measure the cost of reusing a finite set of masks (S2), and prove a separation between masked reconstruction and unmasked reconstruction (S3). Your review identified each of these as a strength.
ICLR has accepted this kind of contribution before:
- Luo et al. [1] studied batch normalization, when it was already standard practice, in a simplified network and used statistical mechanics to characterize it.
- Zhang et al. [2] neither proposed nor improved Mixup. Their contribution was to explain its known benefits, showing that the Mixup loss approximately upper-bounds an adversarial loss and acts as a data-adaptive regularizer.
Our paper does the same for masked training: in a tractable model, it separates masking from resampling and measures what resampling adds.
If this addresses your remaining concern about significance, we would be grateful if you would consider whether it warrants an update to your assessment. Thank you again for engaging with the paper throughout the discussion.
References
[1] Ping Luo et al. Towards Understanding Regularization in Batch Normalization. ICLR, 2019.
[2] Linjun Zhang et al. How Does Mixup Help With Robustness and Generalization? ICLR, 2021.
Rebuttal 1/2
We thank the reviewer for recognizing the paper’s main contributions. We truly believe that masking is a huge enabler of modern self-supervised learning and that understanding when, why, and how masking produces useful representations in a controlled setting is our main contribution.
(W3) What the theory adds
Our claim is not simply that having more prediction tasks could be beneficial. That is of course expected. Our contribution is to show that mask resampling alone provides useful task variability, which is not trivial since different views reuse the same signal and noise, and to establish how large the benefits can be. In particular, we show that:
(W2 and Q1) Mask diversity and pretraining sample size
We thank reviewers 1REi and 4Eh2 for raising this question, which led to new experimental findings that are now included in Fig. 6(b) of the revised manuscript. We ran ViT experiments with different pretraining dataset sizes: 10%, 25%, 50%, and 100% of the total images at different mask diversities (K=1,4,8, and dynamic masking).
Dynamic masking performs best at every tested pretraining dataset size. In particular, dynamic masking with 25% of the pretraining images outperforms both static masking and K=4 trained on the full dataset, and dynamic masking with 10% of the pretraining images outperforms static masking on the full dataset. Because real image datasets do not provide a ground-truth latent feature analogous to u⋆ in our theoretical model, we use downstream performance as the main benchmark to understand the influence of masking as the number of unique pretraining samples varies.
This demonstrates that larger mask diversity can substantially improve pretraining data efficiency in a larger model, consistent with the qualitative prediction of our theory.
(W1 and Q2) Practical implications and choosing K
We don't aim to directly improve on dynamic masking or provide a detailed guide for choosing K. Our contribution is to learning theory: we isolate mask reuse and characterize how finite mask diversity affects feature recovery at a given sample complexity. The key question is why resampling helps even though it introduces no new underlying observations.
We remark that understanding established learning procedures is itself a recognized contribution at ICLR: Han et al. (2022) [1] and Kadkhodaie et al. (2024) [2] received Outstanding Paper Awards for their studies of neural-collapse dynamics and diffusion-model generalization, even though the main contributions of their works were not focused on providing immediate indications for practitioners. Our contribution serves the same purpose: explaining why an established learning procedure helps and quantifying its benefits.
(Q1) Noise adaptation
Motivated by the noise-adaptation mechanism, we evaluate static and dynamic ViTs on the ImageNet-100 subset of ImageNet-C (Fig. 10). Both models were pretrained on clean images. Dynamic masking achieves higher accuracy on these unseen corrupted inputs, showing that its advantage extends beyond clean images.