acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Advantage of Mask Resampling: A Theory of Masked Autoencoders

Abstract

Why can masked prediction learn useful representations that unmasked reconstruction misses? We study this question in a high-dimensional model of a masked autoencoder (MAE) trained on data with shared latent structure and heterogeneous noise. We prove that masked linear reconstruction can recover the latent feature at linear sample complexity in regimes where unmasked linear reconstruction, equivalent to PCA, fails. The analysis also quantifies the statistical advantage of mask resampling, an established ingredient of masked pretraining. By introducing a fixed collection of K masks per sample, we characterize its effect on feature recovery and downstream performance, identifying regimes where greater mask diversity lowers sample complexity. Guided by this prediction, we find that random cropping and flipping in standard image-training pipelines can obscure the advantage of mask resampling by renewing the prediction task even when the patch mask is fixed. Removing these transformations reveals a downstream advantage for dynamic over static masking in CNN autoencoders and vision transformers. A complementary BERT pilot finds benefits from greater mask diversity on downstream language tasks. Our results separate the benefit of the masked prediction objective from that of mask diversity, and show how a tractable theory can guide experiments that uncover advantages hidden by standard training practices.

Primary area:
learning theory
Keywords:
High-dimensional statistics, high-dimensional limit, spiked models, nonlinear autoencoders, statistical physics of learning, replica theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4Eh2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the effect of masking in representation learning, specifically distinguishing between static masks per sample, a fixed set of K randomly drawn masks per sample, and full dynamic masking that redraws masks every time a sample is drawn. The effect is mainly investigated in a setting that uses an autoencoder d -> 1 -> d trained to reconstruct hidden input values, with its learned representation evaluated on how well it recovers a shared latent signal under static, finite K and dynamic masking. They prove under specific assumptions that masked reconstruction can find useful patterns that ordinary PCA cannot, and predict using replica analysis that increased mask variety reduces the data needed to recover the hidden pattern. Synthetic experiments test the theoretical predictions, while CNNs on CIFAR-10, ViTs on ImageNet-100, and BERT on GLUE test the broader trend. They also find that cropping and flipping can partly compensate for fixed masks.

Critical strengths

S1. The paper separates masking itself from resampling masks in a simplified framework to understand the individual components.

S2. The work actually quantifies the cost of repeatedly using a limited collection of masks.

S3. As far as I can tell the quantitative analysis of finite mask reuse is novel. The rigorous result separating masked reconstruction from ordinary PCA is also a meaningful contribution.

Critical weaknesses

W1. It is unclear to me what practical guidance I can derive from the paper's conclusions. How does the insight help improve actual masked representation learning? Apart from confirming the benefit of mask diversity, which is already standard practice, the paper does not show how the theory would change or improve an existing training setup.

W2. The connection between the theory and the experiments on larger models is limited. The theory predicts changes in the sample complexity of recovering a latent feature and suggests that dynamic masking can favor less noisy coordinates. However, the ViT and BERT experiments mainly compare downstream accuracy and do not directly test either prediction. The main ViT augmentation comparison also uses only one seed.

W3. While the quantitative analysis of finite mask reuse appears novel, the contribution seems rather slim, given that the core mechanism of masking has been explored in prior work. I would have liked a clearer explanation of what new insight the theory provides beyond the expected benefit of having more prediction tasks.

W4. The paper is unnecessarily hard to read, with many sentences carrying too much information at once and making the overall story hard to follow. The text is interleaved with a lot of references to the 70 page appendix. Important assumptions and experimental details often require going to the appendix, making the main paper less self-contained.

Questions for authors

Q1. Can the authors test whether the predicted reduction in sample complexity and the noise-adaptation mechanism also occur in larger models?

Q2. Does the theory offer a useful rule for choosing K, or a masking strategy that improves on standard dynamic masking?

Q3. Does the interaction between augmentation and mask resampling in ViTs hold across multiple seeds and longer training schedules?

Q4. What happens when the data contain several shared factors, including predictable patterns that are not useful for the downstream task?

4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank the reviewer for recognizing the paper’s main contributions. We truly believe that masking is a huge enabler of modern self-supervised learning and that understanding when, why, and how masking produces useful representations in a controlled setting is our main contribution.

(W3) What the theory adds

I would have liked a clearer explanation of what new insight the theory provides beyond the expected benefit of having more prediction tasks.

Our claim is not simply that having more prediction tasks could be beneficial. That is of course expected. Our contribution is to show that mask resampling alone provides useful task variability, which is not trivial since different views reuse the same signal and noise, and to establish how large the benefits can be. In particular, we show that:

  • The gains can be substantial. This is the first main conclusion, predicted by our theory of the spiked covariance model, where dynamic masking substantially outperforms static masking across different noise distributions. These predictions are experimentally confirmed across architectures and data distributions.
  • The theory explains the diminishing returns in KK. Eq. (7) predicts that the extra samples needed because of finite mask reuse decrease as 1/K1/K: the first additional masks bring the largest gains. This qualitative trend is confirmed in CNN and ViT simulations.
  • Resampling changes how the available information is used. Dynamic masking can favor cleaner coordinates as recovery begins, without being supplied their noise levels. In the analysed linear regime, this adaptation can reduce the sample requirement to less than half that of even the best static masking ratio.

(W2 and Q1) Mask diversity and pretraining sample size

Can the authors test whether the predicted reduction in sample complexity (...) also occur in larger models?

The connection between the theory and the experiments on larger models is limited. The theory predicts changes in the sample complexity of recovering a latent feature (...)

We thank reviewers 1REi and 4Eh2 for raising this question, which led to new experimental findings that are now included in Fig. 6(b) of the revised manuscript. We ran ViT experiments with different pretraining dataset sizes: 10%, 25%, 50%, and 100% of the total images at different mask diversities (K=1,4,8K=1,4,8, and dynamic masking).

Dynamic masking performs best at every tested pretraining dataset size. In particular, dynamic masking with 25% of the pretraining images outperforms both static masking and K=4K=4 trained on the full dataset, and dynamic masking with 10% of the pretraining images outperforms static masking on the full dataset. Because real image datasets do not provide a ground-truth latent feature analogous to u⋆u^\star in our theoretical model, we use downstream performance as the main benchmark to understand the influence of masking as the number of unique pretraining samples varies.

This demonstrates that larger mask diversity can substantially improve pretraining data efficiency in a larger model, consistent with the qualitative prediction of our theory.

(W1 and Q2) Practical implications and choosing KK

It is unclear to me what practical guidance I can derive from the paper's conclusions.

Does the theory offer a useful rule for choosing K, or a masking strategy that improves on standard dynamic masking?

We don't aim to directly improve on dynamic masking or provide a detailed guide for choosing KK. Our contribution is to learning theory: we isolate mask reuse and characterize how finite mask diversity affects feature recovery at a given sample complexity. The key question is why resampling helps even though it introduces no new underlying observations.

We remark that understanding established learning procedures is itself a recognized contribution at ICLR: Han et al. (2022) [1] and Kadkhodaie et al. (2024) [2] received Outstanding Paper Awards for their studies of neural-collapse dynamics and diffusion-model generalization, even though the main contributions of their works were not focused on providing immediate indications for practitioners. Our contribution serves the same purpose: explaining why an established learning procedure helps and quantifying its benefits.

(Q1) Noise adaptation

Can the authors test whether (...) the noise-adaptation mechanism also occur in larger models?

Motivated by the noise-adaptation mechanism, we evaluate static and dynamic ViTs on the ImageNet-100 subset of ImageNet-C (Fig. 10). Both models were pretrained on clean images. Dynamic masking achieves higher accuracy on these unseen corrupted inputs, showing that its advantage extends beyond clean images.

AuthorsOpenReview ↗

Rebuttal 2/2

(Q3 and W2) Multiple seeds and longer training

The main ViT augmentation comparison also uses only one seed.

Does the interaction between augmentation and mask resampling in ViTs hold across multiple seeds (...)?

Table 5 repeats the experiments at ρ=0.75\rho=0.75 with four seeds:

Pretraining augmentationStatic top-5 accuracyDynamic top-5 accuracy
Without cropping/flipping52.42±1.19%52.42 \pm 1.19\%68.17±1.01%68.17 \pm 1.01\%
With cropping/flipping67.56±0.87%67.56 \pm 0.87\%67.62±0.68%67.62 \pm 0.68\%

Values are means ±\pm standard deviation. Across four seeds, cropping and flipping largely remove the static–dynamic gap by improving static masking.

Does the interaction (...) hold across (...) longer training schedules?

Fig. 9 extends static and dynamic pretraining to 200 epochs. Dynamic masking reaches approximately 75% accuracy, while static masking remains near 50%. Relative to the 100-epoch reference in Fig. 9, the gap between dynamic and static masking increases from 18% to 25%. Thus, extending training does not allow static masking to catch up under this protocol.

(Q4) Several shared factors

What happens when the data contain several shared factors, including predictable patterns that are not useful for the downstream task?

Our rank-one model links the downstream task to the single vector to isolate the effect of masking and mask reuse. Competition between several useful and irrelevant shared factors is an interesting, distinct question that remains open for our setting and could be at the center of future work.

However, the theory that we develop already allows us to form informed guesses on the behavior of the resulting model: the starting point is that a bottleneck autoencoder with limited representation capacity will favor the structure that best reduces masked prediction error, whether or not it is useful for the downstream task. If the additional shared structure is itself low rank, an extension of our population objective provides a concrete starting point for studying the competition between downstream-relevant and non-relevant factors.

(W4) Presentation

Important assumptions and experimental details often require going to the appendix, making the main paper less self-contained.

We appreciate the concern. The paper brings together several contributions, and explaining them within the main-text page limit required considerable compression. We prioritized the central results and their interpretation in the main text, with the complete derivations and experimental details in the appendix. This balance is challenging, and the revised manuscript gives more space to explaining the main mechanisms and making the key assumptions explicit.

We hope we have answered your questions and that our additional results resolve your concerns. If so, we would kindly ask you to reconsider the rating of our paper. We look forward to discussing any remaining questions.

References

[1] X. Y. Han, Vardan Papyan, and David L. Donoho. Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path. ICLR, 2022.

[2] Zahra Kadkhodaie, Florentin Guth, Eero P. Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. ICLR, 2024.

Thank you to the authors for the detailed responses and additional experiments, and apologies for my late response.

I appreciate the effort put into addressing my concerns. Regarding my original points:

  • W1 (Practical implications): I appreciate the new dataset-size experiments demonstrating improved sample efficiency. However, my main concern remains: the work does not provide a clear, actionable takeaway for improving masked representation learning beyond the already established practice of dynamic masking.
  • W2 (Theory–experiment connection): The additional experiments substantially strengthen the empirical evaluation. However, the proposed noise-adaptation mechanism is still not directly demonstrated in larger models.
  • W3 (Novelty and significance): I recognize the value of the quantitative theoretical analysis and appreciate the authors' clarifications. Nevertheless, I remain unconvinced by the significance of the resulting insights. In particular, the analysis consistently favors dynamic masking, which is already the standard approach, without providing a clear direction for improving upon it.
  • W4 (Presentation): I appreciate the revisions and clarifications, although I still find that the paper could be easier to follow.
  • Q4 (Multiple shared factors): I agree that extending the theory beyond a single shared signal is an interesting direction for future work rather than a necessary addition to this paper.

I also find that the concerns raised by the other reviewers, particularly regarding the validity conditions of the theoretical analysis, have been sufficiently addressed.

Ultimately, while I recognize the theoretical effort and the improvements made during rebuttal, I still find the contribution too limited in terms of new, practically useful insights to warrant acceptance. I therefore maintain my Weak Reject rating. However, I recognize that theoretical understanding is valuable in its own right, and I would not oppose acceptance if the other reviewers and AC consider the contribution sufficiently significant for ICLR.

Thank you for the comment and for checking the theoretical revisions we made in response to W3Fd’s review. We are glad the concerns are now resolved.

On significance: mask resampling was already used in practice, but why it is so useful was not understood. Different views of a sample reuse the same signal and noise, so it is not obvious that resampling the masks should help at all. Our contribution is to show that mask resampling alone provides useful task variability, and to establish how large the benefits can be. Concretely, we separate the effect of masking from the effect of resampling (S1), measure the cost of reusing a finite set of KK masks (S2), and prove a separation between masked reconstruction and unmasked reconstruction (S3). Your review identified each of these as a strength.

ICLR has accepted this kind of contribution before:

  • Luo et al. [1] studied batch normalization, when it was already standard practice, in a simplified network and used statistical mechanics to characterize it.
  • Zhang et al. [2] neither proposed nor improved Mixup. Their contribution was to explain its known benefits, showing that the Mixup loss approximately upper-bounds an adversarial loss and acts as a data-adaptive regularizer.

Our paper does the same for masked training: in a tractable model, it separates masking from resampling and measures what resampling adds.

If this addresses your remaining concern about significance, we would be grateful if you would consider whether it warrants an update to your assessment. Thank you again for engaging with the paper throughout the discussion.

References

[1] Ping Luo et al. Towards Understanding Regularization in Batch Normalization. ICLR, 2019.
[2] Linjun Zhang et al. How Does Mixup Help With Robustness and Generalization? ICLR, 2021.

Reviewer_1REiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how mask resampling affects representation learning in a spiked model with coordinate-wise heteroskedastic noise. Under specified conditions, it proves that every global minimizer of the exact mask-averaged linear objective weakly recovers the latent direction in regimes where ordinary PCA fails. A non-rigorous replica-symmetric analysis predicts learning curves and recovery thresholds as the number of fixed masks per sample varies. CNN and ViT experiments show how geometric augmentation can narrow the static–dynamic masking gap, while a BERT pilot provides additional evidence for the benefits of mask diversity.

Critical strengths
  • A rigorous separation from ordinary PCA. Theorem 2.1 establishes recovery for all global minimizers of the exact mask-averaged linear objective under explicit assumptions, including certain bounded-noise settings where ordinary PCA fails. This provides a substantive theoretical foundation for studying masking.

  • Insight into the underlying mechanism. The analysis explains why the masked reward excludes diagonal covariance contributions. In the linear model, the replica analysis also predicts that cleaner coordinates can respond more strongly to the shared feature near continuous recovery onset, under suitable masking conditions. These results deepen our understanding of how masking and mask diversity affect recovery.

  • Experiments that address relevant confounds. The fixed-learning-rate CNN comparison at ρ=0.625\rho=0.625 uses five paired seeds and a common 125k-update checkpoint (Fig. 17(b)). The ViT probe-augmentation ablation (Table 4) addresses an additional evaluation confound, while the crop/flip experiments highlight an interaction relevant to assessing mask resampling.

Critical weaknesses

The practical scope of the ViT advantage remains insufficiently established. The main matched static–dynamic comparisons use 100-epoch pretraining (App. H.1). Table 3 reports extended training recipes only for dynamic masking, demonstrating attainable performance but leaving open whether the gap persists under matched longer training. Moreover, although the unaugmented curves in Fig. 1(c) average three encoder initializations, the augmentation comparison in Fig. 5 uses one run per condition at each masking ratio, with a common seed shared across conditions. This leaves uncertainty about the robustness of the masking–augmentation interaction. These limitations reduce my confidence in how broadly the reported practical advantage extends across training settings.

Questions for authors
  • I wonder whether the performance gap between static and dynamic masking persists with longer training. A comparison under an extended training recipe would help clarify how sensitive the observed advantage is to the training setup.
  • I am curious how the benefit of mask diversity changes with the amount of pretraining data. Experiments varying both dataset size and mask count would help connect the observed performance gains to the sample-efficiency perspective developed in the theory.
2 comments

We thank the reviewer for the insightful comments. We address the concerns and questions below.

(Weakness and Q1) Longer training

I wonder whether the performance gap between static and dynamic masking persists with longer training.

In the revised manuscript, we added Figure 9 and Appendix H.5, which now compare static and dynamic masking at ρ=0.75\rho = 0.75 for a 200-epoch pretraining schedule (twice as many epochs), without stochastic cropping or flipping. Dynamic masking reaches approximately 75% accuracy, while static masking remains near 50%. Compared with the 100-epoch simulations done in most of the ViT experiments (dashed lines in Fig. 9), the gap between dynamic and static masking at the end of training went from 18 to 25 percentage points with longer training. So longer training widens the gap rather than closing it.

The augmentation comparison in Fig. 5 uses one run per condition at each masking ratio (...)

Table 5 and Fig. 6(a), formerly Fig. 5, repeat all four masking and augmentation conditions at ρ=0.75\rho = 0.75 across four runs:

Pretraining augmentationStatic top-5 accuracyDynamic top-5 accuracy
Without cropping/flipping52.42±1.19%52.42 \pm 1.19\%68.17±1.01%68.17 \pm 1.01\%
With cropping/flipping67.56±0.87%67.56 \pm 0.87\%67.62±0.68%67.62 \pm 0.68\%

Values are means with standard deviations. This supports the conclusions of the earlier single-seed experiments and strengthens their statistical validity.

(Q2) Mask diversity and pretraining data

I am curious how the benefit of mask diversity changes with the amount of pretraining data.

We thank reviewers 1Rei and 4Eh2 for asking this question, which led to new experimental findings. We have run simulations of our ViT setup at varying numbers of pretraining samples: 10%, 25%, 50%, and 100%, for static K=1K=1, K=4K=4, K=8K=8, and dynamic masking. We have added these results in Fig. 6(b) of the updated manuscript. Downstream accuracy increases with mask diversity KK at every tested pretraining sample size. In particular, we want to highlight that dynamic masking with 25% of the pretraining images outperforms both static masking and K=4K=4 pretrained on the full dataset. Every probe uses the same full labeled training set, so the comparison isolates the effect of the number of pretraining samples.

As an additional check of practical scope, we have added Figure 10, which evaluates the existing pretrained ViTs and frozen probes across four paired initializations on 15 ImageNet-C corruption types at five severity levels. Dynamic masking retains higher average accuracy on these unseen corrupted inputs, extending its representation learning advantage to unseen corrupted inputs.

Thank you for conducting controlled experiments to address my questions and for sharing the results. These experiments have resolved all of my concerns, and I will maintain my current rating of Weak Accept. Thank you.

Reviewer_W3FdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when masked prediction learns a useful direction that ordinary reconstruction misses, and whether reusing only a small collection of masks hurts learning. The analysis uses a rank-one spiked model with coordinate-dependent noise and a single-unit tied autoencoder.

There are three main results. Theorem 2.1 shows that every global minimizer of the exact mask-averaged linear objective weakly recovers the latent direction at a sufficiently large but finite sample ratio, while ordinary PCA fails in the corresponding regimes. The replica analysis predicts recovery and downstream performance when each example has a fixed collection of K reused masks; the weak-signal excess threshold contains a 1/K term. CNN, ViT, and BERT experiments then compare static and dynamic masking and examine how cropping and flipping change the comparison.

Critical strengths

Theorem 2.1 is a real contribution. It covers every global minimizer of a specific hidden-target, tied-weight objective, including cases with unbounded noise variance. The direction-dependent prediction penalty makes this estimator different from diagonal-deletion PCA, even though the two share an off-diagonal reward.

The fixed-K setup is also useful. It separates the number of original examples from the number of prediction tasks available during training. The coefficient in Eq. (7) goes beyond the generic observation that averaging K random masks reduces variance by 1/K. The spectral baselines and controlled training budgets make the empirical part more credible as well.

Critical weaknesses

1. The contribution is narrower than the broad “masked recovery and multi-mask benefit” framing

Wortsman Zurich et al., 2026 has already analyzed masked regression in terms of joint predictors, spectra, and recovery transitions. RoBERTa, data2vec 2.0, and Masked Siamese ConvNets have also shown benefits from multiple masks, while MADE used a finite mask list much earlier. These points should limit the novelty claims, not obscure the actual result.

The bounded-noise separation needs particular care. A simple deleted-diagonal PCA can also recover at a sufficiently large finite sample ratio while ordinary PCA fails under A3. The new content is therefore the guarantee for this particular objective, its extension to unbounded noise, and the fixed-K analysis. The introduction should say this directly and cite MADE and Masked Siamese ConvNets.

2. The finite-K branch conditions are not yet handled cleanly

Eq. (7) should be restricted to an admissible regular continuous weak-signal branch. For the example with gamma=(1+Exp(1))/2 and rho=1/4, D.138 does not cover the regular dynamic continuous crossing, although sequentially taking K to infinity still produces a coefficient. That coefficient cannot automatically be interpreted as the global threshold of actual dynamic ERM or SGD.

There is a more specific positivity issue as well. For fixed K>=2, the second-order term is B=-27(1-K^-2)v^2/512+O(v^3), which conflicts with the positivity requirement in D.17 when the variance support is unbounded. This shows that the assumed regular small-field branch is inadmissible in this example. It does not show that actual ERM fails, nor does it affect Theorem 2.1, but the paper must state where the finite-K analysis applies.

The Gaussian-channel step also needs a clearer account of weighted self-averaging. Euclidean delocalization alone is not enough. A sufficient condition such as max_i gamma_i w_i^2/d -> 0 would clarify the required weight profile. The finite-K analysis therefore needs a narrower validity domain, while Theorem 2.1 remains intact.

The experiments mostly support qualitative trends. Figure 18 checks the full replica equations rather than direct ERM at small beta, and the ViT crop/flip comparison uses only one shared seed. Direct small-beta validation and a repeated ViT interaction experiment would be more useful than adding another architecture.

Questions for authors
  1. Can you provide a compact comparison of SSR, HeteroPCA, and this work in terms of the estimator, noise assumptions, observable, and theorem quantifiers? In particular, what is the new capability once bounded diagonal-deletion is allowed?
  2. For Eq. (7), what exactly guarantees existence of the regular branch, and how is the sequential dynamic coefficient related to the actual dynamic objective?
  3. Which weighted self-averaging condition holds on the regular RS branch? Can you report the relevant weighted concentration quantities?
  4. Can you add direct small-beta ERM results and repeat the ViT crop/flip comparison at one pre-specified masking ratio?
3 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We thank the reviewer for the detailed feedback and for recognizing the strengths of our theoretical contributions.

A particularly informative addition, following suggestions from Reviewers 1REi and 4Eh2, is the ViT dataset-size experiment in Fig. 6(b). We compare K=1,4,8K=1,4,8 and dynamic masking using 10%, 25%, 50%, and 100% of the pretraining data under matched training-update budgets. Dynamic masking with only one quarter of the images outperforms both static masking and K=4K=4 trained on the full dataset. This highlights a setting where mask resampling is especially valuable: learning from limited pretraining data. It also connects our theoretical reduction in sample requirements to a substantial improvement in a larger model.

(W1 and Q1) Relation to SSR, HeteroPCA, diagonal deletion, and earlier masking methods

Can you provide a compact comparison of SSR, HeteroPCA, and this work in terms of the estimator, noise assumptions, observable, and theorem quantifiers?

The comparison below separates the estimator, masking protocol, assumptions, and type of result.

MethodEstimatorMasksData/noise assumptionsObservable and result
SSRJoint matrix of separately parameterized leave-one-coordinate-out ridge predictors; no bottleneckAll dd single-coordinate prediction tasks, each using every sampleRisk results: bounded covariance and independent standardized entries with bounded 4+ε4+\varepsilon moments; Gaussian spectral proof; isotropic-noise spiked modelProportional-limit prediction risks and spectrum; ridgeless BBP eigenvalue-outlier transition for a delocalized spike and α>1\alpha>1
HeteroPCAIterative covariance-diagonal imputation using rank-rr SVD; leading singular subspaceNone in the spiked-covariance settingIndependent heteroskedastic coordinate noise; standardized sub-Gaussian tails, bounded signal condition number, and incoherence conditionsFinite-sample bounds on expected subspace error
Theorem 2.1 (ours)Tied single-unit linear autoencoder; hidden-coordinate squared loss; λR=0\lambda_R=0Exact average over independent Bernoulli$(\rho)$ mask entriesGaussian latent priors and Gaussian zμiz_{\mu i}; i.i.d. coordinate variances from a fixed mean-one law on [a,∞)[a,\infty), a>0a>0, with finite second momentFor every fixed α>Asuff\alpha>A_{\rm suff}, with probability tending to one, global minimizers exist and every minimizer weakly recovers. The sufficient bound is non-sharp. Ordinary PCA has vanishing alignment for every fixed α>0\alpha>0 under unbounded variance support or bounded-noise condition (A3)
Replica analysis (ours)Tied single-unit hidden-target autoencoder with linear or nonlinear activation; ridge allowedKK independent masks per sample, held fixed and reused; also dynamic mask averagingModel (1): centered unit-variance latent priors, Gaussian zμiz_{\mu i}, and positive i.i.d. coordinate variances with unit mean and finite second moment; stated RS and self-averaging conditionsConditional RS learning curves, continuous-instability thresholds, and frozen-representation downstream accuracy; the weak-signal law additionally requires the assumptions of App. D.2.1

For Theorem 2.1, β>0\beta>0 and 0<ρ<10<\rho<1 are fixed, and weak recovery means θu⋆2≥η>0\theta_{u^\star}^2\geq\eta>0. The replica analysis takes the proportional limit at fixed KK; App. B.2.2 states the required self-averaging conditions.

In particular, what is the new capability once bounded diagonal-deletion is allowed?

HeteroPCA provides subspace-recovery guarantees through covariance correction specifically designed for heterogeneous noise. Diagonal-deletion PCA can also recover at sufficiently large finite α\alpha in the bounded setting. These results establish that suitable spectral estimators can recover; our contribution is to establish that a general-purpose autoencoder trained by masked prediction learns a useful representation without noise-specific architectural changes or preprocessing. Theorem 2.1 provides this guarantee for every global minimizer of the exact mask-averaged linear objective, including variance laws with unbounded support. Thus, robustness emerges from the prediction objective itself.

The masked objective combines an off-diagonal reward with a direction-dependent prediction penalty (Eq. (6)), so diagonal deletion does not solve the same optimization. The new main-text Fig. 3 makes this distinction quantitative: dynamic masking achieves higher mean feature alignment than both HeteroPCA and diagonal deletion in the tested bounded- and unbounded-noise settings. This complements our theoretical recovery guarantee with an empirical advantage over these stronger spectral baselines.

AuthorsOpenReview ↗

Rebuttal 2/3

SSR already analyzes masked regression and its joint predictor.

The main distinctions are parameter sharing and the masking regime. SSR's optimization separates across target-specific coefficient vectors. Our objective instead couples all hidden targets and mask patterns through the same tied encoder-decoder and learned bottleneck. Even in the linear case, the coefficients have the constrained form wiwj/dw_iw_j/d, so the coordinate optimizations do not decouple.

Additionally, each SSR task hides one coordinate, a fraction 1/d→01/d\to0. We instead fix ρ∈(0,1)\rho\in(0,1), hiding ρd\rho d coordinates on average per sample, and vary the number KK of masks assigned to each sample and reused throughout training. SSR’s formulation therefore leaves open how masking ratio and mask reuse shape a shared representation learned jointly across hidden targets. Our finite-KK theory addresses precisely this question. Appendix A, p. 18, discusses the comparison in more detail.

The introduction should say this directly and cite MADE and Masked Siamese ConvNets.

We decided to keep these references in the appendix rather than the introduction because neither isolates reuse of input masks in the sense studied in our manuscript. MADE masks network weights to impose an autoregressive task rather than masking the input samples themselves. Masked Siamese ConvNets studies masked views under a joint-embedding objective, including masking on top of random augmentation, but does not isolate the effect of using multiple masks versus a single mask for each original observation.

We thank the reviewer for pointing out these references. We retain the detailed distinctions in the appendix, while the introduction discusses earlier mask-resampling results that are closer to our setting, namely data2vec 2.0 and RoBERTa.

Earlier methods already demonstrate benefits from multiple masks; these precedents should delimit the novelty claims.

The revised introduction discusses RoBERTa and data2vec 2.0 as precedents for the empirical benefits of mask diversity. RoBERTa’s static baseline already uses ten pregenerated masks per sequence, leaving the single-mask regime untested. Our BERT-Medium pilot fills this comparison: under a common pretraining budget, the average across eight task-specific development scores is 62.7, 65.8, and 66.4 for K=1K=1, K=10K=10, and dynamic masking, respectively (Table 1). The larger gain occurs between one and ten masks. Thus, RoBERTa’s modest improvement from ten masks to dynamic masking is compatible with a substantial benefit of resampling relative to a single fixed mask.

Data2vec 2.0 instead varies the number of masked student views sharing a teacher target within an update. This does not isolate how many distinct masks an observation encounters across training, which is the quantity controlled by our KK. Our contribution is to quantify how this finite collection of prediction tasks determines recovery and sample efficiency, connecting the empirical benefit of mask diversity to a theoretical account of when and why it helps.

(W2 and Q2) Branch existence, admissibility, and the dynamic limit

For Eq. (7), what exactly guarantees existence of the regular branch, and how is the sequential dynamic coefficient related to the actual dynamic objective?

We make explicit the branch assumptions already implicit in the continuous weak-signal expansion underlying Eq. (7), now collected in App. D.2.1. The dynamic condition cited in the review is given in Eqs. (D.143)–(D.144), App. D.3.2.

There is a more specific positivity issue as well. For fixed K>=2, ...

Appendix D.2.2 explicitly treats this obstruction. At ρ=1/4\rho=1/4,

BK,0=−27512(1−K−2)vΓ2+O(vΓ3).B_{K,0}=-\frac{27}{512}(1-K^{-2})v_\Gamma^2+O(v_\Gamma^3).

For fixed K≥2K\geq2, unbounded upper variance support makes this incompatible with positivity of the input precisions throughout the support. The assumed regular small-field branch is therefore inadmissible. This does not imply failure of recovery or affect Theorem 2.1.

Proposition D.1 provides a positive existence result: for a fixed variance law bounded above and away from zero, linear activation, zero ridge, a fixed masking ratio, and sufficiently small signal, the RS equations admit a regular uninformative branch whose teacher-overlap gain crosses one. This is established both for fixed finite KK and directly for the dynamic output channel. The leading dynamic coefficient agrees with the sequential limit of the finite-KK coefficient, without interchanging the limits in the branch construction.

AuthorsOpenReview ↗

Rebuttal 3/3

(W2 and Q3) Weighted self-averaging

Which weighted self-averaging condition holds on the regular RS branch?

Euclidean delocalization alone is insufficient. Appendix B.2.2, Eqs. (B.22)–(B.23), now includes the sufficient conditions

QΓ,d=1d∑iγi⋆wi2=O(1),LΓ,d=max⁡iγi⋆wi2d⟶0.Q_{\Gamma,d}=\frac{1}{d}\sum_i\gamma_i^\star w_i^2=O(1),\qquad L_{\Gamma,d}=\frac{\max_i\gamma_i^\star w_i^2}{d}\longrightarrow0.

These imply

JΓ,d=1d2∑i(γi⋆)2wi4≤LΓ,dQΓ,d⟶0,J_{\Gamma,d}=\frac{1}{d^2}\sum_i(\gamma_i^\star)^2w_i^4\leq L_{\Gamma,d}Q_{\Gamma,d}\longrightarrow0,

which controls the variance-weighted mask-covariance fluctuations.

Additionally, App. C.1.7, immediately after Eq. (C.74), explains compatibility with the replica single-site product law. Finite expectations of w2w^2 and γw2\gamma w^2 provide the integrability needed for the maximum-over-dimension conditions for i.i.d. single-site coordinates. This establishes consistency within that ensemble; it is not a concentration theorem for data-dependent empirical minimizers.

Can you report the relevant weighted concentration quantities?

Figure 22 and Appendix H.17 report all three quantities on directly optimized weights under heterogeneous noise. QΓ,dQ_{\Gamma,d} remains close to its replica prediction, while LΓ,dL_{\Gamma,d} and JΓ,dJ_{\Gamma,d} decrease with dimension, supporting their required zero limits in the tested heterogeneous setting.

(W2 and Q4) Direct small-signal optimization and repeated ViT interaction

Can you add direct small-beta ERM results (...) ?

Appendix H.16 (Fig. 21) now trains linear masked autoencoders directly at small signal β=0.2\beta=0.2. The measured cosine similarities broadly follow the replica predictions, with finite-dimensional deviations near the transition. This complements Fig. 20 (the Figure 18 cited in the review), which compares numerical instability thresholds with their weak-signal expansion.

Can you (...) repeat the ViT crop/flip comparison at one pre-specified masking ratio?

Table 5 repeats all four pretraining conditions at ρ=0.75\rho=0.75 across four encoder initializations. We report mean validation accuracy ±\pm one sample standard deviation:

Pretraining augmentationStatic top-5 accuracyDynamic top-5 accuracy
Without cropping/flipping52.42±1.19%52.42 \pm 1.19\%68.17±1.01%68.17 \pm 1.01\%
With cropping/flipping67.56±0.87%67.56 \pm 0.87\%67.62±0.68%67.62 \pm 0.68\%

These repeated experiments support the robustness of the interaction reported in Fig. 6(a): without cropping/flipping, dynamic masking substantially outperforms static masking, whereas stochastic image augmentation largely closes this gap under the tested recipe.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.