acceptodds
Withdrawn from ICLR 2027

JEPA LEARNS WHAT THE MASK LEAVES UNRECOVERABLE

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint-embedding predictive architectures are unusually sensitive to how the input is masked: block masks work, scattered masks do not, and the explanations are empirical. We give a measurement account. A mask is a linear measurement, and in a compactly supported wavelet basis every atom whose support lies inside the hidden region falls in the measurement's null space and leaves no trace in the data. The JEPA loss asks only that the encoded context suffice for the target, so a target that a low-level prior can recover admits a shortcut, one the moving-average target encoder can make self-consistent. What removes the shortcut is the coarse-scale content the mask leaves unrecoverable, provided enough context stays within reach of each target. We score that content before training and test the account's distinctive predictions in 151 pre-training runs. On ImageNet-100, strip masks match blocks in area and contiguity yet are recoverable, and they land at 40.3% linear top-1, beside random masks at 40.8%, against 64.3% for blocks; within one geometry family, the placements that leave the least unrecoverable content lose 6.5 points to those that leave the most, over five seed pairs; pixel targets span 7 points where latent targets span 25; and against a frozen target the gap between random and block masks, 19 points on the same kind of GPU, closes to 1.5, so the geometry acts through the target the encoder produces for itself. On UCF101 the masking ratio decides which condition, content or reach, binds; removing whole frames, unrecoverable in space but recoverable from neighbouring frames, is worst at both ratios; and on V-JEPA's own masks, batching them intact instead of truncated changes little (36.0% against 35.1%), whereas making 100 target tokens inside the blocks visible lifts them to 48.7% and hiding 100 context tokens outside the blocks does not (33.7%).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, I-JEPA, V-JEPA, masking, masked image modeling, self-distillation, representation learning, video representation learning, compressed sensing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eFHaOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • JEPA models learn good representations when large blocks of the input are hidden, but poor ones when the same area is hidden as scattered patches. Prior work treats this as an empirical recipe. The paper asks why.

  • If every hidden patch has visible neighbours, as with scattered masks, its content can be filled in by simple local interpolation. JEPA's loss only asks the encoder to make the target predictable, so in that case the encoder can get away with low-level features and never learns what is in the image. Because the targets come from a moving average of the encoder itself, this shortcut reinforces itself. Large blocks hide regions that are too big to interpolate, so the target can only be predicted by understanding the content. The paper quantifies this with ν, a score computed from the mask alone before training: roughly, the share of large hidden regions that contain no visible patch.

  • The explanation, that masks work by making the target unrecoverable by low-level shortcuts, is well supported by experiments. Less established is the claim that ν predicts performance better than simpler geometric measures, and the claim that "enough hidden content" and "enough nearby context" are two separable conditions on images. Experiments are at small scale (ViT-S, ImageNet-100, UCF101).

Critical strengths

Complementary controls support a mechanism beyond a mask–accuracy correlation.

  • Three controls each remove one candidate cause and the probe follows the unrecoverable content. Specifically, strips perform like scattered masks despite being contiguous (40.29% versus 40.77%) while four blocks reach 64.30% (Table 1); a pixel target shrinks the mask-family spread from 25.1 to 6.8 points (Table 2); a frozen pretrained target closes the random-versus-block gap from about 19 to 1.5 points (Appendix I), with a wide interval (−3.9 to 6.9) that supports a large reduction but not exact equality.

The video anchor intervention yields a large, same-platform improvement.

  • Making 100 target tokens visible is a concrete intervention with a large effect. Specifically, with intact official-style masks, exposing 100 tokens inside the target blocks raises linear-probe accuracy from 36.04% to 48.72%, three seeds per condition on RTX 5090 (Table 4), while hiding 100 context tokens outside the blocks produces no corresponding gain; this comparison is same-platform and distinct from the smaller cross-platform public-collator versus intact comparison.
Critical weaknesses

The independent roles of hidden-content amount and reachability remain insufficiently separated on images.

  • The image sweep does not show that ν predicts beyond simpler geometry, nor that the two conditions are two. Specifically, ν, the inscribed square and AvgDist give similar predictive correlations while ν and reachability correlate at ρ = −0.995 (Appendices D and O), and the two thresholds of §5.5 were chosen after the runs.

  • The one separating test is suggestive but confounded and underpowered. Specifically, the pools also differ in reachability, the comparison is three high-ν runs against two low-ν runs after a third collapsed (Welch p = 0.17), and Appendix D describes a reachability-matched pool, built from 400,000 candidates, whose results are not reported. A mechanistic measure can still be useful when a simpler proxy predicts as well, so the claims should separate explanatory from predictive value.

Questions for authors

See weaknesses.

Reviewer_93hTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper develops a measure that relates masking-related choices in joint embedding predictive architectures (JEPAs) to the quality of learned representations. The paper claims that in addition to have sufficient context, JEPAs avoid learning a low-level shortcut only when the mask removes information that cannot be recovered using a low-level prior. Experiments with image and video data suggest that there may be thresholds above which JEPAs can learn good representations.

Critical strengths
  • The paper studies masking choices that affect the quality of representations obtained with JEPAs. Careful mask design has shown to be key for empirical success of JEPAs (I-JEPA vs V-JEPA) so a study that aims to quantify what is a good mask is useful

  • Empirical results suggests that masking is critical to EMA methods like I-JEPA and V-JEPA while it's not as critical to frozen target encoder-based approaches. This empirical observation is likely to be of interest to researchers and practitioners in the JEPA community.

  • The choices for ablation studies in the paper appear to be reasonable in terms of masking choices to keep results comparable

Critical weaknesses
  • The paper is very difficult to follow. The paper cites quantities and terms without clearly defining them

    • What is a wavelet atom? (I am aware of it as I have some familiarity with wavelet literature but I had to look it up)
    • The abstract is fairly dense and packed with numbers (accuracies) but does not offer a clear picture of what role the developed measure plays with masking choices.
    • The development of ν\nu in Section 3.2 Equation 3 is unclear and could use more intuitive arguments. Furthermore, what wsw_s stands for is not clear and the reader is left to infer these from text
  • The paper compares the new measure with existing ones like AvgDist but does not include this comparison in main text. The correlation coefficient, in Appendix D, is comparable to AvDist which would make the reader wonder about the need for this measure.

  • Furthermore, there is a large body of work that uses embedding rank-related information [1, 2, 3] to determine hyperparameters choice and its effect on downstream performance. These comparisons are missing.

[1] α\alpha-Req https://openreview.net/forum?id=ii9X4vtZGTZ [2] RankMe https://proceedings.mlr.press/v202/garrido23a/garrido23a.pdf [3] LiDAR https://openreview.net/forum?id=f3g5XpL9Kb

  • The video-variant of proposed metric does not appear to correlate well with downstream performance.
Questions for authors

My biggest concern is that the proposed measure does not improve upon existing measure like AvgDist. Also, there are no comparisons with existing metrics. I would like to understand how the proposed measure adds new insight that could be helpful to JEPAs literature

The writing and presentation needs to be improved significantly so that the paper is useful to a general reader in ICLR community. I have noted some issues I had as I tried to carefully read the paper in Critical Weaknesses section.

Reviewer_ro6wOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates why different masking strategies affect the representations learned by JEPA. The authors introduce a wavelet-based measurement account and a metric, ν, intended to estimate coarse-scale content that a mask leaves unrecoverable to simple priors. They argue that effective masks prevent low-level reconstruction shortcuts while preserving enough visible context for meaningful prediction. The paper evaluates this explanation through image, video and audio experiments, with audio reported as a setting where the account does not hold.

Critical strengths

S1. The frozen-teacher experiment reduces the random-versus-block gap from 19.1 to 1.5 points (Appendix I). The visible-anchor experiment also gives a large improvement (Table 4). These are useful observations about the role of prediction targets and visible context.

S2. Table 1 shows that strips behave like scattered masks rather than blocks, providing evidence that contiguity alone does not explain performance. The within-family placement experiment fixes token counts and training settings and finds a positive high-versus-low-ν effect across all five paired seeds (Section 5.4).

S3. The authors disclose retrospective choices, collapsed runs, hardware differences and the failure of the account on audio (Appendices L–O). This makes it easier to assess what the evidence supports, although disclosure does not resolve these limitations.

Critical weaknesses

W1. The central conclusions rely on retrospective analysis. The thresholds in Section 5.5, the ν = 0.1 split and the successful video statistic were chosen after examining the results (Appendix O). The video analysis also tries several statistics without correcting for multiple comparisons. The two tests specified before training are useful, but do not independently validate the fitted thresholds or the selected video statistic. The evidence therefore supports an exploratory explanation more strongly than a general predictive rule.

W2. The paper presents limited novelty and the added value of ν is unclear. I-JEPA and SimMIM already discuss the need to avoid trivial prediction while retaining informative context [1, 2]. ν ranks configurations much like AvgDist and the largest inscribed square, while the pool experiment does not separate ν from reachability (Appendices D and O). Section 5.4 compares high- and low-ν selection, but not selection using those alternative measures. The paper therefore does not clearly establish added explanatory or practical value beyond existing accounts.

W3. The proposed mechanism is not established. Section 3 shows that fully hidden wavelet components are unobserved, but does not explain why the encoder should learn semantic representations. A pretrained frozen teacher changes target quality as well as target updating, while the pixel-target experiment also changes optimization settings (Appendices I and N). The anchor experiment changes token counts as well as positions (Appendix F). These controls show that targets and context matter, but do not isolate the proposed self-reinforcing shortcut or justify the strongest causal claims.

W4. The paper has narrow scope and fragile comparisons. The main visual evaluation uses ViT-S on ImageNet-100 and UCF101, with two seeds for most configurations. The distinguishing pool test has only five successful runs (p = 0.17), and some video comparisons cross hardware/software setups (Appendix O). ν does not consistently predict video performance, and the broader account fails on audio (Table 3; Appendix L). The reversal of the original V-JEPA mask ordering further leaves the applicability to larger-scale training unresolved.

W5. The paper has substantial clarity issues. The paper is difficult to follow throughout, making it difficult to identify the central research question and contributions. The contributions primarily list experimental findings without clearly explaining what new knowledge they establish. Important concepts, mathematical notation, and experimental controls are introduced without sufficient explanation, and the writing frequently obscures the conclusions being made. Strong statements in the main text are substantially qualified in Appendix O, making it difficult to distinguish demonstrated conclusions from hypotheses.

References [1] Assran et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. CVPR, 2023. [2] Xie et al. SimMIM: A Simple Framework for Masked Image Modeling. CVPR, 2022.

Questions for authors

Q1. Can the thresholds and video statistic be fixed before evaluating new mask configurations, with predictive success assessed on those held-out configurations rather than the fitting runs?

Q2. Can you report the reachability-matched pool experiment described in Appendix D and compare ν-based mask selection with AvgDist and largest-inscribed-square selection under the same candidate and training budgets? What does v capture that existing metrics do not?

Q3. What conclusion about learned representations follows from the null-space argument beyond the absence of direct observations? How does the frozen-teacher experiment distinguish the proposed shortcut feedback from distillation using an already semantic teacher?

Q4. Can the frozen-target comparison start from the same pretrained teacher in both the frozen and updating conditions? Can the anchor experiment exchange internal and external visible tokens while keeping context and target counts fixed?

Q5. Do the main conclusions hold with more seeds and a larger model or dataset, particularly in a setting that reproduces V-JEPA’s original mask ordering? Please include uncertainty estimates and collapse rates for the key comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.