- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
JEPA models learn good representations when large blocks of the input are hidden, but poor ones when the same area is hidden as scattered patches. Prior work treats this as an empirical recipe. The paper asks why.
-
If every hidden patch has visible neighbours, as with scattered masks, its content can be filled in by simple local interpolation. JEPA's loss only asks the encoder to make the target predictable, so in that case the encoder can get away with low-level features and never learns what is in the image. Because the targets come from a moving average of the encoder itself, this shortcut reinforces itself. Large blocks hide regions that are too big to interpolate, so the target can only be predicted by understanding the content. The paper quantifies this with ν, a score computed from the mask alone before training: roughly, the share of large hidden regions that contain no visible patch.
-
The explanation, that masks work by making the target unrecoverable by low-level shortcuts, is well supported by experiments. Less established is the claim that ν predicts performance better than simpler geometric measures, and the claim that "enough hidden content" and "enough nearby context" are two separable conditions on images. Experiments are at small scale (ViT-S, ImageNet-100, UCF101).
Critical strengths
Complementary controls support a mechanism beyond a mask–accuracy correlation.
- Three controls each remove one candidate cause and the probe follows the unrecoverable content. Specifically, strips perform like scattered masks despite being contiguous (40.29% versus 40.77%) while four blocks reach 64.30% (Table 1); a pixel target shrinks the mask-family spread from 25.1 to 6.8 points (Table 2); a frozen pretrained target closes the random-versus-block gap from about 19 to 1.5 points (Appendix I), with a wide interval (−3.9 to 6.9) that supports a large reduction but not exact equality.
The video anchor intervention yields a large, same-platform improvement.
- Making 100 target tokens visible is a concrete intervention with a large effect. Specifically, with intact official-style masks, exposing 100 tokens inside the target blocks raises linear-probe accuracy from 36.04% to 48.72%, three seeds per condition on RTX 5090 (Table 4), while hiding 100 context tokens outside the blocks produces no corresponding gain; this comparison is same-platform and distinct from the smaller cross-platform public-collator versus intact comparison.
Critical weaknesses
The independent roles of hidden-content amount and reachability remain insufficiently separated on images.
-
The image sweep does not show that ν predicts beyond simpler geometry, nor that the two conditions are two. Specifically, ν, the inscribed square and AvgDist give similar predictive correlations while ν and reachability correlate at ρ = −0.995 (Appendices D and O), and the two thresholds of §5.5 were chosen after the runs.
-
The one separating test is suggestive but confounded and underpowered. Specifically, the pools also differ in reachability, the comparison is three high-ν runs against two low-ν runs after a third collapsed (Welch p = 0.17), and Appendix D describes a reachability-matched pool, built from 400,000 candidates, whose results are not reported. A mechanistic measure can still be useful when a simpler proxy predicts as well, so the claims should separate explanatory from predictive value.
Questions for authors
See weaknesses.