- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work compares learned representations of joint-embedding predictive architectures (JEPAs) and masked autoencoders (MAEs). Specifically, the learned representations of physical simulations (i.e., video data) are compared using the Well dataset. Learned representations are probed for physical regime parameters to assess their quality. In contrast to previous work, this evaluation cleanly separates whether current clip prediction/ reconstruction or future clip prediction/ reconstruction leads to better representations. The results confirm that latent prediction with JEPA leads to better representations than masked autoencoding, while highlighting that JEPA yields significantly better representations when predicting future rather than current or past representations.
Critical strengths
- The work includes a controlled comparison of objectives that cleanly separates the general learning objective (JEPA vs. MAE) from using future or current/ past targets. The result that JEPA-like objectives are only superior when predicting the future is significant for the research community and challenges related frame-wise objectives like DINO (Caron et al., 2021).
- The result that probing results can even improve for JEPA representations when the target offset increases (see Figure 1) is an interesting finding that could lead to further research.
Caron et al., 2021: Emerging properties in self-supervised vision transformers. In ICCV
Critical weaknesses
- As mentioned in the limitations, the comparison only includes one model architecture (a ViT-S). Additionally, a vanilla ViT is primarily designed for image data, while the comparison focus on video data. Thus, evaluating more models like ViViT (Arnab et al., 2021) that are specifically designed for video understanding would improve the work.
- The evaluated JEPA objective is a bit dated. Using the recent LeJEPA (Balestriero & LeCun, 2025) objective would improve the work.
Arnab et al., 2021: ViViT: A Video Vision Transformer. In ICCV
Balestriero & LeCun, 2025: LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. In arXiv
Questions for authors
These questions are clarification questions and suggestions for the rebuttal:
-
Why does the plot in the center of Figure 3 not include results for the pixel-current target or are they identical to the results for the pixel-future target?
-
Do the results change when increasing the encoder capacity? I.e., using a ViT-B, ViT-L, ViT-H, etc.
-
Can both objectives be combined? I.e. adding both a predictor for the JEPA objective and a decoder for pixel reconstruction and training them jointly.
3 comments
Thank you for your response. The new caption of Figure 3 improves the clarity and the new results for using both objectives are interesting. More seeded runs and different model architectures would further improve the work. Overall, I’m leaning toward increasing my rating to 4.
Thanks for this! If you're still leaning towards a four we would really appreciate you changing it to that before the deadline?
Thanks for the review!
Pixel-Current in the centre panel of the depth figure
You’re right, they overlap, and we’ve adjusted this to be clearer!
Larger encoders and alternative architectures
We agree that evaluating larger encoders and alternative video architectures would be valuable. We did not run ViT-B/L/H, ViViT or LeJEPA experiments. These would require substantially more compute and time. We appreciate it’s a limitation and scale our claims with this in mind.
Combining latent prediction with pixel reconstruction
We additionally trained an encoder using the Latent-Future objective together with Pixel-Current's masked-reconstruction loss, with equal loss weights and no tuning (Appendix F.6, Table 14). This is a one seed experiment so far but will be brought to 3 seeds by the CRC.
The combined objective preserves Latent-Future's long-horizon performance:
It also retains some complementary strengths of pixel reconstruction. In active matter, it matches Pixel-Current on the current state; in shear flow, it gives the best encoder result at each future horizon. This suggests that adding a pixel reconstruction loss does not necessarily displace the information learned by latent-future prediction. Rather, the combined objective can preserve long-horizon information while adding complementary detail, although this finding requires confirmation across further seeds. A very cool idea worth exploring further!