- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LeVJEPA, an efficient self-supervised video pretraining framework that extends LeJEPA to video representation learning. The method uses a single video encoder trained with global-local view alignment and SIGReg regularization, removing the need for a target encoder, predictor, or masked reconstruction. It further introduces aggressive random token dropping and block-causal attention to reduce training costs. Experiments demonstrate competitive downstream performance with substantially lower pretraining compute than existing video representation learning methods.
Critical strengths
-
The framework is simple and computationally efficient. Removing the teacher encoder and predictor significantly simplifies video pretraining, while the reported 5.6–20.8× reduction in compute is impressive.
-
The experimental comparisons are reasonably comprehensive, covering different model sizes, matched training epochs, and matched FLOPs. The results provide useful insights into random token dropping, temporal tokenization, and causal attention.
Critical weaknesses
-
The main learning objective is directly inherited from LeJEPA, while global-local augmentation, random token dropping, and causal attention are established techniques. The contribution mainly lies in their combination and empirical evaluation.
-
The emergence of meaningful patch-level representations is demonstrated primarily through qualitative PCA and similarity visualizations. Quantitative evaluations on dense prediction tasks would be more convincing.
Questions for authors
-
How does SIGReg compare with other regularization methods, such as VICReg, when using the same video encoder, augmentations, and token budget?
-
Can the authors provide quantitative evidence that the learned patch tokens preserve spatial and temporal information, particularly for tracking, segmentation, or motion correspondence?
3 comments
Thank you for the response and additional experiments. The new DAVIS-2017 results provide quantitative evidence for patch-level representations, and the LeJEPA image baseline helps clarify the benefits of video pretraining.
However, two concerns remain:
- Limited novelty: The method primarily combines existing techniques, and the additional experiments validate the effectiveness of this combination rather than introduce a new learning mechanism. As such, the work feels more like an empirical study than a substantial methodological contribution.
- Lack of regularization ablation: Without a controlled comparison against VICReg, it remains unclear whether SIGReg provides distinct performance or efficiency advantages, or whether similar results could be achieved with alternative regularizers.
Overall, I will maintain my original rating of Weak Acceptance.
Response to Reviewer 5C3f
We thank the reviewer for the follow-up and for acknowledging the new results. We agree that LeVJEPA builds on existing components (e.g., the invariance loss and SIGReg regularization introduced by LeJEPA [1]). However, it was not clear a priori whether, and how, this recipe would transfer to video. First, clips contain an order of magnitude more tokens than images, which makes efficiency a central concern. Second, established design choices in video pretraining were developed around collapse-prevention mechanisms (target encoder, predictor, stop-gradient) or pixel reconstruction, so it is unclear which of these choices remain necessary once these mechanisms are removed.
Furthermore, we would like to emphasise that novelty in science does not only stem from new techniques: applying an existing method in a way that had not been considered before can be novel, and replacing a complex algorithm with a simpler one can provide new insight [2]. Both apply to our work, as LeVJEPA replaces the architectural asymmetries and design choices of current video pretraining methods with a single encoder trained with a single loss. In this spirit, novel aspects of our paper include that we (i) introduce the first extension of LeJEPA to video, showing that SIGReg alone is sufficient to prevent collapse; (ii) characterize the design space of video pretraining under this objective, showing that standard choices such as tube masking and temporal patch aggregation are unnecessary or even harmful; (iii) show that the encoder can be trained with block-causal attention at no accuracy cost; (iv) demonstrate considerable efficiency gains over existing video methods, up to pretraining on a single consumer GPU; and (v) show, at matched compute and on identical data, that video pretraining can approach image pretraining on appearance while clearly outperforming it on motion.
[1] Randall Balestriero, Yann LeCun. “LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics.” 2025.
[2] Michael J. Black, “Novelty in Science”, https://perceiving-systems.blog/en/news/novelty-in-science, 2022
We thank the reviewer for the positive assessment and the helpful suggestions. We appreciate that the reviewer found the framework simple and computationally efficient, considered the 5.6 to 20.8× reduction in pretraining compute impressive, and found that the experiments, which span model sizes and both matched epochs and matched FLOPs, provide useful insights into random token dropping, temporal tokenization and causal attention. We address the reviewer's comments below.
Novelty. We agree that the objective is inherited from LeJEPA, and the paper states this explicitly. Our contribution is to show how video pretraining changes once the architectural asymmetries (target encoder, predictor, stop-gradient) are removed. Several of the resulting findings run against standard practice: uniform random token dropping outperforms tube masking by 11.1 points on IN1K and keeps improving accuracy up to 95% dropping; per-frame tokenization outperforms temporal patch aggregation at the same token budget, also on SSv2; and block-causal attention comes at no accuracy cost. To further separate the effect of the objective itself, we now also include a new LeJEPA image baseline, trained on frames of the same videos at matched FLOPs (Table 3). It reaches 56.4 on IN1K and 18.4 on SSv2, compared to 51.2 and 30.7 for LeVJEPA. With the objective held fixed, training on video instead of frames thus adds 12.3 points on SSv2, at the cost of part of the appearance accuracy. Taken together, these results characterize video pretraining under the objective inherited from LeJEPA in a way that cannot be inferred from the individual components.
SIGReg vs. VICReg. We did not run this comparison and leave it to future work. Our choice of SIGReg is motivated by three properties: it matches the full embedding distribution rather than only second-order statistics, it has a single hyperparameter that we keep at its published default, and its cost scales linearly with batch size and embedding dimension, so it can be computed on the full global batch with a single small all-reduce (Appendix A).
Quantitative evidence for patch tokens. We have now added DAVIS-2017 video object segmentation via label propagation of frozen patch tokens (Appendix F), which tests both spatial and temporal correspondence. Against the released ViT-L checkpoints, LeVJEPA matches V-JEPA (57.0 J&F) and outperforms V-JEPA 2 (50.7), although its patch tokens receive no direct training signal. When frames are encoded together with surrounding frames, LeVJEPA, which sees only past frames, outperforms both bidirectional baselines (51.8 vs. 49.9 and 41.3 J&F).