acceptodds
Under review as a conference paper at ICLR 2027

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

13 views

Abstract

Video carries the temporal structure of the physical world, yet learning representations from it remains computationally expensive. Current self-supervised methods either prevent representation collapse through architectural asymmetries (an exponential-moving-average target encoder, a stop-gradient, and a predictor), or avoid it by reconstructing masked content in pixel space with a dedicated decoder. We introduce LeVJEPA, the first extension of LeJEPA to video. A single encoder is trained with an invariance loss between global and local views of a clip, while collapse is prevented by SIGReg with provable guarantees. Thus, neither architectural asymmetries nor pixel reconstruction is needed. This formulation has two properties that we study in this paper. First, the pretraining cost depends only on the number of tokens the encoder observes. Dropping tokens uniformly at random reduces this number and, at the same time, improves downstream accuracy. At matched epochs on identical data, LeVJEPA matches or outperforms V-JEPA 2 across ViT-S/B/L with to less pretraining compute. At matched FLOPs, it exceeds the strongest video baseline by points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no target encoder or predictor is involved, the encoder can be trained with block-causal attention without loss in accuracy, so that the representation of each frame depends only on past frames. Finally, compared to DINOv2 trained on frames of the same videos with matched compute, LeVJEPA remains within points on ImageNet-1K while nearly doubling the accuracy of DINOv2 on Something-Something-v2. These results suggest that, once its computational overhead is removed, video is a viable substrate for general-purpose visual pretraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, video pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5C3fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LeVJEPA, an efficient self-supervised video pretraining framework that extends LeJEPA to video representation learning. The method uses a single video encoder trained with global-local view alignment and SIGReg regularization, removing the need for a target encoder, predictor, or masked reconstruction. It further introduces aggressive random token dropping and block-causal attention to reduce training costs. Experiments demonstrate competitive downstream performance with substantially lower pretraining compute than existing video representation learning methods.

Critical strengths
  1. The framework is simple and computationally efficient. Removing the teacher encoder and predictor significantly simplifies video pretraining, while the reported 5.6–20.8× reduction in compute is impressive.

  2. The experimental comparisons are reasonably comprehensive, covering different model sizes, matched training epochs, and matched FLOPs. The results provide useful insights into random token dropping, temporal tokenization, and causal attention.

Critical weaknesses
  1. The main learning objective is directly inherited from LeJEPA, while global-local augmentation, random token dropping, and causal attention are established techniques. The contribution mainly lies in their combination and empirical evaluation.

  2. The emergence of meaningful patch-level representations is demonstrated primarily through qualitative PCA and similarity visualizations. Quantitative evaluations on dense prediction tasks would be more convincing.

Questions for authors
  1. How does SIGReg compare with other regularization methods, such as VICReg, when using the same video encoder, augmentations, and token budget?

  2. Can the authors provide quantitative evidence that the learned patch tokens preserve spatial and temporal information, particularly for tracking, segmentation, or motion correspondence?

3 comments

We thank the reviewer for the positive assessment and the helpful suggestions. We appreciate that the reviewer found the framework simple and computationally efficient, considered the 5.6 to 20.8× reduction in pretraining compute impressive, and found that the experiments, which span model sizes and both matched epochs and matched FLOPs, provide useful insights into random token dropping, temporal tokenization and causal attention. We address the reviewer's comments below.

Novelty. We agree that the objective is inherited from LeJEPA, and the paper states this explicitly. Our contribution is to show how video pretraining changes once the architectural asymmetries (target encoder, predictor, stop-gradient) are removed. Several of the resulting findings run against standard practice: uniform random token dropping outperforms tube masking by 11.1 points on IN1K and keeps improving accuracy up to 95% dropping; per-frame tokenization outperforms temporal patch aggregation at the same token budget, also on SSv2; and block-causal attention comes at no accuracy cost. To further separate the effect of the objective itself, we now also include a new LeJEPA image baseline, trained on frames of the same videos at matched FLOPs (Table 3). It reaches 56.4 on IN1K and 18.4 on SSv2, compared to 51.2 and 30.7 for LeVJEPA. With the objective held fixed, training on video instead of frames thus adds 12.3 points on SSv2, at the cost of part of the appearance accuracy. Taken together, these results characterize video pretraining under the objective inherited from LeJEPA in a way that cannot be inferred from the individual components.

SIGReg vs. VICReg. We did not run this comparison and leave it to future work. Our choice of SIGReg is motivated by three properties: it matches the full embedding distribution rather than only second-order statistics, it has a single hyperparameter that we keep at its published default, and its cost scales linearly with batch size and embedding dimension, so it can be computed on the full global batch with a single small all-reduce (Appendix A).

Quantitative evidence for patch tokens. We have now added DAVIS-2017 video object segmentation via label propagation of frozen patch tokens (Appendix F), which tests both spatial and temporal correspondence. Against the released ViT-L checkpoints, LeVJEPA matches V-JEPA (57.0 J&F) and outperforms V-JEPA 2 (50.7), although its patch tokens receive no direct training signal. When frames are encoded together with surrounding frames, LeVJEPA, which sees only past frames, outperforms both bidirectional baselines (51.8 vs. 49.9 and 41.3 J&F).

Thank you for the response and additional experiments. The new DAVIS-2017 results provide quantitative evidence for patch-level representations, and the LeJEPA image baseline helps clarify the benefits of video pretraining.

However, two concerns remain:

  1. Limited novelty: The method primarily combines existing techniques, and the additional experiments validate the effectiveness of this combination rather than introduce a new learning mechanism. As such, the work feels more like an empirical study than a substantial methodological contribution.
  2. Lack of regularization ablation: Without a controlled comparison against VICReg, it remains unclear whether SIGReg provides distinct performance or efficiency advantages, or whether similar results could be achieved with alternative regularizers.

Overall, I will maintain my original rating of Weak Acceptance.

AuthorsOpenReview ↗

Response to Reviewer 5C3f

We thank the reviewer for the follow-up and for acknowledging the new results. We agree that LeVJEPA builds on existing components (e.g., the invariance loss and SIGReg regularization introduced by LeJEPA [1]). However, it was not clear a priori whether, and how, this recipe would transfer to video. First, clips contain an order of magnitude more tokens than images, which makes efficiency a central concern. Second, established design choices in video pretraining were developed around collapse-prevention mechanisms (target encoder, predictor, stop-gradient) or pixel reconstruction, so it is unclear which of these choices remain necessary once these mechanisms are removed.

Furthermore, we would like to emphasise that novelty in science does not only stem from new techniques: applying an existing method in a way that had not been considered before can be novel, and replacing a complex algorithm with a simpler one can provide new insight [2]. Both apply to our work, as LeVJEPA replaces the architectural asymmetries and design choices of current video pretraining methods with a single encoder trained with a single loss. In this spirit, novel aspects of our paper include that we (i) introduce the first extension of LeJEPA to video, showing that SIGReg alone is sufficient to prevent collapse; (ii) characterize the design space of video pretraining under this objective, showing that standard choices such as tube masking and temporal patch aggregation are unnecessary or even harmful; (iii) show that the encoder can be trained with block-causal attention at no accuracy cost; (iv) demonstrate considerable efficiency gains over existing video methods, up to pretraining on a single consumer GPU; and (v) show, at matched compute and on identical data, that video pretraining can approach image pretraining on appearance while clearly outperforming it on motion.

[1] Randall Balestriero, Yann LeCun. “LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics.” 2025.

[2] Michael J. Black, “Novelty in Science”, https://perceiving-systems.blog/en/news/novelty-in-science, 2022

Reviewer_7Zn2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts the LeJEPA single-encoder objective and SIGReg to video pretraining. The method combines global and local views with 95% uniform token dropping, per-frame tokenization, and block-causal attention. It does not use a target encoder, predictor, or stop-gradient during training; the paper states that EMA is used only for evaluation checkpoints. Under the reported FLOP-matched setting, LeVJEPA reaches 61.0 on IN1K and 44.6 on K400, which are 7.6 and 3.9 points above the strongest listed baselines, respectively. Its SSv2 result is 40.4, which is 3.2 points below VideoMAEv2. The main contribution is an efficient video training recipe rather than a new learning objective or anti-collapse theory.

Critical strengths
  1. The paper reports useful same-data comparisons under both epoch-matched and FLOP-matched protocols. The strong IN1K and K400 results make the simplified shared-encoder recipe practically interesting, and the paper also reports its weaker SSv2 result.

  2. The ablations support several important design choices. Higher token dropping improves the reported IN1K result, and per-frame tokenization outperforms temporal patch aggregation under the compared token budget. The ViT-Tiny experiment on one RTX 5080 also shows that the recipe can be useful with limited hardware.

Critical weaknesses
  1. The video-specific contribution is not isolated from the inherited LeJEPA objective and the rest of the training recipe. SIGReg and the single-encoder objective come from LeJEPA, while the full method also changes token dropping, view construction, tokenization, and attention. The DINOv2 comparison changes several of these factors at once. A compute-matched LeJEPA image model trained on frames from the same videos is needed to show how much of the gain comes from temporal video modeling.

  2. The main efficiency claim is not fully auditable from the paper. The exact per-method configuration, step count, learning-rate schedule, view and token count, temporal sampling, and included FLOPs are not given in one place. This matters because the FLOP-matched LeVJEPA run uses 1,085 epochs and ten local views. Figure 3 reports only IN1K, so epoch-matched K400 and SSv2 results are also needed. In addition, Table 3 reports 50.7/30.4, the same values as the bidirectional row in Table 1, although the paper says later experiments use block-causal attention. This checkpoint or configuration should be clarified.

Questions for authors
  1. Under the same videos, ViT-B backbone, total FLOPs, augmentations, token budget, and frozen evaluation, how does a single-frame or per-frame LeJEPA baseline compare with full LeVJEPA?

  2. Can you provide one exact configuration table for Figure 3 and Tables 2-3, including checkpoints, attention mode, frame stride, views, retained tokens, epochs, steps, learning-rate schedules, and all counted FLOPs? Please also report epoch-matched K400 and SSv2 results and explain why Table 3 matches the bidirectional row in Table 1.

  3. Can you provide the exact loss pseudocode, including all reductions over dimensions, views, and batch samples, and explain the source and scaling of lambda = 0.02 and the exact scope of the anti-collapse guarantee?

3 comments

We thank the reviewer for the thorough and precise review. We appreciate that the reviewer found the same-data comparisons under both epoch-matched and FLOP-matched protocols useful, considered the strong IN1K and K400 results to make the simplified shared-encoder recipe practically interesting, and noted that the ablations support several important design choices. We are also grateful for the concrete and actionable requests. The suggested LeJEPA image baseline and the consolidated configuration table directly strengthen the paper, and we added both. We address each point below.

Isolating the video contribution. We agree that this control was missing. We trained an image LeJEPA baseline on frames of the same videos, with the same ViT-B, augmentations, total FLOPs and frozen evaluation as LeVJEPA (Table 3, revised):

ViT-B, same videos, matched FLOPsIN1KSSv2
DINOv2 (frames)53.816.9
LeJEPA (frames)56.418.4
LeVJEPA51.230.7

With the objective held fixed, training on video instead of frames adds 12.3 points on SSv2, so the motion gains come from video pretraining and not from the objective alone. The image model keeps an advantage on IN1K, and we revised Section 5.2 to present this as a trade-off between appearance and motion at equal compute. This new baseline also confirms that the inherited objective is competitive on images, outperforming DINOv2 on IN1K at the same budget.

Configuration and FLOPs. Appendix E now provides a single configuration table for Figure 3 and Tables 2 and 3. For every run, it lists checkpoint, attention mode, frame stride, views, retained tokens, epochs, steps, learning-rate schedule and counted FLOPs. FLOPs count the forward and backward passes of all networks evaluated during pretraining at the sequence lengths actually processed, including the target-encoder forward pass and the predictor for V-JEPA 2 and the decoder for VideoMAEv2.

Epoch-matched K400 and SSv2. At ViT-B, the epoch-matched comparison on SSv2 can already be read from the paper: the V-JEPA 2 row of Table 2 is the 240-epoch run of Figure 3 (Appendix E), and Table 1 reports the 240-epoch LeVJEPA. At matched epochs, LeVJEPA reaches 30.4 on SSv2 (30.7 with block-causal attention), compared to 42.5 for V-JEPA 2, while using 7.6× less compute (4.8 vs. 36.4 ExaFLOPs). When the saved compute is spent on a longer schedule, LeVJEPA reaches 40.4 and comes within 2.1 points of V-JEPA 2 at equal compute (Table 2). The efficiency advantage at matched epochs therefore holds on IN1K, whereas on SSv2 LeVJEPA needs a longer schedule, which still fits within the compute budget of the baselines. We revised the abstract to state that the epoch-matched claim refers to IN1K. Attentive probing on SSv2 and linear probing on K400 for all nine checkpoints of Figure 3 did not fit into the discussion period, and we will add them for all model sizes and methods in the final version.

Table 3. This was an error, thank you. Table 3 now reports the block-causal model (51.2 / 30.7).

Loss, λ\lambda, and guarantee. The exact pseudocode is in Appendix A and the supplementary code, and reproduced here for convenience:

# x: micro-batch of clips, V: local views, K = 256, M = 1024, lam = 0.02

# encode views; only the [cls] token enters the loss
z0 = proj(enc(drop(global_view(x)))[:, 0])     # (b, K)
zl = proj(enc(drop(local_views(x)))[:, 0])     # (b, V, K)
z  = cat([z0[:, None], zl], dim=1)             # (b, V+1, K)

# invariance: mean over samples, V+1 views and K dims, no stop-gradient
L_inv = ((z[:, :1] - z) ** 2).mean()

# SIGReg: random directions, resampled every forward pass
A = randn(K, M)
A = A / A.norm(dim=0)

# Epps-Pulley quadrature on t in [0, 3]
t   = linspace(0, 3, 17)
phi = exp(-t ** 2 / 2)
w   = trapezoid_weights(t) * phi

# empirical characteristic function per view, on the global batch
p   = (z.transpose(0, 1) @ A)[..., None] * t   # (V+1, b, M, 17)
ecf = stack([cos(p).mean(1), sin(p).mean(1)])  # mean over local batch
ecf = all_reduce_mean(ecf)                     # mean over global batch

# statistic, scaled by n (global batch size); mean over views and directions
err   = (ecf[0] - phi) ** 2 + ecf[1] ** 2      # (V+1, M, 17)
L_sig = (n * (err @ w)).mean()

L = L_inv + lam * L_sig

In words: The invariance term is the squared error between the global and each view embedding, averaged over the K=256K=256 embedding dimensions, the V+1V+1 views (the global view's own term is zero) and the samples, with gradients through all views and no stop-gradient. For SIGReg, the empirical characteristic function is computed per view and averaged over the global batch with a single all-reduce. The Epps-Pulley statistic is then evaluated with 17 trapezoidal knots on t∈[0,3]t \in [0, 3], multiplied by the number of samples nn as in the standard statistic, and averaged over M=1024M=1024 random directions (resampled at every step) and over the views. We also corrected Equations 2 and 3 so that they state these normalizations explicitly. We use λ=0.02\lambda = 0.02, the LeJEPA default (Balestriero & LeCun, 2025), and did not tune it for video. The anti-collapse guarantee applies to the distribution of the projected [cls] embeddings, whose optimum is an isotropic Gaussian and thus excludes complete and dimensional collapse of these embeddings. It does not directly constrain the patch tokens. We now state this scope in Appendix A.

Dear Reviewer,

Thank you again for your time and effort. As the end of the rebuttal phase is approaching, we would appreciate hearing whether our responses and the additions to the paper address your concerns. If so, we would be grateful if you would consider updating your assessment. We are happy to answer any further questions.

Sincerely,

The Authors

Reviewer_LMGmOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LeVJEPA extends LeJEPA to video through a shared encoder, an invariance objective, and SIGReg. It combines aggressive random token dropping with block causal attention to reduce pretraining cost. The paper studies this design through frozen recognition benchmarks, comparisons with video models and DINOv2, and qualitative analysis of patch representations.

Critical strengths
  1. The paper is clearly organized, with a convincing motivation for affordable video pretraining.
  2. The simple encoder design removes substantial training overhead.
  3. Controlled baseline comparisons and the consumer GPU experiment give the work practical value.
Critical weaknesses
  1. The main conclusions lean heavily on ImageNet. Section 4 acknowledges that aggressive token dropping hurts SSv2 under short schedules, but neither the main text nor the appendix shows how longer training recovers this gap. At equal compute, Table 2 still trails VideoMAEv2 on SSv2, 40.4 versus 43.6. The benefits appear more task dependent than the overall narrative suggests.

  2. Figures 4 and 5 offer only a few static visualizations. Dense performance can decline during longer training even as classification improves [1, 2], making this concern relevant to LeVJEPA's longer schedules. SIGReg's constraint on projected clip embeddings does not by itself establish stable patch representations. I would welcome further discussion of whether LeVJEPA is susceptible to this degradation. A small segmentation or tracking evaluation across checkpoints would make the patch quality claims more convincing.

[1] Dai et al. Exploring Structural Degradation in Dense Representations for Self-supervised Learning. NeurIPS 2025.

[2] Siméoni et al. DINOv3. 2025.

Questions for authors
  1. How are DINOv2 frame features aggregated for SSv2, and is temporal positional information added? I have checked the Appendix C and the supplementary code, but this is not specified. If independently encoded frames are pooled without temporal information, the readout cannot distinguish frame order. Clarifying this would help assess whether the reported gap partly reflects the evaluation protocol.

  2. Table 3 reports 50.7 on ImageNet and 30.4 on SSv2, matching the bidirectional model in Table 1. Section 4 says subsequent experiments use block causal attention, whose scores are 51.2 and 30.7. Is it a typo here?

4 comments

We thank the reviewer for the careful and detailed review. We appreciate that the reviewer found the paper clearly organized with a convincing motivation for affordable video pretraining, the simple encoder design effective in removing substantial training overhead, and the controlled baseline comparisons and consumer GPU experiment of practical value.

SSv2 and task dependence. We agree that the gains are larger on appearance than on motion, and we revised the text to state this more clearly. SSv2 accuracy benefits from longer training at equal compute: the same ViT-B goes from 30.7 at 240 epochs (Table 1) to 40.4 in the FLOP-matched run (Table 2, which also uses V=10V=10 local views), while staying within the compute budget of the baselines. We attribute the remaining gap mainly to view construction. The training target is a single clip-level embedding predicted from small local crops with 95% of tokens dropped, so fine-grained, localized motion (such as the hand-object interactions in SSv2) is often absent from a local view and contributes little to the objective. This is in line with recent analyses of which features JEPA objectives retain (Levy et al., 2026). Scaling the model and the data raises SSv2 to 55.0 (Section 5.4), so the gap is not a fixed limit of the objective. Constructing views that preserve temporal correspondences are a natural next step, as discussed in Section 6.

Dense quality over training. We agree that SIGReg on the projected clip embeddings does not by itself guarantee stable patch representations, and we now state this explicitly in Appendix A and in the limitations. We also added the structural degradation reported by Dai et al. (2025) and Siméoni et al. (2025) to the discussion. To evaluate patch quality quantitatively, we added DAVIS-2017 video object segmentation via label propagation of frozen patch tokens (Appendix F), comparing against the released ViT-L checkpoints of V-JEPA and V-JEPA 2:

ViT-LJ&FJFJ&F with temporal context
V-JEPA57.055.858.349.9 (past and future)
V-JEPA 250.748.253.141.3 (past and future)
LeVJEPA57.055.358.751.8 (past only)

At the end of its full training schedule, LeVJEPA matches V-JEPA and outperforms V-JEPA 2 by 6.3 J&F, although only the [cls] token is supervised. When frames are encoded together with surrounding frames, all models decline, but LeVJEPA declines least and outperforms both baselines while seeing only past frames.

DINOv2 aggregation on SSv2. We follow V-JEPA 2 (Assran et al., 2025): Each frame is encoded independently, and the patch tokens of all frames are passed to the same attentive probe as for the video models, that is no temporal positional embedding is added. This is intentional: the probe should read out what the frozen encoder represents, and adding temporal information in the probe would move temporal modeling from the encoder into a shallow probe. In LeVJEPA, temporal order is encoded by the encoder itself through 3D RoPE and block-causal attention. A new LeJEPA image baseline, trained on frames of the same videos at matched FLOPs, is evaluated with the same protocol to further separate the effect of the SIGReg-based objective inherited from LeJEPA. It reaches 56.4 on IN1K and 18.4 on SSv2, compared to 53.8 / 16.9 for DINOv2 and 51.2 / 30.7 for LeVJEPA. With the objective held fixed, training on clips instead of frames thus adds 12.3 points on SSv2, so the gap to image encoders reflects what the encoder learns from video rather than the objective. We describe the protocol in Appendix C and report the new baseline in Table 3.

Table 3. Thank you for catching this error. Table 3 now reports the block-causal model (51.2 / 30.7).

Dear Authors,

Thank you for your detailed responses. However, the PDF appears to remain unrevised. Could you please double-check whether the correct revised version has been uploaded?

Best,
Reviewer

Thank you for letting us know, and apologies for the inconvenience. The upload of the revised PDF did not go through due to an OpenReview error. We have now uploaded it again and verified that the revised version is available. All changes are marked in blue, and the main revisions are summarized in our general response.

Thank you for submitting the revised manuscript. I have carefully reviewed both the manuscript and your response. The revisions and clarifications address most of my concerns, and I have raised my score accordingly.

Comments on the paper

1 comment

We sincerely thank all reviewers for their time and for their thoughtful and constructive reviews. We are encouraged that the reviewers found the paper clearly organized and easy to follow (LMGm, 7Zn2, 5C3f), with a convincing motivation for affordable video pretraining (LMGm). They described the single-encoder design as simple and computationally efficient, noting that removing the teacher encoder and predictor substantially reduces training overhead (LMGm, 5C3f). They also considered the same-data comparisons under both epoch-matched and FLOP-matched protocols useful and reasonably comprehensive (7Zn2, 5C3f), found that the ablations support the key design choices of token dropping, per-frame tokenization and causal attention (7Zn2, 5C3f), and saw practical value in pretraining on a single consumer GPU (LMGm, 7Zn2).

The reviewers' comments helped us sharpen the paper, in particular by isolating the contribution of video pretraining, making the compute comparison fully auditable, and stating the scope of our claims more precisely. We uploaded a revised PDF with all changes marked in blue, and summarize the main revisions below:

  1. New LeJEPA image baseline (Reviewer 7Zn2). This new baseline is trained on frames of the same videos at matched FLOPs, with the same ViT-B, augmentations and frozen evaluation (Table 3), and isolates the contribution of video pretraining from the objective inherited from LeJEPA.
  2. Corrected Table 3 (Reviewers LMGm, 7Zn2) now reports the block-causal LeVJEPA: 51.2 on IN1K and 30.7 on SSv2.
  3. Full configuration table (Reviewer 7Zn2) for Figure 3 and Tables 2 and 3, now reports checkpoints, attention mode, views, retained tokens, epochs, steps, learning-rate schedules and counted FLOPs (Appendix E).
  4. Exact loss pseudocode (Reviewer 7Zn2), with all reductions, in Appendix A and the supplementary code.
  5. Clarified probing protocol for image encoders on SSv2 (Reviewer LMGm, Appendix C).
  6. Quantitative dense evaluation (Reviewers LMGm, 5C3f). DAVIS-2017 label propagation against the released ViT-L checkpoints of V-JEPA and V-JEPA 2 (Appendix F). Although its patch tokens receive no training signal, LeVJEPA matches V-JEPA (57.0 J&F) and outperforms V-JEPA 2 by 6.3 points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.