acceptodds
Under review as a conference paper at ICLR 2027

Weak Supervision Helps JEPA

Abstract

When a hand picks up a cup, understanding the action requires distinguishing the hand from the cup while capturing their interaction. Yet global video self-supervision leaves these object-level distinctions implicit. We show that weak object supervision during pretraining improves the quality and data efficiency of LeVJEPA representations. We use object masks generated offline by pretrained SAM2 to supervise an auxiliary instance-separation objective, bringing together patch features belonging to the same object while separating them from other objects and the background. Furthermore, we show that instance separation alone is the best-performing auxiliary objective, outperforming variants incorporating object-level SIGReg regularization and additional object-level alignment. The encoder architecture and retained-token budget remain unchanged. SAM2 is used only for preprocessing, and the auxiliary head is removed after pretraining, leaving no additional inference cost. Across matched experiments with ViT-Tiny, ViT-Small, and ViT-Large pretrained on 50K to 1M videos, our method improves over standard LeVJEPA at every evaluated scale. Identical attentive probes on frozen encoders achieve higher early-epoch and final accuracy. At 50K videos, Top-1 accuracy improves by approximately 9.4 percentage points on both Something-Something-v2 and Kinetics-400. The SSv2 advantage persists as pretraining data increases, reaching 6.7 points at 200K videos and 4.7 points at 1M. These gains also translate into improved data efficiency. On nested datasets, our model pretrained on 50K videos outperforms standard LeVJEPA pretrained on 95K by 6.1 SSv2 Top-1 points. In a complementary cross-scale comparison, our ViT-Small reaches 51.9% SSv2 accuracy on an SSv2-enriched 200K-video corpus, within 4.3 points of the ViT-Large baseline pretrained on 1M videos. Together, these findings identify local instance separation as an effective complement to global video self-supervision: a single auxiliary objective provides useful object structure during pretraining and enables stronger representations from limited data.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unsupervised, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_GmALOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper studies if local object supervision (through a segmentation mask) helps video pre-training methods such as LeVJEPA and VJEPA 2.1. This supervision is instantiated using supervised contrastive loss where patches from object mask represent positives and from non-object mask represent negatives.
  • This mask supervision improves action recognition and mask propagation in all four matched LeVJEPA settings, across three encoder sizes.
  • The authors experiment with varying the model size and the size of the training data. The object supervision achieves similar/better performance than the baseline with 50% of the data.
Critical strengths
  • Using additional mask supervision may help in case of smaller models (which may not have the capacity to learn generalizable features from self-supervision alone) - this is where this idea of weak supervision can be useful.
  • The experiments are designed and conducted systematically and thoroughly.
Critical weaknesses
  • It is not entirely surprising that object mask supervision during pre-training helps action recognition. LeVJEPA or VJEPA is designed to be self-supervised in the first place -- its strength comes from learning without any supervision. By using mask supervision, it is not clear what the paper is trying to achieve. Yes, it may give a few % benefit on action recognition, but it defeats the purpose of self-supervision.
  • For large models, the difference shrinks - indicating that this inductive bias may not be necessary in the first place once again re-inforcing that pure self-supervision scales more favorably than segmentation supervision. It can help with small models though.
  • Much of the ideas of this work have been studied in [1] as also stated in the introduction: "Object-centric LeJEPA (Geusen & Konukoglu, 2026) studies image pretraining on still images from COCO." What new challenges/insights does studying the same ideas for video tasks bring?
  • (Minor) I think Zhou et al [2] must be referenced since it is quite related.

[1] Object-centric LeJEPA. Jakob Geusen, Ender Konukoglu. ArXiv 2026. [2] How can objects help action recognition? Xingyi Zhou, Anurag Arnab, Chen Sun, Cordelia Schmid. CVPR 2023

Questions for authors
  • From the introduction, what does object membership mean? This does not seem like a standard term. "Video JEPA methods learn from videos without action labels (Assran et al., 2025; Kuhn et al., 2026), while object membership remains implicit in their training objectives. We make this information explicit through a local instance-separation loss"
Reviewer_mHaxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides the insight that weak instance supervision helps JEPA in better differentialting between objects and understanding the interactions between different objects. They adopt a simple group-level contrastive objective that matches patches of the same instance while pushing away different instances' patches. They get instance-level masks from SAM2 and demonstrate the effectiveness of this method on LeVJEPA and VJEPA, on action recognition (Kinetics-400 and SSv2), and video object segmentation to measure object correspondence (DAVIS and YouTube-VOS).

Critical strengths
  1. A very clear solution to a problem that is well-written for the most part.
  2. A very simple but effective pipeline to tackle the problem. They also test it on 2 different JEPA models which is appreciable.
  3. The experiments conducted are majorly rigorous and convincing, except for one point mentioned in weaknesses.
Critical weaknesses

This is a mix of weaknesses and questions I have about the paper.

  1. Writing is not clear in the methodology section at places. Like for exampl, L_{inst} is not defined but before that \lambda_{inst} number is mentioned and also the entire objective is given in terms of L_{inst} in L .
  2. Results in Tables 3 and 4 are very confusing. Table 3 shows that LeVJEPA trained using ViT-Small and 200K pretraining data is better on DAVIS and YouTube-VOS than LeVJEPA trained using ViT-Large and 1M pretraining data. This goes against the scaling law on both axes: data and model size. So more clarity on this is necessary as this reduces the impact of the experiments, and therefore, main claim in general.
  3. It is mentioned in the limitations that current method only uses local supervision and future works can look at temporal instance supervision (adding temporal positives and negatives in contrastive objective), meaning that the authors have already identified this as a potential idea that they have not tried. Given that SAM2 already provides masklets, isn’t it straightforward to implement temporal contrastive learning as well? Is there a reason to have not compared it because given that the authors recognize the idea and it is straightforward, it seems a bit surprising to me as it could strengthen the claim of the paper. Could you implement it and provide results now?
  4. Also, there are no ablations to assess the need for instance supervision? Are simpler and easy-to-acquire forms of mask supervision like foreground/background mask supervision sufficient?
Questions for authors

See weaknesses. If the experimental result confusion is resolved majorly, I will be open to raising my score.

Reviewer_FpTrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether weak object-level supervision can improve representations learned by video JEPA models. The authors use masks generated by a frozen SAM2 model to introduce an auxiliary instance-separation loss, encouraging patches from the same object to have similar features while separating different objects and the background.

The method is evaluated on LeVJEPA across several model sizes and pretraining data scales, with additional experiments on V-JEPA2. Results show consistent improvements in action recognition and video mask propagation. The method also preserves the original encoder architecture and introduces no additional inference cost.

Critical strengths

1. Consistent improvements across different settings.

The experimental results are convincing in terms of the consistency of the improvements. The proposed loss improves both action recognition and mask propagation across different model sizes and dataset scales. I also appreciate the additional V-JEPA2 experiments, which suggest that the benefits are not limited to LeVJEPA's original training objective.

2. Simple method with practical benefits.

The proposed method is straightforward and does not require architectural changes or additional inference computation. The improvements on DAVIS and YouTube-VOS are particularly encouraging, suggesting that even relatively simple object-level supervision can improve the quality of learned video representations.

Critical weaknesses

1. Limited methodological novelty.

My main concern is the novelty of the proposed method. The instance-separation objective is closely related to prior work, particularly Object-centric LeJEPA. The main difference appears to be applying this supervision during video pretraining. Moreover, the proposed loss only uses positive pairs within the same frame and view, rather than explicitly learning temporal object correspondence. It is therefore unclear how much of the improvement comes from video-specific learning, as opposed to simply introducing object-level supervision into the encoder.

2. Limited analysis of robustness and data efficiency.

The paper reports only one pretraining seed per configuration, making it difficult to assess the variability of the results. Although the improvements are consistent, additional runs would strengthen the conclusions. I also think the data-efficiency claim needs some qualification, since generating the object masks requires a pretrained SAM2 model and additional offline computation. The current comparisons mainly demonstrate improved efficiency in terms of student training videos, rather than the total resources required by the pipeline.

Questions for authors

Q1. Since the instance-separation loss operates within individual frames and views, would independently generated per-frame object masks achieve similar results? It would be useful to understand whether temporal object tracking from SAM2 contributes to the improvement.

Q2. Could the authors provide results from multiple pretraining seeds for at least one representative setting? This would help assess the robustness of the reported improvements.

Q3. What is the computational cost of generating SAM2 masks for the pretraining datasets? How would including this cost affect the claimed data-efficiency advantage?

Q4. The improvement numbers in the OpenReview abstract appear inconsistent with those in the submitted PDF. For example, the abstract reports approximately 9.4 percentage points of improvement on SSv2 at 50K videos, whereas Table 1 reports 7.24 points. Could the authors clarify which results correspond to the final experiments?

Reviewer_f7ZLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores whether weak object supervision from a frozen segmentation model can improve joint video representation learning under high token dropping.

The authors introduce an auxiliary instance-separation loss that uses offline pseudo-masks from SAM 2 to contrast patch tokens from the same object with tokens from other objects and the background.

A key contribution is showing that this local object signal improves representations even when 95% of spatial patches are randomly dropped during sparse LeVJEPA pretraining.

The auxiliary projection head is discarded after training, so encoder architecture and inference cost remain unchanged.

Extensive experiments on action recognition (SSv2, Kinetics-400) and video object segmentation (DAVIS, YouTube-VOS) achieve decent gains across model scales, datasets, and objectives, including predictive V-JEPA 2.

Critical strengths
  • Adding local instance separation provides dramatic improvements in dense video tracking (e.g., +11 to +13 points on DAVIS J&F across scales) while also lifting global action recognition. Furthermore, a model trained with weak supervision on 95K videos outperforms a baseline trained on 190K videos on mask propagation, proving that explicitly injecting object boundaries makes video pretraining far more data-efficient.
  • Because the method removes the auxiliary projection head after pretraining, the student encoder adds no latency or parameter overhead during inference. Moreover, the authors show that this local objective transfers beyond LeVJEPA to predictive masked-feature models (V-JEPA 2), demonstrating possible generalization across various self-supervised paradigms.
Critical weaknesses
  • The proposed loss contains no temporal supervision of its own. It directly ports the 2D within-image contrastive loss from Object-centric LeJEPA and applies it independently within single frames, leaving 90.62% of frame–view groups with zero positive pairs under 95% token dropping (Appendix A.3) even though SAM 2 object tracks are available across frames. Because the supervision is purely 2D spatial grouping inside each frame, almost all large gains occur on spatial mask propagation (DAVIS / YouTube-VOS), whereas improvements on temporal action recognition at scale (200K–1M) are much smaller. This limits the work’s novelty to simply applying image-model segmentation supervision frame-by-frame to video models, without truly adapting the objective to the temporal nature of video tasks.

  • Although the paper claims instance supervision consistently improves dense features across scales, dense tracking performance actually drops sharply at the largest scale. Moving from ViT-Small on 200K videos (Table 3) to ViT-Large on 1M videos (Table 4) causes the instance-supervised model to -10 on DAVIS (47.41→37.66) and -8 on YouTube-VOS (48.04→39.93) Performing worse than even the smallest ViT-Tiny model trained on just 50K videos (45.02on DAVIS, 47.01 on YouTube-VOS, Table 1). At the same time, the relative gain from instance supervision is cut roughly in half at 1M (DAVIS gain from +13.11 at 50K to +6.37 at 1M; SSv2 gain from +7.24 to +2.13). Because the 1M setting simultaneously changes the backbone size, data mixture and batch size, the paper leaves it unclear why scaling up hurts dense representations so severely.

Questions for authors
  • In Appendix A.2, you run SAM 2 video tracking so that every object has a consistent Track ID across the frames of a clip. However, your main loss restricts positive pairs to a single frame, providing no cross-frame temporal supervision and, with a 95% token drop, leaving 90.62% of frame–view groups with zero positive pairs. On the other hand, your cross-frame/cross-view ablation in Table 6 (object alignment) uses an unnormalized MSE pull on pooled object vectors with no negative pairs at all. Why not follow the standard video contrastive setup and form cross-frame positive pairs for the same tracked object (using SAM 2 Track IDs), contrasted against single-frame different-object/background negatives (or cross-video negatives)? Did you test this, and would it resolve the 90.6% empty-frame sparsity while adding true temporal supervision?

  • When scaling from ViT-Small on 200K videos (Table 3) to ViT-Large on 1M videos (Table 4), why do both the baseline and instance-supervised models drop by 5–10 points on DAVIS and YouTube-VOS, ending up below the 50K ViT-Tiny model? Also, why does the DAVIS gain from instance supervision shrink from +11.20~13.11 points at 50K–200K to +6.37 points at 1M? Does the ViT-Large architecture cause this regression, the shift in dataset mixture, or the larger batch size, and have you tested ViT-Small on the 1M mixture or ViT-Large on the 200K split to isolate the cause?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.