acceptodds
Under review as a conference paper at ICLR 2027

MV-JEPA: Post-Training Video Encoders for Cross-View Understanding

Abstract

JEPA-based video encoders learn strong representations for video and image understanding through prediction in latent space. However, their standard pretraining predicts features within individual images or video clips, without explicitly exploiting paired camera views of the same object or moment. Consequently, the learned representations may not reliably support matching across different camera viewpoints. To solve this problem, we introduce multi-view predictive post-training to strengthen cross-view instance and temporal correspondence while retaining independent single-view inference. A shared encoder processes masked context views independently, and a training-time predictor jointly predicts latent tokens for masked context regions and a fully held-out view. We alternate cross-view prediction with single-view masked prediction on general image and video data to help retain the encoder’s pretrained capabilities. After only 9,000 post-training updates on a single GPU, MV-JEPA improves cross-view instance retrieval and coarse cross-camera moment retrieval on various datasets, with further gains in zero-shot action-phase retrieval and frozen-feature classification on AE2. Frozen-probe evaluations further indicate that post-training largely preserves motion recognition, image recognition, and dense semantic segmentation performance on the evaluated benchmarks. Together, these results demonstrate that multi-view predictive post-training can strengthen instance and coarse temporal correspondence in JEPA video encoders without modifying the encoder architecture or single-view inference interface.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, JEPA, multi-view learning, cross-view prediction, viewpoint robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_pryrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MV-JEPA, which enhances the cross-view correspondence capability of the pre-trained JEPA video encoder through post-training with multi-view prediction. The shared encoder processes each contextual view independently, whilst the joint predictor reconstructs occluded contextual features and predicts features from another fully visible view; simultaneously, single-view prediction training is interspersed to preserve the original visual capabilities. After 9,000 updates, the model demonstrated improvements in cross-view instance retrieval, cross-camera-time retrieval, and certain ego–exo action understanding tasks, recognition and semantic segmentation performance remain largely preserved.

Critical strengths
  1. Significant cross-perspective improvements were achieved, demonstrating a degree of transferability. V-JEPA2-L achieved improvements of 9.36 and 14.64 percentage points on ABO and Ego-Exo4D respectively, whilst also yielding gains on SOP, Assembly101 and AE2—datasets not used in the post-training phase. This indicates that the results extend beyond mere adaptation to a single training dataset.
  2. It simultaneously examines adaptation gains and the preservation of existing capabilities. The paper evaluates action recognition, image classification and semantic segmentation, whilst maintaining the encoder architecture and an independent single-view inference interface. For post-training schemes on pre-trained models, this evaluation is more persuasive than simply reporting improvements on the target task.
Critical weaknesses
  1. In the pre-processing stage of MVImgNet 2.0, the authors set the target foreground area ratio to [0.35, 0.55]. Whilst this setting may reduce the difficulty of cross-view prediction by emphasising the foreground and minimising background interference, its impact on the model’s generalisation ability remains unclear. It is recommended that the authors analyse the model’s sensitivity to this range of ratios.
  2. In the held-out view predictions, there is a lack of ablation experiments comparing the use and non-use of camera pose information; consequently, it remains unclear whether pose information significantly affects prediction performance and the final cross-view correspondence capability.
  3. The main comparative experiments use only the original pre-trained model as a baseline, lacking comparisons with other cross-view learning or post-training methods. Consequently, whilst the current results demonstrate an improvement over the initialised model, they are insufficient to fully illustrate the method’s advantages over existing approaches.
Questions for authors

As indicated in Critical Weaknesses.

1 comment

Thank you for the review. Shared results are in the general response (G1–G3).

W1. Foreground ratio of the MVImgNet training crop. The crop is used only for the MVImgNet training views; no evaluation uses it. Three results show that the gain does not depend on this framing.

  • Uncropped test images. The MVImgNet test images are the original frames, without this crop, and instance retrieval there rises from 50.6 to 66.4. ABO, SOP, Assembly101 and AE2 never use the crop or MVImgNet, and the gains hold: +9.4 and +3.6 for instance retrieval on ABO and SOP, and +8.3 for moment retrieval on Assembly101.
  • Different framing at test time. On MVImgNet we changed how much of the query image the object fills. The gain of +15.8 becomes +16.6 and +16.0 when the query is zoomed out by 1.25× and 1.5× (smaller object, more background), and +15.4 and +13.4 with centre crops that keep 0.85 and 0.70 of the image (larger object). It stays positive even with a tight 0.40 crop (+5.1).
  • No background. With the background of both query and gallery replaced by gray, the gain remains (+9.3).

We will add a sweep of the training range itself to the appendix.

W2. Camera pose ablation. We ran it. Each view's relative pose and intrinsics, held-out view included, are embedded by an MLP and added to its predictor tokens; everything else is unchanged.

ViT-LMVImgNetABOSOPEgo-Exo4DAssembly101
MV-JEPA (no pose)66.462.247.644.839.7
MV-JEPA + pose and intrinsics67.162.247.444.739.4
  • Prediction. Pose lowers the training loss on the held-out view only slightly: 0.541 vs 0.545 in the last epoch.
  • Cross-view correspondence. Every result changes by less than one point.
  • Other camera designs. In a 3,000-update screen of seven camera-input designs, instance retrieval varied by 1.3 points, less than the 1.95 points between two seeds of one design.

Pose input therefore has no noticeable effect, and MV-JEPA can be trained on multi-view data without camera calibration.

W3. Comparison with other cross-view learning and post-training methods. We compared MV-JEPA with three alternatives under identical conditions: the same encoder, data, replay and budget (G1, G2).

Post-training without a cross-view target (already in Table 5 of the paper). Continuing V-JEPA 2's own single-view training on the same data gives no gain: instance retrieval is 50.2 (MVImgNet) and moment retrieval 30.0 (Ego-Exo4D), against 50.6 and 30.1 for V-JEPA 2. Post-training by itself is not the source of the improvement.

Contrastive post-training. We trained multi-view contrastive learning in its most common form, pooled cross-view InfoNCE (one embedding per clip). Its pooled instance retrieval is higher (MVImgNet +21.4, ABO +11.0, SOP +5.6), but by design this objective works against a general encoder in two ways:

  • It is pooled: it compares one averaged vector per clip, exactly the feature that instance retrieval reads out, and never constrains a patch token.
  • It is contrastive: it pushes apart all other samples, including clips of the same action from other videos.

Neither protects patch detail or alignment across videos, and after contrastive post-training both are worse than before: fine-grained spatial correspondence (PCK@16: 31.9 vs 33.5 on ViT-L, 35.4 vs 39.1 on ViT-g) and frame-level action understanding across views (6 of 7 AE2 columns below the starting checkpoint). MV-JEPA keeps or improves both. Moment retrieval and transfer are on par, and recognition is slightly higher for MV-JEPA. The contrastive objective gets its retrieval number by turning the general encoder into a retrieval-specific one.

Predictive cross-view learning with a partly visible target (BYOV-style). BYOV keeps 20% of the target view visible. With the same 20% visible in our setting, almost the whole gain is gone: instance retrieval falls from 66.4 to 51.8 (MVImgNet) and moment retrieval from 44.8 to 31.0 (Ego-Exo4D), close to V-JEPA 2. When the other views show the same object or moment, any visible part of the target is a shortcut: the model completes it from its own patches without using the other views. The gain comes from hiding the target view completely.

What MV-JEPA offers over these alternatives. Of the four, only MV-JEPA both improves cross-view correspondence and keeps the encoder general. Instance retrieval rises by +15.8 (MVImgNet) and moment retrieval by +14.6 (Ego-Exo4D). Image recognition stays at 76.6 (IN1K; V-JEPA 2: 76.7), and fine-grained spatial correspondence and frame-level action understanding are kept or improved. Published numbers of methods such as BYOV cannot be compared directly: they train a task-specific model on the benchmark's own videos, while MV-JEPA post-trains a general encoder once and uses it frozen on unseen data.

We hope this answers your concerns and would be grateful if you would reconsider your score.

Reviewer_Rb23OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MV-JEPA, a multi-view predictive post-training framework for improving cross-view correspondence in pre-trained JEPA video encoders. The method independently encodes multiple masked context views and employs a joint predictor to reconstruct masked context tokens and predict the latent representation of an entirely held-out view. An EMA encoder supplies prediction targets, while single-view masked prediction replay is interleaved with multi-view updates to help retain pre-trained visual capabilities. The post-trained encoder preserves its original single-view inference interface. Experiments on V-JEPA2-L and V-JEPA2.1-g demonstrate improvements in cross-view instance retrieval and synchronized cross-camera moment retrieval. Additional evaluations on AE2 show improvements in action-phase retrieval, while frozen probes indicate largely preserved recognition and segmentation performance. Ablations further suggest that held-out-view prediction is an important component of the reported retrieval gains.

Critical strengths

[S1] Substantial and transferable improvements in cross-view correspondence. MV-JEPA significantly improves instance retrieval and synchronized moment retrieval over the released V-JEPA2-L encoder. The improvements are observed on datasets excluded from post-training, including ABO, SOP, and Assembly101, and are reproduced with V-JEPA2.1-g. This provides meaningful evidence that the proposed procedure improves representations beyond memorizing the specific training instances. The method retains the original single-view inference architecture, which is practically useful for adapting existing video encoders.

[S2] A practical post-training design with encouraging results. The method requires a relatively short post-training stage and introduces no additional inference-time modules. The paper provides evaluations on both cross-view adaptation and retention of general recognition capabilities, addressing an important trade-off in representation adaptation. In addition, the ablation separating single-view post-training, multi-view context prediction, and held-out-view prediction provides evidence that the complete objective contributes more than additional training on the same data sources. The reported recognition and segmentation results suggest that substantial cross-view improvements can coexist with limited degradation of previously learned capabilities.

Critical weaknesses

[W1] Novelty. The proposed framework combines established techniques in multi-view representation learning, masked latent prediction, and self-supervised post-training. Cross-view correspondence learning has been extensively explored through contrastive objectives, including Time-Contrastive Networks [Sermanet et al., CVPRW’17], and through predictive objectives, including CroCo [Weinzaepfel et al., NeurIPS’22], BYOV [Park et al., CVPR’25], and seq-JEPA [Ghaemi et al., NeurIPS’25]. MV-JEPA differs by adapting pre-trained JEPA video encoders through fully held-out-view prediction and single-view replay, while preserving the original inference interface. However, these distinctions primarily concern the training configuration and adaptation procedure rather than a fundamentally new representation-learning principle. The manuscript provides limited conceptual or analytical insight into why the proposed combination should produce better representations than existing alternatives. Consequently, while the approach is practically reasonable, its methodological novelty and broader scientific contribution appear somewhat incremental.

[W2] Missing comparisons.

  • The experiments demonstrate substantial improvements over the released V-JEPA encoders, but do not sufficiently establish the advantage of the proposed objective over existing cross-view representation learning methods. The current ablations compare single-view post-training, multi-view masked-context prediction, and the complete framework. Although these comparisons support the benefit of held-out-view prediction within the proposed architecture, they do not establish whether the specific predictive formulation is superior to simpler multi-view alignment strategies.
  • The framework combines masked-context reconstruction, fully held-out-view prediction, and single-view masked-prediction replay. However, the available ablations do not independently establish the contributions of all three mechanisms. Table 5 shows that removing held-out-view prediction substantially reduces retrieval performance, but this comparison also removes the corresponding prediction queries, supervised tokens, and associated computation.
  • The central objective requires the model to predict the latent features of an entirely held-out observation using representations from other views. However, the predictor does not receive explicit information about the target camera pose or relative viewpoint. Consequently, multiple possible target observations may be consistent with the same context views, making the prediction problem inherently ambiguous. Under this formulation, the model could improve its objective by emphasizing view-invariant appearance or semantic information rather than learning explicit geometric or temporally precise correspondences. This behavior may be beneficial for instance retrieval, but the current experiments do not distinguish between improved viewpoint invariance, stronger instance discrimination, and more detailed cross-view scene understanding. The manuscript provides limited representation-level analyses explaining which properties are enhanced by post-training.
Questions for authors

[Q1] Could the authors clarify the principal methodological contribution of MV-JEPA relative to established cross-view contrastive and predictive representation-learning methods? In particular, what property of fully held-out-view latent prediction makes it preferable to conventional cross-view alignment or masked cross-view reconstruction? A more explicit conceptual comparison with TCN, CroCo, BYOV, and seq-JEPA would help clarify the novelty and significance of the proposed learning strategy.

[Q2] Could the authors compare MV-JEPA against a cross-view contrastive objective, such as InfoNCE using synchronized observations as positive pairs, under the same pre-trained backbone, training datasets, replay schedule, and optimization budget? Additionally, a comparison with an alternative predictive objective that uses an equivalent number of supervised tokens but does not predict a fully held-out view would help determine whether the improvement is specific to the proposed cross-view prediction mechanism.

[Q3] The held-out-view predictor does not receive explicit information about the target camera pose or relative viewpoint. Under this setting, how does the model distinguish among multiple possible held-out observations that are consistent with the same context views? Could the authors analyze whether the learned representations primarily become more viewpoint-invariant or whether they also preserve viewpoint-dependent geometric information? Comparisons using camera-pose conditioning or geometry-sensitive correspondence evaluations would help establish what the proposed objective actually encourages the encoder to learn.

[Q4] The paper uses single-view masked-prediction replay to help preserve pretrained recognition capabilities, but does not evaluate the full multi-view objective without replay. Could the authors provide such a comparison and report both cross-view retrieval and general recognition performance? It would also be informative to examine different replay ratios under a fixed post-training budget. These results would clarify whether replay is necessary for preventing representation degradation and how it influences the trade-off between cross-view adaptation and general visual performance.

2 comments

Thank you for the careful review. Shared results are in the general response.

W1, Q1. What is new, and why fully held-out latent prediction. The contribution is a learning principle, with evidence for each of its parts: to give a pretrained general encoder cross-view ability, predict an entirely unobserved view, in latent space, token by token.

  • Entirely unobserved target. When part of the target is visible, as in CroCo and BYOV, the model can complete it from its own patches without using the other views. With 10% of the target visible, much of the gain is lost; with 20%, almost all of it (G1).
  • Token-level prediction instead of pooled alignment. A contrastive objective, as in TCN, compares one pooled vector per input. It gives higher pooled retrieval, but the encoder ends below its starting checkpoint in fine-grained spatial correspondence and in frame-level action understanding across views (see Q2 below).
  • Latent targets, no pose. Predicting latents instead of pixels (CroCo) lets the model leave out what the other views cannot predict. A pose or transformation input, as in seq-JEPA, is not needed (see Q3 below).
  • Post-training with replay. None of these methods post-trains a pretrained general video encoder while keeping its abilities. MV-JEPA does: in 9,000 updates, it gains +14.6 on cross-camera moment retrieval (Ego-Exo4D) and, on data it was never post-trained on, +9.4 and +3.6 on instance retrieval (ABO, SOP) and +8.3 on moment retrieval (Assembly101). Image recognition stays at 76.6 (IN1K; V-JEPA 2: 76.7), and single-view inference is unchanged.

W2, Q2. Contrastive objective and an equal-token alternative.

  • InfoNCE with the same backbone, data, replay and budget (G2). Pooled instance retrieval is higher (MVImgNet +21.4, ABO +11.0, SOP +5.6); moment retrieval and transfer are on par. By design, this objective works against a general encoder: being pooled, it never constrains a patch token; being contrastive, it pushes apart all other samples, including the same action in other videos. After contrastive post-training, fine-grained spatial correspondence is worse than before (PCK@16: 31.9 vs 33.5 on ViT-L, 35.4 vs 39.1 on ViT-g), and so is frame-level action understanding across views (6 of 7 AE2 columns below the starting checkpoint). MV-JEPA keeps or improves both.
  • Equal tokens, target 10% visible instead of fully held out (G1). The predictor processes exactly as many tokens as in MV-JEPA, and 95–97% of the tokens are still supervised. Cross-view instance retrieval falls from 66.4 to 58.5 (MVImgNet) and moment retrieval from 44.8 to 32.6 (Ego-Exo4D). The improvement is specific to predicting a fully held-out view.

W2. Contribution of each mechanism.

  • Held-out prediction produces the cross-view gain. Without it, cross-view instance retrieval falls from 66.4 to 53.3 (MVImgNet) and moment retrieval from 44.8 to 31.0 (Ego-Exo4D); see Table 5. The equal-token control above rules out fewer supervised tokens or less computation as the cause.
  • The masked-context loss carries the gain beyond the post-training data. Without it, transfer to unseen data weakens (Assembly101 moment retrieval falls from 39.7 to 37.8), and fine-grained spatial correspondence worsens (PCK@16 falls from 33.6 to 32.0).
  • Replay preserves recognition and out-of-domain transfer. Without it, image recognition falls from 76.6 to 76.0 (IN1K), motion recognition from 82.8 to 81.7 (Diving48) and moment retrieval from 39.7 to 38.4 (Assembly101).

W2, Q3. No pose input: ambiguity, invariance and geometry.

  • Pose conditioning. Each view's relative pose and intrinsics, held-out view included, are embedded by an MLP and added to its predictor tokens. Results change by under one point: MVImgNet +0.7, ABO +0.0, SOP −0.2, Ego-Exo4D −0.0, Assembly101 −0.3. The ambiguity does not hold the encoder back. With a regression loss in latent space, the predictor is rewarded for what the possible target views share, and this is what the encoder learns to encode.
  • Viewpoint invariance. The pooled feature becomes robust to viewpoint. The ABO gain is +9.0, +9.6, +8.2 and +10.7 for azimuth differences of 0–45°, 45–90°, 90–135° and 135–180°.
  • Geometric information is preserved. Cross-view point matching (PCK@16), a geometry-sensitive test on patch features, is 33.6 vs 33.5 for V-JEPA 2 on ViT-L, and 40.4 vs 39.1 on ViT-g.
  • Temporal precision improves. On AE2, Kendall's τ\tau rises from 0.334 to 0.364. With only neighbouring windows as candidates, moment retrieval rises from 52.1 to 64.2.

Q4. Replay. G3 reports 0%, 25% and 49% replay at 9,000 updates, with retrieval and recognition. Without replay, recognition and out-of-domain transfer drop, so replay is needed to prevent degradation. With 25% replay, IN1K is unchanged and retrieval is higher; its video recognition is not finished yet, so we keep 49%.

We hope this answers your concerns and would be grateful if you would reconsider your score.

I would like to thank the authors for the detailed responses and extensive controlled experiments provided during the rebuttal period.

Several of my initial questions regarding controlled ablations and component attribution have been clarified, including the compute-matched partial-visibility controls (10% and 20% target visibility in G1), the no-context-loss ablation, the camera-pose conditioning experiment, and the replay-ratio comparisons (G3).

At the same time, my broader concerns regarding methodological novelty, contrastive baseline design, and what cross-view properties are actually learned remain unresolved. Conceptually, MV-JEPA remains primarily a practical post-training recipe combining V-JEPA latent prediction, full-view cross-view completion, and replay. Regarding [Q2], under identical data and compute, simple pooled InfoNCE substantially outperforms MV-JEPA on cross-view instance retrieval (+21.4 on MVImgNet, +11.0 on ABO, and +5.6 on SOP) and matches or slightly exceeds it on moment retrieval (+1.6 on Ego-Exo4D and +0.0 on Assembly101). Because only a clip-pooled InfoNCE objective without hard temporal negatives or token-level alignment was tested, the degradation observed on PCK@16 and AE2 reflects clip-level pooling rather than an inherent limitation of contrastive or non-JEPA cross-view objectives (such as TCN). Furthermore, regarding [Q3] and [Q4], adding explicit camera pose and intrinsics changes downstream scores by less than one point and PCK@16 on ViT-L remains essentially unchanged from V-JEPA 2 (33.6 vs. 33.5), while the shuffled-frame test on Ego-Exo4D (retaining +11.71 of the +14.64 gain when frame order is randomized) indicates that the post-training gains stem largely from static multi-frame co-occurrence and viewpoint-invariant appearance rather than geometric correspondence or fine-grained temporal dynamics, and the 25% replay video-recognition evaluation remains incomplete.

Taking everything into account, while the rebuttal experiments substantially improve the empirical attribution of the post-training recipe, my main reservations regarding conceptual novelty and the scope of learned geometric and temporal correspondence remain. I would keep my initial score.

Reviewer_esBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces multi-view predictive post-training for V-JEPA to improve cross-view instance and temporal correspondence while preserving its general visual capabilities. Experiments show improvements over the original V-JEPA backbones.

Critical strengths
  • The paper shows that multi-view predictive post-training can improve the cross-view understanding capabilities of pretrained video encoders, with consistent improvements across several benchmarks.
  • The method is relatively efficient (9,000 updates on a single GPU) and does not require modifying the encoder architecture or inference pipeline.
Critical weaknesses
  • Lack of competitive baselines. This is my biggest concern. The paper mainly compares against the original V-JEPA backbones, without comparing with existing cross-view representation learning methods such as BYOV. It also lacks comparisons with alternative post-training objectives, such as contrastive learning.
  • The cross-backbone evaluation only covers two closely related JEPA models, which is limited and is not sufficient enough to list as a single second contribution.
  • The idea of combining self-view and cross-view masked prediction has already been explored in BYOV. However, the paper does not sufficiently discuss how MV-JEPA differs from BYOV beyond adapting a similar framework to pretrained JEPA encoders.
  • Presentation needs improvement. Section 3 mixes methodology with training and implementation details, making it hard to follow. The experimental analysis is also mixed up and hard to follow.
Questions for authors
  1. How does MV-JEPA compare with existing cross-view representation learning methods, such as BYOV, using a similar evaluation protocol?
  2. Can the authors compare the training recipe with other alternatives, such as contrastive learning?
  3. How does the target-view masking ratio affect performance? In particular, how does fully held-out-view prediction compare with BYOV-style heavy masking, where part of the target view remains visible? Beyond this difference and the adaptation to pretrained JEPA encoders, what is the key methodological novelty of MV-JEPA compared with BYOV?
  4. Can the authors provide an ablation without single-view replay to demonstrate its necessity for preserving general visual capabilities?
1 comment

Thank you for the detailed review. Shared results are in the general response (G1–G3).

W1, Q1, Q3. Comparison with BYOV.

Why the published numbers cannot be compared directly. The two methods have different goals. BYOV builds a model for one fine-grained ego-exo task: it keeps a CLIP image encoder frozen and trains a small Transformer on the training videos of the AE2 task on which it is then evaluated. MV-JEPA aims to give a general video encoder cross-view ability: it post-trains the encoder once and uses it frozen on data it was never post-trained on (ABO, SOP, Assembly101, AE2). The two sets of numbers answer different questions, so the fair comparison is between the designs under identical conditions.

BYOV's design in our setting (G1). In BYOV's cross-view prediction the target view is not fully hidden: 20% of it stays visible. With the same encoder, data and budget as MV-JEPA, we changed only this point: 20% (BYOV's ratio) or 10% of the target view stays visible.

ViT-LMVImgNetABOEgo-Exo4DAssembly101
V-JEPA 250.652.830.131.4
Target 20% visible (as in BYOV)51.855.031.033.6
Target 10% visible58.558.532.636.3
MV-JEPA (target fully hidden)66.462.244.839.7

With 20% visible, almost the whole gain is gone; with 10%, much of it is gone. BYOV-style partial visibility does not work in our setting: the gain comes from hiding the target completely.

How MV-JEPA differs from BYOV (W3).

  • Fully hidden target. In MV-JEPA the other views show the same object or the same moment, so they determine the target view. Any visible part of the target is then a shortcut: the model completes the target from its own patches without using the other views. In BYOV the other view is an independently recorded video, which cannot determine the target, so part of it must stay visible; BYOV's own ablation finds 80% masking better than 100%.
  • Patch-level prediction. MV-JEPA predicts the patch-level features of the whole unseen view. BYOV predicts one CLIP feature per frame.
  • A general encoder instead of a task model. One short post-training run (9,000 updates on one GPU) improves the encoder itself. The same frozen encoder then transfers without further training: +9.4 on ABO, +3.6 on SOP and +8.3 on Assembly101, none of which it was post-trained on, and +14.6 on moment retrieval for unseen Ego-Exo4D captures. With replay, IN1K stays at 76.6 (V-JEPA 2: 76.7). The architecture and the single-view inference are unchanged, so it replaces the original encoder directly.

W1, Q2. Contrastive learning. We trained multi-view contrastive learning in its most common form, pooled cross-view InfoNCE (one embedding per clip), on exactly MV-JEPA's inputs (G2). Its pooled instance retrieval is higher (MVImgNet +21.4, ABO +11.0, SOP +5.6), but this does not make it the better way to post-train the encoder. By design, this objective works against a general encoder in two ways:

  • It is pooled. It compares one averaged vector per clip, exactly the feature that instance retrieval reads out, and never constrains a patch token.
  • It is contrastive. It pushes apart all other samples, including clips of the same action from other videos.

Neither protects patch detail or alignment across videos, and after contrastive post-training both are worse than before. Cross-view point matching (PCK@16) is 31.9 after vs 33.5 before on ViT-L, and 35.4 vs 39.1 on ViT-g. On AE2, 6 of 7 classification, frame-retrieval and Kendall's τ\tau columns fall below the starting checkpoint. MV-JEPA predicts every patch feature of an unseen view and uses no negatives, and it keeps or improves both. Moment retrieval and transfer are on par, and recognition is slightly higher for MV-JEPA. The contrastive objective gets its retrieval number by turning the general encoder into a retrieval-specific one.

Q4. Ablation without replay. See G3. Without replay, IN1K drops from 76.6 to 76.0, Diving48 from 82.8 to 81.7 and out-of-domain Assembly101 from 39.7 to 38.4. Replay is therefore needed to preserve the general capabilities.

W2. Two backbones. The method needs a pretrained latent predictor, which JEPA models provide. The two backbones differ in size (ViT-L with 0.3B parameters, ViT-g with 1.0B) and in pretraining generation (V-JEPA 2 and V-JEPA 2.1), and the gain appears on both. On ViT-g, MVImgNet rises from 61.8 to 73.4, ABO from 46.7 to 51.8 and SOP from 45.9 to 48.2. We will present this as a transfer result of the method instead of a separate contribution.

W4. Presentation. We will split Section 3 into the method (objective and replay) and an implementation part with a table of all settings. We will also organize the experiments by question: cross-view correspondence, where the gain comes from, and what is retained.

We hope this answers your concerns, and we would be grateful if you would reconsider your score.

Reviewer_JUbbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Overall, the paper has improved the cross-view retrieval performance through short-term post-training. During reasoning, only a single view is still required. The overall idea is relatively clear and it has certain practical value. However, at present, more sufficient comparisons are needed to illustrate the advantages of the method compared to other solutions, as well as the respective roles of each design. The details of training and feature extraction also need to be supplemented.

Critical strengths

The author used other perspectives to predict the perspective features of the flow, and then interspersed single-perspective replays for post-training. After 9000 steps of training on a single GPU, the ABO instance retrieval and Ego-Exo4D moment retrieval of V-JEPA2-L were improved by 9.36 and 14.64 percentage points respectively, and the inference only required single-perspective.

Critical weaknesses
  1. The cross-perspective experiments in the paper mainly compared with the original JEPA and its text variants. I currently cannot see the advantages of this method compared to other cross-perspective training. It is recommended to add multi-perspective contrast learning methods as a control under the same encoder data and training budget conditions.
  2. Table 5 of the paper shows that the retained perspective branch is useful, but when it is removed, the corresponding queries and predictions also disappear. Although the training steps are the same, the number of supervised tokens and the amount of computation are different. It is recommended to add a control with similar supervision quantity and computation amount to further confirm the role of the prediction target itself.
  3. All the post-training variants in Table 5 of the paper retained the single-perspective replay. It is not clear how much help and contribution it has to retain the original capabilities. It is recommended to further keep the total number of steps unchanged, compare the effect of having replay and different replay ratios, and also report the results of cross-perspective retrieval and general recognition.
  4. In Section 4.2, only one out of six fragments was selected each time. If there are obvious differences in the action stage, the picture may also select the correct one. Although there is the AE2 evaluation, I still hope to see the results when the candidate moments are closer, as well as how much the retrieval performance changes when using only a single frame or shuffling the frame order.
  5. I did not find the learning rate settings and adjustment methods for post-training in the paper, nor did I see how to summarize the temporal and spatial tokens into feature vectors when performing instance and moment retrieval. These details will be used for reproduction, and I hope the author can provide the specific configuration and processing steps.
Questions for authors

Look at the weaknesses section

1 comment

Thank you for the concrete suggestions; we ran all of them. Shared results are in the general response (G1–G3).

W1. Comparison with multi-view contrastive learning. We trained multi-view contrastive learning in its most common form, pooled cross-view InfoNCE (one embedding per clip), with the same encoder, data, replay and budget (G2). Its pooled instance retrieval is higher (MVImgNet +21.4, ABO +11.0, SOP +5.6), but this does not make it the better way to post-train the encoder. By design, this objective works against a general encoder in two ways:

  • It is pooled. It compares one averaged vector per clip, exactly the feature that instance retrieval reads out, and never constrains a patch token.
  • It is contrastive. It pushes apart all other samples, including clips of the same action from other videos.

Neither protects patch detail or alignment across videos, and after contrastive post-training both are worse than before:

  • Cross-view point matching (PCK@16): 31.9 after vs 33.5 before on ViT-L, and 35.4 vs 39.1 on ViT-g. MV-JEPA keeps or raises it (33.6 and 40.4).
  • AE2: below its starting checkpoint in 6 of 7 classification, frame-retrieval and Kendall's τ\tau columns, and below MV-JEPA in all 7 (τ\tau 0.280 vs 0.364).

Its advantage is limited to pooled object retrieval: moment retrieval and transfer are on par, and recognition is slightly lower (IN1K 76.1 vs 76.6, SSv2 73.5 vs 73.9).

The paper's setting is a general video encoder that is used frozen for many tasks, so post-training should add cross-view ability without taking away what the encoder already has. MV-JEPA predicts every patch feature of a view the encoder never sees and uses no negatives: retrieval, moment retrieval and transfer improve, while recognition and fine-grained performance are kept. The contrastive objective gets its retrieval number by turning the general encoder into a retrieval-specific one.

W2. Control with matched supervision and compute. We added it (G1). The held-out view is no longer fully hidden: the encoder sees 10% of its patches and predicts the rest. The predictor processes exactly as many tokens as in MV-JEPA, and 95–97% of the tokens are still supervised. Much of the gain disappears: MVImgNet 58.5 vs 66.4 and Ego-Exo4D 32.6 vs 44.8 (V-JEPA 2: 50.6 and 30.1). With 20% visible it is almost gone (51.8 and 31.0). In the other direction, supervising only the held-out view (27–53% of the tokens) keeps the gain (70.5 and 47.2). The gain therefore comes from the prediction target itself, a view the encoder never sees, and not from the amount of supervision or compute.

W3. Replay at a fixed number of steps. See G3 (0%, 25% and 49% replay, with retrieval and recognition). Replay is what retains the original capabilities. Without it, IN1K drops from 76.6 to 76.0, Diving48 from 82.8 to 81.7 and Assembly101 from 39.7 to 38.4, while retrieval rises because the multi-view updates double. With 25% replay, IN1K is unchanged (76.5 vs 76.6) and retrieval is higher. Its video recognition benchmarks are not finished yet, so we keep 49% and will report the full sweep.

W4. Closer candidate moments, single frame, shuffled frames. Ego-Exo4D, 81 takes from 71 captures not used in post-training, Top-1 (%):

Candidates / inputV-JEPA 2MV-JEPAGain
Six windows (paper; chance 16.7%)30.1144.75+14.64
True window and its neighbours only (chance 38.9%)52.1164.16+12.05
Centre frame repeated 16 times21.0223.18+2.16
Frames shuffled within each clip31.2943.00+11.71
  • Closer candidates. Neighbouring windows are about 14 s apart. The gain is still +12.05, and the two controls of Table 5 stay at V-JEPA 2's level (51.89 and 53.34). The gain does not rely on obviously different action stages.
  • Single frame. Every model falls close to chance, so one picture is not enough to pick the right moment, and the gain shrinks to +2.16.
  • Shuffled frames. +11.71 of the +14.64 remains. The gain comes from the content of several frames, not from their order.

W5. Learning rate and feature extraction.

  • Optimization. AdamW, β=(0.9,0.999)\beta = (0.9, 0.999), weight decay 0.04. The learning rate increases linearly from 10−610^{-6} to 3.775×10−53.775\times10^{-5} over the 9,000 updates, so early updates to the pretrained weights are small. EMA momentum is 0.99925 (constant). Each step uses 48 multi-view groups or 24 replay clips.
  • Features. Instance and moment retrieval use the same feature from the frozen EMA encoder. We take all 2,048 final-layer spatio-temporal tokens of a 16-frame, 256-px clip, apply LayerNorm without learnable parameters to each token, average over all tokens, L2-normalize, and compare with cosine similarity.

We will add these details to Section 3 and release code and configs.

We hope this answers your concerns, and we would be grateful if you would reconsider your score.

Comments on the paper

1 comment

We thank the reviewers and ran requested experiment. Each new model starts from the same public checkpoint as MV-JEPA, is trained with the paper's data, replay, optimizer, schedule, seed and 9,000-update budget, changes one factor, and is evaluated with the paper's protocols. All results below will be added to the revised paper, and we will release code and checkpoints.

Summary

  1. The gain comes from predicting a view the student never sees, not from the amount of supervision.
  2. A contrastive baseline (InfoNCE) on the same views is better at pooled instance retrieval, but falls below its starting checkpoint on the paper's fine-grained AE2 benchmark and on cross-view point matching, where MV-JEPA is better. Recognition is slightly higher for MV-JEPA, and moment retrieval and transfer are on par.
  3. Replay is what preserves recognition and out-of-domain transfer; without it, both drop.

G1. What produces the gain

In the "Target 10% / 20% visible" rows, the held-out view is no longer fully hidden: the encoder sees 10% or 20% of its patches (20% as in BYOV) and predicts the rest. With 10% visible, the predictor processes exactly as many tokens as in MV-JEPA, so compute is matched. In the "no context loss" row, only the held-out view is predicted. "Supervised tokens" is the number of tokens with a prediction loss, relative to MV-JEPA, for 2–6 context views.

ViT-LSupervised tokensMVImgNetABOEgo-Exo4DAssembly101
V-JEPA 2–50.652.830.131.4
No held-out view (Tab. 5)47–73%53.355.831.033.7
Target 20% visible89–95%51.855.031.033.6
Target 10% visible95–97%58.558.532.636.3
MV-JEPA (held out)100%66.462.244.839.7
Held out, no context loss27–53%70.562.247.237.8

Revealing 20% of the target keeps ≥89%\geq 89\% of the supervision but removes almost the whole gain (Ego-Exo4D 31.0 vs 30.1 for V-JEPA 2). Removing half or more of the supervision (no context loss) keeps the gain. Scores even rise on the two post-training datasets (MVImgNet, Ego-Exo4D), but fall on other data: Assembly101 drops by 1.9 and PCK@16 by 1.6. (PCK@16, cross-view point matching, measures fine-grained spatial correspondence, i.e. the quality of dense patch features: on 200 held-out MVImgNet objects, points are matched across views by nearest patch feature.) The context loss is what makes the gain carry beyond the post-training data, so we keep it.

G2. Contrastive baseline

We trained pooled cross-view InfoNCE (T=0.2T = 0.2; one embedding per clip) on exactly MV-JEPA's inputs. InfoNCE minus MV-JEPA:

  • Instance retrieval: MVImgNet +21.4, ABO +11.0, SOP +5.6.
  • Moment retrieval and transfer: Ego-Exo4D +1.6, Assembly101 +0.0.
  • Recognition, MV-JEPA vs InfoNCE: IN1K 76.6 vs 76.1, SSv2 73.9 vs 73.5, Jester 97.0 vs 96.9, Diving48 82.8 vs 83.0.

On the paper's AE2 benchmark, InfoNCE is below MV-JEPA in every classification, frame-retrieval and Kendall's τ\tau column, averaged over the four tasks.

F1 RegularF1 Ego2ExoF1 Exo2EgomAP@10 RegularmAP@10 Ego2ExomAP@10 Exo2EgoKendall's τ\tau
MV-JEPA78.4241.0852.8074.9752.6856.860.3644
InfoNCE74.6839.4546.4868.7250.0846.140.2799
Difference−3.74−1.63−6.32−6.25−2.60−10.72−0.0845

On PCK@16, starting checkpoint / MV-JEPA / InfoNCE score 33.5 / 33.6 / 31.9 on ViT-L and 39.1 / 40.4 / 35.4 on ViT-g.

How we read this. By design, this objective works against a general encoder in two ways. It is pooled: it compares one averaged vector per clip, the feature that retrieval reads out, and never constrains a patch token. It is contrastive: it pushes apart all other samples, including clips of the same action from other videos. Neither protects patch detail or alignment across videos, and both degrade: PCK and 6 of these 7 AE2 columns fall below the starting checkpoint. MV-JEPA predicts every patch latent of an unseen view and uses no negatives. It adds view robustness while keeping the encoder's recognition and fine-grained performance. The contrastive objective instead turns the general encoder into a retrieval-specific one.

G3. Replay at 9,000 updates

ReplayMVImgNetEgo-Exo4DAssembly101IN1KDiving48
0%70.651.438.476.081.7
25%68.248.040.876.5–
49% (paper)66.444.839.776.682.8
V-JEPA 250.630.131.476.783.3

Without replay, the multi-view updates double and retrieval rises, but IN1K drops by 0.5, Diving48 falls 1.6 below V-JEPA 2 (MV-JEPA: 0.5 below) and out-of-domain Assembly101 drops by 1.3. A 25% share keeps IN1K (76.5 vs 76.6) and raises retrieval and Assembly101. Its video recognition benchmarks are not finished yet, so we keep the paper's 49% and will report the full sweep.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.