- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MV-JEPA, which enhances the cross-view correspondence capability of the pre-trained JEPA video encoder through post-training with multi-view prediction. The shared encoder processes each contextual view independently, whilst the joint predictor reconstructs occluded contextual features and predicts features from another fully visible view; simultaneously, single-view prediction training is interspersed to preserve the original visual capabilities. After 9,000 updates, the model demonstrated improvements in cross-view instance retrieval, cross-camera-time retrieval, and certain ego–exo action understanding tasks, recognition and semantic segmentation performance remain largely preserved.
Critical strengths
- Significant cross-perspective improvements were achieved, demonstrating a degree of transferability. V-JEPA2-L achieved improvements of 9.36 and 14.64 percentage points on ABO and Ego-Exo4D respectively, whilst also yielding gains on SOP, Assembly101 and AE2—datasets not used in the post-training phase. This indicates that the results extend beyond mere adaptation to a single training dataset.
- It simultaneously examines adaptation gains and the preservation of existing capabilities. The paper evaluates action recognition, image classification and semantic segmentation, whilst maintaining the encoder architecture and an independent single-view inference interface. For post-training schemes on pre-trained models, this evaluation is more persuasive than simply reporting improvements on the target task.
Critical weaknesses
- In the pre-processing stage of MVImgNet 2.0, the authors set the target foreground area ratio to [0.35, 0.55]. Whilst this setting may reduce the difficulty of cross-view prediction by emphasising the foreground and minimising background interference, its impact on the model’s generalisation ability remains unclear. It is recommended that the authors analyse the model’s sensitivity to this range of ratios.
- In the held-out view predictions, there is a lack of ablation experiments comparing the use and non-use of camera pose information; consequently, it remains unclear whether pose information significantly affects prediction performance and the final cross-view correspondence capability.
- The main comparative experiments use only the original pre-trained model as a baseline, lacking comparisons with other cross-view learning or post-training methods. Consequently, whilst the current results demonstrate an improvement over the initialised model, they are insufficient to fully illustrate the method’s advantages over existing approaches.
Questions for authors
As indicated in Critical Weaknesses.
Thank you for the review. Shared results are in the general response (G1–G3).
W1. Foreground ratio of the MVImgNet training crop. The crop is used only for the MVImgNet training views; no evaluation uses it. Three results show that the gain does not depend on this framing.
We will add a sweep of the training range itself to the appendix.
W2. Camera pose ablation. We ran it. Each view's relative pose and intrinsics, held-out view included, are embedded by an MLP and added to its predictor tokens; everything else is unchanged.
Pose input therefore has no noticeable effect, and MV-JEPA can be trained on multi-view data without camera calibration.
W3. Comparison with other cross-view learning and post-training methods. We compared MV-JEPA with three alternatives under identical conditions: the same encoder, data, replay and budget (G1, G2).
Post-training without a cross-view target (already in Table 5 of the paper). Continuing V-JEPA 2's own single-view training on the same data gives no gain: instance retrieval is 50.2 (MVImgNet) and moment retrieval 30.0 (Ego-Exo4D), against 50.6 and 30.1 for V-JEPA 2. Post-training by itself is not the source of the improvement.
Contrastive post-training. We trained multi-view contrastive learning in its most common form, pooled cross-view InfoNCE (one embedding per clip). Its pooled instance retrieval is higher (MVImgNet +21.4, ABO +11.0, SOP +5.6), but by design this objective works against a general encoder in two ways:
Neither protects patch detail or alignment across videos, and after contrastive post-training both are worse than before: fine-grained spatial correspondence (PCK@16: 31.9 vs 33.5 on ViT-L, 35.4 vs 39.1 on ViT-g) and frame-level action understanding across views (6 of 7 AE2 columns below the starting checkpoint). MV-JEPA keeps or improves both. Moment retrieval and transfer are on par, and recognition is slightly higher for MV-JEPA. The contrastive objective gets its retrieval number by turning the general encoder into a retrieval-specific one.
Predictive cross-view learning with a partly visible target (BYOV-style). BYOV keeps 20% of the target view visible. With the same 20% visible in our setting, almost the whole gain is gone: instance retrieval falls from 66.4 to 51.8 (MVImgNet) and moment retrieval from 44.8 to 31.0 (Ego-Exo4D), close to V-JEPA 2. When the other views show the same object or moment, any visible part of the target is a shortcut: the model completes it from its own patches without using the other views. The gain comes from hiding the target view completely.
What MV-JEPA offers over these alternatives. Of the four, only MV-JEPA both improves cross-view correspondence and keeps the encoder general. Instance retrieval rises by +15.8 (MVImgNet) and moment retrieval by +14.6 (Ego-Exo4D). Image recognition stays at 76.6 (IN1K; V-JEPA 2: 76.7), and fine-grained spatial correspondence and frame-level action understanding are kept or improved. Published numbers of methods such as BYOV cannot be compared directly: they train a task-specific model on the benchmark's own videos, while MV-JEPA post-trains a general encoder once and uses it frozen on unseen data.
We hope this answers your concerns and would be grateful if you would reconsider your score.