- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes W2Rep to learn visual representations from videos. A shared encoder processes a source image and a masked video separately. A predictor combines source features, video context, and temporal offsets to predict masked EMA target features. The method is evaluated on both image and video tasks.
Critical strengths
- Learning useful visual representations from video supervision is a meaningful problem.
- W2Rep shows promising classification results at both the image and video levels under the main comparison protocol.
- The paper provides useful ablations on the source gradient path and temporal conditioning, and acknowledges its limitations in dense prediction.
Critical weaknesses
- The novelty over closely related methods is not sufficiently clear. SiamMAE[1] already uses a shared encoder to independently encode frames and reconstruct masked future-frame regions through cross-attention. RVM[2]also learns spatial and temporal representations through video reconstruction. Neither is discussed in Section 2 or included in the comparisons. Although W2Rep introduces EMA feature targets, video-context conditioning, and signed temporal offsets, the paper should explain more clearly what these changes contribute beyond prior work.
- The video classification advantage depends on the comparison budget. In Table 4, forward-MAC-matched VideoMAE exceeds W2Rep on joint SSv2 (30.46% vs. 25.38%) and UCF101 (59.56% vs. 56.60%). This qualification should be more visible alongside the primary results in Table 1.
- The evidence is stronger for recognition than for dense prediction or correspondence. In Table 1(a), W2Rep trails VideoMAE on ADE20K by 3.58 mIoU at ViT-B and 3.51 mIoU at ViT-L, as the authors acknowledge. Figure 3 only provides qualitative cross-frame similarity maps. A quantitative correspondence evaluation, such as VOS, would help assess this ability and support comparison with SiamMAE.
[1] Siamese Masked Autoencoders (NIPS2023)
[2] Recurrent Video Masked Autoencoders(CVPR 2026)
Questions for authors
- What are the main differences from SiamMAE and RVM, and which results show the benefit of these differences?
- Since pretraining is usually performed once and the encoder can be reused across tasks, could the authors clarify the practical advantage of W2Rep over compute-matched VideoMAE, given the results in Table 4?
4 comments
Response to Reviewer pVhH (Part 2: Q2 - Compute & Reuse)
Q2: Comparison budget in Table 4 (Appendix B) and practical advantage of reusing one W2Rep model.
A2: Thanks for raising this important question. We respectfully clarify that W2Rep's advantage does not depend on comparison budget: our main comparison in Table 1 (under equal training data exposure) is already completely fair, whereas Table 4 in Appendix B was included solely to test the boundary of our model. That is why we placed Table 4 in Appendix B rather than in the main text, and we will clarify its boundary-test purpose more explicitly in Appendix B of the revised manuscript. The key points are summarized below:
1. Per-step MAC differences come from JEPA vs. MAE, not our algorithm.
Unlike MAE (which has no teacher and encodes only 10% visible patches), every JEPA model requires an EMA teacher to encode 100% unmasked target inputs. Within the JEPA family, W2Rep actually uses fewer MACs per step than V-JEPA.
2. What Table 4 (Appendix B) actually measures.
When matched by forward MACs, VideoMAE sees 8× more video data (799k vs. 100k updates) and consumes 2.5× more GPU hours than W2Rep. Thus, Table 4 in the Appendix was included solely to test our model's boundary under an extreme 8×-data regime rather than as an equal-data comparison.
3. Practical advantage of reusing one W2Rep checkpoint.
At downstream inference, W2Rep has the exact same cost as baselines (one standard ViT). Even against 799k-step VideoMAE (8× data, 2.5× GPU hours), reusing a single 100k-step W2Rep checkpoint still wins on ImageNet (+3.59%), DAVIS VOS (+3.8), and per-frame action probing, directly confirming that our method leverages video temporal cues to strengthen image-level representations. Meanwhile, full video fine-tuning nearly closes the video gap (81.84% vs. 82.43% Top-5):
| Method (Pretraining Budget) | ImageNet | DAVIS VOS | SSv2 (Frozen) | SSv2 Full FT (Top-1 / Top-5) |
|---|---|---|---|---|
| I-JEPA (MAC-matched, 520k) | 28.50 | 52.8 | -- | 12.23 / 28.43 |
| V-JEPA (MAC-matched, 95k) | 25.17 | 54.2 | 12.83 | 43.65 / 68.58 |
| VideoMAE (Equal-data, 100k) | 26.70 | 50.8 | 24.27 | 55.16 / 79.13 |
| VideoMAE (MAC-matched, 8× data, 2.5× GPU-hr) | 31.01 | 54.3 | 30.46 | 60.16 / 82.43 |
| W2Rep (Equal-data, 1× data, 1× GPU-hr) | 34.60 | 58.1 | 25.38 | 58.77 / 81.84 |
We appreciate your helpful comment, and will expand this boundary analysis in Appendix B of the revised manuscript.
Response to Reviewer pVhH (Part 3: Q3 - VOS & ADE20K)
Q3: Quantitative correspondence evaluation (DAVIS VOS) and dense prediction on ADE20K.
A3: Thanks for your constructive suggestion to evaluate quantitative correspondence via VOS. Following your advice, we evaluated frozen label propagation on DAVIS-2017 VOS across all models, and will add these results to Section 4.2 of the revised manuscript. The key findings are summarized below:
Strong quantitative correspondence on DAVIS-2017 VOS. W2Rep ranks #1 across all metrics (58.1 J&F) with the lowest temporal decay (14.3), outperforming SiamJEPA (+4.9) as well as VideoMAE under both equal-data (+7.3) and 8×-data MAC-matched (+3.8) budgets:
| Method (Pretraining Budget) | J&F ↑ | J-Mean ↑ | F-Mean ↑ | J-Decay ↓ |
|---|---|---|---|---|
| I-JEPA (MAC-matched, 520k) | 52.8 | 51.7 | 53.9 | 18.0 |
| V-JEPA (Equal-data, 100k) | 48.3 | 46.7 | 49.8 | 21.3 |
| SiamJEPA (Equal-data, 100k) | 53.2 | 52.6 | 53.8 | 22.0 |
| VideoMAE (Equal-data, 100k) | 50.8 | 50.5 | 51.1 | 22.6 |
| VideoMAE (MAC-matched, 799k) | 54.3 | 53.8 | 54.9 | 20.4 |
| W2Rep (Equal-data, 100k) | 58.1 | 57.7 | 58.5 | 14.3 |
We believe that the gap between W2Rep and VideoMAE on ADE20K actually stems from architectural differences between the JEPA and MAE paradigms. VideoMAE's edge on ADE20K stems from pixel-level reconstruction rather than temporal representation quality. As analyzed in V-JEPA 2.1 (Section 2.2, Figures 1 & 3, Table 1), supervising only masked latent targets gives unmasked context tokens little incentive to preserve local pixel boundaries. Our W2Rep already achieved the best among all JEPA based methods in Table 1a.
V-JEPA 2.1 shows that closing this dense-prediction gap requires orthogonal add-ons (context loss and multi-layer self-supervision, Section 2.3), which lie outside our single-innovation scope. We included ADE20K to candidly document this boundary for future work.
We sincerely appreciate your suggestions, and will also tighten the exposition across Sections 2–4 in the revised manuscript to make it easier to read.
Dear Reviewer pVhH,
Thank you again for your careful review. As the discussion period is coming to an end, we would be glad to clarify any remaining questions about our response.
Best regards,
The Authors
Response to Reviewer pVhH (Part 1: Q1 - SiamMAE & RVM)
Thank you for your constructive review, positive remarks on our results and ablations, and valuable suggestions. We respond in three short parts (1/3: Q1, 2/3: Q2, 3/3: Q3):
A1: Thanks for your valuable suggestion. While SiamMAE, RVM, and W2Rep all learn from video, significant differences exist in what their encoders share, their backbone architectures, and their learning objectives. We will add a dedicated discussion in Section 2 and the controlled comparison below in Appendix C. The key distinctions are summarized as follows:
1. Different meaning of the "shared encoder" (vs. SiamMAE). SiamMAE shares Siamese weights only between two isolated single frames. In contrast, W2Rep unifies image-level and video-level representations within one encoder, leveraging temporal dynamics in video to improve single-image features while simultaneously learning multi-frame video representations.
2. Different research focus and backbone architecture (vs. RVM). RVM redesigns the visual backbone into a recurrent Transformer-RNN (with causal pixel reconstruction) to process long videos sequentially at linear cost. In contrast, W2Rep retains one standard, unmodified ViT and unifies image and video representation learning in latent space via video context and signed temporal offsets.
More importantly, W2Rep fills the gap between I-JEPA (image-only, unable to model multi-frame video) and V-JEPA (video-focused, weaker image features): within one standard ViT, it both learns strong video-level representations and leverages temporal information in video to learn stronger image-level representations.
3. Controlled same-architecture comparison (SiamJEPA vs. W2Rep). RVM has not released its pretraining code, making controlled reproduction difficult. For SiamMAE, directly comparing its pixel-based model would entangle MAE-vs.-JEPA differences with our method. To isolate our algorithmic design cleanly, we implemented SiamJEPA, running SiamMAE's algorithm on the exact same ViT-B/16 JEPA architecture as W2Rep (100k steps):
SiamMAE's pairwise algorithm cannot benefit from multi-frame temporal cues (11.03% vs. 25.38% on SSv2), whereas W2Rep substantially improves both video understanding and single-image features (+6.87% on ImageNet, +4.9 on DAVIS VOS).
We appreciate your valuable suggestion, which helps clarify our contribution in Section 2 and Appendix C.