acceptodds
Under review as a conference paper at ICLR 2027

W2Rep: Learning Visual Representations by Watching the World Change

Abstract

Images capture the world at one moment, whereas video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead improve features available from one image without sacrificing the ability to represent video. We introduce W2REP, a masked feature-prediction framework in which an independently encoded source image participates in prediction at the same or another moment. The predictor is conditioned on visible video context, the queried location, and the signed time interval between source and target. This gives the cross-frame objective two complementary roles: the image path learns features that remain useful across time, while the video path must gather evidence that is missing from the source image. Across model scales and downstream tasks, W2REP improves frozen and fine-tuned recognition under our comparison protocol, while joint video encoding provides further gains over frame-wise aggregation. Controlled experiments show that these gains depend on directly updating the source-image features and on using both video context and temporal displacement. Overall, change across a video can supervise a visual encoder whose representations remain useful at either image or video granularity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pVhHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes W2Rep to learn visual representations from videos. A shared encoder processes a source image and a masked video separately. A predictor combines source features, video context, and temporal offsets to predict masked EMA target features. The method is evaluated on both image and video tasks.

Critical strengths
  1. Learning useful visual representations from video supervision is a meaningful problem.
  2. W2Rep shows promising classification results at both the image and video levels under the main comparison protocol.
  3. The paper provides useful ablations on the source gradient path and temporal conditioning, and acknowledges its limitations in dense prediction.
Critical weaknesses
  1. The novelty over closely related methods is not sufficiently clear. SiamMAE[1] already uses a shared encoder to independently encode frames and reconstruct masked future-frame regions through cross-attention. RVM[2]also learns spatial and temporal representations through video reconstruction. Neither is discussed in Section 2 or included in the comparisons. Although W2Rep introduces EMA feature targets, video-context conditioning, and signed temporal offsets, the paper should explain more clearly what these changes contribute beyond prior work.
  2. The video classification advantage depends on the comparison budget. In Table 4, forward-MAC-matched VideoMAE exceeds W2Rep on joint SSv2 (30.46% vs. 25.38%) and UCF101 (59.56% vs. 56.60%). This qualification should be more visible alongside the primary results in Table 1.
  3. The evidence is stronger for recognition than for dense prediction or correspondence. In Table 1(a), W2Rep trails VideoMAE on ADE20K by 3.58 mIoU at ViT-B and 3.51 mIoU at ViT-L, as the authors acknowledge. Figure 3 only provides qualitative cross-frame similarity maps. A quantitative correspondence evaluation, such as VOS, would help assess this ability and support comparison with SiamMAE.

[1] Siamese Masked Autoencoders (NIPS2023)

[2] Recurrent Video Masked Autoencoders(CVPR 2026)

Questions for authors
  1. What are the main differences from SiamMAE and RVM, and which results show the benefit of these differences?
  2. Since pretraining is usually performed once and the encoder can be reused across tasks, could the authors clarify the practical advantage of W2Rep over compute-matched VideoMAE, given the results in Table 4?
4 comments
AuthorsOpenReview ↗

Response to Reviewer pVhH (Part 1: Q1 - SiamMAE & RVM)

Thank you for your constructive review, positive remarks on our results and ablations, and valuable suggestions. We respond in three short parts (1/3: Q1, 2/3: Q2, 3/3: Q3):

Q1: Novelty and empirical comparison with closely related methods (SiamMAE and RVM).

A1: Thanks for your valuable suggestion. While SiamMAE, RVM, and W2Rep all learn from video, significant differences exist in what their encoders share, their backbone architectures, and their learning objectives. We will add a dedicated discussion in Section 2 and the controlled comparison below in Appendix C. The key distinctions are summarized as follows:

1. Different meaning of the "shared encoder" (vs. SiamMAE). SiamMAE shares Siamese weights only between two isolated single frames. In contrast, W2Rep unifies image-level and video-level representations within one encoder, leveraging temporal dynamics in video to improve single-image features while simultaneously learning multi-frame video representations.

2. Different research focus and backbone architecture (vs. RVM). RVM redesigns the visual backbone into a recurrent Transformer-RNN (with causal pixel reconstruction) to process long videos sequentially at linear cost. In contrast, W2Rep retains one standard, unmodified ViT and unifies image and video representation learning in latent space via video context and signed temporal offsets.

More importantly, W2Rep fills the gap between I-JEPA (image-only, unable to model multi-frame video) and V-JEPA (video-focused, weaker image features): within one standard ViT, it both learns strong video-level representations and leverages temporal information in video to learn stronger image-level representations.

3. Controlled same-architecture comparison (SiamJEPA vs. W2Rep). RVM has not released its pretraining code, making controlled reproduction difficult. For SiamMAE, directly comparing its pixel-based model would entangle MAE-vs.-JEPA differences with our method. To isolate our algorithmic design cleanly, we implemented SiamJEPA, running SiamMAE's algorithm on the exact same ViT-B/16 JEPA architecture as W2Rep (100k steps):

Method (Same JEPA Arch.)ImageNetDAVIS VOS (J&F)SSv2 (Video)UCF101 (Video)
SiamJEPA27.7353.211.0346.52
W2Rep (Ours)34.6058.125.3856.60
Gain+6.87+4.9+14.35+10.08

SiamMAE's pairwise algorithm cannot benefit from multi-frame temporal cues (11.03% vs. 25.38% on SSv2), whereas W2Rep substantially improves both video understanding and single-image features (+6.87% on ImageNet, +4.9 on DAVIS VOS).

We appreciate your valuable suggestion, which helps clarify our contribution in Section 2 and Appendix C.

AuthorsOpenReview ↗

Response to Reviewer pVhH (Part 2: Q2 - Compute & Reuse)

Q2: Comparison budget in Table 4 (Appendix B) and practical advantage of reusing one W2Rep model.

A2: Thanks for raising this important question. We respectfully clarify that W2Rep's advantage does not depend on comparison budget: our main comparison in Table 1 (under equal training data exposure) is already completely fair, whereas Table 4 in Appendix B was included solely to test the boundary of our model. That is why we placed Table 4 in Appendix B rather than in the main text, and we will clarify its boundary-test purpose more explicitly in Appendix B of the revised manuscript. The key points are summarized below:

1. Per-step MAC differences come from JEPA vs. MAE, not our algorithm.

Unlike MAE (which has no teacher and encodes only 10% visible patches), every JEPA model requires an EMA teacher to encode 100% unmasked target inputs. Within the JEPA family, W2Rep actually uses fewer MACs per step than V-JEPA.

2. What Table 4 (Appendix B) actually measures.

When matched by forward MACs, VideoMAE sees 8× more video data (799k vs. 100k updates) and consumes 2.5× more GPU hours than W2Rep. Thus, Table 4 in the Appendix was included solely to test our model's boundary under an extreme 8×-data regime rather than as an equal-data comparison.

3. Practical advantage of reusing one W2Rep checkpoint.

At downstream inference, W2Rep has the exact same cost as baselines (one standard ViT). Even against 799k-step VideoMAE (8× data, 2.5× GPU hours), reusing a single 100k-step W2Rep checkpoint still wins on ImageNet (+3.59%), DAVIS VOS (+3.8), and per-frame action probing, directly confirming that our method leverages video temporal cues to strengthen image-level representations. Meanwhile, full video fine-tuning nearly closes the video gap (81.84% vs. 82.43% Top-5):

Method (Pretraining Budget)ImageNetDAVIS VOSSSv2 (Frozen)SSv2 Full FT (Top-1 / Top-5)
I-JEPA (MAC-matched, 520k)28.5052.8--12.23 / 28.43
V-JEPA (MAC-matched, 95k)25.1754.212.8343.65 / 68.58
VideoMAE (Equal-data, 100k)26.7050.824.2755.16 / 79.13
VideoMAE (MAC-matched, 8× data, 2.5× GPU-hr)31.0154.330.4660.16 / 82.43
W2Rep (Equal-data, 1× data, 1× GPU-hr)34.6058.125.3858.77 / 81.84

We appreciate your helpful comment, and will expand this boundary analysis in Appendix B of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer pVhH (Part 3: Q3 - VOS & ADE20K)

Q3: Quantitative correspondence evaluation (DAVIS VOS) and dense prediction on ADE20K.

A3: Thanks for your constructive suggestion to evaluate quantitative correspondence via VOS. Following your advice, we evaluated frozen label propagation on DAVIS-2017 VOS across all models, and will add these results to Section 4.2 of the revised manuscript. The key findings are summarized below:

Strong quantitative correspondence on DAVIS-2017 VOS. W2Rep ranks #1 across all metrics (58.1 J&F) with the lowest temporal decay (14.3), outperforming SiamJEPA (+4.9) as well as VideoMAE under both equal-data (+7.3) and 8×-data MAC-matched (+3.8) budgets:

Method (Pretraining Budget)J&F ↑J-Mean ↑F-Mean ↑J-Decay ↓
I-JEPA (MAC-matched, 520k)52.851.753.918.0
V-JEPA (Equal-data, 100k)48.346.749.821.3
SiamJEPA (Equal-data, 100k)53.252.653.822.0
VideoMAE (Equal-data, 100k)50.850.551.122.6
VideoMAE (MAC-matched, 799k)54.353.854.920.4
W2Rep (Equal-data, 100k)58.157.758.514.3

We believe that the gap between W2Rep and VideoMAE on ADE20K actually stems from architectural differences between the JEPA and MAE paradigms. VideoMAE's edge on ADE20K stems from pixel-level reconstruction rather than temporal representation quality. As analyzed in V-JEPA 2.1 (Section 2.2, Figures 1 & 3, Table 1), supervising only masked latent targets gives unmasked context tokens little incentive to preserve local pixel boundaries. Our W2Rep already achieved the best among all JEPA based methods in Table 1a.

V-JEPA 2.1 shows that closing this dense-prediction gap requires orthogonal add-ons (context loss and multi-layer self-supervision, Section 2.3), which lie outside our single-innovation scope. We included ADE20K to candidly document this boundary for future work.

We sincerely appreciate your suggestions, and will also tighten the exposition across Sections 2–4 in the revised manuscript to make it easier to read.

Dear Reviewer pVhH,

Thank you again for your careful review. As the discussion period is coming to an end, we would be glad to clarify any remaining questions about our response.

Best regards,

The Authors

Reviewer_WmvnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes W2Rep, a self-supervised visual representation learning framework that leverages temporal information to improve image representations while preserving video understanding capabilities. Specifically, W2Rep employs a shared ViT encoder with separate encoding paths for source images and video context, and a conditional predictor to predict masked target features across time. Experiments on ImageNet-1K, ADE20K, SSv2, UCF101, and Diving48 show W2Rep achieves strong performance in image classification and comparable performance in video action recognition.

Critical strengths

1.The motivation of improving image representations by leveraging temporal information in videos is reasonable and meaningful. 2.The overall framework is well designed. W2Rep combines three independent data path, enabling the shared encoder to learn from both spatial and temporal information. 3.The extensive experiments across multiple downstream tasks and ablation studies provide empirical evidence supporting its underlying motivation to some extents.

Critical weaknesses

1.Insufficient comparison with recent unified image-video representation learning methods, such as RVM[1] and TimeBridge[2]. 2.The complementary relationship between same-frame and cross-frame prediction is unclear. As shown in Table 2, the cross-frame-only variant outperform the full W2Rep model on all three action recognition datasets, which raises concerns about whether the two objectives is conflicting. 3.Although W2Rep outperforms VideoMAE under the primary step-matched comparison, Table 4 show they just achive comparable performance under the Forward-MAC-matched setting.

[1]Recurrent Video Masked Autoencoders [2]TimeBridge: Self-Supervised Video Representation Learning via Start-End Joint Embedding and In-Between Frame Prediction

Questions for authors

1.How does W2Rep fundamentally differ from recent video self-supervised learning methods such as RVM and TimeBridge? 2.Why does adding the same-frame prediction objective degrade action recognition performance? Does this indicate a conflict between learning spatial representations and temporal representations? 3.Given that VideoMAE achieves stronger joint video recognition results under the Forward-MAC-matched setting, how should the performance advantages of W2Rep be interpreted?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Wmvn (Part 1: Q1 - RVM & TimeBridge)

Thank you for your constructive review, recognition of our motivation, framework design, and experiments, and valuable suggestions. We respond in three short parts (1/3: Q1, 2/3: Q2, 3/3: Q3):

Q1: Fundamental differences and comparisons with recent methods (RVM and TimeBridge).

A1: Thanks for your valuable suggestion. While RVM, TimeBridge, and W2Rep all learn representations from video, significant differences exist in their retained backbones, encoder inputs, and temporal learning mechanisms. We will add a dedicated discussion in Section 2 and the controlled comparison below in Appendix C of the revised manuscript. The key distinctions are summarized below:

Different backbone architecture and goal (vs. RVM, CVPR'26). RVM redesigns the visual backbone into a recurrent Transformer-RNN (trained with causal pixel reconstruction) to process long videos sequentially at linear cost. In contrast, W2Rep retains one standard, unmodified ViT and unifies single-image and multi-frame video representation learning in latent space.

Different encoder input and temporal modeling (vs. TimeBridge, CVPR'26). TimeBridge extends image-level iBOT: its ViT encodes only isolated start and end frames, reconstructing intermediate pixels through a discarded decoder. Thus, its encoder never jointly models multi-frame video clips and only interpolates between two fixed endpoints. In contrast, W2Rep's encoder processes both single images and multi-frame clips, modeling bidirectional dynamics via video context and signed temporal offsets.

Controlled empirical comparison (via SiamMAE / SiamJEPA). Neither RVM nor TimeBridge has publicly released its pretraining code, making direct reproduction infeasible. However, following another reviewer's suggestion, we reproduced SiamMAE (which shares TimeBridge's core paradigm of encoding isolated frame pairs without joint multi-frame encoding in the retained ViT) on the exact same ViT-B/16 JEPA architecture as W2Rep (SiamJEPA, 100k steps), alongside I-JEPA and V-JEPA:

Method (JEPA Family, ViT-B/16)ImageNetDAVIS VOS (J&F)SSv2 (Video)UCF101 (Video)
I-JEPA28.5052.8----
V-JEPA24.9048.312.6649.23
SiamJEPA (SiamMAE on JEPA arch.)27.7353.211.0346.52
W2Rep (Ours)34.6058.125.3856.60

Pairwise isolated-frame learning (SiamJEPA) fails to capture multi-frame video dynamics and also lags substantially on image transfer. More importantly, W2Rep fills the gap between I-JEPA (image-only, unable to model multi-frame video) and V-JEPA (video-focused, weaker image features): within one standard ViT, it both learns strong video-level representations and leverages temporal information in video to learn stronger image-level representations.

We appreciate your valuable suggestion, and will include this discussion and comparison in Section 2 and Appendix C of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Wmvn (Part 2: Q2 - Same vs. Cross-Frame)

Q2: Relationship between same-frame and cross-frame prediction in Table 2.

A2: Thanks for this insightful question. Spatial and temporal learning do not conflict; rather, they strongly complement each other on spatial representations, while the frozen action gap mainly comes from 25% less effective video-branch budget. To verify this directly, we ran an additional 75k video-budget-matched control and will include it in Section 4.3 of the revised manuscript. The key findings are summarized below:

1. Synergy (not conflict) on spatial representations.

Cross-frame only already outperforms Same-frame only by +4.18% on ImageNet and +0.7 on DAVIS VOS, showing that temporal prediction directly strengthens spatial semantics. Combining both in Final W2Rep (34.60% ImageNet, 22.21 ADE20K, 58.1 DAVIS VOS) surpasses either objective alone, ruling out inherent conflict between the two objectives.

2. 75k video-budget-matched control & remaining SSv2 gap.

Because same-frame prediction (25% of targets, Section 3.3) zeroes out video context, the video branch receives an effective budget of 75k steps. Matched at 75k, Final W2Rep edges out Diving48 and nearly matches UCF101 while boosting ImageNet by +3.58%. For the remaining SSv2 gap, notably, V-JEPA 2.1 (Figure 5 & Table 1) observes the same phenomenon, where adding fine-grained spatial supervision improves image tasks while lowering frozen SSv2. We hypothesize this is because appearance-agnostic SSv2 favors pure motion over fine spatial detail under frozen probing:

VariantSame-Frame ShareEffective Video BudgetImageNetADE20KDAVIS VOSSSv2UCF101Diving48
Same-frame only (100k)100%028.0821.8952.011.2147.837.99
Cross-frame only (100k, Table 2)0%100k32.2621.6152.729.7758.1511.54
Cross-frame only (75k, budget-matched)0%75k31.02----28.5357.2611.13
Final W2Rep (100k, Both)25%75k34.6022.2158.125.3856.6011.29

We appreciate your question and will add this budget-matched control and analysis to Section 4.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Wmvn (Part 3: Q3 - Compute & Table 4)

Q3: Interpreting W2Rep's advantages given Forward-MAC-matched VideoMAE (Table 4, Appendix B).

A3: Thanks for raising this important question. We clarify that our primary comparison in Table 1 (under equal training data exposure) is already the fair comparison, whereas Table 4 in Appendix B was included solely to test the boundary of our model. We will clarify this boundary-test purpose and add the full fine-tuning results below in Appendix B of the revised manuscript. The key points are summarized below:

1. Per-step MAC differences come from JEPA vs. MAE, not our algorithm.

Unlike MAE (which has no teacher and encodes only 10% visible patches), every JEPA model adds an EMA teacher to encode 100% unmasked target inputs. Within the JEPA family, W2Rep actually uses fewer MACs per step than V-JEPA.

2. What Table 4 (Appendix B) actually measures.

Under equal forward MACs, VideoMAE sees 8× more video data (799k vs. 100k updates) and consumes 2.5× more GPU hours than W2Rep. Thus, Table 4 in the Appendix serves solely to test our model's boundary under an asymmetric 8×-data regime rather than as an equal-data comparison.

3. How to interpret W2Rep's practical advantage.

Downstream inference cost is identical (one standard ViT). Even against 799k-step VideoMAE (8× data, 2.5× GPU hours), reusing a single 100k-step W2Rep checkpoint still wins on ImageNet (+3.59%), DAVIS VOS (+3.8), and per-frame action probing, directly confirming that our method leverages video temporal cues to strengthen image-level representations. Meanwhile, full video fine-tuning nearly closes the video gap (81.84% vs. 82.43% Top-5):

Method (Pretraining Budget)ImageNetDAVIS VOSSSv2 (Frozen)SSv2 Full FT (Top-1 / Top-5)
I-JEPA (MAC-matched, 520k)28.5052.8--12.23 / 28.43
V-JEPA (MAC-matched, 95k)25.1754.212.8343.65 / 68.58
VideoMAE (Equal-data, 100k)26.7050.824.2755.16 / 79.13
VideoMAE (MAC-matched, 8× data, 2.5× GPU-hr)31.0154.330.4660.16 / 82.43
W2Rep (Equal-data, 1× data, 1× GPU-hr)34.6058.125.3858.77 / 81.84

We sincerely appreciate your feedback, and will clarify this interpretation in Appendix B and tighten presentation throughout the revised manuscript.

Reviewer_w9axOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a self-supervised method, W2Rep to learn single-image representations using temporal changes observed in video.

During pretraining, a ViT encodes an isolated source frame while also processing masked video context using auxiliary tokens.

A conditional predictor uses source features, video context, spatial queries, and signed temporal offsets to predict target frame features.

After pretraining, the predictor and auxiliary tokens are dropped, leaving an encoder that functions on both single images and full video clips.

Experiments on ImageNet, ADE20K, and action benchmarks show gains in frozen and fine-tuned settings over step-matched baselines, along with temporal ablations.

Critical strengths
  • A single trained backbone performs well on both independent single-frame tasks (like ImageNet classification) and multi-frame video actions. This avoids forcing a choice between an image-only model and a dedicated video-only model.

  • Through extensive ablation experiments (wrong-sign offsets, zeroed latents, and shuffled frames), the paper shows that performance gains come directly from signed time offsets and ordered video context, not unmodeled feature leakage.

Critical weaknesses
  • In Table 4, when VideoMAE receives the same forward-MAC compute budget as W2Rep, VideoMAE outperforms W2Rep on joint SSv2 (30.46% vs. 25.38%) and joint UCF101 (59.56% vs. 56.60%). This makes it hard to understand whether the video-level advantages under the primary 100k-horizon schedule stem from the objective design or simply from higher compute per update.
  • Pretraining is performed exclusively on SSv2. Because SSv2 has strong motion cues and a fixed recording format, it remains unproven whether this dual image/video objective transfers effectively when pretrained on large-scale, uncurated in-the-wild web video (e.g., Kinetics or WebVid).
Questions for authors
  • When VideoMAE is given the same total computation (forward MACs) as W2Rep, it beats W2Rep on all three video benchmarks when representations are frozen (Table 4). Does W2Rep still win on full video fine-tuning (Table 5) if you compare it against this compute-matched VideoMAE model instead of the 100k-step one? Also, how well does V-JEPA perform when matched for the same compute budget?

  • The model uses the exact same box coordinates across all frames, even if the camera moves or objects shift. If the camera moves a lot so that objects in the source frame shift out of view or pop up elsewhere, does the model stop relying on the source image ctc_t and just get its answers from the video context z(V)z(V)? Have you tested how well this method works on videos with heavy camera motion, like Kinetics?

7 comments
AuthorsOpenReview ↗

Response to Reviewer w9ax (Part 1: Q1 - Compute & Fine-Tuning)

Thank you for your thoughtful review, recognition of our unified single-backbone performance and temporal ablations, and valuable questions. We respond in two short parts (1: Q1, 2: Q2):

Q1: Objective design vs. per-update compute (Table 4, Appendix B), full fine-tuning against compute-matched VideoMAE, and compute-matched V-JEPA.

A1: Thanks for your thoughtful questions. W2Rep's video gains stem from objective design rather than higher compute per update: Table 1 (equal training data exposure) is already the fair comparison, whereas Table 4 in Appendix B was included solely to test the boundary of our model. We will add the full fine-tuning and compute comparisons below to Appendix B of the revised manuscript. The key results are summarized as follows:

1. JEPA vs. MAE architecture

Unlike MAE (which has no teacher and encodes only 10% visible patches), every JEPA model adds an EMA teacher to encode 100% unmasked target inputs, so per-step MAC differences come from the JEPA vs. MAE paradigm rather than our algorithm. In fact, V-JEPA uses ~5% more MACs per step than our W2Rep.

2. Full video fine-tuning

At equal forward MACs, VideoMAE sees 8× more video data (799k vs. 100k steps) and consumes 2.5× more GPU hours than W2Rep, making Table 4 in the Appendix an extreme boundary test rather than an equal-data comparison. Even against this 8×-data checkpoint, 50-epoch full fine-tuning (Table 5 recipe) nearly closes the video gap (81.84% vs. 82.43% Top-5), while W2Rep still wins on ImageNet (+3.59%) and DAVIS VOS (+3.8), which directly confirms that our method leverages video temporal cues to strengthen image-level representations:

Method (Pretraining Budget)ImageNetDAVIS VOSSSv2 (Frozen)SSv2 Full FT (Top-1 / Top-5)
I-JEPA (MAC-matched, 520k)28.5052.8--12.23 / 28.43
V-JEPA (MAC-matched, 95k)25.1754.212.8343.65 / 68.58
VideoMAE (Equal-data, 100k)26.7050.824.2755.16 / 79.13
VideoMAE (MAC-matched, 8× data, 2.5× GPU-hr)31.0154.330.4660.16 / 82.43
W2Rep (Equal-data, 1× data, 1× GPU-hr)34.6058.125.3858.77 / 81.84

We appreciate your valuable questions, and will add these results and clarifications to Appendix B of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer w9ax (Part 2: Q2 - Camera Motion & VOS)

Q2: Shared query coordinates under heavy camera motion and transfer to in-the-wild videos.

A2: Thanks for raising this insightful question. We evaluated heavy-motion correspondence on DAVIS-2017 VOS and compared camera-motion magnitudes between SSv2 and Kinetics, and will add these results, visual comparisons, and a dedicated discussion to Section 4 and the Appendix of the revised manuscript. Our response is summarized below:

1. Preventing context-copying shortcuts while modeling continuous motion. Extreme view shifts also challenge similar methods (e.g., SiamMAE), which can only blindly guess unseen regions when two frames diverge. Instead, W2Rep models the clip as a continuous motion process, with three explicit designs (Section 3) preventing the model from trivially copying video context and bypassing the source image:

  • Tube masking & discarded video patches: Target boxes are masked across the clip, and dense video patch tokens are discarded.
  • Compact 16-token bottleneck: Video context is compressed into just 16 position-free tokens with norm regularization, which is far too small to copy dense frame features.
  • Target-blind video pass & relative offsets: The video branch is not told which frame is the source or target, and the predictor receives only a relative time offset (not a frame index), so it must read the source image to anchor moment t before stepping to t + Δ.

2. Moderate content context in video context is beneficial by design.

On the other hand, we do not claim or constrain video context to be strictly disentangled from visual content. It naturally carries moderate content context alongside camera and object motion. In fact, handling the exact scenario you raised (where objects shift out of view or new regions appear) is part of why we designed video context this way, as it avoids ill-posed blind guessing from the source image alone.

3. Heavy camera motion: SSv2 vs. Kinetics and DAVIS-2017 VOS.

Comparing SSv2 with Kinetics, we found that handheld crowdsourced SSv2 already contains substantial data with comparable camera motion to Kinetics (we will add visual comparisons to the Appendix). Together with W2Rep ranking #1 overall (58.1 J&F) on real-world heavy-motion DAVIS-2017 VOS below, along with consistent gains on YouTube sports/diving videos (UCF101 & Diving48), our experiments confirm strong performance under heavy camera motion despite our compute limits preventing full Kinetics pretraining:

Method (Pretraining Budget)J&F ↑J-Mean ↑F-Mean ↑J-Decay ↓
I-JEPA (MAC-matched, 520k)52.851.753.918.0
V-JEPA (Equal-data, 100k)48.346.749.821.3
VideoMAE (Equal-data, 100k)50.850.551.122.6
VideoMAE (MAC-matched, 799k)54.353.854.920.4
W2Rep (Equal-data, 100k)58.157.758.514.3

Still, how to learn even more effectively under wild camera motion and large-scale uncurated web video remains a valuable direction for future work. We sincerely appreciate your suggestion and will include this discussion in the revised manuscript.

In Part 2, you state that SSv2 contains substantial data with camera motion comparable to Kinetics, and argue by architecture design that the predictor still relies on the source image rather than z(V) under heavy motion. Do you have quantitative measurements supporting either point, or are both currently qualitative/design-level arguments?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for AC

Dear Area Chair,

We sincerely thank you for the time and effort you have devoted to handling our submission, and we are grateful to Reviewers w9ax, Wmvn, and pVhH for their valuable feedback. We have thoroughly clarified and empirically answered every question and concern raised by all three reviewers, and after two rounds of active discussion, Reviewer w9ax confirmed having no remaining major concerns. Below is a concise summary for your meta-review:

Core Significance and Value of W2Rep

Existing visual self-supervision is split across image-only (I-JEPA, unable to model multi-frame video), video-focused (V-JEPA and VideoMAE, weaker single-image features), and isolated frame-pair (SiamMAE and TimeBridge, no multi-frame modeling in the retained backbone) paradigms. W2Rep bridges this gap within one unmodified ViT:

  • Scientific significance: Watching how a scene changes over time provides strong supervision for single-image representations and cross-frame correspondence, while simultaneously training the same attention layers for multi-frame video reasoning.
  • Practical value: Discarding all auxiliary modules after pretraining leaves one drop-in standard ViT with zero additional inference overhead that transfers effectively to both single-image and multi-frame video tasks.

Resolution of All Reviewer Concerns and Questions

1. Related Works and Same-Architecture Comparison (Reviewers pVhH & Wmvn)

We clarified how W2Rep's unified single-ViT design differs from isolated two-frame (SiamMAE, TimeBridge) and recurrent (RVM) backbones. Through a controlled same-architecture comparison (implementing SiamMAE's pairwise algorithm on the exact same JEPA architecture), we further showed that pairwise frame prediction (like SiamMAE) fails to benefit from multi-frame video inputs, whereas W2Rep gains substantially from multi-frame temporal cues while also learning stronger single-image and correspondence representations across all benchmarks.

2. Correspondence (DAVIS-2017 VOS), ADE20K, and Heavy Camera Motion (Reviewers pVhH & w9ax)

We added frozen label-propagation experiments on DAVIS-2017 video object segmentation (VOS), where W2Rep ranks #1 overall with the lowest temporal decay across all baselines, and clarified why the ADE20K segmentation gap reflects the general JEPA-vs.-MAE pixel-supervision boundary analyzed in V-JEPA 2.1. We also quantitatively verified both W2Rep's effectiveness and its sustained reliance on the source image under heavy camera motion, resolving Reviewer w9ax's follow-up questions.

3. Fairness of the Main Comparison, Boundary Test, and Single-Model Reuse (Reviewers w9ax, Wmvn, & pVhH)

Our main comparison under equal data exposure is already the fair comparison: per-step MAC differences come from the JEPA-vs.-MAE architecture (an EMA target encoder vs. no target encoder) rather than from our algorithm, and W2Rep also outperforms compute-matched V-JEPA across all image and video tasks. Compute-matched VideoMAE trains on ~8× more video clips at equal forward MACs and was included in the Appendix to test the boundary of our model: while it achieves higher joint video accuracy, reusing one W2Rep checkpoint still clearly leads on single-image recognition, per-frame probing, and cross-frame correspondence while retaining competitive end-to-end video fine-tuning performance, and we also tightened our main-text claims accordingly.

4. Synergy Between Same-Frame and Cross-Frame Prediction (Reviewer Wmvn)

We showed that combining same-frame and cross-frame prediction improves spatial classification, segmentation, and correspondence over either objective alone, and verified via a matched video-branch budget control that the two objectives do not conflict (with the remaining gap under frozen probing on SSv2 mirroring a similar phenomenon observed in V-JEPA 2.1, where adding fine-grained spatial supervision improves frozen image transfer while lowering frozen SSv2 probing accuracy).

All reviewer questions have been fully addressed, and all corresponding experiments and clarifications have been synchronized into our updated manuscript. Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.