- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces VANE, a self-supervised image and video encoder that learns visual representations by predicting the next frame in a pretrained feature space. It trains a causal vision transformer to encode past frames and a predictor to estimate the next frame’s representation. It uses frozen DINOv2 features as targets for the predictor. An auxiliary generative decoder also predicts the next frame’s pixels, and a local-to-global crop objective extends the approach to images as well. The authors train a 1B model on a large collection of 145M images and 24M videos, reporting strong results across action anticipation, video recognition, image classification, and dense perception; e.g., VANE outperforms V-JEPA 2.1 on several benchmarks, with notable gains on action anticipation.
Critical strengths
-
The pretraining objective is clean, simple, and effective.
-
VANE performs well as a frozen encoder across action anticipation, video recognition, image classification, and dense perception.
-
The paper explains the method and its motivation clearly. The architecture figures make the attention mechanism easy to follow.
-
The paper provides substantial detail on the architecture, training hyperparameters, masking strategies, data mixture, and compute requirements. This makes the method easier to understand and provides a useful starting point for reproducing the experiments.
Critical weaknesses
-
The paper's visual world modeling claim is not sufficiently supported. EK100 and Ego4D STA evaluate frozen encoder features with supervised probes, not the model’s ability to predict future states. In fact the predictor is discarded after pretraining, and no future forecasting, multi-step prediction, or planning is evaluated. To justify its framing as a world model, VANE should evaluate its predictor on dense future forecasting (like future semantic segmentation as in DINO-Foresight and DINO-world) and/or action-conditioned planning using a predictor trained on frozen VANE features (like V-JEPA 2-AC or DINO-WM). Without such experiments, VANE is better described as a predictive pretraining method for visual representation learning.
-
It is unclear to me how much VANE actually learns from its current future prediction. For most cases, DINOv2 features of consecutive frames are highly similar. A useful ablation would be to predict the current frame’s DINOv2 features instead of the next frame’s and compare the resulting representations. I would also expect a method that learns meaningful temporal dynamics to show larger gains over V-JEPA 2.1 on motion-focused benchmarks such as SSv2. However, the comparison shows very close performance, leaving the contribution of temporal prediction unclear.
-
Prior works like SiamMAE (Gupta et al. 2023) and FRAME (TV et al. 2025) have already explored predicting future visual information from past frames. Similarly, CropMAE (Eymaël et al. 2024) has explored reconstruction across image crops. The related work section should discuss these works and clarify what is new beyond the proposed combination of components and its larger training scale. Comparing VANE with SiamMAE and FRAME on visual correspondence would further improve its positioning as a predictive pretraining method for visual representation learning.
Questions for authors
-
If authors want to present VANE as a world model: How does VANE compare to V-JEPA-AC and DINO-WM on action conditioning? How does it compare with DINO-world and DINO-Foresight on dense future forecasting? If they want to position it as predictive pretraining for vision encoder: How does VANE compare to SiamMAE and FRAME on tasks like visual correspondence and action classification?
-
Is causal encoder and temporal modeling contributing to the performance? How much? Why are the gains on SSv2 modest compared to V-JEPA?
-
Is joint image-video training necessary to achieve the reported gains? The paper presents it as a step toward unifying visual pretraining, but does not ablate image-only vs video-only vs joint image-video training.
I am willing to raise my score if the authors adequately address these questions.
3 comments
Response to Reviewer TgzR
Q3. Joint image–video training.
We compare image-only, video-only, and joint image–video pretraining:
| Pretraining data | IN1K ↑ | SSv2 ↑ | ADE20K ↑ |
|---|---|---|---|
| Images only | 78.6 | 60.8 | 43.5 |
| Videos only | 82.2 | 70.1 | 43.2 |
| Images + Videos | 84.5 | 72.3 | 45.1 |
Joint training improves IN1K, SSv2, and ADE20K over video-only training by 2.3, 2.2, and 1.9 points, respectively. It also improves SSv2 over image-only training by 11.5 points. These results support the complementary benefits of image and video data for learning unified spatial and temporal representations.
W2. Next-frame prediction versus current-frame learning.
Following the reviewer’s suggestion, we compare the two objectives:
| Training objective | IN1K ↑ | SSv2 ↑ | ADE20K ↑ |
|---|---|---|---|
| DINOv2 ViT-g | 85.9 | 50.7 | 49.5 |
| Current-frame prediction | 84.8 | 59.7 | 48.2 |
| Next-frame prediction — VANE | 87.5 | 75.9 | 51.0 |
Next-frame prediction improves IN1K, SSv2, and ADE20K by 2.7, 24.2, and 2.8 points, respectively. Current-frame learning primarily distills/reconstructs teacher features of the observed input and remains below the DINOv2 reference on IN1K and ADE20K. The substantial SSv2 improvement supports the importance of predicting unobserved future content.
We apply temporally shared tube masking to historical frames to reduce copying shortcuts caused by redundancy between consecutive video frames. Masked spatial locations remain unavailable throughout the history, while the complete future frame is supervised. Moreover, DINOv2 features retain motion-related variation. On random sampling SSv2 videos, mean same-location patch cosine similarity is 0.628. An optical-flow-based diagnostic yields a motion-region AUROC of 0.655 [0.611, 0.698]. These diagnostics show that the features contain motion-related changes.
W3. Positioning relative to prior predictive pretraining methods.
We clarify the methodological differences from SiamMAE and FRAME and compare their representation quality on SSv2 and DAVIS-2017 in W1 / Q1, part 3.
Response to Reviewer TgzR
Dear Reviewer,
Thank you again for your thoughtful feedback. We would like to kindly follow up on our responses and the additional experiments we provided in response to your suggestions. These include action-conditioned planning and future semantic segmentation evaluations, next-frame versus current-frame prediction ablations, comparisons with SiamMAE and FRAME, and image-only/video-only/joint-training ablations.
We would greatly appreciate your feedback on whether these results address your main concerns. Please let us know if any questions remain; we would be happy to provide further clarification.
Thank you for your time and consideration.
Response to Reviewer TgzR
We thank the reviewer for the positive assessment of our objective, presentation, and broad evaluation. We address the main concerns below.
W1 / Q1. Comparisons on action conditioning, dense future forecasting, and visual representation learning.
VANE is a representation learning framework for visual world modeling. Here, visual world modeling encompasses representing and understanding the current video state as well as predicting its future evolution. We add experiments to assess world-modeling capabilities through planning and forecasting, and representation quality through action classification and visual correspondence.
1. Comparison with V-JEPA 2-AC and DINO-WM on action conditioning.
Action-conditioned planning. We currently lack access to the robotic hardware required to reproduce the real-world manipulation evaluation of V-JEPA 2-AC. We compare with DINO-WM on simulated control tasks by training an action-conditioned predictor on frozen VANE features and evaluating goal-reaching performance under matched data and planning settings.
SR: success rate (%). CD: Chamfer Distance.
VANE improves over DINO-WM on four tasks, matches it on PointMaze, and performs slightly worse on Reacher. These results show that representations learned through next-frame prediction transfer effectively to action-conditioned planning.
Predictor-assisted anticipation. We additionally evaluate Ego4D short-term action anticipation using predicted future features alongside encoder features:
Adding the jointly trained predictor improves mAP All by 0.09 points, suggesting that explicit future-state predictions complement the anticipation cues already learned by the encoder through next-frame pretraining.
2. Comparison with DINO-world and DINO-Foresight on dense future forecasting.
We evaluate future semantic segmentation on Cityscapes at short-term and mid-term horizons. VANE is pretrained to predict the next frame’s features at its pretraining sampling interval, whereas these benchmarks evaluate predictions at multiple future horizons. We therefore fine-tune VANE's predictor for the benchmark’s temporal intervals and evaluate at the same context and target timestamps.
The gains at both horizons are consistent with next-frame prediction capturing temporal evolution and generative reconstruction preserving spatial detail, together supporting the prediction of future scene content and layout.
3. Comparison with SiamMAE and FRAME.
SiamMAE reconstructs masked future-frame pixels from an unmasked past frame and visible future patches. FRAME first distills a frame encoder from image teachers, then freezes it by default to train temporal memory and prediction modules. VANE formulates visual pretraining as next-frame prediction in embedding space. It learns representations of the current state through future prediction, jointly training a causal encoder and predictor.
We evaluate motion understanding on SSv2 and visual correspondence on DAVIS-2017.
VANE outperforms SiamMAE and FRAME on both SSv2 and DAVIS, consistent with the benefits of learning temporal dynamics through next-frame prediction and preserving spatial detail through generative reconstruction.
Q2. Temporal modeling and the modest SSv2 gain.
V-JEPA 2.1 is a strong state-of-the-art video representation baseline. Our classification evaluation uses a four-layer attentive probe trained on frozen encoder features. V-JEPA uses a bidirectional encoder with bidirectional temporal attention, allowing each token to incorporate both earlier and later observations within the clip. This provides richer temporal context for individual token representations and can benefit offline action classification with attentive probing.
VANE instead uses a causal encoder, where each token can access only the current and preceding observations. This restriction supports future prediction and incremental encoding, but limits the temporal context available to individual encoder tokens. Despite this constraint, VANE surpasses V-JEPA 2.1 on SSv2 under our matched evaluation protocol. The modest gain should therefore be considered alongside this architectural difference and VANE’s stronger improvements on anticipation tasks.