acceptodds
Withdrawn from ICLR 2027

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video Representation Learning, Joint Embedding Predictive Architectures, Generative Pretraining, Self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_TgzROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VANE, a self-supervised image and video encoder that learns visual representations by predicting the next frame in a pretrained feature space. It trains a causal vision transformer to encode past frames and a predictor to estimate the next frame’s representation. It uses frozen DINOv2 features as targets for the predictor. An auxiliary generative decoder also predicts the next frame’s pixels, and a local-to-global crop objective extends the approach to images as well. The authors train a 1B model on a large collection of 145M images and 24M videos, reporting strong results across action anticipation, video recognition, image classification, and dense perception; e.g., VANE outperforms V-JEPA 2.1 on several benchmarks, with notable gains on action anticipation.

Critical strengths
  • The pretraining objective is clean, simple, and effective.

  • VANE performs well as a frozen encoder across action anticipation, video recognition, image classification, and dense perception.

  • The paper explains the method and its motivation clearly. The architecture figures make the attention mechanism easy to follow.

  • The paper provides substantial detail on the architecture, training hyperparameters, masking strategies, data mixture, and compute requirements. This makes the method easier to understand and provides a useful starting point for reproducing the experiments.

Critical weaknesses
  • The paper's visual world modeling claim is not sufficiently supported. EK100 and Ego4D STA evaluate frozen encoder features with supervised probes, not the model’s ability to predict future states. In fact the predictor is discarded after pretraining, and no future forecasting, multi-step prediction, or planning is evaluated. To justify its framing as a world model, VANE should evaluate its predictor on dense future forecasting (like future semantic segmentation as in DINO-Foresight and DINO-world) and/or action-conditioned planning using a predictor trained on frozen VANE features (like V-JEPA 2-AC or DINO-WM). Without such experiments, VANE is better described as a predictive pretraining method for visual representation learning.

  • It is unclear to me how much VANE actually learns from its current future prediction. For most cases, DINOv2 features of consecutive frames are highly similar. A useful ablation would be to predict the current frame’s DINOv2 features instead of the next frame’s and compare the resulting representations. I would also expect a method that learns meaningful temporal dynamics to show larger gains over V-JEPA 2.1 on motion-focused benchmarks such as SSv2. However, the comparison shows very close performance, leaving the contribution of temporal prediction unclear.

  • Prior works like SiamMAE (Gupta et al. 2023) and FRAME (TV et al. 2025) have already explored predicting future visual information from past frames. Similarly, CropMAE (Eymaël et al. 2024) has explored reconstruction across image crops. The related work section should discuss these works and clarify what is new beyond the proposed combination of components and its larger training scale. Comparing VANE with SiamMAE and FRAME on visual correspondence would further improve its positioning as a predictive pretraining method for visual representation learning.

Questions for authors
  • If authors want to present VANE as a world model: How does VANE compare to V-JEPA-AC and DINO-WM on action conditioning? How does it compare with DINO-world and DINO-Foresight on dense future forecasting? If they want to position it as predictive pretraining for vision encoder: How does VANE compare to SiamMAE and FRAME on tasks like visual correspondence and action classification?

  • Is causal encoder and temporal modeling contributing to the performance? How much? Why are the gains on SSv2 modest compared to V-JEPA?

  • Is joint image-video training necessary to achieve the reported gains? The paper presents it as a step toward unifying visual pretraining, but does not ablate image-only vs video-only vs joint image-video training.

I am willing to raise my score if the authors adequately address these questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TgzR

We thank the reviewer for the positive assessment of our objective, presentation, and broad evaluation. We address the main concerns below.

W1 / Q1. Comparisons on action conditioning, dense future forecasting, and visual representation learning.

VANE is a representation learning framework for visual world modeling. Here, visual world modeling encompasses representing and understanding the current video state as well as predicting its future evolution. We add experiments to assess world-modeling capabilities through planning and forecasting, and representation quality through action classification and visual correspondence.

1. Comparison with V-JEPA 2-AC and DINO-WM on action conditioning.

Action-conditioned planning. We currently lack access to the robotic hardware required to reproduce the real-world manipulation evaluation of V-JEPA 2-AC. We compare with DINO-WM on simulated control tasks by training an action-conditioned predictor on frozen VANE features and evaluating goal-reaching performance under matched data and planning settings.

MethodPointMaze SR ↑Wall SR ↑Reacher SR ↑PushT SR ↑Rope CD ↓Granular CD ↓
DINO-WM98.096.092.090.00.410.26
VANE98.098.090.094.00.370.21

SR: success rate (%). CD: Chamfer Distance.

VANE improves over DINO-WM on four tasks, matches it on PointMaze, and performs slightly worse on Reacher. These results show that representations learned through next-frame prediction transfer effectively to action-conditioned planning.

Predictor-assisted anticipation. We additionally evaluate Ego4D short-term action anticipation using predicted future features alongside encoder features:

VANE configurationEgo4D mAP All ↑
Encoder only7.81
Encoder + predictor7.90

Adding the jointly trained predictor improves mAP All by 0.09 points, suggesting that explicit future-state predictions complement the anticipation cues already learned by the encoder through next-frame pretraining.

2. Comparison with DINO-world and DINO-Foresight on dense future forecasting.

We evaluate future semantic segmentation on Cityscapes at short-term and mid-term horizons. VANE is pretrained to predict the next frame’s features at its pretraining sampling interval, whereas these benchmarks evaluate predictions at multiple future horizons. We therefore fine-tune VANE's predictor for the benchmark’s temporal intervals and evaluate at the same context and target timestamps.

MethodShort-term mIoU ↑ (3 frames)Mid-term mIoU ↑ (9 frames)
DINO-Foresight64.557.2
DINO-world64.755.1
VANE67.959.2

The gains at both horizons are consistent with next-frame prediction capturing temporal evolution and generative reconstruction preserving spatial detail, together supporting the prediction of future scene content and layout.

3. Comparison with SiamMAE and FRAME.

SiamMAE reconstructs masked future-frame pixels from an unmasked past frame and visible future patches. FRAME first distills a frame encoder from image teachers, then freezes it by default to train temporal memory and prediction modules. VANE formulates visual pretraining as next-frame prediction in embedding space. It learns representations of the current state through future prediction, jointly training a causal encoder and predictor.

We evaluate motion understanding on SSv2 and visual correspondence on DAVIS-2017.

MethodSSv2 ↑DAVIS J&F ↑
SiamMAE40.062.0
FRAME49.366.3
V-JEPA 2.175.368.1
VANE75.968.8

VANE outperforms SiamMAE and FRAME on both SSv2 and DAVIS, consistent with the benefits of learning temporal dynamics through next-frame prediction and preserving spatial detail through generative reconstruction.

Q2. Temporal modeling and the modest SSv2 gain.

V-JEPA 2.1 is a strong state-of-the-art video representation baseline. Our classification evaluation uses a four-layer attentive probe trained on frozen encoder features. V-JEPA uses a bidirectional encoder with bidirectional temporal attention, allowing each token to incorporate both earlier and later observations within the clip. This provides richer temporal context for individual token representations and can benefit offline action classification with attentive probing.

VANE instead uses a causal encoder, where each token can access only the current and preceding observations. This restriction supports future prediction and incremental encoding, but limits the temporal context available to individual encoder tokens. Despite this constraint, VANE surpasses V-JEPA 2.1 on SSv2 under our matched evaluation protocol. The modest gain should therefore be considered alongside this architectural difference and VANE’s stronger improvements on anticipation tasks.

AuthorsOpenReview ↗

Response to Reviewer TgzR

Q3. Joint image–video training.

We compare image-only, video-only, and joint image–video pretraining:

Pretraining dataIN1K ↑SSv2 ↑ADE20K ↑
Images only78.660.843.5
Videos only82.270.143.2
Images + Videos84.572.345.1

Joint training improves IN1K, SSv2, and ADE20K over video-only training by 2.3, 2.2, and 1.9 points, respectively. It also improves SSv2 over image-only training by 11.5 points. These results support the complementary benefits of image and video data for learning unified spatial and temporal representations.

W2. Next-frame prediction versus current-frame learning.

Following the reviewer’s suggestion, we compare the two objectives:

Training objectiveIN1K ↑SSv2 ↑ADE20K ↑
DINOv2 ViT-g85.950.749.5
Current-frame prediction84.859.748.2
Next-frame prediction — VANE87.575.951.0

Next-frame prediction improves IN1K, SSv2, and ADE20K by 2.7, 24.2, and 2.8 points, respectively. Current-frame learning primarily distills/reconstructs teacher features of the observed input and remains below the DINOv2 reference on IN1K and ADE20K. The substantial SSv2 improvement supports the importance of predicting unobserved future content.

We apply temporally shared tube masking to historical frames to reduce copying shortcuts caused by redundancy between consecutive video frames. Masked spatial locations remain unavailable throughout the history, while the complete future frame is supervised. Moreover, DINOv2 features retain motion-related variation. On random sampling SSv2 videos, mean same-location patch cosine similarity is 0.628. An optical-flow-based diagnostic yields a motion-region AUROC of 0.655 [0.611, 0.698]. These diagnostics show that the features contain motion-related changes.

W3. Positioning relative to prior predictive pretraining methods.

We clarify the methodological differences from SiamMAE and FRAME and compare their representation quality on SSv2 and DAVIS-2017 in W1 / Q1, part 3.

AuthorsOpenReview ↗

Response to Reviewer TgzR

Dear Reviewer,

Thank you again for your thoughtful feedback. We would like to kindly follow up on our responses and the additional experiments we provided in response to your suggestions. These include action-conditioned planning and future semantic segmentation evaluations, next-frame versus current-frame prediction ablations, comparisons with SiamMAE and FRAME, and image-only/video-only/joint-training ablations.

We would greatly appreciate your feedback on whether these results address your main concerns. Please let us know if any questions remain; we would be happy to provide further clarification.

Thank you for your time and consideration.

Reviewer_ntCvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper summary: The overall idea of the paper is about training an image-video encoder that uses next frame prediction as its objective where prediction takes place in a frozen semantic embedding space (specifically DINOv2). It uses a causal encoder that takes as input a set of context frames and produces an encoding representation for each one of them. These encoded frames are used by a predictor model to predict the frozen semantic tokens of the next frame. In parallel, the predictor output is also used as conditioning for a diffusion-based generative decoder to reconstruct the next frame. For image encoding they treat images as local to global crops to apply the same idea.

Critical strengths

Positive points:

  • Paper is well-written and easy to read
  • The idea of using local-to-global crops for images is interesting and provides a unified approach for images and video
  • Good experimental results (particularly action anticipation EPIC-KITCHENS-100, object-interaction anticipation on Ego4D STA v2, and dense perception results, e.g., depth)
Critical weaknesses

Negative points:

  • The proposed vision encoder already relies on a strong pre-existing vision encoder (DINOv2). This raises the question of how fair the comparison is wrt vision encoders like V-JEPA that do not make use of such a strong teacher. There is only a small comparison in the appendix Table 9 that compares with V-JEPA when using the same teacher but it is for a ViT-L model and shows no anticipation results. Moreover, no comparison with other methods that make use of a frozen teacher is included (e.g. Wang et al., CVPR 2026 - InternVideo-Next).
  • The paper emphasizes in its intro and motivation the idea of next frame prediction vs masked completion but at the end of the day it also uses masked context similar to V-JEPA (appendix C3). This also raises concerns about how novel the proposed method really is.
  • The paper also talks about next frame prediction improving world modeling but the trained predictor is thrown away and is never evaluated about how accurate it is at test time.
  • For training their model the authors use their own dataset (called VISTA), which is curated to contain data that matches the distribution of the training data for the target evaluation tasks. There is only a small scale comparison at matched pretraining in table 9 but this does not include any anticipation results as already noted above (it is thus unclear if the large anticipation gains persist in this case). This raises the question whether the big performance improvements come mainly from this dataset choice. A fully matched 1B comparison is missing.
Questions for authors

The authors should convincingly address the issues raised in the weakness section

1 comment
AuthorsOpenReview ↗

Response to Reviewer ntCv

We thank the reviewer for recognizing the clear presentation, unified local-to-global formulation, and strong anticipation and dense perception results. We jointly address the teacher and data concerns in W1 and W4, followed by context masking and predictor evaluation.

W1 & W4. Teacher and data fairness.

Teacher and data. VANE trains its encoder and predictor from scratch, with frozen DINOv2 providing prediction space. Since V-JEPA 2.1's full corpus is unavailable, we independently construct VISTA following its curation approach, using only public training splits for pretraining and retrieval.

Controlled 1B anticipation. Both methods use ViT-g (1B) encoders pretrained on VISTA for 50K iterations, with the same downstream evaluation protocol:

MethodSemantic targetEK100 action mean-class recall@5 ↑Ego4D STA v2 mAP All ↑
V-JEPA 2.1EMA34.75.88
V-JEPA 2.1Frozen DINOv2 ViT-L33.25.99
VANEFrozen DINOv2 ViT-L42.36.98

With the same DINOv2 teacher, VANE gains 9.1 / 0.99 points on EK100/Ego4D (7.6 / 1.10 over the EMA baseline). Thus, anticipation gains persist with data, teacher, encoder scale, and training iterations matched. Existing ViT-L controls (Table 9 and Table 5(c)) also show recognition and segmentation gains under matched data, targets, and iterations, without a generative decoder in either method.

Comparison with InternVideo-Next. We add the requested comparison with this frozen-teacher method. InternVideo-Next uses a frozen SigLIP2 teacher for feature distillation in Stage 1, followed by masked prediction with a frozen Stage-1 teacher. VANE instead predicts unseen future features in DINOv2 space, rather than distilling features of observed content.

MethodEK100 action mean-class recall@5 ↑SSv2 top-1 ↑
InternVideo-Next34.073.0
VANE45.575.9

VANE improves EK100 action anticipation by 11.5 points and SSv2 action recognition by 2.9 points, supporting the effectiveness of next-frame prediction for learning transferable temporal representations.

W2. Context masking and the distinction from masked completion.

We apply tube masking to historical frames (Appendix C.3) to increase prediction difficulty and reduce copying shortcuts from redundant historical observations. VANE still predicts the complete unseen next frame using only past observations, whereas V-JEPA recovers masked content from bidirectional clip context.

W3. Evaluating the predictor and the world-modeling claim.

The main experiments evaluate frozen encoder features to measure the representations learned through future prediction. We agree that assessing the predictor itself provides complementary evidence, and include two evaluations that use its outputs.

Predictor-assisted anticipation. We evaluate Ego4D STA v2 using encoder features alone or together with predicted future features:

VANE configurationEgo4D mAP All ↑
Encoder only7.81
Encoder + predictor7.90

Adding the predictor improves mAP All by 0.09 points, showing a modest additional benefit from predicted future features at inference.

Dense future forecasting. VANE is pretrained to predict the next frame at its pretraining sampling interval, which differs from the short- and mid-term forecasting horizons in Cityscapes. We therefore fine-tune the predictor to adapt to these temporal intervals and evaluate at the same context and target timestamps as the baselines.

MethodShort-term mIoU ↑ (3 frames)Mid-term mIoU ↑ (9 frames)
DINO-Foresight64.557.2
DINO-world64.755.1
VANE67.959.2

VANE improves over DINO-Foresight by 3.4 / 2.0 mIoU and DINO-world by 3.2 / 4.1 mIoU. The gains at both horizons are consistent with next-frame prediction capturing temporal evolution and generative reconstruction preserving spatial detail for dense future forecasting. These results evaluate the usefulness of predicted future features after benchmark adaptation.

Reviewer_H5WfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Masked video pretraining only recovers missing content rather than predicting the future, while autoregressive methods predict token by token and lag in representation quality. To address this, the paper proposes VANE, which predicts the entire next frame in a semantic embedding space. A frame-causal encoder and a context-isolated predictor regress the features of a frozen DINOv2 encoder, with an auxiliary flow-matching pixel decoder, and images are handled as local-to-global crop sequences, which yields a single image–video encoder. With a 1B encoder, VANE outperforms V-JEPA 2.1 on most anticipation, classification, and dense benchmarks, including 7.1% gain on EK100 action anticipation, and scales with model and data size.

Critical strengths
  • Clear motivation and simple formulation. The paper motivates frame-level prediction in embedding space well. The resulting design is simple and coherent, and extending it to still images through local-to-global crops is a natural way to unify image and video pretraining under a single objective.
  • Informative ablations. The ablations cover the key design choices: prediction granularity, target space, image objectives, and predictor attention. The comparison with V-JEPA 2.1 trained on the same data and with the same target is particularly helpful for isolating the effect of the objective.
Critical weaknesses
  • Limited conceptual novelty. Predicting next-frame embeddings from past observations has been explored in recent latent world models such as DINO-WM, DINO-world, and DINO-Foresight. The paper distinguishes VANE from these works by noting that it trains the encoder itself rather than learning dynamics on top of a fixed encoder, which is a fair distinction. That said, LeWM appears closely related, since it also trains a JEPA end-to-end from pixels with next-embedding prediction, and it would be helpful if the authors could discuss it and clarify how VANE differs. A clearer positioning against this line of work would help readers better understand the specific contribution of VANE.
  • Dependence on a pretrained teacher. One aspect I found somewhat unfortunate is that the prediction targets come from a frozen, separately pretrained DINOv2 ViT-L, whereas the main baselines, V-JEPA 2 and V-JEPA 2.1, require no such external model and rely only on an EMA target encoder. This makes the comparison less direct. It also brings the approach somewhat closer to feature distillation onto future frames than the self-supervised framing suggests, and the reported compute does not account for pretraining this teacher. Given the use of such a strong teacher, the gains on recognition and dense perception over V-JEPA 2.1 also seem relatively modest. Moreover, when the teacher is replaced with an online EMA target, VANE appears to fall below V-JEPA 2.1 under the matched setting (Table 5a vs. Table 9). This suggests that its advantage may depend in part on the teacher. It would be helpful if the authors could include DINOv2 ViT-L itself as a baseline and discuss this dependence.
Questions for authors
  • How does VANE differ from LeWM, which also learns next-embedding prediction end-to-end from pixels?
  • Decoder in the matched comparison. Does V-JEPA 2.1 also benefit from the same auxiliary generative decoder? This would help separate the effect of the decoder from that of next-frame prediction, particularly on dense tasks.
3 comments
AuthorsOpenReview ↗

Response to Reviewer H5Wf

We thank the reviewer for recognizing the motivation, unified image–video formulation, and controlled ablations. We address the main concerns below.

W1 / Q1. Methodological novelty and comparisons with prior world models.

1. Comparison with DINO-WM, DINO-world, and DINO-Foresight.

VANE is a representation learning framework for visual world modeling, trained through next-frame prediction in embedding space. Here, visual world modeling encompasses representing and understanding the current world state as well as predicting its future evolution. In contrast, DINO-WM, DINO-world, and DINO-Foresight learn dynamics or forecasting modules over fixed representations of the current visual state, extracted by frozen pretrained encoders. DINO-WM focuses on action-conditioned planning, while DINO-world and DINO-Foresight evaluate future scene understanding.

We add experiments to compare downstream forecasting and planning performance. For future semantic segmentation on Cityscapes, we fine-tune VANE‘s predictor for the benchmark's temporal intervals and use the same context/target timestamps.

MethodShort-term mIoU ↑ (3 frames)Mid-term mIoU ↑ (9 frames)
DINO-Foresight64.557.2
DINO-world64.755.1
VANE67.959.2

VANE improves over DINO-Foresight by 3.4 / 2.0 mIoU and over DINO-world by 3.2 / 4.1 mIoU at short/mid horizons. The gains at both horizons are consistent with next-frame prediction capturing temporal evolution and generative reconstruction preserving spatial detail for dense future forecasting.

For planning, we train an action-conditioned predictor on frozen VANE features using matched downstream data and planning settings:

MethodPointMaze SR ↑Wall SR ↑Reacher SR ↑PushT SR ↑Rope CD ↓Granular CD ↓
DINO-WM98.096.092.090.00.410.26
VANE98.098.090.094.00.370.21

SR: success rate (%); CD: Chamfer Distance.

VANE improves over DINO-WM on four tasks, matches it on PointMaze, and performs slightly worse on Reacher. These results show that representations learned through next-frame prediction transfer effectively to action-conditioned planning.

2. Comparison with LeWM.

LeWM and VANE differ in both objectives and methodology. LeWM learns action-conditioned dynamics for control and planning, predicting a single global CLS-based embedding using jointly learned targets and SIGReg. VANE instead learns general-purpose visual representations for visual world modeling through next-frame prediction without action conditioning. Its learning objective uses dense semantic patch targets covering the entire unseen next frame, with a frozen pretrained DINOv2 tokenizer providing a stable prediction space. VANE’s learned representations support a broad range of tasks, including motion and appearance understanding, dense prediction, action prediction, and planning. We hope these substantive differences in objectives and methodology help clarify VANE’s novelty relative to LeWM.

AuthorsOpenReview ↗

Response to Reviewer H5Wf

W2. Dependence on a pretrained tokenizer.

Rationale for using a fixed prediction space. A frozen tokenizer provides a stationary semantic space for learning through future prediction, analogous to video generation in a frozen VAE space or GPT-style prediction of fixed token identities. VANE learns its own encoder from historical frames to predict unseen future content; the tokenizer defines the targets, while the learned encoder supplies the downstream representations.

EMA targets evolve with the learner and may co-adapt toward easier prediction. To examine the role of fixed supervision in training stability, we compare:

VANE semantic targetGenerative decoderIN1K ↑SSv2 ↑ADE20K ↑
EMANo67.542.819.7
EMAYes81.067.535.3
Frozen DINOv2 ViT-LNo84.372.042.8
Frozen DINOv2 ViT-LYes84.572.345.1

With EMA, removing the decoder eliminates the remaining fixed pixel-space supervision, and training collapses. Frozen semantic targets maintain stable training even without the decoder, supporting their stabilizing role in our training setup.

Although VANE uses frozen DINOv2 ViT-L targets, it surpasses the larger DINOv2 ViT-g by 1.6 / 25.2 / 1.5 points on IN1K/SSv2/ADE20K, showing that future prediction can yield stronger transferable representations:

EncoderIN1K ↑SSv2 ↑ADE20K ↑
DINOv2 ViT-L85.349.847.7
DINOv2 ViT-g85.950.749.5
VANE ViT-g87.575.951.0

Matched targets and training stability. Table 9 compares ViT-L encoders trained on VISTA for 50K iterations. With the same frozen DINOv2 target, VANE achieves 84.5 / 72.3 / 45.1, versus 82.7 / 70.6 / 41.5 for V-JEPA 2.1. The gains of 1.8 / 1.7 / 3.6 points show that the teacher alone does not explain the improvement. The comparison without decoders is addressed in Q2.

Q2. Auxiliary decoder in the matched comparison.

With the auxiliary decoder disabled in both methods, VANE still outperforms V-JEPA 2.1 by 1.6, 1.4, and 1.3 points on IN1K, SSv2, and ADE20K, respectively (Table 5c). Enabling the decoder in VANE yields further gains of 0.2, 0.3, and 2.3 points, respectively (Table 5c). The decoder provides reconstruction supervision for VANE’s predicted future states. In V-JEPA 2.1, the predictor estimates features from multiple layers of the EMA target encoder. We tried training the same auxiliary decoder on V-JEPA 2.1’s predicted features from either multiple layers or a single layer, but training collapsed in both cases and yielded no usable representations. One possible explanation is that adding the reconstruction objective to V-JEPA 2.1’s multi-layer prediction setup may introduce shortcuts that lead to representation collapse.

AuthorsOpenReview ↗

Response to Reviewer H5Wf

Dear Reviewer H5Wf,

Thank you again for your thoughtful feedback. We would like to follow up on our responses to your concerns. We have clarified the differences between VANE and LeWM, provided further analysis of the role of frozen DINOv2 targets, and reported our attempts to add the same auxiliary generative decoder to V-JEPA 2.1, alongside a controlled comparison with the decoder disabled in both methods.

We would greatly appreciate your feedback on whether these clarifications and results address your main concerns. Please let us know if any questions remain; we would be happy to provide further clarification.

Thank you for your time and consideration.

Reviewer_LqtYOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VANE, a self-supervised visual pretraining framework based on next-frame prediction in embedding space. Instead of predicting individual visual tokens or reconstructing masked regions, VANE uses a causal encoder and a state predictor to predict the semantic representation of the entire next frame. A frozen semantic tokenizer provides the prediction targets, while an auxiliary generative decoder helps preserve fine-grained spatial information. The framework also extends to still images through local-to-global crop prediction, enabling unified image-video pretraining. Experiments cover action anticipation, image and video recognition, semantic segmentation, and depth estimation. VANE achieves competitive or improved performance across most benchmarks.

Critical strengths
  1. Predicting the next frame in semantic embedding space is a reasonable approach to learning temporal representations. The distinction between frame-level prediction and token-level autoregression is particularly interesting. The ablation results provide strong evidence for this design: under the same DINOv2 target space, next-frame prediction substantially improves SSv2 accuracy (72.3 vs. 53.2) and ADE20K mIoU (45.1 vs. 22.4) compared with next-token prediction. These results support the main methodological choice.

  2. The evaluation covers a broad range of tasks, including anticipation, recognition, segmentation, and depth estimation. VANE consistently performs well against recent unified image-video encoders, with particularly encouraging results on EPIC-KITCHENS-100 and Ego4D. The ablations on semantic targets, image prediction objectives, and the auxiliary generative decoder are also useful. They help explain which components contribute to the observed improvements.

Critical weaknesses
  1. Limited methodological novelty relative to existing predictive representation learning. Several key ingredients, including causal attention, future feature prediction, frozen semantic targets, and generative reconstruction, have been explored in prior work. Related approaches such as DINO-Foresight, DINO-world, and other predictive visual learning methods already investigate future prediction in representation space. Although VANE combines these ideas into a unified framework, its conceptual novelty beyond existing methods could be better established. More direct comparisons with closely related future-feature prediction approaches would strengthen the contribution.

  2. Incomplete isolation of the learning objective and limited validation of world-modeling capabilities. The main model uses a large-scale curated dataset and a multi-stage training recipe involving model scaling, long-video post-training, and high-resolution training. Although the paper includes useful small-scale ablations, it remains difficult to determine how much of the final improvement comes specifically from next-frame prediction rather than differences in data, compute, or training strategy. Furthermore, most evaluations rely on frozen encoder features and downstream probes. These results demonstrate strong representation learning, but provide limited direct evidence of long-horizon prediction or temporal dynamics modeling. The broader claims about visual world modeling would benefit from more direct evaluation of the learned predictor.

Questions for authors

Q1: Could the authors further clarify the differences between VANE and recent future-feature prediction methods, particularly DINO-Foresight and DINO-world? What is the key methodological advantage beyond using frame-level prediction with a frozen semantic tokenizer?

Q2 : How do the pretraining data, total training tokens, and computational budgets compare with V-JEPA 2 and V-JEPA 2.1? Could the authors provide a more controlled comparison under matched data and compute budgets to better isolate the benefits of the proposed objective?

Q3 : The main evaluations focus on frozen encoder representations, while the state predictor is not directly evaluated on downstream anticipation tasks. Could the authors provide quantitative evaluations of future-frame feature prediction, particularly for multi-step prediction or longer temporal horizons?

Q4: How much of the performance improvement can be attributed to the pretrained DINOv2 tokenizer? Although different target spaces are compared, a more detailed analysis of the knowledge transferred from the frozen teacher would help clarify what VANE learns from temporal prediction.

Q5 : The improvements on SSv2 and Diving-48 are relatively modest compared with those on ImageNet and Kinetics-400. Could the authors further analyze whether VANE learns stronger motion-sensitive representations, rather than primarily benefiting from semantic and appearance information?

Q6 : The full training recipe requires substantial computational resources. Could the authors report training throughput, memory consumption, and performance as a function of total training compute? This would help substantiate the scalability claim beyond improvements obtained by increasing model size and training data.

Q7 : Future frames can have multiple plausible outcomes, whereas the semantic prediction objective uses deterministic L1 regression. How does VANE handle uncertainty in future states, particularly during complex interactions or long-horizon prediction?

Q8 : The auxiliary generative decoder improves dense prediction performance, particularly on ADE20K. Could the authors clarify whether its contribution comes mainly from preserving spatial details or from providing additional pixel-level supervision? A comparison with simpler reconstruction objectives would be informative.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LqtY

We thank the reviewer for recognizing the motivation, broad evaluation, and evidence supporting frame-level prediction. We address the main concerns below.

W1 / Q1 / Q3. Methodological novelty and evaluation of planning and forecasting.

VANE is a representation learning framework for visual world modeling, trained through future prediction in feature space. Here, visual world modeling encompasses understanding the current world state and predicting its future evolution. VANE jointly trains a causal encoder and predictor to learn transferable image–video representations, whereas DINO-WM, DINO-world, and DINO-Foresight learn dynamics or forecasting modules over fixed representations from frozen pretrained encoders. We further evaluate these capabilities through action-conditioned planning and future semantic segmentation.

Action-conditioned planning. We train an action-conditioned predictor on frozen VANE features and compare with DINO-WM under matched downstream data and planning settings. VANE improves over DINO-WM on four tasks, matches it on PointMaze, and performs slightly worse on Reacher. These results show that representations learned through next-frame prediction transfer effectively to action-conditioned planning.

MethodPointMaze SR ↑Wall SR ↑Reacher SR ↑PushT SR ↑Rope CD ↓Granular CD ↓
DINO-WM98.096.092.090.00.410.26
VANE98.098.090.094.00.370.21

SR: success rate (%); CD: Chamfer Distance.

Future semantic segmentation. We fine-tune VANE for the Cityscapes forecasting intervals and evaluate at the same context and target timestamps as the baselines. Following the linear-head protocol, a segmentation head trained on observed DINOv2 features is applied to predicted future features.

MethodShort-term mIoU ↑ (3 frames)Mid-term mIoU ↑ (9 frames)
DINO-Foresight64.557.2
DINO-world64.755.1
VANE67.959.2

VANE improves over DINO-Foresight by 3.4 / 2.0 mIoU and DINO-world by 3.2 / 4.1 mIoU. The gains at both horizons are consistent with next-frame prediction capturing temporal evolution and generative reconstruction preserving spatial detail for dense future forecasting.

These results establish the usefulness of predicted features after adaptation, but do not demonstrate zero-shot multi-step rollout of the original predictor. The 3/9-frame horizons denote target-frame offsets, not recursive prediction steps.

W2. Isolating the objective and evidence for world modeling.

The contribution of the predictive objective is already examined through controlled ablations. As the reviewer acknowledges in Strength 1, next-frame prediction substantially improves over next-token prediction with the same frozen DINOv2 target: 72.3 vs. 53.2 on SSv2 and 45.1 vs. 22.4 on ADE20K. Table 5 also examines the target space, image objective, and auxiliary decoder individually. The comparison with V-JEPA 2.1 further matches the corpus, encoder scale, and iteration count, with gains persisting in the decoder-free, same-target setting (Q2).

For direct prediction evaluation, please see W1 / Q1 / Q3. VANE fine-tuned for Cityscapes forecasting reaches 67.9 / 59.2 mIoU at short-term and mid-term horizons, exceeding DINO-Foresight and DINO-world. We also report action-conditioned planning using an additional predictor trained on frozen VANE features. These experiments complement the frozen-encoder evaluations with results on future segmentation and planning.

Q2. Pretraining data and controlled comparison with V-JEPA 2.1.

Data availability and scale. Since V-JEPA 2.1's full pretraining corpus is not publicly available, we independently construct VISTA following V-JEPA 2.1's data-curation methodology, with approximately 145M images and 24M video samples, comparable in scale to VisionMix163M.

Controlled comparison. This experiment is already reported in Table 9 of the paper: both methods use ViT-L encoders trained on the same VISTA corpus for 50K iterations, with identical downstream evaluation protocols. With the same frozen DINOv2 ViT-L target, VANE achieves 84.5 / 72.3 / 45.1 on IN1K/SSv2/ADE20K, compared with 82.7 / 70.6 / 41.5 for V-JEPA 2.1. Table 5(c) further shows that decoder-free VANE achieves 84.3 / 72.0 / 42.8, still outperforming the same-target baseline. These results show that the gains persist under matched data, encoder scale, iteration count, and semantic targets, even without the auxiliary decoder.

AuthorsOpenReview ↗

Response to Reviewer LqtY

Q4. Contribution of the pretrained DINOv2 tokenizer.

VANE learns representations through future prediction in feature space, with the frozen DINOv2 tokenizer providing stable prediction space. Freezing the tokenizer keeps the semantic target space stationary throughout training. The encoder and predictor are jointly optimized to infer these future targets from historical observations, and the learned encoder supplies the downstream representations.

The tokenizer does not impose a ceiling on downstream representation quality. Although VANE uses DINOv2 ViT-L as its tokenizer, it outperforms the larger DINOv2 ViT-g on both SSv2 (75.9 vs. 50.7) and ADE20K (51.0 vs. 49.5), as reported in Tables 3 and 4.

Q5. Modest gains on SSv2 and Diving-48 and motion sensitivity.

Our classification evaluation uses a four-layer attentive probe on frozen encoder features. V-JEPA 2.1 uses bidirectional temporal attention, allowing each token to incorporate earlier and later observations within the clip. This richer context can benefit offline motion recognition. VANE instead uses a causal encoder, restricting each token to current and preceding observations to support future prediction and incremental encoding.

Despite this constraint, VANE improves SSv2 from 75.3 to 75.9 and Diving-48 from 89.0 to 89.7 over V-JEPA 2.1 under the same evaluation protocol. These modest gains should be considered alongside the architectural difference and VANE's larger improvements on anticipation tasks.

Q6. Training epochs and computational cost.

We use the same number of pretraining epochs as V-JEPA 2.1. As reported in Implementation details and the supplementary Multi-Stage Training Strategy, warm-up and base pretraining require approximately 26.6K B200 GPU-hours, followed by 30.7K B200 GPU-hours for long-video post-training and high-resolution cooldown, totaling 57.3K B200 GPU-hours.

Q7. Uncertainty under deterministic L1 semantic prediction.

Following V-JEPA 2 and V-JEPA 2.1, we use L1 regression between predicted and target features for semantic supervision. In VANE, the loss is averaged over supervised patches and transitions. This objective supports representation learning through future prediction, with the semantic predictor producing a deterministic point estimate. The auxiliary flow decoder models conditional pixel-level variability; explicitly modeling multiple plausible semantic futures remains future work.

Q8. Auxiliary reconstruction for spatial detail.

The distinction raised in this question is unclear to us. Our decoder is itself a lightweight auxiliary reconstruction module, so we are unsure which alternative reconstruction mechanism the reviewer is proposing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.