acceptodds
Under review as a conference paper at ICLR 2027

Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities

Abstract

Video modeling captures how the world is structured in space and how it evolves over time. Most approaches model this structure in a single representation space: RGB or a semantic representation (e.g., DINOv2, V-JEPA). However, different spaces capture different aspects of the underlying world and serve complementary roles, e.g., DINOv2 and V-JEPA excel at high-level semantic understanding, while RGB preserves fine-grained detail. In this work, we move beyond modeling the world in a single representation space and study video modeling jointly across time and modalities, each serving as a representation space (e.g., RGB, depth, optical flow, feature maps, and bounding boxes etc.). We develop an any-to-any multimodal video model that can predict any modality at any point in time, given any combination of input modalities at any time steps. Flexibly traversing this time-modality grid gives rise to higher-level inference capabilities that are not supported by specialist video models. First, in chained generation, we decompose a task (e.g., caption-to-video) into a chain of potentially simpler sub-tasks of predicting intermediate modalities, and find that this yields higher-quality generations than direct generation. Second, we study future prediction in different modality spaces and find that no single space is optimal: the best prediction space varies across samples, and selecting it per sample yields substantially more accurate forecasts than committing to any single space. Finally, we show that the flexible modality-time conditioning enables counterfactual generation, where conditioning modalities act as actions that steer generation, from coarse (e.g., captions) to fine-grained (e.g., depth or bounding boxes) control.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video modeling, multimodal learning, multi-task learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_1Hh3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends 4M-style multimodal masked modeling to video. It constructs aligned pseudo-labels on Kinetics-600, tokenizes dense and sequential representations, and trains a shared encoder-decoder with temporal embeddings and forecasting masks. The resulting Video-4M supports cross-modal prediction, generation through intermediate modalities, and forecasting in different representation spaces. Experiments cover several generation and perception tasks, compare direct and chained inference, and evaluate probabilistic forecasts using energy scores.

Critical strengths
  1. The breadth of capabilities in one model is a meaningful contribution. RGB, geometry, semantic features, and structured trajectories share a prediction interface, and the temporal masking design makes this interface useful for forecasting as well as cross-modal completion. Although the backbone follows 4M, the video tokenization, aligned dataset, and temporal training setup constitute substantial work. The value here is the flexible model rather than winning every specialist benchmark.

  2. The same-model comparisons provide convincing evidence that intermediate representations can help generation. For the large model, Table 2 reduces one-frame prediction FVD from 188.1 to 129.5 and caption-conditioned FVD from 220.8 to 180.2. I also appreciate the use of distributional forecasting scores and the explicit separation of the best-chain oracle from the validation-selected modality mixture in Appendix D.1. These make the forecasting analysis more informative than a single deterministic reconstruction score.

Critical weaknesses
  1. The generation gains are not separated from the additional inference budget. Chained inference generates several intermediate sequences before RGB, whereas direct inference only generates the target. The matched training schedule and same-model comparison are useful controls, but they do not establish whether the improvement comes from the choice of representation or simply from spending more computation at inference. This matters because decomposition through modalities is a central claim. Sampling iterations, generated token counts, and latency for both settings, together with a direct baseline at a comparable budget, would make the evidence substantially stronger.

  2. The proposed coarse-to-fine recipe has a less uniform advantage than the headline suggests. In Table 3, semantic-only chaining gives better caption-to-video FVD than the full chain (167.1 versus 180.2). In Appendix C.3, omitting depth also gives better class-conditioned FVD (215.6 versus 225.2), with a small accuracy trade-off. The paper does acknowledge these trade-offs, but the evidence supports task-dependent chain design more strongly than a generally preferred ordering. The validation-selected forecasting mixture is promising; its selection procedure and performance uncertainty need to be specified more precisely before drawing a broader conclusion about choosing representation spaces.

Questions for authors
  1. At a matched inference budget, how much of the Table 2 improvement remains? Please specify the sampling schedule, generated token budget, and measured latency of the direct and chained variants.

  2. For the oracle-independent mixture in Appendix D.1, what exact candidate chains, mixture weights or sample allocations, and validation selection procedure were used? Can you provide numerical energy-score improvements with uncertainty, and distinguish this result from the per-sample oracle?

Overall, I lean toward weak acceptance. The unified video model is useful and the empirical study contains several informative findings, even though the efficiency and chain-selection claims need a more careful qualification.

4 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer 1Hh3 (1/2)

We thank reviewer 1Hh3 for the positive feedback. We address their remaining questions below.

At a matched inference budget, how much of the Table 2 improvement remains? Please specify the sampling schedule, generated token budget, and measured latency of the direct and chained variants.

We refer the reviewer to our full response in Sec. 2 of the general response, which reports decoding steps, generated tokens, FLOPs and measured latency for different task settings (Table 1). To summarize:

  • The improvement remains at matched compute. At matched compute budgets, chaining still clearly improves FVD over direct generation on all tasks, and giving direct generation more decoding steps does not close the gap to Tab. 2 of the main paper.
  • Sampling schedule. All dense modalities (RGB, SigLIP, depth) are decoded with ROAR masked decoding and a linear token schedule over NN steps. Direct generation uses N=100N=100 for class→video and N=60N=60 for frame-conditioned generation, as in the paper, with sampling hyperparameters selected by grid search in this setting. In the chain, SigLIP, depth and RGB use 50, 50 and 100 masked steps, and the sequential modalities (caption, transcription, bounding boxes) are decoded autoregressively until end-of-sequence. For the matched-compute comparison, we increase NN for direct generation until its FLOPs match the chain; all other sampling settings are unchanged.
  • Generated tokens. For class→video, direct generation produces 1,280 RGB tokens and the →SigLIP→RGB chain 2,260 (980 SigLIP + 1,280 RGB). More tokens alone do not explain the gain: chains through less informative intermediates also generate more tokens, yet are worse than direct generation (class→video in Tab. 3 of the main paper: Depth→RGB generates 2,560 tokens and reaches 446.3 FVD, vs. 330.6 for direct generation with 1,280 tokens in Tab. 2 of the main paper).

We have added the full breakdown and quality-cost plots to App. C.5 of the revised manuscript.

The proposed coarse-to-fine recipe has a less uniform advantage than the headline suggests. […] the evidence supports task-dependent chain design more strongly than a generally preferred ordering.

We thank the reviewer for this careful reading, and we agree that the evidence supports task-dependent chain design. We would like to clarify that the main goal of this work is to show that chained generation, made possible by training a single video model across the modality-time grid, improves over direct generation. As we already acknowledge in the main paper (paragraph "Optimal chain design" in Sec. 3.4), the ideal chain is likely task-dependent, and with nn modalities there are n!n! possible orderings, so a systematic search per task is beyond the scope of this work. We therefore use a single, general coarse-to-fine chain for all tasks, which aims to achieve the best performance trade-off across tasks, instead of optimizing the chain for each task or metric. In our experiments, this coarse-to-fine chain improves upon direct generation on all tasks (Tab. 2 of the main paper), and among the configurations in Tab. 3 of the main paper, it gives the best FVD on 3 of the 4 tasks.

At the same time, different intermediate modalities improve different aspects of generation: semantic modalities improve FVD and alignment, while depth improves structural detail (Tab. 3 of the main paper and Tab. 12 of the revised manuscript). Since Video-4M supports any prediction route at inference without retraining, the chain can also be chosen for a specific task or metric, e.g., a semantic-only chain when caption alignment is the priority. We have highlighted these trade-offs upfront in the chained generation section (Sec. 4.2) of the revised manuscript.

AuthorsOpenReview ↗

Authors’ Response to Reviewer 1Hh3 (2/2)

For the oracle-independent mixture in Appendix D.1, what exact candidate chains, mixture weights or sample allocations, and validation selection procedure were used? Can you provide numerical energy-score improvements with uncertainty […]? The validation-selected forecasting mixture is promising; its selection procedure and performance uncertainty need to be specified more precisely before drawing a broader conclusion about choosing representation spaces.

We thank the reviewer for finding the validation-selected mixture promising. As suggested, we detail the exact candidate chains, the sample allocations for every target, and the validation selection procedure in Sec. 3 of the general response. In short, the allocation of 14 samples across seven intermediate routes is selected by exhaustive search (38,760 candidates) on 528 held-out validation clips, and evaluated without modification on 472 test clips.

With paired bootstrap 95% confidence intervals, the mixture improves significantly over the best fixed route for RGB, depth and DINOv2, but not for bounding boxes, and recovers 5–48% of the oracle gain (Table 2 of the general response). We therefore phrase the conclusion more precisely: the best forecasting representation depends on the clip (oracle), and pooling complementary routes already gives a measurable oracle-free gain, while choosing the representation per clip without access to the future remains open. We have added this specification to App. D.1 of the revised manuscript.

Reviewer_1Hh3OpenReview ↗

Post-rebuttal assessment

Thank you for the detailed response. I have raised my rating from 3 to 4. The new compute-matched comparisons support the benefit of chaining beyond extra decoding, and the forecasting response specifies the validation-selected mixture, fixed rollout allocations, and paired confidence intervals. Together with the clearer discussion of task-dependent chain choices, these additions address my main concerns.

AuthorsOpenReview ↗

Thank you!

We sincerely thank the reviewer for taking the time to provide constructive feedback and for raising the score. We have incorporated these discussions and results into the revised manuscript, and we believe they have further strengthened the paper.

Reviewer_59ygOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Video-4M, an any-to-any video model supporting 12 modalities through discrete tokenization and multimodal masked modeling. It extends 4M with temporal positional embeddings and forecasting-oriented masking. The main experiments study chained generation through intermediate modalities, forecasting in different representation spaces, and controllable generation. Chaining improves generation FVD across tasks and model sizes, while forecasting experiments suggest that useful intermediate representations vary across samples.

Critical strengths

Video-4M provides a useful extension of any-to-any modeling to video. A single model supports perception, generation, and forecasting across diverse representations. The temporal masking ablations also support the design choices needed for forecasting, making this more than a straightforward application of image-based 4M.

The chained-generation results are convincing within the evaluated setting. Using the same Large model, chaining reduces caption-to-video FVD from 220.8 to 180.2 and one-frame-conditioned FVD from 188.1 to 129.5. Improvements also hold at Base scale. The modality-group and ordering ablations provide useful evidence about which intermediate representations help.

  1. The positioning against related work is incomplete. Section 2 states that any-to-any modeling has not been explored for video, but does not discuss UnityVideo or UniVidX. UnityVideo studies joint learning across video modalities and tasks, while UniVidX explicitly supports omni-directional conditional generation through modality masking. GenCeption (Video Generation Models are General-Purpose Vision Learners) and One4D also provide relevant context for unified perception and generation/reconstruction. These works differ in scope, but should be cited and discussed to clarify what is new about Video-4M’s temporal conditioning and chained inference. The broad priority claim should be narrowed accordingly.

  2. The counterfactual-control claim is supported mainly by selected examples. Section 4.4 demonstrates caption, bounding-box, and depth-based control, but does not quantify whether generated outputs follow the edited conditions while preserving unaffected content. This is a distinct capability that the generation FVD results do not establish. A systematic evaluation of trajectory adherence and preservation would make this part of the contribution more convincing.

Critical weaknesses

The benefit of chaining is not separated from its additional inference cost. The same-checkpoint comparison controls training, but generating several intermediate modalities requires extra computation. Without latency, token counts, or a direct-generation baseline given a comparable inference budget, the quality–cost trade-off remains unclear. Improvements are also not uniform across metrics: caption alignment drops from 21.6 to 20.8 for the Large model despite better FVD.

The strongest forecasting gains rely on selecting the best chain using the realized future. This establishes potential rather than an inference-time selection method. Appendix D.1 provides a useful oracle-free mixture, but does not clearly quantify how much of the oracle gain it recovers or its uncertainty. The practical forecasting benefit is therefore less established than the chained-generation result.

Questions for authors

Does chaining retain an advantage when the direct baseline receives a comparable inference budget?

For the oracle-free modality mixture, could you report numerical improvements and confidence intervals against the best fixed chain, and clarify whether the total number of sampled rollouts is matched?

4 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer 59yg (1/3)

We thank reviewer 59yg for the positive feedback. We address their questions below.

Does chaining retain an advantage when the direct baseline receives a comparable inference budget?

Yes. Please see Sec. 2 of the general response, where we compare chained and direct generation at two matched compute budgets in detail. Giving direct generation more decoding steps does not meaningfully improve it, while chaining at the same budget clearly improves FVD on every task. At the full chain's budget, direct generation is far behind on all tasks.

Improvements are also not uniform across metrics: caption alignment drops from 21.6 to 20.8 for the Large model despite better FVD.

We note that the drop in alignment comes from which intermediate modalities the chain goes through, not from chaining itself. Our runs of the chains in Tab. 3 of the main paper also logged ViCLIP [1] alignment metric:

Table. Caption→video FVD and ViCLIP alignment (Video-4M-L).

ConfigurationFVD ↓ViCLIP ↑
Direct220.821.65
Sparse only (Caption → Transc. → RGB)337.618.41
Semantic only (SigLIP 2 → BBox → RGB)167.121.94
Geometric only (Depth → RGB)282.320.28
Coarse-to-fine chain (default)180.220.83

Semantic intermediates improve both FVD and alignment over direct generation, while the sparse-only (generated transcription, 18.41) and geometric-only (depth, 20.28) chains lower alignment. Depth added after the semantic modalities does not hurt alignment: for class→video, it slightly improves top-1 accuracy (69.3 → 70.1) and edge sharpness, at a small FVD cost (Tab. 12 of the revised manuscript).

With the default coarse-to-fine chain, we aimed for the best trade-off across tasks and metrics rather than the best value of a single metric: it improves FVD over direct generation on all four tasks (Tab. 2 of the main paper), outperforms the semantic-only chain on class, 1-frame and 5-frame conditioning (Tab. 3 of the main paper), and gives the best structural detail (Tab. 12 of the revised manuscript), while the semantic-only chain is best for caption conditioning. Since Video-4M supports any prediction route without retraining, the semantic-only route can be used when caption alignment is the priority.

The positioning against related work is incomplete. […] The broad priority claim should be narrowed accordingly.

We thank the reviewer for the pointers, and we agree that the broad priority claim should be narrowed. We would like to clarify that the unique positioning of Video-4M lies in its joint traversal of both the modality and time axes, i.e., conditioning on any modality at arbitrary time steps, while prior and concurrent multimodal video models mainly traverse across modalities. This joint traversal of the modality–time grid expands the breadth of tasks, which allows us to explore forecasting in any modality and chained generation within one model.

In particular, UnityVideo [2] uses auxiliary modalities mainly to improve RGB generation; UniVidX [3] allows different subsets of four pixel-aligned graphics modalities to act as conditions or targets, but over the whole clip; GenCeption [4] (concurrent) maps RGB to perception outputs only; and One4D [5] generates RGB and point maps. One4D also supports conditioning on sparse RGB frames, which relates to our temporal conditioning. However, its outputs are limited to RGB and 3D geometry and its conditioning is limited to RGB frames, whereas Video-4M can condition on any modality at any time step.

We have now cited and discussed UnityVideo, UniVidX, GenCeption and One4D in the revised manuscript, and narrowed the claim to: "Prior and concurrent work has explored multimodal video modeling, through masked any-to-any pretraining over a few egocentric modalities (EgoM2P [6]) or through video diffusion models that jointly generate a small set of pixel-aligned modalities with whole-clip or RGB-only conditioning (OmniVDiff [7], UnityVideo, UniVidX, One4D). To our knowledge, Video-4M is the first single model that performs any-to-any generation across a diverse set of modalities in video domain, with conditioning on any subset of modalities at any time step."

References

[1] InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation, Wang et al., 2024

[2] UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation, Huang et al., 2025

[3] UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors, Chen et al., 2026

[4] Video Generation Models are General-Purpose Vision Learners, Wang et al., 2026

[5] One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control, Mi et al., 2025

[6] EgoM2P: Egocentric Multimodal Multitask Pretraining, Li et al., 2025

[7] OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding, Xi et al., 2025

AuthorsOpenReview ↗

Authors’ Response to Reviewer 59yg (2/3)

The counterfactual-control claim is supported mainly by selected examples. […] A systematic evaluation of trajectory adherence and preservation would make this part of the contribution more convincing.

We note that our qualitative examples in our paper are representative of the model's performance rather than rare successes, and our supplementary video material shows many further counterfactual generations beyond those in the paper. We agree, however, that a systematic evaluation makes this contribution more convincing, and ran it for bounding-box, caption, and depth controls, with the default chain and the same settings as the paper's counterfactual examples (1,000 test videos, first frame given as condition).

Bounding-box edits. Given the first frame and the boxes of all 17 frames, we shift one object's boxes horizontally (linearly from 0 at the first frame to 30% of the frame width at the last) and leave all other boxes unchanged; 428 videos contain a suitable object. An open-vocabulary detector (Grounding DINO [8], prompted with the object's label) locates the object in the second half of the generated clip. Success is the fraction of videos where it overlaps the edited (or original) path with IoU ≥ 0.5; preservation is the PSNR to the ground-truth video outside the object's boxes.

Table. Counterfactual bounding-box control (428 test videos with a tracked object).

Boxes givenSuccess, edited path ↑Success, original pathPSNR outside boxes ↑FVD ↓
None8.4%43.5%14.7436.4
Original7.5%73.0%17.1390.4
Edited54.0%20.2%16.6396.6

With edited boxes, the generated object follows the new trajectory (54.0% vs. 7.5% with the original boxes) and largely leaves its original path (20.2% vs. 73.0%). Content outside the edited region stays close to generation with the original boxes (PSNR 16.6 vs. 17.1), and realism is essentially unchanged (FVD 396.6 vs. 390.4), so the edit acts mostly locally.

Caption edits. We generate each video from its own first frame but with the caption of a different test video, and measure ViCLIP [1] alignment to this counterfactual caption and to the original one.

Table. Counterfactual caption control (1 frame + caption → video).

Caption givenViCLIP to given caption ↑ViCLIP to original captionΔ ↑
Own caption (reference)20.7220.72–
Counterfactual caption15.0312.03+3.00

The generated video follows the caption it is given rather than the content of its first frame, and its alignment to the original caption falls from 20.7 to 12.0 although the first frame is unchanged. Since a swapped caption describes an entirely different scene, the model typically keeps the given frame and then moves to a new scene consistent with the caption.

Depth edits. We performed the same analysis with depth as the conditioning modality, on a smaller set of 17 test videos. To obtain different depth trajectories from the same first frame, we first generated three different videos starting from that frame with OpenAI's video generation API and extracted their depth with Video Depth Anything [9]; each of these depth videos is then used to condition Video-4M. The generated video follows the depth it is given (Spearman ρ to the input depth 0.93 vs. 0.71 to the two alternative depths, close to the 0.95 obtained by the depth estimator on the source videos), matches its own depth best in 50 of 51 generations (chance 33%), and is always closest to the source video that corresponds to its depth (PSNR 17.9 vs. 12.5).

Overall, these experiments empirically support that Video-4M adheres to the input control across bounding-box, caption and depth conditioning. We have added these studies to App. B.1.

References

[1] InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation, Wang et al., 2024

[8] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, Liu et al., 2024

[9] Video Depth Anything: Consistent Depth Estimation for Super-Long Videos, Chen et al., 2025

AuthorsOpenReview ↗

Authors’ Response to Reviewer 59yg (3/3)

The strongest forecasting gains rely on selecting the best chain using the realized future. .... For the oracle-free modality mixture, could you report numerical improvements and confidence intervals against the best fixed chain, and clarify whether the total number of sampled rollouts is matched?

We agree that selecting the best chain per sample is not a practical inference-time method, and we see our contribution here as two-fold. First, the oracle analysis is itself a finding: the best forecasting representation is scene-dependent, and selecting the right one per clip makes a significant difference, which can only be observed with a single model that forecasts through any modality. Second, the validation-selected mixture turns part of this potential into a practical, oracle-free gain. As suggested by the reviewer, we specify the modality mixture details and report it separately from the oracle in Sec. 3 of the general response (Table 2). The number of rollouts is matched: the direct forecast, every fixed route, the mixture and the oracle all use 14 forecasts per clip.

Against the best fixed chain, the mixture improves the energy score significantly for RGB, depth and DINOv2 (mixture − best fixed, paired bootstrap 95% CIs: -0.063 [-0.085, -0.041], -0.011 [-0.015, -0.007] and -7.46 [-8.95, -6.03]), recovering 20.1%, 23.9% and 47.8% of the oracle gain. For bounding boxes, the difference is within noise (5.3% [−9.1, 17.0] of the oracle gain). In the revised manuscript, we now clearly separate the oracle analysis from the practical mixture and report the mixture's gains with their uncertainty.

AuthorsOpenReview ↗

Gentle Reminder: Author Response

We thank the reviewer again for their detailed and constructive feedback. As the author-reviewer discussion period is nearing its end, we wanted to kindly check whether the reviewer has had a chance to go through our response. We would be happy to clarify any remaining questions.

Reviewer_X6o2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper propose Video-4M, which extends 4M’s multimodal masked modeling approach to video, mapping between RGB, semantic, geometric, and textual modalities. It studies direct prediction and chains that generate intermediate modalities before the final output. Experiments cover generation, understanding, and probabilistic forecasting, with ablations of modality groups, chain order, and temporal masking.

Critical strengths
  1. Predicting video through intermediate modalities is an interesting direction. The shared-model comparison in Table 2 is convincing evidence that the prediction route matters. The modality-group and ordering ablations make these differences easier to interpret.
  2. The treatment of probabilistic forecasting broadens the study, and the authors distinguish oracle chain selection from a validation-selected mixture. The presentation is mostly clear about the trade-offs and tokenizer limitations.
Critical weaknesses
  1. Chaining costs more at inference. It generates extra modalities through additional model calls, but Table 2 compares quality without matching these budgets. Total decoding iterations, generated tokens, and latency are missing from this comparison. The table mixes the benefit of intermediate representations with the benefit of extra decoding.
  2. I cannot reconstruct the forecasting policy that would actually be used. Lines 428–432 identify the oracle, and Appendix D.1 describes a validation-selected mixture. The mixture's composition, weights, rollout allocation, and test-time budget are still unclear. Its measured gains and total cost need to be reported separately from the oracle upper bound.
Questions for authors
  1. How do direct and chained prediction compare under similar inference budgets?
  2. What exactly is the final modality mixture, how many samples come from each chain, and what is its total cost?
  3. Removing depth improves FVD in Tables 6–8. How should users choose the trade-off between distribution metrics, alignment, and structural detail?
3 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer X6o2 (1/2)

We thank reviewer X6o2 for the overall positive review. We address their remaining questions below.

Chaining costs more at inference. It generates extra modalities through additional model calls .... The table mixes the benefit of intermediate representations with the benefit of extra decoding.

Thank you for this important point. We ran a compute-matched study that reports decoding steps, generated tokens, FLOPs and measured latency for different task settings. Please refer to Sec. 2 and Table 1 of the general response. In summary:

  • Increasing the inference compute budget for direct generation using decoding iterations does not meaningfully improve its performance.
  • At matched compute budgets, chaining is clearly better on all tasks and at different compute budgets.
  • We also note that the performance gains come from the choice and order of intermediate modalities in the chain, not from the number of generated tokens. For instance, chains through less informative intermediates perform worse than direct generation despite generating more tokens (class→video in Tab. 3 of the main paper: Depth→RGB 446.3 FVD with 2,560 generated tokens, Caption→Transcription→RGB 425.6 with ≈1,460, vs. 330.6 for direct with 1,280 in Tab. 2 of the main paper).

This separates the benefit of the intermediate representation from the benefit of extra decoding, as the reviewer suggested.

I cannot reconstruct the forecasting policy that would actually be used. Lines 428–432 identify the oracle ...

Thank you for the careful reading. We specify the full forecasting policy in Sec. 3 of the general response. To summarize, the policy that would actually be used is the validation-selected mixture; the per-sample oracle is an upper bound for analysis which was first used to study that optimal forecasting modalities vary per input sample.

  • Composition and weights: for RGB, 10 bounding-box, 3 SigLIP 2 and 1 transcription rollouts (weights 10/14, 3/14, 1/14); the allocations for depth, DINOv2 and bounding boxes are listed in Sec. 3.
  • Rollout allocation: selected by exhaustive search over all 38,760 allocations of 14 samples on 528 held-out validation clips, and fixed before test time, so no per-clip selection or access to the future is needed.
  • Test-time budget: 14 rollouts per clip, the same as the direct forecast and any fixed prediction route.
  • Measured gains, separate from the oracle: the mixture improves over direct forecasting on all four targets (RGB, depth, DINOv2 and bounding boxes), and significantly over the best fixed route for RGB, depth and DINOv2 (paired bootstrap 95% CIs, Table 2 of the general response); the oracle remains better on all four.

We have added this specification and the table to App. D.1.

AuthorsOpenReview ↗

Authors’ Response to Reviewer X6o2 (2/2)

Removing depth improves FVD in Tables 6–8. How should users choose the trade-off between distribution metrics, alignment, and structural detail?

We note that no single metric captures every aspect of generated video quality, and each metric is sensitive to one property of the generated samples: FVD, built on I3D [1] features trained for action recognition, mainly reflects motion and overall distribution realism; ViCLIP [2] score and top-1 accuracy reflect alignment with the input condition; and edge sharpness reflects spatial structure and fine-grained detail.

Our experiments show that each modality group in the chain mainly helps one of these properties. In Tab. 3 of the main paper, the semantic group gives the best FVD among single groups on every task (e.g., 167.1 for caption→video, vs. 337.6 sparse and 282.3 geometric) and the best text alignment (21.9, vs. 18.4 sparse, 20.3 geometric and 21.6 direct). The geometric group mainly adds structural detail: adding depth to the chain raises RGB edge sharpness from 68.39 to 70.81 (Tab. 12 of the revised manuscript, which was Tab. 6 in the original submission).

Our recommendation for users follows directly:

  • Realism and alignment: a semantic-first chain such as →SigLIP→RGB, which gives the best alignment and strong FVD at a low cost.
  • Fine spatial structure (e.g., boundaries, thin objects, background clarity): add geometric modalities such as depth to the chain.
  • Balanced trade-off across tasks and metrics: our default coarse-to-fine chain.

Since Video-4M supports any prediction route at inference without retraining, users can choose the intermediate modalities that best reflect their downstream needs. With the default coarse-to-fine chain, we aimed for the best trade-off across these properties rather than the best value of a single metric. Tab. 12 of the revised manuscript shows this: the default chain gives the best top-1 accuracy (70.1) and edge sharpness (70.81 in RGB, 18.47 in depth), at a small FVD cost compared to the chain without depth (225.2 vs. 215.6), and it improves over direct generation in both FVD and top-1 accuracy (Tab. 2 of the main paper).

References

[1] Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset, Carreira and Zisserman, 2017

[2] InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation, Wang et al., 2024

AuthorsOpenReview ↗

Gentle Reminder: Author Response

We thank the reviewer again for their detailed and constructive review. As we are approaching the end of the author-reviewer discussion period, we wanted to kindly check whether the reviewer has had a chance to look at our response. We would be happy to address any further questions.

Reviewer_aVdaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Video-4M, a method that extends the 4M work to videos and allows video modeling across multiple modalities like RGB, depth, captions, bounding boxes, and more. The method can be used for chained generation, where a task is broken down into a coarse-to-fine chain of tasks. It can also be used for forecasting across modalities, and the paper studies which modalities work best for forecasting different scenes. The method can also be used for counterfactual video generation using conditions from different modalities.

The authors have created a dataset, K600-MM, with 12 temporally aligned modalities. They use it to train the model with multiple masking strategies, including three new temporal masking techniques: FCFF, FCAF, and CFP, to support different types of forecasting.

The method processes different input modalities and converts them into discrete tokens that can be fed into an encoder-decoder Transformer to predict masked tokens. Dense modalities are tokenized using FSQ-based autoencoders that compress them both spatially and temporally. Text uses WordPiece, bounding boxes use discrete coordinate tokens, and human poses use a VQ-VAE-based representation. The authors also use spatiotemporal positional embeddings to support predictions across time.

The main contributions include unified video modeling across 12 modalities, the K600-MM dataset, new temporal masking strategies, and experiments showing that chained video generation using intermediate modalities improves video quality compared to direct generation. The paper also shows that the best modality for forecasting depends on the particular scene.

Critical strengths
  1. The paper brings together 12 modalities for video-based tasks, including RGB, depth, captions, and bounding boxes. The authors also create K600-MM, a dataset with 12 temporally aligned modalities. The unified model achieves competitive results compared to specialist models on several tasks.

  2. The authors introduce FCFF, FCAF, and CFP masking strategies to support predicting future frames, selected parts of future frames, and subsequent tokens in sequential modalities, respectively. The ablation results show that these strategies help improve forecasting.

  3. The paper shows that the best modality for forecasting varies across different video samples. This is an interesting finding that shows how different representations can help predict the future, although the best modality is selected using an oracle.

Critical weaknesses
  1. The paper does not clearly compare its approach with existing multimodal video models such as EgoM2P(though it is specifically for egocentric videos, it solves some of the same issues as the paper about predicting modalities across time) , diffusion-based models like OmniVDiff, or models combining text and video generation like UniVideo. It is unclear why the proposed discrete-token Transformer is better than diffusion or flow matching approaches. The authors do not clearly show what advantage Video-4M over these approaches or over combining these approaches to increase the modalities. Comparisons on common tasks, video quality, and inference cost would help establish the advantages of the proposed approach when compared to closest multimodal video generation papers.

  2. The paper uses FSQ-based autoencoders to compress dense video modalities, but does not compare them with other video tokenization methods like Transformer-based tokenizers( eg: ViTok-v2). Since the quality of these compressed representations directly affects the final predictions, ablations comparing these would help.

Questions for authors
  1. Can the authors provide experimental comparisons with existing multimodal video models(mentioned in weaknesses), including diffusion and flow-matching approaches, to better establish the advantages of Video-4M?

  2. Can the authors provide ablations using alternative video tokenizer methods(mentioned in weaknesses) instead of only FSQ-based ones, to show how the choice of compression method affects the results?

  3. Can the authors compare direct and chained generation under the same inference computation budget to show whether the improvements from chaining still hold or is the improvements just because of extra compute used ?

5 comments
AuthorsOpenReview ↗

Authors’ Response to Reviewer aVda (1/3)

We thank reviewer aVda for their constructive feedback. We address their questions below.

The paper does not clearly compare its approach with existing multimodal video models such as EgoM2P, OmniVDiff, or UniVideo. The authors do not clearly show what advantage Video-4M over these approaches or over combining these approaches to increase the modalities.

We agree that positioning Video-4M against these multimodal video models clarifies our contributions, and summarize the key differences below.

  • EgoM2P [1] is the closest related work to ours and follows a 4M-style masked training approach. It covers four egocentric modalities trained purely on egocentric data, with investigation focused on pairwise cross-modal prediction tasks. It does not explore the broader implications of multimodal video modeling at inference time, i.e., how a model that spans many modalities can use them beyond direct cross-modal prediction, such as chained generation, future prediction in different modality spaces, and counterfactual generation with modalities as actions, which are the main focus of our work.
  • OmniVDiff [2] is a diffusion approach that jointly diffuses four modalities with whole-clip conditioning and always decodes all four modalities even if one prediction modality is desired.
  • UniVideo [3] unifies understanding, generation and editing, but its inputs and outputs are restricted to text and RGB.

Table. Comparison with closely related multimodal video models. ✓: supported and shown; ○: possible in principle but not investigated; ✗: not supported by design.

MethodModalitiesInput → outputCond. on any modalityCond. at arbitrary time stepsChained generationForecasting in non-RGB modalities
EgoM2P4any → any✓○○○
OmniVDiff4one → rest✓✗✓✗
UniVideo2 (text, RGB)text ↔ RGB✓✗○✗
Video-4M12any → any✓✓✓✓

Video-4M is the only model among these that demonstrates conditioning at arbitrary time steps, which enables forecasting in any modality and counterfactual control with any modality at chosen time steps (e.g., per-frame boxes or depth), beyond text-based steering of the whole clip as in UniVideo. Moreover, combining these models would not give the same capabilities. The three models are built on different modeling frameworks (masked modeling for EgoM2P, video diffusion for OmniVDiff, and a multimodal LLM with a diffusion generator for UniVideo), each with its own set of modalities and its own way of providing conditions (unmasked input tokens for EgoM2P, one clean modality over the whole clip for OmniVDiff, and text or visual inputs to the MLLM for UniVideo), so their combination is not designed to jointly condition on an arbitrary subset of modalities at arbitrary time steps. We have added a related-work discussion of EgoM2P, OmniVDiff, UniVideo to the revised manuscript.

References

[1] EgoM2P: Egocentric Multimodal Multitask Pretraining, Li et al., 2025

[2] OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding, Xi et al., 2025

[3] UniVideo: Unified Understanding, Generation, and Editing for Videos, Wei et al., 2025

AuthorsOpenReview ↗

Authors’ Response to Reviewer aVda (2/3)

Comparisons on common tasks, video quality, and inference cost would help establish the advantages of the proposed approach when compared to closest multimodal video generation papers.

As discussed in our above comment, the main advantage of Video-4M over these works lies in the breadth of modalities and higher-level inference schemes which these models do not offer or present. Beyond this, a direct comparison is not trivial since these models differ in training data, resolution, and model size (5B and ≈20B parameters for OmniVDiff [2] and UniVideo [3] vs. 0.84B for Video-4M). Where tasks overlap on the same data distributions, we provide quantitative comparisons below.

Table. Depth estimation and inference cost on NYUv2 and DIODE. Latency per 17-frame clip on one A100 (80GB), batch size 1, including encoding, generation and decoding (mean over 10 clips after 2 warm-up clips, excluding model loading and file I/O).

MethodParamsTraining videosInputNYUv2 AbsRel↓NYUv2 δ1↑DIODE AbsRel↓DIODE δ1↑Latency (s/clip)↓
OmniVDiff5B35M + 2B images (CogVideoX [8]) + 0.4MOriginal RGB0.21282.50.58862.932.4
Video-4M0.84B≈0.4M (K600, from scratch)Tokenized 128×128 RGB0.20683.40.58065.59.5

Video-4M is competitive to OmniVDiff across all four depth metrics, while operating on tokenized 128×128 RGB rather than full-resolution input as used by OmniVDiff. It is also about 3.4× faster (9.5 s vs. 32.4 s per clip) and needs about 4× less GPU memory (6.4 GB vs. 25.2 GB peak).

Table. Video captioning comparison on the MSVD test set. Latency per clip on one H100 (80GB), batch size 1. Random caption: for each test video, the ground-truth caption of a randomly selected validation video is used as the prediction.

MethodParamsTraining videosMETEOR ↑BERTScore F1 ↑Latency (s/clip) ↓
Random caption (lower bound)––7.6884.06–
UniVideo≈20B (7B+13B)not reported23.9789.5159.47
Video-4M0.84B≈0.4M (K600)22.1289.509.41

Note that UniVideo keeps its MLLM (Qwen2.5-VL-7B [9]) frozen during training and reports its understanding results with this MLLM backbone, so its captioning ability comes from this pretrained vision-language model rather than from UniVideo's own training. Video-4M is within 1.9 METEOR of UniVideo, while simultaneously supporting 11 other modalities, and runs about 6× faster (9.41 s vs. 59.47 s per clip).

Why is the proposed discrete-token Transformer better than diffusion or flow matching approaches?

We see diffusion and discrete masked modeling as complementary rather than competing. The focus of our work is introducing any-to-any modeling in the video domain, not the underlying generative framework. A diffusion-based any-to-any video model could equally benefit from these capabilities.

References

[2] OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding, Xi et al., 2025

[3] UniVideo: Unified Understanding, Generation, and Editing for Videos, Wei et al., 2025

[8] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer, Yang et al., 2024

[9] Qwen2.5-VL Technical Report, Bai et al., 2025

AuthorsOpenReview ↗

Authors’ Response to Reviewer aVda (3/3)

Ablations using alternative video tokenization methods would help show how the choice of compression method affects the results.

We agree that tokenizer quality bounds the final predictions. During the tokenizer finalization phase in our project, we indeed ran ablations across different tokenizer types and compared both FSQ- and VQ-based variants to select the best tokenization scheme. Below we compare our RGB tokenizer with publicly available discrete video tokenizers on 2,000 K600 test clips in our setting (17 frames, 128×128). All tokenizers in the comparison have the same compression ratio (8×8 spatial and 4× temporal), so they produce the same number of tokens per clip (1,280).

Table. Tokenizer reconstruction on 2,000 Kinetics-600 test clips (17 frames, 128×128, 1,280 tokens per clip).

TokenizerArchitecture / quantizerrFVD ↓rFID ↓PSNR ↑LPIPS ↓
OmniTokenizer [10]Transformer / VQ171.723.924.050.229
Cosmos-Tokenizer [11] DV4x8x8CNN + attention / FSQ180.423.225.130.245
VidTok [12] (released)3D CNN / FSQ76.36.925.460.165
Ours (VidTok fine-tuned)3D CNN / FSQ49.55.425.410.160

The FSQ-based VidTok tokenizer improves over both OmniTokenizer and Cosmos-Tokenizer already before fine-tuning (rFVD 76.3 vs. 171.7 and 180.4), and fine-tuning it on modality-specific data, as we do for Video-4M, improves it further. Compared to VQ, FSQ-based tokenizers are also easier to train, since the codebook is fixed and implicit: no codebook learning, commitment loss, or handling of codebook collapse is required.

Regarding ViTok-v2, which the reviewer suggested: the public release is an image autoencoder with continuous latents, and its limitations section notes that video is not yet supported, so it cannot produce discrete video tokens for Video-4M. The Video-4M framework itself is agnostic to the choice of discrete tokenizer, Transformer-based or not (App. H), and can directly benefit from stronger discrete video tokenizers as they become available. We have added this comparison to App. C.2.

Can the authors compare direct and chained generation under the same inference computation budget to show whether the improvements from chaining still hold or is the improvements just because of extra compute used?

We thank the reviewer for raising this. Please see Sec. 2 of the general response, where we compare chained and direct generation at two matched compute budgets and report decoding steps, generated tokens, FLOPs and latency (Table 1 of the general response). Increasing the number of decoding steps does not meaningfully improve direct generation, while chaining at the same budget clearly improves FVD on all tasks. The improvement therefore comes from the choice of intermediate modality.

References

[10] OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation, Wang et al., 2024

[11] Cosmos World Foundation Model Platform for Physical AI, Agarwal et al., 2025

[12] VidTok: A Versatile and Open-Source Video Tokenizer, Tang et al., 2024

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/3): Summary of Rebuttal and Revisions

We thank the reviewers for their time and insightful comments, and we are glad that the breadth of capabilities of any-to-any video modeling was broadly appreciated.

  • Reviewer aVda finds the unified modeling across 12 modalities and the K600-MM dataset a “meaningful contribution” and notes that the temporal masking ablations “support the design choices needed for forecasting.”
  • Reviewer X6o2 found the shared-model comparison in Table 2 “convincing evidence that the prediction route matters” and the probabilistic forecasting treatment a useful broadening of the study.
  • Reviewer 59yg found the chained-generation results “convincing within the evaluated setting” and appreciates the explicit separation of the oracle from the validation-selected mixture in forecasting experiments.
  • Reviewer 1Hh3 appreciates the “breadth of capabilities in one model” and the use of distributional forecasting scores; the forecasting analysis is highlighted as “more informative than a single deterministic reconstruction score.”

1. Summary of rebuttal & clarifications

We address each reviewer’s individual questions in separate replies to their reviews. In summary, the main additions are:

  • All reviewers raised the question of whether chaining improves generation only due to extra compute at inference time. We ran chained and direct generation at two matched budgets and showed that increasing the compute budget beyond the paper setting (i.e., increasing decoding iterations) does not help direct generation. FVD changes little and non-monotonically with more iterations, while chaining at a matched compute budget gives significant FVD gains. We report the full comparison in terms of decoding steps, FLOPs, and wall-clock latency on one H200 GPU (Sec. 2 of this general response).
  • Reviewer aVda. The reviewer noted missing comparisons with EgoM2P, OmniVDiff, and UniVideo, and requested a tokenizer ablation. We discuss how these methods differ in their scope and training setups, which make direct comparisons non-trivial. Where tasks overlap on the same data distributions, we provide quantitative comparisons. We also compare our tokenizer against other tokenization methods to motivate our design choice of using FSQ-based tokenization.
  • Reviewers X6o2, 59yg, and 1Hh3. Reviewers asked for more details on the forecasting mixture’s composition, selection procedure, test-time cost, and uncertainty. We address these in detail in Sec. 3 of this general response.
  • Reviewer 59yg. The reviewer suggested discussing and positioning Video-4M against closely related works. We now discuss UnityVideo, UniVidX, GenCeption, and One4D in the related work section. While these works explore unified or bidirectional video modeling, Video-4M is uniquely positioned by its joint traversal of both the modality and temporal axes, with higher-level capabilities such as chained generations with arbitrary selection of modalities and future prediction in multiple spaces that are not been explored in prior video work. We have narrowed the positioning claim in the revised manuscript accordingly. The reviewer also requested a more systematic counterfactual evaluation; we provide quantitative results for bounding-box, caption and depth controls for conditional RGB generation.
  • Revised manuscript. We have also updated the manuscript based on the reviewers' comments and uploaded the revised version, with the changes reflected in the main paper and the appendix.
AuthorsOpenReview ↗

General Response (2/3): Chained vs. Direct Generation at Matched Compute

2. Common question: does chained generation help only because it uses more inference compute? To answer this, we ran chained and direct generation at two compute budgets using the same Video-4M-Large checkpoint. For direct generation, we vary the inference compute by changing the number of decoding steps; for chaining, we vary the intermediate modalities to control the compute budget. We match budgets based on FLOPs and report latency, decoding steps, generated tokens, and FVD. FLOPs are measured with PyTorch's FLOP counter over the complete inference steps required to generate all target tokens (including classifier-free guidance, excluding tokenizer decoding). All runs use ROAR-style decoding (4M; Mizrahi et al., 2023).

Low budget: We use a single intermediate modality (→SigLIP 2→RGB) and match direct generation to the same FLOPs budget.

High budget: We use the full coarse-to-fine chain (Sec. 3.4, main paper) and match direct generation accordingly by increasing decoding steps. For frame-conditioned tasks, direct generation reaches its maximum of one token per step (1,024 / 768 steps), which is still below the chain's FLOPs.

Table 1. Chained vs. direct generation at matched compute budgets (Video-4M-Large). D = Direct, C = Chain. FVD lower is better, mean of 3 seeds. *Paper setting, from Tab. 2 of the main paper. ≈: approximate average, since caption, transcription and bounding-box tokens are decoded autoregressively until end-of-sequence and their length varies per video (≈43, ≈134 and ≈400 tokens on average, respectively).

BudgetTaskTFLOPs DTFLOPs CSteps DSteps CLatency (s) DLatency (s) CTokens DTokens CFVD↓ DFVD↓ C
Paper* (direct only)Class→video104–100–4.1–1,280–330.6–
1 frame→video37–60–1.9–1,024–188.1–
5 frames→video44–60–2.1–768–117.1–
Low (→SigLIP 2→RGB)Class→video2382342251508.07.51,2802,260330.7254.1
1 frame→video1931863001506.35.71,0242,004184.8144.4
5 frames→video2172112901507.06.47681,748122.9106.3
High (full chain)Class→video1,1811,1841,000≈78036.931.91,280≈4,100330.9225.4
1 frame→video6247431,024≈78019.924.11,024≈3,900209.9130.0
5 frames→video551833768≈78017.725.9768≈3,600125.7101.6

The results show three things:

  • More decoding steps do not meaningfully help direct generation. FVD changes little and non-monotonically as we increase the budget (e.g., class FVD between 314.1 and 330.9 from 100 to 1,000 steps, App. C.5) and is worst at the highest budget for frame-conditioned generation, while alignment also drops (class top-1 68.8 at 100 steps vs. 64.8 at 1,000 steps, mean of 3 seeds). We note that this is consistent with prior masked generative models, where performance usually saturates at high numbers of decoding iterations (MaskGIT; Chang et al., 2022).
  • At matched compute, chaining is clearly better across all tasks and both budgets.
  • The number of generated tokens does not explain the gain. The chain generates more tokens than direct (e.g., 2,260 vs. 1,280 for class at low budget), yet chains through less useful intermediates are worse than direct despite also generating more tokens, as shown in the main paper for class→video (Table 3: Depth→RGB 446.3 FVD with 2,560 generated tokens, Caption→Transcription→RGB 425.6 FVD with ≈1,460 tokens, vs. 330.6 FVD for direct with 1,280 tokens in Table 2). This suggests that performance gains come from the choice of intermediate representation space and their order, not from extra compute alone.

We have added this table and the full decoding-step sweep to the appendix (App. C.5) and referenced it in the main paper. Note that in this analysis, we compare chained and direct generation at matched FLOPs. In Fig. 15 (App. C.5) of the revised manuscript, we show that the same observations hold when we instead compare them in terms of other inference compute metrics, namely latency and the number of decoding steps.

AuthorsOpenReview ↗

General Response (3/3): Oracle-Free Modality Mixture

3. Common question: how exactly is the oracle-free modality mixture built, and how does it compare to the oracle?

Method. We observe the first five RGB frames and the goal is to forecast the future RGB frames. Our Video-4M can take different routes, e.g., observed RGB → future bounding boxes → future RGB, observed RGB → future SigLIP 2→ future RGB, etc. Each route generates RGB futures, but its intermediate representation emphasizes different information: object locations or semantic content etc. The mixture pools forecasts from several routes, since different routes can produce complementary plausible futures, which can describe uncertainty better than generating all rollouts on one route.

Selection procedure. On a validation set of 528 clips held out from the development set (not used for training), we exhaustively search all 38,760 allocations of 14 samples across the seven intermediate modalities, compute the mean energy score of each allocation, and select the one with the lowest score. The selected allocation is then evaluated without modification on 472 held-out test clips. For RGB, the selected mixture is 10 bounding-box, 3 SigLIP 2 and 1 transcription rollouts, i.e.,

pmix(Y∣X)=1014pBBox(Y∣X)+314pSigLIP2(Y∣X)+114pTranscription(Y∣X),p_{\mathrm{mix}}(Y \mid X) = \tfrac{10}{14} p_{\mathrm{BBox}}(Y \mid X) + \tfrac{3}{14} p_{\mathrm{SigLIP 2}}(Y \mid X) + \tfrac{1}{14} p_{\mathrm{Transcription}}(Y \mid X),

where XX is the observed prefix and YY a possible future. The other targets are selected in the same way (depth: 5 bounding-box, 3 SigLIP 2, 2 optical flow, 2 RGB, 2 transcription; DINOv2: 5 bounding-box, 4 transcription, 3 RGB, 1 depth, 1 SigLIP 2; bounding boxes: 5 SigLIP 2, 4 transcription, 3 depth, 2 normal).

Test-time budget. Every method uses the same 14 forecasts per clip, so the mixture has exactly the same number of rollouts as the direct forecast, any fixed route, and the oracle. The allocation is fixed before test time, so the mixture needs no per-clip selection and no access to the future.

Table 2. Mean energy score on 472 held-out test clips, averaged over frames 6–16, with 14 forecasts per method (lower is better; scales differ across targets). Brackets: paired bootstrap 95% CIs over test clips (10,000 resamples). Gap closed = (best fixed − mixture) / (best fixed − oracle).

Forecast targetDirectBest fixed routeMixtureOracleMixture − best fixedGap closed
RGB8.3667.850 (BBox)7.7877.538-0.063 [-0.085, -0.041]20.1% [14.3, 25.6]
Depth0.4550.430 (BBox)0.4190.384-0.011 [-0.015, -0.007]23.9% [15.6, 31.2]
DINOv2254.50251.59 (optical flow)244.13236.00-7.46 [-8.95, -6.03]47.8% [40.6, 54.7]
BBox0.2580.251 (RGB)0.2490.210-0.002 [-0.007, +0.003]5.3% [−9.1, 17.0]

We make three observations:

  • The mixture improves over direct forecasting on all four targets (RGB, depth, DINOv2 and bounding boxes).
  • It has a lower energy score than the best fixed route on every target (by 0.8–3.0%); the paired confidence intervals show that this improvement is significant for RGB, depth and DINOv2, where the mixture is better on 61%, 59% and 71% of clips, while for bounding boxes the difference is within noise.
  • The oracle, which selects the route per clip using the realized future, remains better on all four targets, and the mixture recovers 5–48% of this gap, the most for DINOv2.

This suggests that pooling complementary routes already gives a practical, oracle-free gain, and the remaining headroom motivates clip-specific route selection without access to the future. We have now added this specification and the table to App. D.1.

AuthorsOpenReview ↗

General Response and Significance Statement

We thank the reviewers for their time and constructive feedback, and the AC for handling our submission. We believe that the additional analyses and clarifications during the rebuttal have further strengthened our work. Below, we also provide a statement on its potential broader impact.

Significance Statement

Relevance to other sequential domains. We believe that the broader impact of our work goes beyond extending 4M to the video domain (still significant on its own). Importantly, we propose a thoroughly evaluated design for handling an additional sequential structure in the data, which is directly relevant to other domains such as audio, time series, or interleaved data. Prior any-to-any models such as 4M were mainly developed for static images and offer limited design guidance for such domains. Similarly, prior multimodal video models mainly traverse modalities, and the exploration of higher-level emergent capabilities through joint traversal across the modality-time grid is limited. Video-4M helps address this gap, which required non-trivial design changes to handle the temporal axis, e.g., forecasting-oriented masking modes for both sequential and non-sequential modalities that improve the model’s forecasting capabilities (Sec. 3.3 and Tab. 8).

A step towards world models in multimodal abstract spaces. Furthermore, we see this work as a step towards world models operating in (multimodal) abstract spaces. It is an open question whether world modeling should happen in raw sensory space (e.g., RGB video), in a latent space (as in JEPA-style world models), or or in other structured abstractions (e.g., bounding boxes, depth). Our work propose a joint multimodal modeling as a principled alternative to an either-or choice: a single model represents one underlying physical reality through many modalities at once and can traverse across them without committing to any one, so raw observations, latents, and other abstractions can all be present simultaneously. This shifts the question from which space to model in to how to effectively model a rich hierarchy of modalities that accommodate all ranks of abstraction. Our findings on chained generation (Sec. 4.2) and forecasting across representation spaces (Sec. 4.3) are promising steps in this direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.