- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper extends 4M-style multimodal masked modeling to video. It constructs aligned pseudo-labels on Kinetics-600, tokenizes dense and sequential representations, and trains a shared encoder-decoder with temporal embeddings and forecasting masks. The resulting Video-4M supports cross-modal prediction, generation through intermediate modalities, and forecasting in different representation spaces. Experiments cover several generation and perception tasks, compare direct and chained inference, and evaluate probabilistic forecasts using energy scores.
Critical strengths
-
The breadth of capabilities in one model is a meaningful contribution. RGB, geometry, semantic features, and structured trajectories share a prediction interface, and the temporal masking design makes this interface useful for forecasting as well as cross-modal completion. Although the backbone follows 4M, the video tokenization, aligned dataset, and temporal training setup constitute substantial work. The value here is the flexible model rather than winning every specialist benchmark.
-
The same-model comparisons provide convincing evidence that intermediate representations can help generation. For the large model, Table 2 reduces one-frame prediction FVD from 188.1 to 129.5 and caption-conditioned FVD from 220.8 to 180.2. I also appreciate the use of distributional forecasting scores and the explicit separation of the best-chain oracle from the validation-selected modality mixture in Appendix D.1. These make the forecasting analysis more informative than a single deterministic reconstruction score.
Critical weaknesses
-
The generation gains are not separated from the additional inference budget. Chained inference generates several intermediate sequences before RGB, whereas direct inference only generates the target. The matched training schedule and same-model comparison are useful controls, but they do not establish whether the improvement comes from the choice of representation or simply from spending more computation at inference. This matters because decomposition through modalities is a central claim. Sampling iterations, generated token counts, and latency for both settings, together with a direct baseline at a comparable budget, would make the evidence substantially stronger.
-
The proposed coarse-to-fine recipe has a less uniform advantage than the headline suggests. In Table 3, semantic-only chaining gives better caption-to-video FVD than the full chain (167.1 versus 180.2). In Appendix C.3, omitting depth also gives better class-conditioned FVD (215.6 versus 225.2), with a small accuracy trade-off. The paper does acknowledge these trade-offs, but the evidence supports task-dependent chain design more strongly than a generally preferred ordering. The validation-selected forecasting mixture is promising; its selection procedure and performance uncertainty need to be specified more precisely before drawing a broader conclusion about choosing representation spaces.
Questions for authors
-
At a matched inference budget, how much of the Table 2 improvement remains? Please specify the sampling schedule, generated token budget, and measured latency of the direct and chained variants.
-
For the oracle-independent mixture in Appendix D.1, what exact candidate chains, mixture weights or sample allocations, and validation selection procedure were used? Can you provide numerical energy-score improvements with uncertainty, and distinguish this result from the per-sample oracle?
Overall, I lean toward weak acceptance. The unified video model is useful and the empirical study contains several informative findings, even though the efficiency and chain-selection claims need a more careful qualification.
4 comments
Authors’ Response to Reviewer 1Hh3 (2/2)
For the oracle-independent mixture in Appendix D.1, what exact candidate chains, mixture weights or sample allocations, and validation selection procedure were used? Can you provide numerical energy-score improvements with uncertainty […]? The validation-selected forecasting mixture is promising; its selection procedure and performance uncertainty need to be specified more precisely before drawing a broader conclusion about choosing representation spaces.
We thank the reviewer for finding the validation-selected mixture promising. As suggested, we detail the exact candidate chains, the sample allocations for every target, and the validation selection procedure in Sec. 3 of the general response. In short, the allocation of 14 samples across seven intermediate routes is selected by exhaustive search (38,760 candidates) on 528 held-out validation clips, and evaluated without modification on 472 test clips.
With paired bootstrap 95% confidence intervals, the mixture improves significantly over the best fixed route for RGB, depth and DINOv2, but not for bounding boxes, and recovers 5–48% of the oracle gain (Table 2 of the general response). We therefore phrase the conclusion more precisely: the best forecasting representation depends on the clip (oracle), and pooling complementary routes already gives a measurable oracle-free gain, while choosing the representation per clip without access to the future remains open. We have added this specification to App. D.1 of the revised manuscript.
Post-rebuttal assessment
Thank you for the detailed response. I have raised my rating from 3 to 4. The new compute-matched comparisons support the benefit of chaining beyond extra decoding, and the forecasting response specifies the validation-selected mixture, fixed rollout allocations, and paired confidence intervals. Together with the clearer discussion of task-dependent chain choices, these additions address my main concerns.
Thank you!
We sincerely thank the reviewer for taking the time to provide constructive feedback and for raising the score. We have incorporated these discussions and results into the revised manuscript, and we believe they have further strengthened the paper.
Authors’ Response to Reviewer 1Hh3 (1/2)
We thank reviewer 1Hh3 for the positive feedback. We address their remaining questions below.
We refer the reviewer to our full response in Sec. 2 of the general response, which reports decoding steps, generated tokens, FLOPs and measured latency for different task settings (Table 1). To summarize:
We have added the full breakdown and quality-cost plots to App. C.5 of the revised manuscript.
We thank the reviewer for this careful reading, and we agree that the evidence supports task-dependent chain design. We would like to clarify that the main goal of this work is to show that chained generation, made possible by training a single video model across the modality-time grid, improves over direct generation. As we already acknowledge in the main paper (paragraph "Optimal chain design" in Sec. 3.4), the ideal chain is likely task-dependent, and with n modalities there are n! possible orderings, so a systematic search per task is beyond the scope of this work. We therefore use a single, general coarse-to-fine chain for all tasks, which aims to achieve the best performance trade-off across tasks, instead of optimizing the chain for each task or metric. In our experiments, this coarse-to-fine chain improves upon direct generation on all tasks (Tab. 2 of the main paper), and among the configurations in Tab. 3 of the main paper, it gives the best FVD on 3 of the 4 tasks.
At the same time, different intermediate modalities improve different aspects of generation: semantic modalities improve FVD and alignment, while depth improves structural detail (Tab. 3 of the main paper and Tab. 12 of the revised manuscript). Since Video-4M supports any prediction route at inference without retraining, the chain can also be chosen for a specific task or metric, e.g., a semantic-only chain when caption alignment is the priority. We have highlighted these trade-offs upfront in the chained generation section (Sec. 4.2) of the revised manuscript.