- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Video-Point Model (VPM), a framework that jointly forecasts future multi-view videos and persistent 3D point trajectories. Built upon Cosmos 3, the method introduces camera-conditioned multi-view processing and an auxiliary trajectory prediction head, using point trajectory and correspondence supervision to improve the spatial and temporal consistency of video generation. The authors also introduce VPM-Data and VP-WorldBench, covering simulated and real-world robot manipulation and urban dynamics. Extensive experiments demonstrate improvements in video generation, 3D motion prediction, and downstream robot manipulation.
Critical strengths
-
Jointly forecasting future videos and 3D point trajectories is a meaningful direction for 4D world modeling. The paper presents a coherent framework that connects visual prediction, geometric motion modeling, and robot action learning.
-
The experimental evaluation is extensive. In particular, the comparison between VPM (V) and VPM (V+T) provides clear evidence that trajectory supervision improves video generation and geometric prediction. The additional comparison with per-frame depth and pointmap supervision further supports the benefits of persistent point trajectories.
-
The paper contributes both a method and new data/benchmark resources. The evaluation covers different domains, including robot manipulation and general dynamic scenes, as well as simulated and real-world environments. The inclusion of downstream robot manipulation experiments further strengthens the practical relevance of the work.
Critical weaknesses
-
Lack of direct validation of video–trajectory consistency. A central claim of VPM is that jointly forecasting future videos and persistent 3D point trajectories encourages a coherent representation of future 4D scene dynamics. However, the consistency between these two predicted outputs is not directly evaluated. While the trajectory loss supervises future point positions and the correspondence loss encourages cross-view feature alignment, neither directly enforces agreement between predicted 3D trajectories and the motion depicted in generated videos. Tables 2 and 19 separately evaluate video-derived motion and trajectory-head predictions against ground truth, but do not measure their mutual agreement. Figure 4 provides qualitative evidence, which is insufficient to establish this claim quantitatively. A direct comparison between the predicted point trajectories and trajectories independently recovered from the generated videos would provide stronger evidence of coherent joint forecasting.
-
Limited methodological novelty beyond existing video–geometry modeling frameworks. The methodological novelty of VPM remains insufficiently established. Prior works have already explored integrating geometric and motion supervision into video world models. In particular, MVTrack4Gen employs a closely related design, and 4D-WAM has investigated trajectory-based supervision for world-action modeling. The limited comparisons with related geometry- and trajectory-aware methods make VPM’s advantages unclear.
Questions for authors
-
How consistency between these two predicted outputs is explicitly established beyond joint supervision of the shared backbone. Although the correspondence loss encourages cross-view feature alignment, it does not directly enforce consistency between generated videos and predicted 3D trajectories. The experiments mainly evaluate video quality and trajectory accuracy separately, without directly measuring their mutual consistency.
-
Concern about some claims. Some statements in Sec.Intro like “Unlike conventional tracking, both predictions extend beyond the observed history” (Line 63) does not sufficiently distinguish the proposed method from existing future world modeling approaches. Prior works have already explored joint prediction or modeling of future visual appearance and 3D geometry or motion. The motivation should more clearly establish what remains unresolved by these approaches and what specific advantages persistent 3D trajectory forecasting provides.
-
Future camera-conditioning protocol. Section 3.1 assumes camera parameters for the requested future views, while Section 6.1 states that future camera parameters are unavailable in the main evaluation. Could the authors clarify how camera-dependent embeddings for future video tokens are constructed when future camera poses are unknown, and how this differs from the camera-conditioned evaluation in Appendix E?
2 comments
Thank you for your reply. I have no more question about this paper.
Response to Reviewer KFjt
Dear reviewer KFjt,
We sincerely appreciate your thoughtful and positive review. We address each of your concerns below. In the revised draft, we mark our major revisions as “blue”.
[W1 & Q1] Video–trajectory consistency.
While we address this in General Response [G1], we include the response here as well, for completeness.
Reviewers asked whether the predicted trajectories and the generated videos describe the same motion. Our current metrics do not test this directly: MEt3R measures consistency across generated views, and Tables 2 and 19 compare each output with ground truth, not with each other. Below, we first explain why the two outputs should agree, and then following reviewers’ suggestion, we measure how closely they do.
Why the outputs should agree. VPM has no hard constraint between them, but they are tied by design. The trajectory head reads the same intermediate features the generator uses to produce the video, and it locates each point by matching these features across views and frames. Lcorr further trains these features to associate the same physical point. At inference, the trajectories are read from the sample being generated, so they describe that sample rather than a separate forecast.
Additional Experimental Setup. Under the protocol of Table 2, we track the head’s query points through each generated video with the same MoGe-2 + MVTracker pipeline. We then evaluate Eq. (9) with these video-derived tracks as reference. If the head describes its own video, agreement should exceed that against the real future video (tracked the same way) and approach the tracker's ceiling on real videos (Table 16).
Table R1. Agreement (δavg3D ↑)
The head agrees more with its own video than with the real future and approaches the tracker ceiling. It therefore describes the generated motion rather than making a separate forecast, and it remains the most accurate against ground truth (Table 19). Agreement is lower for VPM (V), so point supervision tightens the coupling.
We incorporated these updated results into the revised manuscript (see Table 34).
[W2 & Q2] Novelty and the advantage of persistent 3D trajectories.
We agree the designs are related: our tracking and correspondence objectives follow MVTrack4Gen, a concurrent ICLR 2027 submission, and we extend tracking from 2D to metric 3D (Sec. 3.4). Our contribution lies in the problem and the supervision:
In short, prior approaches supervise per-frame geometry, which does not link a point across time; 2D tracks, which lack metric depth; or feature alignment, which yields no explicit trajectories. Persistent 3D trajectory forecasting provides all three, and our controlled comparisons show that persistence and metric depth each contribute substantially (Tables 23, 30).
L63 was meant to contrast with tracking, not with world modeling. We revised it to state this motivation in the updated manuscript.
[Q3] Future camera-conditioning protocol.
Sec. 3.1 describes the general formulation, in which future poses are optional. For a fair comparison with baselines, the main evaluation (Tables 2–4) does not provide them. Future tokens then carry no camera embedding and are identified only by the view axis of our 4D RoPE, so the model infers the camera motion itself. In Appendix E ("+ camera cond."), the same model receives ground-truth future poses through the same camera embedding as the history, which improves every metric on every split (Table 16).