acceptodds
Under review as a conference paper at ICLR 2027

Learning the 4D World through Point Trajectory Forecasting

15 views

Abstract

Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. We introduce Video-Point Model (VPM), which, given posed multi-view video histories and a language instruction, jointly forecasts future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Because each trajectory follows a single point across all views, supervising it alongside video generation ties appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, so the policy is built on the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises manipulation success of the action-enabled model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
world modeling, 3d point forecasting, video generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KFjtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Video-Point Model (VPM), a framework that jointly forecasts future multi-view videos and persistent 3D point trajectories. Built upon Cosmos 3, the method introduces camera-conditioned multi-view processing and an auxiliary trajectory prediction head, using point trajectory and correspondence supervision to improve the spatial and temporal consistency of video generation. The authors also introduce VPM-Data and VP-WorldBench, covering simulated and real-world robot manipulation and urban dynamics. Extensive experiments demonstrate improvements in video generation, 3D motion prediction, and downstream robot manipulation.

Critical strengths
  1. Jointly forecasting future videos and 3D point trajectories is a meaningful direction for 4D world modeling. The paper presents a coherent framework that connects visual prediction, geometric motion modeling, and robot action learning.

  2. The experimental evaluation is extensive. In particular, the comparison between VPM (V) and VPM (V+T) provides clear evidence that trajectory supervision improves video generation and geometric prediction. The additional comparison with per-frame depth and pointmap supervision further supports the benefits of persistent point trajectories.

  3. The paper contributes both a method and new data/benchmark resources. The evaluation covers different domains, including robot manipulation and general dynamic scenes, as well as simulated and real-world environments. The inclusion of downstream robot manipulation experiments further strengthens the practical relevance of the work.

Critical weaknesses
  1. Lack of direct validation of video–trajectory consistency. A central claim of VPM is that jointly forecasting future videos and persistent 3D point trajectories encourages a coherent representation of future 4D scene dynamics. However, the consistency between these two predicted outputs is not directly evaluated. While the trajectory loss supervises future point positions and the correspondence loss encourages cross-view feature alignment, neither directly enforces agreement between predicted 3D trajectories and the motion depicted in generated videos. Tables 2 and 19 separately evaluate video-derived motion and trajectory-head predictions against ground truth, but do not measure their mutual agreement. Figure 4 provides qualitative evidence, which is insufficient to establish this claim quantitatively. A direct comparison between the predicted point trajectories and trajectories independently recovered from the generated videos would provide stronger evidence of coherent joint forecasting.

  2. Limited methodological novelty beyond existing video–geometry modeling frameworks. The methodological novelty of VPM remains insufficiently established. Prior works have already explored integrating geometric and motion supervision into video world models. In particular, MVTrack4Gen employs a closely related design, and 4D-WAM has investigated trajectory-based supervision for world-action modeling. The limited comparisons with related geometry- and trajectory-aware methods make VPM’s advantages unclear.

Questions for authors
  1. How consistency between these two predicted outputs is explicitly established beyond joint supervision of the shared backbone. Although the correspondence loss encourages cross-view feature alignment, it does not directly enforce consistency between generated videos and predicted 3D trajectories. The experiments mainly evaluate video quality and trajectory accuracy separately, without directly measuring their mutual consistency.

  2. Concern about some claims. Some statements in Sec.Intro like “Unlike conventional tracking, both predictions extend beyond the observed history” (Line 63) does not sufficiently distinguish the proposed method from existing future world modeling approaches. Prior works have already explored joint prediction or modeling of future visual appearance and 3D geometry or motion. The motivation should more clearly establish what remains unresolved by these approaches and what specific advantages persistent 3D trajectory forecasting provides.

  3. Future camera-conditioning protocol. Section 3.1 assumes camera parameters for the requested future views, while Section 6.1 states that future camera parameters are unavailable in the main evaluation. Could the authors clarify how camera-dependent embeddings for future video tokens are constructed when future camera poses are unknown, and how this differs from the camera-conditioned evaluation in Appendix E?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KFjt

Dear reviewer KFjt,

We sincerely appreciate your thoughtful and positive review. We address each of your concerns below. In the revised draft, we mark our major revisions as “blue”.


[W1 & Q1] Video–trajectory consistency.

While we address this in General Response [G1], we include the response here as well, for completeness.

Reviewers asked whether the predicted trajectories and the generated videos describe the same motion. Our current metrics do not test this directly: MEt3R measures consistency across generated views, and Tables 2 and 19 compare each output with ground truth, not with each other. Below, we first explain why the two outputs should agree, and then following reviewers’ suggestion, we measure how closely they do.

Why the outputs should agree. VPM has no hard constraint between them, but they are tied by design. The trajectory head reads the same intermediate features the generator uses to produce the video, and it locates each point by matching these features across views and frames. Lcorr\mathcal{L}_{\text{corr}} further trains these features to associate the same physical point. At inference, the trajectories are read from the sample being generated, so they describe that sample rather than a separate forecast.

Additional Experimental Setup. Under the protocol of Table 2, we track the head’s query points through each generated video with the same MoGe-2 + MVTracker pipeline. We then evaluate Eq. (9) with these video-derived tracks as reference. If the head describes its own video, agreement should exceed that against the real future video (tracked the same way) and approach the tracker's ceiling on real videos (Table 16).

Table R1. Agreement (δavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow)

Sim RobotSim GeneralReal RobotReal General
VPM (V+T): head vs. own video67.9155.4072.9451.23
VPM (V+T): head vs. real future video47.5933.5663.4131.08
VPM (V): head + own video63.3550.6168.2446.86
Ceiling: tracker on real video vs. GT73.2761.8780.4057.98

The head agrees more with its own video than with the real future and approaches the tracker ceiling. It therefore describes the generated motion rather than making a separate forecast, and it remains the most accurate against ground truth (Table 19). Agreement is lower for VPM (V), so point supervision tightens the coupling.

We incorporated these updated results into the revised manuscript (see Table 34).


[W2 & Q2] Novelty and the advantage of persistent 3D trajectories.

We agree the designs are related: our tracking and correspondence objectives follow MVTrack4Gen, a concurrent ICLR 2027 submission, and we extend tracking from 2D to metric 3D (Sec. 3.4). Our contribution lies in the problem and the supervision:

  • vs. MVTrack4Gen: it synthesizes novel views of an observed video with 2D tracks, whereas VPM forecasts unseen future frames across posed views with metric 3D trajectories and extends to action learning.
  • vs. 4D-WAM (concurrent arXiv work): it aligns features with trajectory cues, whereas VPM explicitly forecasts persistent trajectories, enabling direct consistency checks (Table R1) and zero-shot control (App. A.4).
  • vs. Robot4DGen and X-WAM (the latter also a concurrent ICLR 2027 submission): both forecast per-frame geometry (pointmaps and RGB-D, respectively). VPM outperforms both (Tables 2, 4, 5). Notably, VPM (V) trails Robot4DGen on geometric consistency, while VPM (V+T) surpasses it (Table 4).
  • What persistence and 3D add: under an identical setup, replacing our trajectories with per-frame depth or pointmap supervision recovers only 19% or 28% of the FVD gain (Table 23). Adding metric depth to 2D tracks raises the recovered gain from 48% to 68% (Table 30).

In short, prior approaches supervise per-frame geometry, which does not link a point across time; 2D tracks, which lack metric depth; or feature alignment, which yields no explicit trajectories. Persistent 3D trajectory forecasting provides all three, and our controlled comparisons show that persistence and metric depth each contribute substantially (Tables 23, 30).

L63 was meant to contrast with tracking, not with world modeling. We revised it to state this motivation in the updated manuscript.


[Q3] Future camera-conditioning protocol.

Sec. 3.1 describes the general formulation, in which future poses are optional. For a fair comparison with baselines, the main evaluation (Tables 2–4) does not provide them. Future tokens then carry no camera embedding and are identified only by the view axis of our 4D RoPE, so the model infers the camera motion itself. In Appendix E ("+ camera cond."), the same model receives ground-truth future poses through the same camera embedding as the history, which improves every metric on every split (Table 16).

Thank you for your reply. I have no more question about this paper.

Reviewer_WrLmOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Video-Point Model (VPM), which, based on multi-view historical videos, camera parameters and language instructions, jointly predicts future videos and continuous three-dimensional point trajectories, and extends to the generation of robot actions. The method enhances the motion representation of the video model through trajectory prediction and cross-view correspondence supervision. Meanwhile, VPM-Data and VP-WorldBench are constructed for training and evaluating the capabilities of visual prediction, three-dimensional motion and robot control.

Critical strengths

The method introduces continuous three-dimensional point trajectory supervision into video prediction, providing a clear learning signal for cross-view and cross-time motion modeling. The experiment covers simulation, real scenarios and robot control, and through comparison with the model and dissolution with frame-by-frame geometric supervision, it fully supports the value of point trajectory supervision. The assessment simultaneously includes video quality, 3D motion accuracy and task success rate, providing a relatively complete empirical support for the contribution.

Critical weaknesses

None

Questions for authors
  1. Can the trajectories predicted by the trajectory head be quantitatively evaluated as consistent with those extracted in the same generated video to confirm whether they describe the same future movement?
  2. Furthermore, could explicitly enhancing this consistency (such as adding consistency loss) further improve the performance of motion prediction and robot control?
3 comments
AuthorsOpenReview ↗

Response to Reviewer WrLm

Dear reviewer WrLm,

We sincerely appreciate your thoughtful and positive review. We address each of your questions below. In the revised draft, we mark our major revisions as "blue".


[Q1] Are the head's trajectories consistent with those extracted from the same generated video?

While we address this in General Response [G1], we include the response here as well, for completeness.

Reviewers asked whether the predicted trajectories and the generated videos describe the same motion. Our current metrics do not test this directly: MEt3R measures consistency across generated views, and Tables 2 and 19 compare each output with ground truth, not with each other. Below, we first explain why the two outputs should agree, and then following reviewers’ suggestion, we measure how closely they do.

Why the outputs should agree. VPM has no hard constraint between them, but they are tied by design. The trajectory head reads the same intermediate features the generator uses to produce the video, and it locates each point by matching these features across views and frames. Lcorr\mathcal{L}_{\text{corr}} further trains these features to associate the same physical point. At inference, the trajectories are read from the sample being generated, so they describe that sample rather than a separate forecast.

Additional Experimental Setup. Under the protocol of Table 2, we track the head’s query points through each generated video with the same MoGe-2 + MVTracker pipeline. We then evaluate Eq. (9) with these video-derived tracks as reference. If the head describes its own video, agreement should exceed that against the real future video (tracked the same way) and approach the tracker's ceiling on real videos (Table 16).

Table R1. Agreement (δavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow)

Sim RobotSim GeneralReal RobotReal General
VPM (V+T): head vs. own video67.9155.4072.9451.23
VPM (V+T): head vs. real future video47.5933.5663.4131.08
VPM (V): head + own video63.3550.6168.2446.86
Ceiling: tracker on real video vs. GT73.2761.8780.4057.98

The head agrees more with its own video than with the real future and approaches the tracker ceiling. It therefore describes the generated motion rather than making a separate forecast, and it remains the most accurate against ground truth (Table 19). Agreement is lower for VPM (V), so point supervision tightens the coupling.

We incorporated these results into the revised manuscript (see Table 34).


[Q2] Would an explicit consistency loss further improve motion prediction and robot control?

Following your suggestion, we add an explicit consistency loss Lcons\mathcal{L}_{\text{cons}} to VPM (V+T), keeping all other settings unchanged: Lcons=1∣Ωf∣∑(n,v,τ)∈Ωfρ(S(x^0,τv,u^τ,nv)−S(x0,τv,uτ,nv))\displaystyle \mathcal{L}_{\text{cons}} = \frac{1}{|\Omega_f|} \sum_{(n,v,\tau)\in\Omega_f} \rho\Big( \mathcal{S}\big(\hat{\mathbf{x}}^{v}_{0,\tau}, \hat{\mathbf{u}}^{v}_{\tau,n}\big) - \mathcal{S}\big(\mathbf{x}^{v}_{0,\tau}, \mathbf{u}^{v}_{\tau,n}\big) \Big)

where x^0=xσ−σvθ\hat{\mathbf{x}}_0 = \mathbf{x}_\sigma - \sigma v_\theta is the denoised video latent, S(⋅,u)\mathcal{S}(\cdot, \mathbf{u}) samples a latent at position u\mathbf{u}, u^\hat{\mathbf{u}} and u\mathbf{u} are the predicted and ground-truth point positions, ρ\rho is the Huber penalty, and Ωf\Omega_f is the set of visible future points.


Table R3. Effect of Lcons\mathcal{L}_{\text{cons}} on VP-WorldBench.

Sim Robot

MethodFVD ↓\downarrowPSNR ↑\uparrowMEt3R ↓\downarrowδavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow
VPM (V+T)67.3522.080.03450.05
VPM (V+T) + Lcons\mathcal{L}_{\text{cons}}67.0122.140.03450.47

Sim General

MethodFVD ↓\downarrowPSNR ↑\uparrowMEt3R ↓\downarrowδavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow
VPM (V+T)81.9219.160.04636.80
VPM (V+T) + Lcons\mathcal{L}_{\text{cons}}81.4919.180.04537.11

Real Robot

MethodFVD ↓\downarrowPSNR ↑\uparrowMEt3R ↓\downarrowδavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow
VPM (V+T)255.1920.570.05264.75
VPM (V+T) + Lcons\mathcal{L}_{\text{cons}}252.9820.640.05265.16

Real General

MethodFVD ↓\downarrowPSNR ↑\uparrowMEt3R ↓\downarrowδavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow
VPM (V+T)56.4822.100.05533.92
VPM (V+T) + Lcons\mathcal{L}_{\text{cons}}54.8922.130.05435.00

Lcons\mathcal{L}_{\text{cons}} further improves forecasting, indicating that tighter coupling between the two outputs is beneficial. Since action tokens share this backbone (Sec. 5), we expect the gain to carry over to robot control, as point supervision does (Table 5).

We incorporated these results into the revised manuscript (see Table 22).

Thanks for your detailed reply and your updated manuscript. My concerns have been addressed. Overall, the research topic is interesting, and the paper’s contributions are substantial, so I will maintain my score

Dear reviewer WrLm,

Thank you for your thoughtful comments and for taking the time to review our manuscript. We are pleased to hear that our replies and the updated manuscript have addressed your concerns, and we sincerely appreciate your recognition of the significance of our research topic and contributions.

We believe that VPM can make a meaningful contribution to future directions in 4D world modeling, and we are committed to making it a valuable resource for the research community through the revised manuscript and our open-source release.

Best regards,
Authors

Reviewer_6bP6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary:

This paper deals with forecasting videos from arbitrary viewpoints and 3D point tracks of persistent scene points given posed multi-view video histories and a language instruction. Towards this end, this work introduces Video-Point Model (VPM) which extends a pre-trained world model with camera-conditioned multi-view processing by integrating complementary observations across cameras, and adding a trajectory head that predicts how physical points move through time. In order to support the training and evaluation of the model, this work proposes VPM-Data - a simulation corpus which pairs synchronized multi-view videos with persistent point trajectories and VP-WorldBench - a forecasting benchmark spanning simulated and real environments. Model performances of competing techniques are evaluated on VP-WorldBench and other 4D generation benchmarks. Additionally, the work also extends VPM to robot control by adding robot actions as a token modality to the generator.

Critical strengths

The main strengths of the paper are as follows:

  1. Unlike approaches that only estimate geometry or track points within observed videos, VPM forecasts future 3D positions of persistent scene points while simultaneously generating future frames.
  2. Besides supervising the trajectory head, the cross-view correspondence supervision provides geometric information directly to the features used by the video generator, rather than relying exclusively on the trajectory head's final predictions.
  3. The VP-WorldBench benchmark is a useful contribution to the community. In particular, the benchmark contains scenarios where even if the motion of scene objects is occluded in some views, they are visible in others. This allows to test the capability of approaches to use multi-view information, since independent per-camera video prediction is likely to be inadequate.
  4. The proposed model exhibits strong performance compared to competing methods on several benchmarks.
Critical weaknesses

The key weaknesses of the paper are as follows:

  1. The proposed formulation does not explicitly guarantee consistency between the generated videos and predicted 3D trajectories.
  2. The 3D point supervision does not cover all kinds of dynamic objects, for example cloth and hair.
  3. A related prior work that has not been referenced [a].

References:

[a] Bharadhwaj, H. and Jangir, Y., 2026. MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction. arXiv preprint arXiv:2607.16192.

Questions for authors

Comments/Questions:

  1. How are camera poses for the future frames handled? Are they provided as input? Are they assumed to be the same as the last frame?
  2. Does 3D trajectory supervision provide greater benefits when targets are occluded?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6bP6

Dear reviewer 6bP6,

We sincerely appreciate your thoughtful and positive review. We address each of your concerns below. In the revised draft, we mark our major revisions as "blue".


[W1] No explicit guarantee of consistency between generated videos and predicted trajectories.

We agree there is no hard constraint. However, consistency is encouraged by design: the head is a correlation-based tracker on the generator's own features, LcorrL_{\text{corr}} supervises those features, and trajectories are decoded from the generated sample itself. Moreover, following the reviewer's valuable suggestion, we now measure the consistency directly (G1, Table R1).

Table R1. Agreement (δavg3D\delta^{\text{3D}}_{\text{avg}} ↑\uparrow)

Sim RobotSim GeneralReal RobotReal General
VPM (V+T): head vs. own video67.9155.4072.9451.23
VPM (V+T): head vs. real future video47.5933.5663.4131.08
VPM (V): head + own video63.3550.6168.2446.86
Ceiling: tracker on real video vs. GT73.2761.8780.4057.98

The head agrees more with its own video than with the real future and approaches the tracker ceiling. It therefore describes the generated motion rather than making a separate forecast, and it remains the most accurate against ground truth (Table 19). Agreement is lower for VPM (V), so point supervision tightens the coupling.

We incorporated these updated experimental results into the revised manuscript (see Table 34).


[W2] Point supervision does not cover all dynamic content (e.g., cloth, hair).

We agree and this is a limitation addressed in Sec. 7. We note that it stems from the annotation source rather than the method's supervision: our simulated humans are annotated through skeletal skinning, which does not capture independent cloth or hair motion, whereas the trajectory head and losses accept any persistent tracks. Hair is particularly challenging: its motion occurs at millimeter scale, which is extremely difficult to annotate accurately. Using such annotations would also require training at HD resolution or higher, whereas our model is trained at 256–480px (Table 7), where individual strands are sub-pixel. Point supervision nonetheless remains beneficial on non-rigid real scenes. VPM (V+T) improves over VPM (V) in both FVD and δavg3D\delta^{\text{3D}}_{\text{avg}} on real-world General (Table 2) and CMU Panoptic (Table 3), which feature real people in everyday clothing. Our trajectory head also achieves the lowest ADE and FDE on DAVIS (Table 24). Extending annotations to cloth simulation is a natural next step.


[W3] Missing reference to MotionForesight [a].

We kindly refer the reviewer to Sec. 2 (L123-124), where we already cited this concurrent work. MotionForesight forecasts only 3D tracks from monocular video, whereas VPM jointly forecasts multi-view videos and 3D trajectories and uses the latter to improve the former.


[Q1] How are future camera poses handled?

Future poses are an optional input during both training and inference. For a fair comparison with baselines, the main evaluation (Tables 2–4) does not provide them. Future tokens then carry no camera embedding and are identified only by the view axis of our 4D RoPE, so the model infers the camera motion itself rather than assuming the last pose. When ground-truth future poses are provided ("+ camera cond.", App. E), VPM improves every metric on every split (Table 16). A single model supports both settings. We incorporated details into the revised manuscript (see Sec. 3.1 and App. A.1).


[Q2] Does point supervision help more under occlusion?

Yes. VP-WorldBench's hard subsets contain clips in which individual cameras lose sight of the target for at least 30 consecutive frames while another camera still sees it (Sec. 4.2). Table R2 compares VPM (V) and VPM (V+T) on these clips and on the rest.

Table R2. VPM (V) vs. VPM (V+T) on normal and hard clips (I2V protocol of Table 2; gain over VPM (V) in parentheses; FVD is omitted as it is unreliable on small subsets).

MetricModelSim Robot Normal (473)Sim Robot Hard (27)Sim General Normal (405)Sim General Hard (95)
PSNR ↑\uparrowVPM (V)21.5219.6217.3215.37
VPM (V+T)22.16 (+0.64)20.71 (+1.09)19.39 (+2.07)18.19 (+2.82)
δavg3D\delta^{3D}_{\text{avg}} ↑\uparrowVPM (V)44.1234.9030.3321.84
VPM (V+T)50.36 (+6.24)44.60 (+9.70)37.82 (+7.49)32.46 (+10.62)

The gain from point supervision is larger on hard clips in both domains. Because each trajectory follows the same point across all cameras, the model can forecast a target’s motion from the views that still see it.

AuthorsOpenReview ↗

Gentle Reminder

Dear reviewer 6bP6,

We sincerely appreciate your time and thoughtful efforts in reviewing our manuscript. As the discussion period will end in three days, we would like to gently remind you in case you have any remaining comments. We believe that we have sincerely and successfully addressed your concerns, supported by the corresponding additional experimental results.

If you have any questions or concerns, please do not hesitate to let us know. We remain fully available for further discussion and are prepared to conduct additional experiments to address any remaining concerns during the discussion period.

Best regards,
Authors

Reviewer_6bP6OpenReview ↗

Response to Rebuttal

I thank the authors for addressing my principal concerns. I hold my rating.

Comments on the paper

1 comment

Dear reviewers,

Thanks for your continued efforts. The authors have provided a rebuttal to the initial reviews. Please review it carefully, along with all other reviews and the corresponding author responses, and consider whether you have any further questions for the authors or whether the rebuttal has addressed your concerns. There are only a few days remaining before the discussion period closes, so please use this time to seek any further clarifications or raise any additional questions about this paper.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.