acceptodds
Under review as a conference paper at ICLR 2027

Momentum: Persistent Dynamics for Video World Models

Abstract

Video world models must capture how a scene evolves beyond the current camera view to generate consistent observations when its content reappears. Existing approaches rely on observation history, static point clouds, or image caches constructed from past frames. These representations are often expensive and, more importantly, cannot update dynamic scene content outside the field of view: generated vehicles and pedestrians may be forgotten after leaving the field of view or reappear in physically inaccurate states. We represent an evolving scene through synchronized videos from reference-camera rigs, each comprising eight cameras that jointly cover the surrounding scene. We connect these rigs in a graph and propagate video generation between neighboring rigs to coordinate scene dynamics across locations and time. We introduce Momentum, a two-stage pipeline: WorldUpdate jointly generates reference-camera videos as a shared memory of scene dynamics, and WorldInteract grounds camera-controlled video generation in this evolving memory, allowing moving content to remain represented outside the target view and reappear consistently. To support training and evaluation, we develop a CARLA pipeline that captures synchronized videos from reference camera rigs and a moving target camera, together with an evaluation framework measuring consistency within the generated world state and the target observations. Experiments on generated street scenes show that grounding target videos in scene dynamics generated by Momentum improves dynamic scene consistency by 126.2% in unseen scenes over static image conditioning, while also improving visual fidelity. Our models and source code will be made publicly available.

Primary area:
generative models
Keywords:
world models, memory, video generation, content consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ymwSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper present Momentum, a two-stage framework for keeping scene dynamics persistent in video world models. In the first stage, WorldUpdate generate synchronized videos from reference camera rigs spread out across scene, and pass information between neighboring rig through geometric reprojection. In the second stage, WorldInteract use these evolving reference videos to guide video generation under camera control. The authors also builds a data generation pipeline on top of CARLA, and they evaluates visual fidelity, camera control, and how consistent the dynamic scene stay. In both seen and unseen towns, the method do better than conditioning on static images, especially on dynamic scene consistency.

Critical strengths

1.The paper tackle a real limitation of current video world models, which is keeping scene dynamics going beyond what camera can currently see. Splitting the problem into two stages, one that evolve the world state and one that generates the target view, give a useful framework for modeling persistent dynamic scene without having to track objects explicitly. 2.The multi-rig representation and the CARLA data pipeline makes it possible to generate videos that stays synchronized across viewpoint and over time. In the end-to-end experiments, the method beat static grounding on visual fidelity and dynamic scene consistency, and this hold on unseen town as well.

Critical weaknesses

1.The method isn't very novel. WorldUpdate build on pretrained multi-view diffusion and geometric reprojection, and WorldInteract is basically SWM extended with dynamic video conditioning. The paper don't isolate what graph propagation contribute well enough, so it's still unclear whether simpler way of grounding on dynamics could get similar gain. 2.The evaluation don't convincingly show that the dynamics persists and stay physically consistent. On SCS-dyn, WorldUpdate actually do worse than frozen-frame baseline, and the improvement in staticness that the paper report doesn't show that objects follows the right trajectories. The experiments are also limited to CARLA and doesn't analyze inference cost, so how well the method generalizes to real world and how well it scales are still not properly tested.

Questions for authors
  1. In Appendix C.1, WorldUpdate achieves lower SCS-dyn than the frozen-frame baseline. How do the authors explain this result, and what evidence shows that generated objects follow correct motion trajectories rather than merely exhibiting similar amounts of motion?

  2. Can the authors provide ablations isolating the contribution of graph propagation, including independent rig generation without neighbor conditioning and alternative propagation orders?

  3. WorldUpdate uses ground-truth neighboring videos during training but generated neighboring videos during inference. How does this training–inference gap affect consistency as information propagates across more rigs and longer sequences?

  4. How does Momentum preserve dynamics in regions that remain outside the target camera's vicinity? The data pipeline freezes reference observations when the target camera is more than 25 meters away.

  5. Can the authors quantitatively evaluate object identity, trajectory consistency, and dynamic interactions across different reference rigs and target-camera views, beyond category-level mask overlap?

  6. How does the inference cost scale with the number of rigs and target cameras? Can the authors report GPU memory usage, generation latency, and the number of target trajectories needed to amortize WorldUpdate's cost?

3 comments

We thank the reviewer for the detailed questions. We address each concern in turn. New results use Test 3: 30 new unseen-town sessions with ground-truth object instance masks.

Q1, W2. Lower SCS-dyn than the frozen frame. We show in App. C1.1 that SCS-dyn has a bias to favor static images rather than any motion, which affects static cameras like our rigs. Given the new per-object masks this becomes apparent, as well as that WorldUpdate follows moving objects better than a persistent baseline:

Rig views, per-object PSNR ↑all (387)stationary (54)moving (289)
WorldUpdate16.5518.4616.14
Persisten baseline18.0231.7115.64
LiveWorld monitors14.4917.4813.92

In contrast to the staticness metric, this metric concludes that not only the model moves more, but the motion is correct.

Q2. Isolating graph propagation (end-to-end, Test 3, target view): We isolate cross-rig conditioning by running WorldUpdate with and without conditioning on its neighbouring rig. To score against ground truth, the neighbour is given as its recorded video, so the conditioning carries the true scene dynamics. On Test 3, conditioning on the neighbouring rig raises per-object PSNR in the rig views from 16.61 to 16.71 dB (387 objects).

While in the current setup cross-rig conditioning mostly has a supportive role, holding the scene state in rig videos is mainly what lets Momentum simulate up to 35 moving agents off screen without detection, tracking or per-object state, at a cost that does not grow with their number.

Q3. Training/inference gap of WorldUpdate conditioning. We measure that there is no measurable effect of the training/inference gap. In training, each rig is conditioned on its neighbour's recorded video; at inference, on its neighbour's generated video. If this mismatch introduced errors, neighbouring rigs would drift apart. We therefore measure how well neighbouring rigs agree: one rig's video is reprojected into its neighbour's views and compared on the pixels it covers (105 rig pairs, Test 3). Generated rigs agree with each other as well as the recordings do (16.32 vs. 16.35 dB PSNR). The neighbour warp is dropped for 20% of training samples, and each rig is anchored on its own history.

Q4. The 25 m rule. The rule concerns only data generation for training in order to save storage and rendering time by avoiding rendering rigs where no ground truth target camera is rendered for training. Training was performed without using any of the frozen rigs. At inference, WorldUpdate has no 25m limitation: it advances every rig from its own history and its neighbours, so far regions keep evolving.

Q5. Identity, trajectory, consistency across rigs and target view. To measure identity we introduce object-level metrics which use CARLA object instance masks. This measures moving object identity. We also measure objects which reenter the target camera view after at least a second being out of camera to further measure consistency. WorldInteract is given the recorded rig videos as world state and compared with a frozen frame conditioning.

Test 3, target view, per-object PSNR ↑ / LPIPS ↓all objects (594)re-entering objects (335)
World state: time-aligned rig video19.03 / 0.22318.93 / 0.223
World state: same rigs, frozen18.07 / 0.24917.83 / 0.251
LiveWorld observer14.92 / 0.30314.66 / 0.307

The target view places objects where the world state has them: freezing the state costs 1.0 dB per object, and SCS-dyn drops from 0.256 to 0.070. An object that was out of view comes back as accurately as one that stayed visible (18.93 vs. 19.03 dB); with a frozen state it comes back 1.1 dB worse. Momentum is 3.6 dB above LiveWorld.

Q6. Cost We measure the performance and compare with a baseline LiveWorld one H200:

GPU-h / sessionTFLOPs / sessionPeak memoryLatency
WorldUpdate (ours)0.21200,01056 GiB19 s per rig update (8 × 28 frames)
LiveWorld monitors0.831,805,620 (9.0×)68 GiB8.3 min per round (1 × 64 frames)
WorldInteract (ours)0.04847,57447 GiB1.5 min per 77 frames
LiveWorld observer0.89 (≈18×)1,074,996 (≈23×)79 GiB14 min per round (64 frames)
Total: Momentum0.26247,584
Total: LiveWorld1.72 (6.6×)2,880,616 (≈12×)

WorldUpdate is linear in the number of rigs, with constant memory, and independent of the number of objects. WorldInteract is independent of the number of rigs. In comparison, LiveWorld is much more computationally demanding.

W1. Novelty and simpler alternatives. The contribution is a scalable world state that evolves at all locations with one model, without entity tracking or a static/dynamic split. This is novel and we consider this a major contribution. Previous methods are significantly more computationally demanding, offer less space coverage and are better fit for fewer entities.

We demonstrated significantly higher performance than LiveWorld - a related work we use as a baseline. Simpler alternatives are time-aligned video given to the statically trained model (SCS-dyn 0.127 vs. 0.265, Tab. 2) and a single-view model for the state, which tends to freeze (Tab. 5). We add LiveWorld (entity-centric): lower on every measure at 6.6x our GPU time.

W2. CARLA only. At inference the method needs one image per rig view, which makes it usable with panorama images which are available at scale. However, no open dataset has synchronized real multi-view video with a moving target camera, which leads us to the synthetic setting. However, the base model, trained on CARLA only, is competitive on real MARS data (Tab. 1).

Thank you for the responses and revisions. The clarifications of the metrics generally make sense. I will maintain my original score.

Reviewer_rKmUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce Momentum, addressing the issue of out-of-view objects disappearing or returning in the states when using static memory caches. Momentum represents the world state as synchronized multi-vieew videos captured by fixed reference camera rigs linked in a graph. Momentum has two main stages: WorldUpdate rolls the rig videos forward chunk-by-chunk using a breadh-first traversal conditioned on reprojected neighbor views, while WorldInteract renders target camera trajectories directly from the dynamic world state. Evaluated on a CARLA benchmark, Momentum significantly improves dynamic scene consistency.

Critical strengths

[S1] Representing the content that is outside the field of view as synchronized multi-view video across a rig graph is a compelling concept and elegant.

Critical weaknesses

[W1] The core claim that out-of-view content keeps evolving and reappears in a consistent state is never measured directly. The main metric (SCS-dyn) only tracks per-frame category overlap, without accounting for object instances or re-entry.WorldUpdate (Multi-Rig DGWM) is also worse than the Persistent Baseline even on SCS-dyn, which freezes the conditioning frame (Appendix C.1). Also the staticness gap only checks the amount of motion, not whether the motion is correct or not. The paper also provides no check that the target-camera video actually agrees with the world rollout. The main claim rests mostly on qualitative examples. Finally, the dynamic grounding experiments are limited to synthetic CARLA benchmarks, without comparisons to related methods such as LiveWorld [1].

[1] Duan, Zicheng, et al. "Liveworld: Simulating out-of-sight dynamics in generative video world models." arXiv preprint arXiv:2603.07145 (2026).

Questions for authors

[Q1] Does graph propagation help end-to-end compared with generating rigs independently? And how much does SCS-dyn vary across the sampling seeds?

2 comments
AuthorsOpenReview ↗

Clarifications

We thank the reviewer for the insightful questions. We respond to each of the concerns. New results use Test 3: 30 new unseen-town sessions with ground-truth object-level instance masks.

W1a. Object instances. Re-entry, and agreement of the target video with the world rollout. We implement the reviewer's suggestion and implement object instance masks (vehicles and people) scoring. We rerun evaluation comparing within the object masks only and also define 335 objects which return to the view after out of view for ≥ 1 s. We run WorldInteract on our state and compare with the static rigs.

Test 3, target viewper-object PSNR ↑re-entering PSNR ↑SCS-dyn ↑
Dynamic state19.0318.930.256
Static state18.0717.830.070

This establishes the target video follows the dynamic state: removing its motion costs 1.0 dB inside the objects and most of SCS-dyn. Objects that left the view return as accurately as objects that stayed visible, with a static state they are worse.

W1b. Below the Persistent Baseline on SCS-dyn (App. C.1). We show in App. C1.1 that SCS-dyn has a bias to favor static images rather than any motion, which affects static cameras like our rigs. Given the new per-object masks this becomes apparent, as well as that WorldUpdate follows moving objects better than a persistent baseline:

Rig views, per-object PSNR ↑all (387)stationary (54)moving (289)
WorldUpdate16.5518.4616.14
Persisten baseline18.0231.7115.64
LiveWorld monitors14.4917.4813.92

End-to-end, against ground-truth target video, WorldUpdate's rollout more than doubles SCS-dyn (0.086 → 0.198 seen, 0.084 → 0.190 unseen; Tab. 4).

W1c. LiveWorld. We compare with LiveWorld as released, adjusted by launching a monitor at each of our rig views. Momentum is ahead on every measure:

Test 3, target viewper-object PSNR ↑re-entering PSNR ↑SCS-static ↑SCS-dyn ↑GPU-h/session
Momentum18.5518.290.6940.1670.26
LiveWorld14.9214.660.3950.0261.72

In the rig views WorldUpdate scores 16.55 dB per object vs. 14.49 for LiveWorld's monitors. LiveWorld incurs significantly more computational costs.

Q1. SCS-dyn Seeds. Over three seeds, SCS-dyn is 0.172 ± 0.005. This spread is far smaller than the gains the paper reports (0.086 to 0.198, Tab. 4).

Q1. Graph propagation vs. independent rigs: We isolate cross-rig conditioning by running WorldUpdate with and without conditioning on its neighbouring rig. To score against ground truth, the neighbour is given as its recorded video, so the conditioning carries the true scene dynamics. On Test 3, conditioning on the neighbouring rig raises per-object PSNR in the rig views from 16.61 to 16.71 dB (387 objects).

While in the current setup cross-rig conditioning mostly has a supportive role, holding the scene state in rig videos is mainly what lets Momentum simulate up to 35 moving agents off screen without detection, tracking or per-object state, at a cost that does not grow with their number.

Thank you for the clarifications and the additional experiments. While the new metrics are helpful, still it seems the dynamic gains are modest. WorldUpdate improves on moving objects only a small amount, but degrades a lot on stationary compare to the frozen baseline. Even though the other points were largely addressed, the core limitation remains, so I will maintain my score.

Reviewer_3arEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper represents a dynamic street scene as a graph of fixed 8-camera rigs whose synchronized videos serve as an evolving world state. One Cosmos-based multi-view model generates the rig videos, propagating across the graph via depth-reprojected neighbor views, and a second model (a re-implementation of Seoul World Model) renders target-camera video conditioned on the time-aligned rig videos. All dynamic experiments use a custom CARLA dataset with seen- and unseen-town test splits, evaluated with PSNR/LPIPS, camera error, and SAM-based mask-IoU consistency scores.

Critical strengths
  • It targets a real gap in world-model memory: off-screen dynamic content usually freezes or disappears in existing caches. Using time-indexed multi-view video as the state is simple and avoids explicit object tracking or static/dynamic decomposition.
  • The decomposition into world-state update and observation rendering is clean, and one rollout can serve several target trajectories.
  • With GT rig videos, dynamic grounding helps consistently across fidelity, camera accuracy and both SCS metrics, in 1-rig and 4-rig scenes and in seen and unseen towns.
Critical weaknesses
  • Evaluation is synthetic only, and the setup is impractical. All dynamic results are in CARLA, with at most 10 vehicles and 25 pedestrians and 30 test sessions per split. Dense, synchronized 8-camera video rigs, up to 4 per intersection, have no real-world counterpart, so practical relevance is unclear.
  • There is little evidence that the generated dynamics are correct. In Table 5, Multi-Rig DGWM has the lowest SCS-dyn, below both a frozen baseline and the single-view model, and it wins only on the staticness gap, which measures how much the scene moves rather than whether it moves correctly. Category-level mask IoU also cannot verify that a specific car reappears in the right place and state, even though CARLA provides ground-truth agent poses that would support an object-level metric.
  • Comparisons are narrow and the gains are overstated. All baselines are the authors' own SWM variants. There is no comparison to other dynamic-memory methods such as LiveWorld, WorldDirector or HyDRA, or to a 4D reconstruction-and-render baseline.
  • Graph propagation and cost are not analysed. There is no study of error accumulation with graph depth, of conflicts when neighbors disagree, or of the mismatch between training on ground-truth neighbor video and running on generated video at inference. No runtime or compute is reported for generating N rigs × 8 views, and the rig resolution of 192×352 is far below the target resolution of 704×1280.
Questions for authors

Can you add captions and a README to the supplementary material explaining what each row and file shows (e.g., dg / sg / wi / eg / mr)?

1 comment
AuthorsOpenReview ↗

Clarifications

We thank the reviewer for the careful and constructive review. We respond to each of the concerns. New results use Test 3: 30 new unseen-town sessions with ground-truth instance masks.

W1. Synthetic setup. We use CARLA to isolate and demonstrate our contributions in a controllable setup. However, a real world counterpart of our setup exists in the form of street-view panorama images. At inference, WorldUpdate needs one single image per rig view to generate the videos itself. Panoramas exist at scale and are what SWM grounds on with their proprietary dataset. However, no suitable public dataset with multi-view video + target camera exists yet for our training, so we demonstrate our method on a synthetic environment. In contrast to LiveWorld and HyDRA which support a few subjects, our cost does not grow with more entities and we have no such limitation, allowing up to 35 in our setup.

W2. Are the generated dynamics correct? We added the suggested object-level metric: every moving vehicle and pedestrian is scored inside its own ground-truth mask. In addition, we isolate only the objects that leave and re-appear in the frame. We compare WorldInteract's results grounded on WorldUpdate's state and a static state.

Target view, per objectPSNR ↑re-entering objects PSNR ↑SCS-dyn ↑
Dynamic state19.0318.930.256
Static state18.0717.830.070

The objects that are out of view (335 in total) for ≥ 1 s and return come back as accurately as objects that stayed visible (18.93 vs. 19.03 dB) - with a static state they are 1.1 dB worse.

Lowest SCS-dyn in Tab. 5. The role of Tab.5 is to show that SCS-dyn favors staticness, especially in static cameras like our rig setup. After one step a moving object still overlaps most of itself, while a model that moves it is penalised for any deviation from the one recorded future. The proposed by the reviewer per-object metric reveals that WorldUpdate is clearly better at generating moving objects:

Rig views, per-object PSNR ↑all (387)stationary (54)moving (289)
WorldUpdate16.5518.4616.14
Frozen baseline18.0231.7115.64
LiveWorld monitors14.4917.4813.92

In addition, end-to-end, against ground-truth target video, WorldUpdate's state more than doubles SCS-dyn (0.086 → 0.198, Tab. 4); wrong dynamics would lower it.

W3. Comparison with other baselines. We compare with HyDRA and LiveWorld. We ran them as released on our setup - for LiveWorld we add one monitor per rig camera, for HyDRA the history context is given. Momentum is ahead on every measure - it provides the best results while staying efficient:

Test 3, target viewper-object PSNR ↑re-entering PSNR ↑SCS-static ↑SCS-dyn ↑GPU-h
Momentum18.5518.290.6940.1677.7
LiveWorld14.9214.660.3950.02648.4
HyDRA13.6212.830.2310.0193.1

HyDRA is cheaper because it builds no world state. WorldDirector generates under explicit LLM-planned object trajectories, which our setting does not provide. We differentiate ourselves from 4D reconstruct-and-render, as it needs an observed video of the motion, while our input is one static image per view.

W4. Graph propagation and cost. Our proposed dynamic memory is significantly more efficient than existing work:

GPU-h / sessionTFLOPs / sessionPeak memory
WorldUpdate (ours)0.21200,01056 GiB
LiveWorld monitors0.831,805,620 (9.0×)68 GiB
WorldInteract (ours)0.04847,57447 GiB
LiveWorld observer0.89 (≈18×)1,074,996 (≈23×)79 GiB
Total: Momentum0.26247,584
Total: LiveWorld1.72 (6.6×)2,880,616 (≈12×)

We thank the reviewer for spotting the resolution mistake in the appendix: 704×1280 is the pretrained model's resolution. Both grounded WMs are trained and evaluated at 384×704, which is twice the resolution of WorldUpdate. We will make sure to make this correction in text.

Train/inference mismatch and error accumulation: running WorldUpdate with ground-truth neighbours (the training condition) instead of unconditioned ones changes per-object PSNR from 16.61 to 16.71 dB in the rig views. Generated neighbours do not degrade the result. When each rig conditions on one neighbour (its BFS parent), generated neighbouring rigs agree with each other as well as the recordings do (rig-to-rig reprojection PSNR 16.32 vs. 16.35 dB).

Q1. Yes: we added a README with per-video captions (test set, session, model, meaning of each row).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.