- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper present Momentum, a two-stage framework for keeping scene dynamics persistent in video world models. In the first stage, WorldUpdate generate synchronized videos from reference camera rigs spread out across scene, and pass information between neighboring rig through geometric reprojection. In the second stage, WorldInteract use these evolving reference videos to guide video generation under camera control. The authors also builds a data generation pipeline on top of CARLA, and they evaluates visual fidelity, camera control, and how consistent the dynamic scene stay. In both seen and unseen towns, the method do better than conditioning on static images, especially on dynamic scene consistency.
Critical strengths
1.The paper tackle a real limitation of current video world models, which is keeping scene dynamics going beyond what camera can currently see. Splitting the problem into two stages, one that evolve the world state and one that generates the target view, give a useful framework for modeling persistent dynamic scene without having to track objects explicitly. 2.The multi-rig representation and the CARLA data pipeline makes it possible to generate videos that stays synchronized across viewpoint and over time. In the end-to-end experiments, the method beat static grounding on visual fidelity and dynamic scene consistency, and this hold on unseen town as well.
Critical weaknesses
1.The method isn't very novel. WorldUpdate build on pretrained multi-view diffusion and geometric reprojection, and WorldInteract is basically SWM extended with dynamic video conditioning. The paper don't isolate what graph propagation contribute well enough, so it's still unclear whether simpler way of grounding on dynamics could get similar gain. 2.The evaluation don't convincingly show that the dynamics persists and stay physically consistent. On SCS-dyn, WorldUpdate actually do worse than frozen-frame baseline, and the improvement in staticness that the paper report doesn't show that objects follows the right trajectories. The experiments are also limited to CARLA and doesn't analyze inference cost, so how well the method generalizes to real world and how well it scales are still not properly tested.
Questions for authors
-
In Appendix C.1, WorldUpdate achieves lower SCS-dyn than the frozen-frame baseline. How do the authors explain this result, and what evidence shows that generated objects follow correct motion trajectories rather than merely exhibiting similar amounts of motion?
-
Can the authors provide ablations isolating the contribution of graph propagation, including independent rig generation without neighbor conditioning and alternative propagation orders?
-
WorldUpdate uses ground-truth neighboring videos during training but generated neighboring videos during inference. How does this training–inference gap affect consistency as information propagates across more rigs and longer sequences?
-
How does Momentum preserve dynamics in regions that remain outside the target camera's vicinity? The data pipeline freezes reference observations when the target camera is more than 25 meters away.
-
Can the authors quantitatively evaluate object identity, trajectory consistency, and dynamic interactions across different reference rigs and target-camera views, beyond category-level mask overlap?
-
How does the inference cost scale with the number of rigs and target cameras? Can the authors report GPU memory usage, generation latency, and the number of target trajectories needed to amortize WorldUpdate's cost?
3 comments
Q6. Cost We measure the performance and compare with a baseline LiveWorld one H200:
| GPU-h / session | TFLOPs / session | Peak memory | Latency | |
|---|---|---|---|---|
| WorldUpdate (ours) | 0.21 | 200,010 | 56 GiB | 19 s per rig update (8 × 28 frames) |
| LiveWorld monitors | 0.83 | 1,805,620 (9.0×) | 68 GiB | 8.3 min per round (1 × 64 frames) |
| WorldInteract (ours) | 0.048 | 47,574 | 47 GiB | 1.5 min per 77 frames |
| LiveWorld observer | 0.89 (≈18×) | 1,074,996 (≈23×) | 79 GiB | 14 min per round (64 frames) |
| Total: Momentum | 0.26 | 247,584 | ||
| Total: LiveWorld | 1.72 (6.6×) | 2,880,616 (≈12×) |
WorldUpdate is linear in the number of rigs, with constant memory, and independent of the number of objects. WorldInteract is independent of the number of rigs. In comparison, LiveWorld is much more computationally demanding.
W1. Novelty and simpler alternatives. The contribution is a scalable world state that evolves at all locations with one model, without entity tracking or a static/dynamic split. This is novel and we consider this a major contribution. Previous methods are significantly more computationally demanding, offer less space coverage and are better fit for fewer entities.
We demonstrated significantly higher performance than LiveWorld - a related work we use as a baseline. Simpler alternatives are time-aligned video given to the statically trained model (SCS-dyn 0.127 vs. 0.265, Tab. 2) and a single-view model for the state, which tends to freeze (Tab. 5). We add LiveWorld (entity-centric): lower on every measure at 6.6x our GPU time.
W2. CARLA only. At inference the method needs one image per rig view, which makes it usable with panorama images which are available at scale. However, no open dataset has synchronized real multi-view video with a moving target camera, which leads us to the synthetic setting. However, the base model, trained on CARLA only, is competitive on real MARS data (Tab. 1).
Thank you for the responses and revisions. The clarifications of the metrics generally make sense. I will maintain my original score.
We thank the reviewer for the detailed questions. We address each concern in turn. New results use Test 3: 30 new unseen-town sessions with ground-truth object instance masks.
Q1, W2. Lower SCS-dyn than the frozen frame. We show in App. C1.1 that SCS-dyn has a bias to favor static images rather than any motion, which affects static cameras like our rigs. Given the new per-object masks this becomes apparent, as well as that WorldUpdate follows moving objects better than a persistent baseline:
In contrast to the staticness metric, this metric concludes that not only the model moves more, but the motion is correct.
Q2. Isolating graph propagation (end-to-end, Test 3, target view): We isolate cross-rig conditioning by running WorldUpdate with and without conditioning on its neighbouring rig. To score against ground truth, the neighbour is given as its recorded video, so the conditioning carries the true scene dynamics. On Test 3, conditioning on the neighbouring rig raises per-object PSNR in the rig views from 16.61 to 16.71 dB (387 objects).
While in the current setup cross-rig conditioning mostly has a supportive role, holding the scene state in rig videos is mainly what lets Momentum simulate up to 35 moving agents off screen without detection, tracking or per-object state, at a cost that does not grow with their number.
Q3. Training/inference gap of WorldUpdate conditioning. We measure that there is no measurable effect of the training/inference gap. In training, each rig is conditioned on its neighbour's recorded video; at inference, on its neighbour's generated video. If this mismatch introduced errors, neighbouring rigs would drift apart. We therefore measure how well neighbouring rigs agree: one rig's video is reprojected into its neighbour's views and compared on the pixels it covers (105 rig pairs, Test 3). Generated rigs agree with each other as well as the recordings do (16.32 vs. 16.35 dB PSNR). The neighbour warp is dropped for 20% of training samples, and each rig is anchored on its own history.
Q4. The 25 m rule. The rule concerns only data generation for training in order to save storage and rendering time by avoiding rendering rigs where no ground truth target camera is rendered for training. Training was performed without using any of the frozen rigs. At inference, WorldUpdate has no 25m limitation: it advances every rig from its own history and its neighbours, so far regions keep evolving.
Q5. Identity, trajectory, consistency across rigs and target view. To measure identity we introduce object-level metrics which use CARLA object instance masks. This measures moving object identity. We also measure objects which reenter the target camera view after at least a second being out of camera to further measure consistency. WorldInteract is given the recorded rig videos as world state and compared with a frozen frame conditioning.
The target view places objects where the world state has them: freezing the state costs 1.0 dB per object, and SCS-dyn drops from 0.256 to 0.070. An object that was out of view comes back as accurately as one that stayed visible (18.93 vs. 19.03 dB); with a frozen state it comes back 1.1 dB worse. Momentum is 3.6 dB above LiveWorld.