- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents MultiWorld, a real-time multi-agent multi-view video world model: given initial views and per-frame actions from multiple agents, it generates synchronized videos of a shared environment. Three core designs: (1) Agent Identity Embedding (RoPE over the agent axis) binds actions to the correct agents, and a Dynamic World State Model maintains shared geometric memory for cross-view consistency; (2) Causal Teacher Distillation distills a few-step autoregressive generator from a causal rather than bidirectional teacher, aligning supervision with the temporal information boundary of online generation, while Data-Guided Correction adds paired-data supervision under generated histories to prevent error accumulation and collapse; (3) view-parallel inference and pipelined decoding enable ~24.8 FPS interactive generation of two 480×480 views. Experiments on a self-collected ~200 h two-player gameplay dataset and a multi-robot manipulation dataset (including failure trajectories) show improvements in FVD, action following, and 3D consistency over baselines, with ablations validating each component.
Critical strengths
A complete, genuinely real-time system for an important and timely problem — joint multi-agent action control with consistent multi-view observations at 24.8 FPS — plus two datasets with synchronized actions (including failure trajectories) pledged for release; the system and data are independently valuable.
CTD+DGC pinpoints a real failure mode and fixes it, with clean attribution. The collapse mechanism of naive multi-step→few-step switching (sampling errors accumulate → score estimation fails on degraded frames → training collapse) is clearly characterized, and the fix is simple and effective (FVD 397→290 in ablation). This analysis-and-recipe is useful, beyond this paper itself.
Critical weaknesses
The evaluation does not measure the deployed model, and the baselines are weak. Table 1 evaluates all methods — including the authors' own — as bidirectional offline models, so the headline numbers do not reflect the few-step causal model that is the paper's actual product; the deployed model (FVD 290) is nearly 2× worse than its multi-step teacher (154), and this gap is barely analyzed.
In robotics, the superiority claim rests on excluding Concat-View*, which in fact outperforms the proposed model on PSNR (27.44 vs 26.60), action score (92.0 vs 88.7), and RPE (0.82 vs 1.52); even with it excluded, only FVD leads marginally (96 vs COMBO 99).
No comparison is made against concurrent multi-agent world models or established few-step distillation methods the only distillation baseline is a weakened bidirectional teacher.
The incremental contribution boundary is blurry. CTD's core idea — causal teacher, history-conditioned scoring, matching the temporal information boundary — is essentially shared with CMD (which the paper cites) and the Causal Forcing line, yet no differentiation or empirical comparison is given.
Questions for authors
What fundamentally distinguishes CTD from CMD and Causal Forcing? CMD argues scores should be conditioned on the student's realized prefix, whereas you condition on GT history — which is more reliable when the generated history has degraded, and can you provide evidence?
Does DGC's paired-data regression cause memorization of training sequences or reduced diversity on unseen initial frames/actions?
The IDM is trained on the same data as the world model — could this make the action-following metric systematically optimistic? Can you cross-check with an IDM trained on held-out data?
4 comments
We thank the authors for the additional evaluations and baseline comparisons, which partially address our concerns about the distilled model and CTD. However, several important issues remain. The robotics comparison is still not conducted under matched view settings; stating that Concat-View* uses an easier two-view setup does not replace a controlled comparison. Moreover, similar IDM accuracy on training and held-out real videos does not resolve the potential bias of evaluating generated videos using an IDM trained on the same data. DGC’s diversity and generalization are also supported only qualitatively. I therefore retain my rating.
More quantative results for reviewer EpKE [1/2]
Thank you for your thoughtful follow-up comments. Your constructive feedback has inspired us to conduct the remaining controlled experiments and further refine our manuscript. We sincerely appreciate your time and valuable insights, which have significantly strengthened our paper. Below, we provide direct quantitative results addressing each of your three remaining concerns.
1. Matched 2–4 View Robotics Comparison
Matched 2–4 View Setting via Task-Homogeneous Batching. To jointly train and evaluate Concat-View across all 2–4 view robotics tasks under the exact same setting as the other methods, we constrain each mini-batch to sample episodes from a single task (with a uniform view count) before spatially concatenating the views.
As shown in Table R4, when Concat-View is jointly trained across tasks with varying view counts (Full (2–4 Views)), its performance drops significantly across all metrics (e.g., FVD degrades from 106 to 121, PSNR drops from 27.44 to 23.31, and RPE increases from 0.82 to 2.31), indicating that spatial view concatenation struggles with cross-view and cross-task modeling. Under this strictly matched 2–4 view setting, our method remains the most effective approach across all metrics.
Table R4: Matched 2–4 View Robotics Comparison.
| Method | View Setting | FVD ↓ | LPIPS ↓ | SSIM ↑ | PSNR ↑ | Action ↑ | RPE ↓ |
|---|---|---|---|---|---|---|---|
| Concat-View* | 2 Views Only | 106 | 0.06 | 0.90 | 27.44 | 92.0 | 0.82 |
| Standard | Full (2–4 Views) | 100 | 0.07 | 0.90 | 26.39 | 88.2 | 1.60 |
| Concat-View | Full (2–4 Views) | 121 | 0.12 | 0.85 | 23.31 | 87.1 | 2.31 |
| COMBO | Full (2–4 Views) | 99 | 0.08 | 0.90 | 26.49 | 88.5 | 1.54 |
| Ours | Full (2–4 Views) | 96 | 0.07 | 0.90 | 26.60 | 88.7 | 1.52 |
2. Cross-Checking Generated Videos with an Independent Held-out IDM
While our previous check showed that the Original IDM generalizes well to unseen real videos (97.62% on training vs. 95.22% on held-out real videos), to directly rule out any shared-data bias when evaluating generated videos, we further trained an independent IDM exclusively on 1,807 held-out clips (650,520 frames) unseen by the world model (Held-out IDM). Although the Held-out IDM yields slightly lower absolute accuracy across all entries (including real videos) because it is trained on much less data than the Original IDM, it does not change the relative action-following ranking of any method in Table R5. Since both the Original IDM and the independent Held-out IDM produce the exact same method ordering—with our method consistently ranking first and even widening its lead over COMBO from +1.9% to +2.3%—this confirms that our action-following gains on generated videos are not an artifact of shared training data.
Table R5: Action-following accuracy (%) on real evaluation videos and generated videos across methods, evaluated using the Original IDM (trained on the world-model training split) and an independent Held-out IDM (trained exclusively on 1,807 unseen held-out clips).
| Method | Original IDM ↑ (WM Train Split) | Held-out IDM ↑ (1,807 Held-out Clips) | Δ (Held-out − Orig.) |
|---|---|---|---|
| Real Videos (Eval) | 95.2 | 94.1 | -1.1 |
| Standard | 88.4 | 87.3 | -1.1 |
| Concat-View | 89.1 | 88.5 | -0.6 |
| COMBO | 89.3 | 88.6 | -0.7 |
| Ours-Multistep | 91.2 | 90.9 | -0.3 |
| Ours-Fewstep | 88.5 | 87.7 | -0.8 |
More quantative results for reviewer EpKE [2/2]
3. Quantitative Evaluation of DGC Diversity and Generalization
To complement our qualitative results, we quantitatively evaluate CTD w/o DGC and CTD w/ DGC on unseen held-out initial frames under both matched actions () and cross-paired unseen actions () with fixed random seeds (Table R6). Note that cross-paired action following is naturally more challenging than matched-action evaluation because an action sequence transferred from another clip can encounter scene obstacles in the new initial frame. Even so, CTD w/ DGC improves cross-action following from 75.2% to 77.6% and substantially increases action-conditioned trajectory diversity (Pairwise LPIPS between and rollouts rises from 0.300 to 0.403). Furthermore, under cross-paired actions , its LPIPS to the original GT increases sharply from 0.34 (under matched actions , Table 3) to 0.449, confirming strong generalization without sequence memorization.
Table R6: Quantitative evaluation of generalization and action-conditioned trajectory diversity on unseen held-out initial frames under matched () and cross-paired () action sequences.
| Model | Unseen FVD ↓ | Unseen PSNR ↑ | Cross-Action Following ↑ | Action-Conditioned Diversity (Pairwise LPIPS ↑) | Non-Memorization (LPIPS vs. Source GT ↑) |
|---|---|---|---|---|---|
| CTD w/o DGC | 397 | 16.29 | 75.2 | 0.300 | 0.323 |
| CTD w/ DGC (Ours) | 290 | 17.80 | 77.6 | 0.403 | 0.449 |
Response to Reviewer EpKE
EpKE [W1/W3] Complete Evaluation of the Distilled Real-Time Model with more baselines
Complete Evaluation of the Distilled Real-Time Model with more baselines Thank you for this constructive suggestion. We provide action accuracy (Action) and cross-view geometric consistency (RPE) for the distilled few-step model with Self-Forcing [1], CMD[2] and GammaWorld[3] baselines. Table. 1 shows our base model significantly outperforms existing baselines, while 4/8 NFE models remain comparable with the teacher forcing base model. Although the 2-step model has worse FVD than teacher forcing model, it still consistently outperforms Self-Forcing, Gamma-World 4NFE setting.
EpKE [W2] Robotics Results and Methodological Advantages:
Thank you for pointing this out. We clarify the robotics results from three perspectives: 3. Capability Difference Beyond Metrics: MultiWorld is the first work to support variable numbers of agents and views, which the baselines do not support. 4. The robotics scene is visually simpler than the game because of its static cameras. Even so, MultiWorld still achieves clear relative gains over baselines, reducing FVD by 4.0% and RPE by 5.0%. 5. Concat-View* is evaluated on only 2 views whereas MultiWorld is evaluated on 4 views, so it is natural for Concat-View* to have higher scores on a simpler subset.
EpKE [W4/Q1] CTD contribution Clarification.
Thank you for raising these points. As shown in Table R1, we have added the concurrent multi-agent baseline Gamma-World [G2] as well as the distillation baselines Self-Forcing [G1] and CMD [G3] for direct empirical comparison. Regarding the core difference of CTD:
EpKE [Q2]: DGC memorization and action diversity.
Thank you for this thoughtful question. We don't observe that DGC largely hurts diversity, and we added new qualitative figures in the revised paper for evidence. Same Initial Scene with Diverse Actions in Training: Our training datasets naturally contain many instances where the same or visually similar initial scene is paired with different action combinations Qualitative Evidence: We added two new qualitative examples to the revised PDF, confirming action controllability and diversity without severe memorization.
EpKE [Q3]: IDM evaluation
We appreciate this valuable question. To investigate whether the IDM may be biased toward the data used to train the world model, we evaluated its action prediction accuracy on 100 training cases and a held-out validation set. The IDM achieved 97.62% accuracy on the training set and 95.22% on the validation set, with no substantial difference between them. This suggests that the IDM does not exhibit a substantial generalization gap on real videos.