acceptodds
Under review as a conference paper at ICLR 2027

MultiWorld: Interactive Real-Time Multi-Agent Multi-View Video World Models

Abstract

Video world models have achieved remarkable success in simulating environmental dynamics in response to users' actions. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce Agent Identity Embedding to achieve precise multi-agent controllability, and Dynamic Geometry World State to ensure coherent observations across different views. We employ Causal Teacher Distillation for interactive deployment. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Our few-step Multiworld supports 24 FPS generation for both view.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Agent World Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EpKEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MultiWorld, a real-time multi-agent multi-view video world model: given initial views and per-frame actions from multiple agents, it generates synchronized videos of a shared environment. Three core designs: (1) Agent Identity Embedding (RoPE over the agent axis) binds actions to the correct agents, and a Dynamic World State Model maintains shared geometric memory for cross-view consistency; (2) Causal Teacher Distillation distills a few-step autoregressive generator from a causal rather than bidirectional teacher, aligning supervision with the temporal information boundary of online generation, while Data-Guided Correction adds paired-data supervision under generated histories to prevent error accumulation and collapse; (3) view-parallel inference and pipelined decoding enable ~24.8 FPS interactive generation of two 480×480 views. Experiments on a self-collected ~200 h two-player gameplay dataset and a multi-robot manipulation dataset (including failure trajectories) show improvements in FVD, action following, and 3D consistency over baselines, with ablations validating each component.

Critical strengths

A complete, genuinely real-time system for an important and timely problem — joint multi-agent action control with consistent multi-view observations at 24.8 FPS — plus two datasets with synchronized actions (including failure trajectories) pledged for release; the system and data are independently valuable.

CTD+DGC pinpoints a real failure mode and fixes it, with clean attribution. The collapse mechanism of naive multi-step→few-step switching (sampling errors accumulate → score estimation fails on degraded frames → training collapse) is clearly characterized, and the fix is simple and effective (FVD 397→290 in ablation). This analysis-and-recipe is useful, beyond this paper itself.

Critical weaknesses

The evaluation does not measure the deployed model, and the baselines are weak. Table 1 evaluates all methods — including the authors' own — as bidirectional offline models, so the headline numbers do not reflect the few-step causal model that is the paper's actual product; the deployed model (FVD 290) is nearly 2× worse than its multi-step teacher (154), and this gap is barely analyzed.

In robotics, the superiority claim rests on excluding Concat-View*, which in fact outperforms the proposed model on PSNR (27.44 vs 26.60), action score (92.0 vs 88.7), and RPE (0.82 vs 1.52); even with it excluded, only FVD leads marginally (96 vs COMBO 99).

No comparison is made against concurrent multi-agent world models or established few-step distillation methods the only distillation baseline is a weakened bidirectional teacher.

The incremental contribution boundary is blurry. CTD's core idea — causal teacher, history-conditioned scoring, matching the temporal information boundary — is essentially shared with CMD (which the paper cites) and the Causal Forcing line, yet no differentiation or empirical comparison is given.

Questions for authors

What fundamentally distinguishes CTD from CMD and Causal Forcing? CMD argues scores should be conditioned on the student's realized prefix, whereas you condition on GT history — which is more reliable when the generated history has degraded, and can you provide evidence?

Does DGC's paired-data regression cause memorization of training sequences or reduced diversity on unseen initial frames/actions?

The IDM is trained on the same data as the world model — could this make the action-following metric systematically optimistic? Can you cross-check with an IDM trained on held-out data?

4 comments
AuthorsOpenReview ↗

Response to Reviewer EpKE

EpKE [W1/W3] Complete Evaluation of the Distilled Real-Time Model with more baselines

Complete Evaluation of the Distilled Real-Time Model with more baselines Thank you for this constructive suggestion. We provide action accuracy (Action) and cross-view geometric consistency (RPE) for the distilled few-step model with Self-Forcing [1], CMD[2] and GammaWorld[3] baselines. Table. 1 shows our base model significantly outperforms existing baselines, while 4/8 NFE models remain comparable with the teacher forcing base model. Although the 2-step model has worse FVD than teacher forcing model, it still consistently outperforms Self-Forcing, Gamma-World 4NFE setting.

EpKE [W2] Robotics Results and Methodological Advantages:

Thank you for pointing this out. We clarify the robotics results from three perspectives: 3. Capability Difference Beyond Metrics: MultiWorld is the first work to support variable numbers of agents and views, which the baselines do not support. 4. The robotics scene is visually simpler than the game because of its static cameras. Even so, MultiWorld still achieves clear relative gains over baselines, reducing FVD by 4.0% and RPE by 5.0%. 5. Concat-View* is evaluated on only 2 views whereas MultiWorld is evaluated on 4 views, so it is natural for Concat-View* to have higher scores on a simpler subset.

EpKE [W4/Q1] CTD contribution Clarification.

Thank you for raising these points. As shown in Table R1, we have added the concurrent multi-agent baseline Gamma-World [G2] as well as the distillation baselines Self-Forcing [G1] and CMD [G3] for direct empirical comparison. Regarding the core difference of CTD:

  • Vs. Self-Forcing [G1] & Causal Forcing [G6]: Instead of relying on a bidirectional teacher for score distillation, CTD directly uses a causal teacher aligned with the temporal boundary of per-frame action control.
  • Vs. CMD [G3]: While CMD conditions real and fake score estimation on the student's degraded prefix, CTD conditions the score models on clean GT history h<iGTh_{<i}^{\text{GT}}). Clean GT condition preserves the teacher's exact pretraining condition and prevents student rollout errors from drifting the real-score distribution, yielding more accurate score estimation. As evidenced in Table 1 above, our CTD clearly outperforms CMD [G3] (e.g., FVD 212 vs. 345).

EpKE [Q2]: DGC memorization and action diversity.

Thank you for this thoughtful question. We don't observe that DGC largely hurts diversity, and we added new qualitative figures in the revised paper for evidence. Same Initial Scene with Diverse Actions in Training: Our training datasets naturally contain many instances where the same or visually similar initial scene is paired with different action combinations Qualitative Evidence: We added two new qualitative examples to the revised PDF, confirming action controllability and diversity without severe memorization.

EpKE [Q3]: IDM evaluation

We appreciate this valuable question. To investigate whether the IDM may be biased toward the data used to train the world model, we evaluated its action prediction accuracy on 100 training cases and a held-out validation set. The IDM achieved 97.62% accuracy on the training set and 95.22% on the validation set, with no substantial difference between them. This suggests that the IDM does not exhibit a substantial generalization gap on real videos.

We thank the authors for the additional evaluations and baseline comparisons, which partially address our concerns about the distilled model and CTD. However, several important issues remain. The robotics comparison is still not conducted under matched view settings; stating that Concat-View* uses an easier two-view setup does not replace a controlled comparison. Moreover, similar IDM accuracy on training and held-out real videos does not resolve the potential bias of evaluating generated videos using an IDM trained on the same data. DGC’s diversity and generalization are also supported only qualitatively. I therefore retain my rating.

AuthorsOpenReview ↗

More quantative results for reviewer EpKE [1/2]

Thank you for your thoughtful follow-up comments. Your constructive feedback has inspired us to conduct the remaining controlled experiments and further refine our manuscript. We sincerely appreciate your time and valuable insights, which have significantly strengthened our paper. Below, we provide direct quantitative results addressing each of your three remaining concerns.


1. Matched 2–4 View Robotics Comparison

Matched 2–4 View Setting via Task-Homogeneous Batching. To jointly train and evaluate Concat-View across all 2–4 view robotics tasks under the exact same setting as the other methods, we constrain each mini-batch to sample episodes from a single task (with a uniform view count) before spatially concatenating the views.

As shown in Table R4, when Concat-View is jointly trained across tasks with varying view counts (Full (2–4 Views)), its performance drops significantly across all metrics (e.g., FVD degrades from 106 to 121, PSNR drops from 27.44 to 23.31, and RPE increases from 0.82 to 2.31), indicating that spatial view concatenation struggles with cross-view and cross-task modeling. Under this strictly matched 2–4 view setting, our method remains the most effective approach across all metrics.

Table R4: Matched 2–4 View Robotics Comparison.

MethodView SettingFVD ↓LPIPS ↓SSIM ↑PSNR ↑Action ↑RPE ↓
Concat-View*2 Views Only1060.060.9027.4492.00.82
StandardFull (2–4 Views)1000.070.9026.3988.21.60
Concat-ViewFull (2–4 Views)1210.120.8523.3187.12.31
COMBOFull (2–4 Views)990.080.9026.4988.51.54
OursFull (2–4 Views)960.070.9026.6088.71.52

2. Cross-Checking Generated Videos with an Independent Held-out IDM

While our previous check showed that the Original IDM generalizes well to unseen real videos (97.62% on training vs. 95.22% on held-out real videos), to directly rule out any shared-data bias when evaluating generated videos, we further trained an independent IDM exclusively on 1,807 held-out clips (650,520 frames) unseen by the world model (Held-out IDM). Although the Held-out IDM yields slightly lower absolute accuracy across all entries (including real videos) because it is trained on much less data than the Original IDM, it does not change the relative action-following ranking of any method in Table R5. Since both the Original IDM and the independent Held-out IDM produce the exact same method ordering—with our method consistently ranking first and even widening its lead over COMBO from +1.9% to +2.3%—this confirms that our action-following gains on generated videos are not an artifact of shared training data.

Table R5: Action-following accuracy (%) on real evaluation videos and generated videos across methods, evaluated using the Original IDM (trained on the world-model training split) and an independent Held-out IDM (trained exclusively on 1,807 unseen held-out clips).

MethodOriginal IDM ↑ (WM Train Split)Held-out IDM ↑ (1,807 Held-out Clips)Δ (Held-out − Orig.)
Real Videos (Eval)95.294.1-1.1
Standard88.487.3-1.1
Concat-View89.188.5-0.6
COMBO89.388.6-0.7
Ours-Multistep91.290.9-0.3
Ours-Fewstep88.587.7-0.8
AuthorsOpenReview ↗

More quantative results for reviewer EpKE [2/2]

3. Quantitative Evaluation of DGC Diversity and Generalization

To complement our qualitative results, we quantitatively evaluate CTD w/o DGC and CTD w/ DGC on unseen held-out initial frames under both matched actions (a1a_1) and cross-paired unseen actions (a2≠a1a_2 \neq a_1) with fixed random seeds (Table R6). Note that cross-paired action following is naturally more challenging than matched-action evaluation because an action sequence transferred from another clip can encounter scene obstacles in the new initial frame. Even so, CTD w/ DGC improves cross-action following from 75.2% to 77.6% and substantially increases action-conditioned trajectory diversity (Pairwise LPIPS between a1a_1 and a2a_2 rollouts rises from 0.300 to 0.403). Furthermore, under cross-paired actions a2a_2, its LPIPS to the original GT increases sharply from 0.34 (under matched actions a1a_1, Table 3) to 0.449, confirming strong generalization without sequence memorization.

Table R6: Quantitative evaluation of generalization and action-conditioned trajectory diversity on unseen held-out initial frames under matched (a1a_1) and cross-paired (a2≠a1a_2 \neq a_1) action sequences.

ModelUnseen FVD ↓Unseen PSNR ↑Cross-Action Following ↑Action-Conditioned Diversity (Pairwise LPIPS ↑)Non-Memorization (LPIPS vs. Source GT ↑)
CTD w/o DGC39716.2975.20.3000.323
CTD w/ DGC (Ours)29017.8077.60.4030.449
Reviewer_wGGWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MultiWorld, a framework that generates synchronized views of a shared environment conditioned on multiple agents' actions. Three modules are presented: Agent Identity Embedding, Dynamic World State Model, Causal Teacher Distillation. Ablation studies show these modules are effective. The framework supports varying numbers of agents and views.

Critical strengths
  • Beyond single agent video world model, this paper explores multi-agent video world model, which will be useful for the community.

  • The paper, including the figures, show promising progress towards multi-agent video world models. The coordination and controllability seems good.

  • Inference efficiency is also discussed and seems to support real-time generation.

Critical weaknesses
  • The introduction does not adequately position the paper in relation to relevant prior work. Also, the discussion is not deep. The authors simply list the names of the proposed modules without explaining their origins or connections to existing work. This makes it difficult for readers to understand the paper’s contributions.

  • The experiment section is a bit messy. It's hard to follow the experiment settings and understand the contributions of each proposed modules. For example, in table 3, VGGT-Ω\Omega seems to be the authors' proposed final model, then it would be better to highlight it (now, it's hidden in the Method section). I would suggest the authors to rethink what message should be conveyed to the reader.

  • The experiment includes few baselines, including two different settings (Standard, Concat-view) and one third-party work: COMBO. Are there more baselines to compare? For example, in line 121-122, authors already list some. Is there any specific reason that they are excluded in the experiments?

Questions for authors
  1. In table 2, the numbers does not match with table 1. Could the authors elaborate more on the experiment settings? If the final model is in table 2, why there's no match with table 1? If not, why not compare the final model with the ablation settings?

  2. In experiment section, ablations are conducted on each individual proposed module. Is it possible to try different combinations of the modules? For example, different encoder + AIE + distillation settings.

  3. I would suggest the authors to explain the evaluation metrics a bit (e.g., FVD, LPIPS, etc) and the intuition from the experiment results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wGGW

wGGW [W1] Position and Motivation of MultiWorld

Thanks for the feedback. We respectfully note that our problem positioning, motivations, and connections to prior work are already presented in Section 1 (Lines 056–081) and Section 2 (Lines 095–123), and both @8chu and @EpKE reviewers recognized that our framework is "reasonably motivated," with components that "serve clear purposes" and are "clear and easy to follow." To make these connections even clearer, our paper precedes the formulation and solution of multi-agent, multi-view video world modeling. Each module is directly task-driven:

  • Agent Identity Embedding (AIE) for agent controllability: Extending single-agent world models to multiple agents causes action-binding ambiguity; AIE adapts RoPE [G5] along the agent axis to bind actions to specific agents and support variable agent counts.
  • Dynamic World State Model for multi-view consistency: We distill a causal, multi-view model from VGGT-Ω\Omega [G4] that continuously integrates synchronized views in the video generation backbone.
  • Real-Time Deployment: Causal Teacher Distillation is designed for real-time interactive deployment and is connected to distillation works such as Self-Forcing [G1] and CMD [G3].

wGGW [W2/Q1] Clarification of Experiment Organization.

Thanks for the suggestion. We would like to clarify that our experiments in Section 4.2 are organized as a progressive, step-by-step ablation following our framework pipeline:

  1. Action Condition Ablation (Table 2): We first evaluate AIE (with different RoPE bases) on the naive baseline to isolate the effect of multi-agent action control. Thus, the metrics in Table 2 do not match those in Table 1 due to no Geometry Encoder. In Table 3, the first line (None Encoder) is the AIE result matching with the final model in Table 1.
  2. Ablation on Geometry Encoders (Table 3): Based on the optimal AIE setting in Table 2, we add the Dynamic World State Model and compare different geometry encoders (Wan VAE, DINOv2, VGGT, and our final Causal VGGT-Ω), resulting in the full model in Table 1.

wGGW [W3] Response to Comparison with Additional Baselines

Thank you for this helpful suggestion. The works mentioned in Lines 120–122 are concurrent, and most have not yet released their training code for reproduction. Although reproducing all of these works remains challenging, we made our best effort to implement and evaluate Gamma-World [G2] and CMD [G3] on our dataset. As shown in Table R1, MultiWorld consistently outperforms both Gamma-World and CMD across FVD, action following (Action), and 3D cross-view consistency (RPE), while achieving top perceptual and structural quality.

wGGw[Q2] Different encoder + AIE + distillation settings.

Thank you very much for this thoughtful suggestion. As explained in our response to [W2], because our proposed modules are orthogonal across stages, we adopt a progressive ablation setting that builds upon the best configuration at each step. If you have any particular combination of interest, please let us know and we would be happy to run and report it during the discussion period.

wGGw [Q3] Metric Explanations

Thank you for the suggestion. As detailed in Section 4.1 and Appendix B, our metrics cover three complementary aspects:

  • Visual Quality (FVD, LPIPS, SSIM, PSNR): Evaluate video-level frame-level visual fidelity; Our improvements reflect better visual appearance.
  • Action Following (Action): Uses an Inverse Dynamics Model (IDM) to measure how accurately videos execute each agent's input controls.
  • Cross-View Consistency: Measures 3D reprojection error (RPE) across views via DROID-SLAM, where our lower RPE confirms better geometric alignment across viewpoints.

Thank you for your response to my questions. While partial of my concerns are addressed, several issues remain:

  • Experiment part (w2, q1, q2): The progressive experiments do not establish whether these modules interact. Authors mention the modules are 'orthogonal' across stages, but this explanation does not necessarily mean that they do not interfere with each other.

  • W2/Q1: In the response to W2/Q1, you state that the first row of tab. 3 matches the final model in tab. 1. However, this row reports FVD 228 and PSNR 17.56, matching tab. 2 instead. The final model in tab. 1 reports FVD 159 and PSNR 17.58, matching the Causal VGGT-Ω\Omega row of tab. 3. The explanations does not clarify the experiment settings or results.

Therefore, I'll maintain the current score.

AuthorsOpenReview ↗

Kindly Remind

Dear Reviewer,

As the author–reviewer discussion period is coming to a close, we wanted to send a kind reminder to check if our response, revised manuscript, and additional experiments have addressed your concerns. Please let us know if you have any further questions—we would be very happy to clarify before the discussion deadline. Thank you again for your time and constructive feedback!

AuthorsOpenReview ↗

Follow up clarification of our result.

Thank you for your careful reading and follow-up feedback.

1. Progressive Ablation and No Module Interference

  • Progressive ablation: Evaluating the full combinatorial grid across the two stages (3×5=153 \times 5 = 15 combinations) is computationally prohibitive; following standard practice in video world models (e.g., Self Forcing [G1] and others), we adopt a progressive ablation that retains the optimal setting at each step.
  • No negative interference: As shown in the combined Table 2 & Table 3 below, stacking our optimal modules yields monotonic improvements across all metrics without degrading earlier stages—adding Causal VGGT-Ω onto the optimal AIE (base = 20) (Row 7 vs. Row 3) improves FVD (228 → 159) and RPE (0.75 → 0.59) while further boosting AIE’s Action accuracy (89.7% → 91.2%), confirming that the modules do not negatively interfere.

2. Clarification of Table Typo and Unified Table 2 & Table 3 (W2/Q1)

We sincerely apologize for the typo in our previous response: "matching with the final model in Table 1" was a typo for "matching with the final model in Table 2". Your reading is completely correct: the first row of Table 3 (FVD 228, PSNR 17.56) corresponds to the best AIE configuration in Table 2, whereas the last row of Table 3 (Causal VGGT-Ω, FVD 159, PSNR 17.58) corresponds to our final model in Table 1.

To make the progression completely clear, we provide the joint, end-to-end version of Table 2 and Table 3 below:

RowAgent Identity Embedding (AIE)World State EncoderFVD ↓LPIPS ↓SSIM ↑PSNR ↑Action ↑RPE ↓Reference in Paper
1w/o AIENone2360.370.4717.5288.60.75Table 2, row 1
2AIE (base=10k)None2340.380.4817.5389.20.75Table 2, row 2
3AIE (base=20)None (AIE only)2280.360.5117.5689.70.75Table 2, row 3 = Table 3, row 1
4AIE (base=20)Wan VAE2560.360.5017.3888.90.71Table 3, row 2
5AIE (base=20)DINOv22320.360.5017.4889.40.72Table 3, row 3
6AIE (base=20)VGGT1790.350.5117.7290.60.67Table 3, row 4
7AIE (base=20)Causal VGGT-Ω (Ours)1590.330.5217.5891.20.59Table 3, row 5 = Ours in Table 1

Thank you again for your valuable feedback and time in reviewing our work. We hope these clarifications resolve your remaining concerns and kindly ask if you would reconsider your assessment of our paper.

Reviewer_8chuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents MultiWorld, a framework for interactive multi-agent, multi-view video world modeling. The method combines agent-specific action conditioning with a shared dynamic world state to generate consistent observations across different viewpoints. The authors introduce Causal Teacher Distillation (CTD) and Data-Guided Correction (DGC), together with several inference optimizations to support real-time generation. Experiments on a two-player game and simulated multi-robot manipulation tasks show improvements in visual quality, action following, and cross-view consistency.

Critical strengths

Fisrt, a meaningful and practical problem. The paper addresses an important challenge in interactive world modeling. Supporting multiple agents with different viewpoints is considerably more difficult than single-agent video generation, especially when their actions affect the same environment. I appreciate that the authors consider action control, shared-world consistency, and real-time performance together rather than treating them as separate problems.

Second I would say a well-designed and reasonably motivated framework. The overall method is coherent, and the different components serve clear purposes. In particular, the Dynamic World State Model provides a useful way to share information across viewpoints, while CTD and DGC address the difficulties of few-step autoregressive generation. The ablation results are encouraging, especially the improvement from adding DGC in Table 4.

Third is that the paper has a good experimental results in the multiplayer setting. MultiWorld performs well against the evaluated baselines, with clear improvements in FVD, action following, and reprojection error. The qualitative results are also helpful, showing interactions between agents and some shared physical effects across views. I also appreciate the inclusion of failure trajectories in the robotics experiments.

Last bu not least, a convincing effort toward real-time deployment. The authors go beyond offline video generation and implement an interactive system. Achieving 24.8 FPS for two synchronized views is a meaningful engineering result. The discussion of parallel generation and pipelined decoding also makes the deployment design relatively easy to follow.

Critical weaknesses

First problem is that the evaluation of the final real-time model could be more complete. Table 1 compares bidirectional models, while the deployed system uses the distilled few-step model. Although Table 4 evaluates distillation quality, it does not provide the same action-following and cross-view consistency metrics as Table 1. It is therefore somewhat difficult to judge how much of the original model's performance is preserved after distillation, especially at the two-step setting used for real-time generation.

Also, the improvements in the robotics experiments are relatively modest. While the results in the multiplayer game are convincing, the gains in the robotics setting are much smaller. For example, the differences in FVD, action following, and RPE between MultiWorld and COMBO are limited. Concat-View also performs better on some metrics, although it uses a different training setup. A more detailed discussion of these results would help clarify where the proposed method provides the most benefit.

Questions for authors
  1. How important is the continuous update of the shared world state, compared with simply extracting geometric features from recent observations? An ablation comparing dynamic memory updates with a fixed or short-window representation would help clarify this contribution.

  2. Could the authors report action-following accuracy, RPE, and video quality for the final two-step model used in the real-time system? A comparison between the teacher-forcing model and the deployed model under the same evaluation protocol would be particularly helpful.

  3. How does generation quality change as the number of agents or viewpoints increases? Also, are there particular interaction patterns, such as collisions or tightly coordinated actions, where MultiWorld still struggles to maintain consistency?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8chu

8chu [W1/Q2] Complete Evaluation of the Distilled Real-Time Model with more baselines

Thank you for this constructive suggestion. We provide action accuracy (Action) and cross-view geometric consistency (RPE) for the distilled few-step model with Self-Forcing [G1], CMD [G2] and GammaWorld [G3] baselines. Table. R1 shows our base model significantly outperforms existing baselines, while 4/8 NFE models remain comparable with the teacher forcing base model. Although the 2-step model has worse FVD than teacher forcing model, it still consistently outperforms Self-Forcing, Gamma-World 4NFE setting.

8CHU [W2] Robotics Results and Methodological Advantages:

Thank you for pointing this out. We clarify the robotics results from three perspectives:

  1. Capability Difference Beyond Metrics: MultiWorld is the first work to support variable numbers of agents and views, which the baselines do not support.
  2. The robotics scene is visually simpler than the game because of its static cameras. Even so, MultiWorld still achieves clear relative gains over baselines, reducing FVD by 4.0% and RPE by 5.0%.
  3. Concat-View* is evaluated on only 2 views whereas MultiWorld is evaluated on 4 views, so it is natural for Concat-View* to have higher scores on a simpler subset.

8Chu [Q1] Ablation on Dynamic World State Update Ratio

Thank you for this constructive suggestion. Continuous memory updates are essential when the camera moves into regions unseen in initial observation. To clarify this effect, we conducted an ablation study comparing three update frequencies: (1) No update (fixed to the initial frame), (2) Updating every 4 latent frames, and (3) Updating every 1 latent frame (our default setting). As shown in Table R2, more frequent memory updates consistently yield better performance, with the gain most prominently reflected in cross-view 3D consistency (RPE↓\text{RPE}\downarrow) as the evolving shared geometry stays accurately synchronized across viewpoints.

Table R2. Ablation on Dynamic World State Update Ratio.

SettingFVD ↓\downarrowLPIPS ↓\downarrowSSIM ↑\uparrowPSNR ↑\uparrowAction ↑\uparrowRPE ↓\downarrow
No Memory Updates1810.330.5217.2890.10.65
Memory Updates Every 4 Latent1730.330.5117.3490.30.64
Memory Updates Every 1 Latent1590.320.5117.4691.20.59

8Chu [Q3] Increasing Agents & Failure Case Analysis

We thank the reviewer for the thoughtful question.

  1. Scaling with More Agents and Viewpoints: Increasing the number of agents and viewpoints does not noticeably degrade generation quality. As shown in Table R3, increasing from 2 agents to 4 agents maintains consistent visual fidelity and action-following performance.
  2. Failure Cases: As discussed in Figure 13, distant agents occupying very small image regions can appear blurry.

Table. R3. Ablation on agent amount during inference.

SettingFVD ↓\downarrowLPIPS ↓\downarrowSSIM ↑\uparrowPSNR ↑\uparrowAction ↑\uparrowRPE ↓\downarrow
2-agents950.070.9026.5188.31.21
4-agents960.070.9026.6088.71.52

Thanks for your rebuttal! Most of my concerns have been resolved, therefore I remain my current rating.

We also thanks for your thoughtful and comprehensive review. Best regrads.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response [1/2] for All Reviewers and AC

We sincerely thank all reviewers (@8chu, @wGGW, @EpKE) for their thoughtful and professional feedback, which has greatly helped us polish and strengthen our work. Below, we summarize the core strengths recognized by the reviewers, the major questions raised, and our corresponding responses and new experiments. All changes in the revised PDF are highlighted in blue.

Highlighted Strengths from Kindly Reviewers:

  • Timely Problem & Valuable Datasets (@8chu, @wGGW, @EpKE): All reviewers agree that interactive multi-agent, multi-view world modeling is an important and practical frontier beyond single-agent settings, with @8chu and @EpKE highlighting the independent value of our released datasets (including robotic failure trajectories).
  • Well-Motivated Framework & Effective Solutions(@8chu, @wGGW, @EpKE): Reviewers commend our coherent design for joint action control and shared-world consistency, specifically praising Causal Teacher Distillation with Data-Guided Correction for resolving training collapse in few-step autoregressive distillation (@8chu, @EpKE).
  • Strong Performance & Genuinely Real-Time System (@8chu, @wGGW, @EpKE): All reviewers recognize our clear gains in FVD, action controllability, and cross-view 3D consistency in game setting and praise our complete, genuinely real-time interactive system (24.8 FPS across synchronized views).

Our contributions

  • MultiWorld Framework: A unified world model supporting joint action control across variable numbers of agents and views.
  • Agent Identity Embedding & Dynamic World State: RoPE-based identity encoding for per-agent control and causal geometry memory for 3D cross-view consistency.
  • Real-Time Deployment: Causal Teacher Distillation (CTD) with Data-Guided Correction to prevent rollout collapse in few-step generation. View-parallel pipelined inference achieving 24.8 FPS.

Highlighted Experiments and Responses

  • Complete Few-step Model Metrics & New Baselines (@8chu, @wGGW, @EpKE): We completed the full set of evaluation metrics—including Action-following and RPE—for our distilled few-step models across 2/4/8-step settings, and comparisons against distillation baselines (Self-Forcing, CMD) and concurrent multi-agent world models (Gamma-World).
  • Clarification of Robotics Metrics & Capabilities (@8chu, @EpKE): We explained why visual metrics saturate in the static-camera robotics scene while highlighting our consistent relative gains (4.0% in FVD, 5.0% in RPE), clarified that Concat-View* is only evaluated on a simpler 2-view subset rather than 4 views, and emphasized our unique capability to support variable agent/view counts with a shared 3D environment state.
  • Ablation on Evolving World State Updates (@8chu): We added an ablation comparing different environment update ratios, demonstrating that continuous updates are crucial for preserving cross-view 3D consistency.
  • Contribution of Causal Teacher Distillation (@EpKE, @wGGW): CTD differs from Self-Forcing by using a causal teacher, and differs from CMD by conditioning on GT context, yielding more accurate score estimation and better performance.
  • Data-Guided Correction & Action Controllability (@EpKE): We don't observe that DGC largely hurts diversity, and we added new qualitative figures in the revised paper for evidence.

Experiement

Table R1. Full evaluation with distilled baselines and 2/4/4 NFE results.

MethodFVD ↓\downarrowPSNR ↑\uparrowLPIPS ↓\downarrowSSIM ↑\uparrowAction ↑\uparrowRPE ↓\downarrow
Ours-TF-base15917.580.330.5291.20.59
Self-Forcing-4NFE30917.950.330.5388.10.76
GammaWorld-4NFE35616.070.390.4789.50.69
CMD-4NFE34517.800.350.5388.30.73
Ours-8NFE17617.360.340.5090.30.60
Ours-4NFE21217.740.330.5490.10.62
Ours-2NFE29017.800.340.5388.50.65
AuthorsOpenReview ↗

General Response [2/2] for All Reviewers and AC

Global Citations

[G1] Huang et al., "Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion," 2025.

[G2] Liu et al., "Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players," 2026.

[G3] Bandyopadhyay et al., "Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation," 2026.

[G4] Wang et al., "VGGT-Ω: Scaling Visual Geometry Grounded Transformer," 2026.

[G5] Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding," 2024.

[G6] Zhu et al., "Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation," 2026.

We are glad to address the reviewers' questions and sincerely hope our responses and revisions have resolved all concerns. Please let us know if there are any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.