acceptodds
Under review as a conference paper at ICLR 2027

Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

Abstract

Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings. We present WorldWeaver, a streaming multi-agent video diffusion model that augments rollout with recurrent cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. We ground these registers with supervision signals spanning individual agent status, global state views including bird’s-eye views, and scene text. We further improve the architecture with a Mixture-of-Transformers design that uses separate weights for world state modeling and visual frame modeling. Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality.

Primary area:
generative models
Keywords:
Generative model, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k9VfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldWeaver, a streaming multi-agent video diffusion model with recurrent World State Registers (WSRs). The registers are intended to maintain shared world information across agents and generated video chunks. They are trained using auxiliary supervision from agent statistics, bird's-eye-view features, and scene descriptions. The method also uses a Mixture-of-Transformers (MoT) architecture to separate visual and state processing, followed by self-forcing training. Experiments in two-player Minecraft environments show improvements over Solaris and a frame-concatenation baseline.

Critical strengths

Maintaining a shared world state is a reasonable approach to multi-agent video generation. In this setting, individual agents have only partial observations, and a limited history of generated frames may not be enough to preserve information across different viewpoints. The proposed registers offer a separate mechanism for carrying such information between generation steps. Table 1 shows clear improvements over the retrained Solaris baseline: Grounding increases from 81.3 to 93.8, Building from 9.4 to 28.1, and Consistency from 57.8 to 76.6. These are encouraging results, particularly given the paper's emphasis on cross-agent consistency.

The ablations also provide some support for the architectural choices. In Table 2, adding registers without explicit state supervision already improves Memory and Building performance, suggesting that the recurrent register mechanism is useful on its own. Table 3 further shows that MoT outperforms the dense transformer across all five VLM evaluation categories under scene-text supervision. This is a useful observation, although it remains unclear how much of the benefit comes from the architectural separation versus the way register supervision is applied.

Critical weaknesses

My main concern is how the register supervision works during self-forcing. Section 3.4 states that Stage 3 uses self-generated frames and updated registers while retaining the supervision loss from Stage 2. However, I could not determine how the supervision targets are handled when the generated trajectory differs from the recorded trajectory. For example, if the generated agent moves to a different position than in the ground-truth rollout, would the corresponding register still be supervised using the original simulator state? If so, the register target might no longer agree with the generated observation. This is not necessarily an error in the method, but the paper needs to explain the target assignment more clearly. Otherwise, it is difficult to understand whether the registers represent the evolving generated world or mainly reconstruct states from the reference trajectory.

The combined-supervision results are also difficult to interpret. In Table 2, Memory accuracy drops from 62.5 with registers only to 46.9 with all three supervision signals, despite an improvement in the overall WorldScore. This is a fairly large decrease for a method designed to preserve world information. Moreover, Appendix D.1 indicates that the single-supervision variants can use all 256 register tokens, while the combined setting allocates them as 64/64/128 across different prediction heads. The comparison therefore changes both the supervision signals and the register capacity available to each head. Without controlling for this difference, I am not sure the ablation establishes that combining the three supervision objectives is beneficial.

I would also like to see more direct evidence of what the registers actually learn. The current evaluation mainly measures generation quality and consistency through downstream metrics. These results are useful, but they do not directly establish whether the recurrent representations accurately track the shared world state. Since explicit world-state modeling is the central contribution, measurements such as agent-position prediction error, BEV agreement, or cross-agent state consistency would help validate this claim. It would be particularly interesting to see whether register accuracy deteriorates as the rollout becomes longer.

Questions for authors
  1. During Stage 3 self-forcing, how are register supervision targets assigned when generated observations deviate from the ground-truth trajectory? Are the registers supervised to represent the generated world or the recorded simulator state?

  2. What explains the drop in Memory accuracy from 62.5 to 46.9 under combined supervision? Could the authors provide a capacity-controlled ablation, keeping the register allocation comparable between single-supervision and combined-supervision settings?

  3. Since the method explicitly models a shared world state, could the authors report direct measurements of register accuracy, such as agent-position prediction error, BEV agreement, or cross-agent state consistency? How do these measurements change over longer rollouts?

Reviewer_HM7dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WorldWeaver, a multi-agent video world model that uses world state registers to record the state of multi-agents. World state registers include learnable tokens that store shared world information, and can track individual agent status. They are dynamically updated after each generated chunk. WorldWeaver train these registers with supervision signal from agent status, global state views and scene text. Experiments show that WorldWeaver improves logical consistency and generation quality.

Critical strengths
  • The paper introduces registers to record the agents' state to help multi-agent world models generate videos. The idea is interesting.

  • Writing is clear and easy to follow. Researchers could reproduce the method given the introductions in the paper.

  • Experiments show the proposed method performs better than the listed baselines (frame concat and Solaris). Architecture ablation shows that choosing MoT over dense model is more reasonable.

Critical weaknesses
  • In table 2, model performance on Memory drops significantly when adding all component to the register. The paper lacks a discussion on why introducing agent stats + bird's-eye view to the model could result in such a performance decline.

  • In table 5, the WSR number for different experiment settings might be unfair. For example, in combined setting, agent states only gets 64 tokens, which is less than agent states alone (256 tokens). The representation power for different token numbers is different. More experiments are needed to show that the performance is not related to the number of tokens.

  • The registers are inspected qualitatively and evaluated indirectly through video generation performance, but their state tracking accuracy and temporal drift are not quantitatively evaluated.

  • Authors cite Gamma-World, MultiWorld and MultiGen in the related work section, and these works also focus on memory design in world models. But these baselines do not appear in the experiment settings. No explanation is provided why these baselines are not included in the experiment. Missing experiments weakens the claim made by the paper.

Questions for authors
  • The supervision signal for the registers include agent stats. I wonder if such mechanism could be applied to single-agent world models. Because this method seems to apply a direct supervision over the agent's physical status (partially), can it improve the controllability or physical plausibility for general (single-agent) video world models?
Reviewer_FXwdOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorldWeaver, a streaming multi-agent autoregressive video diffusion model designed to improve long-term memory and cross-view consistency. Its central component is a set of recurrent World State Registers (WSRs) that maintain and update shared information across agents and generation steps. The registers are grounded through auxiliary supervision from agent statistics, bird's-eye-view representations, and scene descriptions. The framework further incorporates a Mixture-of-Transformers (MoT) architecture to separate state and visual processing, together with Self-Forcing training to reduce autoregressive rollout errors.

Experiments on two-agent Minecraft environments demonstrate improvements in video generation quality and logical consistency over the Solaris baseline.

Critical strengths

Strengths

S1. The paper addresses an important limitation of existing multi-agent video world models: maintaining a consistent shared world state across agents and over long generation horizons. Explicitly representing world states beyond local visual observations is a meaningful research direction.

S2. The proposed WSR provides a mechanism for maintaining and updating shared latent information across agents and rollout steps. Supervising the registers with agent statistics, BEV representations, and scene descriptions is a reasonable way to encourage structured state representations.

S3. The proposed method achieves improvements over the retrained Solaris baseline, particularly in Grounding, Building, and Consistency. Table 1 reports increases in VLM accuracy from 81.3 to 93.8 for Grounding, 9.4 to 28.1 for Building, and 57.8 to 76.6 for Consistency.

Critical weaknesses

Weaknesses

W1. Previous memory-based video generation approaches have explored recurrent latent representations for preserving historical information. Although WorldWeaver extends this idea to multi-agent world modeling and introduces explicit world-state supervision, the paper does not sufficiently distinguish the proposed WSR mechanism from existing recurrent memory approaches.

The authors should clarify which properties of WSR are technically distinctive beyond introducing a shared recurrent latent state, particularly regarding cross-agent information aggregation, state updating, and explicit supervision.

W2. The main contribution is the ability to maintain a persistent world state beyond the local observation window. However, the current evaluation primarily measures generated video quality and VLM-based task accuracy, rather than directly evaluating the accuracy and persistence of the stored world state.

Although the qualitative register readouts are informative, they do not fully establish whether WSR reliably preserves information that is no longer visible to any agent.

Experiments measuring state retention under increasing observation gaps, or controlled register-reset and register-replacement experiments, would provide stronger evidence that the learned registers function as persistent world memory.

W3. Table 2 shows that the benefits of different supervision signals are highly task-dependent. For example, the Registers-only variant achieves a Memory VLM accuracy of 62.5, whereas combining all supervision signals reduces it to 46.9. Similarly, agent-stat supervision reduces Grounding accuracy from 81.3 to 59.4.

Although the combined setting achieves the highest auxiliary WorldScore, it does not consistently outperform individual supervision configurations.

These results suggest potential interference between supervision objectives. A more detailed analysis of register capacity, loss weighting, and competing state representations would help explain why additional supervision sometimes degrades performance.

W4. MoT and Self-Forcing are established techniques. While their integration into the proposed framework is reasonable, the paper should clarify whether the adaptations introduce distinctive technical contributions for multi-agent world-state modeling.

Furthermore, Table 3 indicates that MoT does not consistently outperform the dense transformer. Without explicit register supervision, the Dense model achieves a higher WorldScore (95.5 versus 93.8). The advantages of MoT become more evident under scene-text supervision.

This suggests that the benefit of separating visual and state pathways depends on the supervision configuration. A parameter-matched comparison and additional analysis of representation interference would strengthen the architectural justification.

W5. Long-term memory and cross-agent consistency are central motivations of the paper. However, the reported experiments focus on two-agent Minecraft scenarios and primarily use VLM accuracy and FID.

More direct evaluations of world-state consistency, such as cross-agent spatial agreement, state prediction errors, and performance under increasing rollout lengths, would help establish whether the proposed framework maintains coherent world representations over time.

In particular, it would be useful to distinguish improvements in visual generation quality from improvements in the underlying geometric and logical consistency of the shared world.

Questions for authors

Q1. What is the key technical distinction between WSR and existing recurrent latent memory approaches? Is the primary novelty the shared multi-agent setting, explicit state supervision, or a new state-update mechanism?

Q2. Can the authors provide direct evidence that WSR retains information beyond the local context window, for example through register-reset experiments or evaluations with increasing observation gaps?

Q3. Why does combining all supervision signals reduce Memory performance compared with Registers-only? Does this indicate interference between different state-supervision objectives?

Q4. Can the authors evaluate cross-agent geometric or state consistency directly, rather than relying primarily on VLM-based evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.