- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces WorldWeaver, a streaming multi-agent video diffusion model with recurrent World State Registers (WSRs). The registers are intended to maintain shared world information across agents and generated video chunks. They are trained using auxiliary supervision from agent statistics, bird's-eye-view features, and scene descriptions. The method also uses a Mixture-of-Transformers (MoT) architecture to separate visual and state processing, followed by self-forcing training. Experiments in two-player Minecraft environments show improvements over Solaris and a frame-concatenation baseline.
Critical strengths
Maintaining a shared world state is a reasonable approach to multi-agent video generation. In this setting, individual agents have only partial observations, and a limited history of generated frames may not be enough to preserve information across different viewpoints. The proposed registers offer a separate mechanism for carrying such information between generation steps. Table 1 shows clear improvements over the retrained Solaris baseline: Grounding increases from 81.3 to 93.8, Building from 9.4 to 28.1, and Consistency from 57.8 to 76.6. These are encouraging results, particularly given the paper's emphasis on cross-agent consistency.
The ablations also provide some support for the architectural choices. In Table 2, adding registers without explicit state supervision already improves Memory and Building performance, suggesting that the recurrent register mechanism is useful on its own. Table 3 further shows that MoT outperforms the dense transformer across all five VLM evaluation categories under scene-text supervision. This is a useful observation, although it remains unclear how much of the benefit comes from the architectural separation versus the way register supervision is applied.
Critical weaknesses
My main concern is how the register supervision works during self-forcing. Section 3.4 states that Stage 3 uses self-generated frames and updated registers while retaining the supervision loss from Stage 2. However, I could not determine how the supervision targets are handled when the generated trajectory differs from the recorded trajectory. For example, if the generated agent moves to a different position than in the ground-truth rollout, would the corresponding register still be supervised using the original simulator state? If so, the register target might no longer agree with the generated observation. This is not necessarily an error in the method, but the paper needs to explain the target assignment more clearly. Otherwise, it is difficult to understand whether the registers represent the evolving generated world or mainly reconstruct states from the reference trajectory.
The combined-supervision results are also difficult to interpret. In Table 2, Memory accuracy drops from 62.5 with registers only to 46.9 with all three supervision signals, despite an improvement in the overall WorldScore. This is a fairly large decrease for a method designed to preserve world information. Moreover, Appendix D.1 indicates that the single-supervision variants can use all 256 register tokens, while the combined setting allocates them as 64/64/128 across different prediction heads. The comparison therefore changes both the supervision signals and the register capacity available to each head. Without controlling for this difference, I am not sure the ablation establishes that combining the three supervision objectives is beneficial.
I would also like to see more direct evidence of what the registers actually learn. The current evaluation mainly measures generation quality and consistency through downstream metrics. These results are useful, but they do not directly establish whether the recurrent representations accurately track the shared world state. Since explicit world-state modeling is the central contribution, measurements such as agent-position prediction error, BEV agreement, or cross-agent state consistency would help validate this claim. It would be particularly interesting to see whether register accuracy deteriorates as the rollout becomes longer.
Questions for authors
-
During Stage 3 self-forcing, how are register supervision targets assigned when generated observations deviate from the ground-truth trajectory? Are the registers supervised to represent the generated world or the recorded simulator state?
-
What explains the drop in Memory accuracy from 62.5 to 46.9 under combined supervision? Could the authors provide a capacity-controlled ablation, keeping the register allocation comparable between single-supervision and combined-supervision settings?
-
Since the method explicitly models a shared world state, could the authors report direct measurements of register accuracy, such as agent-position prediction error, BEV agreement, or cross-agent state consistency? How do these measurements change over longer rollouts?