acceptodds
Under review as a conference paper at ICLR 2027

Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players

Abstract

World models for interactive video generation have largely focused on single-agent settings, where future observations are generated from a single control signal. However, many generated environments require multi-agent interaction: multiple players, robots, or embodied agents act simultaneously within a shared space. Scaling world models to such settings requires a principled multi-agent design: agents should remain independently controllable, permutation-symmetric, and support efficient inference while maintaining consistency across time and perspectives. In this paper, we present our generative multi-agent world model for interactive simulation. It introduces Simplex Rotary Agent Encoding, a parameter-free extension of 3D RoPE that represents agents as vertices of a regular simplex in rotary angle space. This gives each agent a distinct phase while making all agents permutation-equivalent, enabling scalable agent identity without learned per-slot identities or a fixed agent ordering. To avoid dense all-to-all attention across agents, we further propose Sparse Hub Attention, where learnable hub tokens mediate token interaction across agents, reducing cross-agent attention cost from quadratic to linear in the number of agents. For real-time rollout, we distill a full-context diffusion teacher into a causal student that generates temporal blocks sequentially with KV caching, enabling action-responsive generation at 24 FPS. Experiments in multiplayer virtual environments show that our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines, while generalizing from two to four players without additional training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Model, Multi-Players, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DBi3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Gamma-World, a generative multi-agent world model for interactive video generation. The method introduces Simplex Rotary Agent Encoding to represent agent identities using vertices of a regular simplex, and Sparse Hub Attention to reduce the computational cost of cross-agent communication. A bidirectional diffusion teacher is distilled into a causal student for streaming generation with KV caching. Experiments in Minecraft environments show improvements in video quality and inference efficiency, along with qualitative examples of generalization from two to four agents.

Critical strengths

The paper addresses a practical difficulty in scaling interactive world models to multiple agents. Dense attention across agents becomes expensive as the number of viewpoints increases, and the proposed Sparse Hub Attention offers a relatively simple way to reduce this cost. The complexity analysis in Section 3.3 indicates linear scaling with the number of agents when the block size and hub capacity are fixed. Figure 3 provides supporting measurements: with eight agents, DiT latency decreases from 611 ms using dense attention to 246 ms using Sparse Hub Attention. This is a substantial reduction and makes the approach relevant to interactive generation.

The generation results are also promising. In Table 1, Gamma-World reduces FVD from 333.8 to 184.1 on Memory and from 443.1 to 280.0 on Consistency compared with Solaris. Table 2 provides some evidence for the proposed architectural components. Replacing learned view embeddings with Simplex Rotary Agent Encoding reduces FVD from 256.3 to 228.5, and adding Sparse Hub Attention further reduces it to 223.4. The improvements are not uniform across all metrics, but the overall results suggest that the proposed design can improve generation quality while reducing computational overhead.

Critical weaknesses

My main concern is the claimed permutation symmetry of Simplex Rotary Agent Encoding. The paper shows that agents represented by vertices of a regular simplex have equal pairwise distances in the rotary angle space. However, I am not sure this is sufficient to establish permutation equivariance of the complete attention computation. The attention scores also depend on the query and key representations and their relative rotations. Equal geometric distances between agent identities do not by themselves establish that permuting agent assignments produces correspondingly permuted attention outputs. Random assignment during training may improve robustness to agent ordering, but this is different from a formal equivariance guarantee. Since permutation symmetry is one of the main motivations for the encoding, the precise claim needs to be clarified. An explicit evaluation under different agent-identity assignments would also be useful.

The evidence for generalization beyond two agents is limited. Figure 5 shows qualitative four-agent generation results, but the main quantitative evaluations are conducted in the two-agent setting. Figure 3 demonstrates that the architecture scales favorably in terms of computation, which does not necessarily mean that generation quality and cross-agent consistency are maintained as more agents are introduced. In addition, the FID and FVD results in Table 1 do not directly measure whether an agent's actions are correctly reflected in other agents' observations. I would like to see quantitative results for four-agent generation, particularly on action controllability and cross-agent consistency. The real-world robotic examples in Figure 6 are interesting, but they are also qualitative.

I also have some concerns about the fairness of the baseline comparisons. The improvements over Solaris are substantial, but it is not sufficiently clear from the reported comparisons whether the training data, optimization budgets, and evaluation settings are fully matched. This matters because the performance differences could reflect factors beyond the proposed architecture. The encoding ablations are useful, but more explicit experimental configurations would help determine how much of the improvement comes from Simplex Rotary Agent Encoding and Sparse Hub Attention themselves.

Questions for authors
  1. Does Simplex Rotary Agent Encoding guarantee permutation equivariance of the attention computation, or is the intended property robustness to agent permutations learned through random assignment? Could the authors provide either a formal clarification or an experiment comparing outputs under different agent-identity assignments?

  2. Could the authors provide quantitative results for zero-shot four-agent generation? I am particularly interested in how cross-agent consistency, action controllability, and video quality change when moving from two to four agents.

  3. Are the Solaris comparisons and agent-encoding ablations conducted with matched training data, optimization budgets, and evaluation protocols? More details would help assess the contribution of the proposed architectural components.

Reviewer_b4FfOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a three-stage pipeline to train a world model (Gamma-World) that allows multiple players to interact with each other in a shared scene. Gamma-World introduces a novel causal model architecture that uses a sparse hub to share scene-specific information, enabling cross-agent information passing. Compared to existing baselines, Gamma-World shows stronger performance and better scalability with respect to the number of players. Extensive results support the effectiveness of this method.

Critical strengths
  • Novel Idea: The proposed hub tokens serve as a bridge for sharing information across players, and the Simplex Rotary Agent Encoding is clean and effective.

  • Strong Scalability: Gamma-World achieves higher computational efficiency than models that require full attention between the two players.

Critical weaknesses
  • Missing Evaluation: The paper does not report VLM accuracy following the Solaris pipeline, which is critical for evaluating correctness and consistency across the two agents. Including this evaluation would also improve the clarity of the paper.

  • Missing Evaluation of the Hub Tokens: While the hub tokens are proposed to serve as a bridge for cross-player information sharing, they provide only limited improvement over the baseline (FID from 31.5 to 30.2 in Table 6). In addition, there are no experiments quantifying what kind of information is encoded in the hub tokens. The absence of such analysis weakens the support for the paper's central claim.

  • Unfair Main Evaluation: The main evaluation results reported in Table 1 are not a fair comparison. Gamma-World is trained with a larger model size and a longer sequence length for bidirectional model training, as well as with DMD supervision. It therefore remains unclear whether sparse attention based on hub tokens can match the performance of full attention across players with the same base model. Providing a fair baseline would help readers understand the performance–efficiency trade-off.

I'm happy to raise my score if my concern has been addressed.

Questions for authors

It is exciting to see that the model can be extended to more than two players after the three-stage training. However, the visualizations of the four-agent generation results do not show high consistency across players. Can you provide quantitative results to establish a baseline for the four-player setting?

3 comments

Thank you for preparing such a detailed rebuttal.

Q1/Q3. For the VLM evaluation results, could you provide the per-category VLM accuracies rather than reporting only the mean VLM accuracy? Also, I respectfully disagree with the claim that “These results show that Sparse Hub Attention achieves overall generation quality comparable to dense attention.” Gamma-World provides this analysis only for the two-player setting. Please consider narrowing the claim accordingly, rather than presenting it as a general conclusion.

Q2. There could be multiple reasons why the variant without hub attention does not work well, such as a mismatch with the training setting or sensitivity to sequence length. How does changing  K affect the performance gap between models with and without hub attention? This analysis could help the audience better understand the effectiveness of hub tokens.

We thank the reviewer for the helpful follow-up.

[1 (Q1): Per-category VLM results]

Below is the category breakdown of the VLM evaluation reported in our rebuttal. All evaluation data come from the publicly available Solaris evaluation dataset.

CategorySolarisOurs (bidirectional)Ours (distilled)GT calibration
Translation56.25%56.25%62.50%93.75%
Grounding56.25%84.38%84.38%100.00%
Memory43.75%71.88%75.00%96.88%
Building†21.88%50.00%50.00%93.75%
Consistency81.25%75.00%51.56%96.88%
Mean across categories51.88%67.50%64.69%96.25%

Each category contains 32 instances, except Consistency, which combines 32 same-scene and 32 opposite-scene instances. The mean equally weights the five categories. We use gemini-3-flash-preview for all categories except Building. For Building, we use gemini-3.8-flash, based on its higher GT calibration accuracy on identical questions and images: 93.75%, versus 78.125% with gemini-3-flash-preview.

Both our variants achieve higher success rates on grounding, memory, and building than Solaris, contributing to their higher average success.

The Consistency category also uses the Solaris evaluation dataset and VLM prompts. It includes same-scene tests (two players turn to face the same direction, testing consistency in shared scene content) and opposite-scene tests (two players turn to face opposite directions, testing whether their views show different scenery). Both tests ask VLM: “Do these two screenshots show the same scenery?”

Same-scene. A pair passes if the VLM answers “yes” to the scenery question. However, in one scene generated by Solaris, one player’s view shows the other player, but the other player is absent from the corresponding GT view. The VLM still answers “yes”, so the example passes despite incorrect player visibility. We will include the corresponding images in the revision. To complement this judgment, we evaluate the 3D consistency on all 32 same-scene episodes by reprojection. We reproject Player 1 into Player 2 using cameras fitted from GT and static geometry reconstructed from the scene seed. RGB L1 is measured within identical visibility masks, excluding HUD and first-person foreground regions, and averaged equally over episodes.

MethodRGB L1 ↓
GT0.02889
Solaris0.17586
Ours (bidirectional)0.16811
Ours (distilled)0.18962

These results suggest that our models achieve cross-view consistency comparable to Solaris in this reprojection evaluation.

Opposite-scene. The same question is scored against “no”. However, opposite viewing directions can show visually similar terrain, so “same scenery” does not directly test whether each generated viewpoint is correct. Therefore, we believe this VLM test can give an inaccurate assessment of consistency.

We thank the reviewer for prompting this analysis. We will add these evaluation details and qualitative comparisons with Solaris in the revision.

[2 (Q3): Scope of the dense-attention comparison]

We thank the reviewer for pointing out the need to qualify this claim. We agree and will clarify that Sparse Hub Attention achieves generation quality comparable to dense attention in the evaluated two-player setting.

[3 (Q2): Hub capacity and the intervention]

We agree that measuring the with/without-hub gap across K would better characterize hub effectiveness. We also agree that retraining the corresponding variants, rather than disabling hub attention only at inference, would further strengthen this comparison.

To address this question and potential training–inference mismatch, we will compare models trained with different hub capacities against a baseline trained without hub communication, using matched training budgets and evaluation lengths. Due to time and GPU resource constraints, we cannot complete these additional training runs during the rebuttal period and will include this analysis in the revision.

We have updated the revision and added the qualitative examples discussed in our previous response to Appendix H (Figures 7–8). These examples illustrate the limitations of the VLM criterion. Due to time constraints, the current revision does not yet incorporate all the changes discussed in our rebuttal. We will include all relevant experiments and clarifications in the final revision.

Reviewer_dfJdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Gamma-World, a generative multi-agent video world model designed to support independently controllable agents, efficient cross-agent communication, and scalable interactive video generation. The method introduces Simplex Rotary Agent Encoding (SRAE), which assigns agent identities to vertices of a regular simplex in rotary angle space, and Sparse Hub Attention (SHA), which mediates cross-agent communication through a small set of shared hub tokens. The system further adopts a multi-stage teacher–student training and distillation pipeline to support causal video generation with KV caching.

Experiments on multiplayer Minecraft scenarios show improvements in video generation quality over the reported baselines and substantial attention efficiency gains as the number of agents increases. The paper also presents qualitative examples of zero-shot generalization from two-agent training to four-agent inference.

Critical strengths
  1. Relevant and meaningful problem. Extending interactive video world models to multi-agent settings is an important research direction. The requirements of independent action control, cross-perspective consistency, and computational scalability are well motivated.

  2. Promising efficiency improvements. Sparse Hub Attention provides a reasonable architectural approach to reducing dense cross-agent interactions. The complexity analysis in Section 3.3 is sound under fixed hub capacity, and Figure 3 demonstrates meaningful computational savings as the number of agents increases.

  3. Positive architectural ablation results. Table 2 shows that Simplex Encoding consistently improves the reported generation metrics over learned View Embeddings. The comparison with Dense Attention and the hub-token ablation in Table 6 provide useful evidence about the quality–efficiency trade-off.

  4. Practical system design. The integration of multi-agent conditioning, block-causal generation, distillation, and KV caching is valuable from a systems perspective. The reported qualitative results are encouraging.

Critical weaknesses

W1. The claimed permutation-symmetry property of Simplex RoPE is insufficiently characterized.

The paper motivates SRAE primarily through permutation symmetry and establishes that the simplex vertices have equal pairwise Euclidean distances in rotary angle space. However, equal pairwise distances do not, by themselves, establish the claimed symmetry properties of the resulting Transformer. In particular, RoPE attention logits depend on the individual rotary phase differences and query/key features, rather than solely on Euclidean distances between agent embeddings. The paper should distinguish between: Permutation equivariance under a joint reordering of agent streams and their associated identity encodings. Invariance or robustness to reassignment of simplex vertices to otherwise identical agent trajectories. The former may already follow from shared model parameters and a permutation-equivariant attention topology, without requiring simplex geometry. The latter is a stronger property that is not established by the current proof. Moreover, the ablation in Table 2 compares SRAE only against learned View Embeddings. This does not isolate whether the improvements arise specifically from the simplex geometry, the rotary parameterization, or random identity assignment during training.

W2. The experimental metrics do not directly establish the claimed improvements in action controllability and cross-agent consistency.

A central motivation of the paper is that multiple agent observations must remain consistent with the evolution of a shared environment. However, the quantitative comparison in Table 1 reports FVD and FID, including for the evaluation category named "Consistency." These metrics characterize video quality or distributional similarity but do not directly establish whether a physical event generated from one agent's action is correctly reflected in another agent's observation. Similarly, the reported improvements in action controllability are not supported by a dedicated quantitative action-adherence metric. This is particularly important because Solaris provides public evaluation scenarios and VLM-based metrics targeting cross-player movement, grounding, building, and consistency. It would be valuable to evaluate Gamma-World using comparable capability-oriented protocols.

W3. The central claim of zero-shot generalization beyond two agents lacks quantitative validation.

The ability to scale from two-agent training to four-agent inference is one of the most important claims of the paper. Appendix E explains that training uses two active agents sampled from a pool of four simplex vertices, and Figure 5 shows qualitative four-agent generation results. These examples demonstrate that the architecture can produce outputs for additional agents without retraining. However, it remains unclear whether the generated four-agent rollouts preserve video quality, action adherence, and shared-world consistency to a comparable degree. Figure 3 evaluates computational efficiency at different agent counts, but does not establish generation quality or interaction correctness in the four-agent setting.

W4. Important experimental and reproducibility details are insufficiently specified.

Although the appendix provides useful architecture and optimization details, several aspects needed to interpret or reproduce the main results remain unclear. First, the paper does not sufficiently specify the training and evaluation dataset sizes, episode-level splits, test-set composition, or the precise evaluation protocol. Table 1 also lacks a clear account of whether the baseline results were reproduced under matched conditions or taken from previous reports. Since the systems use different pretrained backbones and training pipelines, this information is important for interpreting the reported improvements. Second, the generation-quality experiments do not report uncertainty estimates, making it difficult to assess the reliability of smaller performance differences. Third, the reported 24 FPS interactive generation claim is not accompanied by a sufficiently explicit end-to-end timing breakdown. Figure 3 focuses on DiT and attention latency rather than the complete interactive inference pipeline. Finally, the FVD of the full model in Table 2 (223.4) differs from that of the distilled model in Table 5 (239.7), and the relationship between these evaluation settings is not clearly explained.

Questions for authors

The following questions would be particularly important for updating my assessment.

Q1. Symmetry guarantee: What precise mathematical notion of permutation symmetry does SRAE guarantee at the model level? Can the authors distinguish the effect of simplex geometry from the equivariance potentially induced by shared weights and a symmetric attention mask?

Q2. Direct capability evaluation: Can the authors report action-following and cross-agent event-consistency metrics, ideally using the publicly available Solaris evaluation protocols?

Q3. Four-agent generalization: How does generation quality and cross-view consistency change from two to four agents? Does SRAE outperform simpler agent encodings under the same two-agent training setup?

Q4. Evaluation fairness and latency: Were the baseline results obtained using matched evaluation data and generation protocols? What operations are included in the reported 24 FPS throughput?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.