acceptodds
Under review as a conference paper at ICLR 2027

MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines

Abstract

Video world models have shown immense promise for interactive simulation and entertainment, but current systems still struggle with two important aspects of interactivity: user control over the environment for reproducible, editable experiences, and shared inference where players hold influence over a common world. To address these limitations, we introduce an explicit external memory into the system, a persistent state operating independent of the model's context window, that is continually updated by user actions and queried throughout the generation roll-out. Unlike conventional diffusion game engines that operate as next-frame predictors, our approach decomposes generation into Memory, Observation, and Dynamics modules. This design gives users direct, editable control over environment structure via an editable memory representation, and it naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.

Primary area:
generative models
Keywords:
generative game engines, game design, video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_97ZBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MultiGen decomposes a diffusion-based game engine into Memory, Observation, and Dynamics modules. Memory stores 2D wall geometry and player poses; Observation generates the next frame from ray-cast disparity, recent images, and actions; Dynamics predicts pose increments from diffusion features and state information. The multiplayer version shares player poses and alive/dead states across model instances, with opponent-aware geometry and a kill-prediction head. Experiments in Doom/ViZDoom evaluate layout control and multiplayer visual consistency through image metrics, user preferences, opponent visibility, and bounding-box IoU. The reported real-time rate is approximately 20 FPS with one A100 per player, rather than a constant-hardware guarantee as the player count increases.

Critical strengths

A sensible decomposition with a useful control interface. The static map, evolving poses, and image generator have clear responsibilities. Direct map editing and shared state access place both applications within the same architecture. The paper offers a concrete interaction interface, and this system contribution has practical value beyond simply producing longer videos.

Critical weaknesses

The reliability claims exceed the scope of the quantitative evaluation. The level data comprise 100 maps, but map-disjoint training and test sets are not clearly specified, leaving unseen-layout generalization unresolved; multiplayer training and evaluation explicitly use one map. The main quantitative rollouts extend to 256 frames, which does not establish long-term state correctness. Table 2 improves visibility and location, but does not directly measure identity, victim assignment, or cross-view consistency of death/respawn events. The concern is the scope of reliability that the evidence supports, rather than a demonstrated data leak or method failure.

Questions for authors

Q1: Which maps and trajectories in Table 1 and Figure 4 were entirely excluded from training, and what is the map-level split? Please provide the independent sample counts and clarify whether quantitative pose or event-consistency results exist beyond 256 frames. This would determine whether I can interpret the results as evidence of unseen-layout generalization and sustained reliability.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 97ZB (1/2): map split, sample counts, unseen layouts, pose beyond 256 frames

Thank you for your review. We are glad you found the decomposition sensible, with a useful control interface whose practical value goes beyond producing longer videos. The responses below address each of your concerns.

Unseen-layout generalization is unresolved without a map-disjoint split

The 9 test maps of the submission (Table 1, Fig. 4) were unseen during training; the revision now states this explicitly. We also sampled 30 brand-new maps from the same generator, also never used in training, with 60 rollouts per method. The results are comparable to the original test set, and MultiGen scores best in PSNR, LPIPS and wall match over frames 129–256.

To measure whether generated frames follow the map, we trained a small probe network. From a single frame, it predicts the distance to the wall in each image column. It was trained on real game frames from 4 maps separate from all test maps, and sees only the image, never the map or the generator. Wall match is the share of image columns where the probe's distance is within 25% of the true distance ray-cast from the map, at the pose the method itself is rendering. Real frames score 0.98 (App. S8).

30 new maps, frames 129–256PSNR ↑SSIM ↑LPIPS ↓Wall match ↑
GameNGen17.540.4110.555—
ControlNet18.030.4220.5560.414
IP-Adapter17.790.4130.5500.112
MultiGen18.180.4120.5350.951

Q1. Which maps in Table 1 and Fig. 4 were excluded from training, and what are the sample counts?

As above, all 9 test maps of Table 1 and Fig. 4 were excluded from training, which used 100 other maps from the same generator. Each test map has one recorded episode cut into 6 chunks, so the independent sample size is 9 maps; every CI resamples maps.

SplitMapsRecordingsRollouts per method
Training10010M+ frames—
Original test91 episode each20 for Table 1; 54 for the other results
New test302 episodes each60
Multiplayer test1 (the training map)200 two-player episodes200

Q1. Are there quantitative pose results beyond 256 frames?

Yes. MultiGen's pose error over the first 512 frames on the 30 new maps, in map units (Fig. 12 in App. S9 shows their scale on a test map):

Frames32128256512
Pose error20121247492

For gameplay, what matters is not the distance to a recorded trajectory. Over long rollouts MultiGen's absolute position drifts from the recording (App. S9 reports pose error up to 3,567 frames on the original test maps), but during play no recorded trajectory exists. The player sees the generated frames and chooses the next action from them, so the only position that matters is the one the game is rendering from. What a player needs is that:

  • What they see matches where the game thinks they are. At its own pose, MultiGen's wall match stays at 0.96 over all 4,096 frames.
  • The world stays physically consistent. Its path never crosses a wall.
  • Actions keep working. After 32 frames MultiGen is only 20 map units from the recorded position, so it follows actions step by step; long-run drift is small errors accumulating, not a loss of control.

Drift from a recording is therefore a property of evaluation against logged play, not something a player would notice.

AuthorsOpenReview ↗

Response to Reviewer 97ZB (2/2): multiplayer beyond 256 frames, events and identity, one multiplayer map

Is multiplayer state still correct beyond 256 frames?

In MultiGen, the multiplayer state does not degrade over time: every player's pose and alive/dead status live in one shared memory, updated by the same dynamics and rules for everyone, so all players always agree on it by construction. The question that remains is whether each player's rendered frames stay faithful to that shared state over long horizons, from every view.

We test exactly this over 1,768 frames. Using recorded two-player gameplay with both players moving, we write the players' positions into the shared state and generate each player's view from it. A detector trained on simulator boxes then checks whether the opponent is drawn whenever it should be visible (App. S10):

Opponent drawn when visible (balanced accuracy)0–256256–1k1k–1.8k
MultiGen92.4%96.4%98.0%

Real frames: 97.8% over all frames. Rendering does not drift away from the shared state over time.

Table 2 does not measure identity, victim assignment, or cross-view death/respawn consistency

We agree that Table 2 does not. In MultiGen these events are decided once, in the shared memory: a learned kill head decides when a kill happens, a rule picks the victim, and death, respawn and identity are memory entries shared by every player. All views therefore agree on who died and who respawned by construction. What can be wrong is the events themselves.

To test them, four AI agents played 189 new deathmatches in the Doom simulator, with seeds never used in training (about 4 hours, 702 deaths). We ran the kill head on the recorded inputs and the victim rule at the true positions, and compared both with the simulator's log:

Real deaths the kill head catches (recall, ±2 frames)74%
Kills it fires that match a real death (precision, ±2 frames)84%
Victim rule picks the true victim (chance 33%)90.5%

Multiplayer training and evaluation use only one map

Yes, and Sec. 6 now states this as a limitation. The multiplayer mechanism is map-agnostic: every player's view is rendered from ray-casts of whatever map is in the shared memory, and the dynamics read the map only through those ray-casts. But our multiplayer model was trained on one map, and we have not shown that it generalizes to others. The same memory and rendering interface, trained on 100 maps for level design, does generalize to unseen layouts (first part of this reply).

We hope this clarifies what the evidence supports; if it resolves your question, we would be grateful if you would consider updating your assessment.

Thank you very much for detailed response. The addition of results for previously unseen maps, along with a more extensive analysis of pose and wall-matching, strengthens the evidence regarding layout control and provides greater transparency concerning the system's limitations. The clarifications have effectively addressed my initial concerns. Consequently, I am raising my score from 2 to 3.

Reviewer_cVdtOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MultiGen proposes a memory-centric reformulation of diffusion game engines. Instead of treating a window of past frames as the state ("frames-as-state", as in GameNGen), the authors introduce an explicit external memory holding (i) a static, user-authored top-down level layout represented as 2D vertices/edges, and (ii) the evolving set of player poses. The engine is decomposed into three modules: Memory (ray-casts 320 rays against map geometry and converts depth to disparity 80/d), Observation (a fine-tuned SD v1.4 UNet whose input layer is expanded from 4 to 33 channels — 32 latent context channels from 8 frames plus 1 depth channel — with action and context-noise embeddings injected via cross-attention in place of CLIP text), and Dynamics (an 8-layer transformer over frozen-UNet decoder features plus pose/action-history/omnidirectional-depth tokens, predicting an incremental pose update). Two applications are studied in Doom/ViZDoom: (1) level design, where a user authors or edits a coarse minimap and the model renders first-person observations consistent with it, and (2) multiplayer, where each player runs an independent Observation/Dynamics instance while all read/write the same shared memory, so player count is not baked into the model interface and no multi-view supervision is needed. Reported results: SSIM/PSNR/LPIPS gains over GameNGen/ControlNet/IP-Adapter concentrated in the late (128–256) window of 256-step rollouts; 81–91% 2AFC preference over four baselines including Matrix-Game 2.0; opponent-presence accuracy 75.4% (vs. 65.3% for a split-screen baseline) and mean bbox IoU 0.604 (vs. 0.207). The system runs at ~20 FPS per player on one A100 per player.

Critical strengths
  1. S1 — Well-motivated primitive. Framing shared, writable state as the missing primitive behind both controllability and multi-agent consistency is clean and, I think, correct. The modular factorization (persistent low-dim state vs. high-dim rendering) is simple, interpretable, and builds entirely on off-the-shelf SD components, which makes it easy for others to adopt.
  2. S2 — Genuinely scalable multiplayer design. Distributed per-player generation over a shared state is the paper's most interesting idea: cross-viewpoint consistency emerges from single-viewpoint training data, and arbitrary player counts are supported (demonstrated up to 8) without retraining. This is a real departure from joint "split-screen" models that fix N at training time.
  3. S3 — Practical authoring angle. Conditioning on an editable top-down blueprint addresses reproducibility and editability, which most world-model papers ignore entirely. The interface is realistic for a level designer.
Critical weaknesses
  1. W1 — The level-design gains are marginal, and partly negative early. Against GameNGen the improvements are +0.013 SSIM / +0.55 dB PSNR / −0.018 LPIPS overall, and in the early segment the method is worse: PSNR 20.06 vs. 20.23 and LPIPS 0.400 vs. 0.379 for steps 1–128. The entire "long-horizon structure" claim therefore rests on the 128–256 window plus a subjective study. No variance across rollouts, confidence intervals, significance tests, or the number of held-out maps/trajectories is reported. Please add per-rollout standard errors and an objective layout-adherence measure (e.g., agreement between the depth/layout recoverable from the generated frame and the ray-traced conditioning).

  2. W2 — Baseline fairness: Tab. 3 "w/o Memory" is numerically identical to Tab. 1 "GameNGen" in all nine cells. As written this reads as one set of numbers reused for two different things. Please state explicitly whether GameNGen/ControlNet/IP-Adapter were re-implemented on SD v1.4, trained on the same 10M-frame dataset, with the same context length, the same channel-expansion trick, the same context-noise augmentation and a comparable training budget. If any baseline is a weaker reproduction, the comparison is not informative. The same concern applies to Matrix-Game 2.0: 10K iterations at batch 8 is likely far under-trained relative to your setup, so a preference win over it is weak evidence.

  3. W3 — The titular "editability" is never evaluated. The paper shows rollouts conditioned on an authored map, but no experiment edits a map (move/remove a wall) and shows the observations changing correspondingly, and no human study asks whether edits are reflected. Without this, what is demonstrated is level conditioning, which the ControlNet/IP-Adapter baselines also do. An edit-sensitivity study — generate from M and from edited M′, measure the resulting structural change and its correspondence to the edit — is cheap and would materially strengthen the central claim.

  4. W4 — Multiplayer evaluation is narrow and partly proxy-based. (a) Training and evaluation use a single map (Sec. 5.2), so the generalization claim for shared memory is untested. (b) Opponent presence is judged by GPT-4o with ground-truth and generated frames shown side by side; the judge's own agreement with human labels is not reported, and side-by-side presentation invites bias. 75.4% accuracy is modest in absolute terms. (c) The IoU result (0.604 vs. 0.207) is the most convincing number but rests on 250 single-annotator boxes with no inter-annotator agreement and no statement of how frames were sampled. (d) Kill/respawn appear to be driven by a learned kill head with a 0.995 threshold plus scripted respawn — please state clearly which parts of the interaction are generated versus rule-based, otherwise "mutual consistency" overstates what is learned. (e) There is no cross-view event-consistency metric over time and no accuracy-vs-player-count curve.

Questions for authors
  1. Q1: Are the Tab. 1 baselines trained on your dataset with matched backbone/context/budget? Why is "w/o Memory" identical to GameNGen?
  2. Q2: Can you add an edit-sensitivity experiment and an objective layout-adherence metric (W3)?
  3. Q3: What is the pose error at 256 and 1024 steps, and how much of the late-segment gain survives if ground-truth poses are supplied?
3 comments
AuthorsOpenReview ↗

Response to Reviewer cVdt (1/3): level-design gains, variance and pose error (W1, Q3)

Thank you for your detailed review. We are glad you found shared, writable state a well-motivated primitive, the distributed multiplayer design genuinely scalable, and the editable blueprint a practical authoring interface. The responses below address each of your concerns.

W1. The gains are marginal, worse early, and reported without variance or a layout measure

We don't expect a gain early in a rollout. A rollout starts from real context frames, so early on every method can copy the layout from frames it already sees. The map matters once the player moves into areas that were never in context.

As you suggested, we added 95% confidence intervals, resampled by map, and an objective layout-adherence measure (App. S7, S8). For layout adherence, a small probe network predicts the wall distance in each image column of a single frame. It was trained on real frames from 4 maps outside all test sets. Wall match is the share of columns within 25% of the distance ray-cast from the map. Real frames score 0.988.

PSNR (dB), frames 1–128PSNR (dB), frames 129–256Wall match ↑
GameNGen20.23 (18.96, 21.53)17.32 (16.35, 18.28)0.262 (0.228, 0.295)
ControlNet19.58 (18.51, 20.76)17.46 (16.20, 18.83)0.884 (0.850, 0.914)
IP-Adapter20.30 (19.02, 21.57)17.19 (15.97, 18.58)0.411 (0.333, 0.499)
MultiGen20.06 (18.96, 21.19)18.60 (17.40, 19.73)0.980 (0.972, 0.987)

Q3. What is the pose error at 256 and 1,024 steps, and how much gain survives with ground-truth poses?

Pose error. MultiGen's position error on the original test maps, in map units (App. S9, Fig. 12 shows the scale):

Frames2561,024
Pose error293828

Over long rollouts, MultiGen's position drifts from the recorded trajectory. During play, though, there is no recorded trajectory. The player sees the generated frames and picks the next action from them, so what matters is that the frames match the pose the game is rendering from. At its own pose, MultiGen's wall match stays at 0.966 over 3,567 frames, and its path never crosses a wall.

With ground-truth poses, the gain grows. Driven by the ground-truth pose, MultiGen leads GameNGen by +3.60 dB PSNR (2.66, 4.57) on frames 129–256. Pose prediction is therefore the main source of error. The map-based rendering itself does not limit the gain.

AuthorsOpenReview ↗

Response to Reviewer cVdt (2/3): baseline parity and editability (W2/Q1, W3/Q2)

W2/Q1. Are the baselines matched, and why is "w/o Memory" identical to GameNGen?

Yes, the level-design baselines are matched. All four models are our implementations on SD 1.4. They share the same 10M+ training frames from 100 maps, the same 32-frame context window, the same channel expansion and context-noise augmentation, and 300k training steps each. They differ only in how the map enters the model (App. S11):

MethodMap input
GameNGennone
ControlNettop-down render of the local map, centered at the player's position
IP-Adapterthe same image as ControlNet
MultiGenray-cast disparity of the map

"w/o Memory" is GameNGen by construction. Removing the map channel and the dynamics module from MultiGen leaves exactly our GameNGen model. Table 3 therefore reports the same trained network. The row now reads "w/o Memory (= GameNGen)".

W3/Q2. "Editability" is never evaluated: add an edit-sensitivity study

By editability we mean that a designer can change parts of the level layout that are not in the current view, and the model follows the new structure when the player navigates to the edited area.

MultiGen has this property by construction. The level layout is held in an external memory, separate from the model. At every frame, MultiGen ray-casts the current map from the player's pose and conditions the renderer on the resulting disparity. Nothing about unseen areas is stored in past frames or in the model's weights. When the map is edited, the next ray-cast reads the edited map. Once the player turns toward or walks into the edited area, the renderer receives the new geometry and draws it, with no retraining or re-encoding.

A frames-as-state model such as GameNGen has no such memory. Its only knowledge of the level is the recent frames, so there is nothing for a designer to edit.

AuthorsOpenReview ↗

Response to Reviewer cVdt (3/3): multiplayer evaluation (W4)

W4. The multiplayer evaluation is narrow and partly proxy-based

(a) One map. Although we only evaluate multiplayer on one map, the multiplayer mechanism does not depend on the map: every view is rendered from ray-casts of whatever map is in the shared memory, and the same interface generalizes to unseen layouts in level design (W1). Combining the two settings, with multiplayer play on newly authored levels, is a natural next step that we leave to future work.

(b, c) The judge and the boxes. Every method went through the same protocol: the same judge prompt, the same frames and the same annotation procedure. The judge does not know which method produced a frame, so any bias from side-by-side presentation applies to all methods equally and cannot by itself produce the gap between them. The judge is shown the ground-truth frame because whether an opponent should be visible depends on the true game state. Boxes were drawn on player 0's view at fixed frames (40, 50, …, 90) of each episode. We have added these details to the paper.

(d) Learned vs. rule-based. Pose updates and the rendering of all players, including dead bodies, are learned. A learned kill head decides when a kill happens, a rule picks the victim, and death and respawn are updates to the shared memory. The death screen and scoreboard are overlays. Sec. 5 now states this.

(e) Over time and player count. Events are stored once in the shared memory, so all views agree on them by construction. The detector tests whether each view keeps rendering that state:

Opponent drawn when visible (balanced accuracy)
Baselines53.2–57.8%
MultiGen91.9%
MultiGen, frames 0–256 / 256–1k / 1k–1.8k92.4 / 96.4 / 98.0%
MultiGen, 2 / 4 / 6 / 8 players78.1 / 80.6 / 82.7 / 83.5%

Accuracy holds over 1,800 frames and up to 8 players.

Thank you again. These analyses made the paper substantially more rigorous, and we would welcome your updated assessment.

Reviewer_So5bOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MultiGen extends the action-conditioned generation setup of GameNGen with an editable map and explicit player states. A diffusion model generates each player's view, while a learned dynamics model updates the shared state. The Doom experiments study level control and multiplayer consistency.

Critical strengths
  1. An editable map and shared player states give independently generated views a common reference. This connects level design with multiplayer generation without fixing the player count in a single model's output.

  2. The multiplayer gains are meaningful: it improves opponent-box IoU from the best baseline's 0.246 to 0.604. Later-rollout image quality also improves, supporting the value of persistent geometry.

Critical weaknesses
  1. The qualitative examples are encouraging, but quantitative multiplayer evaluation uses one map and focuses on opponent presence and placement. These metrics do not directly test agreement on kills or deaths, and the map-level split for level design is unclear. The evidence for reliable interactions and new layouts therefore remains limited.

  2. The architectural contribution is difficult to separate from the conditioning interface: MultiGen receives projected geometry, while some baselines receive a top-down map (Section 4.3). Its SD 1.4 implementation also uses UNet features for dynamics (Appendix S1.2), with little explanation of the backbone choice. The comparison with Matrix-Game 2.0 [1] is useful, but does not establish whether the proposed design transfers to DiT.

[1] Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, Baixin Xu, Hao-Xiang Guo, Kaixiong Gong, Size Wu, Wei Li, Xuchen Song, Yang Liu, Yangguang Li, and Yahui Zhou. Matrix-game 2.0: An open-source, real-time, and streaming interactive world model. arXiv:2508.13009v5, 2026.

Questions for authors
  1. Could you clarify whether level-design testing uses unseen maps and whether the available multiplayer results verify agreement on kill/death events, beyond opponent visibility?

  2. What geometric and player-state inputs does each baseline receive? A representative comparison using the same projected geometry would help assess the benefit of the proposed architecture.

  3. What motivated SD 1.4, and which parts of the dynamics readout depend on UNet features? Please clarify what would need to change for a DiT implementation.

1 comment

Thank you for the detailed responses. I appreciate the effort the authors have put into the revision. The new results and clarifications have addressed most of my concerns, and I am inclined to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.