acceptodds
Under review as a conference paper at ICLR 2027

WorldCast: Distributed Multiplayer World Models

Abstract

Multiplayer world models must generate independently controlled views with consistent representations of both players and their shared environment. Most existing approaches coordinate multiple players through joint multi-view generation, whose cost grows with each additional player. We present WorldCast, a distributed multiplayer world model in which each player runs a local client comprising a video generator and a state model. Using recorded player positions and map geometry during training, the state model estimates the player's position from generated video and control inputs. Clients exchange player states and project them into camera-aligned player state fields that guide where and how other players are rendered. Shared scene state enables clients to reuse one another's generated observations to maintain consistent scene appearance across views. Experiments on Counter-Strike 2 demonstrate WorldCast's consistency, real-time performance, and distributed scalability. The camera-aligned player state field improves player rendering rates by over an order of magnitude over joint-generation methods, while shared scene state improves visual consistency over whole rounds. Each client runs in real time and exchanges only player and scene states, enabling scalable multiplayer generation without a centralized computational bottleneck. Image quality remains stable over hour-long rollouts.

Primary area:
generative models
Keywords:
generative models, video world models, diffusion models, multiplayer games, autoregressive video generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4LK5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldCast generates multiplayer video using a separate generator for each player, coordinated through explicit player states and shared appearance memory. It combines state estimation from generated video with a distilled video model, evaluated on Counter-Strike 2.

Critical strengths
  1. The supplementary results look appealing and promising, including examples where one player's movement is reflected in another player's view (eg jumps).
  2. I like the combination of explicit player states and shared generated appearance memory. Exchanging this information without joint denoising is a useful design, and the reported generation throughput is encouraging.
Critical weaknesses
  1. The gap from MultiGen (Po et al) is not sufficiently established. MultiGen already uses separate per-player generators, shared explicit state and viewpoint-dependent geometric conditioning. WorldCast adds a different player representation and shared appearance memory, but there is no direct comparison explaining why these choices are preferable to an adapted MultiGen. I am not yet convinced that the contribution beyond this closely related framework is sufficiently motivated. Similarly, one GPU per player is a deployment choice, and the introduction should distinguish this choice from a requirement for multiplayer generation.
  2. The claims need clearer qualification. FPS excludes synchronization waits, and end-to-end interaction latency and behavior under network delays are not evaluated. The hour-long results use one generated client, recorded states for other players, and round-wise memory/position resets. These limitations should be clearer in the introduction and discussed in the conclusion. I also noticed shooting that was not reflected in the other player's view in multiple examples. Although the paper states hits and deaths are explicitly outside scope, it is unclear why action/weapon consistency is not evaluated alongside player presence. This leaves the demonstrated level of multiplayer consistency a bit uncertain, namely whether it is a design choice or lack of generalization.

Minor: Figure 2 is too packed and hard to follow.

Questions for authors
  1. What is the main advantage of the proposed state representation and update mechanism over MultiGen, beyond differences in backbone and domain?
  2. Which supplementary examples use the four-step model, and what evidence isolates its quality relative to the non-distilled model? Could you point to the longest available closed-loop multiplayer example?
  3. Beyond throughput, what evidence supports consistent generation with more players than seen during training?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4LK5 (1/2)

Thank you for your constructive feedback. We respond below and provide videos and qualitative results at: https://anonymous.4open.science/w/ICLR2027-21616/4LK5/index.html

W1a & Q1: Gap from MultiGen, and the main advantage of the proposed state representation and update mechanism.

As you point out, MultiGen already allows players to generate their views independently using shared poses and map geometry. However, its fixed map does not record how a scene appeared in another player’s generated view, and its pose estimates can drift as errors accumulate. WorldCast addresses both by updating the shared state from the video each client generates.

State representation. MultiGen's state comprises a given 2D map and the pose of each player. The map specifies the layout but not the textures, lighting, or objects, and the paper lists as a limitation that scene properties not represented in the map "are not explicitly preserved." WorldCast is given no map at inference and maintains a scene state, a shared memory that holds the video frames generated by all clients with their cameras and predicted depth. Every client retrieves from it the frames that cover its upcoming view. A region generated by one client is therefore rendered consistently by the clients that visit it later. With the scene state switched off, PSNR after 30 seconds drops by 1.23 dB (Table 3).

For the players, MultiGen's pose specifies only where a player stands and faces, whereas WorldCast's player state additionally contains the weapon and the current action. WorldCast takes a different approach to conditioning generation on player states. We project the other players into the observer's camera and write the state of each player at the image location where it appears, which yields a camera-aligned player state field designed for multiple players. The field explicitly aligns each player with its state. All players share it, so the generator's input does not grow with the number of players. Players hidden behind walls or floors are excluded using the depth predicted for the generated frames.

Update mechanism. In MultiGen, the map is fixed and only the poses are updated, by accumulating predicted increments. The paper notes that "small pose errors may accumulate over long rollouts." In WorldCast, each client updates both scene and player states from the video it generated, once per one-second block. For the scene state, it publishes the generated frames as a new memory entry. For the player state, it publishes the position estimated from these frames, together with the other attributes of the player. The position estimate uses predicted motion, as in MultiGen, and adds an estimate of the absolute position, which limits the accumulated error.

As MultiGen has no public code, we re-implemented its update in our framework. The position error at 50–60 s is 11.4 body widths (7.4 when applied on top of the extrapolation from the controls), against 2.6 for WorldCast.


W1b: One GPU per player is a deployment choice rather than a requirement.

We agree that one GPU per player is a deployment choice, and we have clarified this in the introduction. WorldCast makes this choice practical by allowing each client to generate its own view on a separate machine. In joint generation on one GPU, every added player slows down all players: from 2 to 16 players, Gamma-World falls from 13.0 to 2.2 FPS. Spreading it over GPUs is costly, as its views exchange attention features at every denoising step (149–310 Mb/s for Gamma-World). WorldCast clients exchange only a small state (0.55–3.17 Mb/s), and the frame rate stays at 24.1–28.2 FPS with the waits included (Table B1).


W2a: FPS excludes synchronization waits, and end-to-end latency is not evaluated.

In our evaluation, clients advance in lockstep: before generating each block, a client waits for the previous-block states of all other clients. Lockstep is used only in evaluation, so that runs are reproducible and aligned in time with the recording for scoring. A deployed client does not wait: it extrapolates each other player from the latest state it has received, and a late state only shows that player where it was earlier (W2b). The submitted Table 7 excludes these waits; Table B1 includes them.

Table B1: Frames per second per client with the waits included (one H200 GPU per client; two machines for 10 and 16 clients).

Clients2481016
Code of Table 722.822.521.116.415.1
Optimized client28.227.524.524.124.2
Optimized client with a small display decoder (TAEHV)38.937.632.832.230.7

The optimized client applies engineering optimizations that leave the generated latents unchanged.

The latency from a key press to its effect on screen is 0.73 s at the median with the small display decoder and 0.95 s with the full decoder. Most of it is the wait for the next block.

AuthorsOpenReview ↗

Response to Reviewer 4LK5 (2/2)

W2b: Behavior under network delays is not evaluated.

States are exchanged once per block, so a network delay matters only when a message misses the next block. We therefore simulated, in closed loop, messages that arrive up to two blocks late or are lost. Rendered decreases by at most 0.07, and PSNR changes by less than 0.08 dB (Table B2). A delay only shifts where the other players appear: with every message one block late, they are rendered where they were one second earlier, a median of 3.9 body widths from their current position, as with lag in online games.

Table B2: Effect of delayed and lost messages in closed loop, relative to all messages on time (Rendered 0.806, PSNR 15.66 dB).

Messages between clientsChange in RenderedChange in PSNR (dB)
Every message one block late−0.070−0.075
Random delay of 0–2 blocks−0.031−0.003
30% of messages lost−0.007+0.051

W2c: The hour-long results use one generated client, recorded states for other players, and round-wise resets.

In the original submission, we joined consecutive recorded rounds, each about two minutes long. At the start of each new round, we reset the player’s position to its recorded spawn point and cleared the scene state. Video generation continued from the last generated frames of the previous round, so the full video formed one continuous rollout.

We have now run one-hour sessions that require no reset. Every client is driven by the recorded key presses of its player, joined across consecutive rounds. All clients generate their views concurrently and estimate their own positions from the generated frames. Their positions and the shared scene state carry over from one round to the next without any resets. VBench imaging quality stays within 70.5–72.3 in every ten-minute interval of the hour, with five and with ten clients (videos at the link above).

On a continuous 15-minute track joined from recorded segments, Rendered is 0.800 with recorded positions and 0.643 with estimated positions. Estimated positions are less accurate, which lowers consistency in long closed-loop runs. The conclusion now states this limitation; the introduction claims only stable image quality over one hour.


W2d: Shooting is not reflected in the other player's view, and action/weapon consistency is not evaluated.

Firing is modeled: the attack key and the weapon of every visible player are part of the state given to the generator. Cross-view agreement in weapons and firing is difficult to evaluate quantitatively. The videos at the link above show the same scene generated with the attack key held and released, and the muzzle flash appears only in the former. The flash of a distant shooter is sometimes missing, as such players are small (a median height of 38 px). This is a current limitation.

For facing direction and actions, we applied classifiers trained only on recorded player crops to the same player as rendered by two clients in closed loop. They show the player facing the same direction (within 45°) in 87.5% of the cases (92.3% on the recording), and a jumping player is shown in the air in 85.0% of the views (88.7% on the recording).


Q2: Which supplementary examples use the four-step model, what evidence isolates its quality relative to the non-distilled model, and the longest closed-loop multiplayer example.

All supplementary videos use the four-step model, except the section "Rendering other players", which compares the non-distilled model with the baselines of Table 1. We evaluated the four-step model and the non-distilled model from which it is distilled (the bidirectional model of Table 1) on the same windows and inputs. The four-step model renders other players as reliably as the non-distilled model with comparable image quality, while its FVD is higher (Table B3; side-by-side videos at the link above). The longest closed-loop multiplayer examples are the one-hour runs of W2c.

Table B3: Four-step model versus the non-distilled model on the 510 views of Table 1.

MetricNon-distilled modelFour-step model
Rendered ↑0.8170.870
Unmatched ↓0.0390.028
VBench imaging quality ↑67.969.6
PSNR (dB) ↑16.1716.03
FVD ↓39.947.5

Q3: Evidence for consistent generation with more players than seen during training.

Training matches contain ten players. We built a world of twenty players by placing the players of two recorded rounds on the same map, and ran twenty clients in it, one per player, with recorded positions. Rendered is 0.842 with twenty players, against 0.872 with ten. Videos at the link above show this world, and three scenes in which we place twenty players ourselves: walking in files, standing in a ring, and standing in a crowd.


Minor: Figure 2 is too packed.

We agree and have simplified Figure 2.

All results above are in the revised paper.

Reviewer_cvu7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldCast is a distributed multiplayer video world model in which each client generates its own view while exchanging player states and scene observations. Shared player states are projected into a camera-aligned field to guide the rendering of other players, while a shared memory bank provides previously generated scene content. A state model estimates position and depth from generated video and controls to update these shared states. Experiments on Counter-Strike 2 evaluate player placement, scene consistency, closed-loop generation, and distributed performance.

Critical strengths
  1. The separation of shared state from local video generation is a practical approach to multiplayer scalability. The system combines spatial player conditioning, shared scene observations, and video-based state estimation into a functioning distributed generation loop.

  2. The evaluation usefully separates several sources of error. Matched-backbone comparisons and field ablations support the importance of spatially aligned conditioning, while the scene-memory experiments examine the benefit of sharing observations across clients. Separate GT-state and predicted-state evaluations also help distinguish rendering ability from state-estimation accuracy.

Critical weaknesses
  1. The distinction from prior shared-state multiplayer methods is not sufficiently clear. The controlled baselines focus on joint generation, so they do not directly establish what WorldCast adds over approaches that already separate shared state from view generation. This makes the significance of its distributed state-update and conditioning design difficult to assess.

  2. The evaluation leaves aspects of the claimed consistency and real-time performance untested. The main player-rendering metric checks placement but not cross-view agreement in identity, appearance, or actions. Clients also advance in block-level lockstep, while reported frame rates exclude synchronization waits. The results therefore provide limited evidence of behavioral consistency within each block and end-to-end responsiveness when clients experience different delays.

Questions for authors
  1. Relative to shared-state approaches such as MultiGen and MASS, which aspect of WorldCast constitutes the main methodological advance? Could the authors clarify the differences in state updates, visual conditioning, and distributed execution?

  2. Beyond player placement, what evidence is available for cross-view agreement in player identity, appearance, and actions? How are discrepancies handled when a client’s generated motion differs from the motion extrapolated by other clients within the same block?

  3. What is the end-to-end block latency when synchronization waits are included? How might delayed state updates affect generation quality and responsiveness?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cvu7 (1/2)

Thank you for your constructive feedback. We respond below and provide videos and qualitative results at: https://anonymous.4open.science/w/ICLR2027-21616/cvu7/index.html

W1 & Q1: Main methodological advance over shared-state methods (MultiGen, MASS), and the differences in state updates, visual conditioning, and distributed execution.

Thank you for pointing out where our discussion of prior work could be clearer. MultiGen and MASS also share a state across players. MultiGen shares a given map and the player poses, while MASS maintains a typed state defined by a game-specific schema. Both assume that what must stay consistent across views can be specified in advance, and derive every view from this specification. However, in a game as complex as CS2, no such specification is available: a map gives the geometry of the scene but not its appearance, and the content and rules of the game cannot be enumerated in a schema.

WorldCast instead keeps an explicit state only for the players, and the rest of the world exists only in the generated video. Its main advance is to share this generated content as a scene state, and to update both the scene state and the player state from the generated video, closing the loop between state and video. MultiGen and MASS are also evaluated in simpler domains, namely two players on one ViZDoom map and grid-based arcade games.

State updates. In MultiGen, the map is fixed and only the poses are updated, by accumulating predicted increments. The paper notes that "small pose errors may accumulate over long rollouts." MASS advances its state on a server from the actions, without reading the generated frames. In WorldCast, each client updates both scene and player states from the video it generated, once per one-second block. For the scene state, it publishes the generated frames with their cameras and predicted depth as a new memory entry, which extends the shared scene wherever the players go. For the player state, it publishes the position estimated from these frames, together with the other attributes of the player. The position estimate uses predicted motion, as in MultiGen, and adds an estimate of the absolute position, which limits the accumulated error.

As neither method has public code, we re-implemented both position updates in our framework. At 50–60 s, the position error is 11.4 body widths for the MultiGen-style update (7.4 when applied on top of the extrapolation from the controls) and 12.6 for the MASS-style update, against 2.6 for WorldCast.

Visual conditioning for multiplayer generation. For the scene, MultiGen conditions each generator on depth ray-cast from the given map and lists as a limitation that scene properties not represented in the map "are not explicitly preserved." MASS renders each view from the schema state alone. In WorldCast, each client retrieves from the scene state the frames of any client that cover its upcoming view. A region generated by one client is therefore rendered consistently by the clients that visit it later (videos at the link above). With the scene state switched off, PSNR after 30 seconds drops by 1.23 dB (Table 3).

For the players, MultiGen shares only a pose. WorldCast's player state additionally contains the weapon and the current action. WorldCast also takes a different approach to conditioning generation on player states. We project every other player into the observer's camera and encode its state at the image location where it appears, which yields a camera-aligned player state field designed for multiple players. (i) This explicitly aligns each player with its state. (ii) All players are represented in this one fixed-size field, so the input of the generator does not grow with the number of players. (iii) Players hidden behind walls or floors are excluded using the depth that the state model predicts for the generated frames.

Distributed execution. MultiGen updates the shared poses once per generated frame, and MASS advances the state on a central server at every tick. In WorldCast, each client is the only writer of its own state, publishes it once per block, and extrapolates the other players from their latest records, so it never needs their current state and can generate asynchronously. Lockstep, in which every client waits for the states of all others before each block, is used in our evaluation only to make runs reproducible and aligned in time with the recording for scoring. Without lockstep, a slow or disconnected client does not affect the others, whereas with lockstep a single missing message halts all clients (Table A1).

Table A1: Frames per second of the other seven of eight clients when one client is disturbed (small display decoder).

One clientWith lockstep (evaluation)Without lockstep (deployment)
No disturbance33.136.9
Slowed to 25% speed9.537.4
Disconnected036.1
30% of its messages lost035.6
AuthorsOpenReview ↗

Response to Reviewer cvu7 (2/2)

W2 & Q2: Cross-view agreement in identity, appearance, and actions, and discrepancies between a client's generated motion and the motion extrapolated by other clients.

Cross-view agreement. Following your suggestion, we evaluated whether different clients depict the same player with a consistent facing direction and action. We evaluated whole rounds in closed loop, with each client estimating its own position from its generated video. We used the shared player records to identify the same player in two views, then applied classifiers trained only on player crops from recorded gameplay. The two clients show the player facing the same direction (within 45°) in 87.5% of the cases (92.3% on the recording), and a jumping player is shown in the air in 85.0% of the views (88.7% on the recording). Videos at the link above show the jumps and crouches of a player appearing at the same moments in the views of different clients. Cross-view agreement in weapons and firing is difficult to evaluate quantitatively and is also shown in the videos. The tables there also report player identity and crouching across views, and the weapon and firing in each client's own view.

Motion discrepancies within a block. The other clients extrapolate player j during a one-second block from j's last published position and controls, since the frames that j's client is generating are not yet available to them. If the motion generated by j's client deviates from this extrapolation (e.g., j is stopped by a wall), j's client detects the deviation, as it estimates j's position from its own generated frames, and publishes the corrected position, which all clients adopt from the next block. A discrepancy therefore lasts for at most one block. The median difference between the extrapolated position and the position estimated from the player’s own video grows from 0.35 to 0.90 body widths over the block. These values are measured on our evaluation data, professional matches in which the controls are complex and change very frequently. With controls specified by hand, as in the closed-loop videos at the link above and the section "Scripted play" of the supplementary videos, the corrections cause hardly any visible jump.


W2 & Q3: End-to-end latency including synchronization waits, and the effect of delayed state updates.

Block latency with synchronization. In the submitted evaluation, clients ran in lockstep: before generating the next block, each client waited for the previous-block updates from all other clients. Table 7 excluded this waiting time. Table A2 reports both measurements for the same code, followed by the results for our optimized implementation.

Table A2: Frames per second per client (one H200 GPU per client; 10 and 16 clients run on two machines).

Clients2481016
Table 7 as submitted (waits excluded)23.423.322.521.718.7
Same code, waits included22.822.521.116.415.1
Optimized client, waits included28.227.524.524.124.2
Optimized client with a small display decoder (TAEHV), waits included38.937.632.832.230.7

A block contains one second of video (16 frames). With the waits included, the optimized client therefore generates a block in 0.57–0.66 s, which is faster than real time. The optimized client applies engineering optimizations to the submitted code without changing the generated latents. For example, clients exchange messages over direct network connections instead of a shared file system, which is slow across two machines.

Effect of delayed state updates. A client reads its own controls locally, so a late message never slows its response to its own input. For the other players, states are exchanged once per block, so a network delay matters only when a message misses the next block. We therefore simulated whole-block delays and message loss in closed loop, where a delayed or lost message is replaced by the most recent one received. As shown in Table A3, Rendered decreases by at most 0.07 and image quality is unaffected. A late message makes a client render the other players where they were earlier: when every message is one block late, the median distance to their current position is 3.9 body widths. Videos at the link above show one client's view of another player under these conditions.

Table A3: Effect of delayed and lost messages between clients in closed loop, relative to all messages on time (Rendered 0.806, PSNR 15.66 dB).

Messages between clientsChange in RenderedChange in PSNR (dB)
Every message one block late−0.070−0.075
Random delay of 0–2 blocks−0.031−0.003
10% of messages lost0.000+0.046
30% of messages lost−0.007+0.051
One client always one block late0.000+0.006

All results above have been added to the revised paper.

Reviewer_LviJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldCast proposes a distributed multiplayer world model in which each player runs an independent video generator and state estimator. Clients exchange player states and generated scene observations rather than intermediate network features. A camera-aligned player state field controls the placement and attributes of other players, while a shared scene memory supports appearance consistency across views. The state model estimates position and depth from generated video and controls, replacing engine-provided geometry at inference. Experiments on four Counter-Strike 2 maps include controlled baseline comparisons, component ablations, closed-loop rollouts, and deployment measurements up to 16 clients. The results demonstrate substantial improvements in player rendering and benefits from shared scene memory. However, I lean toward weak rejection because the evidence for sustained multiplayer consistency and practical real-time distributed operation remains insufficient.

Critical strengths

A coherent architecture for decoupling multiplayer generation. Separating local video generation from shared player and scene states addresses an important limitation of joint multi-view generation. The camera-aligned field provides explicit spatial control without increasing the generator’s attention sequence with player count. Its effectiveness is supported by the controlled comparisons and alignment ablation: the rendering rate falls from 0.763 to 0.093 without camera alignment. This is a meaningful technical contribution with potential beyond the demonstrated game.

Substantial experimental effort with useful component-level evidence. The paper retrains comparison models using the same backbone and data, evaluates both ground-truth and predicted states, and studies scene memory over whole rounds. The appendices also provide metric sensitivity analyses and detailed deployment conditions. These experiments establish that the proposed conditioning and memory mechanisms offer real benefits, even though they do not fully validate the broader system claims.

Critical weaknesses

Sustained closed-loop multiplayer consistency is not convincingly established. The strongest rendering comparisons and most whole-round evaluations use ground-truth player states. Predicted-state rendering is evaluated quantitatively over ten-second windows, while the closed-loop whole-round results show considerable remaining disagreement: after 30 seconds, rotation error is 55.6° and depth agreement is 0.120 despite scene memory. Furthermore, the rendering criterion checks proximity to the supplied state but does not verify identity, facing direction, weapon, or action consistency. The hour-long experiments provide limited additional evidence: they run one generated client with other players following recorded states, clear scene memory, and reset position at every round. These results support visual continuity under that protocol, but do not establish hour-long distributed multiplayer consistency. This gap matters because maintaining a coherent shared world is the central contribution.

The deployment evaluation does not establish end-to-end real-time interactivity under distributed conditions. Clients advance in lockstep, but reported FPS excludes synchronization waits. The experiments do not vary network delay, jitter, or loss, and the effect of stale states on rendering is unmeasured. With one-second generation blocks, throughput alone also leaves action-to-display latency unclear. In addition, the scalability figure compares WorldCast using one GPU per player against released joint-generation systems running all players on one GPU; the communication figures for distributed baselines are analytical rather than measured deployments. This demonstrates the appeal of the architecture, but leaves uncertainty about its practical latency and resource-normalized advantage. Since real-time distributed scalability is a headline claim, this uncertainty materially lowers my assessment.

Questions for authors

What are the achieved synchronized FPS and median/tail action-to-display latency, including communication and synchronization, for the 2–16-client deployments? How are controls arriving during a one-second generation block incorporated?

Can you report rendering, unmatched detections, and cross-view player identity/action agreement over whole rounds with predicted states, broken down by elapsed time? The existing scene-agreement measurements do not establish whether clients continue to depict the same players performing compatible actions.

Do you have multi-client rollouts without periodic ground-truth position resets or memory clearing? If so, how do spatial and cross-view consistency evolve? This would directly affect my assessment of whether the method maintains a persistent shared world.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LviJ (1/2)

Thank you for your thorough and constructive feedback. The submission uses recorded positions in most evaluations by design: the generated video can then be compared directly against the recording, and the comparison between methods is controlled. As Reviewer cvu7 notes, this helps "distinguish rendering ability from state-estimation accuracy." Following your comments, the consistency results below are in closed loop, with every client estimating its own position with the state model, and Rendered remains at 0.64–0.80 over whole rounds. We respond below and provide videos and qualitative results at: https://anonymous.4open.science/w/ICLR2027-21616/LviJ/index.html

W1a: Closed-loop scene consistency after 30 seconds (RotErr 55.6°, depth agreement 0.120).

What the measures evaluate. The measures of Table 4a evaluate the consistency of the scene across clients. When two players look at the same part of the map at the same instant, the surfaces that both see should be generated with the same geometry and appearance. Mat. Pix. denotes the number of pixels matched between the two frames, Inliers the number of matched points that fit a single relative camera pose, RotErr the error of the camera rotation recovered from the matches, and Depth the fraction of pixels whose depths agree.

Why the absolute values are low. First, the pairs of Table 4a are candidates rather than pairs known to look at the same place. They are selected offline from the recorded positions and camera directions: two clients are kept if their recorded views share any visible surface, however small or distant. Even in the recording, 71% of the candidates share less than half of their views after 30 seconds, and the recorded video itself reaches only a RotErr of 23.6° and a Depth of 0.350 on them.

Second, in closed loop the positions are estimated from the generated video, so the shared state follows what the clients generate rather than the recording. As autoregressive generation and the position estimate both accumulate error, a closed-loop round gradually departs from the recording. The clients still share one estimated state, but the recording no longer indicates which of them look at the same place: after 30 seconds, 83% of the candidates overlap by less than 10% in closed loop, against 31% in the recording.

Nevertheless, Table 4a keeps the candidates selected from the recording, so that scene state on and off are evaluated on identical pairs. It should be read in relative terms: scene state improves all four measures, with 42% more matched pixels, and the gain is largest for the candidates whose recorded views overlap most (+22.1K matched pixels, against +9.6K for those that overlap least).

Consistency of clients that look at the same place. To evaluate the consistency itself, we chose the pairs from the state shared by the clients in the closed-loop run: two clients within eight body widths, each seeing at least half of the surfaces that the other sees. Table C1 reports their agreement after 30 seconds, with the recorded frames of the same players and unrelated frames as references. The generated frames match three quarters as many pixels as the recorded frames, and their RotErr is 25.3° against 16.3°, far from the 59.8° of unrelated frames.

Table C1: Agreement between two closed-loop clients that look at the same place (after 30 seconds; 144 pairs).

Frames comparedMat. Pix. (K) ↑Inliers ↑RotErr (°) ↓Depth ↑
Generated frames of the two clients252.2150.125.30.247
Recorded frames of the same players331.0316.316.30.397
Unrelated frames (recorded, 10 s apart)91.280.859.80.124

W1b & Q2: Rendered, Unmatched, and cross-view identity and action agreement over whole rounds, by elapsed time.

Following your suggestion, Table C2 reports Rendered and Unmatched in closed loop over whole rounds by elapsed time (five clients, no resets).

Table C2: Rendered and Unmatched in closed loop, by elapsed time.

Seconds into the round0–1010–2020–3030–4040–5050–60
Rendered ↑0.7500.7570.7500.6390.8000.757
Unmatched ↓0.0510.1310.1800.2470.2500.163

Agreement beyond position. Classifiers trained only on recorded player crops read the facing direction and the actions of the same player as rendered by two clients. Over whole rounds in closed loop, the two clients show the player facing the same direction (within 45°) in 87.5% of the cases (92.3% on the recording), and a jumping player is shown in the air in 85.0% of the views (88.7% on the recording). The appendix reports both by elapsed time. Videos at the link above show the jumps, crouches, and firing of a player in the views of different clients. The tables there also report the identity of a player, crouching, and the weapon and firing in each client's own view.

AuthorsOpenReview ↗

Response to Reviewer LviJ (2/2)

W1c & Q3: Multi-client rollouts without position resets or memory clearing.

In the original submission, we joined consecutive recorded rounds, each about two minutes long. At the start of each new round, we reset the player’s position to its recorded spawn point and cleared the scene memory. Video generation continued from the last generated frames of the previous round, so the full video formed one continuous rollout.

We have now run continuous sessions that require no reset. For one hour, every client is driven by the recorded key presses of its player, joined across consecutive rounds. All clients generate their views concurrently and estimate their own positions from the generated frames. Their positions and the shared scene memory carry over from one round to the next without any resets. VBench imaging quality stays within 70.5–72.3 in every ten-minute interval of the hour, with five and with ten clients (videos at the link above).

To follow the consistency over time, we built a continuous 15-minute track by joining recorded segments where the position, view direction, and velocity match, and ran it with recorded and with estimated positions. Over its three five-minute intervals, Rendered is 0.827, 0.795, and 0.789 with recorded positions, and 0.669, 0.533, and 0.655 with estimated positions. With estimated positions, Rendered is lower from the first interval on, as they are less accurate than recorded positions, but it does not decrease further. This result shows that the accumulated error of the player state remains limited. The revised conclusion states this loss of accuracy.


W2a & Q1: Synchronized FPS, action-to-display latency, and controls arriving during a block.

In our evaluation, clients advance in lockstep: before generating each one-second block, a client waits for the previous-block states of all other clients. Lockstep is needed only for evaluation, to make runs reproducible and aligned in time with the recording for scoring. In deployment, a client uses the latest states it has received. Table 7, as submitted, excludes these waits; Table C3 includes them.

Table C3: Frames per second per client with the waits included (one H200 GPU per client; 10 and 16 clients on two machines).

Clients2481016
Code of Table 722.822.521.116.415.1
Optimized client28.227.524.524.124.2
Optimized client with a small display decoder (TAEHV)38.937.632.832.230.7

The optimized client applies engineering optimizations that leave the generated latents unchanged.

Controls are read once per block, before it is generated. A control arriving during a block therefore takes effect in the next block: it waits 0.50 s at the median, and with the small display decoder the first frame of that block appears within 0.2 s. The resulting action-to-display latency, including communication and synchronization, is 0.73 s at the median and 1.18 s at the 95th percentile with the small display decoder, and 0.95 s and 1.41 s with the full decoder, for 2 to 16 clients.


W2b: Network delay, jitter, loss, and stale states.

The frame rates and latencies above include the real network exchange between two machines. As states are exchanged once per block, a network delay matters only when a message misses the next block. We therefore simulated whole-block delays and message loss in closed loop, where a late or lost message is replaced by the most recent one received. Rendered decreases by 0.07 when every message is one block late, and by at most 0.03 under random delays of 0–2 blocks, 30% message loss, or one client that is always one block late. PSNR changes by less than 0.08 dB. A late message makes a client render the other players where they were earlier: the median distance to their current position is 3.9 body widths when every message is one block late, and 1.7 under random delays (videos at the link above).


W2c: Resource-normalized advantage, and analytical communication figures for the baselines.

We agree that the scalability figure gives WorldCast one GPU per player, whereas the joint-generation models run all players on one GPU, as released. Given one GPU per player as well, their views must exchange attention features between GPUs at every denoising step. The submission computed the size of this exchange from the architectures. We have now measured the transfer of data of that size between two machines. For one second of video and 2 to 16 players, the exchange takes 0.40–0.49 ms for WorldCast (0.55–3.17 Mb/s per client), 46–111 ms for Gamma-World (149–310 Mb/s), and 1.37–11.0 s for Solaris (26–389 Gb/s), which alone exceeds real time. For Gamma-World, the exchange is a small overhead within a data center, but over a 100 Mb/s link it would take 1.5–3.1 s per second of video.

All results above are in the revised paper.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF, and summarize the changes below. All new results are placed in a new appendix, "Additional results" (Appendix H), with still frames of the accompanying videos in Appendix J, so that the numbering of the existing tables and figures is unchanged. No number in the submission has been changed. Changes are marked in blue.

1. Relation to MultiGen and MASS (Reviewers cvu7 and 4LK5). Related Work now states the differences explicitly. MASS advances a schema-defined state on a server, and MultiGen accumulates predicted pose increments on a given map. Neither shares the generated appearance. WorldCast makes the generated content part of the shared state and updates the state from the generated video, with no map or schema given at inference. The appendix adds a comparison in which the MultiGen-style and MASS-style position updates are re-implemented in our framework: at 50–60 s, the position error is 2.6 body widths for WorldCast, against 11.4 and 12.6 for these two updates.

2. Frame rate, latency, and communication (all reviewers). Table 7 keeps the submitted rows and adds two rows: the frame rate of the optimized client with the lockstep waits included, and the traffic of WorldCast measured on the wire. The appendix reports the frame rate with and without these waits, for the submitted code and for an optimized client; the latency from a key press to its effect on screen; and the measured time needed to exchange, between two machines, the data that joint generation requires. With the waits included, the optimized client runs at 24.1–28.2 FPS for 2 to 16 clients. The introduction now states that each client is the sole writer of its own state, which permits asynchronous generation, and gives the frame rate as obtained with one GPU per client. Figure 2 is simplified, and its caption states that the frame rates shown exclude lockstep waits.

3. Network conditions (all reviewers). New tables report closed-loop results under simulated network delay and message loss, and the effect of a slow, paused, or disconnected client with and without lockstep. Rendered decreases by at most 0.07 under delay and loss, and without lockstep a slow or disconnected client does not affect the others.

4. Long closed-loop runs (Reviewers 4LK5 and LviJ). New tables report one-hour runs in closed loop without any reset, with five and ten clients, and a continuous 15-minute track with recorded and with estimated positions. Image quality stays stable over the hour (VBench 70.5–72.3), and Rendered on the 15-minute track is 0.800 with recorded and 0.643 with estimated positions. The conclusion and the limitations now state that the error of the estimated positions lowers consistency in long closed-loop runs.

5. Consistency beyond position (all reviewers). New tables report the cross-view agreement in identity, facing direction, and actions (facing within 45° in 87.5% of the cases, against 92.3% on the recording); the weapon and firing in each client's own view; the agreement between closed-loop clients that look at the same place; Rendered and Unmatched in closed loop by elapsed time; the four-step model against the non-distilled model; and worlds of twenty players.

Videos and qualitative results are provided on the anonymous pages linked in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.