- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WorldCast generates multiplayer video using a separate generator for each player, coordinated through explicit player states and shared appearance memory. It combines state estimation from generated video with a distilled video model, evaluated on Counter-Strike 2.
Critical strengths
- The supplementary results look appealing and promising, including examples where one player's movement is reflected in another player's view (eg jumps).
- I like the combination of explicit player states and shared generated appearance memory. Exchanging this information without joint denoising is a useful design, and the reported generation throughput is encouraging.
Critical weaknesses
- The gap from MultiGen (Po et al) is not sufficiently established. MultiGen already uses separate per-player generators, shared explicit state and viewpoint-dependent geometric conditioning. WorldCast adds a different player representation and shared appearance memory, but there is no direct comparison explaining why these choices are preferable to an adapted MultiGen. I am not yet convinced that the contribution beyond this closely related framework is sufficiently motivated. Similarly, one GPU per player is a deployment choice, and the introduction should distinguish this choice from a requirement for multiplayer generation.
- The claims need clearer qualification. FPS excludes synchronization waits, and end-to-end interaction latency and behavior under network delays are not evaluated. The hour-long results use one generated client, recorded states for other players, and round-wise memory/position resets. These limitations should be clearer in the introduction and discussed in the conclusion. I also noticed shooting that was not reflected in the other player's view in multiple examples. Although the paper states hits and deaths are explicitly outside scope, it is unclear why action/weapon consistency is not evaluated alongside player presence. This leaves the demonstrated level of multiplayer consistency a bit uncertain, namely whether it is a design choice or lack of generalization.
Minor: Figure 2 is too packed and hard to follow.
Questions for authors
- What is the main advantage of the proposed state representation and update mechanism over MultiGen, beyond differences in backbone and domain?
- Which supplementary examples use the four-step model, and what evidence isolates its quality relative to the non-distilled model? Could you point to the longest available closed-loop multiplayer example?
- Beyond throughput, what evidence supports consistent generation with more players than seen during training?
2 comments
Response to Reviewer 4LK5 (2/2)
W2b: Behavior under network delays is not evaluated.
States are exchanged once per block, so a network delay matters only when a message misses the next block. We therefore simulated, in closed loop, messages that arrive up to two blocks late or are lost. Rendered decreases by at most 0.07, and PSNR changes by less than 0.08 dB (Table B2). A delay only shifts where the other players appear: with every message one block late, they are rendered where they were one second earlier, a median of 3.9 body widths from their current position, as with lag in online games.
Table B2: Effect of delayed and lost messages in closed loop, relative to all messages on time (Rendered 0.806, PSNR 15.66 dB).
| Messages between clients | Change in Rendered | Change in PSNR (dB) |
|---|---|---|
| Every message one block late | −0.070 | −0.075 |
| Random delay of 0–2 blocks | −0.031 | −0.003 |
| 30% of messages lost | −0.007 | +0.051 |
W2c: The hour-long results use one generated client, recorded states for other players, and round-wise resets.
In the original submission, we joined consecutive recorded rounds, each about two minutes long. At the start of each new round, we reset the player’s position to its recorded spawn point and cleared the scene state. Video generation continued from the last generated frames of the previous round, so the full video formed one continuous rollout.
We have now run one-hour sessions that require no reset. Every client is driven by the recorded key presses of its player, joined across consecutive rounds. All clients generate their views concurrently and estimate their own positions from the generated frames. Their positions and the shared scene state carry over from one round to the next without any resets. VBench imaging quality stays within 70.5–72.3 in every ten-minute interval of the hour, with five and with ten clients (videos at the link above).
On a continuous 15-minute track joined from recorded segments, Rendered is 0.800 with recorded positions and 0.643 with estimated positions. Estimated positions are less accurate, which lowers consistency in long closed-loop runs. The conclusion now states this limitation; the introduction claims only stable image quality over one hour.
W2d: Shooting is not reflected in the other player's view, and action/weapon consistency is not evaluated.
Firing is modeled: the attack key and the weapon of every visible player are part of the state given to the generator. Cross-view agreement in weapons and firing is difficult to evaluate quantitatively. The videos at the link above show the same scene generated with the attack key held and released, and the muzzle flash appears only in the former. The flash of a distant shooter is sometimes missing, as such players are small (a median height of 38 px). This is a current limitation.
For facing direction and actions, we applied classifiers trained only on recorded player crops to the same player as rendered by two clients in closed loop. They show the player facing the same direction (within 45°) in 87.5% of the cases (92.3% on the recording), and a jumping player is shown in the air in 85.0% of the views (88.7% on the recording).
Q2: Which supplementary examples use the four-step model, what evidence isolates its quality relative to the non-distilled model, and the longest closed-loop multiplayer example.
All supplementary videos use the four-step model, except the section "Rendering other players", which compares the non-distilled model with the baselines of Table 1. We evaluated the four-step model and the non-distilled model from which it is distilled (the bidirectional model of Table 1) on the same windows and inputs. The four-step model renders other players as reliably as the non-distilled model with comparable image quality, while its FVD is higher (Table B3; side-by-side videos at the link above). The longest closed-loop multiplayer examples are the one-hour runs of W2c.
Table B3: Four-step model versus the non-distilled model on the 510 views of Table 1.
| Metric | Non-distilled model | Four-step model |
|---|---|---|
| Rendered ↑ | 0.817 | 0.870 |
| Unmatched ↓ | 0.039 | 0.028 |
| VBench imaging quality ↑ | 67.9 | 69.6 |
| PSNR (dB) ↑ | 16.17 | 16.03 |
| FVD ↓ | 39.9 | 47.5 |
Q3: Evidence for consistent generation with more players than seen during training.
Training matches contain ten players. We built a world of twenty players by placing the players of two recorded rounds on the same map, and ran twenty clients in it, one per player, with recorded positions. Rendered is 0.842 with twenty players, against 0.872 with ten. Videos at the link above show this world, and three scenes in which we place twenty players ourselves: walking in files, standing in a ring, and standing in a crowd.
Minor: Figure 2 is too packed.
We agree and have simplified Figure 2.
All results above are in the revised paper.
Response to Reviewer 4LK5 (1/2)
Thank you for your constructive feedback. We respond below and provide videos and qualitative results at: https://anonymous.4open.science/w/ICLR2027-21616/4LK5/index.html
W1a & Q1: Gap from MultiGen, and the main advantage of the proposed state representation and update mechanism.
As you point out, MultiGen already allows players to generate their views independently using shared poses and map geometry. However, its fixed map does not record how a scene appeared in another player’s generated view, and its pose estimates can drift as errors accumulate. WorldCast addresses both by updating the shared state from the video each client generates.
State representation. MultiGen's state comprises a given 2D map and the pose of each player. The map specifies the layout but not the textures, lighting, or objects, and the paper lists as a limitation that scene properties not represented in the map "are not explicitly preserved." WorldCast is given no map at inference and maintains a scene state, a shared memory that holds the video frames generated by all clients with their cameras and predicted depth. Every client retrieves from it the frames that cover its upcoming view. A region generated by one client is therefore rendered consistently by the clients that visit it later. With the scene state switched off, PSNR after 30 seconds drops by 1.23 dB (Table 3).
For the players, MultiGen's pose specifies only where a player stands and faces, whereas WorldCast's player state additionally contains the weapon and the current action. WorldCast takes a different approach to conditioning generation on player states. We project the other players into the observer's camera and write the state of each player at the image location where it appears, which yields a camera-aligned player state field designed for multiple players. The field explicitly aligns each player with its state. All players share it, so the generator's input does not grow with the number of players. Players hidden behind walls or floors are excluded using the depth predicted for the generated frames.
Update mechanism. In MultiGen, the map is fixed and only the poses are updated, by accumulating predicted increments. The paper notes that "small pose errors may accumulate over long rollouts." In WorldCast, each client updates both scene and player states from the video it generated, once per one-second block. For the scene state, it publishes the generated frames as a new memory entry. For the player state, it publishes the position estimated from these frames, together with the other attributes of the player. The position estimate uses predicted motion, as in MultiGen, and adds an estimate of the absolute position, which limits the accumulated error.
As MultiGen has no public code, we re-implemented its update in our framework. The position error at 50–60 s is 11.4 body widths (7.4 when applied on top of the extrapolation from the controls), against 2.6 for WorldCast.
W1b: One GPU per player is a deployment choice rather than a requirement.
We agree that one GPU per player is a deployment choice, and we have clarified this in the introduction. WorldCast makes this choice practical by allowing each client to generate its own view on a separate machine. In joint generation on one GPU, every added player slows down all players: from 2 to 16 players, Gamma-World falls from 13.0 to 2.2 FPS. Spreading it over GPUs is costly, as its views exchange attention features at every denoising step (149–310 Mb/s for Gamma-World). WorldCast clients exchange only a small state (0.55–3.17 Mb/s), and the frame rate stays at 24.1–28.2 FPS with the waits included (Table B1).
W2a: FPS excludes synchronization waits, and end-to-end latency is not evaluated.
In our evaluation, clients advance in lockstep: before generating each block, a client waits for the previous-block states of all other clients. Lockstep is used only in evaluation, so that runs are reproducible and aligned in time with the recording for scoring. A deployed client does not wait: it extrapolates each other player from the latest state it has received, and a late state only shows that player where it was earlier (W2b). The submitted Table 7 excludes these waits; Table B1 includes them.
Table B1: Frames per second per client with the waits included (one H200 GPU per client; two machines for 10 and 16 clients).
The optimized client applies engineering optimizations that leave the generated latents unchanged.
The latency from a key press to its effect on screen is 0.73 s at the median with the small display decoder and 0.95 s with the full decoder. Most of it is the wait for the next block.