- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PAWA, a multiplayer world model that jointly predicts POV and panoramic videos in Minecraft. It introduces a scalable data collection pipeline, a dual-tower architecture with ray-aligned cross-attention, and a quadratic-phase player encoding for handling more players. Experiments show improvements over existing multiplayer world models.
Critical strengths
- The paper introduces an approach to multiplayer world modeling by combining POV and panoramic video prediction. The motivation and technical details are clearly explained, and the paper is well written and easy to follow.
- The paper includes extensive experiments and ablation studies. It also provides useful visualizations and qualitative results to illustrate the improvements.
Critical weaknesses
- The evaluation mainly relies on visual quality metrics such as FVD and LPIPS, while largely overlooking consistency between the generated POV and panoramic videos. For this type of world modeling task, cross-view consistency is more important than the visual quality of each individual stream.
- The paper does not compare ray-aligned cross-attention with simpler alternatives, such as vanilla cross-attention or explicit geometric warping. Since POV and panoramic views share the same camera center, their geometric correspondence can be directly computed. The necessity of the proposed ray-aligned design is therefore not sufficiently demonstrated.
- The experiments use 12s rollouts and at most 16 players. I'm interesting about model's long-term consistency and scalability. In addition, the paper lacks an analysis of computational cost of the two DiT towers.
- The comparison with existing methods is not very fair. PAWA uses additional panoramic inputs and a dual-tower architecture, while the baselines only use POV inputs.
Questions for authors
The experiments mainly focus on co-player tracking and action following, with limited evaluation of changes to the shared environment. I'm curious about another important aspect: whether changes made by one player, such as placing or breaking blocks, are correctly reflected in other players' views. This aspect is not evaluated.
3 comments
(Continued from 1/2.)
W3: Long rollouts, scalability and cost. The paper evaluates 12 s rollouts, 2.6× the causal training window (93 frames) and 5.3× the distillation window (45 frames). In Table 13's last band (frames 189–239), PAWA alone remains closer to the reference than a static first frame, both over the whole POV and within co-player regions.
We extended PAWA to continuous 24 s rollouts on 30 two-player episodes, with no mid-rollout reset. For PAWA (Distilled), POV PSNR decreases from 15.2512 to 13.5818 and LPIPS increases from 0.4717 to 0.5720 between the first and second 12 s windows of the same rollout. PAWA (Distilled) thus doubles the paper's rollout horizon without retraining, with moderate degradation in POV PSNR and LPIPS.
For scalability, PAWA is the only model that supports 16 players. There, PAWA (Causal) reaches a POV FVD of 92.2 (Table 2), and for 73–83% of the players their own actions best explain the camera motion of their generated video, against 6% by chance (Table 8). For cost, PAWA has 4.60B parameters: 2.18B in each tower and 0.24B (5%) in the seven cross-attention sites; Gamma-World has 2.18B. The table reports the average time per generated frame (sampling and decoding) and the peak memory on a single NVIDIA B200 GPU, over a 12 s rollout (6 s for 16 players); one frame comprises the POV and panorama of every player:
| Model | Players | Steps | Time per frame | Peak memory |
|---|---|---|---|---|
| Gamma-World (Causal) | 2 | 35 | 0.37 s | 41 GiB |
| Gamma-World (Distilled) | 2 | 4 | 0.07 s | 31 GiB |
| PAWA (Causal) | 2 | 35 | 0.86 s | 86 GiB |
| PAWA (Distilled) | 2 | 4 | 0.23 s | 58 GiB |
| PAWA (Distilled), cross-attention removed | 2 | 4 | 0.22 s | 57.5 GiB |
| PAWA (Distilled) | 4 | 4 | 0.39 s | 91 GiB |
| PAWA (Distilled) | 8 | 4 | 0.73 s | 157 GiB |
| PAWA (Distilled) | 16 | 4 | 1.28 s | 165 GiB |
The cross-attention adds about 5% time and 1% memory; most of the additional cost over Gamma-World comes from the second tower, which predicts the panoramic stream. From 2 to 16 players, the time per frame grows roughly linearly with the number of players (0.23 s to 1.28 s).
W4: Fairness of the comparison. Predicting the panorama jointly with the POV is the central proposal of the paper rather than an auxiliary input: our claim is that, in multiplayer scenes, the panoramic stream gives each player's prediction access to what happens outside the current view. A comparison that removes all panoramic information would therefore evaluate a different model from the one we propose. We nevertheless ran such controls, to show what each difference contributes. Dual-tower architecture. Without cross-attention (Tables 9 and 11), the POV stream has no access to panoramic information, since pano→POV cross-attention is the only path from the panorama to the POV stream. The POV stream is then computed by a single tower initialized from Gamma-World and receives the same visual input as the baselines. Under the same training data and procedure, the arm of Table 11 that keeps the cross-attention reaches POV FVD 138.4 and LPIPS 0.455, against 191.7 and 0.511 for the arm without it. Additional panoramic input. We re-ran PAWA (Distilled) on the same 30 two-player episodes with the initial panorama replaced by a projection of the initial POV alone (regions not covered by the POV are filled with uniform gray), so that the model receives the same initial observation as the baselines. It still outperforms all three Gamma-World variants on every POV metric: PSNR 15.35, LPIPS 0.482, FID 30.2 and FVD 129.4, against 13.08, 0.567, 52.0 and 267.1 for the best Gamma-World variant on each metric. With the full initial panorama, the same re-run gives FVD 124.2 (138.1 in Table 1, which used a separate evaluation setup), so the advantage over the baselines does not depend on the additional initial panoramic observation.
Q: Block edits made by other players. We ran a preliminary analysis using the block placements logged during data collection. In the interaction tier, 7 placements occur within the forecast window and later become visible to the other player. Scoring the placed cell in that player's view against the recorded video, PAWA (Distilled) is closer to the recorded edit than a static copy of the conditioning frame (in which the block is absent) for 6 of the 7 events, in both the POV and the panorama; its POV LPIPS on the edited cell is 0.662, against 0.691 and 0.722 for Gamma-World (Bidirectional) and (Distilled). Our evaluation tiers were selected for co-visibility and camera motion, not for edits, so this sample is small. The final version will include a dedicated edit tier, covering both placing and breaking, in which one player edits the world outside the other's view; our pipeline supports this directly, since it records edits in the event log and renders any viewpoint offline.
Thank you for the rebuttal. The new results address most of my concerns about cost, long rollouts and fairness.
However, I am still not convinced by the cross-view consistency evaluation: the improvement over the time-shifted control is limited and does not directly show a consistent shared world across players. Also, the advantage of ray-aligned cross-attention over simpler alternatives remains unclear without direct comparisons.
Overall, I appreciate the authors' effort, but my concerns remain. I will raise my score to 3, but my actual assessment is more like 2.5.
We thank all reviewers for their thoughtful reviews. We are glad that the reviewers acknowledge our contributions, including the clear motivation of modeling out-of-view interactions, the usefulness of the dataset and collection pipeline, the well-motivated geometric design, and the extensive experiments. Below we respond to your comments point by point.
W1: Cross-view consistency. We agree that a direct measure of agreement between the two generated streams is valuable, and we report one here. (The paper evaluates each stream against its own ground truth, in the POV and panorama rows of Table 1.) Since a player's POV camera and panorama share the same center, we reproject each generated panorama into the POV frustum using the camera orientation of that frame and compare it with the POV generated by the same model (LPIPS ↓, excluding the HUD rows). Because the two streams differ in resolution and the panorama contains no hand or HUD, the metric does not reach zero even for the recorded videos. We therefore report two reference levels measured on the recorded videos: the attainable level (recorded POV vs. recorded panorama) and the level of unrelated views (each POV frame paired with the panorama of the same video 6 s away). Results on the 30 two-player interaction episodes (frames 1–239):
Both PAWA variants are significantly more consistent than their time-shifted controls, on 30 of 30 (Causal) and 29 of 30 (Distilled) episodes, and PAWA (Causal) remains so in all four time bands of the 12 s rollout (frames 189–239: −0.033 [−0.042, −0.026]). A generated POV thus agrees with its own panorama at the same moment significantly better than with an unrelated view of the same scene: the two towers generate the same world rather than two independently plausible ones. Prior models do not predict a panoramic stream, so the metric is undefined for them. We will add this metric to the paper.
W2: Ray-aligned vs. simpler cross-attention. We agree that Tables 9–11 primarily show the value of exchanging information, and that a vanilla cross-attention trained under the same conditions is the most direct test. Each such control requires retraining the model at full scale, which does not fit within the rebuttal period with the compute currently available to us. We will therefore add this comparison to the final paper: a vanilla cross-attention with image-coordinate 2D RoPE and an explicit-warping baseline, both trained under the same conditions as the full model. We add two points here:
What the ray alignment adds. The correspondence between the two image grids changes with yaw and pitch at every frame. A plain cross-attention would have to learn this mapping from a per-frame global orientation signal, whereas the ray phase supplies it in closed form and without parameters, so the attention starts from the correct geometric prior. The trained attention follows this geometry: at sites 19 and 23, POV queries place 43–57% of their attention inside their own frustum (10% by chance), and the most attended key lies 5°–9° from the query ray (Table 12).
Explicit warping. Warping imposes a fixed one-to-one pixel transfer, but the two streams differ in resolution, projection and content. The panorama covers 360°×180° at 512×256 pixels (about 1.4 px/°), whereas the POV covers 102°×70° at 480×320 (about 4.7 px/°), so a warp can supply at most about a third of the POV's linear resolution and nothing for the HUD and hand. Moreover, both streams are noisy at every denoising step, so copying pixels would carry the panorama's errors directly into the POV. The ray-aligned attention keeps the geometric prior while letting the model learn how much to read from the other stream through zero-initialized gates.