acceptodds
Under review as a conference paper at ICLR 2027

Multi-Agent World Models Beyond What Agents See

Abstract

Despite the success of video world models, extending them to multiple agents is difficult because it requires generating several agent-specific video streams that are consistent in time and between agents. To address this gap, we introduce OverSeer, a multi-agent world model that augments egocentric agent views with auxiliary camera views. We suggest that, by fixing the auxiliary views in space, they act as a proxy for a world-centric representation: Unlike the agents' views, which can change entirely due to a simple change in agent position, changes in the auxiliary views directly map to localized changes in the world state. We show that these fixed views substantially help the model to maintain consistency. We also show that, due to their redundant and semi-static nature, the auxiliary views can be efficiently compressed with a novel delta-VAE that encodes only the changes relative to a static reference image, reducing the number of tokens required to process them.

Primary area:
generative models
Keywords:
Video World Model, Multi-player

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_dZSNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript presents OverSeer, a multi-agent video world model designed to improve temporal and cross-agent consistency. The central idea is to introduce fixed-pose auxiliary camera views that are jointly generated with the agents' egocentric observations, providing a persistent visual reference for the shared environment. To reduce the additional computational cost, the authors propose δ-VAE, which compresses auxiliary video streams by encoding changes relative to their initial reference frames.

The framework is evaluated on Minecraft and Doom, with additional qualitative demonstrations on real-world multi-camera captures. The results show substantial improvements in spatial memory and multi-agent consistency over existing approaches.

Critical strengths

The manuscript is well motivated and addresses an important challenge in multi-agent world modeling. Using fixed auxiliary views as a proxy for a shared world-centric representation is intuitive and offers an interesting alternative to approaches that rely exclusively on egocentric observations or explicit geometric reconstruction.

The proposed δ-VAE is an interesting contribution. By exploiting the temporal redundancy of stationary viewpoints, the framework reduces the token overhead associated with generating multiple auxiliary streams. The cross-scene residual transfer experiments also provide useful evidence that the compressed representations can separate dynamic changes from relatively static scene content.

OverSeer achieves substantial improvements on the newly introduced MAC-Bench, particularly for agent reunion and dynamic event consistency. The ablations on auxiliary camera count and latent compression further support the main design choices. The qualitative demonstrations across Minecraft, Doom, and real-world captures are also informative.

Critical weaknesses

My main concern is that the experiments do not fully isolate the contribution of fixed viewpoints themselves. The ablation without auxiliary views demonstrates the benefit of introducing additional visual streams, but it does not establish whether fixed viewpoints are substantially better than alternative multi-view configurations. For example, comparing against moving auxiliary cameras or additional egocentric views under a comparable token budget would help determine whether the improvements specifically originate from persistent spatial references rather than additional visual supervision and modeling capacity.

Another concern is the evaluation of world and agent consistency. The proposed MAC-Bench provides meaningful tests of spatial memory, but its quantitative evaluation relies primarily on VLM-based judgments of relatively simple questions, such as whether another agent is visible or whether an environmental event has progressed. These metrics do not fully characterize geometric, temporal, or causal consistency across generated views. More direct measurements, such as cross-view reprojection consistency, object trajectory agreement, or action-conditioned state transition accuracy, would strengthen the central claims.

The generation of auxiliary views at inference time deserves further investigation. Since the model receives only initial agent observations and sampled camera poses, the content of previously unseen regions must be inferred rather than directly observed. Consequently, generated auxiliary views may remain temporally consistent without necessarily representing the correct underlying environment. Moreover, the camera sampling strategy does not explicitly consider scene occupancy or occlusion. Evaluating sensitivity to camera placement and limited spatial coverage would provide a clearer understanding of the approach's practical boundaries.

The efficiency–quality trade-off of δ-VAE is also significant in more dynamic environments. In Doom, removing δ-VAE improves Reunite at Start accuracy from 62.0% to 78.0% and Reunite Far accuracy from 67.0% to 84.0%. Although the computational savings are meaningful, these differences suggest that compression may discard important dynamic information. A more detailed discussion of adaptive compression or reference-frame updating would strengthen this contribution.

The manuscript could also benefit from a broader discussion of agentic world modeling beyond multi-view video generation. World models can serve not only as visual predictors but also as interactive simulators that support action-conditioned forecasting, counterfactual reasoning, long-horizon memory, multi-agent coordination, and decision-making. While OverSeer improves visual consistency across agents and time, it remains less clear how these improvements translate into more general agentic capabilities, such as anticipating the consequences of alternative actions, supporting cooperative planning, or providing reliable feedback for downstream decisions. Discussing how the proposed framework relates to these broader directions, as well as its potential limitations, would help better position its contribution within the evolving world modeling literature.

Questions for authors

Q1. Could the authors clarify whether the improvements primarily originate from the fixed camera poses or from introducing additional visual streams? A comparison with moving auxiliary cameras or additional egocentric views under a comparable token budget would help isolate the key contribution.


Q2. How reliably do the VLM-based metrics in MAC-Bench reflect actual geometric and temporal consistency? Have the authors considered additional quantitative measurements or human verification to distinguish visually plausible generations from genuinely consistent world states?


Q3. Since auxiliary views are generated without observing the underlying environment at inference time, how does the framework ensure that these views represent the correct scene rather than merely maintaining internal consistency? How sensitive is the performance to auxiliary camera placement, occlusion, and limited spatial coverage?


Q4. The uncompressed variant substantially outperforms the complete model on Doom consistency tasks. Could the authors further clarify whether alternative compression settings can achieve a better balance between computational efficiency and consistency, particularly in highly dynamic environments?

2 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

We thank the reviewer for the encouraging assessment of the fixed-view design and δ-VAE. Your suggestions helped us improve the comparisons and explain the limits of our consistency claims more carefully (General Response Q1–Q4).

Q1. Would moving or egocentric views help at the same token budget?

Moving cameras (General Response Q3). As suggested, we report two development variants with six random-spline or agent-following cameras. Motion violates δ-VAE's sparse residual assumption, so both use uncompressed latents (7.2× more auxiliary tokens). They gain 11–33 points over w/o Aux. Views, but fixed views lead on Reunite at Start and Dynamic Event. Agent-following is slightly better on sparsely covered Reunite Far. We acknowledge the controls are incomplete; there is no consistent benefit at about 5× the cost.

Extra egocentric views move with the agent and largely overlap its view, leaving previous regions behind. Each costs a full view's tokens, whereas all six fixed views together cost 0.84 view-equivalents.

Token budget. We agree a matched comparison would be stronger, but time and compute prevented training it during rebuttal. xcVW Q1 discusses the available evidence: extra tokens without new views, the token-matched Lalign\mathcal{L}_{\text{align}} ablation and task-specific gains. A full comparison remains future work.

Q2. How reliable are the consistency metrics?

Human verification (General Response Q4). Annotators unaware of the method agree with the VLM on 93/96 episodes (κ = 0.94). The VLM accepts only 2/48 episodes they judged incorrect. Expected answers come from the ground truth (App. F.1), so the queries test world states beyond visual plausibility.

Geometry (General Response Q1). Reprojection checks agent positions at every frame against their true 3D trajectories; epipolar consistency checks camera geometry. Egocentric reprojection, where we can compare with baselines, reaches 88.4% hits versus 63.9%/55.7%.

Action-conditioned transitions. Solaris movement and building test action following. Here OverSeer trails the best baseline, and auxiliary views offer little benefit. Dynamic Event instead tests a scripted change off-screen, independent of actions (40.5 vs. 12.5).

Q3. Are auxiliary views correct and robust to camera placement?

Correctness. Auxiliary streams learn from ground-truth videos; evaluation checks true event states and 3D agent positions. Reprojection reaches 79.7% hits versus 95.8% on ground-truth video (General Response Q1). These checks cannot guarantee every scene detail: initially unseen regions can only be imagined plausibly and consistently.

Camera count. Increasing cameras from 0 to 6 improves accuracy and FID on MAC-Bench and Solaris world-state tasks (Figure 5). Correct partner placement rises from 70.7% to 76.8/83.7/88.4% with 2/4/6 cameras (General Response Q1).

Placement and occlusion. Default placement ignores occlusion (App. B), so the results include blocked views. Excluding answer regions retains about 80% of accuracy, above w/o Aux. Views (General Response Q2). Reunite Far reaches 37.9 vs. 6.2/0.0 despite sparse coverage at the periphery. Uncovered regions remain harder, motivating adaptive placement (App. E).

AuthorsOpenReview ↗

Rebuttal Part 2/2

Q4. Could compression adapt to more dynamic scenes?

Ratios. We tested 1×–16× compression for reconstruction and transfer (Figure 6). Mover transfer is best at 8× (IoU 26.7, identity 0.66), with little leakage (0.05). Downstream tests compare 8× with uncompressed (Table 2): Minecraft accuracy differs by at most 2.7 points, with about 5× less compute but some loss of visual and geometric fidelity.

Doom. Lighting changes and enemy motion cause large changes from the anchor (Sec. 4.2). Frame-wide lighting may be especially hard to compress, suggesting re-anchoring could help beyond changing the ratio. We have not verified this explanation or tested other ratios downstream (Uqyo Q4).

Adaptive compression. We appreciate this suggestion. Re-anchoring (App. E) could be triggered by the unnormalized change map M~t\widetilde{M}_t (Eq. 4), computed after decoding generated residuals because ground-truth latents are unavailable at inference. A new anchor costs as many tokens as four residual codes, so occasional updates would retain most savings. We will discuss this and per-chunk adaptive ratios as future work.

Q5. What does this enable beyond visual consistency?

Thank you for encouraging this broader perspective. OverSeer conditions rollouts on actions. Solaris movement and building test action following; changing actions from the same start allows counterfactuals, not yet evaluated quantitatively. Memory gains appear on 12-second Reunite rollouts and Solaris memory (87.5 vs. 20.8). A consistent shared world is needed for coordination; partner placement reaches 88.4% (General Response Q1), but decision-making and closed-loop planning remain untested. Quantitative evaluation covers 12 seconds (30 s only qualitatively), two agents and coverage-dependent gains. We will discuss policy learning and evaluation, re-anchoring for unbounded worlds and more agents as extensions.

Reviewer_xcVWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes OverSeer, a two-player video world model built on Wan2.2-TI2V-5B. It jointly generates the egocentric agent views and several fixed-pose auxiliary camera views, which serve as a proxy for a world-centric state. A δ-VAE compresses each auxiliary stream into a first-frame anchor and residual codes, reducing the number of tokens by eight times. Experiments in Minecraft and Doom show large gains on tasks where agents separate and later revisit the same place.

Critical strengths

S1: The motivation is clear. Unlike prior work that mainly expands what a world model conditions on, the paper focuses on what the model generates, which offers a fresh perspective for multi-agent world models. S2: The framework is simple. Jointly generating fixed auxiliary views gives all agents a shared and persistent reference without explicit global state reconstruction or pose-based memory retrieval, and δ-VAE keeps the extra cost low.

Critical weaknesses

W1: It is unclear whether the gain comes from the fixed views themselves or simply from the extra tokens they add. W2: The main benchmark is designed by the authors, and its test events happen where the cameras are placed.

Questions for authors

Refer to Weakness.

2 comments

We thank the reviewer for the encouraging assessment and helpful questions. Your concerns prompted us to test coverage more directly and clarify what the existing controls can establish (General Response Q1–Q3).

Q1. Fixed views or extra tokens?

We agree that a matched comparison is needed to isolate these effects. We could not train one within the rebuttal budget, but the following evidence may help:

  • Token count (Table 2). Keeping the same six views but removing δ-VAE uses 7.2× more auxiliary tokens and changes Minecraft accuracy by at most 2.7 points. Adding views with only 42% more tokens improves accuracy by 35.2/21.2/19.7 points. On Doom, however, uncompressed views gain 16–17 points, possibly because of rapid scene-wide changes (Uqyo Q4).
  • Moving cameras (General Response Q3). With 7.2× more auxiliary tokens, random-spline cameras score lower on all three tasks and agent-following cameras on two, although this comparison is not fully controlled.
  • Alignment loss (Table 4). Removing Lalign\mathcal{L}_{\text{align}} leaves token count unchanged but shifts capacity toward background, reducing accuracy by 2.4–4.5 points on all tasks.
  • Task-specific gains (Figure 5). More cameras help grounding, memory, consistency and MAC-Bench, while movement and building remain flat. This pattern is consistent with better access to persistent state.
  • Geometry (General Response Q1). Correct partner placement rises from 70.7% to 88.4% with six auxiliary views; 7.2× more auxiliary tokens improve it by a further 2.9 points.

These results suggest that persistent auxiliary information contributes beyond token count. We agree they do not fully isolate the effect of fixed viewpoints, and will make this limitation and the need for a matched comparison explicit.

Q2. Does camera placement favor our benchmark?

Thank you for raising this concern. Default placement samples poses from a scene-independent prior around agent 1's start, then greedily maximizes coverage without scene or event information. All tasks use this rule (App. B).

Coverage test (General Response Q2). Reunite at Start and Dynamic Event do take place near the densely covered start area. When no camera sees the answer region, accuracy remains 46.9/30.6/32.5 versus 60.0/37.5/40.0 by default: about 80%, still above w/o Aux. Views (24.4/16.3/20.6) and both baselines.

External benchmark (Table 1). On Savva et al.'s Solaris benchmark, OverSeer improves grounding (91.7 vs. 66.7), memory (87.5 vs. 20.8) and consistency (60.4 vs. 56.3), while trailing the best baseline on movement and building. The gains thus also appear on an independently designed benchmark.

Thank you for the detailed rebuttal. The responses address my questions well. I have no remaining concerns, and I will keep my original positive score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

We thank the reviewer for the careful reading and recognition of our control. Your questions helped us improve the evaluation and clarify details we had overlooked (General Response Q1–Q4).

Q1. How reliable is the evaluation?

Exclusions. Each Minecraft task has 32 × 5 = 160 episode-level evaluations per method. A run excludes episodes with unparsable judge responses (outside the accepted words; App. A); no generation failed. Exclusions range from 1 to 5 per task across methods (0.63%–3.13%), with only 1–2 for OverSeer.

Counting these as failures changes OverSeer from 60.2/37.9/40.5 to 60.0/37.5/40.0 and the best baseline from 12.5 to 11.9. Rankings are unchanged, as is FID, which uses all videos. We will report both protocols in the camera-ready version.

Run variation. Below we report the mean ± sample std over 5 independent runs.

MethodReunite at Start: VLM ↑FID ↓Reunite Far: VLM ↑FID ↓Dynamic Event: VLM ↑FID ↓
Solaris0.0 ± 0.0017.06 ± 2.816.2 ± 7.2320.23 ± 1.7912.5 ± 4.8726.52 ± 2.86
γ-World0.0 ± 0.0023.83 ± 4.720.0 ± 0.0025.71 ± 2.758.3 ± 3.8331.72 ± 3.99
OverSeer60.2 ± 13.3813.99 ± 1.2137.9 ± 6.1912.79 ± 1.7940.5 ± 5.8517.61 ± 1.57

OverSeer has the best mean on each metric. Its VLM margins of 28–60 points are more than four times the run-level std.

Judge reliability (General Response Q4). Annotators unaware of the method agree with the VLM on 93/96 episodes (96.9%, κ = 0.94), supporting its reliability on the MAC-Bench questions.

Claim scope. We agree that visibility and event state alone cannot establish geometric consistency; we should have been clearer. The new checks show better egocentric reprojection for OverSeer than for the baselines (General Response Q1). We will claim cross-view consistency of agent positions and scene state, rather than dense 3D reconstruction.

Q2. How are the transfer metrics computed?

We apologize for omitting these definitions. For each donor–host pair (A,B)(A,B), we decode the host anchor with its own residual (control CC) or the donor's (transfer TT). On frames T={20,…,120}\mathcal{T}=\lbrace 20,\ldots,120\rbrace, we use grayscale g(⋅)∈[0,1]g(\cdot)\in[0,1], Gaussian smoothing (σ = 2 px) and threshold τ = 0.06 to obtain the donor motion mask MtAM^A_t (MtBM^B_t likewise for the host). The mask MtTM^T_t captures changes from replacing the residual, including removed host motion, which can conservatively penalize Import IoU:

MtA=1[Gσ∗∣g(At)−g(A0)∣>τ],MtT=1[Gσ∗∣g(Tt)−g(Ct)∣>τ]M^A_t=\mathbf{1}\left[G_\sigma\ast\lvert g(A_t)-g(A_0)\rvert>\tau\right],\qquad M^T_t=\mathbf{1}\left[G_\sigma\ast\lvert g(T_t)-g(C_t)\rvert>\tau\right]

ImportIoU=100∣T∣∑t∈T∣MtA∩MtT∣∣MtA∪MtT∣\mathrm{ImportIoU}=\frac{100}{\lvert\mathcal{T}\rvert}\sum_{t\in\mathcal{T}}\frac{\lvert M^A_t\cap M^T_t\rvert}{\lvert M^A_t\cup M^T_t\rvert}

MoverId=1∣T′∣∑t∈T′1∣Pt∣∑p∈Pt⟨ϕp(At),ϕp(Tt)⟩,T′={20,25,…,120}\mathrm{MoverId}=\frac{1}{\lvert\mathcal{T}'\rvert}\sum_{t\in\mathcal{T}'}\frac{1}{\lvert P_t\rvert}\sum_{p\in P_t}\langle\phi_p(A_t),\phi_p(T_t)\rangle,\qquad \mathcal{T}'=\lbrace 20,25,\ldots,120\rbrace

Leak=1∣Ts∣∑t∈Tsρ([g(Tt)−g(Ct)]St,[g(A0)−g(B0)]St)\mathrm{Leak}=\frac{1}{\lvert\mathcal{T}_s\rvert}\sum_{t\in\mathcal{T}_s}\rho\left(\left[g(T_t)-g(C_t)\right]_{S_t},\left[g(A_0)-g(B_0)\right]_{S_t}\right)

Here ϕp\phi_p are L2-normalized DINOv2 ViT-B/14 patch features (with registers). PtP_t contains patches with > 50% of their pixels in MtAM^A_t, so comparing features at the same location rewards both appearance and placement. StS_t excludes the 7 px dilation of MtA∪MtBM^A_t\cup M^B_t; Ts={t:∣St∣>500}\mathcal{T}_s=\lbrace t:\lvert S_t\rvert>500\rbrace. We use Pearson correlation ρ to measure donor background leakage, with positive values indicating leakage. Only Mover Id. uses a learned model; the others use pixel differences. Per domain (Minecraft and Doom), we sample 100 test episodes and form 100 pairs, both randomly, then average over pairs (200 total). We will add this to App. A.

AuthorsOpenReview ↗

Rebuttal Part 2/2

Q3. How are costs and resolutions measured?

Workload. Figure 6 estimates Wan2.2-TI2V-5B DiT and video-VAE decoding FLOPs for two agents and KK auxiliary views: 241 frames per view at 512×384 (11,712 DiT tokens per view), 25 denoising steps and global attention over all views. This compares the representations under one-shot generation; our actual inference is causal and autoregressive.

Anchors. Thank you for catching the omitted anchor cost. Including one per auxiliary view gives 0.139 view-equivalents: 7.2× fewer tokens, rather than 8×. At six views, cost rises from 73.86 to 77.19 PFLOPs, reducing savings over 413.81 PFLOPs from 5.6× to 5.4×. Overhead rises from 37.5% to 41.8%. We will correct Figure 6, Table 2's caption and the text.

End-to-end savings. On an NVIDIA GB300, a 241-frame autoregressive rollout with six auxiliary views takes 60–105 s with δ-VAE versus 330–550 s without it, roughly a 5× speedup.

Resolution. We apologize for the inconsistency in App. A. Minecraft and real captures use 576×384, not 576×320; Doom uses 480×320. Minecraft experiments, Minecraft transfer clips and the token example all use 576×384 (864 = (576/16) × (384/16) video-VAE positions per frame, further patchified 2×2 by the DiT). Only the FLOPs curve uses 512×384. Token reduction is independent of resolution, and the computed FLOPs ratios remain 5.3–5.5×. We will clarify App. A in the camera-ready version.

Q4. Why 8× compression, and are baseline comparisons fair?

Why 8×. We chose it as a compromise. Import IoU and Mover Id. peak at 26.7 and 0.66, while Donor Leak falls from 0.55 at 1× to 0.05; mover transfer degrades at 16× (Figure 6). It saves about 5× compute (Q3) with at most 2.7 points' difference in Minecraft accuracy, within run variation. We acknowledge the cost in visual and geometric fidelity (Table 2; General Response Q1).

Doom. Uncompressed accuracy is 16–17 points higher. Rapid lighting changes and enemy motion cause larger changes from the anchor than in Minecraft (Sec. 4.2). Compression may lose frame-wide lighting changes, but we have not isolated this cause. If confirmed, it would motivate re-anchoring (App. E), beyond changing the ratio. We will clarify this in Sec. 4.2.

Matched training. We agree this would be the ideal control and regret that we could not retrain Solaris and γ-World during rebuttal. Our closest control, w/o Aux. Views (Table 2), uses the same backbone, episodes and step budget, removing only auxiliary views and δ-VAE to leave two directly coupled agent streams. Adding views raises accuracy from 25.0/16.7/20.8 to 60.2/37.9/40.5 in Minecraft and 12.0/0.0 to 62.0/67.0 in Doom. The Minecraft gain accounts for 58–70% of the gap to the best baseline. We cannot separate the remaining effects of backbone, data and budget without retraining.

Reviewer_UqyoOpenReview ↗

Thanks for the additional experiment.

The evaluation questions are answered, and the concern no longer holds. The transfer metrics are now defined in enough detail to reproduce, the cost accounting also moved against the paper. I accept all three answers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.