- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript presents OverSeer, a multi-agent video world model designed to improve temporal and cross-agent consistency. The central idea is to introduce fixed-pose auxiliary camera views that are jointly generated with the agents' egocentric observations, providing a persistent visual reference for the shared environment. To reduce the additional computational cost, the authors propose δ-VAE, which compresses auxiliary video streams by encoding changes relative to their initial reference frames.
The framework is evaluated on Minecraft and Doom, with additional qualitative demonstrations on real-world multi-camera captures. The results show substantial improvements in spatial memory and multi-agent consistency over existing approaches.
Critical strengths
The manuscript is well motivated and addresses an important challenge in multi-agent world modeling. Using fixed auxiliary views as a proxy for a shared world-centric representation is intuitive and offers an interesting alternative to approaches that rely exclusively on egocentric observations or explicit geometric reconstruction.
The proposed δ-VAE is an interesting contribution. By exploiting the temporal redundancy of stationary viewpoints, the framework reduces the token overhead associated with generating multiple auxiliary streams. The cross-scene residual transfer experiments also provide useful evidence that the compressed representations can separate dynamic changes from relatively static scene content.
OverSeer achieves substantial improvements on the newly introduced MAC-Bench, particularly for agent reunion and dynamic event consistency. The ablations on auxiliary camera count and latent compression further support the main design choices. The qualitative demonstrations across Minecraft, Doom, and real-world captures are also informative.
Critical weaknesses
My main concern is that the experiments do not fully isolate the contribution of fixed viewpoints themselves. The ablation without auxiliary views demonstrates the benefit of introducing additional visual streams, but it does not establish whether fixed viewpoints are substantially better than alternative multi-view configurations. For example, comparing against moving auxiliary cameras or additional egocentric views under a comparable token budget would help determine whether the improvements specifically originate from persistent spatial references rather than additional visual supervision and modeling capacity.
Another concern is the evaluation of world and agent consistency. The proposed MAC-Bench provides meaningful tests of spatial memory, but its quantitative evaluation relies primarily on VLM-based judgments of relatively simple questions, such as whether another agent is visible or whether an environmental event has progressed. These metrics do not fully characterize geometric, temporal, or causal consistency across generated views. More direct measurements, such as cross-view reprojection consistency, object trajectory agreement, or action-conditioned state transition accuracy, would strengthen the central claims.
The generation of auxiliary views at inference time deserves further investigation. Since the model receives only initial agent observations and sampled camera poses, the content of previously unseen regions must be inferred rather than directly observed. Consequently, generated auxiliary views may remain temporally consistent without necessarily representing the correct underlying environment. Moreover, the camera sampling strategy does not explicitly consider scene occupancy or occlusion. Evaluating sensitivity to camera placement and limited spatial coverage would provide a clearer understanding of the approach's practical boundaries.
The efficiency–quality trade-off of δ-VAE is also significant in more dynamic environments. In Doom, removing δ-VAE improves Reunite at Start accuracy from 62.0% to 78.0% and Reunite Far accuracy from 67.0% to 84.0%. Although the computational savings are meaningful, these differences suggest that compression may discard important dynamic information. A more detailed discussion of adaptive compression or reference-frame updating would strengthen this contribution.
The manuscript could also benefit from a broader discussion of agentic world modeling beyond multi-view video generation. World models can serve not only as visual predictors but also as interactive simulators that support action-conditioned forecasting, counterfactual reasoning, long-horizon memory, multi-agent coordination, and decision-making. While OverSeer improves visual consistency across agents and time, it remains less clear how these improvements translate into more general agentic capabilities, such as anticipating the consequences of alternative actions, supporting cooperative planning, or providing reliable feedback for downstream decisions. Discussing how the proposed framework relates to these broader directions, as well as its potential limitations, would help better position its contribution within the evolving world modeling literature.
Questions for authors
Q1. Could the authors clarify whether the improvements primarily originate from the fixed camera poses or from introducing additional visual streams? A comparison with moving auxiliary cameras or additional egocentric views under a comparable token budget would help isolate the key contribution.
Q2. How reliably do the VLM-based metrics in MAC-Bench reflect actual geometric and temporal consistency? Have the authors considered additional quantitative measurements or human verification to distinguish visually plausible generations from genuinely consistent world states?
Q3. Since auxiliary views are generated without observing the underlying environment at inference time, how does the framework ensure that these views represent the correct scene rather than merely maintaining internal consistency? How sensitive is the performance to auxiliary camera placement, occlusion, and limited spatial coverage?
Q4. The uncompressed variant substantially outperforms the complete model on Doom consistency tasks. Could the authors further clarify whether alternative compression settings can achieve a better balance between computational efficiency and consistency, particularly in highly dynamic environments?
2 comments
Rebuttal Part 2/2
Q4. Could compression adapt to more dynamic scenes?
Ratios. We tested 1×–16× compression for reconstruction and transfer (Figure 6). Mover transfer is best at 8× (IoU 26.7, identity 0.66), with little leakage (0.05). Downstream tests compare 8× with uncompressed (Table 2): Minecraft accuracy differs by at most 2.7 points, with about 5× less compute but some loss of visual and geometric fidelity.
Doom. Lighting changes and enemy motion cause large changes from the anchor (Sec. 4.2). Frame-wide lighting may be especially hard to compress, suggesting re-anchoring could help beyond changing the ratio. We have not verified this explanation or tested other ratios downstream (Uqyo Q4).
Adaptive compression. We appreciate this suggestion. Re-anchoring (App. E) could be triggered by the unnormalized change map (Eq. 4), computed after decoding generated residuals because ground-truth latents are unavailable at inference. A new anchor costs as many tokens as four residual codes, so occasional updates would retain most savings. We will discuss this and per-chunk adaptive ratios as future work.
Q5. What does this enable beyond visual consistency?
Thank you for encouraging this broader perspective. OverSeer conditions rollouts on actions. Solaris movement and building test action following; changing actions from the same start allows counterfactuals, not yet evaluated quantitatively. Memory gains appear on 12-second Reunite rollouts and Solaris memory (87.5 vs. 20.8). A consistent shared world is needed for coordination; partner placement reaches 88.4% (General Response Q1), but decision-making and closed-loop planning remain untested. Quantitative evaluation covers 12 seconds (30 s only qualitatively), two agents and coverage-dependent gains. We will discuss policy learning and evaluation, re-anchoring for unbounded worlds and more agents as extensions.
Rebuttal Part 1/2
We thank the reviewer for the encouraging assessment of the fixed-view design and δ-VAE. Your suggestions helped us improve the comparisons and explain the limits of our consistency claims more carefully (General Response Q1–Q4).
Q1. Would moving or egocentric views help at the same token budget?
Moving cameras (General Response Q3). As suggested, we report two development variants with six random-spline or agent-following cameras. Motion violates δ-VAE's sparse residual assumption, so both use uncompressed latents (7.2× more auxiliary tokens). They gain 11–33 points over w/o Aux. Views, but fixed views lead on Reunite at Start and Dynamic Event. Agent-following is slightly better on sparsely covered Reunite Far. We acknowledge the controls are incomplete; there is no consistent benefit at about 5× the cost.
Extra egocentric views move with the agent and largely overlap its view, leaving previous regions behind. Each costs a full view's tokens, whereas all six fixed views together cost 0.84 view-equivalents.
Token budget. We agree a matched comparison would be stronger, but time and compute prevented training it during rebuttal. xcVW Q1 discusses the available evidence: extra tokens without new views, the token-matched Lalign ablation and task-specific gains. A full comparison remains future work.
Q2. How reliable are the consistency metrics?
Human verification (General Response Q4). Annotators unaware of the method agree with the VLM on 93/96 episodes (κ = 0.94). The VLM accepts only 2/48 episodes they judged incorrect. Expected answers come from the ground truth (App. F.1), so the queries test world states beyond visual plausibility.
Geometry (General Response Q1). Reprojection checks agent positions at every frame against their true 3D trajectories; epipolar consistency checks camera geometry. Egocentric reprojection, where we can compare with baselines, reaches 88.4% hits versus 63.9%/55.7%.
Action-conditioned transitions. Solaris movement and building test action following. Here OverSeer trails the best baseline, and auxiliary views offer little benefit. Dynamic Event instead tests a scripted change off-screen, independent of actions (40.5 vs. 12.5).
Q3. Are auxiliary views correct and robust to camera placement?
Correctness. Auxiliary streams learn from ground-truth videos; evaluation checks true event states and 3D agent positions. Reprojection reaches 79.7% hits versus 95.8% on ground-truth video (General Response Q1). These checks cannot guarantee every scene detail: initially unseen regions can only be imagined plausibly and consistently.
Camera count. Increasing cameras from 0 to 6 improves accuracy and FID on MAC-Bench and Solaris world-state tasks (Figure 5). Correct partner placement rises from 70.7% to 76.8/83.7/88.4% with 2/4/6 cameras (General Response Q1).
Placement and occlusion. Default placement ignores occlusion (App. B), so the results include blocked views. Excluding answer regions retains about 80% of accuracy, above w/o Aux. Views (General Response Q2). Reunite Far reaches 37.9 vs. 6.2/0.0 despite sparse coverage at the periphery. Uncovered regions remain harder, motivating adaptive placement (App. E).