acceptodds
Withdrawn from ICLR 2027

World Observer: Joint Actor-Observer Generation for Persistent World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How can a world model continuously observe regions beyond the actor’s current view? Video world models simulate how an environment evolves from an agent's actions, yet remain actor-centric. Once an object leaves the actor's view, they lose direct evidence of its evolution, often failing to preserve its state and dynamics upon re-entry. To address this, we introduce World Observer, which decouples observing from acting by jointly generating a perspective actor for the agent-centric view with one or more panoramic observers that watch selected world regions. This allows objects that leave the actor's view to remain visually evolving in an observer, so their updated states are reflected when they re-enter. We ground both streams in the initial panorama through warping and introduce an Observer Sink of high-resolution perspective references that restores fine appearance when a region re-enters the actor view. Since the observers are decoupled from the actor, they can be placed freely across the scene and driven by control signals to steer out-of-view evolution. To evaluate out-of-view evolution, we further introduce world-space metrics and a benchmark spanning real and synthetic scenes. World Observer substantially improves out-of-view dynamics while remaining competitive in visual fidelity, camera control, and 3D adherence.

Primary area:
generative models
Keywords:
World Models, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gUSaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces World Observer, a video world modeling framework designed to maintain dynamic scene states beyond an actor's current field of view. The method jointly generates a perspective actor stream and one or more panoramic observer streams using a shared diffusion transformer. The observer streams continuously represent out-of-view regions, allowing dynamic objects to evolve and re-enter the actor's view with updated states. The framework also introduces flexible observer placement, an Observer Sink for preserving high-resolution appearance, and a benchmark with 3D displacement-based metrics for evaluating out-of-view dynamics.

Critical strengths
  1. The paper addresses the important problem of maintaining dynamic states beyond an actor's current field of view. Jointly generating perspective actor and panoramic observer streams provides a promising alternative to relying solely on historical memory or state extrapolation. Flexible observer placement further extends the framework to regions that cannot be continuously observed by the actor.

  2. The paper introduces real and synthetic benchmarks alongside 3D displacement-based metrics to evaluate object re-entry and motion consistency. This provides a useful evaluation perspective beyond conventional video quality metrics. The experiments and ablations also demonstrate promising improvements in visual fidelity and out-of-view consistency.

Critical weaknesses
  1. The main quantitative evaluation is limited to 77-frame sequences generated within a single chunk. Although the paper presents longer qualitative examples, it does not quantitatively evaluate state persistence across multiple autoregressive chunks or repeated out-of-view intervals.

  2. OOV-Dgt and OOV-Dself measure displacement between object exit and re-entry, but do not directly evaluate the full trajectory or state evolution during the unobserved interval. OOV-Dself also assumes consistency with pre-exit motion, which may not hold for turning, accelerating, or interacting objects.

Questions for authors

Q1. The main evaluation uses 77-frame sequences within a single generation chunk. Can the authors provide quantitative results for longer autoregressive rollouts, including repeated object disappearance and re-entry across multiple chunks?

Q2. OOV-Dgt and OOV-Dself rely on displacement between exit and re-entry. How do these metrics account for nonlinear motion, acceleration, or interactions that change an object's trajectory while it is out of view?

Q3. The 3D displacement metrics rely on estimated depth and object segmentation. Have the authors validated their accuracy against simulator-provided ground-truth positions or trajectories in CARLA?

Q4. In Table 1, the multi-observer model improves OOV-F from 0.580 to 0.722 but reduces OOV-Dself from 0.526 to 0.457. What causes this degradation in motion consistency when additional observers are introduced?

Q5. The real-world training data uses co-located actor and observer views, while spatially decoupled and multi-observer configurations rely on synthetic data. How well does the model generalize to real-world scenes with independently positioned observers?

Q6. The baselines use different input conditions, and HyDRA is evaluated with a different reconstruction protocol. How do the authors ensure that the comparisons fairly isolate the contribution of joint actor-observer generation?

Q7. The framework jointly generates actor and observer streams, but the evaluation does not directly measure their cross-view consistency. Can the authors quantify object identity, position, and appearance consistency between the two generated streams?

Q8. How does inference cost scale with the number of observers? Can the authors report GPU memory usage, generation latency, and quality changes as additional observer streams are introduced?

Reviewer_aCGzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper tackles the problem of out-of-view memory and progression in video generation-based world models from a multi-view video generation aspect. They propose to explicitly model the parts of the scene which are hidden from an actor's view, by placing observers at various points in the scene, mostly as panoramic viewers in those locations for maximum coverage. They also introduce their own benchmark to evaluate models' performance in these out-of-view agent behaviours.

Critical strengths

The problem the paper tries to tackle is interesting in itself; many previous work try to address memory for static objects (e.g. when an agent turns 90 degrees left and comes back, the scene often changes), but fewer works have explored evolving agents out of FoV. I think this problem is an interesting one which the community should think about and evaluate.

Critical weaknesses
  1. The practicality of the proposed approach is questionable in 4 ways: firstly, the additional view generations increases training/inference costs hugely. Secondly, the real videos are panoramic ones used for the observer and a crop of them used as the actor video, which is not very interesting for most world models, specially the driving ones, since they are already multi-view, so the panoramic view doesn't add much. Thirdly, the more interesting case which is movable observers at different locations requires a simulator or very complicated real-world setups, making the approach less scalable. Finally, and perhaps most importantly, this approach requires manually deciding the location/trajectory of the observer(s). There is no automatic way for the model to decide where is the most uncertainty and where it makes more sense to put an observer, while we often design world models to be a very scalable model of the world, not requiring manual placement of cameras/observers.

  2. The evaluation approach seems flawed to me. The authors extract object movements in 3D from GT videos and generated videos, and compare them for objects that leave field of view. However the movement of these objects is not deterministic, rather probabilistic and follow some distribution. Therefore comparison with GT, and even comparing with the the same object's trajectory while being in the field of view (FOV), is not a principled evaluation; imagine a car when exiting the FOV would break, therefore have a different 3D trajectory than the GT or its movement before exiting the FOV. A distribution-based metric on a number of generations could improve the benchmark in accurately evaluating performance in this task. Maybe that also partially explains why the OOV numbers in Table 1 don't have a big gap with second best methods.

Questions for authors
  1. The case where observer and actor's camera centers are not on aligned is more interesting, but only CARLA synthetic data was used to train model for such cases. Could you provide any metric/video examples to demonstrate generalization to real videos in this case?

  2. Improving the benchmark to account for distribution rather than direct deterministic comparison is a great addition and might improve your main results (Table 1). Could you please improve these metrics, fully describe the details of how you implement them, and share the new results?

  3. Addressing any of the weaknesses above is also appreciated, specially the question regarding how to decide where to put the observer(s) at inference time.

Reviewer_9pkpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on one problem of current video models: the dynamic world state is hard to preserve when the camera moves away. The authors classify the problem into four modes: frame-locked, lost, frozen, and impostor.

To address this problem, this paper proposes World Observer. It uses a single DiT (Cosmos-Predict2.5) to jointly generate a perspective actor video and one or more panoramic observer videos. To evaluate the model, this paper proposes a benchmark with three new metrics for 3D out-of-view dynamic consistency and valid leave-and-return cases. The method significantly improves out-of-view dynamics while being competitive on visual fidelity, camera control, and 3D adherence.

Critical strengths
  1. The design of using a panoramic video to keep track of global world state is novel and practical. The panoramic video only needs to be generated at a lower resolution. Prior works mostly infer the state change via implicit memory or generative prior.

  2. The paper clearly defines different modes of out-of-view failures and designs different metrics in the benchmark to evaluate them separately. The proposed metrics are deterministic and better than VLM-based metrics in prior out-of-view benchmarks.

Critical weaknesses
  1. Extra input is needed in the current system. The trajectories of both the actor and the observers must be specified in advance. The system also needs a panorama with depth as input. These conditions are not easy to get when applying to a new scene.

The extra inputs may also affect the comparison in Table 1. Most baselines receive only a perspective first frame. World Observer’s advantage in mPSNR, FID/FVD, and camera error likely comes from the panorama and depth. Compared with OmniRoam, which also takes a panorama, the real-set mPSNR is nearly identical (18.574 vs. 18.523).

  1. The proposed benchmark and metrics are still limited. It requires real video as ground truth. Also, current metrics can only reflect simple motion. OOV-D_{self} assumes the object continues its motion. The benchmark is restricted to simple scenes: ~1.26 target objects per clip, and prompts that almost all describe walking or driving straight. Turning, stopping, interactions, and other state changes are not covered.
Questions for authors
  1. How is the initial position and trajectory of the observer determined? Can it be set to the same trajectory as the actor?

  2. In the multi-observer setting, one observer is set at the final position of the actor trajectory. Does this setting leak additional information to the actor?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.