acceptodds
Under review as a conference paper at ICLR 2027

Cross-View World Models

Abstract

World models enable agents to plan by imagining future states, but existing approaches operate from a single viewpoint, typically egocentric, even when other perspectives would make planning easier. We introduce Cross-View World Models (XVWM), trained with a cross-view prediction objective: given frames from one viewpoint, predict the future state from the same or a different viewpoint after an action is taken. Because the input and output views may share little or no visual overlap, the model must learn view-invariant representations of the environment's 3D structure and the agent's place in it. Trained on synchronized multi-view gameplay data from Aimlabs, the model localizes itself on an overhead map from egocentric frames alone and renders egocentric scenes from only a small marker on that map. Probing the diffusion transformer layer by layer pinpoints where these representations emerge and how actions transform them. In early layers, the encoding of the agent's position and heading depends on the input view; in the middle layers it converges onto a view-invariant code, where a linear probe trained on egocentric inputs decodes pose from bird's-eye inputs without retraining, and vice versa. Within this code, heading lies on a ring distributed across many units. When we inject a yaw action, the ring rotates rigidly by the commanded angle, while the heading encoded in the earlier, view-dependent layers is left untouched: the model applies the turn to its view-invariant code, not to the view-specific encoding that precedes it. Together, these results give a mechanistic account of the model's “cognitive map”: how it tracks where it is and which way it is facing in a known environment. Such mechanistic understanding of how world models represent an agent's spatial state, and how actions transform it, is essential for reliable embodied agents.

Primary area:
interpretability and explainable AI
Keywords:
World models, Self-supervised learning, Mechanistic interpretability, View-invariant representations, Diffusion transformers, Computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FcVTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Cross-View World Models (XVWM), which train an action-conditioned diffusion world model to predict a future frame from either the same or a different camera view. The experiments use 4,186 synchronized Aimlabs gameplay episodes with egocentric, bird's-eye, over-the-shoulder, and front views. The paper compares Single-View, Two-View, and Four-View models, evaluates cross-view generation and bird's-eye marker localization, and probes hidden states for a shared linear code of allocentric position and heading. The main interpretability result is that linear heading and position probes transfer between egocentric and bird's-eye contexts in middle transformer blocks, and that injecting a yaw action rotates the decoded heading in this interval.

Critical strengths

1.The paper proposes a simple and well-motivated cross-view prediction objective and evaluates it beyond qualitative samples. In particular, Table 1 measures ego-to-BEV localization on 1,257 test samples, and Figure 6 evaluates ego-target perceptual quality with LPIPS and DreamSim, including bootstrap 95% confidence intervals. The Two-View model obtains a 2.0-pixel median BEV marker error, while the marker is approximately 17 pixels on its long side. These results substantiate that the model can associate egocentric context with global position in the evaluated setting.

2.The interpretability analysis is unusually careful for this type of model. The authors use episode-grouped cross-validation, test zero-shot transfer of a probe between ego and BEV contexts, include shuffled-label controls, and apply frozen probes to disjoint episodes under yaw interventions. Figures 3–5 give convergent evidence that, in the reported middle-layer interval, pose is linearly decodable across views and the decoded heading changes approximately proportionally to the injected yaw. This is a meaningful characterization of the model's internal representation.

Critical weaknesses
  1. The evidence is limited to an episode-level 90:10 split within a known Aimlabs environment. The paper does not report a held-out environment, map, or spatial-region split, and it explicitly identifies transfer to unseen environments as an open question. This leaves an important alternative explanation for the strong localization and BEV-to-ego results: the model may learn a representation specialized to the training environment and its fixed map rather than a more general cross-view spatial representation. The current claims should therefore be limited to spatial grounding in a known environment unless stronger generalization experiments are provided.

  2. The intervention supports action-equivariance of a probed heading readout, but it does not yet establish that this code causally mediates image generation or that it is target-view independent. In Section 4.1, the target view is always egocentric during probing; therefore, the shared code observed for ego and BEV contexts may be partly specific to producing an egocentric target. The paper itself notes that changing the action input and reading the representation does not establish that the model uses this code to produce images. This limits the strength of the mechanistic claim that the model applies actions through a general view-invariant cognitive-map code.

Questions for authors

1.Can the authors evaluate a stronger generalization split, such as held-out Aimlabs environments/maps or spatially disjoint regions of a map? At minimum, please clarify how the 90:10 episode split controls overlap in locations and near-duplicate trajectories between training and test data.

2.Can the probing and yaw-intervention analyses be repeated with bird's-eye view as the target view, and ideally with multiple output views? In addition, can the authors test causal mediation more directly, for example by editing the identified heading-code direction at an intermediate layer and measuring the resulting change in generated images?

3.In Figure 6(b,c), the exposure-matched Single-View results are intermediate checkpoints, whereas the Two-View model is fully trained. Please report the matched total update count and total view-frame exposure, and clarify whether a control can hold total optimization compute fixed while isolating the effect of cross-view targets. This is needed to support the claim that the reported improvement is specifically positive transfer from cross-view training.

Reviewer_qxdpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To tackle the challenge of learning video world models across different camera views without using camera calibration or explicit 3D geometry, the authors propose Cross-View World Models (XVWM), an action conditioned video diffusion model predicting future frames across different camera viewpoints (such as egocentric/top-down).

Given context frames from one view, and a relative action, XVWM predicts future frames in a target view that's specified via a learnable target view embedding. The authors also probe the model's internal transformer layers and find that the middle layers form a shared pose representation across views.

XVWM is then validated on multi view gameplay videos in Aimlabs under single-view, two-view and four-view setups.

Critical strengths
  1. The paper tackles a practical issue.

  2. The proposed target view conditioning is fairly simple, and does not require explicit 3D / camera calibration. There seems to be good empirical results on localization and view generation.

Critical weaknesses
  1. The paper is quite difficult to follow and would really benefit from a better organization and clearer writing. For instance, after the section 2 "Related work", we jump directly to section 3 "Results". There are also a lot of direct references to the Appendices while the content haven't even been exposed in the main paper. For instance, l.186-187 mentions "In brief (Appendices B and C):...", and the details about the model and training are in the appendix". Section 3.1 starts with "As discussed in Appendix B...", which presupposes that the reader has read the Appendix before. This makes the paper really hard to follow.

  2. All the experiments are conducted in a single static Aimlabs scene and the trajectory, where the top down view map directly renders a ground truth position and marker, so the generalizability of XVWM is unclear.

Questions for authors

For the main points, please refer to the section on weaknesses.

Reviewer_xK7eOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a cross-view prediction objective for world modeling. Specifically, given the context from egocentric views / BEV views, the objective is to predict the BEV / egocentric views after a given action is taken. This is combined with the standard next-state prediction objective. The paper trains a conditional diffusion transformer model to serve as a cross-view world model (XVWM) in a simulated game setting (Aimlabs -- first person shooter game). Experiments are performed to demonstrate that the XVWM can effectively predict the next state (primarily based on the accuracy of the next pose in the BEV). Further probing experiments demonstrate that the CDiT network learns view-invariant representations of the agent's pose in the environment in later stages of the network (i.e., the pose is decodable from ego / BEV views given the same probe), and that the angular pose representation is cleanly modulated by rotation actions in the environment.

Critical strengths

S1. The motivation is strong, and cross-view world modeling is a promising route to better spatial representations

The paper trains a world model to predict an action's consequences from a viewpoint other than the input. Its aim is to push the model toward representations of the environment's structure and the agent's place in it (L012–016). The goal is well motivated, and the mechanistic analysis in Section 4 is an appealing way to study what such a model learns.

S2. The emergent cross-view invariance is an interesting and novel finding

A linear probe trained on ego-context states decodes pose from BEV-context states without refitting, and vice versa, in blocks 7–16 (Figure 3 and Section 4.2). It is interesting that this invariance emerges without an explicit invariance objective.

Critical weaknesses

W1. Writing clarity is poor

  • Critical information sits in the appendix. Section 3 compresses the experimental setup and main results into a few sentences and defers to Appendices A–C. The positive-transfer result, listed as a contribution (L117–119), appears only in Appendix B (Figure 6).
  • Key components are not described in the main text.
    • CDiT-L is named (L183) but described only by reference to NWM in Appendix A.
    • The data collection is not described: how the 4,186 Aimlabs episodes were collected, what the simulator produces, whether episodes come from the same or different environments, and whether any test environment is held out. Appendix A states only a "90:10 train-test split" (L659).
  • Terms are used before they are defined. "OS" appears in Figure 1 but is defined only in Appendix A. DreamSim is used in the main text (L187–188) without explanation; it is cited only in Appendix B (L706).
  • Several procedures are unclear.
    • How a BEV trajectory is encoded as context, and how the model renders egocentric scenes from it (L190–191).
    • How the probing data is selected (L259–265).
    • The R²R^² notation (L267).
  • Motivating claims lack citations or are outdated.
    • The argument about human perspective-taking is anecdotal and uncited (L046–048).
    • "Mammalian brains solved navigation differently ... track itself in space" has no citation (L053–082).
    • "Map-based navigation is more robust than the route-based planning that current world models perform" has no citation (L083–084).
    • The statements that current world models predict from a single, almost always egocentric, perspective (L041–043) and plan with ensembles of autoregressive rollouts (L051–053) rest on 2024 works. This is an inaccurate characterization of the current state of world models.
      • L041–043: multi-view and non-egocentric world models already exist. The paper itself cites MV-MWM, ReViWo and Ctrl-World (L148–150); uncited works include BEVWorld (2024) and MVISTA-4D (ICML 2026).
      • L051–053: this describes NWM's planner, not world models in general. For example, MVISTA-4D (ICML 2026) optimizes actions by gradient instead of brute-force rollouts.

W2. Novelty claims are overblown

  • The paper states that "this is the first use of cross-view prediction as a self-supervised objective for training world model dynamics" (L090–091, L114-116). Three groups of prior work bear on this claim.
    • Prior world models already use cross-view or multi-view objectives. MV-MWM (Seo et al., ICML 2023), ReViWo (Pang et al., ICLR 2025) and Ctrl-World (Guo et al., ICLR 2026) are discussed only in Appendix D, pointed to from one sentence in Section 2 (L148–150). BEV-WM (Lekkala et al., 2023) translates first-person views to BEV for navigation world models and is not cited.
    • Prior generative models already predict other viewpoints across time from a single input. Examples are 4DiM (Watson et al., ICLR 2025), GCD (Van Hoorick et al., ECCV 2024, including ego → top-down) and Consistent GQN (Kumar et al., 2018). None of these are cited. Section 2 dismisses cross-view synthesis as "without action conditioning" (L146–148). Against these works, the delta reduces to conditioning on agent actions plus a discrete view ID instead of camera poses, which is not a significant difference in my opinion.
    • MVISTA-4D (Wang et al., ICML 2026) generates action-conditioned future observations for other viewpoints from a single-view input. It was published in ICML 2026 and is not cited nor discussed.
  • The spatial-grounding claim has precedent in AI. The paper says "we show that cross-view training with a bird's-eye view can yield analogous spatial grounding" (L172–173). Artificial agents have already been shown to develop spatial representations like those of the hippocampal–entorhinal system (Banino et al., Nature 2018), and this work is not cited.
  • Parts of the probing findings repeat prior work. Joseph et al. (2026), cited in Section 2, already report mid-depth emergence and a distributed circular code for direction in video world models. Cueva et al. (ICLR 2020), not cited, show head-direction rings that are updated by angular-velocity input.

W3. The experiments are poorly designed

The experiments use a very limited setting, no baseline comparisons, and no control studies. The world model's usefulness is never tested.

  • The setting is very limited: a single environment and no generalization test.
    • The paper describes its results as being in a known environment (L029, L462; Figure 10). The BEV is "a fixed overhead view" (L654–655), the data uses a 90:10 train-test split (L659), and transfer to unseen environments is left open (L462–464). This restriction is not justified: the base architecture, NWM, was trained across multiple environments and evaluated on a held-out one.
    • The data choice (Aimlabs, L017) is not motivated, and only 2D planar motion (Δx, Δy, Δϕ) is supported (L017–019, L185).
  • There are no baseline comparisons.
    • The only baseline is the authors' own Single-View model (Appendix A, L674), plus an LPIPS reference to NWM's number on a different dataset (L710–711).
    • The closest multi-view world models are contrasted only in prose (Appendix D, L926–949). The claims that cross-view prediction is "strictly harder" and acts as "geometric regularization" (L945–949; L089–090) are never tested against them.
    • Table 1 (L194) omits single-view baselines for the same input and output views.
    • The conclusion that "cross-view prediction is an effective self-supervised signal for world models" (L448) has no support from standard benchmarks or comparisons with state-of-the-art methods.
  • There are no control studies for the claims made.
    • The untrained-network control is asserted in a contribution (L123–124) and in Section 4.2, but its results are "not shown" (L350–351).
    • Section 4 attributes the view-invariant code to "predicting one view from another" (L250–254), but two controls are missing that would test this:
      1. The same probes and yaw intervention on the Single-View model. The yaw-driven rotation (Section 4.3, L370) may also occur in a regular world model.
      2. A Two-View model trained only on same-view pairs (ego → ego, BEV → BEV), to separate the cross-view objective from multi-view data exposure.
    • As the authors acknowledge (L308–311), the intervention changes the action input and reads the representation. It does not show that the model uses this code to produce its predictions.
  • The world model's usefulness is never tested. The introduction motivates the work by planning efficiency (L050–053, L082–084), but no planning or downstream experiment is reported. The localization results (Table 1, L211–215) are not connected to any downstream use, so their significance is unclear.

References

Questions for authors
  1. (W2) What is the contribution relative to MV-MWM, ReViWo, Ctrl-World, BEV-WM, 4DiM, GCD, Consistent GQN and MVISTA-4D? Please restate or narrow the "first use" claim (L090–091) and contribution 1 (L114–116).
  2. (W2, W3) Can you compare experimentally, on the same data and backbone, against the objectives of the closest multi-view world models? For example: joint multi-view prediction (Ctrl-World style), contrastive view invariance (ReViWo style), and a cross-view-trained encoder with same-view dynamics (MV-MWM style). Without this, the claims of being "strictly harder" and of "geometric regularization" (L945–949) are unsupported.
  3. (W2) How does XVWM differ from novel view synthesis methods such as NeRF or 3D Gaussian Splatting, which also render novel views of a known environment from images? Is the method closer to novel view synthesis than to world modeling? The action-conditioning only changes the camera-pose, and this can be done programmatically.
  4. (W3) How was the Aimlabs data collected? Do all episodes come from one environment or several? Is the 90:10 split (L659) by episode within the same environment, and is any test environment held out? Why Aimlabs (L017)?
  5. (W3) Can you evaluate on held-out environments, as NWM does? Does cross-view prediction, particularly BEV → ego (L190–191), work when the environment has not been seen in training?
  6. (W3) Please show the untrained-network results (L350–351). Do the heading ring and yaw-driven rotation (Section 4.3) also appear in the Single-View model? Does a Two-View model trained only on same-view pairs develop the same cross-view-invariant code?
  7. (W3) Can Table 1 include single-view baselines with the same input and output views?
  8. (W3) Can you report a planning or other downstream experiment that supports the planning motivation (L050–053) and the "effective" claim in the conclusion (L448)?
  9. (W1) There are significant clarity issues with the paper. How do you propose to address these?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.