acceptodds
Under review as a conference paper at ICLR 2027

World Models in Robotics and Autonomous Driving: A Survey of the Latency-Fidelity Trade-off and the Case for Unified Scene-Motion Dynamics

Abstract

World models – learned predictive models that compress an agent's observations into a latent representation and forecast how that representation evolves under action – have become a unifying substrate for both simulated control and autonomous driving. This paper synthesizes thirty such systems (2018–2026) along four axes: the five training objectives in current use (variational latent dynamics, discrete autoregressive token prediction, denoising score matching, joint-embedding feature prediction, and value-equivalent dynamics); four architecture families (recurrent/variational, Transformer/diffusion, Joint-Embedding Predictive Architectures, and driving/robotics-specific systems); the choice of 3D scene representation, from raw pixels to bird's-eye-view latents, occupancy grids, and point clouds; and five largely incommensurable evaluation protocols, under which only about a quarter of latency-relevant papers report any throughput figure at all. A persistent tension between representational richness and inference latency runs through every family: the most expressive generative world models are, by their own authors' account, not yet real-time, while real-time-capable models sacrifice visual fidelity, multi-agent modeling, or explicit 3D grounding. We argue that unifying scene and motion representation within a single, low-latency model – rather than treating world modeling and action planning as separate stages – is the central open challenge for physical, real-time robotic and autonomous-driving deployment, and examine three candidate directions toward it – continuous-time (Neural ODE) dynamics, cross-embodiment generalization, and structured state-space backbones – each grounded in evidence from the surveyed corpus or its margins, though none yet demonstrated at the scale the gap requires.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, model-based reinforcement learning, autonomous driving, scene representation, Joint-Embedding Predictive Architectures, diffusion models, vision-language models, partially observable Markov decision processes, evaluation protocols, latency-fidelity trade-off, structured state-space models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_M6AJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper attempts to categorize world models from the perspectives of training objectives, architectures, scene representation, downstream applications, and evaluation protocols, and proposes that world models for robotics and autonomous driving face a latency–fidelity trade-off, and that unifying scene and motion representations is a key area for breakthroughs. However, the current manuscript presents substantial issues regarding the accuracy of the factual information cited, the comparative methods used, and the core arguments. **These issues affect the conclusions of the review and go beyond the scope of minor editorial revisions.

Critical strengths

S1. Timely deployment-oriented perspective. S2. Broad coverage of modeling paradigms. S3. Useful distinction between downstream roles. S4. Attention to evaluation limitations.

Critical weaknesses

W1. Factual inaccuracies in the literature synthesis. The survey misinterprets V-JEPA 2’s video frame rates as inference throughput and incorrectly states that WorldDrive reports no FID/FVD. These errors undermine the reliability of its core evidence.

W2. Insufficiently grounded latency–fidelity comparison. The proposed Pareto frontier combines runtime measurements from different settings with subjective fidelity rankings. Without comparable measurements and clearly defined metrics, it cannot substantiate the claimed trade-off.

W3. Limited support for the central conclusion. The survey advocates scene–motion unification as the central solution, but does not establish why it should take priority over alternatives. WorldDrive’s efficiency also depends on distillation and bypassing scene generation during inference, complicating this interpretation.

Additionally, there are still noticeable traces of AIGC, e.g., "at the user’s later, explicit request" in Appendix Overview.

Questions for authors

Refer to Critical Weaknesses.

Reviewer_Kdd7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper surveys thirty world-model systems across architectures, scene representations, deployment roles, and evaluation protocols, arguing that a latency–fidelity trade-off motivates unified scene–motion dynamics for robotics and autonomous driving.

Critical strengths
  1. The taxonomy across architectures, representations, and deployment roles provides a clear entry point to the literature, and the discussion of incompatible evaluation protocols highlights a genuine barrier to comparing world models for physical deployment.
Critical weaknesses
  1. The latency numbers backing the central taxonomy are internally inconsistent and misread primary sources. Section 3.1, Table 1, and Figure 4 place V-JEPA 2 robot action inference at 4–16 FPS, whereas Section 5 (p. 7) of the survey itself states that V-JEPA 2 requires 16 seconds per action. In the original V-JEPA 2 paper (arXiv:2506.09985), 16 seconds per action is the actual MPC planning latency on an RTX 4090 (§4.2, Table 3), while 4 FPS is the video frame sampling rate used during training (Appendix B.1). Conflating data sampling rates or backbone throughput with closed-loop planning latency distorts the survey's assessment of real-time viability.

  2. Figure 4 cannot support a quantitative Pareto frontier or the claimed "missing middle." As the caption notes, GAIA-2 and UniSim have no published latency, their horizontal coordinates are assigned qualitatively, and the vertical axis blends visual fidelity with multi-agent expressiveness by hand. Plotting qualitative estimates on a numerical log-latency axis without standardized measurement protocols or reproducible placement criteria turns an informal conceptual sketch into an unsupported empirical claim.

Questions for authors
  1. Where does the 4–16 FPS action-inference figure for V-JEPA 2 come from, and how do you reconcile it with the 16-second planning time cited on p. 7? Will you revise Table 1 and Figure 4 to strictly separate video frame sampling, single-pass forward throughput, and end-to-end planning latency?

  2. Once models without reported latency are removed from the quantitative axis and timing definitions are standardized, what concrete data remains to support the Pareto frontier and "missing middle" in Figure 4?

Reviewer_SLWbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper surveys world models for robotics and autonomous driving, covering architectures, scene representations, applications, and evaluation. Its main focus is the trade-off between fidelity and latency, and it argues for unified scene and motion modeling. The discussion is based on results from existing papers, with no new experiments.

Critical strengths

None

Critical weaknesses

The latency–fidelity trade-off is central to the paper, but the comparisons do not support it well enough. Figure 4 puts fidelity, representation richness, and multi-agent capabilities on a single qualitative scale, while latency numbers are missing for some models and measured under different conditions for others. The paper discusses representation choices, but does not clearly explain why models with different tasks and outputs should be ranked this way. The authors acknowledge these limitations, but the evidence is still too limited for what they present as the survey’s main distinguishing contribution.

Questions for authors
  1. Could the authors explain the vertical ordering in Figure 4? What is the basis for ranking models with different tasks and outputs on a single scale of fidelity, representation richness, and multi-agent capabilities?
  2. Given the differences in tasks, hardware, and inference procedures noted in the paper, which comparisons do the authors consider the strongest evidence for the latency–fidelity trade-off, and why?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.