acceptodds
Under review as a conference paper at ICLR 2027

OccWAM: 4D Occupancy-Augmented World-Action Modeling for End-to-End Autonomous Driving

Abstract

World-Action Models (WAMs) offer a promising paradigm for end-to-end autonomous driving by jointly modeling future visual observations and driving actions. However, world representations primarily based on 2D images or videos remain limited in explicitly modeling the 3D structure required for safe driving. Although some methods forecast future point clouds, their sparse surface samples do not explicitly characterize volumetric occupancy. Driving planning requires understanding how scene geometry and spatial constraints evolve, beyond predicting future sensor observations. To address these limitations, we turn to semantic 3D occupancy, which provides a dense, explicit, and unified representation of scene geometry and semantics. Modeling its temporal evolution directly captures changes in the 3D scene structure. Building on this representation, we propose OccWAM, the first World-Action Model to explicitly incorporate occupancy modeling for 4D occupancy-augmented action generation. Under a flow-matching framework, OccWAM introduces a 4D occupancy generation expert alongside video and action generation experts to jointly predict future visual observations, evolving 3D occupancy, and ego trajectories. These experts interact through cross-modal token attention within a unified Mixture-of-Transformers architecture, providing structured spatial context for action generation. Furthermore, since occupancy representations contain extensive spatial information, directly incorporating them inevitably incurs substantial computational overhead. we introduce Trajectory-Centric Occupancy Routing (TCOR), to dynamically partition the occupancy space into action-relevant regions and redundant background regions, enabling the model to focus on geometric information in areas that directly influence driving behavior. Extensive experiments on nuScenes demonstrate that OccWAM achieves strong trajectory prediction performance, with an average trajectory L2 error of 0.38 m and a state-of-the-art collision rate of 0.002. Moreover, OccWAM achieves computational efficiency comparable to other models while maintaining prediction accuracy.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.