acceptodds
Desk rejected at ICLR 2027

OccWAM: 4D Occupancy-Augmented World-Action Modeling for End-to-End Autonomous Driving

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

World-Action Models (WAMs) offer a promising paradigm for end-to-end autonomous driving by jointly modeling future visual observations and driving actions. However, world representations primarily based on 2D images or videos remain limited in explicitly modeling the 3D structure required for safe driving. Although some methods forecast future point clouds, their sparse surface samples do not explicitly characterize volumetric occupancy. Driving planning requires understanding how scene geometry and spatial constraints evolve, beyond predicting future sensor observations. To address these limitations, we turn to semantic 3D occupancy, which provides a dense, explicit, and unified representation of scene geometry and semantics. Modeling its temporal evolution directly captures changes in the 3D scene structure. Building on this representation, we propose OccWAM, the first World-Action Model to explicitly incorporate occupancy modeling for 4D occupancy-augmented action generation. Under a flow-matching framework, OccWAM introduces a 4D occupancy generation expert alongside video and action generation experts to jointly predict future visual observations, evolving 3D occupancy, and ego trajectories. These experts interact through cross-modal token attention within a unified Mixture-of-Transformers architecture, providing structured spatial context for action generation. Furthermore, since occupancy representations contain extensive spatial information, directly incorporating them inevitably incurs substantial computational overhead. we introduce Trajectory-Centric Occupancy Routing (TCOR), to dynamically partition the occupancy space into action-relevant regions and redundant background regions, enabling the model to focus on geometric information in areas that directly influence driving behavior. Extensive experiments on nuScenes demonstrate that OccWAM achieves strong trajectory prediction performance, with an average trajectory L2 error of 0.38 m and a state-of-the-art collision rate of 0.002. Moreover, OccWAM achieves computational efficiency comparable to other models while maintaining prediction accuracy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving;World-Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dWqGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OccWAM combines video, 4D occupancy, and trajectory generation within a flow-matching mixture-of-transformers architecture. Its trajectory-conditioned occupancy routing mechanism selects a small subset of occupancy tokens around candidate motion corridors. Experiments on NAVSIM and nuScenes study planning quality, occupancy prediction, and the accuracy-efficiency trade-off of routing.

Critical strengths

(+) The motivation for combining appearance with explicit volumetric structure is clear. Occupancy provides a useful spatial representation for planning, and the paper implements a concrete mechanism for reducing the cost of processing it.

(+) The controlled comparisons show consistent planning improvements: the full model reaches 91.1 PDMS versus 89.9 for Video-Action, and reduces average nuScenes L2 error from 0.41 to 0.36 and collision rate from 0.16% to 0.12%.

(+) The routing study includes both full occupancy and random token selection. Selecting 64 of 625 tokens preserves most of the reported displacement accuracy while reducing latency from 4701 ms to 2841 ms. The comparison with ground-truth current occupancy also usefully exposes sensitivity to perception quality.

Critical weaknesses

(-) The existing ablations do not fully isolate the value of predicting future occupancy from the benefit of extra 3D supervision, input information, or model capacity. Static occupancy conditioning and an Occupancy-Action model without the video expert would be particularly informative controls.

(-) The current experiments are primarily conducted on nuScenes and NAVSIM. Would it be possible to consider supplementing them with experiments on NAVSIM v2, so as to further substantiate the effectiveness of the proposed method through results that are more oriented toward closed-loop evaluation?

Questions for authors
  1. Can the authors add static-occupancy and Occupancy-Action controls to separate future prediction from additional perception information and the video expert?

  2. How do FullOCC and routed occupancy compare on collision metrics and cases where a hazard initially lies outside the selected corridors? What fraction of relevant occupied regions is retained?

  3. Which camera views are used by the current-occupancy estimator, and is its runtime included in the latency? Please report hardware, denoising steps, and component-wise inference costs.

Reviewer_FHnKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OccWAM incorporates 4D occupancy into world-action modeling by jointly generating future video, occupancy, and ego trajectories with three experts under a unified flow-matching framework. Trajectory-Centric Occupancy Routing selects occupancy tokens around scene-conditioned trajectory hypotheses to focus cross-expert interaction and reduce computation. Experiments on nuScenes and NAVSIM show planning improvements, while routing retains accuracy close to the full-occupancy variant.

Critical strengths
  1. Well-motivated joint modeling: Joint generation of video, 4D occupancy, and ego trajectories connects visual dynamics with the spatial constraints central to driving.
  2. Empirical improvements: Results on nuScenes and NAVSIM show planning gains, while TCOR reduces computation with little loss relative to full occupancy processing.
Critical weaknesses
  1. Unverified routing generalization: TCOR selects local occupancy around three refined, training-derived anchors, but coverage of rare maneuvers and unfamiliar road geometries is not established.
Questions for authors
  1. How does OccWAM's planning accuracy–latency trade-off compare with autoregressive models such as OccWorld under matched hardware and comparable prediction settings? Please report denoising steps and a latency breakdown.
  2. How sensitive is the accuracy–latency trade-off to the numbers of selected trajectory hypotheses and retained occupancy tokens?
  3. Could you evaluate OccWAM on nuPlan or a comparable large-scale closed-loop benchmark to assess longer-horizon planning and interactions with other agents?
Reviewer_64c4OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

OccWAM jointly models future video, 4D occupancy, and ego trajectories using modality-specific experts coupled through Mixture-of-Transformers attention under a flow-matching objective. Its Trajectory-Centric Occupancy Routing (TCOR) selects occupancy tokens near three predicted trajectory hypotheses to reduce computation. Experiments report 91.1 PDMS on NAVSIM v1 and 0.36 m average trajectory error with a 0.12% collision rate on nuScenes. TCOR retains approximately 10% of occupancy tokens while approaching the full-token model’s planning accuracy.

Critical strengths
  • A well-motivated integration of geometry and visual dynamics. Explicit occupancy provides a useful spatial representation for action generation. Within the proposed architecture, adding the occupancy branch improves NAVSIM PDMS from 89.9 to 91.1 and reduces nuScenes average L2 error from 0.41 to 0.36 m (Table 4), providing evidence that occupancy augmentation benefits planning.

  • Controlled evidence supporting trajectory-guided token selection. With the same 64-token budget, TCOR achieves 0.36 m average L2 error versus 0.45 m for random selection, approaching the full model’s 0.35 m (Table 6). This supports the value of selecting spatially relevant tokens and demonstrates a useful accuracy–computation trade-off.

Critical weaknesses
  1. Important stronger NAVSIM v1 baselines are omitted, leaving competitiveness unestablished. Table 1 excludes RAP-DINO and DrivoR etc recent papers, which report 93.8 and 93.1 PDMS, respectively—both above OccWAM’s 91.1. These results are documented in RAP, Table 1 and DrivoR’s official results. Training-data and sensor differences require explicit qualification, including RAP’s additional rasterized training data. Nevertheless, the current comparison does not demonstrate a clear performance advantage over stronger existing planners.

  2. NAVSIM v2 evaluation is missing. Section 4 evaluates only NAVSIM v1 PDMS and nuScenes planning metrics. NAVSIM v2 EPDMS results, with a clearly specified split and evaluation protocol, would materially strengthen the assessment of planning robustness. Given the limited advantage established on v1, this omission leaves the broader effectiveness of occupancy-augmented world–action modeling insufficiently validated.

  3. The benefit of future occupancy modeling is not isolated. Table 4 introduces current occupancy inputs, future occupancy generation, and TCOR together. Consequently, the improvement could partly reflect additional current-scene geometry or selector supervision. A baseline retaining identical current occupancy conditioning and selector supervision while removing future occupancy prediction is needed to substantiate the central claim about modeling temporal occupancy evolution.

  4. The deployed routing variant lacks sufficient forecasting and robustness validation. Table 3 evaluates occupancy forecasting with TCOR disabled, whereas the main planner uses TCOR. Appendix A.5 specifies that unselected tokens receive neither subsequent occupancy processing nor occupancy supervision. Thus, full-model forecasting results do not establish the quality of the routed model’s predictions. Robustness when the selected anchors poorly cover the required maneuver also remains untested.

  5. Absolute inference cost remains substantial. Although TCOR reduces occupancy-token count by 89.8%, end-to-end latency decreases by 39.6%, from 4701 to 2841 ms (Table 6). The remaining latency raises questions about practical online planning. Denoising-step counts, timing conditions, and a breakdown including occupancy estimation are needed to assess the complete computational cost.

Questions for authors
  1. Why are RAP-DINO and DrivoR omitted from Table 1? Please compare their results while documenting training splits, additional data, pretraining, and all camera inputs, including those used by the occupancy predictor.

  2. Can you report NAVSIM v2 EPDMS and component metrics, clearly identifying the evaluation split and whether the protocol uses one or two stages?

  3. With current occupancy inputs and selector supervision held fixed, how much improvement specifically comes from future occupancy prediction?

  4. What forecasting accuracy and collision performance does the actual TCOR variant achieve, particularly when its top-three anchors poorly cover the desired maneuver?

  5. What inference hardware, batch size, and denoising-step count produce the reported latency, and does it include current occupancy estimation? What planning accuracy is retained with fewer denoising steps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.