acceptodds
Under review as a conference paper at ICLR 2027

HGWM: Hierarchical 3D Gaussian World Model with 4D Occupancy Forecasting for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving requires a world state that preserves explicit three-dimensional geometry, evolves coherently over time, and remains directly accessible to the planner. Existing driving world models commonly rely on dense grids or latent representations, making future geometry and its connection to planning less explicit. We propose the Hierarchical Gaussian World Model(HGWM), which formulates semantic 3D Gaussian primitives as an explicit recurrent world state shared by 4D occupancy forecasting and end-to-end trajectory planning. Rather than evolving all primitives uniformly, HGWM decomposes the scene into comparatively stable background and actor-centric foreground fields and models their distinct intra- and cross-field interactions through geometry-aware hierarchical communication and sparse local refinement. A residual transition operator jointly evolves Gaussian geometric, feature, opacity, and semantic attributes, while sparse expert routing accommodates heterogeneous foreground dynamics. Crucially, the same autoregressively evolved Gaussian state is rendered into future semantic occupancy and directly queried to generate multi-modal ego trajectories, coupling scene forecasting and planning through a shared geometric state. Under the reported camera-only nuScenes protocol, HGWM achieves 25.49% semantic mIoU and 49.86% binary IoU at 3s, together with an average planning L2 error of 0.28m and a 0.11% collision rate. It further reaches a PDMS of 92.1 on NAVSIM and a Driving Score of 82.6 with a 61.8% Success Rate on Bench2Drive. These results support hierarchical Gaussian evolution as an explicit shared world-state formulation for future scene prediction and trajectory planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
4D occupancy forecasting, end-to-end autonomous driving, world models, 3D Gaussian representation, temporal scene modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GrSiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Using an explicit recurrent semantic Gaussian world state to unify 4D occupancy prediction and trajectory planning. The topic is valuable, the design is complete, and the experiments cover three benchmarks: nuScenes, NAVSIM, and Bench2Drive.

Critical strengths
  1. The datasets used in the experiments are relatively comprehensive.
  2. The foreground–background separation aligns with the distinct evolution patterns of dynamic and static structures in AD tasks.
Critical weaknesses
  1. The paper claims that a “shared state” supports forecasting and planning, but the authors themselves admit that the current ablations do not isolate the causal effect of the shared state on planning.
  2. The order of the four directed interactions is hand-designed and lacks theoretical justification or ablation support.
  3. DS-SAGA relies on a strong sparse-neighborhood assumption: K=16 is fixed and lacks sensitivity analysis.
  4. The residual evolution is overly generic and lacks an explicit motion model.
  5. Occupancy and planning merely read the same Gaussian tensor, but no experiment demonstrates that planning truly depends on the predicted geometry.
  6. The fixed set of 6,400 Gaussians cannot add or remove actors. Does this affect dynamic scenarios such as actors appearing after occlusion, entering or leaving the field of view, and long-horizon interactions?
  7. It is unclear how the ego query aggregates information from the Gaussians, among the five candidates, only the lowest-error one is supervised, and there are no ego-collision, comfort, or dynamics-feasibility constraints, so the planning loss is relatively weak.
Questions for authors

Please refer to weaknesses.

Reviewer_uWAmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed a 3D Guassian-based world model with 4D forecasting for autonomous driving scenes. The authors choose the semantic Gaussian primitives as the representation and establish a field-aware transition combining both foreground and background. They also validate the effectiveness on nuScenes, Bench2Drive, and NAVSIM.

Critical strengths
  1. The basic motivation is clear: the paper would introduce explicit 3D world state to strengthen the geometry awareness and planning ability.

  2. Hierachical interactions between foreground and background information are reasonable. For the autonomous driving generation field, agent and background are usually decoupled.

  3. The experiments on three benchmarks are sufficient, although the comparison baselines are somehow limited.

Critical weaknesses
  1. The main concern is limited novelty, since the Gaussian representation is commonly chosen in recent world models. Could the authors explain the main difference with other works, such as GaussianDWM (CVPR'26)?

  2. The method part is hard to follow, where for each designed novelty, authors should introduce the high-level insight first and then the detailed method details. However, in Section 3.1-3.3, the network details are directly described. I recommend the authors add a pipeline overview in the first paragraph.

  3. The comparison baselines are limited. For example, on NAVSIM, some representative works such as DrivingLAW, Epona, etc are not compared. This weakens the effectiveness and sota claim in this paper.

  4. I wonder whether the method can be evaluated on NAVSIM V2, which is adopted commonly in AR world models for closed-loop verification.

  5. More efficiency comparisons are needed, since for the real-world planning, the latency is also a crucial metric.

  6. The ablation studies cannot cover all the main designs. More results on key contributions are required.

Questions for authors

Please refer to the weakness part, and I also question why the manuscript has less than 9 pages. The visualization samples are quite short (only one figure), and also the ablations are limited to support all the contributions.

Reviewer_XPpMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HGWM represents a driving scene with semantic 3D Gaussians and evolves them autoregressively as a shared world state for 4D occupancy forecasting and trajectory planning. It separates foreground and background fields, models their interactions through geometry-aware hierarchical attention and sparse local refinement, and uses sparse experts and residual updates to predict future states. Occupancy and multimodal ego trajectories are decoded from the same predicted states. The paper reports results on nuScenes, NAVSIM, and Bench2Drive.

Critical strengths
  1. Extensive experimental evaluation: Experiments cover occupancy forecasting and planning across three benchmarks, including closed-loop driving, with additional component ablations.
  2. Clear presentation: The writing, figures, and tables clearly explain the framework, its components, and the experimental results.
Critical weaknesses
  1. Unisolated benefit of state sharing: The ablations do not separate the value of a shared future Gaussian state from the effects of added interaction modules, supervision, and regularization.
  2. Limited handling of scene changes: The fixed Gaussian set lacks explicit actor creation and removal mechanisms; robustness to newly revealed objects and longer rollouts remains insufficiently validated.
Questions for authors
  1. How robust is foreground–background grouping when Gaussian supports overlap object boundaries? Could grouping errors cause background drift or distorted foreground motion during rollouts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.