acceptodds
Under review as a conference paper at ICLR 2027

SwiftWM: An Event-Guided World Model for Fast Motion Planning in Autonomous Driving

Abstract

Timely adaptation to changing traffic is essential for autonomous driving. Dynamic vision sensors (DVS) capture sparse, asynchronous brightness changes with fine temporal resolution. Frame aggregation can obscure event timing and weaken their low-latency advantage. Iterative diffusion planning adds computational cost to trajectory updates. We propose SwiftWM, an event-guided latent world model for fast motion planning in autonomous driving. SwiftWM encodes dynamic spatiotemporal event graphs with a spiking graph neural network. Geometry-guided alignment and gated fusion integrate event features with RGB–LiDAR context and radar observations into a persistent bird's-eye-view latent state. DVS packets incrementally update affected regions while retaining the state elsewhere. A predictor forecasts future latents from the updated state. At each planning update, a diffusion planner conditions on current and predicted future latents to generate a new path with a single solver step, while a separate head predicts target speed. Predictive and planning supervision jointly optimize the world model and planner over causal sequences. We evaluate SwiftWM through closed-loop driving on Bench2Drive and recorded-input diagnostics of future-state prediction and computational cost. SwiftWM achieves a driving score of 89.82 and a success rate of 76.36% on Bench2Drive, while delivering a 76.38% reduction in state-maintenance computation through incremental execution compared with full recomputation in recorded-input evaluations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous driving, Latent world models, Dynamic vision sensors, Spiking graph neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NomEOpenReview β†—
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SwiftWM, an event-guided latent world model framework for fast motion planning in autonomous driving that combines RGB-LiDAR data with Dynamic Vision Sensors (DVS). The proposed architecture comprises a BEV Encoder, a SAG Encoder, a Context Encoder, a Predictor, and a Diffusion Planner, which collectively perform multi-modal fusion to collaboratively predict the vehicle's trajectory and target speed. The authors evaluate their method on the closed-loop Bench2Drive benchmark and provide a breakdown of computational costs across each stage.

Critical strengths
  1. The paper addresses a meaningful and practical problem by exploring the fusion of RGB-LiDAR and event camera (DVS) data. Leveraging the complementary properties of standard frame-based cameras and event sensors is a well-motivated direction for autonomous driving.
  2. The authors provide insightful qualitative visualizations across driving scenarios, displaying telemetry plots (speed, acceleration, and braking) side-by-side against baseline methods.
  3. The ablation studies are thorough, examining computational overhead and planning latency, the number of solver steps for the diffusion path planner, the impact of event data inputs, and the encoder design.
Critical weaknesses
  1. Limited benchmark diversity: Evaluation is confined to a single synthetic closed-loop benchmark (Bench2Drive). The paper does not assess the model on standard real-world driving benchmarks such as NAVSIM (likely constrained by the absence of event camera data in current real-world datasets).
  2. Fairness of comparison and source of performance gain: Comparing against prior baselines raises fairness concerns, as none of them have access to DVS inputs. More critically, the ablation shows that SwiftWM without DVS (SwiftWM w/o DVS) underperforms existing baselines under identical sensor settings. This strongly suggests that the reported gains stem primarily from the addition of extra sensor hardware rather than core architectural contributions.
  3. Lack of sensor modality breakdown in benchmarks: Because the proposed framework leverages an additional modality, Table 2 should explicitly list the input modalities/sensor configurations utilized by each baseline to ensure a fair and transparent comparison.
  4. Unclear formulation and mathematical notation: Section 3 and Algorithm 1 are quite difficult to follow due to several undefined variables and mathematical operators. Furthermore, there is conflicting (e.g., π‘ˆπ‘˜π‘ˆ_π‘˜ vs. Uk\mathcal{U}_k) and overloaded notation, where identical symbols denote different entities between the equations and the pseudocode.
  5. Missing recent baselines: The empirical comparison lacks key recent baselines, such as HiP-AD [1], DiffRefiner [2], Bridgedrive [3] (in Table 4.).
Questions for authors
  1. Since SwiftWM underperforms existing baselines on an identical sensor footprint (without DVS), is the performance gain driven solely by the additional sensor hardware? Furthermore, while the abstract highlights a 76.3876.38% reduction in computation within isolated state-maintenance, Table 3 reveals that actual policy runtime only improves by 1.24×1.24 \times (178ms→144ms178 ms \to 144 ms). How does this 144 ms end-to-end latency compare against the competing baselines in Table 2, and is 144 ms fast enough to claim "fast motion planning" for closed-loop reactive driving?
  2. Can you add comparisons against some missing baselines (such as HiP-AD [1], DiffRefiner [2], Bridgedrive [3]) on Bench2Drive in Table 4., and also add extra column to explicitly specify the sensor modalities used by each baseline?
  3. Can you thoroughly revise and streamline the mathematical notation and Algorithm 1 across Section 3, ensuring all variables and operators are clearly defined and consistently used, to make the method significantly easier for readers to follow?
  4. Given that the evaluation is currently restricted to the synthetic Bench2Drive benchmark, is it possible to provide validation on a real-world driving dataset to demonstrate how well your approach handles the sim-to-real gap same as previous baselines?

[1] Y. Tang, Z. Xu, Z. Meng and E. Cheng, "HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder," 2025 IEEE/CVF International Conference on Computer Vision (ICCV)

[2] Yin, Liuhan, et al. "Diffrefiner: Coarse to fine trajectory planning via diffusion refinement with semantic interaction for end to end autonomous driving." Proceedings of the AAAI Conference on Artificial Intelligence.

[3] Liu, Shu, et al. "Bridgedrive: Diffusion bridge policy for closed-loop trajectory planning in autonomous driving." International Conference on Learning Representations. Vol. 2026. 2026.

Reviewer_ptPWOpenReview β†—
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dynamic Vision Sensors (DVS) are able to capture changes between image frames, complementing appearance and geometry. However, the heavy computational cost of diffusion planning weakens the low-latency advantage of DVS. To this end, the authors propose an event-guided latent world model for fast motion planning in autonomous driving, namely SwiftWM. To be specific, SwiftWM encodes dynamic spatiotemporal event graphs with a spiking graph neural network, incrementally updates a persistent BEV latent state and generates a path through single-step diffusion conditioned on predicted latents. All experiments are conducted on the Bench2Drive and LEAD-DVS benchmark based on the CARLA simulator, showing obvious closed-loop model performance improvement. However, I still have several concerns, as detailed below.

Critical strengths
  1. In the field of end-to-end autonomous driving, this paper provides extensive closed-loop evaluation (i.e., within CARLA) to validate the effectiveness of the proposed SwiftWM, showing sufficient performance gains.

  2. Such validation in an interactive closed-loop environment also alleviates the concern about the deployment of SwiftWM in real-world scenarios.

  3. The paper is clear and well-written, making the approach easy to understand.

Critical weaknesses
  1. The Broad applicability of SwiftWM. It is unclear whether the proposed method is limited to diffusion-based planners. Since diffusion planners are only one component of an end-to-end driving pipeline, restricting the method to diffusion planning would limit its contribution. Can it also improve other types of planners while maintaining low latency?

  2. The design of experiments has several issues: i) Figure 4 compares SwiftWM with a simple persistence baseline that does not model object motion, whereas SwiftWM is explicitly trained to predict future states. However, even basic motion extrapolation could substantially outperform persistence for moving vehicles. More reliable baselines are necessary to assess the effectiveness of the proposed prediction method.

    ii) The lack of ablations for key modules. Graph representation, spiking graph encoder, and geometry-guided gated fusion are central to SwiftWM, but their individual contributions are not sufficiently isolated through ablations on Bench2Drive (i.e., only w/o DVS in Table 4).

    iii) As shown in Figure 5, the speed of SwiftWM generally increases over the interval covered by the displayed images (βˆ’1.5 to +1.5 s), with the main slowdown occurring later. No unsafe event is shown for SwiftWM despite its higher speed, and thus the slower DriveTransformer and Hydra-NeXt may require less braking. Moreover, the speed profile suggests a mild initial slowdown followed by much sharper deceleration, raising concerns about the timeliness and smoothness of its response. Therefore, this visualization does not clearly demonstrate an advantage.

  3. In Figure 8, the predicted turning direction of Hydra-NeXt appears inconsistent with the intended route. Given the explicit navigation guidance in CARLA, it would be better if the authors verify the configuration are correct for this baseline.

  4. 20-step diffusion planning yields higher ADE than single-step planning in both Table 1 and 2. Please consider to provide more discussions on this part.

  5. Avoid splitting the word 'memory' across lines 390–391.

Questions for authors

See weaknesses

Reviewer_rusbOpenReview β†—
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper target an important problem in autonomous driving: incorporating high-temporal-resolution event signals into a planning-oriented world model. The overall system is carefully designed and combines event processing, multimodal fusion, future-state prediction, and motion planning in a coherent way. The system is competent and the results are encouraging, but the novelty is not yet strong enough for me, and the current ablations do not convincingly support the paper's broader claims about event-guided world modeling and fast planning.

Critical strengths

The closed-loop results on Bench2Drive are promising. In particular, the improvements over the no-DVS variant suggest that event input can be useful for time-sensitive driving behaviors such as emergency braking and merging.

The paper includes useful implementation diagnostics, including future-state prediction, solver-budget comparisons, and runtime measurements. These analyses make the system easier to inspect than many end-to-end driving papers.

Critical weaknesses
  1. The main contribution feels largely like an engineering combination of existing components, including event graphs, spiking encoders, gated fusion, recurrent BEV states, future latent prediction, and truncated diffusion planning. The paper does not clearly establish a new principle beyond this combination, nor does it isolate which components are responsible for the final gains.

  2. The experimental evidence for the world-model component is limited. Future prediction is mainly compared against current-state persistence, and the effect of future latents on closed-loop driving is demonstrated only through a few intervention examples rather than a systematic Full-versus-Current closed-loop evaluation.

  3. The reported DVS benefit is somewhat difficult to interpret. The closed-loop gap between SwiftWM and SwiftWM w/o DVS is substantial, while withholding DVS at inference causes only a small drop in occupancy AP; this discrepancy deserves a stronger controlled analysis.

  4. The efficiency claims should be stated more carefully. The 76.38% reduction applies to state-maintenance computation rather than end-to-end runtime, while the reported full-policy speedup is only 1.24x and a nontrivial fraction of updates still exceed the 50 ms event-packet interval.

Questions for authors

All my questions and concerns are present in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.