acceptodds
Under review as a conference paper at ICLR 2027

MindDrive: An All-in-One Framework Bridging World Model and Vision-Language Model for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving (E2E-AD) has emerged as a new paradigm, where trajectory planning plays a crucial role. Existing studies mainly follow two directions: trajectory generation-oriented approaches, which focuses on pro- ducing high-quality trajectories with simple decision mechanisms, and trajectory selection-oriented approaches, which perform multi-dimensional evaluation to select the best trajectory yet lack sufficient generative capability. In this work, we propose MindDrive, a harmonized framework that integrates high-quality trajectory generation with comprehensive decision reasoning. It establishes a structured reasoning paradigm of “what-if simulation – candidate generation – multi-objective trade-off.” In particular, the proposed Future-aware Trajectory Generator (FaTG), based on a World Action Model (WAM), performs ego-conditioned “what-if” simulations to predict potential future scenes and generate foresighted trajectory candidates. Building upon this, the VLM-oriented Evaluator (VLoE) leverages the reasoning capability of a large vision–language model to conduct multi-objective evaluations across safety, comfort, and efficiency dimensions, leading to reasoned and human-aligned decision- making. Extensive experiments on the NAVSIM-v1 and NAVSIM- v2 benchmarks demonstrate that MindDrive achieves state-of- the-art performance across multi-dimensional driving metrics, significantly enhancing safety, compliance, and generalization, and offering a promising path toward interpretable and cogni- tively guided autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, End-to-End Autonomous Driving, World Model, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_a8DBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MindDrive, an end-to-end autonomous driving framework that combines a Future-aware Trajectory Generator (FaTG) powered by a Transformer-Mamba World Action Model (WAM) for ego-conditioned BEV scene rollouts with a VLM-oriented Evaluator (VLoE) based on Tiny-LLaVA-1B to score and select trajectory candidates, evaluated on the NAVSIM-v1 and NAVSIM-v2 benchmarks.

Critical strengths
  • Coupling ego-conditioned world model rollouts with multi-objective candidate evaluation addresses a practical trade-off between trajectory generation quality and selection accuracy in end-to-end planning.
  • The empirical evaluation covers both NAVSIM-v1 and the newer NAVSIM-v2 benchmarks (including the safety-critical Navsafe and two-stage Navhard splits), demonstrating consistent improvements in aggregate PDMS and EPDMS metrics.
Critical weaknesses
  • Unaddressed Computational Overhead of 256-Way BEV Rollout and 1B VLM Inference: According to Section 3.2.1 and Appendix A.2, WAM constructs a separate BEV feature map for each of the N=256 trajectory anchors (forming a 4D tensor of shape N by H by W by C) and rolls out a Transformer-Mamba-Transformer network across multiple future steps, followed by running a 1B-parameter VLM (Tiny-LLaVA-1B) for candidate scoring. Despite this heavy pipeline, the paper provides zero analysis of inference latency (ms per frame), memory footprint, or FLOPs, making it unclear whether the framework is practical for real-time closed-loop driving.
  • Disconnect Between Claimed Cognitive Reasoning and Actual Scorer Distillation: While the paper claims VLoE performs interpretable semantic reasoning, the VLM does not generate any textual explanations or chain-of-thought deliberation. Instead, it processes a static template prompt, extracts the hidden state of a single score token, and uses an MLP head to regress ground-truth simulator scores precomputed on the 256 fixed anchors (Appendix A.2). This essentially uses a 1B VLM as a black-box feature extractor to distill the rule-based NAVSIM scorer (similar to Hydra-MDP and WoTE), and introduces a mismatch between scoring fixed anchors and evaluating the dynamically offset trajectories produced by FaTG.
  • Complete Omission of Training Objectives and Loss Formulations: Neither the main paper nor the appendix provides the mathematical definitions or weighting schemes for the training objectives, including the simulator BEV supervision loss for WAM, the trajectory offset regression loss for FaTG, and the multi-metric scoring loss for VLoE. In addition, Equation 13 compresses all N=256 candidates into K metric queries and Equation 15 extracts a single score token hidden state, leaving it unclear how per-candidate multi-metric scores are decoded and matched to the 256 candidates.
  • Sensor Modality Contradictions on Navhard and Marginal Gains on Navtest: MindDrive is designed as a Camera-LiDAR fusion model, yet Table 3 reports Stage II results on Navhard where synthetic future observations are strictly camera-only (as noted in Appendix A.3), without explaining how the missing LiDAR input is handled in Table 3 or why the sensor setup differs from Table 8 and Table 9. Moreover, on NAVSIM-v1 Navtest (Table 1), adding the 1B VLM and 256-way world model rollout yields only a 0.6 PDMS gain over WoTE and DIVER while slightly trailing them on No-at-fault Collision (98.4 vs. 98.5) and Comfort (99.9 vs. 100.0).
Questions for authors
  1. What is the exact per-frame inference latency (in ms), peak GPU memory usage, and parameter count of MindDrive broken down by Perception, WAM (for N=256 scene variants), and VLoE (Tiny-LLaVA-1B), and how does its runtime compare to WoTE and DiffusionDrive?
  2. In Equation 4, does WAM process all N=256 full-resolution BEV feature maps independently through the Transformer and Mamba blocks, and how does VLoE map a single score token hidden state (Equation 15) to multi-objective scores for 256 distinct trajectory candidates?
  3. Could the authors provide the explicit mathematical equations and loss weights for all training objectives in Stage 1 and Stage 2 (Appendix A.2), and clarify how VLoE accounts for the trajectory offsets added in Equation 9 when its supervision is precomputed on the 256 fixed anchors?
  4. How is the Camera-LiDAR MindDrive model evaluated in Table 3 where Stage II of Navhard provides only camera-based synthetic observations without LiDAR point clouds, and what is the exact sensor configuration difference between Table 3 and Table 8?
Reviewer_UjVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MindDrive is an end-to-end autonomous driving framework that combines high-quality trajectory generation with comprehensive decision reasoning. It uses a "what-if simulation – candidate generation – multi-objective trade-off" approach, featuring the Future-aware Trajectory Generator (FaTG) for foresighted trajectory creation and the VLM-oriented Evaluator (VLoE) for multi-dimensional evaluation across safety, comfort, and efficiency. Experiments show MindDrive outperforms existing methods in safety, compliance, and generalization on NAVSIM benchmarks.

Critical strengths
  • A very systematic approach to the work, it seems that the entire system was built up piece by piece in a complete manner, rather than being derived from minor tweaks to a baseline method.
  • Relatively good ablation study experiments; Tables 4 through 8 validate the effectiveness of the main modules.
Critical weaknesses

The main shortcomings of this work are concentrated in two aspects: first, the methodology appears to be a combination of several existing modules pieced together. Although the engineering effort is relatively significant, there is limited technical innovation; second, on the experimental side, there is a lack of strong evidence to demonstrate that the method is sufficiently effective.

  • Technically, this work generates future trajectories through a simulator and supervises future features at the trajectory level. This approach has also been used in many world model works for end-to-end autonomous driving, such as WoTE and LAW. Using VLM as a critic for trajectory evaluation is not novel either, and there is a notable lack of sufficient experimental support in the ablation study for this critic.
  • Experimentally, some of the comparison methods are somewhat outdated—the most recent ones being from 2025—and on the NAVSIMv1 leaderboard, top results far exceed 88.9 despite differences in backbone models. Comparing how different VLMs as critics affect results would be necessary.
Questions for authors
  • Lack of results on closed-loop benchmarks, with the benchmarks focused on NAVSIM, although NAVSIM remains the most mainstream experimental benchmark for end-to-end autonomous driving.
  • Providing some visualizations would be quite helpful, especially to show in which typical cases the method performs better and what obvious failure patterns exist. Visualizing these can help readers better understand the method.
Reviewer_jhFXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provided enhanced E2E framework for both trajectory generation (WAM-based) and selection (VLM-based). It reports state-of-the-art results on NAVSIM.

Critical strengths

Clear idea based on good intuition on generation vs selection-focused approach; Sound benchmarking results and ablation study with consistent gains observed.

Critical weaknesses

Novelty is incremental as the contribution is mainly the combination The contribution of VLM-based trajectory selection was not clear, albeit showing improvement in the ablation study. Unclear whether the improvement is due to the inclusion of VLM, or simply due to the increase of model capacity.

Questions for authors

Can you provide some runtime figures such as latency or FPS, in comparison with other baseline models in the benchmarking dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.