acceptodds
Withdrawn from ICLR 2027

DriveWLA: Unifying Future Dynamics and Scene Semantics through World-Language-Action Modeling for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, World Model, Vision-Language-Action Model

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_yQjYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveWLA, a driving framework that combines a video-pretrained World Expert, a vision-language model with a trainable adapter, and an Action Expert for trajectory generation. Cross-Expert Attention allows the World and Language branches to exchange information while the Action branch uses their representations to predict motion. The model is trained through trajectory imitation and then refined with Flow-GRPO using separately normalized driving rewards and a variance-based adjustment to the policy update. Experiments report 92.3 PDMS on NAVSIM v1, 90.1 EPDMS on NAVSIM v2, and a zero-shot HD-Score of 41.47 across 436 closed-loop HUGSIM scenarios, with additional open-loop transfer results on nuScenes and Bench2Drive.

Critical strengths
  1. The expert and attention ablations provide controlled evidence that combining the World and Language representations improves planning. Removing the World Expert reduces PDMS from 90.8 to 86.2, removing the Language Expert reduces it to 89.1, and disabling information exchange between them reduces it to 88.4 (Tables 5 and 7). These comparisons support both the value of the two branches and the additional benefit of allowing them to interact, strengthening the contribution beyond the performance of the complete system alone.

  2. The paper evaluates the learned policy in closed-loop HUGSIM scenarios, where its actions affect the observations it subsequently receives. Across 436 scenarios from four source datasets, DriveWLA achieves an HD-Score of 41.47 compared with 32.50 for the strongest listed baseline, while also improving time-to-collision and route-completion scores (Table 3). This evaluation provides evidence of driving capability beyond matching recorded trajectories, and Appendix B.3 describes the shared controller and evaluation protocol while acknowledging differences in camera inputs.

Critical weaknesses
  1. The experiments show that including the World Expert improves trajectory planning. What remains unclear is whether this improvement comes from predicting future scene dynamics, as the paper argues, or from providing additional visual features learned during video pretraining. The model is fine-tuned using only trajectory supervision, and removing the entire World Expert also removes its capacity and pretrained features, so this ablation does not distinguish these explanations. Comparing against a branch of similar capacity that processes only observed frames, and changing or removing the future-latent tokens, would help establish whether predictive dynamics contribute to the resulting trajectories.

  2. The paper lacks of video illustrations and visualizations of the open-loop and closed-loop results, especially in closed-loop scenarios it would be interesting to see how the proposed method deal with visual artifact caused by HUGSIM and whether the artifacts would affect the ablation conclusions.

Questions for authors
  1. Please clarify the weakness 2 setting, and if the authors could provide more visualizations of closed-loop behavior improvements (like having less compounding errors than other baselines/recover from failure, etc.)

  2. How do you establish that the World Expert improves planning through predicted future dynamics rather than additional pretrained visual features? Have you compared it with a branch of similar capacity using only observed frames, or modified the future-latent tokens and measured how the trajectory changes? Please also clarify whether Figure 4 visualizes features inferred from observed inputs and noise or features encoded from ground-truth future frames, given the description in Appendix D.

2 comments
AuthorsOpenReview ↗

Response to Reviewer yQjY (1/2)

We thank the reviewer for the constructive feedback and for recognizing our expert ablations and closed-loop evaluation.

Question1:\color{red}{\mathit{Question}\\ 1:} Could the authors provide more visualizations of closed-loop behavior, such as recovery from failure or less compounding error?

Response1:\color{blue}{\mathit{Response}\\ 1:} With our unified World–Language–Action architecture, DriveWLA achieves the highest overall HD-Score and route completion among the methods in Tab. 3. It also leads on Hard and Extreme HD-Score in the zero-shot HUGSIM evaluation:

MethodHard HD ↑Extreme HD ↑Overall HD ↑Overall RC ↑
VAD7.706.3013.9030.10
UniAD21.006.3031.2043.80
LTF12.0011.7023.1038.00
DrivoR20.0014.1032.5047.20
DriveWLA31.4115.8741.4754.99

Compared with DrivoR, DriveWLA improves overall HD-Score by 8.97 points and route completion by 7.79 points. In closed-loop evaluation, we also observe cases where the model corrects its trajectory or recovers from difficult situations. We will add corresponding visualizations to the updated paper PDF to show these situations.

Question2:\color{red}{\mathit{Question}\\ 2:} How does the method deal with visual artifacts caused by HUGSIM, and could these artifacts affect the ablation conclusions?

Response2:\color{blue}{\mathit{Response}\\ 2:} We evaluate DriveWLA zero-shot on HUGSIM’s rendered camera images. Visual artifacts may affect the policy’s observations and closed-loop behavior. However, our expert and information-flow ablations are conducted on NAVSIM (Tabs. 5 and 7) and do not use HUGSIM rendering, so their conclusions are unaffected by these artifacts. We will include representative rendered frames with planned-trajectory overlays in the updated paper PDF to make these cases clearer.

Question3:\color{red}{\mathit{Question}\\ 3:} Have you compared the World Expert with a branch of similar capacity using only observed frames, or modified the future-latent tokens and measured how the trajectory changes?

Response3:\color{blue}{\mathit{Response}\\ 3:} We evaluate two additional inference-time ablations of the future latents. The full model and both ablations use the same imitation-learning (IL) checkpoint, matching the ten-step setting in Tab. 8.

In the full DriveWLA (IL), each Cross-Expert Attention block uses World features from four observed-frame latents and two Gaussian-initialized future latents. Zero future latents sets the future-latent features to zero before each block without changing token positions or attention structure. Remove future latents leaves only the four observed-frame latents in the World branch.

SettingNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
Full DriveWLA (IL)99.297.797.2100.084.090.8
Zero future latents99.397.397.8100.081.289.7 (−1.1)
Remove future latents98.293.693.2100.077.784.1 (−6.7)

PDMS drops by 1.1 points when the future-latent features are set to zero and by 6.7 points when the future latents are removed. These results show that the planner benefits from the future-latent pathway.

We also train a history-latent-only variant from the same video-pretrained backbone. It uses latents from the four observed frames during training and inference, with no future latents passed to Cross-Expert Attention. The Language and Action branches, trajectory objective, and parameter capacity are unchanged.

ModelNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
History-latent-only99.197.096.5100.083.389.8
Full DriveWLA (IL)99.297.797.2100.084.090.8

The history-latent-only variant achieves 89.8 PDMS, compared with 90.8 for the full model, and scores lower on DAC, TTC, and EP. These results indicate that future latents contribute to planning performance.

Question4:\color{red}{\mathit{Question}\\ 4:} How do you establish that the World Expert retains future-dynamics information rather than only additional pretrained visual features?

Response4:\color{blue}{\mathit{Response}\\ 4:} We train two MLP probes to predict the future positions of objects in the surrounding scene over 4 s, supervised by their ground-truth future positions. We use only past and current images as visual input to the World Expert and extract features at its future-latent positions just before the final Cross-Expert Attention block. These features serve as the World features in both probes.

AuthorsOpenReview ↗

Response to Reviewer yQjY (2/2)

We train the probes on navtrain and evaluate them on the navtest split. ADE@2s and ADE@4s measure the average position error over future time points up to 2 s and 4 s, respectively. FDE@4s measures the position error at 4 s. All errors are in meters; lower is better. Moving denotes objects with an observed speed above 1 m/s.

Probe 1: Direct prediction. The baseline uses only current position. The other two experiments in the table add World features from the pretrained video model (before IL) or the fine-tuned World Expert (after IL). None receives explicit past positions or velocity.

Probe 1: inputADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Position only (baseline)2.3454.0347.0625.0138.71815.798
Position + World (before IL)1.9633.4086.0093.7896.61911.983
Position + World (after IL)1.9093.2855.7453.4586.01310.800

On moving objects, IL fine-tuning reduces ADE@4s from 6.619 to 6.013 m and FDE@4s from 11.983 to 10.800 m. These results show that the World features retain future-motion information after IL.

Probe 2: Prediction then correction. We first train a motion predictor (baseline) on past positions and velocity, then freeze it. We separately train an MLP to correct its predictions using current position and World features. The correction is added to the predictor’s output. Only the MLP is updated during training. In the zero-World experiment, we train the same MLP with the World features set to zero. This separates the benefit of World features from that of the added MLP and current-position input.

Probe 2: predictorADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Predictor Only (baseline)0.4571.0662.4020.7151.7954.310
Predictor + MLP (zero World features)0.4331.0262.3320.6981.7704.268
Predictor + MLP (World before IL)0.4290.9972.2430.6661.6393.901
Predictor + MLP (World after IL)0.4250.9862.2170.6551.5963.776

Compared with the zero-World experiment, adding World features after IL reduces moving-object ADE@4s from 1.770 to 1.596 m (9.8%) and FDE@4s from 4.268 to 3.776 m (11.5%). World features improve prediction even when past motion is already available.

To illustrate the World Expert’s future prediction ability, Fig. 4 visualizes its features after IL fine-tuning. Each row shows one driving scene. From left to right, the columns show the front-view image, PCA maps of history latents 1 and 4, and PCA maps of future latents 1 and 2. The history maps represent observed frames; the two future maps represent successive 2 s intervals over the 4 s prediction horizon. The two future maps are predicted from Gaussian-initialized latents, not encoded from ground-truth future images. Their structured patterns differ from the history maps and change between the two future intervals. Together with the motion probes, these visualizations indicate that the World Expert retains the ability to model future scene evolution after IL.

Question5:\color{red}{\mathit{Question}\\ 5:} Does Figure 4 visualize features inferred from observed inputs and noise, or features encoded from ground-truth future frames?

Response5:\color{blue}{\mathit{Response}\\ 5:} The future features in Fig. 4 are predicted by the IL-fine-tuned World Expert, not encoded from ground-truth future images. The model receives past and current images and Gaussian-initialized future latents. The last two columns visualize the resulting future features for successive 2 s intervals. Their structured patterns evolve over these intervals, providing qualitative support for retained future modeling alongside the quantitative probes above. We will make the prediction process explicit in the caption and Appendix D.

Reviewer_f3VqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DriveWLA aims to integrate World, Language, and Action experts into a unified framework. It introduces Cross-Expert Attention to enable information exchange among the three experts and further employs Flow-GRPO reinforcement learning to improve the driving policy.

Critical strengths
  1. The paper is well organized and easy to follow, with a clear overall architecture and methodology.

  2. The ablation studies are comprehensive and well designed. The analysis of expert composition, information flow, and training strategies helps explain the contributions of different components.

Critical weaknesses
  1. The additional contribution of the modified Flow-GRPO is relatively limited. It improves PDMS from 92.1 to 92.3 compared with the original Flow-GRPO. Moreover, increasing the number of flow-matching sampling steps introduces considerable inference latency, with only marginal improvements in planning performance.

  2. The effectiveness of future dynamics modeling is not sufficiently validated. Although the World Expert improves downstream planning performance, the paper lacks direct evaluation of whether its learned representations accurately capture future scene dynamics.

Questions for authors
  1. Could the authors provide further analysis or visualizations of the Language Expert to better understand how its semantic representations contribute to trajectory planning?

  2. Could the authors quantitatively evaluate the quality of future scene prediction to better validate the effectiveness of the World Expert?

3 comments
AuthorsOpenReview ↗

Response to Reviewer f3Vq (1/2)

We thank the reviewer for the constructive feedback and positive assessment of the paper's clarity and ablation studies.

Question1:\color{red}{\mathit{Question}\\ 1:} Could the authors provide further analysis or visualizations of the Language Expert to better understand how its semantic representations contribute to trajectory planning?

Response1:\color{blue}{\mathit{Response}\\ 1:} To better understand the Language Expert’s contribution, we test whether its planning benefit can be recovered by giving its driving prompt directly to the World Expert. We keep the World Expert’s fixed caption and add the same scene-specific prompt used by the Language Expert. We evaluate this setting with the Language Expert retained or removed (✓: retained; ✗: removed):

Language ExpertWorld Expert textPDMS ↑
✓Caption90.8
✗Caption89.1
✗Caption + driving prompt89.7
✓Caption + driving prompt90.8

The first two rows show the full-model and Language-Expert removal results from Tab. 5. The last two rows report the new experiments with the driving prompt added to the World Expert.

Without the Language Expert, adding the driving prompt improves PDMS from 89.1 to 89.7, but remains below the full model’s 90.8. With the Language Expert retained, the added prompt gives no further improvement. The driving prompt alone does not recover the planning benefit of the Language Expert’s learned features.

Question2:\color{red}{\mathit{Question}\\ 2:} Could the authors quantitatively evaluate the quality of future scene prediction to better validate the effectiveness of the World Expert?

Response2:\color{blue}{\mathit{Response}\\ 2:} To evaluate future scene prediction quantitatively, we probe the World Expert for information about the future motion of surrounding objects. We train two MLP probes to predict their positions over 4 s, supervised by ground-truth future positions. We extract features at the World Expert’s future-latent positions just before the final Cross-Expert Attention block, using only past and current images as visual input. These are the "World features" used below.

We train the probes on navtrain and evaluate them on navtest. ADE@2s and ADE@4s measure the average position error over future time points up to 2 s and 4 s, respectively. FDE@4s measures the position error at 4 s. All errors are in meters; lower is better. Moving denotes objects with an observed speed above 1 m/s.

Probe 1: Direct prediction. This probe tests whether future object motion can be predicted from World features without an explicit motion history. The baseline uses only current position. The other two experiments add World features from the pretrained video model (before IL) or the fine-tuned World Expert (after IL). None receives explicit past positions or velocity.

Probe 1: inputADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Position only (baseline)2.3454.0347.0625.0138.71815.798
Position + World (before IL)1.9633.4086.0093.7896.61911.983
Position + World (after IL)1.9093.2855.7453.4586.01310.800

For moving objects, adding World features after IL reduces ADE@4s from 8.718 to 6.013 m and FDE@4s from 15.798 to 10.800 m compared with position alone. Both errors are also lower than the pretrained features(before IL). These results show that the fine-tuned World Expert retains useful information about future object motion.

AuthorsOpenReview ↗

Response to Reviewer f3Vq (2/2)

Probe 2: Prediction then correction. Unlike Probe 1, which does not explicitly use motion history, Probe 2 evaluates whether World features provide additional predictive information beyond past motion. We first train a baseline predictor using historical positions, then freeze its parameters. A separate MLP takes the current position and World features as input to predict a residual correction to the baseline trajectory. Only this correction MLP is trained. For the zero-World baseline, we use the same architecture and training procedure but replace the World features with zeros. This controlled comparison isolates the additional contribution of World features while keeping the motion-history predictor, correction network, and position input unchanged.

Probe 2: predictorADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Predictor Only (baseline)0.4571.0662.4020.7151.7954.310
Predictor + MLP (zero World features)0.4331.0262.3320.6981.7704.268
Predictor + MLP (World before IL)0.4290.9972.2430.6661.6393.901
Predictor + MLP (World after IL)0.4250.9862.2170.6551.5963.776

Compared with the zero-World experiment, World features after IL reduce moving-object ADE@4s from 1.770 to 1.596 m (9.8%) and FDE@4s from 4.268 to 3.776 m (11.5%). The World Expert provides useful future-motion information even when observed motion is already available.

Fig. 4 complements these quantitative results by visualizing the World Expert’s features after IL. Each row shows one scene; the columns show the front-view image, PCA maps of history latents 1 and 4, and PCA maps of future latents 1 and 2. The history maps represent observed frames, while the future maps cover successive 2 s intervals over the 4 s horizon. The future maps are predicted from Gaussian-initialized latents, not encoded from ground-truth future images. These feature maps exhibit structured spatial patterns that evolve across the predicted intervals, providing complementary evidence that the World Expert retains information relevant to future dynamics after IL.

Thank you for the response. The authors have addressed my concerns. I agree to accept it and will maintain my score at 3.

Reviewer_WfjTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveWLA, an autonomous driving framework that combines video-generation priors and vision-language features for trajectory planning. It contains three experts: a World Expert initialized from a pretrained video model, a Language Expert with a frozen VLM and a trainable adapter, and a flow-matching Action Expert. Through layer-wise Cross-Expert Attention, the World and Language Experts exchange information, while the Action Expert reads features from both without feeding its features back. These hidden features are directly used for planning, without generating future videos or text. The model is first trained through trajectory imitation learning, followed by Flow-GRPO refinement of the Action Expert with factorized advantage estimation and variance-based shrinkage. The authors report 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2. They also evaluate zero-shot transfer through closed-loop testing on HUGSIM and open-loop testing on nuScenes and Bench2Drive.

Critical strengths
  1. The paper presents a clear architecture that integrates World, Language, and Action Experts through layer-wise Cross-Expert Attention.
  2. The evaluation shows promising cross-domain generalization, covering NAVSIM, open-loop transfer to nuScenes and Bench2Drive, and closed-loop transfer to HUGSIM.
  3. The paper provides an effective training strategy supported by informative ablations. These ablations help clarify the effects of individual design choices and provide useful guidance for subsequent work.
Critical weaknesses
  1. First, I am not fully convinced by the contribution of the Language Expert. The World Expert receives four front-view images and text conditions, so it should already be capable of modeling rich scene information. Questions are: (1) Why is a VLM still needed to establish correspondences between image and text? (2) What exactly do the World Expert’s text conditions contain, and how do they differ from the “driving prompts” given to the Language Expert? (3) Why not include these driving prompts in the World Expert’s text conditions as well? (4) The improvement from 89.1 to 90.8 PDMS in the ablation study is not sufficient to establish the effectiveness of the Language Expert. Further clarification would help me better understand the distinct contribution of the Language Expert.
  2. Second, the World Expert performs only one forward pass and directly provides its hidden features to the Action Expert, without sampling or decoding for efficiency. After fine-tuning, it essentially acts as a visual encoder? I think the authors could conduct additional experiments, such as varying the number of sampling steps, removing the future noise latents, or even replacing the World Expert with a standard temporal visual encoder. It would also be useful to measure how well these features predict actual future motion.
  3. Please clarify the end-to-end wall-clock latency during the HUGSIM closed-loop evaluation and explain how it relates to the latency reported in Table 8?
Questions for authors

Please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer WfjT (1/3)

We thank the reviewer for the constructive feedback and for recognizing our architecture, generalization results, and ablation studies.

Question1:\color{red}{\mathit{Question}\\ 1:} Why is a VLM still needed to establish correspondences between image and text?

Response1:\color{blue}{\mathit{Response}\\ 1:} The Language Expert provides features learned through driving-oriented visual-language pretraining, while the World Expert is initialized from video pretraining. We use the VLM hidden states from the current image and driving prompt directly, without generating text. These features interact with World features through Cross-Expert Attention.

Question2:\color{red}{\mathit{Question}\\ 2:} What exactly do the World Expert’s text conditions contain, and how do they differ from the “driving prompts” given to the Language Expert?

Response2:\color{blue}{\mathit{Response}\\ 2:} The World Expert receives the same video-generation caption during training and inference:

A high-quality, photorealistic dashboard camera view of autonomous driving. Based on the past 2 seconds videos, predict and generate the next 4 seconds of realistic driving continuation, Maintain temporal consistency, stable camera perspective, natural motion flow without jitter or artifacts, clear details, and realistic physics.

The Language Expert receives the current front-view image and a scene-specific prompt containing four-step ego-motion history, the navigation command (TURN LEFT / GO STRAIGHT / TURN RIGHT), and the output format for eight (x, y, heading) waypoints. The fixed World caption contains no scene-specific navigation command.

Question3:\color{red}{\mathit{Question}\\ 3:} Why not include these driving prompts in the World Expert’s text conditions as well?

Response3:\color{blue}{\mathit{Response}\\ 3:} We additionally give the World Expert the scene-specific driving prompt, alongside its fixed caption, and test this with and without the Language Expert:

Language ExpertWorld textPDMS ↑
✓Caption90.8
✗Caption89.1
✗Caption + driving prompt89.7
✓Caption + driving prompt90.8

The caption-only rows are from Tab. 5; the prompt-added rows are additional experiments.

Without the Language Expert, adding the prompt improves PDMS from 89.1 to 89.7, but remains below the full model’s 90.8. With the Language Expert retained, it gives no further improvement. Providing the driving prompt to World does not recover the full benefit of the Language Expert in this comparison.

Question4:\color{red}{\mathit{Question}\\ 4:} What evidence beyond the improvement from 89.1 to 90.8 PDMS supports the Language Expert’s contribution?

Response4:\color{blue}{\mathit{Response}\\ 4:} Tab. 7 ablates World–Language interaction while keeping both branches present:

World–Language interactionPDMS ↑
No exchange88.4
World → Language90.1
Language → World89.8
Bidirectional90.8

The arrow indicates that the receiving expert attends to the source expert’s features. Disconnecting the branches gives 88.4 PDMS, whereas bidirectional interaction reaches 90.8. Exchanging World and Language features improves planning performance. The additional experiment in Response 3 supports the Language Expert’s contribution from another angle: giving the driving prompt to World does not recover the full model’s performance when Language is removed.

Question5:\color{red}{\mathit{Question}\\ 5:} After fine-tuning, does the World Expert essentially act as a visual encoder? Could you vary sampling steps, remove the future noise latents, or compare a temporal visual encoder? How well do these features predict actual future motion?

Response5:\color{blue}{\mathit{Response}\\ 5:} The World Expert supplies latent features without decoding future video; the sampling steps in Tab. 8 belong to Action trajectory generation. We test the contribution of future latents with inference-time interventions and a history-latent-only model.

We evaluate two additional inference-time ablations of the future latents. The full model and both ablations use the same imitation-learning (IL) checkpoint, matching the ten-step setting in Tab. 8.

AuthorsOpenReview ↗

Response to Reviewer WfjT (2/3)

In the full DriveWLA (IL), each Cross-Expert Attention block uses World features from four observed-frame latents and two Gaussian-initialized future latents. Zero future latents sets the future-latent features to zero before each block without changing token positions or attention structure. Remove future latents leaves only the four observed-frame latents in the World branch.

SettingNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
Full DriveWLA (IL)99.297.797.2100.084.090.8
Zero future latents99.397.397.8100.081.289.7 (−1.1)
Remove future latents98.293.693.2100.077.784.1 (−6.7)

PDMS drops by 1.1 points when the future-latent features are set to zero and by 6.7 points when the future latents are removed. These results show that the planner benefits from the future-latent pathway.

We also train a history-latent-only variant from the same video-pretrained backbone. It uses latents from the four observed frames during training and inference, with no future latents passed to Cross-Expert Attention. The Language and Action branches, trajectory objective, and parameter capacity are unchanged.

ModelNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
History-latent-only99.197.096.5100.083.389.8
Full DriveWLA (IL)99.297.797.2100.084.090.8

The history-latent-only variant achieves 89.8 PDMS, compared with 90.8 for the full model, and scores lower on DAC, TTC, and EP. These results indicate that future latents contribute to planning performance.

We train two MLP probes to predict the future positions of objects in the surrounding scene over 4 s, supervised by their ground-truth future positions. We use only past and current images as visual input to the World Expert and extract features at its future-latent positions just before the final Cross-Expert Attention block. These features serve as the World features in both probes.

We train the probes on navtrain and evaluate them on the navtest split. ADE@2s and ADE@4s measure the average position error over future time points up to 2 s and 4 s, respectively. FDE@4s measures the position error at 4 s. All errors are in meters; lower is better. Moving denotes objects with an observed speed above 1 m/s.

Probe 1: Direct prediction. The baseline uses only current position. The other two experiments in the table add World features from the pretrained video model (before IL) or the fine-tuned World Expert (after IL). None receives explicit past positions or velocity.

Probe 1: inputADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Position only (baseline)2.3454.0347.0625.0138.71815.798
Position + World (before IL)1.9633.4086.0093.7896.61911.983
Position + World (after IL)1.9093.2855.7453.4586.01310.800

On moving objects, IL fine-tuning reduces ADE@4s from 6.619 to 6.013 m and FDE@4s from 11.983 to 10.800 m. These results show that the World features retain future-motion information after IL.

Probe 2: Prediction then correction. We first train a motion predictor (baseline) on past positions and velocity, then freeze it. We separately train an MLP to correct its predictions using current position and World features. The correction is added to the predictor’s output. Only the MLP is updated during training. In the zero-World experiment, we train the same MLP with the World features set to zero. This separates the benefit of World features from that of the added MLP and current-position input.

Probe 2: predictorADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Predictor Only (baseline)0.4571.0662.4020.7151.7954.310
Predictor + MLP (zero World features)0.4331.0262.3320.6981.7704.268
Predictor + MLP (World before IL)0.4290.9972.2430.6661.6393.901
Predictor + MLP (World after IL)0.4250.9862.2170.6551.5963.776
AuthorsOpenReview ↗

Response to Reviewer WfjT (3/3)

Compared with the zero-World experiment, adding World features after IL reduces moving-object ADE@4s from 1.770 to 1.596 m (9.8%) and FDE@4s from 4.268 to 3.776 m (11.5%). World features improve prediction even when past motion is already available.

To illustrate the World Expert’s future prediction ability, Fig. 4 visualizes its features after IL fine-tuning. Each row shows one driving scene. From left to right, the columns show the front-view image, PCA maps of history latents 1 and 4, and PCA maps of future latents 1 and 2. The history maps represent observed frames; the two future maps represent successive 2 s intervals over the 4 s prediction horizon. The two future maps are predicted from Gaussian-initialized latents, not encoded from ground-truth future images. Their structured patterns differ from the history maps and change between the two future intervals. Together with the motion probes, these visualizations indicate that the World Expert retains the ability to model future scene evolution after IL.

Question6:\color{red}{\mathit{Question}\\ 6:} Please clarify the end-to-end wall-clock latency during the HUGSIM closed-loop evaluation and explain how it relates to the latency reported in Table 8.

Response6:\color{blue}{\mathit{Response}\\ 6:} Tab. 8 measures model inference on an NVIDIA A100, including the VLM forward pass, trajectory inference, and action decoding. However, a HUGSIM closed-loop cycle also requires 3DGS scene rendering and image preprocessing. Including these stages, we measured average end-to-end inference times of 1.61 s for five Action Euler steps and 1.73 s for ten during closed-loop evaluation.

We use synchronous evaluation for the original HUGSIM results in Tab. 3. The compared methods report the same setting. The simulator pauses while the model computes a trajectory.

We additionally evaluate asynchronous execution on all 436 HUGSIM scenarios. The simulator continues to advance and render frames during inference. Each new trajectory is applied as soon as it becomes available.

ExecutionStepsNC ↑DAC ↑TTC ↑Comf. ↑PDMS ↑RC ↑
Synchronous567.4793.7858.4748.9549.4257.25
Synchronous1065.8095.0659.2775.7854.9254.99
Asynchronous572.7293.6561.3418.8145.6041.29
Asynchronous1073.3093.2663.4841.2050.6542.70
ExecutionStepsEasy HD ↑Medium HD ↑Hard HD ↑Extreme HD ↑Overall HD ↑
Synchronous564.8944.8128.0517.6538.39
Synchronous1073.8747.9131.4115.8741.47
Asynchronous553.3033.2813.646.5426.31
Asynchronous1060.0036.9714.458.0329.40

Asynchronous execution lowers HD-Score by about 12 points in both settings. Route completion falls by 15.96 points with five steps and 12.29 with ten. Comfort also decreases, while NC and TTC increase. The ten-step planner achieves higher PDMS and HD-Score under both protocols.

Reviewer_mafQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DriveWLA combines a pretrained video World Expert, a frozen driving VLM with a trainable adapter, and a flow-matching Action Expert. Masked cross-expert attention allows bidirectional interaction between world and language features, while the action branch reads both without sending information back. During imitation learning, the trainable components are optimized through the trajectory objective. A subsequent Flow-GRPO stage updates action-branch LoRA parameters, using factorized reward advantages and variance shrinkage. The model reports 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, alongside zero-shot closed-loop results on HUGSIM and open-loop transfer to nuScenes and Bench2Drive. Ablations examine expert composition, information flow, training stages, and sampling cost.

Critical strengths
  1. The evaluation goes beyond a single benchmark score. HUGSIM provides an actual closed-loop transfer test across 436 scenarios, while the paper clearly identifies Bench2Drive as an open-loop evaluation here. The expert and attention-mask ablations also support the practical value of combining the two feature sources: removing world–language interaction lowers imitation-stage PDMS from 90.8 to 88.4. Reporting the imitation checkpoint separately from RL makes these architectural gains easier to assess.
Critical weaknesses
  1. Future-dynamics modeling is not directly established. Appendix A.1 confirms that the World Expert is fine-tuned only through trajectory loss, without a future-prediction objective. At inference, Gaussian future latents receive one forward pass, and Appendix D offers PCA maps rather than a quantitative test of predicted dynamics. Removing the entire expert also removes substantial pretrained capacity. Consequently, the results establish that the branch is useful, but do not distinguish explicit future modeling from a strong video feature extractor adapted for planning.

  2. The runtime implications for closed-loop driving are unresolved. Table 8 reports 970 ms for ten sampling steps and 810 ms for five, while one step sharply reduces PDMS to 49.2. This is a substantial observation-to-action delay. The HUGSIM protocol describes the controller and evaluation scenarios, but not whether measured inference time advances the simulated scene or how stale observations are handled. Without that information, the closed-loop scores do not show how the system behaves at its measured execution speed.

Questions for authors
  1. Can you compare the World Expert against a capacity-matched video encoder and a version without future-latent tokens? A quantitative probe of future agent motion or scene state before and after planning fine-tuning would also help establish what predictive information the branch retains.

  2. What hardware and timing assumptions underlie Table 8 and the HUGSIM rollouts? Could you evaluate the five- and ten-step planners with their measured computation delays inserted into the control loop, and report the resulting safety and route-completion tradeoffs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mafQ (1/2)

We thank the reviewer for the constructive feedback and positive assessment of our evaluations and ablation studies.

Question1:\color{red}{\mathit{Question}\\ 1:} Can you compare the World Expert against a capacity-matched video encoder and a version without future-latent tokens?

Response1:\color{blue}{\mathit{Response}\\ 1:} We evaluate two additional inference-time ablations of the future latents. The full model and both ablations use the same imitation-learning (IL) checkpoint, matching the ten-step setting in Tab. 8.

In the full DriveWLA (IL), each Cross-Expert Attention block uses World features from four observed-frame latents and two Gaussian-initialized future latents. Zero future latents sets the future-latent features to zero before each block without changing token positions or attention structure. Remove future latents leaves only the four observed-frame latents in the World branch.

SettingNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
Full DriveWLA (IL)99.297.797.2100.084.090.8
Zero future latents99.397.397.8100.081.289.7 (−1.1)
Remove future latents98.293.693.2100.077.784.1 (−6.7)

PDMS drops by 1.1 points when the future-latent features are set to zero and by 6.7 points when the future latents are removed. These results show that the planner benefits from the future-latent pathway.

We also train a history-latent-only variant from the same video-pretrained backbone. It uses latents from the four observed frames during training and inference, with no future latents passed to Cross-Expert Attention. The Language and Action branches, trajectory objective, and parameter capacity are unchanged.

ModelNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
History-latent-only99.197.096.5100.083.389.8 (−1.0)
Full DriveWLA (IL)99.297.797.2100.084.090.8

The history-latent-only variant achieves 89.8 PDMS, compared with 90.8 for the full model, and scores lower on DAC, TTC, and EP. These results indicate that future latents contribute to planning performance.

Question2:\color{red}{\mathit{Question}\\ 2:} A quantitative probe of future agent motion or scene state before and after planning fine-tuning would also help establish what predictive information the branch retains.

Response2:\color{blue}{\mathit{Response}\\ 2:} We train two MLP probes to predict the future positions of objects in the surrounding scene over 4 s, supervised by their ground-truth future positions. We use only past and current images as visual input to the World Expert and extract features at its future-latent positions just before the final Cross-Expert Attention block. These features serve as the World features in both probes.

We train the probes on navtrain and evaluate them on navtest. ADE@2s and ADE@4s measure the average position error over future time points up to 2 s and 4 s, respectively. FDE@4s measures the position error at 4 s. All errors are in meters; lower is better. Moving denotes objects with an observed speed above 1 m/s.

Probe 1: Direct prediction. The baseline uses only current position. The other two experiments in the table add World features from the pretrained video model (before IL) or the fine-tuned World Expert (after IL). None receives explicit past positions or velocity.

Probe 1: inputADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Position only (baseline)2.3454.0347.0625.0138.71815.798
Position + World (before IL)1.9633.4086.0093.7896.61911.983
Position + World (after IL)1.9093.2855.7453.4586.01310.800

On moving objects, IL fine-tuning reduces ADE@4s from 6.619 to 6.013 m and FDE@4s from 11.983 to 10.800 m. These results show that the World features retain future-motion information after IL.

AuthorsOpenReview ↗

Response to Reviewer mafQ (2/2)

Probe 2: Prediction then correction. We first train a motion predictor (baseline) on past positions and velocity, then freeze it. We separately train an MLP to correct its predictions using current position and World features. The correction is added to the predictor’s output. Only the MLP is updated during training. In the zero-World experiment, we train the same MLP with the World features set to zero. This separates the benefit of World features from that of the added MLP and current-position input.

Probe 2: predictorADE@2s ↓ADE@4s ↓FDE@4s ↓Moving ADE@2s ↓Moving ADE@4s ↓Moving FDE@4s ↓
Predictor Only (baseline)0.4571.0662.4020.7151.7954.310
Predictor + MLP (zero World features)0.4331.0262.3320.6981.7704.268
Predictor + MLP (World before IL)0.4290.9972.2430.6661.6393.901
Predictor + MLP (World after IL)0.4250.9862.2170.6551.5963.776

Compared with the zero-World experiment, adding World features after IL reduces moving-object ADE@4s from 1.770 to 1.596 m (9.8%) and FDE@4s from 4.268 to 3.776 m (11.5%). World features improve prediction even when past motion is already available.

To illustrate the World Expert’s future prediction ability, Fig. 4 visualizes its features after IL fine-tuning. Each row shows one driving scene. From left to right, the columns show the front-view image, PCA maps of history latents 1 and 4, and PCA maps of future latents 1 and 2. The history maps represent observed frames; the two future maps represent successive 2 s intervals over the 4 s prediction horizon. The two future maps are predicted from Gaussian-initialized latents, not encoded from ground-truth future images. Their structured patterns differ from the history maps and change between the two future intervals. Together with the motion probes, these visualizations indicate that the World Expert retains the ability to model future scene evolution after IL.

Question3:\color{red}{\mathit{Question}\\ 3:} What hardware and timing assumptions underlie Table 8 and the HUGSIM rollouts?

Response3:\color{blue}{\mathit{Response}\\ 3:} Tab. 8 measures model inference on an NVIDIA A100, including the VLM forward pass, trajectory inference, and action decoding. However, a HUGSIM closed-loop cycle also requires 3DGS scene rendering and image preprocessing. Including these stages, we measured average end-to-end inference times of 1.61 s for five Action Euler steps and 1.73 s for ten during closed-loop evaluation.

We use synchronous evaluation for the original HUGSIM results in Tab. 3. The compared methods report the same setting. The simulator pauses while the model computes a trajectory.

Question4:\color{red}{\mathit{Question}\\ 4:} Could you evaluate the five- and ten-step planners with their measured computation delays inserted into the control loop, and report the resulting safety and route-completion tradeoffs?

Response4:\color{blue}{\mathit{Response}\\ 4:} We additionally evaluate asynchronous execution on all 436 HUGSIM scenarios. The simulator continues to advance and render frames during inference. Each new trajectory is applied as soon as it becomes available.

ExecutionStepsNC ↑DAC ↑TTC ↑Comf. ↑PDMS ↑RC ↑
Synchronous567.4793.7858.4748.9549.4257.25
Synchronous1065.8095.0659.2775.7854.9254.99
Asynchronous572.7293.6561.3418.8145.6041.29
Asynchronous1073.3093.2663.4841.2050.6542.70
ExecutionStepsEasy HD ↑Medium HD ↑Hard HD ↑Extreme HD ↑Overall HD ↑
Synchronous564.8944.8128.0517.6538.39
Synchronous1073.8747.9131.4115.8741.47
Asynchronous553.3033.2813.646.5426.31
Asynchronous1060.0036.9714.458.0329.40

Asynchronous execution lowers HD-Score by about 12 points in both settings. Route completion falls by 15.96 points with five steps and 12.29 with ten. Comfort also decreases, while NC and TTC increase. The ten-step planner achieves higher PDMS and HD-Score under both protocols.

Comments on the paper

1 comment
Area_Chair_zFkCOpenReview ↗

Discussion Phase

Dear Reviewers, the authors have posted their rebuttal. Please review their responses and engage in the discussion (if you have not done so).

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.