- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DriveWLA, a driving framework that combines a video-pretrained World Expert, a vision-language model with a trainable adapter, and an Action Expert for trajectory generation. Cross-Expert Attention allows the World and Language branches to exchange information while the Action branch uses their representations to predict motion. The model is trained through trajectory imitation and then refined with Flow-GRPO using separately normalized driving rewards and a variance-based adjustment to the policy update. Experiments report 92.3 PDMS on NAVSIM v1, 90.1 EPDMS on NAVSIM v2, and a zero-shot HD-Score of 41.47 across 436 closed-loop HUGSIM scenarios, with additional open-loop transfer results on nuScenes and Bench2Drive.
Critical strengths
-
The expert and attention ablations provide controlled evidence that combining the World and Language representations improves planning. Removing the World Expert reduces PDMS from 90.8 to 86.2, removing the Language Expert reduces it to 89.1, and disabling information exchange between them reduces it to 88.4 (Tables 5 and 7). These comparisons support both the value of the two branches and the additional benefit of allowing them to interact, strengthening the contribution beyond the performance of the complete system alone.
-
The paper evaluates the learned policy in closed-loop HUGSIM scenarios, where its actions affect the observations it subsequently receives. Across 436 scenarios from four source datasets, DriveWLA achieves an HD-Score of 41.47 compared with 32.50 for the strongest listed baseline, while also improving time-to-collision and route-completion scores (Table 3). This evaluation provides evidence of driving capability beyond matching recorded trajectories, and Appendix B.3 describes the shared controller and evaluation protocol while acknowledging differences in camera inputs.
Critical weaknesses
-
The experiments show that including the World Expert improves trajectory planning. What remains unclear is whether this improvement comes from predicting future scene dynamics, as the paper argues, or from providing additional visual features learned during video pretraining. The model is fine-tuned using only trajectory supervision, and removing the entire World Expert also removes its capacity and pretrained features, so this ablation does not distinguish these explanations. Comparing against a branch of similar capacity that processes only observed frames, and changing or removing the future-latent tokens, would help establish whether predictive dynamics contribute to the resulting trajectories.
-
The paper lacks of video illustrations and visualizations of the open-loop and closed-loop results, especially in closed-loop scenarios it would be interesting to see how the proposed method deal with visual artifact caused by HUGSIM and whether the artifacts would affect the ablation conclusions.
Questions for authors
-
Please clarify the weakness 2 setting, and if the authors could provide more visualizations of closed-loop behavior improvements (like having less compounding errors than other baselines/recover from failure, etc.)
-
How do you establish that the World Expert improves planning through predicted future dynamics rather than additional pretrained visual features? Have you compared it with a branch of similar capacity using only observed frames, or modified the future-latent tokens and measured how the trajectory changes? Please also clarify whether Figure 4 visualizes features inferred from observed inputs and noise or features encoded from ground-truth future frames, given the description in Appendix D.
2 comments
Response to Reviewer yQjY (2/2)
We train the probes on navtrain and evaluate them on the navtest split. ADE@2s and ADE@4s measure the average position error over future time points up to 2 s and 4 s, respectively. FDE@4s measures the position error at 4 s. All errors are in meters; lower is better. Moving denotes objects with an observed speed above 1 m/s.
Probe 1: Direct prediction. The baseline uses only current position. The other two experiments in the table add World features from the pretrained video model (before IL) or the fine-tuned World Expert (after IL). None receives explicit past positions or velocity.
| Probe 1: input | ADE@2s ↓ | ADE@4s ↓ | FDE@4s ↓ | Moving ADE@2s ↓ | Moving ADE@4s ↓ | Moving FDE@4s ↓ |
|---|---|---|---|---|---|---|
| Position only (baseline) | 2.345 | 4.034 | 7.062 | 5.013 | 8.718 | 15.798 |
| Position + World (before IL) | 1.963 | 3.408 | 6.009 | 3.789 | 6.619 | 11.983 |
| Position + World (after IL) | 1.909 | 3.285 | 5.745 | 3.458 | 6.013 | 10.800 |
On moving objects, IL fine-tuning reduces ADE@4s from 6.619 to 6.013 m and FDE@4s from 11.983 to 10.800 m. These results show that the World features retain future-motion information after IL.
Probe 2: Prediction then correction. We first train a motion predictor (baseline) on past positions and velocity, then freeze it. We separately train an MLP to correct its predictions using current position and World features. The correction is added to the predictor’s output. Only the MLP is updated during training. In the zero-World experiment, we train the same MLP with the World features set to zero. This separates the benefit of World features from that of the added MLP and current-position input.
| Probe 2: predictor | ADE@2s ↓ | ADE@4s ↓ | FDE@4s ↓ | Moving ADE@2s ↓ | Moving ADE@4s ↓ | Moving FDE@4s ↓ |
|---|---|---|---|---|---|---|
| Predictor Only (baseline) | 0.457 | 1.066 | 2.402 | 0.715 | 1.795 | 4.310 |
| Predictor + MLP (zero World features) | 0.433 | 1.026 | 2.332 | 0.698 | 1.770 | 4.268 |
| Predictor + MLP (World before IL) | 0.429 | 0.997 | 2.243 | 0.666 | 1.639 | 3.901 |
| Predictor + MLP (World after IL) | 0.425 | 0.986 | 2.217 | 0.655 | 1.596 | 3.776 |
Compared with the zero-World experiment, adding World features after IL reduces moving-object ADE@4s from 1.770 to 1.596 m (9.8%) and FDE@4s from 4.268 to 3.776 m (11.5%). World features improve prediction even when past motion is already available.
To illustrate the World Expert’s future prediction ability, Fig. 4 visualizes its features after IL fine-tuning. Each row shows one driving scene. From left to right, the columns show the front-view image, PCA maps of history latents 1 and 4, and PCA maps of future latents 1 and 2. The history maps represent observed frames; the two future maps represent successive 2 s intervals over the 4 s prediction horizon. The two future maps are predicted from Gaussian-initialized latents, not encoded from ground-truth future images. Their structured patterns differ from the history maps and change between the two future intervals. Together with the motion probes, these visualizations indicate that the World Expert retains the ability to model future scene evolution after IL.
Does Figure 4 visualize features inferred from observed inputs and noise, or features encoded from ground-truth future frames?
The future features in Fig. 4 are predicted by the IL-fine-tuned World Expert, not encoded from ground-truth future images. The model receives past and current images and Gaussian-initialized future latents. The last two columns visualize the resulting future features for successive 2 s intervals. Their structured patterns evolve over these intervals, providing qualitative support for retained future modeling alongside the quantitative probes above. We will make the prediction process explicit in the caption and Appendix D.
Response to Reviewer yQjY (1/2)
We thank the reviewer for the constructive feedback and for recognizing our expert ablations and closed-loop evaluation.
Question1: Could the authors provide more visualizations of closed-loop behavior, such as recovery from failure or less compounding error?
Response1: With our unified World–Language–Action architecture, DriveWLA achieves the highest overall HD-Score and route completion among the methods in Tab. 3. It also leads on Hard and Extreme HD-Score in the zero-shot HUGSIM evaluation:
Compared with DrivoR, DriveWLA improves overall HD-Score by 8.97 points and route completion by 7.79 points. In closed-loop evaluation, we also observe cases where the model corrects its trajectory or recovers from difficult situations. We will add corresponding visualizations to the updated paper PDF to show these situations.
Question2: How does the method deal with visual artifacts caused by HUGSIM, and could these artifacts affect the ablation conclusions?
Response2: We evaluate DriveWLA zero-shot on HUGSIM’s rendered camera images. Visual artifacts may affect the policy’s observations and closed-loop behavior. However, our expert and information-flow ablations are conducted on NAVSIM (Tabs. 5 and 7) and do not use HUGSIM rendering, so their conclusions are unaffected by these artifacts. We will include representative rendered frames with planned-trajectory overlays in the updated paper PDF to make these cases clearer.
Question3: Have you compared the World Expert with a branch of similar capacity using only observed frames, or modified the future-latent tokens and measured how the trajectory changes?
Response3: We evaluate two additional inference-time ablations of the future latents. The full model and both ablations use the same imitation-learning (IL) checkpoint, matching the ten-step setting in Tab. 8.
In the full DriveWLA (IL), each Cross-Expert Attention block uses World features from four observed-frame latents and two Gaussian-initialized future latents. Zero future latents sets the future-latent features to zero before each block without changing token positions or attention structure. Remove future latents leaves only the four observed-frame latents in the World branch.
PDMS drops by 1.1 points when the future-latent features are set to zero and by 6.7 points when the future latents are removed. These results show that the planner benefits from the future-latent pathway.
We also train a history-latent-only variant from the same video-pretrained backbone. It uses latents from the four observed frames during training and inference, with no future latents passed to Cross-Expert Attention. The Language and Action branches, trajectory objective, and parameter capacity are unchanged.
The history-latent-only variant achieves 89.8 PDMS, compared with 90.8 for the full model, and scores lower on DAC, TTC, and EP. These results indicate that future latents contribute to planning performance.
Question4: How do you establish that the World Expert retains future-dynamics information rather than only additional pretrained visual features?
Response4: We train two MLP probes to predict the future positions of objects in the surrounding scene over 4 s, supervised by their ground-truth future positions. We use only past and current images as visual input to the World Expert and extract features at its future-latent positions just before the final Cross-Expert Attention block. These features serve as the World features in both probes.