- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WAM‑Bench is a real‑world closed‑loop benchmark built for evaluating open‑ended natural‑language instruction‑following for vision‑language‑action autonomous‑driving models. It introduces multiple test splits and a new IF metric, and conducts systematic evaluation over representative state‑of‑the‑art VLA baselines.
Critical strengths
This work presents a closed-loop autonomous driving benchmark for open-ended natural language instructions. Differing from conventional datasets that only support fixed one-hot action commands, it carries potential value for future research on instruction-driven autonomous driving.
Critical weaknesses
-
It only supports single front‑camera rendering without surround multi‑camera inputs, whereas real‑world end‑to‑end autonomous driving relies on surround‑view perception. Rendered images generated by the world model OmniDreams suffer from noise and hallucinations, and closed‑loop errors accumulate continuously. As a result, closed‑loop simulation results cannot reliably equate to real‑world performance, which greatly undermines the credibility of this closed‑loop benchmark.
-
Key maneuvers such as U‑Turn and Pull‑into have extremely few samples; the test set contains merely 5 U‑turn segments. The overall test split comprises only 233 clips, leading to insufficient statistical confidence. Several fine‑grained action subclasses suffer from tiny sample sizes, so evaluation results are vulnerable to interference from individual samples.
-
Weak baseline comparisons with inadequate SOTA benchmarks. The paper mainly compares Qwen3.5 and Alpamayo‑1.5, lacking comprehensive comparisons against mainstream recent driving‑oriented VLA models including Hybrid‑Flow Drive and DriveVLM. No systematic leaderboard for open‑source models is established on this benchmark, and its practical utility remains insufficiently demonstrated.
Questions for authors
See Weaknesses.
5 comments
Response to Weakness 2
W2: Few samples in test (u-turn, pull over) and OOD (pull into, slight turn).
In the updated Table 4, we expand the U-turn and pull-over categories in the test split to 20 samples each, bringing the total to 260 samples with 20 per action class. U-turn remains the most challenging category for Alpamayo-1.5, with an IF@0m score of 20.0. This difficulty may stem from the high-curvature trajectories and nonlinear vehicle dynamics involved in U-turns. Improving the VLA model for such maneuvers is an important direction for future work.
| Action | Num | IF@0m ↑ | IF@1m ↑ | IF@2m ↑ | DAC ↑ | NC ↑ | CF ↑ | EP ↑ |
|---|---|---|---|---|---|---|---|---|
| Stop | 20 | 85 | 90 | 90 | 95 | 85 | 95 | - |
| Continue | 20 | 80 | 90 | 90 | 95 | 100 | 100 | 98.8 |
| Follow | 20 | 80 | 85 | 90 | 100 | 100 | 100 | 97.8 |
| Overtake | 20 | 70 | 75 | 85 | 100 | 100 | 95 | 99.1 |
| Pull Over | 20 | 65 | 65 | 90 | - | 75 | 100 | 93.7 |
| Lane Change | 20 | 60 | 75 | 85 | 95 | 95 | 95 | 99.2 |
| Ramp | 20 | 60 | 65 | 80 | 100 | 95 | 90 | 96.3 |
| Go Straight | 20 | 55 | 60 | 70 | 95 | 95 | 100 | 95.4 |
| Merge | 20 | 55 | 65 | 70 | 70 | 100 | 90 | 94.9 |
| Roundabout | 20 | 55 | 55 | 70 | 75 | 90 | 100 | 95 |
| Start Moving | 20 | 55 | 60 | 65 | 95 | 85 | 95 | 91.6 |
| Turn | 20 | 35 | 50 | 50 | 80 | 50 | 90 | 85.5 |
| U-Turn | 20 | 20 | 20 | 20 | 70 | 65 | 95 | 42.2 |
| All | 260 | 59.6 | 65.8 | 73.5 | 89.2 | 87.3 | 95.8 | 90.8 |
Table 4. Performance across different action classes on the WAM-Bench test split, sorted by IF@0m in descending order.
In the updated Table 6, we expand OOD instruction split to 40 samples, with 20 each for slight turn and pull into. Alpamayo-1.5 achieves an IF@0m score of 35.0, indicating a certain degree of generalization to unseen instruction patterns.
| Action | Num | IF@0m ↑ | IF@1m ↑ | IF@2m ↑ | DAC ↑ | NC ↑ | CF ↑ | EP ↑ |
|---|---|---|---|---|---|---|---|---|
| Pull Into | 20 | 20 | 40 | 40 | - | 100 | 100 | 64.3 |
| Slight Turn | 20 | 50 | 70 | 70 | 80 | 80 | 100 | 95.7 |
| All | 40 | 35 | 55 | 55 | 80 | 90 | 100 | 81.8 |
Table 6. Performance across different action classes on the WAM-Bench OOD instructions split.
Response to Weakness 3
W3: Baseline coverage.
We have expanded the test split from 233 to 260 samples, with 20 samples per action class. We update the results in Table 3 and the corresponding references throughout the manuscript accordingly.
In the updated Table 3, we compare 10 SOTA methods, including three open-source VLA paradigms: AR-based methods (Qwen3.5 series, OpenDriveVLA, and AutoVLA), diffusion-based methods (DiffusionDrive and WAM-Diff), and AR-diffusion hybrid methods (RecogDrive and Alpamayo-1.5). Additionally, we benchmarks three closed-source commercial models: GLM-5.3, DeepSeek-V4.1-Flash-1017, and GPT-6 Luna. For Alpamayo-1.5, FT denotes training from scratch, while Oracle denotes fine-tuning from NVIDIA’s official pretrained weights. Experimental results show that fine-tuned Alpamayo-1.5 outperforms the other zero-shot and fine-tuned baselines in instruction following, highlighting the effectiveness of task-specific training. Nevertheless, existing open-source VLA models still face significant challenges in following open-ended instructions.
Finally, we have been unable to identify the work titled “Hybrid-Flow Drive.” We would appreciate a specific reference, such as an arXiv ID or publication venue, to help us locate it and assess its relevance for comparison.
| Method | Open | Setup | IF@0m ↑ | IF@1m ↑ | IF@2m ↑ | DAC ↑ | NC ↑ | CF ↑ | EP ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Human | 100 | 100 | 100 | 100 | 100 | 100 | 100 | ||
| Constant Velocity | 19.7 | 26.6 | 28.8 | 52.9 | 72.1 | 100 | 76.7 | ||
| Predefined Instruction | |||||||||
| RecogDrive-8B | ✓ | FT | 10.8 | 15.3 | 23.5 | 30.9 | 67.8 | 84 | 72.4 |
| DiffusionDrive | ✓ | FT | 18.9 | 21.5 | 26.2 | 41.3 | 81.5 | 95.7 | 84.6 |
| Open-ended Instruction | |||||||||
| Deepseek-V4.1 Flash | ✗ | ZS | 12.9 | 15.5 | 16.3 | 32.9 | 67.4 | 27 | 57.8 |
| GLM-5.3 | ✗ | ZS | 13.3 | 18.2 | 20.8 | 34 | 70.7 | 58.8 | 50.4 |
| Alpamayo-1.5 | ✓ | ZS | 14.6 | 18 | 21.5 | 39.1 | 76 | 82.8 | 75.9 |
| GPT-6 Luna | ✗ | ZS | 15 | 20.6 | 23.2 | 32.9 | 70.8 | 66.5 | 69.1 |
| Open-ended Instruction | |||||||||
| OpenDriveVLA | ✓ | FT | 22.7 | 29.6 | 32.2 | 36.6 | 78.4 | 92.3 | 87.8 |
| AutoVLA | ✓ | FT | 25.8 | 31.3 | 34.3 | 39.1 | 81.5 | 81.5 | 83.7 |
| RecogDrive-8B | ✓ | FT | 26.2 | 36.9 | 42.1 | 43.1 | 70 | 59.2 | 84.3 |
| WAM-Diff-8B | ✓ | FT | 27.5 | 34.3 | 38.2 | 50.7 | 68.7 | 78.5 | 81.8 |
| Qwen3.5-2B | ✓ | FT | 31.6 | 40.4 | 46.9 | 70.8 | 79.1 | 97.5 | 88.6 |
| Qwen3.5-4B | ✓ | FT | 39.8 | 45.9 | 51.9 | 66.3 | 82.2 | 92.6 | 89.8 |
| Alpamayo-1.5 | ✓ | FT | 46.4 | 55 | 61.8 | 67.2 | 79.4 | 96.7 | 91.1 |
| Alpamayo-1.5 | ✓ | Oracle | 59.6 | 65.8 | 73.5 | 89.2 | 87.3 | 95.8 | 90.8 |
Table 3. Quantitative comparison of planning models on the WAM-Bench test split. Predefined Instruction: methods using one-hot encoded action classes as instructions. Abbreviation: Open (open-source), ZS (zero-shot), FT (fine-tuning). Oracle: fine-tuned from official weights pretrained on NVIDIA’s internal data. The best results for each metric are highlighted.
Response to Weakness 1
We sincerely thank Reviewer rdH2 for your careful reading and positive comments, for recognizing the value of our closed-loop benchmark for open-ended natural-language instructions beyond fixed one-hot action commands, and its potential to support future research on instruction-driven autonomous driving. We also appreciate the insightful suggestions. We have conducted all the suggested experiments and will incorporate them into the revised manuscript.
Due to a 5,000-word limit per comment, we will provide our response in multiple comment. We have summarized your comments, where "W1" stands for "weakness 1."
W1: Reliability of Closed-Loop Evaluation.
We acknowledge the limitation of single front-camera observations and discuss it in Appendix F. We adopt OmniDreams because, to the best of our knowledge as of the end of October, we have not identified a better-performing video world model on the PAI-AV dataset. In future work, we plan to design a multi-view video world model to address this limitation.
To assess the reliability of rendered observations, we conduct an ablation study on the WAM-Bench long-horizon split. For each video segment, observations are rendered continuously from the beginning along the GT ego trajectory to account for potential accumulated rendering errors. We make a single trajectory prediction at each of three time points: 10, 7.5, and 5 seconds before the segment ends. At each time point, we provide either GT (real) or rendered image history at the same GT ego poses, while keeping all other inputs identical.
The results show comparable performance with real and rendered observations, with no consistent degradation across the three prediction times. IF@0m changes from 68.3 to 64.3, from 61.9 to 66.7, and remains at 63.5, respectively. Rendered observations also match or outperform real observations on several planning metrics. These results suggest that the difficulty of long-horizon planning and instruction following is a more substantial factor than rendering artifacts under this controlled setting, supporting the reliability of rendered observations.
Table 10. Comparison of real and rendered observations on the WAM-Bench long-horizon split. A single trajectory prediction is made at each of three time points: 10s, 7.5s, and 5s before the video segment ends. Rendered observations are generated continuously from the beginning of each segment along the GT ego trajectory. All other model inputs are identical.