acceptodds
Under review as a conference paper at ICLR 2027

WAM-Bench: A Closed-Loop Benchmark for Open-Ended Instruction Following in Autonomous Driving

Abstract

Recently, Vision-Language-Action (VLA) models have shown great potential in advancing end-to-end autonomous driving (E2E-AD). However, existing E2E-AD benchmarks typically provide only two predefined instructions (go straight and turn), leaving the ability of VLA models to follow human instructions largely unexplored. To address this limitation, we introduce WAM-Bench, a real-world closed-loop benchmark for evaluating open-ended instruction following in E2E-AD. It contains 37 hours of driving videos paired with human-reviewed open-ended instructions and covers 13 action classes. We further establish a comprehensive evaluation framework with representative VLA baselines, and propose an instruction-following metric (IF) for closed-loop evaluation. Extensive experiments across diverse settings, including the standard test split (with various action classes), long-horizon execution, changed instructions, and OOD instructions, reveal that open-ended instruction following remains challenging for current VLA models. The code is available at -.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous driving, instruction following, open-ended instruction, closed-loop benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_rdH2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WAM‑Bench is a real‑world closed‑loop benchmark built for evaluating open‑ended natural‑language instruction‑following for vision‑language‑action autonomous‑driving models. It introduces multiple test splits and a new IF metric, and conducts systematic evaluation over representative state‑of‑the‑art VLA baselines.

Critical strengths

This work presents a closed-loop autonomous driving benchmark for open-ended natural language instructions. Differing from conventional datasets that only support fixed one-hot action commands, it carries potential value for future research on instruction-driven autonomous driving.

Critical weaknesses
  1. It only supports single front‑camera rendering without surround multi‑camera inputs, whereas real‑world end‑to‑end autonomous driving relies on surround‑view perception. Rendered images generated by the world model OmniDreams suffer from noise and hallucinations, and closed‑loop errors accumulate continuously. As a result, closed‑loop simulation results cannot reliably equate to real‑world performance, which greatly undermines the credibility of this closed‑loop benchmark.

  2. Key maneuvers such as U‑Turn and Pull‑into have extremely few samples; the test set contains merely 5 U‑turn segments. The overall test split comprises only 233 clips, leading to insufficient statistical confidence. Several fine‑grained action subclasses suffer from tiny sample sizes, so evaluation results are vulnerable to interference from individual samples.

  3. Weak baseline comparisons with inadequate SOTA benchmarks. The paper mainly compares Qwen3.5 and Alpamayo‑1.5, lacking comprehensive comparisons against mainstream recent driving‑oriented VLA models including Hybrid‑Flow Drive and DriveVLM. No systematic leaderboard for open‑source models is established on this benchmark, and its practical utility remains insufficiently demonstrated.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

We sincerely thank Reviewer rdH2 for your careful reading and positive comments, for recognizing the value of our closed-loop benchmark for open-ended natural-language instructions beyond fixed one-hot action commands, and its potential to support future research on instruction-driven autonomous driving. We also appreciate the insightful suggestions. We have conducted all the suggested experiments and will incorporate them into the revised manuscript.

Due to a 5,000-word limit per comment, we will provide our response in multiple comment. We have summarized your comments, where "W1" stands for "weakness 1."



W1: Reliability of Closed-Loop Evaluation.

We acknowledge the limitation of single front-camera observations and discuss it in Appendix F. We adopt OmniDreams because, to the best of our knowledge as of the end of October, we have not identified a better-performing video world model on the PAI-AV dataset. In future work, we plan to design a multi-view video world model to address this limitation.

To assess the reliability of rendered observations, we conduct an ablation study on the WAM-Bench long-horizon split. For each video segment, observations are rendered continuously from the beginning along the GT ego trajectory to account for potential accumulated rendering errors. We make a single trajectory prediction at each of three time points: 10, 7.5, and 5 seconds before the segment ends. At each time point, we provide either GT (real) or rendered image history at the same GT ego poses, while keeping all other inputs identical.

The results show comparable performance with real and rendered observations, with no consistent degradation across the three prediction times. IF@0m changes from 68.3 to 64.3, from 61.9 to 66.7, and remains at 63.5, respectively. Rendered observations also match or outperform real observations on several planning metrics. These results suggest that the difficulty of long-horizon planning and instruction following is a more substantial factor than rendering artifacts under this controlled setting, supporting the reliability of rendered observations.

Prediction TimeInputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
10s before the endGT Image68.376.28178.796.896.892.1
Rendered Image64.368.379.486.996.898.491
Prediction TimeInputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
7.5s before the endGT Image61.97384.18296.810092.3
Rendered Image66.78185.683.695.210095.1
Prediction TimeInputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
5s before the endGT Image63.581.585.285.285.710093.8
Rendered Image63.576.284.186.484.110095.5

Table 10. Comparison of real and rendered observations on the WAM-Bench long-horizon split. A single trajectory prediction is made at each of three time points: 10s, 7.5s, and 5s before the video segment ends. Rendered observations are generated continuously from the beginning of each segment along the GT ego trajectory. All other model inputs are identical.

AuthorsOpenReview ↗

Response to Weakness 2

W2: Few samples in test (u-turn, pull over) and OOD (pull into, slight turn).

In the updated Table 4, we expand the U-turn and pull-over categories in the test split to 20 samples each, bringing the total to 260 samples with 20 per action class. U-turn remains the most challenging category for Alpamayo-1.5, with an IF@0m score of 20.0. This difficulty may stem from the high-curvature trajectories and nonlinear vehicle dynamics involved in U-turns. Improving the VLA model for such maneuvers is an important direction for future work.

ActionNumIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Stop20859090958595-
Continue208090909510010098.8
Follow2080859010010010097.8
Overtake207075851001009599.1
Pull Over20656590-7510093.7
Lane Change2060758595959599.2
Ramp20606580100959096.3
Go Straight20556070959510095.4
Merge20556570701009094.9
Roundabout20555570759010095
Start Moving2055606595859591.6
Turn2035505080509085.5
U-Turn2020202070659542.2
All26059.665.873.589.287.395.890.8

Table 4. Performance across different action classes on the WAM-Bench test split, sorted by IF@0m in descending order.

In the updated Table 6, we expand OOD instruction split to 40 samples, with 20 each for slight turn and pull into. Alpamayo-1.5 achieves an IF@0m score of 35.0, indicating a certain degree of generalization to unseen instruction patterns.

ActionNumIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Pull Into20204040-10010064.3
Slight Turn20507070808010095.7
All40355555809010081.8

Table 6. Performance across different action classes on the WAM-Bench OOD instructions split.

AuthorsOpenReview ↗

Response to Weakness 3

W3: Baseline coverage.

We have expanded the test split from 233 to 260 samples, with 20 samples per action class. We update the results in Table 3 and the corresponding references throughout the manuscript accordingly.

In the updated Table 3, we compare 10 SOTA methods, including three open-source VLA paradigms: AR-based methods (Qwen3.5 series, OpenDriveVLA, and AutoVLA), diffusion-based methods (DiffusionDrive and WAM-Diff), and AR-diffusion hybrid methods (RecogDrive and Alpamayo-1.5). Additionally, we benchmarks three closed-source commercial models: GLM-5.3, DeepSeek-V4.1-Flash-1017, and GPT-6 Luna. For Alpamayo-1.5, FT denotes training from scratch, while Oracle denotes fine-tuning from NVIDIA’s official pretrained weights. Experimental results show that fine-tuned Alpamayo-1.5 outperforms the other zero-shot and fine-tuned baselines in instruction following, highlighting the effectiveness of task-specific training. Nevertheless, existing open-source VLA models still face significant challenges in following open-ended instructions.

Finally, we have been unable to identify the work titled “Hybrid-Flow Drive.” We would appreciate a specific reference, such as an arXiv ID or publication venue, to help us locate it and assess its relevance for comparison.

MethodOpenSetupIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Human100100100100100100100
Constant Velocity19.726.628.852.972.110076.7
Predefined Instruction
RecogDrive-8B✓FT10.815.323.530.967.88472.4
DiffusionDrive✓FT18.921.526.241.381.595.784.6
Open-ended Instruction
Deepseek-V4.1 Flash✗ZS12.915.516.332.967.42757.8
GLM-5.3✗ZS13.318.220.83470.758.850.4
Alpamayo-1.5✓ZS14.61821.539.17682.875.9
GPT-6 Luna✗ZS1520.623.232.970.866.569.1
Open-ended Instruction
OpenDriveVLA✓FT22.729.632.236.678.492.387.8
AutoVLA✓FT25.831.334.339.181.581.583.7
RecogDrive-8B✓FT26.236.942.143.17059.284.3
WAM-Diff-8B✓FT27.534.338.250.768.778.581.8
Qwen3.5-2B✓FT31.640.446.970.879.197.588.6
Qwen3.5-4B✓FT39.845.951.966.382.292.689.8
Alpamayo-1.5✓FT46.45561.867.279.496.791.1
Alpamayo-1.5✓Oracle59.665.873.589.287.395.890.8

Table 3. Quantitative comparison of planning models on the WAM-Bench test split. Predefined Instruction: methods using one-hot encoded action classes as instructions. Abbreviation: Open (open-source), ZS (zero-shot), FT (fine-tuning). Oracle: fine-tuned from official weights pretrained on NVIDIA’s internal data. The best results for each metric are highlighted.

Reviewer_RxeUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WAM-Bench, a closed-loop benchmark for testing whether autonomous driving models can follow natural language instructions. It contains 37 hours of real driving data with manually checked instructions covering 13 types of driving actions. The authors build a simulation framework and propose a metric that checks whether the vehicle reaches the endpoint of a reference trajectory. They evaluate several models, including Qwen3.5 and Alpamayo-1.5, on standard tasks, longer driving sequences, changed instructions, and unseen actions. The results show that the tested models still struggle to follow instructions consistently, especially over longer sequences and when the instructions change.

Critical strengths
  • The motivation makes sense. I agree that limiting language inputs to predefined commands essentially degrades VLA models into VA models.
  • The experimental results are detailed, and the analysis covers many aspects, including different actions, long driving sequences, changed instructions, and OOD instructions.
  • Nice writing and clear figures. The paper is easy to follow.
Critical weaknesses
  • Is this really the “first” real-world closed-loop benchmark for open-ended instruction following in autonomous driving? As far as I know, there are some earlier papers on this topic: [1] Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles, CVPR 2026. [2] Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous and Instruction-guided Driving, CVPR 2024.
  • Why does the open-ended instruction comparison only include Qwen3.5 and Alpamayo-1.5? More model families would make the results more convincing, including closed-source API models, other open-source general VLMs, and models trained for driving.
  • Some test sets are quite small. There are only 5 U-turn cases and 10 OOD cases, with just one pull-into case. It is hard to draw strong conclusions about these abilities from so few examples.
Questions for authors
  • WAM-Bench or WAM-Bench2? Figure 1 uses a different name from the rest of the paper. Please make the naming consistent.
  • The IF metric needs more explanation. It mainly checks whether the vehicle reaches the reference endpoint, but reaching the endpoint does not necessarily mean following the full instruction. Please clarify how it handles cases such as taking the wrong route or reaching the target without stopping.
5 comments
AuthorsOpenReview ↗

Response to Weakness 1-2

We sincerely thank Reviewer RxeU for your constructive feedback and for recognizing our motivation for open-ended instructions, comprehensive experimental analysis, and clear presentation. Below, we address your concerns and questions point by point.

Due to a 5,000-word limit per comment, we will provide our response in multiple comment. We have summarized your comments, where "W1" stands for "weakness 1."



W1: Is this really the “first” real-world closed-loop benchmark for open-ended instruction following in autonomous driving? As far as I know, there are some earlier papers on this topic: [1] Open-Ended Instruction Realization with LLM-Enabled Multi-Planner Scheduling in Autonomous Vehicles, CVPR 2026. [2] Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous and Instruction-guided Driving, CVPR 2024.

Thank you for highlighting these important works [1][2]. We will include them in Related Work and clarify the differences. However, these works [1][2] use only language instructions and structured scene information, such as maps and vehicle states, as inputs, without visual observations.

In contrast, WAM-Bench provides visual observations throughout closed-loop evaluation, requiring VLAs to ground open-ended instructions in visible objects and locations. In addition, WAM-Bench also provides systematic evaluation across 13 action classes and 21 subclasses, with dedicated test, long-horizon, changed-instruction, and OOD instruction splits.

Therefore, we refine our claim to “the first visual, real-world, closed-loop benchmark for open-ended instruction following in autonomous driving.”



W2: Baseline coverage.

We have expanded the test split from 233 to 260 samples, with 20 samples per action class. We update the results in Table 3 and the corresponding references throughout the manuscript accordingly.

In the updated Table 3, we compare 10 SOTA methods, including three open-source VLA paradigms: AR-based methods (Qwen3.5 series, OpenDriveVLA, and AutoVLA), diffusion-based methods (DiffusionDrive and WAM-Diff), and AR-diffusion hybrid methods (RecogDrive and Alpamayo-1.5). Additionally, we benchmarks three closed-source commercial models: GLM-5.3, DeepSeek-V4.1-Flash-1017, and GPT-6 Luna. For Alpamayo-1.5, FT denotes training from scratch, while Oracle denotes fine-tuning from NVIDIA’s official pretrained weights. Experimental results show that fine-tuned Alpamayo-1.5 outperforms the other zero-shot and fine-tuned baselines in instruction following, highlighting the effectiveness of task-specific training. Nevertheless, existing open-source VLA models still face significant challenges in following open-ended instructions.

MethodOpenSetupIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Human100100100100100100100
Constant Velocity19.726.628.852.972.110076.7
Predefined Instruction
RecogDrive-8B✓FT10.815.323.530.967.88472.4
DiffusionDrive✓FT18.921.526.241.381.595.784.6
Open-ended Instruction
Deepseek-V4.1 Flash✗ZS12.915.516.332.967.42757.8
GLM-5.3✗ZS13.318.220.83470.758.850.4
Alpamayo-1.5✓ZS14.61821.539.17682.875.9
GPT-6 Luna✗ZS1520.623.232.970.866.569.1
Open-ended Instruction
OpenDriveVLA✓FT22.729.632.236.678.492.387.8
AutoVLA✓FT25.831.334.339.181.581.583.7
RecogDrive-8B✓FT26.236.942.143.17059.284.3
WAM-Diff-8B✓FT27.534.338.250.768.778.581.8
Qwen3.5-2B✓FT31.640.446.970.879.197.588.6
Qwen3.5-4B✓FT39.845.951.966.382.292.689.8
Alpamayo-1.5✓FT46.45561.867.279.496.791.1
Alpamayo-1.5✓Oracle59.665.873.589.287.395.890.8

Table 3. Quantitative comparison of planning models on the WAM-Bench test split. Predefined Instruction: methods using one-hot encoded action classes as instructions. Abbreviation: Open (open-source), ZS (zero-shot), FT (fine-tuning). Oracle: fine-tuned from official weights pretrained on NVIDIA’s internal data. The best results for each metric are highlighted.

AuthorsOpenReview ↗

Response to Weakness 3 and Questions

W3: Few samples in OOD instruction split.

In the updated Table 6, we expand OOD instruction split to 40 samples, with 20 each for slight turn and pull into. Alpamayo-1.5 achieves an IF@0m score of 35.0, indicating a certain degree of generalization to unseen instruction patterns.

ActionNumIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Pull Into20204040-10010064.3
Slight Turn20507070808010095.7
All40355555809010081.8

Table 6. Performance across different action classes on the WAM-Bench OOD instructions split.



Q1: WAM-Bench or WAM-Bench2?

It is WAM-Bench. The figure contained a typo, which we have corrected in the updated teaser (Figure 1).



Q2: Please clarify how the IF metric handles cases such as taking the wrong route or reaching the target without stopping.

For instructions requiring a stationary final state, such as stop, pull into, and pull over, we check both the final predicted trajectory BBox against the GT endpoint BBox and the final ego speed. IF = 1 only when the vehicle reaches the target and satisfies the stopping criterion (ego-vehicle speed is less than the threshold). Merely passing through the target without stopping does not count as success.

For instructions without a stopping requirement, such as go straight. IF = 1 if any predicted trajectory BBox overlaps the GT endpoint BBox. Passing the endpoint is acceptable because stopping is not requested.

Taking an alternative route to the same endpoint is constrained by the short evaluation horizon and road topology. Taking a wrong route typically causes the vehicle to leave the drivable area or collide, resulting in DAC = 0 or NC = 0, respectively. For long-horizon episodes, we compute IF metric for each segment using its corresponding endpoint and multiply the segment-level scores. Thus, reaching the final endpoint alone is insufficient.

Dear Reviewer,

Thank you again for your thoughtful feedback and for recognizing our motivation for open-ended instructions, detailed experimental analysis, and clear writing and figures. We have addressed your concerns in our responses above and incorporated the corresponding clarifications and additional results into the revised manuscript.

As the discussion deadline is approaching, we would appreciate it if you could review our responses and let us know whether any concerns remain. If the revisions have resolved your concerns, we would be grateful if you could update your assessment and score accordingly.

We remain available to provide any further clarification. Thank you for your time and consideration.

Best regards,

WAM-Bench author team

Reviewer_6XVwOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces WAM-Bench, a closed-loop benchmark for evaluating how well autonomous driving models follow open-ended language instructions. The dataset contains 37 hours of real driving videos paired with human-reviewed instructions, covering 13 action classes and 21 subclasses. During evaluation, a VLA model predicts a trajectory from the instruction, historical images, and ego states. The simulator executes the predicted motion, and newly rendered observations are fed back to the model so its actions influence subsequent predictions.

The paper also proposes an instruction-following metric based on reaching a reference endpoint and reports planning metrics such as collision avoidance, drivable-area compliance, and comfort. The authors evaluate several baselines, including Qwen3.5 and Alpamayo-1.5, across standard, long-horizon, changed-instruction, and unseen-action settings. Alpamayo-1.5 performs best on most metrics on the standard test split, but performance drops on the more challenging splits, suggesting that consistently executing open-ended driving instructions remains difficult.

Critical strengths

The closed-loop evaluation grounded in real driving scenes is a great strength. Feeding rendered observations back to the policy allows the benchmark to expose accumulated errors and assess whether a model can recover from its own actions. This is also a useful contribution for other researchers to use in this field since there seems to be an intent to release the code.

Another strength is the diversity of driving instructions, covering 13 action classes and 21 subclasses beyond basic commands such as going straight or turning. The manual review of instructions and segment boundaries, along with checks that referenced objects and locations are observable in the historical inputs, also helps improve the quality of the dataset. This provides a useful resource for studying more specific driving commands that involve both an action and a target in the scene.

Also, reporting different metrics such as collision avoidance, drivable-area compliance, and comfort alongside instruction-following scores provides a broader assessment of execution quality and where strengths and weaknesses may lie in the current pipeline.

Critical weaknesses

The proposed instruction-following metric in Equation 1 seems to measure whether the vehicle reaches a reference endpoint during execution more than whether it satisfies all parts of the instruction. It checks the minimum bounding-box distance to the reference endpoint across time, with a final-position check for stationary actions such as stopping and pulling over. However, reaching that location does not necessarily mean the requested behavior was followed along the way as there is also the scene context and the ego’s history. Therefore, a model could potentially reach the reference endpoint without relying on language. The changed-instruction split and same-scene qualitative examples are valuable controls, but an evaluation with instructions removed or shuffled would help establish how much performance depends on language.

Section 5.1 describes training Alpamayo-1.5 on WAM-Bench, but the paper does not report performance of the original pretrained checkpoint under the same evaluation protocol. It would have been great to include this comparison to quantify the contribution of task-specific training, particularly on the changed-instruction split. This would help establish whether training improved the model’s ability to execute alternative commands in the same scene.

In Section 5.3, the comparison between 62.2% on the full test split and 40.0% on the changed-instruction split may combine differences in scene selection and requested maneuvers, since the full test split has 233 video segments and the changed-instruction split has only 40. If this is correct, then it would be ideal to also report performance under the original instructions for those same 40 segments. Although the paper provides qualitative same-scene comparisons, this paired quantitative comparison would help establish how performance changes when the instruction is modified while the initial scene and ego history are held fixed.

Since one of the bigger contributions was WAM-Bench, it would be great for a quality check on the generated observations during closed-loop evaluation, particularly for the long-horizon split. Since rendered images are repeatedly fed back to the model, unrealistic or inconsistent scenes could affect the model’s predictions. It would be helpful to understand whether the drop in long-horizon performance comes from the policy’s difficulty following instructions over time, errors accumulating in the generated observations, or a combination of both.

Questions for authors

For the changed-instruction split, how was the new trajectory verified after the modified instructions were given from the initial scene and ego state? The paper mentions collecting corresponding reference trajectories using an external steering wheel but were these trajectories checked for feasibility, safety, instruction compliance, and how were their endpoints selected?

6 comments
AuthorsOpenReview ↗

Response to Weakness 1-3

We sincerely thank Reviewer 6XVw for the positive assessment of our benchmark's real-world closed-loop design, instruction diversity, and comprehensive metrics, as well as for the insightful suggestions. We have conducted all the suggested experiments and will incorporate them into the revised manuscript.

Due to a 5,000-word limit per comment, we will provide our response in multiple comment. We have summarized your comments, where "W1" stands for "weakness 1."



W1: Evaluation with instructions removed or shuffled.

In the updated Table 9, we add evaluations with instructions removed or shuffled. The endpoint for each segment remains the same across all instruction settings. We also expand the test split from 233 to 260 samples, with 20 samples per action class, and update all reported results accordingly.

Experimental results show that Alpamayo-1.5 achieves the highest instruction-following (IF) scores with the original open-ended instruction, followed by instructions removed and then shuffled. Specifically, IF@0m drops from 59.6 to 49.2 when instructions are removed, and further to 27.9 when they are shuffled. These results indicate that the model uses language to guide trajectory prediction: removing instructions reduces performance. Providing an incorrect instruction causes the ego vehicle to follow it toward its associated endpoint rather than the endpoint of the original instruction. This supports the model’s dependence on language.

InputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Error (Shuffled)27.935.140.672.478.994.881.9
None (Removed)49.259.265.881.785.894.387.7
Open-ended59.665.873.589.287.395.890.8

Table 9. Ablation study of instruction inputs on the WAM-Bench test split. Error: randomly shuffle instruction. None: remove instruction. Open-ended: open-ended instruction.



W2: Performance of the original pretrained Alpamayo-1.5 across different data splits.

In the updated Table 5, we report zero-shot (ZS) results for the original pretrained Alpamayo-1.5 checkpoint across different data splits. We also report results after fine-tuning this checkpoint on WAM-Bench under the same evaluation protocol. Task-specific training substantially improves performance, particularly on the changed-instruction split, where IF@0m increases from 15.0 to 40.0. This comparison quantifies the contribution of task-specific training and shows that training improves the model’s ability to execute alternative commands in the same scene.

SplitSetupIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
TestZS14.61821.539.17682.875.9
Oracle59.665.873.589.287.395.890.8
Long-HorizonZS9.59.59.521.350.871.463
Oracle31.736.547.683.68193.789.3
Changed InstructionsZS1517.527.547.2759073
Oracle4042.552.578.977.59083.1
OOD InstructionsZS10101033.37010064.2
Oracle355555809010081.8

Table 5. Quantitative comparison on different data splits of WAM-Bench. ZS: zero-shot inference using official weights pretrained on NVIDIA’s internal data. Oracle: fine-tuned from official weights pretrained on NVIDIA’s internal data.



W3: Report performance under the original instructions for 40 segments (changed instructions).

The quantitative comparison of Alpamayo-1.5 on the changed-instruction split and the corresponding samples with their original instructions (test split), as shown in the table below. The results show that with changed instructions, IF@0m and DAC decreases from 50.0 to 40.0 and 82.6 to 78.9, respectively. This indicates that instruction following remains a challenging task for the model.

SplitNumIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Test (Original Instructions)405057.56082.672.58085.4
Change Instructions404042.552.578.977.59083.1
AuthorsOpenReview ↗

Response to Weakness 4 and Questions

W4: The drop in long-horizon performance comes from the policy’s difficulty following instructions over time, errors accumulating in the generated observations, or a combination of both.

To assess the reliability of rendered observations, we conduct an ablation study on the WAM-Bench long-horizon split. For each video segment, observations are rendered continuously from the beginning along the GT ego trajectory to account for potential accumulated rendering errors. We make a single trajectory prediction at each of three time points: 10, 7.5, and 5 seconds before the segment ends. At each time point, we provide either GT (real) or rendered image history at the same GT ego poses, while keeping all other inputs identical.

The results show comparable performance with real and rendered observations, with no consistent degradation across the three prediction times. IF@0m changes from 68.3 to 64.3, from 61.9 to 66.7, and remains at 63.5, respectively. Rendered observations also match or outperform real observations on several planning metrics. These results suggest that the difficulty of long-horizon planning and instruction following is a more substantial factor than rendering artifacts under this controlled setting, supporting the reliability of rendered observations.

Prediction TimeInputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
10s before the endGT Image68.376.28178.796.896.892.1
Rendered Image64.368.379.486.996.898.491
Prediction TimeInputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
7.5s before the endGT Image61.97384.18296.810092.3
Rendered Image66.78185.683.695.210095.1
Prediction TimeInputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
5s before the endGT Image63.581.585.285.285.710093.8
Rendered Image63.576.284.186.484.110095.5

Table 10. Comparison of real and rendered observations on the WAM-Bench long-horizon split. A single trajectory prediction is made at each of three time points: 10s, 7.5s, and 5s before the video segment ends. Rendered observations are generated continuously from the beginning of each segment along the GT ego trajectory. All other model inputs are identical.



Q1: For the changed-instruction split, how was the new trajectory verified after the modified instructions were given from the initial scene and ego state? The paper mentions collecting corresponding reference trajectories using an external steering wheel but were these trajectories checked for feasibility, safety, instruction compliance, and how were their endpoints selected?

All reference trajectories in the changed-instruction split undergo two-stage verification for feasibility, safety, and instruction compliance. First, we compute the corresponding metrics to ensure that each trajectory achieves IF = DAC = NC = CF = 1, indicating instruction compliance, drivable-area compliance, collision avoidance, and comfort, respectively.

Then, we manually review both BEV and ego front-view videos to confirm that the vehicle follows traffic rules and the trajectory fully matches the modified instruction. We select the final frame, after the maneuver is fully completed and the vehicle’s pose has stabilized, as the reference endpoint.

SplitMethodIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Changed InstructionsHuman100100100100100100100

Dear Reviewer,

Thank you again for your insightful suggestions, and for the positive assessment of our benchmark's real-world closed-loop design, instruction diversity, and comprehensive metrics. We have addressed your concerns in our responses above and incorporated the corresponding clarifications and additional results into the revised manuscript.

As the discussion deadline is approaching, we would appreciate it if you could review our responses and let us know whether any concerns remain. If the revisions have resolved your concerns, we would be grateful if you could update your assessment and score accordingly.

We remain available to provide any further clarification. Thank you for your time and consideration.

Best regards,

WAM-Bench author team

Reviewer_hbDDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WAM-Bench provides a simulation benchmark to assess VLAs' instruction-following capabilities in closed-loop, end-to-end autonomous driving. The benchmark comprises 37 hours (36 for training and 1 for testing) of diverse driving scenarios, 13 action classes, and 21 subclasses.

The authors also propose an instruction-following evaluation metric. The metric evaluates the instruction-following ability based on prediction's proximity to the endpoint bounding box of the Ground-Truth trajectory.

Critical strengths

This paper proposed a useful closed-loop benchmark for open-ended instruction following. The authors curate real driving scenarios paired with open-ended instructions through data filtering, automated annotation, VLM-based segment refinement, and human review. The manual review checks instruction accuracy, segment boundaries, causality, and ambiguity, providing a valuable quality-control step. Combined with coverage of 13 action classes and 21 subclasses, the closed-loop evaluation framework provides a useful resource for assessing whether VLAs can execute language instructions over time and identifying their limitations across different driving maneuvers.

Critical weaknesses

W1. Limited baseline coverage and missing training details. The conclusion that open-ended instruction following remains challenging for current VLA models is based on only two open-ended model families. Qwen3.5, a general-purpose VLM, is evaluated at two sizes, while Alpamayo-1.5 is the only driving-specific model assessed with open-ended instructions, despite the paper referencing several other driving VLAs that are not evaluated (e.g., OpenDriveVLA, AutoVLA, WAM-Diff). These findings highlight limitations in the evaluated models but offer limited insight into VLA approaches more broadly. Additionally, the paper does not provide training configurations for Qwen3.5, DiffusionDrive, or RecogDrive. For Alpamayo-1.5, only the trained modules, number of epochs, and hardware are reported, with no information on objectives or optimization settings. This lack of detail affects the validity of reported comparisons. For instance, RecogDrive performs worse than constant velocity on IF@0m (10.8 vs. 19.7) and DAC (30.9 vs. 52.9), which may indicate an adaptation issue rather than a limitation of one-hot instructions. Without comprehensive training details, the results are difficult to reproduce, and it remains unclear to what extent the performance gap is attributable to model capability versus training and adaptation choices.

W2. Insufficient characterization of open-ended and free-form instructions. Although the paper emphasizes open-ended, free-form human instructions, it does not clearly define the scope of language diversity encompassed by these terms. Instructions are initially generated by Qwen and subsequently reviewed by humans, which introduces uncertainty regarding their fidelity to naturally expressed human requests. The appendix presents statistics on instruction length and distributions of referenced objects and locations, indicating that most instructions pertain to maneuver-level tasks; for example, Fig. 11(b) shows “vehicle” at 15.2% and all other objects at 4.3% or less. However, beyond these statistics, the benchmark does not assess robustness to paraphrasing, novel combinations of familiar requirements, or changes in the referenced target within the same maneuver. The paper also does not evaluate whether models perform similarly when provided only with the action-subclass label. The changed-instruction split partially addresses this issue but includes only 40 segments, and its examples alter the maneuver (Fig. 8) rather than the referent. Additionally, the out-of-distribution (OOD) evaluation primarily examines unseen action categories with only ten examples, offering limited evidence for broader language generalization. Consequently, the paper does not yet show that WAM-Bench measures capabilities beyond maneuver-category recognition, which requires a more precise definition of “open-ended” and targeted evaluations that isolate language variations to substantiate this claim.

Questions for authors

Q1: On the test split, how do Alpamayo-1.5 and Qwen3.5-4B score when the instruction is (a) removed, (b) replaced by its action-subclass name (e.g.,Turn left), or (c) paraphrased?

Q2: In manual review, what fraction of Qwen-generated instructions were edited or discarded, by how many annotators, and with what agreement? Were any instructions written from scratch by humans?

Q3: How closely does the IF metric agree with human judgments of successful instruction execution?

Q4: Long-horizon scoring: For concatenated episodes containing multiple instructions, how are instruction switching and IF aggregation implemented? Must each instruction be completed in the correct order, or can reaching the final endpoint suffice?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1.1

We sincerely thank Reviewer hbDD for the thorough review and constructive feedback, and for recognizing the value of our closed-loop benchmark, data quality control, and instruction-following metric. Following your suggestions, we have expanded our baseline evaluations, added comprehensive training details, and conducted targeted ablations on open-ended instructions and metric validity. Below, we address each of your comments in detail.


Due to a 5,000-word limit per comment, we will provide our response in multiple comment. We have summarized your comments, where "W1" stands for "weakness 1."


W1.1 Baseline coverage.

We have expanded the test split from 233 to 260 samples, with 20 samples per action class. We update the results in Table 3 and the corresponding references throughout the manuscript accordingly.

In the updated Table 3, we compare 10 SOTA methods, including three open-source VLA paradigms: AR-based methods (Qwen3.5 series, OpenDriveVLA, and AutoVLA), diffusion-based methods (DiffusionDrive and WAM-Diff), and AR-diffusion hybrid methods (RecogDrive and Alpamayo-1.5). Additionally, we benchmarks three closed-source commercial models: GLM-5.3, DeepSeek-V4.1-Flash-1017, and GPT-6 Luna. For Alpamayo-1.5, FT denotes training from scratch, while Oracle denotes fine-tuning from NVIDIA’s official pretrained weights. Experimental results indicate that existing open-source VLA models still face significant challenges in instruction following.

MethodOpenSetupIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Human100100100100100100100
Constant Velocity19.726.628.852.972.110076.7
Predefined Instruction
RecogDrive-8B✓FT10.815.323.530.967.88472.4
DiffusionDrive✓FT18.921.526.241.381.595.784.6
Open-ended Instruction
Deepseek-V4.1 Flash✗ZS12.915.516.332.967.42757.8
GLM-5.3✗ZS13.318.220.83470.758.850.4
Alpamayo-1.5✓ZS14.61821.539.17682.875.9
GPT-6 Luna✗ZS1520.623.232.970.866.569.1
Open-ended Instruction
OpenDriveVLA✓FT22.729.632.236.678.492.387.8
AutoVLA✓FT25.831.334.339.181.581.583.7
RecogDrive-8B✓FT26.236.942.143.17059.284.3
WAM-Diff-8B✓FT27.534.338.250.768.778.581.8
Qwen3.5-2B✓FT31.640.446.970.879.197.588.6
Qwen3.5-4B✓FT39.845.951.966.382.292.689.8
Alpamayo-1.5✓FT46.45561.867.279.496.791.1
Alpamayo-1.5✓Oracle59.665.873.589.287.395.890.8

Table 3. Quantitative comparison of planning models on the WAM-Bench test split. Predefined Instruction: methods using one-hot encoded action classes as instructions. Abbreviation: Open (open-source), ZS (zero-shot), FT (fine-tuning). Oracle: fine-tuned from official weights pretrained on NVIDIA’s internal data. The best results for each metric are highlighted.

AuthorsOpenReview ↗

Response to Weakness 1.2-1.3

W1.2 Training configurations

In the updated Appendix E (Table 11), we report the training configurations for all models (Alpamayo-1.5, Qwen3.5 series, DiffusionDrive, RecogDrive, OpenDriveVLA, AutoVLA and WAM-Diff). All unlisted hyper-parameters follow the default settings in their respective GitHub repositories.

ModelTrainable ModulesOptimizerLearning RateEpochsBatch SizeWeight DecayWarmupSchedulerObjective
ReCogDrive-8BStage 1: VLMAdamW1e-431280.0510%CosineCE
ReCogDrive-8BStage 2: Action Expert (diffusion)AdamW1e-42001281e-43 epochsCosineNoise prediction MSE
DiffusionDriveDiffusion ModelAdamW6e-41001281e-43 epochsCosineTrajectory L1 + Focal Loss
OpenDriveVLAStage 1: ProjectorAdamW1e-4325603%CosineCE
OpenDriveVLAStage 2: VLMAdamW2e-53320.0510%CosineCE
AutoVLAVLMAdamW2e-53320.01500 stepsStep decayCE
WAM-Diff-8BVLMAdamW1e-536402%CosineCE
Qwen3.5-2BVLMAdamW2e-5101280.12%CosineCE
Qwen3.5-4BVLMAdamW2e-5101280.12%CosineCE
Alpamayo-1.5Stage 1: VLMAdamW1e-55320500 stepsCosineCE
Alpamayo-1.5Stage 2: Action Expert (flow matching)AdamW1e-43320500 stepsCosineFlow matching MSE

Table 11. Training configurations of models fine-tuned on WAM-Bench. Abbreviations: LR (learning rate), BSZ (global batch size), CE (cross-entropy), MSE (mean squared error).


W1.3 RecogDrive (predefined instruction) vs.RecogDrive (open-ended instruction)

We follow your suggestion of the one-hot embedding limitation and change the setting. In the updated Table 3, we add the result for RecogDrive trained with open-ended instruction. The training configuration for RecogDrive follows its official settings, as detailed in W1.2. The results show that RecogDrive performs better when trained using open-ended instruction compared to when it is trained using predefined instruction.

MethodOpenSetupIF@0m ↑IF@1m ↑DAC ↑NC ↑CF ↑EP ↑
Predefined Instruction
RecogDrive-8B✓FT10.815.330.967.88472.4
Open-ended Instruction
RecogDrive-8B✓FT26.236.943.17059.284.3

Table 3. Quantitative comparison of planning models on the WAM-Bench test split. Predefined Instruction: methods using one-hot encoded action classes as instructions. Abbreviation: Open (open-source), FT (fine-tuning).

AuthorsOpenReview ↗

Response to Weakness 2

W2.1 Definition of "open-ended and free-form instructions".

“Open-ended” instructions go beyond predefined instructions. They can specify target objects, locations, or combinations of driving actions. “Free-form” means that instructions use natural language without fixed sentence templates. The same intent can be expressed in different ways. Predefined instructions specify only the action subclass. For example, “pull over” does not specify which vehicle to stop behind. An open-ended instruction can be “Pull over and stop behind the first white vehicle on the right.”



W2.2 Evaluation of changes in the referenced target within the same maneuver.

As shown in the table below, we report results on the changed-instruction split grouped by the type of modification. Of the 40 samples, 26 modify the referenced target while keeping the maneuver unchanged: 13 change the referenced object and 13 change the referenced location. The remaining 14 modify only the action. Alpamayo-1.5 achieves IF@0m scores of 31.7, 43.2, and 44.7, respectively. These results show that the split evaluates target grounding beyond maneuver-category recognition, with changes to the referenced object posing the greatest challenge.

Modification TypeNumIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Reference Object1331.737.649.680.676.910076.8
Reference Location1343.244.153.585.384.684.687.6
Only Action1444.745.654.371.471.485.783.8
All404042.552.578.977.59083.1


W2.3 Action-subclass Label vs. Novel Combination vs. Paraphrased Instruction vs. Open-ended Instruction

In the updated Table 9, we have added ablation studies for Alpamayo-1.5 under various instruction settings. The endpoint for each segment remains the same across all instruction settings. The results show the overall trend: Open-ended ≈ Paraphrased > Novel Combination > Predefined (action-subclass) >None > Error. That is, paraphrased instructions yield performance comparable to the original open-ended instructions, while using only action-subclass labels reduces instruction-following performance. Novel combinations yield lower performance than the original open-ended instructions. Removing instructions leads to a further decline, while providing erroneous instructions produces the lowest IF scores—even lower than providing no instructions. These results suggest that Alpamayo-1.5 uses language to guide trajectory prediction and is relatively robust to paraphrasing.

InputIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Error27.935.140.672.478.994.881.9
None49.259.265.881.785.894.387.7
Predefined54.760.871.280.886.293.589
Novel Combination56.861.268.384.185.49589.7
Paraphrased57.764.672.989.287.795.390
Open-ended59.665.873.589.287.395.890.8

Table 9. Ablation study of instruction inputs on the WAM-Bench test split. Error: random erroneous instruction. None: remove instruction. Predefined: action subclass only. Novel Combination: combining two consecutive instructions into a single instruction. Paraphrase: paraphrasing of open-ended instruction. Open-ended: open-ended instruction.



W2.4 Few samples in OOD instruction split.

In the updated Table 6, we expand OOD instruction split to 40 samples, with 20 each for slight turn and pull into. Alpamayo-1.5 achieves an IF@0m score of 35.0, indicating a certain degree of generalization to unseen instruction patterns.

ActionNumIF@0m ↑IF@1m ↑IF@2m ↑DAC ↑NC ↑CF ↑EP ↑
Pull Into20204040-10010064.3
Slight Turn20507070808010095.7
All40355555809010081.8

Table 6. Performance across different action classes on the WAM-Bench OOD instructions split.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion Phase

Dear Area Chair,

We are very grateful to you for overseeing our submission. We sincerely thank all the reviewers for their time and constructive feedback on WAM-Bench. As the discussion period ends, we summarize their assessments and the key evidence added during the rebuttal.

Reviewer Status

  • 6XVw (Rating 4 / Clear Acceptance). Confirmed that their concerns were addressed.
  • RxeU (Rating 3 / Weak Acceptance). Confirmed that main concerns were addressed.
  • hbDD (Rating 3 / Weak Acceptance). Confirmed that main concerns were addressed.
  • rdH2 (Rating 3 / Weak Acceptance). No follow-up feedback is available.

All four reviewers recommend acceptance, and all three who provided follow-up feedback raised their ratings.

Strengths by Reviewers

  • Motivation and benchmark value. "The motivation makes sense" (RxeU). "The first real-world, closed-loop driving benchmark for open-ended instruction following" (6XVw, hbDD, RxeU, rdH2). "Recognized for WAM-Bench evaluation grounded in real driving scenes" (6XVw, hbDD, RxeU, rdH2) and its “potential value for future research on instruction-driven autonomous driving” (rdH2).
  • Instruction diversity and quality control. “The diversity of driving instructions, covering 13 action classes and 21 subclasses” (6XVw, hbDD, RxeU), supported by manual review that provides “a valuable quality-control step” (hbDD).
  • Experiments, metrics, and presentation. “The experimental results are detailed” and “Nice writing and clear figures” (RxeU). Complementary metrics provide “a broader assessment of execution quality” (6XVw, rdH2).

Concerns Addressed

  • Expanded test and OOD splits (rdH2, RxeU, hbDD). Expanded the test split from 233 to 260 clips, with 20 per action class, and the OOD split from 10 to 40 clips, with 20 each for Pull Into and Slight Turn (Tables 4 and 6).

  • Baseline coverage and training configurations (rdH2, RxeU, hbDD, 6XVw). Added 10 SOTA methods covering AR-based, diffusion-based, and hybrid methods, along with three commercial models. Reported training configurations, RecogDrive results with open-ended instructions, and zero-shot pretrained Alpamayo-1.5 results across all four splits (Tables 3, 5, and 11).

  • Instruction ablations and target grounding (6XVw, hbDD). IF@0m drops from 59.6 to 49.2 when instructions are removed and further to 27.9 when shuffled. Added action-subclass and paraphrase ablations, paired original-versus-changed instruction results, and 26 cases that modify the referenced target while keeping the maneuver unchanged.

  • IF metric validation (RxeU, hbDD, 6XVw). IF@0m agrees with majority human judgment on 258 of 260 samples (99.2%). Clarified stopping-speed checks, multiplicative segment-level IF, and complementary DAC/NC metrics.

  • Rendered-observation reliability (6XVw, rdH2). Real and continuously rendered observations yield comparable performance (Table 10), suggesting that long-horizon planning and instruction following are more substantial factors than rendering artifacts under this controlled setting.

  • Annotation, trajectory verification, and related work (hbDD, 6XVw, RxeU). Reported annotation statistics and manual-review procedures, detailed two-stage reference-trajectory verification, and refined our novelty claim through comparisons with prior works.

Thank you very much for considering the reviewers’ positive assessments and the progress made during the rebuttal.

Sincerely,

The WAM-Bench Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.