- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Language2Drive, a benchmark for evaluating instruction following in autonomous-driving vision-language-action (VLA) models. Unlike conventional driving benchmarks, which primarily score safety, route completion, and driving quality, Language2Drive centers on the Paired Counterfactual Unit (PCU): the same scene is held fixed while only the instruction changes. The authors report Strict Instruction Success Rate (SISR) for individual requests and Counterfactual Paired Success (CFPS) for matched alternatives. Across closed-loop CARLA, reconstructed AlpaSim scenes, and open-loop tests on nuScenes, NAVSIM, and WOD-E2E, they find a consistent gap: strong driving performance does not imply reliable language control. Most successful rollouts are one-sided—the model completes one preferred behavior but fails the other feasible instruction, even when safety and driving scores remain high.
Critical strengths
-
The paper identifies a genuine evaluation risk: high driving scores can overstate instruction-following ability, with direct implications for language-conditioned driving systems.
-
This paper introduces PCU. By comparing SISR and CFPS, it explicitly differentiates isolated task success from true language‑driven behavioral selection.
-
The authors build extended CARLA‑Language2Drive with six purpose‑designed maps, curated AlpaSim real‑world reconstructed scenes, plus open‑loop evaluation pipelines on three public driving datasets. Tasks span four complexity levels from simple atomic maneuvers to complex conditional long‑horizon program instructions. They also provide paraphrase robustness tests to check consistency across semantically equivalent wording.
-
Extensive closed‑loop and open‑loop experiments across multiple state‑of‑the‑art driving VLA models empirically demonstrate the prevalence of one‑sided completion failure. Analyses on paraphrase sensitivity, long‑horizon program failure and safety‑conflict cases provide rich diagnostic insights.
Critical weaknesses
-
Most of the core evaluations are conducted in simulation, using CARLA and reconstructed scenes from AlpaSim. Although the authors additionally use real-world logs for open-loop evaluation, there is no on-vehicle, closed-loop evaluation in real-world driving scenarios. As a result, it remains unclear whether the reported improvements transfer to realistic perception noise, sensor imperfections, and vehicle dynamics. Existing benchmarks such as NAVSIM and NeuroNCAP could provide complementary evaluations on more realistic driving scenarios.
-
Reproducing the full benchmark appears to require custom CARLA maps as well as private evaluation sidecars and specialized evaluator assets. This substantially increases the setup and engineering burden for other research groups, making it difficult to independently reproduce the reported results and adopt Language2Drive as a standard benchmark.
-
The paper provides a thorough characterization of observed failures, including one-sided completion, paraphrase sensitivity, and failures in long-horizon reasoning. However, it provides limited controlled ablations to determine why these failures occur. In particular, the current experiments do not clearly disentangle errors arising from perception, language understanding, planning, and low-level controller tracking.
-
The paper asks whether language determines which feasible behavior a vehicle executes, and the PCU protocol evaluates matched execution of two mutually exclusive contracts under a shared initialization. However, Appendix A.1 acknowledges that an instruction-invariant policy with uncoupled randomness can still achieve an expected CFPS of up to 25% while completely ignoring the language instruction. The paper further notes that instruction-ablated repetitions would provide an additional test of causal language dependence, but such an experiment does not appear to be included. Therefore, the abstract's claim of “weak language control” seems stronger than what the current experimental design can causally establish. At present, the results demonstrate a gap in matched instruction-conditioned behavior, but do not cleanly establish that the gap is caused by insufficient language conditioning.
-
Section 3.1 compares native driving stacks (e.g., LMDrive, SimLingo, and Alpamayo) with generic VLM/VLA bridges (e.g., Qwen-Drive, OneVL, and Impromptu), but these systems do not necessarily share the same camera configuration, control interface, or low-level controller. In particular, the generic bridges use a different camera rig and a pure-pursuit-style tracker with a configured 1 Hz invocation interval. The Senna CARLA results further introduce a “no Senna-E2E planner” configuration, while the RealSim experiments exclude the official Senna and LMDrive implementations. Consequently, failures may arise from differences in sensing, tracking, interfaces, or system integration rather than from the VLA's inability to follow language instructions. This makes the broad conclusion that “current driving VLAs themselves lack reliable instruction following” difficult to identify from the current comparisons.
-
The paper states that SISR and CFPS are the primary measures of complete execution, yet their detailed definitions, interpretation, and empirical analysis receive limited space in the nine-page main paper. Given that these metrics underpin the central claims, the paper would benefit from a more explicit explanation of their design, failure modes, and relationship to conventional driving metrics.
-
The statement that “93.2% of successful rollouts lack a successful opposite-instruction counterpart” is a particularly important quantitative claim because it directly supports the paper's argument about weak language control. However, the source, calculation procedure, and statistical basis of this number should be made explicit in the main text rather than leaving readers to infer its derivation from the appendix or supplementary material.
-
A substantial portion of the paper's text appears to rely heavily on LLM-generated descriptions and analyses.
Questions for authors
- Does the benchmark actually establish that language causes the choice of driving behavior, rather than merely correlating with successful execution?
- Are the observed failures genuinely caused by VLA language understanding, or could they arise from perception, planning, tracking, sensor configuration, or system integration?
- Do the conclusions hold in realistic real-world closed-loop driving rather than primarily in simulation?
- Can independent researchers reproduce the benchmark and evaluation without private assets or specialized infrastructure?
- Are SISR and CFPS sufficiently validated and analyzed to serve as the primary evidence for the paper's conclusions?
- Are claims such as “weak language control” and “current driving VLAs themselves lack reliable instruction following” fully supported by the experimental design?