acceptodds
Under review as a conference paper at ICLR 2027

Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

92 views

Abstract

Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests. Effective evaluation must therefore measure not only whether a VLA can drive successfully, but whether changing the instruction reliably alters which feasible behavior it selects and successfully executes.

Primary area:
datasets and benchmarks
Keywords:
autonomous driving;Vision-Language-Action

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hr3gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Language2Drive, a benchmark for evaluating instruction following in autonomous-driving vision-language-action (VLA) models. Unlike conventional driving benchmarks, which primarily score safety, route completion, and driving quality, Language2Drive centers on the Paired Counterfactual Unit (PCU): the same scene is held fixed while only the instruction changes. The authors report Strict Instruction Success Rate (SISR) for individual requests and Counterfactual Paired Success (CFPS) for matched alternatives. Across closed-loop CARLA, reconstructed AlpaSim scenes, and open-loop tests on nuScenes, NAVSIM, and WOD-E2E, they find a consistent gap: strong driving performance does not imply reliable language control. Most successful rollouts are one-sided—the model completes one preferred behavior but fails the other feasible instruction, even when safety and driving scores remain high.

Critical strengths
  1. The paper identifies a genuine evaluation risk: high driving scores can overstate instruction-following ability, with direct implications for language-conditioned driving systems.

  2. This paper introduces PCU. By comparing SISR and CFPS, it explicitly differentiates isolated task success from true language‑driven behavioral selection.

  3. The authors build extended CARLA‑Language2Drive with six purpose‑designed maps, curated AlpaSim real‑world reconstructed scenes, plus open‑loop evaluation pipelines on three public driving datasets. Tasks span four complexity levels from simple atomic maneuvers to complex conditional long‑horizon program instructions. They also provide paraphrase robustness tests to check consistency across semantically equivalent wording.

  4. Extensive closed‑loop and open‑loop experiments across multiple state‑of‑the‑art driving VLA models empirically demonstrate the prevalence of one‑sided completion failure. Analyses on paraphrase sensitivity, long‑horizon program failure and safety‑conflict cases provide rich diagnostic insights.

Critical weaknesses
  1. Most of the core evaluations are conducted in simulation, using CARLA and reconstructed scenes from AlpaSim. Although the authors additionally use real-world logs for open-loop evaluation, there is no on-vehicle, closed-loop evaluation in real-world driving scenarios. As a result, it remains unclear whether the reported improvements transfer to realistic perception noise, sensor imperfections, and vehicle dynamics. Existing benchmarks such as NAVSIM and NeuroNCAP could provide complementary evaluations on more realistic driving scenarios.

  2. Reproducing the full benchmark appears to require custom CARLA maps as well as private evaluation sidecars and specialized evaluator assets. This substantially increases the setup and engineering burden for other research groups, making it difficult to independently reproduce the reported results and adopt Language2Drive as a standard benchmark.

  3. The paper provides a thorough characterization of observed failures, including one-sided completion, paraphrase sensitivity, and failures in long-horizon reasoning. However, it provides limited controlled ablations to determine why these failures occur. In particular, the current experiments do not clearly disentangle errors arising from perception, language understanding, planning, and low-level controller tracking.

  4. The paper asks whether language determines which feasible behavior a vehicle executes, and the PCU protocol evaluates matched execution of two mutually exclusive contracts under a shared initialization. However, Appendix A.1 acknowledges that an instruction-invariant policy with uncoupled randomness can still achieve an expected CFPS of up to 25% while completely ignoring the language instruction. The paper further notes that instruction-ablated repetitions would provide an additional test of causal language dependence, but such an experiment does not appear to be included. Therefore, the abstract's claim of “weak language control” seems stronger than what the current experimental design can causally establish. At present, the results demonstrate a gap in matched instruction-conditioned behavior, but do not cleanly establish that the gap is caused by insufficient language conditioning.

  5. Section 3.1 compares native driving stacks (e.g., LMDrive, SimLingo, and Alpamayo) with generic VLM/VLA bridges (e.g., Qwen-Drive, OneVL, and Impromptu), but these systems do not necessarily share the same camera configuration, control interface, or low-level controller. In particular, the generic bridges use a different camera rig and a pure-pursuit-style tracker with a configured 1 Hz invocation interval. The Senna CARLA results further introduce a “no Senna-E2E planner” configuration, while the RealSim experiments exclude the official Senna and LMDrive implementations. Consequently, failures may arise from differences in sensing, tracking, interfaces, or system integration rather than from the VLA's inability to follow language instructions. This makes the broad conclusion that “current driving VLAs themselves lack reliable instruction following” difficult to identify from the current comparisons.

  6. The paper states that SISR and CFPS are the primary measures of complete execution, yet their detailed definitions, interpretation, and empirical analysis receive limited space in the nine-page main paper. Given that these metrics underpin the central claims, the paper would benefit from a more explicit explanation of their design, failure modes, and relationship to conventional driving metrics.

  7. The statement that “93.2% of successful rollouts lack a successful opposite-instruction counterpart” is a particularly important quantitative claim because it directly supports the paper's argument about weak language control. However, the source, calculation procedure, and statistical basis of this number should be made explicit in the main text rather than leaving readers to infer its derivation from the appendix or supplementary material.

  8. A substantial portion of the paper's text appears to rely heavily on LLM-generated descriptions and analyses.

Questions for authors
  1. Does the benchmark actually establish that language causes the choice of driving behavior, rather than merely correlating with successful execution?
  2. Are the observed failures genuinely caused by VLA language understanding, or could they arise from perception, planning, tracking, sensor configuration, or system integration?
  3. Do the conclusions hold in realistic real-world closed-loop driving rather than primarily in simulation?
  4. Can independent researchers reproduce the benchmark and evaluation without private assets or specialized infrastructure?
  5. Are SISR and CFPS sufficiently validated and analyzed to serve as the primary evidence for the paper's conclusions?
  6. Are claims such as “weak language control” and “current driving VLAs themselves lack reliable instruction following” fully supported by the experimental design?
Reviewer_zex9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

I find the instruction-following ability of current driving VLA models an important and interesting research question. The proposed Paired Counterfactual Unit (PCU) provides a useful way to distinguish instruction-dependent behavior from default, scene-driven behavior.

However, I still have several concerns:

C1: The benchmark involves so much efforts, yet the analysis remains descriptive. For instance, “Most successful requests have no successful counterpart” is an observation rather than an insightful explanation. I would expect more insightful viewpoints that reveal about the underlying limitations of current VLAs.

C2: While I appreciate this benchmark, the paper is currently difficult to read and needs substantial polishing. I suggest the author revise it with a general audience in mind.

C3: More recent advances in instruction-following driving methods should be introduced.

C4: Could hierarchical or modular approaches offer stronger instruction-following capabilities? More discussion of models such as EMMA and Senna, would provide useful insights beyond benchmark scores.

Overall, I consider Language2Drive a promising benchmark with a meaningful research objective. However, the paper would greatly benefit from content concisement, more focused scientific writing, and deeper experimental analysis.

Critical strengths

See above

Critical weaknesses

See above

Questions for authors

See above

Reviewer_QECtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes PCU (Pairwise Counterfactual Unit), a counterfactual evaluation paradigm for language-conditioned autonomous driving. The authors build a multi-track benchmark suite, which includes closed-loop evaluation in CARLA, RealSim built from real driving logs, open-loop PCU tests on nuScenes, NAVSIM and WOD-E2E, together with auxiliary FDQ and Safety-Q diagnostic subsets. The core observation of this paper is that high single-instruction success rate (SISR) does not indicate genuine language-driven control. Many successful-looking cases belong to one-sided completion: the model follows the default scene behavior instead of selecting actions according to natural language commands.

Overall, This paper proposes a valuable counterfactual evaluation paradigm PCU and reveals an important failure pattern named one-sided completion for language-conditioned autonomous driving models. The core research insight is interesting and the PCU construction workflow is carefully designed. However, the benchmark suffers from major structural design concerns. The multi-track, cross-heterogeneous-environment architecture creates unnecessary domain gaps and heavy reproduction cost. The paradigm itself has inherent coverage limits, and there is no proper mechanism to separate basic driving skill and language instruction following. In addition, root-cause ablation experiments are missing and the custom CARLA maps raise concerns about conclusion transferability. These issues substantially weaken the reliability and usability of the benchmark. Major revisions including benchmark consolidation, ablation studies and robustness verification are required before acceptance.

Critical strengths
  • The paper identifies a critical flaw in existing VLA driving evaluation: relying only on SISR will substantially overestimate a model’s language controllability. The proposed PCU counterfactual paradigm provides a useful warning for the community.
  • The closed-loop and open-loop experiments together demonstrate that the one-sided completion failure occurs at the trajectory prediction stage, rather than being an artifact caused by controller tracking errors.
  • The discussion analyzes potential failure mechanisms and defines clear criteria to judge meaningful progress for future research on language-conditioned autonomous driving.
  • The PCU construction pipeline is systematic. Every candidate case passes oracle simulation validation to guarantee that the two alternative behaviors are physically feasible and mutually exclusive before being added into the benchmark.
Critical weaknesses
  1. The benchmark adopts a fragmented multi-environment design instead of deepening evaluation within one unified environment. The paper evaluates models across heterogeneous environments including CARLA, RealSim and multiple external open-loop datasets, trying to cover many separate benchmark tracks rather than expanding capability within a single environment. Different tracks cover disjoint task subsets, and metrics across tracks cannot be compared directly. This multi-track design introduces domain confounding effects and greatly increases computational cost for reproduction. If open-loop validation is required, open-loop data could be sampled from CARLA to maintain consistent dynamics. RealSim may be moved to the appendix only as a small sanity check instead of a core benchmark track.
  2. The PCU paradigm has inherent coverage limitations and lacks a mechanism to disentangle basic driving skill and language-following ability. PCU requires scenarios containing two mutually exclusive feasible behaviors, so it cannot evaluate most real-world driving cases where only one safe action exists. FDQ can partially assess basic driving ability, but it is an independent diagnostic set and not integrated into core PCU evaluation. Readers may mistakenly attribute poor PCU performance to insufficient basic driving capability instead of weak language instruction following.
  3. Root-cause analysis of failure modes is insufficient, and the main experiment lacks robustness verification. The paper observes multiple failure modes but only provides speculative explanations without targeted controlled ablation experiments. It cannot pinpoint whether failures stem from language grounding, planning modules or the controller. The main closed-loop experiments use only a single random seed. With low CFPS values, the results may be unstable under simulation randomness. There is no uncertainty quantification across multiple seeds.
  4. The custom-built CARLA maps bring domain-shift risks and hinder comparisons with prior work. The authors create entirely new custom CARLA maps instead of reusing and modifying widely adopted official CARLA Towns. Although custom maps simplify controlled construction of PCU decision forks, it remains unvalidated whether conclusions drawn from these maps transfer to standard CARLA Towns used in previous driving research. This custom-map design also complicates quantitative comparison against existing CARLA baselines.
Questions for authors
  1. Why choose this multi-track cross-environment design instead of consolidating evaluation within one environment and building internal generalization splits?
  2. How do you prevent readers from confusing poor PCU performance with insufficient basic driving capability, and how to decouple basic driving skill and language-following ability?
  3. Can ablation studies be designed to locate whether failures come from language grounding, planning or controller modules? Will you add multi-seed experiments and quantify result uncertainty?
  4. What is the main reason to build custom CARLA maps from scratch rather than editing official CARLA Towns? How do you verify the conclusion transferability to standard CARLA Towns?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.