acceptodds
Under review as a conference paper at ICLR 2027

DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?

Abstract

Frontier models match expert humans across many digital benchmarks, yet real-world driving, a skill any new driver has, remains untested for them. We present DrivingBench, the first benchmark where general-purpose vision-language models must drive a real car. Through three tools, the models see camera frames from a Toyota Corolla and directly command its steering and speed around a parking lot cone course. The car may continue moving while the model thinks and new commands replace the currently running one, so inference latency is part of the task, testing the models' abilities to observe, act, monitor, recover, and complete a long-horizon objective under such constraints. We benchmark GPT-6 Astra, Claude Fable 5.1, GPT-5.6 Sol, and Grok 4.6 in vendor-native harnesses (Codex, Claude Code, Cursor) with up to three attempts each in one conversation; Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course. Two of the four models improved materially across attempts with retained context. We release our harness, prompts, course map, and traces with video and telemetry for reproducibility.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Vision Language Models, Physical AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R5B8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents DrivingBench, a harness that allows vision-language models to drive a real Toyota Corolla using three MCP tools: observe, set_motion, and stop_now. Four model–harness pairs are evaluated on a 127 m cone course, with up to three attempts each. Only GPT-6 Astra completes the course, succeeding on its second attempt in 5:22, while most attempts fail within the first 15 m. The paper identifies several key challenges, including spatial misinterpretation, premature decisions, control limitations, inference latency, and failure to apply lessons from reflection.

Critical strengths
  1. What makes this work valuable to me is that it studies closed-loop control of a full-size vehicle in a world that keeps moving during deliberation. This setting exposes behaviors that static or turn-based benchmarks cannot: stop-and-go strategies lose steering authority because the EPS can hardly turn the wheel from a standstill (median time to 90%: 4.0 s, versus 1.9 s while rolling; Fig. 8b); models act on frames captured roughly 4 m earlier (Fig. 7b); and accurate self-diagnosis often fails to translate into changed control (§5.4).
  2. The harness, prompts, per-attempt telemetry, tool calls with the model’s stated reasons, and videos have been released. The authors also openly acknowledge several limitations: the model–harness confound (§4.2), n = 1 (§6), and the inferred centerline (Fig. 3). As a result, the case study is auditable, even though it cannot be reproduced without the specific hardware and course.
Critical weaknesses
  1. The empirical claims rest on one trial per model, and the three attempts within a trial share context. Under those conditions the ranking and the in-context-learning claim cannot be separated from run-to-run variance. The failures are also confounded with the actuation stack: from rest, a 1 m/s command produces measured peaks of 1.9–2.8 m/s (Figs. 12–13, Table 3) and the wheel barely turns until the car is rolling (Fig. 8b), yet 8 of 11 failures occur in section A right after a standing start and §5.3 attributes them to perception. Without repeated trials and an analysis that separates the car's response from the model's decisions, the paper's conclusions are not reliable.

  2. The work is a case study rather than a benchmark, and lacks the baselines needed to interpret it. There is one course, one vehicle, one trial per system and no baselines: no human teleoperating through the same MCP interface, no scripted or classical controller. Model and harness are confounded (§4.2), and the harnesses differ in system prompt and latency, which §5.2 shows is decisive.

Questions for authors
  1. How many development or pilot runs were conducted, with which models, and on the same course? Was the final prompt tuned mainly against GPT-6 Astra? Please report the refusal rate of each model before and after the drivingbench_sandbox renaming and the other prompt changes.
  2. Quantify per-start launch speed overshoot (commanded vs. measured) and, for the eight Section A failures, determine whether perfect tracking of commanded speed and steering would have kept the car on course—i.e., how much of each failure was due to the model’s decisions versus the actuation stack.
  3. Have you run any additional independent trials since submission? Even 3–5 per model would materially change how much confidence the ranking and in-context-learning claims can support.
  4. What was the performance of a human operator driving through the identical MCP interface (same tools, same latency, same images)? This would establish what fraction of the gap to human driving is due to the interface rather than the models.
  5. Why was the turning reference in the prompt (~60 s for 90°) not corrected after it was found to disagree with telemetry, and were all trials run under the same lighting and weather conditions and on the same day?
Reviewer_mnFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DrivingBench tests whether general-purpose vision-language models can directly drive a real car around a cone course using three simple tools, with the car moving while the model thinks. Along with the benchmark, its driving harness and recorded runs are provided. The paper examines why models fail, with reasons such as misreading cone boundaries, committing to plans too early, and acting on outdated camera frames. Of the four frontier models tested, only GPT-6 Astra finished the course.

Critical strengths
  1. This is a novel real-world closed-loop evaluation. In the set up off-the-shelf frontier models have been put in direct control of a real car, seeing through its cameras and sending steering and speed commands themselves. Prior real-vehicle work kept a conventional driving stack in the loop.
  2. The harness, prompts, and other data are available, and the failure analysis points to specific moments in the traces rather than just reporting final scores. This will be helpful to future research works.
Critical weaknesses
  1. Each model gets a single chat session with up to three attempts, but since all three happen in the same conversation, each attempt builds on the previous ones, so there is really only one independent trial per model. With so few samples, it's hard to get more detailed or insightful comparison between models, since differences could just be what happened in that session but not in general. I think more testing is needed: multiple independent runs per model, and ideally different courses or scenarios. That would also make the benchmark much more useful to future researchers.

  2. Much of the outcome seems to depend on how the tools were designed. Several of the failures the paper describes, such as Fable's stop-and-go driving and Grok's commands arriving after the car had already stopped, look like problems of fitting into this interface rather than driving mistakes. The paper never tests alternatives, for example, longer or chained commands, streamed frames, or keeping the car rolling between commands. Without that, it's hard to tell whether the benchmark measures driving or how well a model adapts to the current design.

Questions for authors
  1. Have you run any models more than once? If so, how consistent were the results?
  2. Did you try other interface designs during development, such as longer or chained commands? How did the models do with them?
  3. Which models did you use when iterating on the prompt? Was it tuned mainly to get GPT-6 Astra to drive?
  4. Do you have results from additional runs, or from other courses or scenarios, that could be included?
Reviewer_5kGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DrivingBench evaluates general-purpose vision-language agents controlling a real Toyota Corolla through a low-speed parking-lot cone course. Agents receive camera images and vehicle telemetry through three tools for observation, motion commands, and stopping. Motion can continue while the model deliberates, making observation staleness, inference latency, and actuator response part of the task.

Four model–harness systems are evaluated, each in one independent conversation with up to three context-sharing attempts. GPT-6 Astra with Codex completes the course on its second attempt in 5 minutes and 22 seconds. The other systems reach at most 45% progress. The paper provides telemetry-based failure analyses and releases the harness, prompts, transcripts, and recorded traces.

Critical strengths
  1. The evaluation exposes meaningful difficulties that static or open-loop benchmarks miss. Agents must connect visual interpretation to physical actions while accounting for delayed observations, ongoing motion, and limited steering response. This is a valuable setting for studying the gap between verbal reasoning and effective embodied control.

  2. The instrumentation supports concrete analysis of failures. Synchronized camera frames, commands, vehicle telemetry, and transcripts allow the authors to examine stale observations, command expiry, steering buildup, and unsuccessful adaptation across attempts. The explicit reporting of operator interventions, hardware limitations, and additional operator messages improves transparency.

Critical weaknesses
  1. The experimental coverage is too limited to establish a reliable benchmark characterization. Each system has only one independent conversation; its subsequent attempts share context and therefore do not constitute independent repetitions. The evaluation also uses one vehicle and one fixed course. The successful run establishes that completion is possible under these conditions, but does not estimate repeatable success rates or support robust system rankings. Likewise, improvement across attempts cannot isolate the effects of reflection from course familiarity or stochastic variation. The authors acknowledge the limited sample size, but it remains a central limitation of the benchmark evidence.

  2. Calibration, scoring, and baseline gaps limit interpretation of the results. The prompt’s steering example suggests approximately 60 seconds for a 90-degree turn at 1 m/s, whereas the reported evaluation telemetry implies approximately 25 seconds. This discrepancy may affect the systems differently depending on how they interpret and adapt to the guidance. Progress is measured from GPS relative to an inferred centerline, while failure termination depends on operator intervention; measurement sensitivity and intervention consistency are not quantified. Finally, the comparison with human completion time is not supported by a measured human baseline, and a simple controller baseline is absent. These controls would help distinguish model limitations from the difficulty introduced by the interface and vehicle dynamics.

Questions for authors
  1. Can you provide repeated independent conversations and additional course layouts, reporting completion rates and uncertainty? A retry condition without the reflection prompt would also help distinguish reflection-specific benefits from repeated exposure to the same course.

  2. Can you provide a calibrated vehicle-response specification, measured human or simple-controller baselines, and a sensitivity analysis of the GPS progress metric and operator termination protocol? In particular, how does correcting the steering guidance affect performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.