- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents VehicleArena, a 3D urban-driving benchmark that integrates LLM-based driving, passenger requests, cockpit control, and multi-agent interactions. It includes 112 evaluation tasks (80 single-agent and 32 multi-agent) and evaluates nine models on driving performance, passenger satisfaction, traffic impact, and inference cost. The benchmark highlights the challenges of coordinating independent driving agents in a shared environment. However, the novelty over existing simulators and the generality of the experimental findings remain unclear.
Critical strengths
- The paper explores an interesting setting where driving agents must respond to passenger requests while considering their effects on surrounding traffic.
- The integration of Driving Agents, Personal Agents, and Judges provides a clear framework for connecting passenger requests, vehicle actions, and task evaluation.
- Evaluating traffic externalities, rather than focusing only on ego-vehicle performance, is a useful addition. The results across nine models reveal interesting differences between driving performance, passenger satisfaction, and trip completion.
Critical weaknesses
- VehicleArena combines SUMO-based traffic simulation, 3D rendering, LLM-based driving, and cockpit interaction. Similar capabilities have been explored in CARLA, SMARTS, MetaDrive, LMDrive, LimSim++, and VehicleWorld. The main contribution appears to be system integration rather than a substantially new simulation or interaction mechanism. Moreover, the claimed realism is not quantitatively validated against real-world driving data or higher-fidelity simulators.
- Only 32 multi-agent tasks are included, with each evaluated agent interacting with three fixed Qwen3.8-27B-controlled vehicles. The routes are relatively short, and the multi-agent scenarios contain no pedestrians. This setting provides limited evidence for the claimed generality of independent-objective physical coexistence.
- Since Personal Agents generate requests dynamically based on the evolving simulation, different driving models may receive requests of different difficulty. This could affect the comparability of Request and Cabin scores. In addition, the reliability of the LLM-based Judge has not been sufficiently validated.
- Task deadlines are determined from SUMO reference rollouts with an additional margin of usually 10 seconds, potentially penalizing conservative driving policies. Similarly, the reported traffic externalities are measured primarily against native SUMO control, making it difficult to distinguish limitations of LLM agents from differences in driving behavior and control interfaces.
- Although each task is evaluated three times, the main results lack confidence intervals or standard deviations. The efficiency analysis also focuses on token usage and tool calls without reporting actual decision latency, which is important for time-critical driving tasks.
Questions for authors
- The reliability of the LLM-based Judge has not been sufficiently validated.
- The reported traffic externalities are measured primarily against native SUMO control, making it difficult to distinguish limitations of LLM agents from differences in driving behavior and control interfaces.
- The main results lack confidence intervals or standard deviations.
5 comments
Response to reviewer arfN(2/3)
Since Personal Agents generate requests dynamically based on the evolving simulation, different driving models may receive requests of different difficulty. This could affect the comparability of Request and Cabin scores. In addition, the reliability of the LLM-based Judge has not been sufficiently validated.
Thank you for the suggestion. PA generates requests dynamically to better reflect real passenger–driver interactions: passenger needs evolve with vehicle behavior and environmental conditions rather than remaining independent of the driving process. For example, hard braking may prompt a request for smoother driving, while weather changes may prompt requests to operate cabin devices. Differences in the requests received by different models therefore partly reflect feedback arising from their own driving behavior, which is an intended component of our evaluation. A fixed request sequence facilitates comparison of instruction execution under identical inputs, but does not fully capture this closed-loop interaction.
All evaluated models use the same task configurations, PA model, and request-generation rules. We agree that this does not guarantee identical request difficulty across models. Accordingly, the Request score measures end-to-end fulfillment during dynamic interaction, rather than execution ability on a fixed set of requests. Cabin is evaluated using consistent automatic rules rather than the LLM Judge, although passenger requests may still affect how agents handle cabin obligations. We will clarify these distinctions and interpret the results alongside driving performance, arrival rates, and request distributions.
To validate Judge reliability, we sampled 200 cases from the execution traces of DeepSeek-V4.1-Flash, Qwen3.8-Max, and MIMO-V2.6-Pro. Human experts received the corresponding passenger requests, the same grading criteria used by the LLM Judge, and videos beginning when PA issued each request. They assessed fulfillment based on the requests and the vehicle’s observed behavior, using the following rubric:
| Grade | Score | Criteria |
|---|---|---|
| A | 100 | All core and secondary requirements are satisfied within the expected response time. |
| B | 80 | All requirements are satisfied after the expected response time but within the validity period. |
| C | 60 | All core requirements are satisfied, with minor omissions or insufficient verification evidence for secondary requirements. |
| D | 40 | Some core requirements are successfully executed, or the required state is not maintained. |
| E | 20 | Relevant actions are executed, but no useful fulfillment outcome is achieved. |
| F | 0 | No relevant action is executed, actions are unrelated, or the outcome contradicts the request. |
Exact grade agreement between human experts and the LLM Judge was 88%, and agreement within one grade was 90%. These results provide direct evidence of the Judge’s reliability on the sampled requests. We will include the human-validation procedure, grading rubric, and agreement results in the revision.
Response to reviewer arfN(3/3)
Task deadlines are determined from SUMO reference rollouts with an additional margin of usually 10 seconds, potentially penalizing conservative driving policies. Similarly, the reported traffic externalities are measured primarily against native SUMO control, making it difficult to distinguish limitations of LLM agents from differences in driving behavior and control interfaces.
Thank you for raising this concern. VehicleArena evaluates how agents balance driving efficiency, passenger comfort, and safety, rather than optimizing arrival time alone. The native SUMO controller provides a driving-oriented reference without the ego passenger-service workload; it is not intended as the optimal policy for this broader objective. Our primary comparison is among LLM agents evaluated under the same task configurations, deadline construction, and scoring protocol. We therefore report arrival, driving quality, passenger-request fulfillment, and cabin compliance separately to expose their trade-offs: slower progress relative to SUMO does not, by itself, imply inferior overall performance.
Although each task is evaluated three times, the main results lack confidence intervals or standard deviations. The efficiency analysis also focuses on token usage and tool calls without reporting actual decision latency, which is important for time-critical driving tasks.
Thank you for the suggestion. We will report the mean and standard deviation across the three repeated runs in the revised main results table and clarify the aggregation procedure to show run-to-run variability.
Regarding decision latency, we agree that response time is important for real-world driving deployment. Our token-usage and tool-call analyses primarily measure the inference and interaction costs of completing tasks, rather than real-time deployment performance. Wall-clock latency also depends on model size, inference hardware, service load, network communication, and concurrency settings. Since different models use different inference services, a direct comparison of response times would also reflect infrastructure differences.
Response to reviewer arfN(1/3)
Thank you for the suggestion. We agree that VehicleArena should be positioned more explicitly as an environment and benchmark contribution, rather than a new traffic simulator. Prior work supports traffic simulation, multi-agent driving, language-guided driving, and cockpit interaction. Our contribution organizes these capabilities into an executable and verifiable task setting: each vehicle handles its passenger’s evolving requests and operates cabin devices while driving, with request fulfillment evaluated independently. This exposes coordination failures that separate evaluations of driving and device operation do not directly reveal. For example, Section 5.5 identifies missed cabin obligations during passenger-device requests and trip timeouts following low-speed requests. We will add an explicit comparison with related work to clarify our task mechanisms and evaluation protocol.
Regarding realism, VehicleArena incorporates real-world urban road layouts, SUMO traffic dynamics, synchronized 3D observations, and constraints from weather, lighting, and cabin devices. These features require passenger requests to be fulfilled under concrete traffic conditions. However, they do not establish quantitatively validated real-world fidelity, and 3D rendering alone does not demonstrate physical realism. Our experiments evaluate models within the simulated environment, without establishing transfer to real-world driving. We will narrow the terms “realistic” and “high-fidelity” and acknowledge the lack of validation against real-world data or higher-fidelity simulators.
Thank you for the suggestion. We have expanded the multi-agent evaluation from 32 to 67 tasks, introducing 21 new maps and pedestrian interactions in all 35 additional tasks. These scenarios cover lead-vehicle yielding, pedestrian crossings across parallel lanes, and turning interactions with pedestrians. We completed testing on the additional tasks with three models, as shown below:
Regarding route length, Table 5 in Appendix A.2 reports that the original multi-agent routes range from 75.8 to 427.9 meters, with a mean of 199.6 meters. These tasks primarily cover local interactions at intersections, merges, and encounters with oncoming vehicles, testing how agents coordinate driving and passenger demands during traffic conflicts. We will make this range more explicit and provide route-length statistics for the expanded task set.
Regarding peer configurations, the three Qwen3.8-27B driving agents receive distinct personality prompts that explicitly specify different driving tendencies, rather than identical prompt configurations. Changing the backbone alone does not guarantee distinct, controllable differences in driving style. Keeping the backbone and personality assignments fixed also provides consistent interaction conditions for comparing evaluated models. We will include the specific personality templates in the appendix to make this design clearer and reproducible.
We will also position the paper more explicitly as an environment and benchmark contribution and narrow the claims surrounding “independent-objective physical coexistence.” This term describes an interaction structure in which vehicles serve different passengers, pursue their own objectives, and affect one another through shared traffic. Our contribution includes constructing this task environment and systematically analyzing effects on surrounding vehicles’ delays, collisions, and non-arrivals through matched comparisons. The additional experiments broaden scenario coverage, but we will not interpret the results as establishing generalization to arbitrary route lengths or peer-model combinations.