acceptodds
Withdrawn from ICLR 2027

VehicleArena: A Realistic Urban Environment for Multi-Agent Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-world embodied agents often pursue independent objectives within a shared physical environment, where their actions can alter the conditions faced by others. Existing benchmarks, however, typically assume shared goals or explicitly prescribed interaction protocols, leaving such emergent physical coupling underexplored. We introduce VehicleArena, a 3D urban-driving benchmark for studying independently operating agents in a dynamic shared world. In VehicleArena, LLM-controlled agents must fulfill evolving passenger requests while navigating complex traffic, and each agent's driving decisions can reshape traffic flow, delays, risks, and subsequent observations for surrounding agents. The benchmark provides 112 evaluation tasks spanning single-agent and multi-agent driving. Across nine evaluated models, the highest arrival rates reach only 65.0% on single-agent tasks and 65.6% on multi-agent tasks, while strong passenger-request or cabin scores do not reliably translate into successful trip completion. Moreover, in matched multi-agent runs, every tested focal policy reduces the arrival rate of surrounding vehicles relative to the simulator's native traffic controller, revealing measurable externalities beyond the focal vehicle itself.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multi-Agent Systems, Autonomous Driving, Urban Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_arfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents VehicleArena, a 3D urban-driving benchmark that integrates LLM-based driving, passenger requests, cockpit control, and multi-agent interactions. It includes 112 evaluation tasks (80 single-agent and 32 multi-agent) and evaluates nine models on driving performance, passenger satisfaction, traffic impact, and inference cost. The benchmark highlights the challenges of coordinating independent driving agents in a shared environment. However, the novelty over existing simulators and the generality of the experimental findings remain unclear.

Critical strengths
  1. The paper explores an interesting setting where driving agents must respond to passenger requests while considering their effects on surrounding traffic.
  2. The integration of Driving Agents, Personal Agents, and Judges provides a clear framework for connecting passenger requests, vehicle actions, and task evaluation.
  3. Evaluating traffic externalities, rather than focusing only on ego-vehicle performance, is a useful addition. The results across nine models reveal interesting differences between driving performance, passenger satisfaction, and trip completion.
Critical weaknesses
  1. VehicleArena combines SUMO-based traffic simulation, 3D rendering, LLM-based driving, and cockpit interaction. Similar capabilities have been explored in CARLA, SMARTS, MetaDrive, LMDrive, LimSim++, and VehicleWorld. The main contribution appears to be system integration rather than a substantially new simulation or interaction mechanism. Moreover, the claimed realism is not quantitatively validated against real-world driving data or higher-fidelity simulators.
  2. Only 32 multi-agent tasks are included, with each evaluated agent interacting with three fixed Qwen3.8-27B-controlled vehicles. The routes are relatively short, and the multi-agent scenarios contain no pedestrians. This setting provides limited evidence for the claimed generality of independent-objective physical coexistence.
  3. Since Personal Agents generate requests dynamically based on the evolving simulation, different driving models may receive requests of different difficulty. This could affect the comparability of Request and Cabin scores. In addition, the reliability of the LLM-based Judge has not been sufficiently validated.
  4. Task deadlines are determined from SUMO reference rollouts with an additional margin of usually 10 seconds, potentially penalizing conservative driving policies. Similarly, the reported traffic externalities are measured primarily against native SUMO control, making it difficult to distinguish limitations of LLM agents from differences in driving behavior and control interfaces.
  5. Although each task is evaluated three times, the main results lack confidence intervals or standard deviations. The efficiency analysis also focuses on token usage and tool calls without reporting actual decision latency, which is important for time-critical driving tasks.
Questions for authors
  1. The reliability of the LLM-based Judge has not been sufficiently validated.
  2. The reported traffic externalities are measured primarily against native SUMO control, making it difficult to distinguish limitations of LLM agents from differences in driving behavior and control interfaces.
  3. The main results lack confidence intervals or standard deviations.
5 comments
AuthorsOpenReview ↗

Response to reviewer arfN(1/3)

VehicleArena combines SUMO-based traffic simulation, 3D rendering, LLM-based driving, and cockpit interaction. Similar capabilities have been explored in CARLA, SMARTS, MetaDrive, LMDrive, LimSim++, and VehicleWorld. The main contribution appears to be system integration rather than a substantially new simulation or interaction mechanism. Moreover, the claimed realism is not quantitatively validated against real-world driving data or higher-fidelity simulators.

Thank you for the suggestion. We agree that VehicleArena should be positioned more explicitly as an environment and benchmark contribution, rather than a new traffic simulator. Prior work supports traffic simulation, multi-agent driving, language-guided driving, and cockpit interaction. Our contribution organizes these capabilities into an executable and verifiable task setting: each vehicle handles its passenger’s evolving requests and operates cabin devices while driving, with request fulfillment evaluated independently. This exposes coordination failures that separate evaluations of driving and device operation do not directly reveal. For example, Section 5.5 identifies missed cabin obligations during passenger-device requests and trip timeouts following low-speed requests. We will add an explicit comparison with related work to clarify our task mechanisms and evaluation protocol.

Regarding realism, VehicleArena incorporates real-world urban road layouts, SUMO traffic dynamics, synchronized 3D observations, and constraints from weather, lighting, and cabin devices. These features require passenger requests to be fulfilled under concrete traffic conditions. However, they do not establish quantitatively validated real-world fidelity, and 3D rendering alone does not demonstrate physical realism. Our experiments evaluate models within the simulated environment, without establishing transfer to real-world driving. We will narrow the terms “realistic” and “high-fidelity” and acknowledge the lack of validation against real-world data or higher-fidelity simulators.

Only 32 multi-agent tasks are included, with each evaluated agent interacting with three fixed Qwen3.8-27B-controlled vehicles. The routes are relatively short, and the multi-agent scenarios contain no pedestrians. This setting provides limited evidence for the claimed generality of independent-objective physical coexistence.

Thank you for the suggestion. We have expanded the multi-agent evaluation from 32 to 67 tasks, introducing 21 new maps and pedestrian interactions in all 35 additional tasks. These scenarios cover lead-vehicle yielding, pedestrian crossings across parallel lanes, and turning interactions with pedestrians. We completed testing on the additional tasks with three models, as shown below:

Evaluation SetModelArrival (%) ↑Drive ↑Request ↑Cabin ↑
Additional 35 tasksKimi-K357.150.983.143.9
MIMO-V2.6-Pro62.942.584.149.1
GLM-5.3-Flash65.749.575.748.5
Complete 67 tasksKimi-K349.360.780.546.7
MIMO-V2.6-Pro62.760.485.658.9
GLM-5.3-Flash64.264.579.556.4

Regarding route length, Table 5 in Appendix A.2 reports that the original multi-agent routes range from 75.8 to 427.9 meters, with a mean of 199.6 meters. These tasks primarily cover local interactions at intersections, merges, and encounters with oncoming vehicles, testing how agents coordinate driving and passenger demands during traffic conflicts. We will make this range more explicit and provide route-length statistics for the expanded task set.

Regarding peer configurations, the three Qwen3.8-27B driving agents receive distinct personality prompts that explicitly specify different driving tendencies, rather than identical prompt configurations. Changing the backbone alone does not guarantee distinct, controllable differences in driving style. Keeping the backbone and personality assignments fixed also provides consistent interaction conditions for comparing evaluated models. We will include the specific personality templates in the appendix to make this design clearer and reproducible.

We will also position the paper more explicitly as an environment and benchmark contribution and narrow the claims surrounding “independent-objective physical coexistence.” This term describes an interaction structure in which vehicles serve different passengers, pursue their own objectives, and affect one another through shared traffic. Our contribution includes constructing this task environment and systematically analyzing effects on surrounding vehicles’ delays, collisions, and non-arrivals through matched comparisons. The additional experiments broaden scenario coverage, but we will not interpret the results as establishing generalization to arbitrary route lengths or peer-model combinations.

AuthorsOpenReview ↗

Response to reviewer arfN(2/3)

Since Personal Agents generate requests dynamically based on the evolving simulation, different driving models may receive requests of different difficulty. This could affect the comparability of Request and Cabin scores. In addition, the reliability of the LLM-based Judge has not been sufficiently validated.

Thank you for the suggestion. PA generates requests dynamically to better reflect real passenger–driver interactions: passenger needs evolve with vehicle behavior and environmental conditions rather than remaining independent of the driving process. For example, hard braking may prompt a request for smoother driving, while weather changes may prompt requests to operate cabin devices. Differences in the requests received by different models therefore partly reflect feedback arising from their own driving behavior, which is an intended component of our evaluation. A fixed request sequence facilitates comparison of instruction execution under identical inputs, but does not fully capture this closed-loop interaction.

All evaluated models use the same task configurations, PA model, and request-generation rules. We agree that this does not guarantee identical request difficulty across models. Accordingly, the Request score measures end-to-end fulfillment during dynamic interaction, rather than execution ability on a fixed set of requests. Cabin is evaluated using consistent automatic rules rather than the LLM Judge, although passenger requests may still affect how agents handle cabin obligations. We will clarify these distinctions and interpret the results alongside driving performance, arrival rates, and request distributions.

To validate Judge reliability, we sampled 200 cases from the execution traces of DeepSeek-V4.1-Flash, Qwen3.8-Max, and MIMO-V2.6-Pro. Human experts received the corresponding passenger requests, the same grading criteria used by the LLM Judge, and videos beginning when PA issued each request. They assessed fulfillment based on the requests and the vehicle’s observed behavior, using the following rubric:

GradeScoreCriteria
A100All core and secondary requirements are satisfied within the expected response time.
B80All requirements are satisfied after the expected response time but within the validity period.
C60All core requirements are satisfied, with minor omissions or insufficient verification evidence for secondary requirements.
D40Some core requirements are successfully executed, or the required state is not maintained.
E20Relevant actions are executed, but no useful fulfillment outcome is achieved.
F0No relevant action is executed, actions are unrelated, or the outcome contradicts the request.

Exact grade agreement between human experts and the LLM Judge was 88%, and agreement within one grade was 90%. These results provide direct evidence of the Judge’s reliability on the sampled requests. We will include the human-validation procedure, grading rubric, and agreement results in the revision.

AuthorsOpenReview ↗

Response to reviewer arfN(3/3)

Task deadlines are determined from SUMO reference rollouts with an additional margin of usually 10 seconds, potentially penalizing conservative driving policies. Similarly, the reported traffic externalities are measured primarily against native SUMO control, making it difficult to distinguish limitations of LLM agents from differences in driving behavior and control interfaces.

Thank you for raising this concern. VehicleArena evaluates how agents balance driving efficiency, passenger comfort, and safety, rather than optimizing arrival time alone. The native SUMO controller provides a driving-oriented reference without the ego passenger-service workload; it is not intended as the optimal policy for this broader objective. Our primary comparison is among LLM agents evaluated under the same task configurations, deadline construction, and scoring protocol. We therefore report arrival, driving quality, passenger-request fulfillment, and cabin compliance separately to expose their trade-offs: slower progress relative to SUMO does not, by itself, imply inferior overall performance.

Although each task is evaluated three times, the main results lack confidence intervals or standard deviations. The efficiency analysis also focuses on token usage and tool calls without reporting actual decision latency, which is important for time-critical driving tasks.

Thank you for the suggestion. We will report the mean and standard deviation across the three repeated runs in the revised main results table and clarify the aggregation procedure to show run-to-run variability.

Regarding decision latency, we agree that response time is important for real-world driving deployment. Our token-usage and tool-call analyses primarily measure the inference and interaction costs of completing tasks, rather than real-time deployment performance. Wall-clock latency also depends on model size, inference hardware, service load, network communication, and concurrency settings. Since different models use different inference services, a direct comparison of response times would also reflect infrastructure differences.

Reviewer_jZsYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VehicleArena, a persistent 3D urban-driving benchmark for evaluating LLM-controlled driving agents that must simultaneously drive, satisfy evolving passenger requests, operate cabin devices, and share the road with other independently controlled vehicles. The environment couples SUMO-based traffic simulation with synchronized 3D rendering and defines three roles: a Personal Agent (PA) that issues immediate or delayed passenger requests, a Driving Agent (DA) that drives and executes requests, and a Judge that verifies request satisfaction independently of DA wakes. The benchmark contains 112 held-out tasks, including 80 single-agent BASIC tasks and 32 MULTI-LLM tasks, and evaluates nine DA models with non-compensatory metrics covering arrival, driving quality, passenger fulfillment, cabin compliance, surrounding-traffic externalities, and token cost. The main empirical findings are that arrival rates remain modest (up to 65.0%/65.6%), strong cabin/passenger scores do not reliably imply trip completion, and focal policies can impose measurable externalities on surrounding vehicles.

Critical strengths
  1. The closed-loop benchmark design is a genuine system contribution.
    The paper integrates driving, passenger requests, cabin state, traffic evolution, and independent verification into a single simulation timeline. The non-compensatory metric suite and the surrounding-traffic externality analysis are well aligned with the stated goal of studying physical coexistence, rather than reducing evaluation to a single aggregate score.

  2. The request lifecycle and agent harness are carefully specified.
    The asynchronous DA wakes, typed deferred-request triggers, independent Judge checks, persistent Todo board, and tool/skill discovery mechanism are described in enough detail to make the benchmark reproducible and to support future work. This is more than a thin wrapper around an existing simulator.

Critical weaknesses
  1. The multi-agent claim is substantially stronger than the evidence supports.
    The paper frames “independent-objective physical coexistence” as a central contribution, but the MULTI-LLM evaluation contains only 32 tasks with one ego DA and three fixed peers. The peers use a fixed model (Qwen3.8-27B), fixed personality prompts, and fixed PA configurations. Although their actions evolve with traffic, this setup is closer to evaluating ego robustness against a fixed background population than to studying an open multi-agent ecology. There is no peer-heterogeneity analysis, no cross-play matrix, and no evidence that conclusions hold under different social configurations. This matters because the paper’s main conceptual novelty depends on the multi-agent coexistence setting, not on the driving benchmark alone.

  2. PA and Judge are LLM agents whose reliability is not adequately validated.
    Passenger fulfillment scores are determined by an LLM Judge, and task difficulty is shaped by LLM-generated PA requests. The paper does not report human-Judge agreement, Judge sensitivity to model choice, PA request distributions, or how PA/Judge behavior interacts with DA capability. The PA ablation covers only two models and shows that disabling PA increases arrival and cabin scores, which could reflect either reasonable passenger-induced delays or problematic request design. Without a failure taxonomy separating driving failures, request conflicts, PA/Judge artifacts, and genuine physical-coupling failures, the interpretation of the passenger and multi-agent results remains uncertain.

Questions for authors
  1. Could the paper be repositioned more explicitly as an environment/benchmark contribution, with the multi-agent “physical coexistence” claim stated more cautiously? If the authors retain the stronger claim, can they add peer-heterogeneity experiments, e.g., different peer models, personality prompts, PA configurations, and a cross-play matrix over more MULTI-LLM tasks?
  2. How reliable is the Judge? Can the authors report human agreement, agreement with rule-based checks, and sensitivity to using a different Judge model? Can they report PA request distributions and whether PA systematically favors low-speed or cabin requests?
  3. Can the authors provide a failure taxonomy for the multi-agent setting that distinguishes (i) driving-capability failures, (ii) passenger-request conflicts, (iii) PA/Judge artifacts, and (iv) failures caused specifically by physical coupling? Which failure modes are not exposed by existing driving or multi-agent benchmarks?
  4. Can the main results include confidence intervals or task-level bootstrap statistics? Why does Figure 5 show only seven models while Table 1 evaluates nine? Can the PA ablation be extended beyond two models?
  5. The abstract states that every tested focal policy reduces surrounding vehicles’ arrival rate relative to the native controller. How robust is this conclusion across tasks and peer configurations, and how is it separated from the focal vehicle’s slower progress?
3 comments
AuthorsOpenReview ↗

Response to reviewer jZsY(1/3)

Could the paper be repositioned more explicitly as an environment/benchmark contribution, with the multi-agent “physical coexistence” claim stated more cautiously? If the authors retain the stronger claim, can they add peer-heterogeneity experiments, e.g., different peer models, personality prompts, PA configurations, and a cross-play matrix over more MULTI-LLM tasks?

Thank you for the suggestion. We will position VehicleArena more explicitly as an environment and benchmark contribution and state the “physical coexistence” claim more cautiously. This term describes the interaction structure under study, rather than a new physical-interaction mechanism. Our multi-agent contribution primarily lies in coupling each vehicle’s passenger service with ongoing driving and systematically analyzing the evaluated vehicle’s effects on surrounding vehicles’ delays, collisions, and non-arrivals.

Regarding model configurations, we keep the PA model fixed to maintain a consistent request-generation mechanism, avoiding additional variation in request distributions and difficulty when comparing driving models. For peer driving agents, we use the same Qwen3.8-27B backbone with distinct personality prompts that explicitly specify different driving tendencies. A shared backbone therefore does not imply identical behavioral settings. We will include the specific personality templates in the appendix and distinguish this behavioral variation from backbone diversity.

We have also expanded the multi-agent evaluation from 32 to 67 tasks, introducing 21 new maps and additional traffic scenarios involving pedestrians. We completed testing on the additional tasks with three models, as shown below:

Evaluation SetModelArrival (%) ↑Drive ↑Request ↑Cabin ↑
Additional 35 tasksKimi-K357.150.983.143.9
MIMO-V2.6-Pro62.942.584.149.1
GLM-5.3-Flash65.749.575.748.5
Complete 67 tasksKimi-K349.360.780.546.7
MIMO-V2.6-Pro62.760.485.658.9
GLM-5.3-Flash64.264.579.556.4

How reliable is the Judge? Can the authors report human agreement, agreement with rule-based checks, and sensitivity to using a different Judge model? Can they report PA request distributions and whether PA systematically favors low-speed or cabin requests?

Thank you for these suggestions. To assess the Passenger Judge’s reliability, we sampled 200 traces from DeepSeek V4.1 Flash, Qwen3.8 Max, and MiMo V2.6 Pro for human evaluation. Human experts received the passenger requests and the same six-level scoring rubric used by the LLM Judge, and watched videos beginning when each request was issued. They assessed whether the vehicle’s behavior fulfilled the request. The Judge achieved 88% exact grade agreement with human evaluations, with 90% of judgments differing by no more than one grade, indicating strong agreement on the sampled traces. We will include the evaluation protocol, scoring rubric, and results in the revision.

We have also added a category-level analysis of PA requests:

CategoryLabel CountShare of All LabelsExample
Ride comfort2,54630.91%Turn on the air conditioning, set the temperature to 22°C, and set the fan to level 2.
Lighting and visibility1,50518.27%It is raining and getting dark. Turn on the front fog lights and set the wipers to level 3.
Media and information2,59931.55%Increase the music volume to 45.
Driving style and behavior83910.19%Keep the speed at or below 25 km/h until the rain stops.
Arrival and parking7489.08%Decelerate smoothly when approaching the destination and come to a gentle stop.
Total8,237100.00%

The current distribution is indeed weighted toward ride comfort, lighting and visibility, and media interactions, which together account for 80.73% of all labels. Driving style and behavior, together with arrival and parking, account for the remaining 19.27%. These percentages refer to category labels rather than distinct requests.

Among requests containing explicit numerical speed requirements, we also observe a tendency toward low-speed requirements. We will include these distribution statistics in the revision and clarify the evaluation’s greater emphasis on cabin and auxiliary-control interactions, so that the aggregate results can be interpreted in the context of the request distribution.

AuthorsOpenReview ↗

Response to reviewer jZsY(2/3)

Can the authors provide a failure taxonomy for the multi-agent setting that distinguishes (i) driving-capability failures, (ii) passenger-request conflicts, (iii) PA/Judge artifacts, and (iv) failures caused specifically by physical coupling? Which failure modes are not exposed by existing driving or multi-agent benchmarks?

Thank you for the suggestion. We have expanded our analysis to distinguish driving errors, passenger-request conflicts, PA/Judge artifacts, and failures arising through vehicle interactions.

1. Driving-capability failures

Our driving-penalty records reveal three main types:

Failure typeObserved behaviorConsequence
Poor longitudinal controlHard acceleration and excessive longitudinal jerkAbrupt speed changes and reduced ride comfort
Inadequate intersection handlingNear misses in conflict zones and delayed responsesFailure to respond promptly to crossing or approaching vehicles
Unnecessary stoppingStops without a valid reasonInterrupted progress and reduced traffic efficiency

These concern the vehicle’s own behavior; we distinguish the original driving error from its downstream effects on others.

2. Passenger-request conflicts

Our failure records cover music and volume, lighting, air conditioning, speed and braking, and windows and wipers. An unfulfilled request does not necessarily indicate a conflict.

CaseDistinguishing criterion
Execution failureThe request is feasible, but the vehicle does not act, applies an incorrect setting, or fails to maintain the requested state
Conflict between requestsRequests require incompatible settings or actions
Conflict with safety requirementsFulfilling the request would violate safety requirements under the current road conditions

Device-state checks help identify execution failures through simulator fields for lights, wipers, mirrors, and the HUD. Request completion assesses fulfillment; device-state checks establish what state the equipment actually reached.

Existing statistics identify the request and device categories involved but do not separately quantify conflicts. We will clarify this rather than treating all request failures as conflicts.

3. PA/Judge artifacts

We sampled 200 traces for human evaluation. Experts received the passenger requests and the same scoring rubric as the Judge, and watched videos beginning at request issuance.

Validation measureResult
Exact grade agreement between humans and the Judge88%
Agreement within one grade90%

These results show that most judgments on the sampled traces agreed with human assessments. Device-state checks provide additional execution records independent of the Judge, including whether a device was activated or reached the requested setting.

We distinguish ambiguous or contradictory PA requests from Judge scoring disagreements; the human-agreement study primarily evaluates the latter.

4. Failures caused by physical coupling

We compared videos of LLM-controlled driving with corresponding SUMO baselines to identify ego behavior when surrounding vehicles slowed down, yielded, or became blocked.

Interaction typeHow the effect occursPotential failure consequence
Lane changes or cut-insThe ego vehicle enters space previously available to another vehicleForced braking or increased collision risk
Same-lane followingChanges in the ego vehicle’s speed affect following vehiclesUnnecessary braking and delays propagate backward
Intersection or opposing-traffic encountersOne vehicle’s decision changes when another can proceedUnnecessary waiting or mutual obstruction
Intersection occupation, collisions, or route blockageThe ego vehicle occupies space another vehicle needs to traverseOther vehicles cannot proceed or continue their tasks

These describe observed cross-vehicle effects. Normal following and safe yielding are not failures by themselves; we distinguish necessary traffic responses from unnecessary delays, obstruction, and collisions.

Which failure modes require this joint evaluation?

The key case is that a vehicle may satisfy its passenger’s request while causing other vehicles to slow down, wait, or become blocked. Request completion captures the local outcome; traffic-impact analysis captures effects on others.

Driving-only evaluation does not link actions to passenger requests. Multi-agent benchmarks without shared road space and vehicle dynamics cannot capture the physical effects of cut-ins, following, or intersection occupation. Our evaluation connects passenger requests, vehicle actions, and effects on others within the same episode.

We do not claim that collisions or lane-change conflicts are new failure types. We will clarify which prior benchmarks lack joint evaluation of passenger-request fulfillment and cross-vehicle effects, and add representative examples.

AuthorsOpenReview ↗

Response to reviewer jZsY(3/3)

Can the main results include confidence intervals or task-level bootstrap statistics? Why does Figure 5 show only seven models while Table 1 evaluates nine? Can the PA ablation be extended beyond two models?

Thank you for these suggestions.

Variability of the main results. We will report the mean and standard deviation across repeated independent runs in the revised main results table, specifying the number of runs and aggregation procedure to show run-to-run stability.

Model coverage in Figure 5. Figure 5 reports traffic-impact results for seven of the nine evaluated models, omitting Qwen3-VL-8B and Qwen3-VL-32B. We acknowledge that this coverage is incomplete. In the revision, we will include the traffic-impact results for both omitted models and provide numerical results for all nine models.

Extended PA ablation. We have added MIMO-V2.6-Pro and Qwen3.8-Max, extending the ablation from two models to four. The additional results are shown below. Parentheses indicate changes relative to PA-on; arrival-rate changes are expressed in percentage points.

ModelBasic Arr. (%)Basic DriveBasic CabinMulti-Agent Arr. (%)Multi-Agent DriveMulti-Agent Cabin
MIMO-V2.6-Pro56.3 (+16.3)73.0 (-6.4)68.4 (+10.9)78.1 (+15.6)73.1 (-6.9)68.0 (+1.3)
Qwen3.8-Max75.0 (+10.0)83.9 (+0.8)95.2 (+23.2)68.8 (+3.1)80.0 (-2.4)95.8 (+30.5)

The additional results are consistent with the observations for the original two models: disabling PA improves arrival rates and Cabin scores for both additional models in Basic and Multi-Agent tasks. This further supports our analysis that handling passenger requests adds to the challenge of coordinating trip completion and cabin obligations.

The abstract states that every tested focal policy reduces surrounding vehicles’ arrival rate relative to the native controller. How robust is this conclusion across tasks and peer configurations, and how is it separated from the focal vehicle’s slower progress?

Thank you for the question. We have added a video-based analysis of multi-vehicle scenarios, comparing LLM-controlled driving with the SUMO native-controller baseline to identify what the ego vehicle was doing when surrounding vehicles slowed down or became blocked. We distinguish the following causes:

CauseObserved interactionRelationship to the ego vehicle’s speed
Same-lane followingThe ego vehicle slows down or stops, causing following vehicles to slow downDirectly related to the ego vehicle’s speed
Lane changes or cut-insThe ego vehicle enters another vehicle’s driving space, forcing it to slow down or adjust its positionDepends on the location and timing of the maneuver, not just speed
Intersection or opposing-traffic yieldingThe ego vehicle’s decision changes when another vehicle can proceedInvolves intersection decisions and responses between vehicles
Intersection occupation, collisions, or route blockageThe ego vehicle obstructs space needed by other vehiclesCauses waiting or prevents passage and cannot be explained solely by average ego speed

Across the three models analyzed, 15.0%–56.2% of confirmed effects on NPCs involved lane changes or cut-ins, while 31.2%–50.0% involved same-lane following. Effects on other LLM-controlled vehicles more frequently involved intersection or opposing-traffic yielding, accounting for 47.8%–66.7% of confirmed events. These percentages describe the composition of confirmed interaction events, not overall failure rates; normal yielding is not itself counted as a failure.

These results show that traffic effects occur with both NPC and LLM-controlled surrounding vehicles, and are not explained solely by slower ego progress. The video comparisons distinguish specific mechanisms, including following-related deceleration, cut-ins, and road blockage. We will also add arrival-rate results broken down by task and surrounding-vehicle configuration to assess the consistency of the aggregate finding across settings.

Reviewer_iJYqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VehicleArena, a 3D urban-driving benchmark that combines autonomous driving, passenger requests, and cockpit control in a shared simulation environment. It also supports multiple LLM-controlled vehicles with independent goals, allowing their driving decisions to affect surrounding traffic. The benchmark includes 112 evaluation tasks, with 80 single-agent and 32 multi-agent scenarios. Experiments on nine LLMs show that models often struggle to complete trips, even when they perform well on passenger requests and cabin operations. The paper also studies how these agents affect surrounding vehicles and how passenger requests influence driving performance.

Critical strengths

The paper brings together driving, passenger requests, and cockpit control in one environment, which is an interesting direction. I also like that it looks beyond the ego vehicle and measures how LLM driving decisions affect surrounding traffic. The evaluation covers several aspects of agent performance rather than just whether the vehicle reaches its destination.

Critical weaknesses
  1. The experiments mainly compare different LLMs within VehicleArena, but do not show how the benchmark improves upon existing platforms such as LMDrive, DriveArena, and SMARTS. I would like to see more direct comparisons or experiments showing what VehicleArena can evaluate that existing benchmarks cannot.
  2. The paper emphasizes "independent-objective physical coexistence," but vehicles with independent goals already interact through shared traffic environments in existing simulators. The authors should explain more clearly what is new about this setting.
  3. The multi-agent evaluation includes only 32 tasks, and the other three LLM-controlled vehicles all use Qwen3.8-27B. I think more combinations of LLMs and more diverse traffic scenarios are needed to better evaluate multi-agent interactions.
  4. The ablation study only removes the Personal Agent and Judge together, and it is conducted on just two models. More ablations are needed to understand the effects of individual components, especially passenger requests and multi-agent interactions.
  5. The Passenger Judge uses an LLM, but the paper does not report how well its judgments agree with human evaluations. Also, Figure 5 only shows the traffic impact of seven out of nine models. I suggest adding human validation of the Judge and reporting the complete traffic-impact results.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to reviewer iJYq(1/3)

The experiments mainly compare different LLMs within VehicleArena, but do not show how the benchmark improves upon existing platforms such as LMDrive, DriveArena, and SMARTS. I would like to see more direct comparisons or experiments showing what VehicleArena can evaluate that existing benchmarks cannot.

Thank you for the suggestion. Section 2.2 discusses these differences, which we summarize more explicitly below. The comparison concerns the tasks and evaluation mechanisms provided in the original papers, rather than capabilities that could be added through extensions.

CategoryCapabilityLMDriveDriveArenaSMARTSVehicleArena
Driving environmentClosed-loop driving✓✓✓✓
Natural-language driving instructions✓——✓
Independently controlled driving agents sharing the road——✓✓
Passenger serviceCabin-device operation during driving———✓
Independent passenger agents dynamically generating and updating requests———✓
EvaluationDriving performance and safety✓✓✓✓
Independent verification of passenger-request fulfillment———✓
Evaluation of effects on other vehicles——✓✓

✓ indicates coverage in the original paper; — indicates that the capability is not explicitly provided, rather than being impossible to implement. SMARTS and VehicleArena use different approaches to evaluating effects on other vehicles.

VehicleArena’s main contribution is coupling passenger service with ongoing driving. Each vehicle must complete its trip while handling its passenger’s evolving requests, operating cabin devices, and undergoing independent fulfillment verification. This enables evaluation of how agents balance passenger service, driving efficiency, and safety, alongside their effects on surrounding traffic. Section 5.5 illustrates missed cabin obligations while executing passenger requests, as well as associations between low-speed requests and trip timeouts. These findings expose coordination failures that separate evaluations of driving and device operation do not directly reveal. We will incorporate this comparison into the paper to clarify the additional evaluation value.

The paper emphasizes "independent-objective physical coexistence," but vehicles with independent goals already interact through shared traffic environments in existing simulators. The authors should explain more clearly what is new about this setting.

Thank you for pointing this out. We agree that vehicles with independent destinations already interact through shared roads in simulators such as SMARTS; we do not claim this physical phenomenon itself as new. “Independent-objective physical coexistence” describes the interaction structure studied in VehicleArena. Our contribution is that each vehicle must not only reach its destination but also handle its own passenger’s evolving requests, coordinating driving, cabin operations, and request fulfillment. This enables evaluation of trade-offs between passenger service and driving objectives, alongside the effects of these behaviors on other vehicles. The novelty therefore lies in the integrated tasks and evaluation mechanisms built around this interaction structure, rather than the first realization of physical coexistence among independent vehicles. We will revise the wording to distinguish established traffic-interaction mechanisms from our contribution.

AuthorsOpenReview ↗

Response to reviewer iJYq(2/3)

The multi-agent evaluation includes only 32 tasks, and the other three LLM-controlled vehicles all use Qwen3.8-27B. I think more combinations of LLMs and more diverse traffic scenarios are needed to better evaluate multi-agent interactions.

Thank you for the suggestion. We have expanded the multi-agent evaluation from 32 to 67 tasks, introducing 21 new maps and additional traffic scenarios involving pedestrian interactions, including lead-vehicle yielding, pedestrian crossings across parallel lanes, and turning interactions with pedestrians. We evaluated Kimi-K3, MIMO-V2.6-Pro, and GLM-5.3-Flash on the additional tasks. The results are shown below:

Evaluation SetModelArrival (%) ↑Drive ↑Request ↑Cabin ↑
Additional 35 tasksKimi-K357.150.983.143.9
MIMO-V2.6-Pro62.942.584.149.1
GLM-5.3-Flash65.749.575.748.5
Complete 67 tasksKimi-K349.360.780.546.7
MIMO-V2.6-Pro62.760.485.658.9
GLM-5.3-Flash64.264.579.556.4

Compared with the original tasks, all three models obtain lower Drive and Cabin scores on the additional scenarios, although arrival rates do not uniformly decrease. This indicates that the expanded scenarios expose challenges in driving quality and cabin compliance beyond trip completion, supporting our decision to evaluate these dimensions separately.

Although all three peers use Qwen3.8-27B, they receive distinct personality prompts that explicitly specify different driving tendencies. Sections 4.2 and 5.1 already describe this design, and we will include the specific templates in the appendix. Keeping the peer backbone and personality assignments fixed provides consistent conditions for comparing the evaluated models. The expanded evaluation broadens scenario coverage, but we will also clarify that the results remain conditional on this controlled peer configuration and do not establish generalization to arbitrary peer-model combinations.

The ablation study only removes the Personal Agent and Judge together, and it is conducted on just two models. More ablations are needed to understand the effects of individual components, especially passenger requests and multi-agent interactions.

Thank you for the suggestion. We have extended the PA ablation from Kimi-K3 and GLM-5.3-Flash to include MIMO-V2.6-Pro and Qwen3.8-Max, covering four models in total. The additional results are shown below. Parentheses indicate changes relative to PA-on, with arrival-rate changes expressed in percentage points.

ModelBasic Arr. (%)Basic DriveBasic CabinMulti-Agent Arr. (%)Multi-Agent DriveMulti-Agent Cabin
MIMO-V2.6-Pro56.3 (+16.3)73.0 (-6.4)68.4 (+10.9)78.1 (+15.6)73.1 (-6.9)68.0 (+1.3)
Qwen3.8-Max75.0 (+10.0)83.9 (+0.8)95.2 (+23.2)68.8 (+3.1)80.0 (-2.4)95.8 (+30.5)

The additional results are consistent with the observations for the original two models: disabling PA increases arrival rates and Cabin scores for all four models in both Basic and Multi-Agent tasks. This further supports our analysis that handling ongoing passenger requests adds to the challenge of coordinating trip completion and cabin obligations.

We also clarify the distinct roles of PA and Judge in this ablation. PA generates passenger requests, whereas Judge evaluates fulfillment from execution records without providing control instructions or grading feedback to the Driving Agent. Disabling the ego PA eliminates its passenger requests, so the corresponding ego Judge is also disabled. The experiment therefore primarily examines the behavioral effects of removing passenger requests, rather than jointly ablating Judge as a driving-decision component. Cabin is still evaluated using the same automatic rules, and Request scores are not reported when no requests are generated.

AuthorsOpenReview ↗

Response to reviewer iJYq(3/3)

The Passenger Judge uses an LLM, but the paper does not report how well its judgments agree with human evaluations. Also, Figure 5 only shows the traffic impact of seven out of nine models. I suggest adding human validation of the Judge and reporting the complete traffic-impact results.

Thank you for these suggestions. We have added a human validation study of the Passenger Judge. We sampled 200 traces from DeepSeek V4.1 Flash, Qwen3.8 Max, and MiMo V2.6 Pro. Human experts received the passenger requests and the same scoring rubric used by the LLM judge, and watched videos beginning when each request was issued. They assessed whether the vehicle’s behavior fulfilled the request, using the six-level rubric from A (100) to F (0), which accounts for requirement satisfaction, response timing, and evidence of execution.

GradeScoreCriteria
A100All core and secondary requirements are satisfied within the expected response time.
B80All requirements are satisfied after the expected response time but within the validity period.
C60All core requirements are satisfied, with minor omissions or insufficient verification evidence for secondary requirements.
D40Some core requirements are successfully fulfilled, or the required state is not maintained.
E20Relevant actions are executed, but no useful fulfillment outcome is achieved.
F0No relevant action is executed, actions are unrelated, or the outcome contradicts the request.

The LLM judge and human experts assigned exactly the same grade in 88% of cases, and their grades differed by at most one level in 90% of cases. These results support the judge’s agreement with human assessments on the sampled traces, while also showing that some disagreements remain. We will include the evaluation protocol, rubric, and agreement results in the revised paper. Figure 5 reports traffic-impact results for seven of the nine evaluated models, omitting Qwen3-VL-8B and Qwen3-VL-32B. We acknowledge that this coverage is incomplete. In the revision, we will include the traffic-impact results for both omitted models and provide numerical results for all nine models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.