acceptodds
Under review as a conference paper at ICLR 2027

RoadBench 2.0: Benchmarking Multimodal Agents on End-to-End Urban Road Network Construction for Traffic Simulation

Abstract

Large-scale remote-sensing imagery and tool-using agents make automated urban road-network construction increasingly feasible. However, a simulation-ready network must maintain consistency across junction localization, corridor connectivity, road direction, lane structure, and lane-to-lane movements; existing benchmarks typically evaluate these tasks in isolation and do not reveal how errors accumulate through a construction chain. We introduce RoadBench 2.0, a dataset and evaluation framework for urban road-network construction from remote-sensing imagery, with aligned OSM references. The dataset contains 166 regional cases covering six US cities, and organizes structural construction and SUMO simulation checks into a connected evaluation protocol, including both stage-wise evaluation with controlled inputs and end-to-end evaluation that passes model predictions downstream. We compare five representative models under a unified protocol. The results show that models can perform well on selected individual stages but remain unreliable across the complete chain, with the highest end-to-end success rate reaching only 0.066. Failure analysis shows that errors in junction localization, corridor connection, and simulation checks accumulate downstream, while ablations reveal stage-dependent effects of OSM. These findings identify cross-stage consistency, error recovery, and simulation feedback as central challenges for simulation-ready road-network construction, and establish RoadBench 2.0 as a unified basis for studying them.

Primary area:
datasets and benchmarks
Keywords:
multimodal agents, urban road-network construction, remote-sensing imagery, geospatial reasoning, traffic simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xj1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RoadBench 2.0, a benchmark for evaluating multimodal agents on end-to-end urban road-network construction from aerial imagery and optional OpenStreetMap (OSM) information. Built upon UrbanLaneGraph, it contains 166 regional cases across six US cities and defines six interdependent construction stages (T1–T6) and an end-to-end task (T7). The benchmark combines controlled stage-wise evaluation with prediction-propagating end-to-end evaluation and SUMO-based simulation validation. Experiments on five foundation models reveal a substantial gap between individual-stage performance and end-to-end reliability, with the highest success rate reaching only 0.066. Through failure analysis and ablation studies, the paper highlights error propagation, cross-stage consistency, and the varying effects of OSM and agent execution components.

Critical strengths
  1. A meaningful shift from isolated road perception to end-to-end functional evaluation.

  2. Useful empirical diagnosis of long-horizon agent failures.

Critical weaknesses
  1. The reliability of a benchmark depends critically on whether its metrics faithfully reflect the intended capabilities. In Section 3.1 and Table 2, the junction-localization matching threshold is set to 52.5 meters (350 pixels). Considering that the evaluated imagery has a ground sampling distance of 0.15 m/pixel, this tolerance is relatively large for distinguishing nearby urban intersections. Although the paper specifies one-to-one matching, it does not provide a sensitivity analysis showing how junction detection performance and downstream conclusions vary with the matching threshold.

  2. For T4, Table 2 defines lane-count accuracy using the number of reference junction legs, whereas Appendix A describes a road-length-weighted evaluation. Similarly, the relationship between the count-based formulations of T2 in Table 2 and the length-weighted corridor matching described in Appendix A needs to be made explicit. These ambiguities make it difficult to determine precisely how the reported scores were computed and whether comparisons are fully reproducible.

  3. The component ablations in Section 3.4 do not consistently isolate the contribution of each component under comparable execution conditions. In particular, Appendix B.2 states that removing the harness replaces the agent with a single direct model call without tools, retries, or a validation loop. Appendix D further indicates that the no-harness configurations do not execute the staged pipeline and consequently receive zero scores. Such results primarily demonstrate the dependency of the current evaluation implementation on its orchestration mechanism, rather than establishing the independent performance benefit of the harness.

  4. The benchmark contains 166 regional cases from six US cities, inherited in part from UrbanLaneGraph. While this is a useful starting point, the paper does not provide sufficient evidence about geographic diversity, road-network complexity distributions, or generalization to unseen cities and road layouts. The main component ablations are also conducted using a single model, DeepSeek-V4.1-Flash. Consequently, it remains uncertain whether the observed OSM effects and component dependencies are general properties of multimodal agents or specific to the selected model and dataset.

Questions for authors
  1. What motivates the choice of a 52.5-meter tolerance for junction localization in T1? Could the authors report sensitivity results under smaller thresholds, such as 5, 10, and 20 meters?

  2. Could the authors clarify the exact aggregation procedures for T2 and T4? Specifically, how do the count-based formulations in Table 2 correspond to the length-weighted evaluation descriptions in Appendix A?

  3. Given that the no-harness configuration does not execute the full staged pipeline, how should its zero scores be interpreted as evidence of the harness's contribution? Could the authors provide a comparison against a simpler but executable sequential pipeline, without the proposed orchestration or retry mechanisms, while preserving the same inputs, task definitions, and evaluation criteria?

  4. The removal of OSM improves the end-to-end GP and SR of DeepSeek-V4.1-Flash, whereas its effects vary across individual stages. Does this observation also hold for the other evaluated models or across different cities? Can the authors distinguish performance degradation caused by OSM-image alignment errors from degradation caused by the agent's reasoning or tool-use behavior?

Reviewer_Gw1YOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles a practically relevant problem in constructing roadnetwork dataset, and proposes a useful end-to-end benchmark. The evaluation perspective is the strongest aspect. However, the experimental validity and metric interpretation are the main weaknesses.

Critical strengths
  1. The benchmark connects junction localization, connectivity, and other key elements in roadnetwork construction, with SUMO validation. So the connection helps validate the network compared to road extraction or lane recognition in isolation
  2. Pairing controlled stage-wise evaluation (correct upstream inputs) with end-to-end evaluation (propagated predictions) gives the benchmark diagnostic value beyond an aggregate score.
Critical weaknesses
  1. Removing the harness also removes tools, retries, and the validation loop, so its zero score is meaningless.
  2. Appendix C allows multiple connection points per junction and separate corridors per carriageway, but T1 requires one central point per junction, T2 forbids modifying it, and T3 labels divided two-way streets as bidirectional. The conversion rules should be stated and the output representation shown to be able to express the reference structures.
  3. Without direct OSM-to-SUMO conversion, a fixed specialist pipeline, or specialist tools plus rule-based postprocessing, the 6.6% best SR shows that the evaluated systems perform poorly but not whether agents add value or whether the bottleneck is perception, reasoning, or interface use.
  4. With 166 cases, 6.6% vs. 4.8% is roughly 11 vs. 8 successes, and the OSM gain (1.8% → 3.0%) is about two cases. Repeated runs or confidence intervals are needed, along with city-level distributions, annotation agreement, and splits.
Questions for authors

See above comments

Reviewer_KuzwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Roadbench 2.0 which is a dataset and evaluation framework for urban road-network construction from remote-sensing images and OpenStreet map (OSM) provided data. The evaluation of multi-modal agents on on this dataset for junctions, corridors, directions, lane counts, and other driving-essential cases makes this work useful to the AD community.

Critical strengths
  • Evaluation of cases for downstream tasks with VLA models, and showing the gaps in stage-wise capability and end-to-end reliblity provides useful information about what could be improved
  • The manual annotations for the 166 regional road network cases is quite valuable in further extending work on traffic simulation
  • The combined annotations, visibility handling, and route probes could become a useful community resource.
Critical weaknesses
  • The ablations do not execute the pipeline, so its zero scores cannot substantiate claims about effectiveness of the "no harness" case
  • The prompts appear to limit representation more than the annotation allows for. especiallyfor complex junctions.
  • Some related work needs to be discussed and placed properly in the paper [1,2]

[1] Guo, Hetian, et al. "Automatic Lane-Level Road Network Extraction from Aerial Imagery for Transportation Digital Twins." International Conference on Transportation and Development 2026. 2026. [2] He, Songtao, and Hari Balakrishnan. "Lane-level street map extraction from aerial imagery." 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2022.

Questions for authors

The benchmark addresses an interesting problem, but the current results/discussions do not separate model limitations from protocol and implementation effects

  • I suggest clarifying T6 and scoring specifications as well
  • Clarify the questions from the weaknesses section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.