- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RoadBench 2.0, a benchmark for evaluating multimodal agents on end-to-end urban road-network construction from aerial imagery and optional OpenStreetMap (OSM) information. Built upon UrbanLaneGraph, it contains 166 regional cases across six US cities and defines six interdependent construction stages (T1–T6) and an end-to-end task (T7). The benchmark combines controlled stage-wise evaluation with prediction-propagating end-to-end evaluation and SUMO-based simulation validation. Experiments on five foundation models reveal a substantial gap between individual-stage performance and end-to-end reliability, with the highest success rate reaching only 0.066. Through failure analysis and ablation studies, the paper highlights error propagation, cross-stage consistency, and the varying effects of OSM and agent execution components.
Critical strengths
-
A meaningful shift from isolated road perception to end-to-end functional evaluation.
-
Useful empirical diagnosis of long-horizon agent failures.
Critical weaknesses
-
The reliability of a benchmark depends critically on whether its metrics faithfully reflect the intended capabilities. In Section 3.1 and Table 2, the junction-localization matching threshold is set to 52.5 meters (350 pixels). Considering that the evaluated imagery has a ground sampling distance of 0.15 m/pixel, this tolerance is relatively large for distinguishing nearby urban intersections. Although the paper specifies one-to-one matching, it does not provide a sensitivity analysis showing how junction detection performance and downstream conclusions vary with the matching threshold.
-
For T4, Table 2 defines lane-count accuracy using the number of reference junction legs, whereas Appendix A describes a road-length-weighted evaluation. Similarly, the relationship between the count-based formulations of T2 in Table 2 and the length-weighted corridor matching described in Appendix A needs to be made explicit. These ambiguities make it difficult to determine precisely how the reported scores were computed and whether comparisons are fully reproducible.
-
The component ablations in Section 3.4 do not consistently isolate the contribution of each component under comparable execution conditions. In particular, Appendix B.2 states that removing the harness replaces the agent with a single direct model call without tools, retries, or a validation loop. Appendix D further indicates that the no-harness configurations do not execute the staged pipeline and consequently receive zero scores. Such results primarily demonstrate the dependency of the current evaluation implementation on its orchestration mechanism, rather than establishing the independent performance benefit of the harness.
-
The benchmark contains 166 regional cases from six US cities, inherited in part from UrbanLaneGraph. While this is a useful starting point, the paper does not provide sufficient evidence about geographic diversity, road-network complexity distributions, or generalization to unseen cities and road layouts. The main component ablations are also conducted using a single model, DeepSeek-V4.1-Flash. Consequently, it remains uncertain whether the observed OSM effects and component dependencies are general properties of multimodal agents or specific to the selected model and dataset.
Questions for authors
-
What motivates the choice of a 52.5-meter tolerance for junction localization in T1? Could the authors report sensitivity results under smaller thresholds, such as 5, 10, and 20 meters?
-
Could the authors clarify the exact aggregation procedures for T2 and T4? Specifically, how do the count-based formulations in Table 2 correspond to the length-weighted evaluation descriptions in Appendix A?
-
Given that the no-harness configuration does not execute the full staged pipeline, how should its zero scores be interpreted as evidence of the harness's contribution? Could the authors provide a comparison against a simpler but executable sequential pipeline, without the proposed orchestration or retry mechanisms, while preserving the same inputs, task definitions, and evaluation criteria?
-
The removal of OSM improves the end-to-end GP and SR of DeepSeek-V4.1-Flash, whereas its effects vary across individual stages. Does this observation also hold for the other evaluated models or across different cities? Can the authors distinguish performance degradation caused by OSM-image alignment errors from degradation caused by the agent's reasoning or tool-use behavior?