- Rating
- 4(+1)
- Confidence
- 3(+1)
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
-
This paper proposes UrbanGround, a large-scale urban interaction simulation and evaluation platform built based on real 3D geographic data of Hong Kong. The platform connects real urban geometry, physical collision, pedestrian networks, dynamic pedestrians, weather systems and the closed-loop actions of multimodal large models, and evaluates Spatial Agency through 5 levels and 13 categories of tasks. The authors evaluated multiple mainstream MLLMs on 810 manually verified basic test instances, and found that the models have strong local visual recognition capabilities, but their performance drops significantly in direction judgment, long-term navigation, multi-objective planning and dynamic obstacle adaptation.
-
I think this benchmark paper bears important research questions, high platform engineering value, but the core scientific argumentation is not yet sufficient. Its most valuable contribution is establishing a relatively realistic urban interaction environment and systematically demonstrating the capability gap of current MLLMs from local understanding to long-term autonomous action.
-
However, the core argument of the paper - that models struggle to maintain long-term spatial states - is mainly inferred indirectly through task success rates and navigation trajectories, lacking control experiments that can rule out other failure factors.
-
In addition, the success rate of a large number of high-difficulty tasks is close to zero, causing an obvious evaluation metric saturation problem, which makes it difficult for the benchmark to further distinguish the real capabilities of the models.
Critical strengths
- A physically interactive evaluation environment (simulated) at the real city scale is proposed. UrbanGround integrates real urban 3D geographic data, navigation networks and continuous movement action space, and also adds simulations of conditions such as lighting in different time periods.
- A progressive evaluation from local perception to long-term autonomous spatial action is proposed. The task design is gradually expanded: direction judgment, active exploration, long-distance navigation, multi-task planning and dynamic adaptation. The design of this evaluation task aims to answer the question: At which stage exactly does the model start to fail?
- The paper reveals the compositionality problem in spatial ability of MLLMs. The study finds that the models’ ability to complete target recognition, direction understanding or local movement alone does not mean that it can effectively combine these abilities for long-distance navigation. The accumulation of small local errors may lead to the failure of the entire task.
- The main contributions of UrbanGround are the simulation platform, benchmark and systematic experimental analysis. No new algorithm is proposed.
Critical weaknesses
-
The core scientific claim lacks direct evidence. The paper defines in Eq. 1 of Section 3.1 to represent the workspace state maintained by the model during interaction. However, the authors also explicitly point out that this state is an analytical abstraction, which neither assumes the existence of an explicit memory module in the model nor is directly observed by the evaluator. Therefore, I believe there is an obvious argumentation gap in the current paper: the paper claims to study the model's ability from local perception to sustained spatial intelligence, but it does not fully distinguish between the failure of long-term spatial state maintenance and other factors that lead to navigation failure. The good performance in short-distance navigation and poor performance in long-distance navigation indeed indicate that the model's performance degrades during long-range interaction. However, this does not prove that insufficient spatial memory is the main cause.
-
Extremely low task success rate indicate that model performance Lacks practical implications. As shown in Table 2, all models generally have an extremely low success rate in core tasks such as Long-distance Navigation (LGN), Instruction Navigation (IN), and Constrained Navigation (CN) and others. For example, the best-performing GPT-5.6-Luna only has a 5.0% success rate on the LGN task and a mere 3.3% on the CN task, while GPT-5.5 even achieves a 0.0% success rate on LGN. Such an extremely low success rate indicates that the spatial capabilities of current models in real urban environments are far from reaching a usable level. Although the paper may focus on the construction of the benchmark test, such a low baseline performance calls into question the discriminative power and applicability of this benchmark. If all models can barely complete the tasks, then whether the benchmark can effectively measure the differences in model capabilities remains a question.
-
Lack of reasonable justification for task difficulty settings. This paper claims that all tasks have been manually verified to be completable within the 150-step limit, but the experimental results show that the vast majority of models almost completely failed on long-distance tasks. This suggests that the task difficulty may be set too high, or there is a systematic difference between the verification process and the actual execution of the model. The paper does not provide human performance data to justify the task difficulty. If humans also cannot complete these tasks within 150 steps, the difficulty setting of this benchmark becomes less persuasive. In addition, this paper does not discuss whether the progress of task difficulties between levels is reasonable: whether the difficulty increase from Level 1 to Level 5 is uniform, and whether there are large jumps between some levels.
-
Narrow evaluation metric, lacking analysis of failure modes. This paper mainly reports the task success rate and answer accuracy, but lacks in-depth analysis of the model's failure modes. For example, does the model fail during the path planning stage, or deviate from the route during the execution stage? Does it make mistakes in the visual perception stage, or in the spatial reasoning stage? This paper only provides two case studies of GPT-5.5 in Appendix F, but lacks systematic error classification and quantitative analysis. For benchmark testing, understanding the failure modes of the model is crucial for guiding subsequent research, but the paper's analysis in this regard is obviously insufficient.
-
Limited model coverage, lack of evaluation of open-source small models. Table 2 shows that the evaluated models are mainly concentrated on commercial closed-source models such as the GPT series, Claude series, and Gemini series, as well as a small number of open-source models like GLM-5V-Turbo and Kimi-K3. However, the paper does not evaluate some open-source models widely used in the community (such as LLaVA, Qwen-VL, etc.), nor does it discuss the impact of model scale on performance. In addition, the paper fails to provide a comparison of different models in terms of computing cost, inference latency, etc., which is valuable for model selection in application scenarios. More critically, the paper does not evaluate the performance of the model under limited computing resources, which is crucial for promoting the development of this field in resource-constrained environments.
-
Benchmark is geographically quite limited, and hard to scale. This paper only used the single region of Hong Kong for the benchmarking, and additional geographic regions are necessary to verify its generalization performance. The title is UrbanGround, but Hong Kong is a rather unique city among urban categories. The cost and conditions of creating such a benchmark should be acknowledged so that other researchers are aware of the difficulty of replicating it across cities, and be marked as a limitation that this benchmark is difficult to scale.
Questions for authors
-
If the authors could provide additional evidence to more directly support their claim that the presented navigation failures are mainly due to spatial memory, the paper becomes much better backed up. Control experiments are appreciated.
-
No ground to judge how difficult are some of the tasks, especially long-range navigation ones. Human performance can provide an important baseline. Besides, by providing simple modular agents, e.g., visual localization + topological path planning + low-level motion control. The purpose here is not to defeat general-purpose MLLMs, but to judge whether a navigation system with a simple structure can already solve some tasks that general models fail to complete. Or by providing memory or prompt strategies for MLLMs, e.g., periodic history summary, explicit landmark lists, etc. If these simple strategies can significantly improve the navigation success rate, then the paper's explanation for the insufficient native spatial capabilities of the model needs to be further refined. Conversely, if these strategies are still ineffective, it can also strengthen the author's argument about the difficulty of long-term spatial intelligence. These baselines can substaintially strengthen the paper.
-
The extremely low task success rate leads to three problems. First, it is difficult to distinguish model capabilities. When two models achieve success rates of 0% and 1.3% respectively, it is hard to determine whether there is a meaningful performance difference. Second, it is difficult to apply to subsequent algorithm research. If a new method achieves a 2% success rate on a certain task while the baseline is 0%, this does not necessarily prove that the method has a stable advantage. Finally, it is difficult to locate which capability the improvement comes from. The model may have walked correctly for a long distance, but is still recorded as a failure because it cannot complete the task in the last few meters. The backtracking checkpoint analysis in Appendix D of the paper has a certain mitigating effect, but it is not sufficient to completely solve the problem. I suggest that the authors supplement the rationality argument for this task setting, and preferably add intermediate-level tasks between simple tasks and extremely difficult tasks, so as to make the tasks more practically meaningful.
If the authors carefully incorporate these suggestions and overcome some of the major weaknesses, I will consider raising the score to 4.
13 comments
Official Review of Submission19810 (Part 2)
W2: Extremely low task success rate indicate that model performance Lacks practical implications. As shown in Table 2, all models generally have an extremely low success rate in core tasks such as Long-distance Navigation (LGN), Instruction Navigation (IN), and Constrained Navigation (CN) and others. For example, the best-performing GPT-5.6-Luna only has a 5.0% success rate on the LGN task and a mere 3.3% on the CN task, while GPT-5.5 even achieves a 0.0% success rate on LGN. Such an extremely low success rate indicates that the spatial capabilities of current models in real urban environments are far from reaching a usable level. Although the paper may focus on the construction of the benchmark test, such a low baseline performance calls into question the discriminative power and applicability of this benchmark. If all models can barely complete the tasks, then whether the benchmark can effectively measure the differences in model capabilities remains a question.
The low success rates on challenging navigation tasks show a substantial gap between current MLLMs and the spatial capabilities required for reliable urban exploration. UrbanGround is designed to measure this gap. The benchmark also distinguishes model capabilities through clear performance differences in shorter navigation tasks, while the distance-stratified and partial-progress analyses provide further evidence of differences even when task completion rates are close to zero.
All 810 tasks were completed by human annotators under the same interaction interface, within the 150-step limit, and further verified in a second round. It means that the low model success rates cannot be attributed to infeasible task instances. The large gap between human and model performance points to fundamental weaknesses in sustained spatial reasoning. Figure 9 illustrates this issue, where the agent repeatedly attempts to move forward after encountering an obstacle, which provides directions for improving future models toward reliable long-horizon urban navigation.
W3: Lack of reasonable justification for task difficulty settings. This paper claims that all tasks have been manually verified to be completable within the 150-step limit, but the experimental results show that the vast majority of models almost completely failed on long-distance tasks. This suggests that the task difficulty may be set too high, or there is a systematic difference between the verification process and the actual execution of the model. The paper does not provide human performance data to justify the task difficulty. If humans also cannot complete these tasks within 150 steps, the difficulty setting of this benchmark becomes less persuasive. In addition, this paper does not discuss whether the progress of task difficulties between levels is reasonable: whether the difficulty increase from Level 1 to Level 5 is uniform, and whether there are large jumps between some levels.
The five levels are designed to examine distinct spatial capabilities, without assuming a uniform increase in difficulty. Each level introduces a new requirement, progressing from local grounding to sustained navigation, goal inference, multi-goal planning, and adaptation to environmental changes. Sharp performance drops between levels are therefore meaningful, which reveal where previously acquired capabilities cease to support successful execution.
We further compare success rates across levels to assess whether model failures arise from excessive task difficulty. Human performance reflects the verified manual completions, while the naive baseline uses random guessing for multiple-choice questions and greedy distance-reducing movement for navigation. The comparison establishes human feasibility across all levels and shows that current MLLMs struggle to sustain reliable execution as additional spatial capabilities become necessary.
| Model | L1 | L2 | L3 | L4 | L5 |
|---|---|---|---|---|---|
| GPT-5.6-Luna | 63.2 | 32.5 | 18.3 | 1.7 | 0.0 |
| Claude-Opus-5 | 79.1 | 23.8 | 20.8 | 3.3 | 1.8 |
| Human | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
| Naive Baseline | 24.5 | 4.6 | 0.8 | 0.0 | 0.0 |
Official Review of Submission19810 (Part 3)
W4: Narrow evaluation metric, lacking analysis of failure modes. This paper mainly reports the task success rate and answer accuracy, but lacks in-depth analysis of the model's failure modes. For example, does the model fail during the path planning stage, or deviate from the route during the execution stage? Does it make mistakes in the visual perception stage, or in the spatial reasoning stage? This paper only provides two case studies of GPT-5.5 in Appendix F, but lacks systematic error classification and quantitative analysis. For benchmark testing, understanding the failure modes of the model is crucial for guiding subsequent research, but the paper's analysis in this regard is obviously insufficient.
The evaluation includes pedestrian-network adherence, closure compliance, and pedestrian collisions alongside task success and answer accuracy. Appendix I provides a quantitative classification of GPT-5.5’s LongNav failures, covering premature stopping, failure to establish substantial progress, loss of earlier progress, and incomplete progress at timeout. This analysis extends beyond the individual cases in Appendix F.
We report the controlled interventions on spatial-state maintenance and route planning in our response to W1, and discuss distance-stratified and partial-progress analyses in our response to W2. These interventions examine how restoring spatial state or supplying a verified route affects navigation.
W5: Limited model coverage, lack of evaluation of open-source small models. Table 2 shows that the evaluated models are mainly concentrated on commercial closed-source models such as the GPT series, Claude series, and Gemini series, as well as a small number of open-source models like GLM-5V-Turbo and Kimi-K3. However, the paper does not evaluate some open-source models widely used in the community (such as LLaVA, Qwen-VL, etc.), nor does it discuss the impact of model scale on performance. In addition, the paper fails to provide a comparison of different models in terms of computing cost, inference latency, etc., which is valuable for model selection in application scenarios. More critically, the paper does not evaluate the performance of the model under limited computing resources, which is crucial for promoting the development of this field in resource-constrained environments.
Smaller open-source models show the same gap between local perception and long-range navigation as the twelve models in our paper. We aim to evaluate the native spatial ability of MLLM agents fairly, so every model runs under one fixed protocol. To further cover open models widely used in the community, we evaluate Qwen3-VL-32B-Instruct and LLaVA-OneVision-2-8B-Instruct on all 810 instances. Both score below the twelve models overall.
| Success Rate | VR | OU | AEQ | Short | Long | Instr. | Constr. | Search | Intent | Time | Multi | Closure | Ped. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-32B-Instruct | 60.0 | 30.0 | 56.3 | 26.3 | 0.0 | 4.0 | 0.0 | 6.7 | 3.3 | 0.0 | 0.0 | 0.0 | 0.0 |
| LLaVA-OneVision-2-8B-Instruct | 56.3 | 23.3 | 47.5 | 32.5 | 0.0 | 4.0 | 0.0 | 11.7 | 1.7 | 0.0 | 0.0 | 0.0 | 0.0 |
| PNA | VR | OU | AEQ | Short | Long | Instr. | Constr. | Search | Intent | Time | Multi | Closure | Ped. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-32B-Instruct | 62.4 | 70.3 | 88.7 | 94.6 | 92.8 | 98.2 | 95.4 | 96.1 | 97.9 | 84.1 | 92.0 | 92.3 | 91.6 |
| LLaVA-OneVision-2-8B-Instruct | 63.6 | 71.5 | 89.8 | 95.9 | 94.6 | 98.9 | 96.9 | 97.4 | 98.5 | 86.3 | 93.4 | 93.8 | 93.5 |
We run both open models in BF16 on a single H200 with greedy decoding, while the commercial models are queried through their APIs. LLaVA-OneVision-2-8B-Instruct takes a median of 3.8 s per step, between Gemini-3.8-Flash at 3.7 s and GPT-5.6-Luna at 4.6 s. Qwen3-VL-32B-Instruct takes 7.6 s per step with no gain in navigation.
Official Review of Submission19810 (Part 1)
We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.
The claim of UrbanGround concerns whether spatial information remains usable throughout continuous interaction. In Eq. 1, mt represents the working spatial state inferred from the interaction history, without assuming an explicit internal memory module. Our existing trajectory analysis offers more specific evidence. Agents frequently make substantial progress before losing it, while retrospective checkpoint evaluation recovers much of this progress without changing any executed actions. This indicates that long-range failure often emerges after the agent has established a useful direction.
To investigate this further, we evaluate 20 LongNav instances using GPT-5.6-Luna and Claude-Opus-5. We first divide each verified pedestrian route into 5 or 10 shorter segments. Agents must physically reach each checkpoint before receiving the next target, with the interaction context reset at each transition. End-to-end success improves as the route is divided into shorter segments, confirming that agents can execute navigation segments that they struggle to connect over an extended journey.
Since checkpointing also simplifies route planning, we further distinguish spatial-state maintenance from planning through two controlled interventions on the same LongNav instances. Oracle State periodically restores the agent's current orientation and traversed pedestrian segments without providing future route instructions. Oracle Route supplies a verified pedestrian path while leaving the agent responsible for tracking its own spatial history. The larger improvement from restoring spatial state shows that maintaining a consistent account of previous movement is a major limitation during long-range navigation.