acceptodds
Under review as a conference paper at ICLR 2027

UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

15 views

Abstract

Multimodal large language models (MLLMs) can interpret a street view, but reliable urban action depends on whether such local evidence remains useful after the agent starts to move. In this paper, we investigate how far current MLLM agents can turn local urban perception into reliable action in a real-scale city. We propose UrbanGround, an urban sandbox built from Hong Kong's territory-wide 3D geospatial data. It combines the city's geographic structure with continuous, collision-constrained control through a shared evaluation interface. Agents use first-person observations and an interactive map to select actions across tasks ranging from local question answering to long-horizon navigation. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an agent can gather and interpret local visual evidence to answer spatial questions. Then we ask whether these abilities support navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. MLLM agents usually show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far MLLM agents can explore reliably in open-ended urban environments. Code and data are available at *anonymity*.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multimodal Large Language Models, Spatial Reasoning, City-Scale Simulation, Long-Horizon Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_qwaWOpenReview ↗
Rating
4(+1)
Confidence
3(+1)
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary
  • This paper proposes UrbanGround, a large-scale urban interaction simulation and evaluation platform built based on real 3D geographic data of Hong Kong. The platform connects real urban geometry, physical collision, pedestrian networks, dynamic pedestrians, weather systems and the closed-loop actions of multimodal large models, and evaluates Spatial Agency through 5 levels and 13 categories of tasks. The authors evaluated multiple mainstream MLLMs on 810 manually verified basic test instances, and found that the models have strong local visual recognition capabilities, but their performance drops significantly in direction judgment, long-term navigation, multi-objective planning and dynamic obstacle adaptation.

  • I think this benchmark paper bears important research questions, high platform engineering value, but the core scientific argumentation is not yet sufficient. Its most valuable contribution is establishing a relatively realistic urban interaction environment and systematically demonstrating the capability gap of current MLLMs from local understanding to long-term autonomous action.

  • However, the core argument of the paper - that models struggle to maintain long-term spatial states - is mainly inferred indirectly through task success rates and navigation trajectories, lacking control experiments that can rule out other failure factors.

  • In addition, the success rate of a large number of high-difficulty tasks is close to zero, causing an obvious evaluation metric saturation problem, which makes it difficult for the benchmark to further distinguish the real capabilities of the models.

Critical strengths
  1. A physically interactive evaluation environment (simulated) at the real city scale is proposed. UrbanGround integrates real urban 3D geographic data, navigation networks and continuous movement action space, and also adds simulations of conditions such as lighting in different time periods.
  2. A progressive evaluation from local perception to long-term autonomous spatial action is proposed. The task design is gradually expanded: direction judgment, active exploration, long-distance navigation, multi-task planning and dynamic adaptation. The design of this evaluation task aims to answer the question: At which stage exactly does the model start to fail?
  3. The paper reveals the compositionality problem in spatial ability of MLLMs. The study finds that the models’ ability to complete target recognition, direction understanding or local movement alone does not mean that it can effectively combine these abilities for long-distance navigation. The accumulation of small local errors may lead to the failure of the entire task.
  4. The main contributions of UrbanGround are the simulation platform, benchmark and systematic experimental analysis. No new algorithm is proposed.
Critical weaknesses
  1. The core scientific claim lacks direct evidence.  The paper defines mtm_t in Eq. 1 of Section 3.1 to represent the workspace state maintained by the model during interaction. However, the authors also explicitly point out that this state is an analytical abstraction, which neither assumes the existence of an explicit memory module in the model nor is directly observed by the evaluator. Therefore, I believe there is an obvious argumentation gap in the current paper: the paper claims to study the model's ability from local perception to sustained spatial intelligence, but it does not fully distinguish between the failure of long-term spatial state maintenance and other factors that lead to navigation failure. The good performance in short-distance navigation and poor performance in long-distance navigation indeed indicate that the model's performance degrades during long-range interaction. However, this does not prove that insufficient spatial memory is the main cause.

  2. Extremely low task success rate indicate that model performance Lacks practical implications. As shown in Table 2, all models generally have an extremely low success rate in core tasks such as Long-distance Navigation (LGN), Instruction Navigation (IN), and Constrained Navigation (CN) and others. For example, the best-performing GPT-5.6-Luna only has a 5.0% success rate on the LGN task and a mere 3.3% on the CN task, while GPT-5.5 even achieves a 0.0% success rate on LGN. Such an extremely low success rate indicates that the spatial capabilities of current models in real urban environments are far from reaching a usable level. Although the paper may focus on the construction of the benchmark test, such a low baseline performance calls into question the discriminative power and applicability of this benchmark. If all models can barely complete the tasks, then whether the benchmark can effectively measure the differences in model capabilities remains a question.

  3. Lack of reasonable justification for task difficulty settings. This paper claims that all tasks have been manually verified to be completable within the 150-step limit, but the experimental results show that the vast majority of models almost completely failed on long-distance tasks. This suggests that the task difficulty may be set too high, or there is a systematic difference between the verification process and the actual execution of the model. The paper does not provide human performance data to justify the task difficulty. If humans also cannot complete these tasks within 150 steps, the difficulty setting of this benchmark becomes less persuasive. In addition, this paper does not discuss whether the progress of task difficulties between levels is reasonable: whether the difficulty increase from Level 1 to Level 5 is uniform, and whether there are large jumps between some levels.

  4. Narrow evaluation metric, lacking analysis of failure modes. This paper mainly reports the task success rate and answer accuracy, but lacks in-depth analysis of the model's failure modes. For example, does the model fail during the path planning stage, or deviate from the route during the execution stage? Does it make mistakes in the visual perception stage, or in the spatial reasoning stage? This paper only provides two case studies of GPT-5.5 in Appendix F, but lacks systematic error classification and quantitative analysis. For benchmark testing, understanding the failure modes of the model is crucial for guiding subsequent research, but the paper's analysis in this regard is obviously insufficient.

  5. Limited model coverage, lack of evaluation of open-source small models. Table 2 shows that the evaluated models are mainly concentrated on commercial closed-source models such as the GPT series, Claude series, and Gemini series, as well as a small number of open-source models like GLM-5V-Turbo and Kimi-K3. However, the paper does not evaluate some open-source models widely used in the community (such as LLaVA, Qwen-VL, etc.), nor does it discuss the impact of model scale on performance. In addition, the paper fails to provide a comparison of different models in terms of computing cost, inference latency, etc., which is valuable for model selection in application scenarios. More critically, the paper does not evaluate the performance of the model under limited computing resources, which is crucial for promoting the development of this field in resource-constrained environments.

  6. Benchmark is geographically quite limited, and hard to scale. This paper only used the single region of Hong Kong for the benchmarking, and additional geographic regions are necessary to verify its generalization performance. The title is UrbanGround, but Hong Kong is a rather unique city among urban categories. The cost and conditions of creating such a benchmark should be acknowledged so that other researchers are aware of the difficulty of replicating it across cities, and be marked as a limitation that this benchmark is difficult to scale.

Questions for authors
  1. If the authors could provide additional evidence to more directly support their claim that the presented navigation failures are mainly due to spatial memory, the paper becomes much better backed up. Control experiments are appreciated.

  2. No ground to judge how difficult are some of the tasks, especially long-range navigation ones. Human performance can provide an important baseline. Besides, by providing simple modular agents, e.g., visual localization + topological path planning + low-level motion control. The purpose here is not to defeat general-purpose MLLMs, but to judge whether a navigation system with a simple structure can already solve some tasks that general models fail to complete. Or by providing memory or prompt strategies for MLLMs, e.g., periodic history summary, explicit landmark lists, etc. If these simple strategies can significantly improve the navigation success rate, then the paper's explanation for the insufficient native spatial capabilities of the model needs to be further refined. Conversely, if these strategies are still ineffective, it can also strengthen the author's argument about the difficulty of long-term spatial intelligence. These baselines can substaintially strengthen the paper.

  3. The extremely low task success rate leads to three problems. First, it is difficult to distinguish model capabilities. When two models achieve success rates of 0% and 1.3% respectively, it is hard to determine whether there is a meaningful performance difference. Second, it is difficult to apply to subsequent algorithm research. If a new method achieves a 2% success rate on a certain task while the baseline is 0%, this does not necessarily prove that the method has a stable advantage. Finally, it is difficult to locate which capability the improvement comes from. The model may have walked correctly for a long distance, but is still recorded as a failure because it cannot complete the task in the last few meters. The backtracking checkpoint analysis in Appendix D of the paper has a certain mitigating effect, but it is not sufficient to completely solve the problem. I suggest that the authors supplement the rationality argument for this task setting, and preferably add intermediate-level tasks between simple tasks and extremely difficult tasks, so as to make the tasks more practically meaningful.

If the authors carefully incorporate these suggestions and overcome some of the major weaknesses, I will consider raising the score to 4.

13 comments
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 1)

We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.


W1: The core scientific claim lacks direct evidence. The paper defines mtm_t in Eq. 1 of Section 3.1 to represent the workspace state maintained by the model during interaction. However, the authors also explicitly point out that this state is an analytical abstraction, which neither assumes the existence of an explicit memory module in the model nor is directly observed by the evaluator. Therefore, I believe there is an obvious argumentation gap in the current paper: the paper claims to study the model's ability from local perception to sustained spatial intelligence, but it does not fully distinguish between the failure of long-term spatial state maintenance and other factors that lead to navigation failure. The good performance in short-distance navigation and poor performance in long-distance navigation indeed indicate that the model's performance degrades during long-range interaction. However, this does not prove that insufficient spatial memory is the main cause.

The claim of UrbanGround concerns whether spatial information remains usable throughout continuous interaction. In Eq. 1, mtm_t represents the working spatial state inferred from the interaction history, without assuming an explicit internal memory module. Our existing trajectory analysis offers more specific evidence. Agents frequently make substantial progress before losing it, while retrospective checkpoint evaluation recovers much of this progress without changing any executed actions. This indicates that long-range failure often emerges after the agent has established a useful direction.

To investigate this further, we evaluate 20 LongNav instances using GPT-5.6-Luna and Claude-Opus-5. We first divide each verified pedestrian route into 5 or 10 shorter segments. Agents must physically reach each checkpoint before receiving the next target, with the interaction context reset at each transition. End-to-end success improves as the route is divided into shorter segments, confirming that agents can execute navigation segments that they struggle to connect over an extended journey.

Model0 Checkpoints5 Checkpoints10 Checkpoints
GPT-5.6-Luna0.015.025.0
Claude-Opus-55.015.030.0

Since checkpointing also simplifies route planning, we further distinguish spatial-state maintenance from planning through two controlled interventions on the same LongNav instances. Oracle State periodically restores the agent's current orientation and traversed pedestrian segments without providing future route instructions. Oracle Route supplies a verified pedestrian path while leaving the agent responsible for tracking its own spatial history. The larger improvement from restoring spatial state shows that maintaining a consistent account of previous movement is a major limitation during long-range navigation.

ModelOriginal+ Oracle State+ Oracle Route+ Both
GPT-5.6-Luna0.025.010.025.0
Claude-Opus-55.030.010.040.0
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 2)


W2: Extremely low task success rate indicate that model performance Lacks practical implications. As shown in Table 2, all models generally have an extremely low success rate in core tasks such as Long-distance Navigation (LGN), Instruction Navigation (IN), and Constrained Navigation (CN) and others. For example, the best-performing GPT-5.6-Luna only has a 5.0% success rate on the LGN task and a mere 3.3% on the CN task, while GPT-5.5 even achieves a 0.0% success rate on LGN. Such an extremely low success rate indicates that the spatial capabilities of current models in real urban environments are far from reaching a usable level. Although the paper may focus on the construction of the benchmark test, such a low baseline performance calls into question the discriminative power and applicability of this benchmark. If all models can barely complete the tasks, then whether the benchmark can effectively measure the differences in model capabilities remains a question.

The low success rates on challenging navigation tasks show a substantial gap between current MLLMs and the spatial capabilities required for reliable urban exploration. UrbanGround is designed to measure this gap. The benchmark also distinguishes model capabilities through clear performance differences in shorter navigation tasks, while the distance-stratified and partial-progress analyses provide further evidence of differences even when task completion rates are close to zero.

All 810 tasks were completed by human annotators under the same interaction interface, within the 150-step limit, and further verified in a second round. It means that the low model success rates cannot be attributed to infeasible task instances. The large gap between human and model performance points to fundamental weaknesses in sustained spatial reasoning. Figure 9 illustrates this issue, where the agent repeatedly attempts to move forward after encountering an obstacle, which provides directions for improving future models toward reliable long-horizon urban navigation.


W3: Lack of reasonable justification for task difficulty settings. This paper claims that all tasks have been manually verified to be completable within the 150-step limit, but the experimental results show that the vast majority of models almost completely failed on long-distance tasks. This suggests that the task difficulty may be set too high, or there is a systematic difference between the verification process and the actual execution of the model. The paper does not provide human performance data to justify the task difficulty. If humans also cannot complete these tasks within 150 steps, the difficulty setting of this benchmark becomes less persuasive. In addition, this paper does not discuss whether the progress of task difficulties between levels is reasonable: whether the difficulty increase from Level 1 to Level 5 is uniform, and whether there are large jumps between some levels.

The five levels are designed to examine distinct spatial capabilities, without assuming a uniform increase in difficulty. Each level introduces a new requirement, progressing from local grounding to sustained navigation, goal inference, multi-goal planning, and adaptation to environmental changes. Sharp performance drops between levels are therefore meaningful, which reveal where previously acquired capabilities cease to support successful execution.

We further compare success rates across levels to assess whether model failures arise from excessive task difficulty. Human performance reflects the verified manual completions, while the naive baseline uses random guessing for multiple-choice questions and greedy distance-reducing movement for navigation. The comparison establishes human feasibility across all levels and shows that current MLLMs struggle to sustain reliable execution as additional spatial capabilities become necessary.

ModelL1L2L3L4L5
GPT-5.6-Luna63.232.518.31.70.0
Claude-Opus-579.123.820.83.31.8
Human100.0100.0100.0100.0100.0
Naive Baseline24.54.60.80.00.0
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 3)


W4: Narrow evaluation metric, lacking analysis of failure modes. This paper mainly reports the task success rate and answer accuracy, but lacks in-depth analysis of the model's failure modes. For example, does the model fail during the path planning stage, or deviate from the route during the execution stage? Does it make mistakes in the visual perception stage, or in the spatial reasoning stage? This paper only provides two case studies of GPT-5.5 in Appendix F, but lacks systematic error classification and quantitative analysis. For benchmark testing, understanding the failure modes of the model is crucial for guiding subsequent research, but the paper's analysis in this regard is obviously insufficient.

The evaluation includes pedestrian-network adherence, closure compliance, and pedestrian collisions alongside task success and answer accuracy. Appendix I provides a quantitative classification of GPT-5.5’s LongNav failures, covering premature stopping, failure to establish substantial progress, loss of earlier progress, and incomplete progress at timeout. This analysis extends beyond the individual cases in Appendix F.

We report the controlled interventions on spatial-state maintenance and route planning in our response to W1, and discuss distance-stratified and partial-progress analyses in our response to W2. These interventions examine how restoring spatial state or supplying a verified route affects navigation.


W5: Limited model coverage, lack of evaluation of open-source small models. Table 2 shows that the evaluated models are mainly concentrated on commercial closed-source models such as the GPT series, Claude series, and Gemini series, as well as a small number of open-source models like GLM-5V-Turbo and Kimi-K3. However, the paper does not evaluate some open-source models widely used in the community (such as LLaVA, Qwen-VL, etc.), nor does it discuss the impact of model scale on performance. In addition, the paper fails to provide a comparison of different models in terms of computing cost, inference latency, etc., which is valuable for model selection in application scenarios. More critically, the paper does not evaluate the performance of the model under limited computing resources, which is crucial for promoting the development of this field in resource-constrained environments.

Smaller open-source models show the same gap between local perception and long-range navigation as the twelve models in our paper. We aim to evaluate the native spatial ability of MLLM agents fairly, so every model runs under one fixed protocol. To further cover open models widely used in the community, we evaluate Qwen3-VL-32B-Instruct and LLaVA-OneVision-2-8B-Instruct on all 810 instances. Both score below the twelve models overall.

Success RateVROUAEQShortLongInstr.Constr.SearchIntentTimeMultiClosurePed.
Qwen3-VL-32B-Instruct60.030.056.326.30.04.00.06.73.30.00.00.00.0
LLaVA-OneVision-2-8B-Instruct56.323.347.532.50.04.00.011.71.70.00.00.00.0
PNAVROUAEQShortLongInstr.Constr.SearchIntentTimeMultiClosurePed.
Qwen3-VL-32B-Instruct62.470.388.794.692.898.295.496.197.984.192.092.391.6
LLaVA-OneVision-2-8B-Instruct63.671.589.895.994.698.996.997.498.586.393.493.893.5

We run both open models in BF16 on a single H200 with greedy decoding, while the commercial models are queried through their APIs. LLaVA-OneVision-2-8B-Instruct takes a median of 3.8 s per step, between Gemini-3.8-Flash at 3.7 s and GPT-5.6-Luna at 4.6 s. Qwen3-VL-32B-Instruct takes 7.6 s per step with no gain in navigation.

Reviewer_yyYLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UrbanGround, a city-scale simulation and evaluation framework for studying the spatial agency of multimodal large language model (MLLM) agents. Built upon real-world 3D geospatial data from Hong Kong, the framework integrates continuous first-person movement, physical collision constraints, a georeferenced pedestrian network, and controllable environmental conditions. The authors construct a five-level evaluation benchmark comprising 13 task categories and 810 manually verified instances, covering local visual understanding, navigation, multi-task planning, and adaptation to environmental changes. Experiments across 12 MLLMs reveal a substantial gap between local visual perception and sustained goal-directed behavior, particularly in long-horizon navigation and dynamic environments.

Critical strengths
  1. A valuable integration of real-world urban geometry and physically grounded interaction. UrbanGround provides a useful environment for evaluating MLLM agents in geographically realistic, large-scale urban settings.

  2. A comprehensive and well-structured evaluation framework. The five-level evaluation ladder systematically increases task complexity from local perception to long-horizon navigation and dynamic adaptation.

Critical weaknesses
  1. Despite extensive experiments, the main findings largely reiterate known limitations in embodied navigation, such as the perception-action gap, error accumulation, and weak spatial memory and recovery, as previously observed in CitySeeker and VLN-MME. It remains unclear what fundamentally new insights arise from the physically grounded urban setting. Moreover, the analysis is mostly descriptive, without disentangling failures in localization, memory, planning, and control. Near-zero success rates on complex tasks also introduce floor effects, limiting meaningful comparisons. The trajectory analysis is interesting but insufficient to establish the underlying failure mechanisms.

  2. The reconstructed 3D scenes exhibit uneven visual quality, yet the paper lacks quantitative fidelity assessments or analysis of how reconstruction artifacts affect evaluation reliability. The benchmark is also restricted to Hong Kong, with limited evidence of cross-city generalization or environment extensibility. Furthermore, the experiments underexploit distinctive 3D urban characteristics, such as elevation changes, overpasses, and pedestrian-network topology, which could yield more unique insights. The reproducibility and modifiability of the complete Unity environment also remain unclear. These limitations weaken the broader applicability and research value of the proposed platform.

Questions for authors
  1. What evidence supports spatial-state persistence as the primary cause of long-horizon navigation failures? Could the authors provide controlled diagnostic experiments, potentially on a representative subset, that separately examine localization, spatial memory, route planning, and action execution? For example, comparing performance with oracle localization or route guidance would help determine which factors primarily explain the observed failures.

  2. What unique insights arise from the real-world 3D urban topology? Could the authors analyze navigation performance with respect to topological complexity, elevation changes, pedestrian-network connectivity, or the ratio between shortest feasible walking distance and Euclidean distance? Such analysis would help establish the additional scientific value of UrbanGround beyond existing urban navigation benchmarks.

  3. How are environment fidelity and extensibility validated? Could the authors clarify how geometric and visual reconstruction quality is assessed, whether scene artifacts affect the validity of benchmark results, and which components of the complete Unity environment are available for modification and reproduction? Evidence of portability to another city, or a concrete description of the requirements for such extension, would strengthen the platform's general applicability.

3 comments
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 1)

We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.


W1: Despite extensive experiments, the main findings largely reiterate known limitations in embodied navigation, such as the perception-action gap, error accumulation, and weak spatial memory and recovery, as previously observed in CitySeeker and VLN-MME. It remains unclear what fundamentally new insights arise from the physically grounded urban setting. Moreover, the analysis is mostly descriptive, without disentangling failures in localization, memory, planning, and control. Near-zero success rates on complex tasks also introduce floor effects, limiting meaningful comparisons. The trajectory analysis is interesting but insufficient to establish the underlying failure mechanisms.

UrbanGround also examines whether agents can find a route that the city physically allows. CitySeeker and VLN-MME move agents between sampled viewpoints, where each candidate move is already walkable. In our setting, agents often identify the goal direction but fail to find such a route. To measure this effect, we group the existing ShortNav runs by the ratio of the shortest walking distance on the Lands Department 3D Pedestrian Network to the straight-line distance. Success drops once this ratio exceeds 1.2 at matched start-goal distance, showing that the walkable structure of the city limits navigation even when the goal is visible from the start.

Start-Goal DistanceSR at Ratio ≤ 1.2SR at Ratio > 1.2
< 60 m64.954.2
≥ 60 m42.524.2

To separate localization and memory, we evaluate two oracle interventions on 20 LongNav instances. Oracle Localization restores the agent's current pose. Oracle Memory restores the routes the agent has already traversed. Besides success, we report the substantial-progress rate of Appendix I, which stays well above zero when success is near the floor. The results show that memory brings the largest gain while localization and route guidance add little, confirming that long-range failure comes mainly from keeping track of past movement.

ModelMetricOriginal+ Localization+ Memory
GPT-5.6-LunaSR0.00.010.0
GPT-5.6-LunaProgress40.040.065.0
Claude-Opus-5SR5.010.025.0
Claude-Opus-5Progress45.050.065.0

W2 & Q3: The reconstructed 3D scenes exhibit uneven visual quality, yet the paper lacks quantitative fidelity assessments or analysis of how reconstruction artifacts affect evaluation reliability. The benchmark is also restricted to Hong Kong, with limited evidence of cross-city generalization or environment extensibility. Furthermore, the experiments underexploit distinctive 3D urban characteristics, such as elevation changes, overpasses, and pedestrian-network topology, which could yield more unique insights. The reproducibility and modifiability of the complete Unity environment also remain unclear. These limitations weaken the broader applicability and research value of the proposed platform.

UrbanGround's environment fidelity is grounded in official geospatial data and verified through executable tasks. The same geometry supports visual rendering and physical collision in Unity. We assess visual quality through human inspection of task-relevant landmarks and geometric reliability through actual route execution. All 810 tasks underwent two rounds of human verification, ensuring that the required visual evidence was accessible and the tasks could be completed within 150 steps. To further examine geometric consistency, we manually audited 81 sampled locations against the official 3D Pedestrian Network. The results show 97.5% network coverage and 93.8% agreement with human judgments.

UrbanGround separates geographic assets from the agent evaluation protocol to support extension across cities. The Unity applications support reproducible execution on Windows, macOS, and Linux. Task definitions, agent interaction code, and evaluation scripts are available for modification.

To further validate the scalability, we integrated Kyoto's 2024 3D city model from Project PLATEAU. We constructed 80 short-range navigation tasks with distances matched to 80 Hong Kong instances, retaining the same evaluation protocol. The five evaluated models show consistent performance trends across both cities.

ModelHong Kong SRKyoto SR
GPT-5.6-Luna81.388.8
GPT-5.575.081.3
Claude-Opus-4.675.080.0
Kimi-K367.568.8
GLM-5V-Turbo50.058.8
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 2)


Q1: What evidence supports spatial-state persistence as the primary cause of long-horizon navigation failures? Could the authors provide controlled diagnostic experiments, potentially on a representative subset, that separately examine localization, spatial memory, route planning, and action execution? For example, comparing performance with oracle localization or route guidance would help determine which factors primarily explain the observed failures.

The claim of UrbanGround concerns whether spatial information remains usable throughout continuous interaction. Our existing trajectory analysis offers more specific evidence. Agents frequently make substantial progress before losing it, while retrospective checkpoint evaluation recovers much of this progress without changing any executed actions. This indicates that long-range failure often emerges after the agent has established a useful direction.

To investigate this further, we evaluate 20 LongNav instances using GPT-5.6-Luna and Claude-Opus-5. We first divide each verified pedestrian route into 5 or 10 shorter segments. Agents must physically reach each checkpoint before receiving the next target, with the interaction context reset at each transition. End-to-end success improves as the route is divided into shorter segments, confirming that agents can execute navigation segments that they struggle to connect over an extended journey.

Model0 Checkpoints5 Checkpoints10 Checkpoints
GPT-5.6-Luna0.015.025.0
Claude-Opus-55.015.030.0

Since checkpointing also simplifies route planning, we further distinguish spatial-state maintenance from planning through two controlled interventions on the same LongNav instances. Oracle State periodically restores the agent's current orientation and traversed pedestrian segments without providing future route instructions. Oracle Route supplies a verified pedestrian path while leaving the agent responsible for tracking its own spatial history. The larger improvement from restoring spatial state shows that maintaining a consistent account of previous movement is a major limitation during long-range navigation. Our response to W1 tests separate oracles for localization and memory on these instances, and memory of the traversed segments accounts for most of this gain.

ModelOriginal+ Oracle State+ Oracle Route+ Both
GPT-5.6-Luna0.025.010.025.0
Claude-Opus-55.030.010.040.0

Q2: What unique insights arise from the real-world 3D urban topology? Could the authors analyze navigation performance with respect to topological complexity, elevation changes, pedestrian-network connectivity, or the ratio between shortest feasible walking distance and Euclidean distance? Such analysis would help establish the additional scientific value of UrbanGround beyond existing urban navigation benchmarks.

Navigation performance depends more on how far the walkable route departs from the straight line to the goal than on how many junctions it passes. We measure the four properties raised by the reviewer on the shortest walking route of ShortNav and LongNav instances in the Lands Department 3D Pedestrian Network. For each property, we sort the 80 instances and compare the bottom and top 20%. Elevation change totals the vertical rise and fall along this route, while connectivity counts closed loops per square kilometer of the surrounding network. Detours lower performance on both tasks, with the largest LongNav drop coming from elevation change. Better connectivity raises LongNav progress, while more junctions lower only ShortNav success.

Route PropertyShortNav SR LowShortNav SR HighLongNav Progress LowLongNav Progress High
Detour ratio53.639.139.627.1
Elevation change44.844.336.523.4
Network connectivity44.344.825.534.4
Junctions along route47.938.530.729.7

Agents chase geometric proximity once the walkable route turns away from the goal. For every move of GPT-5.6-Luna, we check on the 3D Pedestrian Network whether it brings the agent closer to the goal in a straight line without shortening its remaining walk. Moves stay rare when the route points toward the goal, but grow on ShortNav once it turns more than 60° away.

ModelTaskRoute Within 30° of Goal DirectionRoute Beyond 60° of Goal Direction
GPT-5.6-LunaShortNav6.644.0
GPT-5.6-LunaLongNav8.613.9

We hope that our explanations above can clarify your doubts and you can consider our work more favorably.

Dear Reviewer,

Thank you for your thoughtful feedback. With only a few hours left in the discussion period, we hope our responses have addressed your concerns and would be glad to discuss any remaining questions.

Thank you again for your time.

Best,

Authors

Reviewer_U8W6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an urban sandbox built from Hong Kong's territory-wide 3D geospatial data. The sandbox combines the city's geographic structure with continuous, collision-constrained control through a shared evaluation interface. Agents use first-person observations and an interactive map to select actions across five task levels, ranging from local question answering to long-horizon navigation. Experiments show visual recognition accuracy of 75.0–93.8%, while orientation understanding is poor and long-range navigation collapses. Agents frequently jaywalk to finish faster, revealing that pedestrian-network constraints are not consistently maintained in spatial reasoning when they compete with rapid task completion.

Critical strengths
  1. One of the most notable contributions is the clear definition of three research questions: whether MLLM agents can establish usable local spatial grounding, how local grounding scales to goal-directed navigation, and how changes in the city affect grounded perception and action. The questions are well defined and make the experimental evaluations easy to follow.
  2. Another notable contribution is the realistic environmental foundation, which is difficult to replicate. The sandbox couples Hong Kong's territory-wide 3D geographic data with continuous, physically collision-constrained first-person movement. Unlike street-view-based urban benchmarks that provide pre-sampled viewpoints without continuous physical interaction, each action produces the next frame, while buildings and terrain constrain movement. The setting, therefore, directly represents the sustained action the paper aims to evaluate.
Critical weaknesses
  1. I find the paper's motivation insignificant and confusing, and its main contributions difficult to identify. What does the paper aim to achieve? Is the goal to provide a more realistic benchmark, with the 3D environment as the main contribution, or to introduce new research questions based on that environment? If the contribution lies in the research questions, the paper needs to compare those questions with the questions addressed by existing benchmarks.
  2. The experiments are unclear and confusing. In particular, the paper needs the comparison mentioned above: an evaluation of the same models on comparable tasks under existing benchmark settings, such as teleporting between sampled viewpoints or static single-image question answering, to show how performance differs from the continuous setting. The authors should explain the reasons for the performance differences and demonstrate why the proposed setting is better. I also question the generality of the reported collapse in long-range navigation, because the paper does not compare different prompting or scaffolding strategies, such as chain-of-thought or self-reflection.
Questions for authors
  1. Do the authors consider it necessary to report a human upper bound and naive lower-bound baselines, beyond the human task-feasibility verification already described? For example, judging how poor a pedestrian-collision rate of 76.3–90.0% is requires a reference point.
  2. I am sceptical about the PNA metric. How complete is the pedestrian network, and could unregistered but legally walkable areas be misclassified as violations? What spatial tolerance is used? Specifically, what distance threshold determines whether an agent's position counts as being on the pedestrian network, and how sensitive are the reported PNA scores to that threshold?
5 comments
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 1)

We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.


W1: I find the paper's motivation insignificant and confusing, and its main contributions difficult to identify. What does the paper aim to achieve? Is the goal to provide a more realistic benchmark, with the 3D environment as the main contribution, or to introduce new research questions based on that environment? If the contribution lies in the research questions, the paper needs to compare those questions with the questions addressed by existing benchmarks.

UrbanGround aims to evaluate whether MLLMs can turn their understanding of urban scenes into reliable action in a real city. A destination may appear close on the map yet require a long detour through an elevated footbridge. An agent may recognize its location correctly but repeatedly move toward an inaccessible path.

VELMA [1], V-IRL [2], CitySeeker [3], and CityNav [4] preserve real geographic context but rely on movement between sampled viewpoints. VLN-MME [5] examines sequential navigation through a simulation-free evaluation framework, while MetaUrban [6], SimWorld-Robotics [7], and Virtual Community [8] support physical interaction in simulated urban environments. Their generated urban assets can produce repetitive building layouts that fail to capture the irregular geometry of real cities, making it difficult to assess how MLLM agents would perform in such complex environments. Overall, these settings do not directly evaluate MLLMs navigating continuously through the actual 3D geometry of a real city, where each movement changes the next observation and the pedestrian network determines which routes are physically feasible.

Our RQs examine whether local spatial understanding can support sustained navigation and adapt when the environment changes. In Figure 8, the agent successfully follows a footbridge to continue toward its destination. In Figure 9, the same model moves toward the goal but becomes trapped by a physical obstacle, repeatedly attempting to move forward without revising its route. The agent can estimate the goal direction but fails to maintain a feasible path through the city. UrbanGround allows us to examine such failures during continuous interaction with real urban geometry, providing a more realistic assessment of current MLLMs' ability to act in complex urban environments.


W2: The experiments are unclear and confusing. In particular, the paper needs the comparison mentioned above: an evaluation of the same models on comparable tasks under existing benchmark settings, such as teleporting between sampled viewpoints or static single-image question answering, to show how performance differs from the continuous setting. The authors should explain the reasons for the performance differences and demonstrate why the proposed setting is better. I also question the generality of the reported collapse in long-range navigation, because the paper does not compare different prompting or scaffolding strategies, such as chain-of-thought or self-reflection.

We randomly sampled 20 ShortNav and 20 LongNav instances and re-ran them under two discrete protocols that mirror existing benchmark settings. The static condition shows the opening view alone and asks a four-option question about which street leads toward the goal. The teleport condition offers four candidate viewpoints on the pedestrian network at each step and moves the agent to its pick.

ModelStatic QA accShort teleport SRShort continuous SRLong teleport SRLong continuous SR
GPT-5.6-Luna90.080.075.00.00.0
Claude-Opus-595.070.050.010.05.0

Short routes fall from static to teleport to continuous. Long routes stay near zero under teleport as well, so removing continuous control leaves the collapse intact.

The evaluated models already run with reasoning enabled, so each action follows an explicit think-before-act step, chain-of-thought in its working form. Holding the scaffold fixed and minimal keeps the score attached to the model rather than to the prompt.

AuthorsOpenReview ↗

Official Review of Submission19810 (Part 2)


Q1: Do the authors consider it necessary to report a human upper bound and naive lower-bound baselines, beyond the human task-feasibility verification already described? For example, judging how poor a pedestrian-collision rate of 76.3–90.0% is requires a reference point.

All 810 task instances were completed by human annotators using the same constraints. We verified each task was solvable within the 150-step limit and checked it in a second round. The human success rate on these verified instances is therefore 100%.

We also recorded pedestrian-network adherence and collision rates for human participants in the dynamic tasks. Even without explicit instructions to avoid pedestrians, humans achieved substantially lower collision rates and better pedestrian-network adherence than the models. Current MLLMs struggle to moving pedestrians during navigation.

For the lower bound, we used random guessing for the multiple-choice QA tasks and a greedy agent for navigation. The greedy agent simply moves in the direction that directly reduces its distance to the goal. Its success rates were also very low, providing a reference for assessing the performance of current models.

Task Success Rate:

ModelVROUAEQShortLongInstr.Constr.SearchIntentTimeMultiClosurePed.
GPT-5.6-Luna78.848.358.881.35.016.03.326.710.01.71.70.00.0
GPT-5.582.540.062.575.00.020.00.035.011.73.30.00.00.0
GPT-5.475.031.757.515.01.320.00.015.011.71.70.00.00.0
GPT-5.277.533.348.825.00.028.00.08.315.01.73.30.01.3
Claude-Opus-591.358.382.548.82.530.03.330.011.73.33.30.02.5
Claude-Opus-4.685.046.773.875.01.328.06.736.715.01.71.73.33.8
Gemini-3.8-Flash92.558.380.045.00.014.00.023.35.00.00.00.00.0
Gemini-3.6-Flash93.856.777.523.80.06.00.023.33.30.01.70.00.0
Gemini-3.1-Pro82.523.346.331.30.08.00.018.36.70.00.00.00.0
Doubao-Seed-2.0-Pro85.036.763.826.31.36.00.015.010.00.00.00.00.0
GLM-5V-Turbo80.036.752.550.00.018.00.025.015.00.00.00.00.0
Kimi-K392.555.076.367.53.842.06.731.715.00.00.03.32.5
Naive Bound27.523.322.512.50.02.00.01.70.00.00.00.00.0
Human (First-try)97.593.395.097.592.590.080.091.788.383.381.780.086.3
Human (Varified)100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0100.0

Dynamic Environment Performance:

ModelClosure SR$\uparrow$Closure PNA$\uparrow$CCR$\uparrow$Ped. SR$\uparrow$Ped. PNA$\uparrow$PCR$\downarrow$
GPT-5.6-Luna0.098.420.00.098.682.5
GPT-5.50.094.713.30.093.883.8
GPT-5.40.096.416.70.098.580.0
GPT-5.20.094.216.71.395.478.8
Claude-Opus-50.093.646.72.594.487.5
Claude-Opus-4.63.393.026.73.892.980.0
Gemini-3.8-Flash0.095.820.00.097.585.0
Gemini-3.6-Flash0.095.416.70.096.090.0
Gemini-3.1-Pro0.093.416.70.094.088.8
Doubao-Seed-2.0-Pro0.093.510.00.095.076.3
GLM-5V-Turbo0.093.320.00.095.982.5
Kimi-K33.395.140.02.590.678.8
Naive Bound0.071.416.70.069.292.5
Human (First-try)80.099.2100.086.398.537.5
Human100.099.2100.0100.098.537.5
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 3)


Q2: I am sceptical about the PNA metric. How complete is the pedestrian network, and could unregistered but legally walkable areas be misclassified as violations? What spatial tolerance is used? Specifically, what distance threshold determines whether an agent's position counts as being on the pedestrian network, and how sensitive are the reported PNA scores to that threshold?

PNA measures whether agents maintain valid pedestrian movement in a real city. We use Hong Kong's official 3D Pedestrian Network, which covers over 9,100 km and is compiled from government road records and field checks. To assess potential omissions, we randomly sampled 10% of the 810 tasks and manually inspected pedestrian-route coverage against the rendered city. The audit found that 79 locations were covered by registered pedestrian connections, with 76 automatic on-network judgments agreeing with human inspection.

For geometric evaluation, we measure the horizontal distance from the agent position after each action to the nearest 3D pedestrian centreline. A position within 6 m counts as on the network, which covers the width of walkways around their centrelines. PNA is weighted by executed action duration. We also re-score the GPT-5.5 trajectories with a 2 m vertical tolerance and then a stricter 4 m buffer. The vertical tolerance changes adherence slightly. The 4 m buffer lowers it by a similar margin in each walking task, so the comparison across tasks holds.

ValidationResult
Pedestrian network coverage79/81 (97.5%)
Agreement with human judgments76/81 (93.8%)
GPT-5.5AEQShortNavLongNav
6 m buffer91.496.994.3
6 m buffer + 2 m vertical tolerance89.695.091.9
4 m buffer + 2 m vertical tolerance78.883.580.6

We hope that our explanations above can clarify your doubts and you can consider our work more favorably.


[1] Schumann et al., VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View, 2024.

[2] Yang et al., V-IRL: Grounding Virtual Intelligence in Real Life, 2024.

[3] Wang et al., CitySeeker: How Do VLMs Explore Embodied Urban Navigation with Implicit Human Needs?, 2026.

[4] Dalal et al., Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge, 2026.

[5] Zhao et al., VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation Agents, 2026.

[6] Wu et al., MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility, 2025.

[7] Zhuang et al., SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration, 2025.

[8] Zhou et al., Virtual Community: An Open World for Humans, Robots, and Society, 2026.

Reviewer_9XBfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UrbanGround, a Unity-based urban benchmark built from Hong Kong's public 3D photogrammetric mesh and 3D pedestrian network. Unlike street-view panorama graphs or synthetic grids, agents control continuous collision-constrained movement alongside an interactive map (without route calculation) across 810 human-verified tasks spanning local QA, multi-scale navigation, and dynamic interventions (weather/lighting, road closures, pedestrians). Evaluating 12 MLLMs, the authors find reasonable performance on local recognition and short line-of-sight navigation, weak directional orientation, and near-zero success on multi-block, multi-stop, and dynamic navigation tasks.

Critical strengths
  1. Realistic 3D urban topology with continuous control. Using Hong Kong's real 3D mesh and multi-level pedestrian network introduces genuine physical traps (footbridges, underpasses, fenced medians) where greedy straight-line movement fails, testing map-to-street alignment much better than discrete street-view graphs or flat synthetic cities.

  2. Informative task ladder and trajectory diagnostics. Even where full navigation success hits 0%, the retrospective checkpoint analysis in Appendices G.1-I adds real value by showing that models often make substantial initial progress toward the goal before losing orientation and regressing later in the episode.

Critical weaknesses
  1. Severe floor effects confounded with an unablated agent harness. Seven of the 10 navigation and dynamic subtasks sit near 0% across all 12 models, yet there are no ablations on the 150-turn cap (where map open/pan/zoom/close actions eat into the movement budget), visual memory (Section 4.1 implies single-frame + text history, while Appendix A.3 mentions an optional frame buffer), or map heading cues, and no quantitative human baseline stats (mean steps, accuracy, adherence) on the street-level oblique aerial meshes.

  2. Prompt-metric mismatches and confounded dynamic setups. Section 4.2 claims agents sacrifice road compliance (PNA) during local QA to answer faster, but the Level 1 prompts (Appendix J.2) never tell agents to stay on sidewalks, the spatial buffer for PNA is undefined, and PNA is oddly lowest on Visual Recognition (~66%) where movement isn't needed. Similarly, Pedestrian Navigation (Table 4) runs on LongNav routes where static success is already ~0%, and reuses the LongNav prompt without mentioning pedestrians at all (Appendix J.6), making both the ~0% success and 76%-90% collision rates hard to interpret.

Questions for authors
  1. Visual history and map orientation cues: In the actual benchmark runs, how many past RGB screenshots (if any) were kept in the model's context window at turn t, or did models only see the current screenshot plus their prior text observation/reason JSON strings? Also, when the agent opens the interactive map, does the "You" marker display the agent's current facing direction/frustum, or only its (x, y) position?

  2. PNA definition and the VR vs. AEQ discrepancy: How exactly is "lies on the registered pedestrian network" computed geometrically in 3D continuous space (horizontal buffer radius and vertical tolerance around the Lands Department 3D polyline)? Why is PNA around 63%-70% on Visual Recognition, where locomotion is generally unnecessary, yet over 90% on Active Exploration and >95% on navigation tasks?

  3. Pedestrian and Road-Closure experimental setup: Can you confirm that NP uses the unmodified LGN prompt without mentioning pedestrian avoidance (Appendix J.6), and clarify whether PCR in Table 4 is the percentage of episodes with at least one pedestrian contact? Why were NP (and DCR, if also long-range) not evaluated on ShortNav-scale routes where models achieve 50%-80% baseline success, so that the marginal degradation from closures and moving pedestrians on goal completion would actually be measurable?

  4. Human baseline statistics and oblique-mesh artifacts: Since human testers completed all 810 instances within 150 steps, can you report their empirical distribution of step counts, map-action counts, PNA, and first-try QA/navigation accuracy? In addition, why do GPT-5.4 (15.0% vs. GPT-5.5's 75.0%) and Claude-Opus-5 (48.8% vs. Opus-4.6's 75.0%) drop so sharply on ShortNav despite strong QA scores -- is this driven by premature terminate calls outside the 15m radius, schema rejections, or getting stuck on street-level photogrammetric mesh geometry?

4 comments
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 1)

We thank the reviewer for the thoughtful feedback. Below we address the main points raised in this review.


W1: Severe floor effects confounded with an unablated agent harness. Seven of the 10 navigation and dynamic subtasks sit near 0% across all 12 models, yet there are no ablations on the 150-turn cap (where map open/pan/zoom/close actions eat into the movement budget), visual memory (Section 4.1 implies single-frame + text history, while Appendix A.3 mentions an optional frame buffer), or map heading cues, and no quantitative human baseline stats (mean steps, accuracy, adherence) on the street-level oblique aerial meshes.

The near-zero success rates are not due to the agent harness. We report the human statistics from the second verification round, where testers used the agents' interface on the same street-level meshes within the 150-step limit. Humans solve most instances on the first attempt.

Human statisticL1L2L3L4L5
Mean steps9.141.639.069.071.3
Map actions1.99.27.98.28.5
PNA87.198.899.598.598.7
First-try accuracy95.592.190.082.584.5

To test the harness directly, we vary its settings on 20 LongNav instances with GPT-5.6-Luna and Claude-Opus-5. We first raise the step cap to 300 so that map actions no longer cut into movement. Separate runs then extend the history from 20 to 40 frames or remove the heading marker. Removing the heading marker lowers progress, so the agents rely on it. Only Oracle Memory improves both metrics by restoring the routes the agent has already traversed.

ModelMetricOriginal300 Steps40 FramesNo Heading Marker+ Oracle Memory
GPT-5.6-LunaSR0.05.00.00.010.0
GPT-5.6-LunaProgress40.045.045.025.065.0
Claude-Opus-5SR5.05.05.00.025.0
Claude-Opus-5Progress45.050.040.030.065.0

W2: Prompt-metric mismatches and confounded dynamic setups. Section 4.2 claims agents sacrifice road compliance (PNA) during local QA to answer faster, but the Level 1 prompts (Appendix J.2) never tell agents to stay on sidewalks, the spatial buffer for PNA is undefined, and PNA is oddly lowest on Visual Recognition (~66%) where movement isn't needed. Similarly, Pedestrian Navigation (Table 4) runs on LongNav routes where static success is already ~0%, and reuses the LongNav prompt without mentioning pedestrians at all (Appendix J.6), making both the ~0% success and 76%-90% collision rates hard to interpret.

We kept Level 1 annotation consistent with the agent input, which carries no explicit walkway requirement. Annotators placed the start poses without access to the pedestrian network. Some start poses therefore lie off the walkways. In these short-range tasks, the start pose decides most of the score in VR and OU, where agents barely move. Answers at the first step lower the VR score further. Longer walks in the other tasks dilute both effects, so the core value of PNA lies in comparing models within the same task type. Rerunning GPT-5.5 on Level 1 with the sidewalk instruction of the navigation prompts barely changes its adherence.

GPT-5.5VROUAEQ
Level 1 prompt63.874.891.4
+ Sidewalk instruction65.375.994.0

The pedestrian setting keeps the LongNav text unchanged to test whether agents avoid the people they see without being told. In our response to Q3, routes with pedestrians show no consistent drop in success while contact stays frequent, even under an explicit avoidance instruction.


Q1: Visual history and map orientation cues: In the actual benchmark runs, how many past RGB screenshots (if any) were kept in the model's context window at turn t, or did models only see the current screenshot plus their prior text observation/reason JSON strings? Also, when the agent opens the interactive map, does the "You" marker display the agent's current facing direction/frustum, or only its (x, y) position?

We retained the 20 most recent RGB screenshots in the model's context at each turn. The agent also had access to its previous reasoning through the text interaction history.

When the interactive map is opened, the "You" marker shows the agent's current position, while a cone-shaped marker indicates its facing direction. Both markers are visible in the first subfigures of Figures 8 and 9, providing the agent with sufficient information to identify its position and orientation on the map.

AuthorsOpenReview ↗

Official Review of Submission19810 (Part 2)


Q2: PNA definition and the VR vs. AEQ discrepancy: How exactly is "lies on the registered pedestrian network" computed geometrically in 3D continuous space (horizontal buffer radius and vertical tolerance around the Lands Department 3D polyline)? Why is PNA around 63%-70% on Visual Recognition, where locomotion is generally unnecessary, yet over 90% on Active Exploration and >95% on navigation tasks?

We compute PNA as the action-duration-weighted fraction of post-action positions within 6 m horizontally of the nearest segment of the registered 3D pedestrian network. We further re-score the GPT-5.5 trajectories in the walking tasks with a 2 m vertical tolerance and then a stricter 4 m buffer. The vertical tolerance changes adherence slightly. The 4 m buffer lowers it by a similar proportion in each walking task, so a stricter threshold mainly shifts the overall level of PNA.

GPT-5.5AEQShortNavLongNav
6 m buffer91.496.994.3
6 m buffer + 2 m vertical tolerance89.695.091.9
4 m buffer + 2 m vertical tolerance78.883.580.6

As for the gap between different types of tasks, annotators placed the start poses and completed the tasks without access to the pedestrian network, matching the input available to the agents. Some start poses therefore lie off the walkways. Agents barely move in VR and OU, so the start pose decides most of the score there. Answers at the first step lower the VR score further. Navigation and AEQ agents spend most of the episode walking, which keeps both effects small. The core value of PNA lies in comparing models within the same task type, not across different task types.


Q3: Pedestrian and Road-Closure experimental setup: Can you confirm that NP uses the unmodified LGN prompt without mentioning pedestrian avoidance (Appendix J.6), and clarify whether PCR in Table 4 is the percentage of episodes with at least one pedestrian contact? Why were NP (and DCR, if also long-range) not evaluated on ShortNav-scale routes where models achieve 50%-80% baseline success, so that the marginal degradation from closures and moving pedestrians on goal completion would actually be measurable?

NP uses the LGN prompt unchanged. Pedestrians enter as part of the environment, with the task text identical to the matched LGN instance. So the setting tests whether an agent avoids the people it sees without being told. PCR is the percentage of NP episodes in which the simulator records at least one contact between the agent and a pedestrian. We compare the pedestrian metrics of the 12 models against the greedy naive bound and human testers under the same text. The comparison shows that model PCR lies far closer to the naive bound than to humans, so current MLLMs seldom treat moving pedestrians as obstacles.

ModelPed. SR$\uparrow$Ped. PNA$\uparrow$PCR$\downarrow$
GPT-5.6-Luna0.098.682.5
GPT-5.50.093.883.8
GPT-5.40.098.580.0
GPT-5.21.395.478.8
Claude-Opus-52.594.487.5
Claude-Opus-4.63.892.980.0
Gemini-3.8-Flash0.097.585.0
Gemini-3.6-Flash0.096.090.0
Gemini-3.1-Pro0.094.088.8
Doubao-Seed-2.0-Pro0.095.076.3
GLM-5V-Turbo0.095.982.5
Kimi-K32.590.678.8
Naive Bound0.069.292.5
Human100.098.537.5

DCR also runs on the long CN routes, with the closure revealed midway. Long routes let both events interrupt sustained navigation, though their static floor hides what each event costs in goal completion. We therefore evaluate both events on the ShortNav routes with GPT-5.5, Claude-Opus-4.6 and GLM-5V-Turbo, whose static success lies between 50% and 80%. Pedestrian runs come from our original batch under the unmodified ShortNav prompt. Each new closure leaves an annotator-verified detour. A further pedestrian condition adds an explicit avoidance instruction. The results show that closures sharply lower success. Under pedestrians, success shows no consistent drop while contact occurs in a large share of episodes. Contact persists under the explicit instruction, confirming that the cost of moving pedestrians lies in collision-aware motion control.

ModelStatic SR$\uparrow$Closure SR$\uparrow$CCR$\uparrow$Ped. SR$\uparrow$PCR$\downarrow$Instructed Ped. SR$\uparrow$Instructed PCR$\downarrow$
GPT-5.575.038.852.576.340.072.533.8
Claude-Opus-4.675.041.357.565.036.363.831.3
GLM-5V-Turbo50.021.341.358.853.856.347.5
AuthorsOpenReview ↗

Official Review of Submission19810 (Part 3)


Q4: Human baseline statistics and oblique-mesh artifacts: Since human testers completed all 810 instances within 150 steps, can you report their empirical distribution of step counts, map-action counts, PNA, and first-try QA/navigation accuracy? In addition, why do GPT-5.4 (15.0% vs. GPT-5.5's 75.0%) and Claude-Opus-5 (48.8% vs. Opus-4.6's 75.0%) drop so sharply on ShortNav despite strong QA scores -- is this driven by premature terminate calls outside the 15m radius, schema rejections, or getting stuck on street-level photogrammetric mesh geometry?

We report these human statistics by level in our response to W1. On the same street-level meshes, humans solve most instances on the first attempt and use fewer than half of the 150 steps on average.

The ShortNav drops of GPT-5.4 and Claude-Opus-5 come mainly from premature terminate calls. Arrival inside the 15 m radius ends an episode automatically, so a terminate call can only end the episode short of the goal. GPT-5.4 often terminates after checking the map with most of the route still ahead, while Claude-Opus-5 approaches the destination and stops just outside the radius. They lack strong distance perception. Question answering does not require this judgment of arrival, which explains the gap between their QA accuracy and ShortNav success.


We hope that our explanations above can clarify your doubts and you can consider our work more favorably.

Dear Reviewer,

Thank you for your thoughtful feedback. With only a few hours left in the discussion period, we hope our responses have addressed your concerns and would be glad to discuss any remaining questions.

Thank you again for your time.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.