acceptodds
Under review as a conference paper at ICLR 2027

CityLoop: Sandbox for Tool-Grounded Urban Navigation

Abstract

Urban navigation requires an embodied agent to connect a user request with global navigation guidance, local visual perception, continuous action, and eventual arrival. Existing research, however, typically studies these components in isolation, making it difficult to evaluate how they work together over complete routes. We introduce CityLoop, a unified request-to-arrival sandbox for tool-grounded urban navigation. CityLoop connects online navigation, 2D Gaussian Splatting (2DGS) simulation, and physical robots through a shared policy interface, while providing controllable localization perturbations and geographically aligned supervision from map and street-view data. Systematic evaluation in CityLoop reveals three limitations of existing models: unreliable execution of navigation instructions even under clean conditions, weak coordination between navigation guidance and egocentric visual evidence, especially when the two disagree, and compounding errors over long-horizon closed-loop execution. To address these limitations, we introduce RouteGround, a capability-structured training framework that develops navigation-context understanding, visual grounding, conflict-aware decision making through counterfactual tool–vision conflicts, and full-route execution. RouteGround improves keypoint decision accuracy over the strongest baseline from 47.6% to 78.4% with accurate localization and from 26.0% to 59.0% under medium localization errors, achieves 46.0% and 32.0% success in complete 2DGS rollouts under clean and medium-error guidance, respectively, and succeeds in 6 of 20 preliminary physical trials after zero-shot transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Urban Navigation, Embodied Navigation, Tool-Grounded Navigation, Visual Grounding, Closed-Loop Navigation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_PXz9OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CityLoop, a request-to-arrival sandbox for tool-grounded urban navigation, along with RouteGround, a training framework built on Qwen3-VL-4B. CityLoop runs the same high-level policy interface in Google-based online navigation, a MatrixCity 2DGS simulator, and a Unitree G1 deployment. A particularly useful part of the setup is its matched perturbation design: the reported position and heading are changed, and the navigation context changes with them, while the physical state, egocentric observation, and oracle action stay fixed. This allows the authors to test how a policy behaves when tool guidance disagrees with what the agent sees. RouteGround is trained using navigation-context, visual-grounding, counterfactual-conflict, and full-route data. It reports 78.4% and 59.0% Keypoint Decision Accuracy under clean and medium-error guidance, with corresponding 2DGS Success Accuracy of 46.0% and 32.0%, and succeeds in 6 of 20 preliminary robot trials. I see CityLoop as the more valuable contribution. My remaining concern is how much confidence to place in the closed-loop findings, given that they come from a single 2DGS scene and that the cross-model comparison is not fully matched.

Critical strengths
  1. CityLoop provides a tightly controlled setting for evaluating conflicts between navigation tools and egocentric visual evidence. The matched perturbation protocol preserves the physical state, egocentric observation, task identity, and oracle action while changing only the navigation context derived from the perturbed pose. It therefore directly tests whether a policy can arbitrate between route guidance and the current visual scene. The shared request-to-arrival interface across online navigation, 2DGS simulation, and physical deployment also supports diagnosis of failures at different stages of the navigation pipeline.
  2. The main results are consistent with the failure mode that RouteGround is designed to address. In Table 3, adding counterfactual conflicts raises Keypoint Decision Accuracy under medium localization errors from 16.6% to 61.0%; adding full-route supervision then raises 2DGS Success Accuracy under medium errors from 20.0% to 32.0%. In Table 4, removing the current egocentric view or the visual history reduces medium-error closed-loop success from 32.0% to 0.0% or 12.0%, while performance remains relatively high under clean guidance. These results indicate that the model uses visual evidence when the navigation guidance is wrong. The physical trials also provide preliminary evidence of cross-platform transfer.
Critical weaknesses
  1. CityLoop's closed-loop success rate is jointly affected by the definition of perturbation difficulty and the route-projection mechanism. The low, medium, and high conditions are defined by pose-offset ranges, but the 2DGS results do not vary monotonically with these labels: RouteGround obtains 32.0% under medium errors and 26.0% under low errors. Table S2 also states that invalid moves are projected to the active route. The current results therefore do not separate the effect of decision-conflict intensity from the sampled perturbation locations or the simulator's route correction, and route projection may prevent an erroneous action from developing into an actual off-route failure. The paper should report how often projection is triggered in successful rollouts, recompute success when the first projection is counted as failure, and report how often each perturbation level changes the next turn or route branch.
  2. The closed-loop evidence is statistically narrow and the cross-model comparison is not fully matched, so the results do not support broad generalization. The 2DGS evaluation uses a fixed set of 50 routes from the single matrixcity_street_clean scene and one deterministic perturbation plan per condition, without repeated perturbations or interval estimates. In Table 2, general multimodal models receive a minimap image, whereas navigation specialists receive the route as text and execute through heterogeneous native-output adapters. In addition, the main text states that all methods use identical action budgets, while Table S2 defines the 2DGS episode budget as droute/1.2 m high-level decisions; it is unclear whether this is a shared budget for all methods or a model-specific conversion, and what quantity is held constant—decision count, physical distance, or expected route coverage. These choices leave the stability of the non-monotonic 26.0% (low-error) versus 32.0% (medium-error) results unclear and do not establish whether the closed-loop advantage transfers across scenes. As the paper itself notes, the navigation specialists are adapted cross-domain references rather than fully matched baselines, so Table 2 does not support a strict model ranking. Multiple perturbation seeds, route-level uncertainty estimates, an additional scene, and a clear definition of the common budget unit would address these concerns. Finally, the curriculum results should be described as evidence for the added data composition rather than proof that the sequential curriculum itself is necessary.
Questions for authors
  1. Please report the episode-level perturbation trigger rate, displacement magnitude, and route-projection rate, together with a sensitivity analysis that separates rollouts with and without projection.
  2. The closed-loop results use a fixed set of 50 routes and one deterministic perturbation plan. Please repeat the evaluation with multiple perturbation seeds and report route-level confidence intervals or bootstrap intervals. In particular, please clarify whether the 26.0% result under low errors, the 32.0% result under medium errors, and the main model ranking are stable.
  3. The 2DGS evaluation uses only the matrixcity_street_clean scene. If feasible, please reproduce the experiment in another 2DGS scene or an independent city block to determine whether RouteGround's closed-loop advantage depends on the road structure or rendering characteristics of the current scene.
  4. Each row in Table 3 changes the data type, sample count, and training process. Is the intended claim that the added data composition is effective, or that the sequential curriculum itself is necessary? Without an order-controlled comparison, the text should limit the conclusion to the gain from adding conflict and full-route data.
  5. Please clarify the common budget unit across models. Is the fixed quantity the number of high-level decisions, the physical distance, or expected route coverage? If the episode budget is droute/1.2 m high-level decisions for all methods, please state so explicitly; if any model-specific conversion based on mean per-step displacement is used, please give the formula and per-model values.
7 comments

We thank Reviewer PXz9 for the constructive feedback and for recognizing that CityLoop provides a tightly controlled setting for evaluating conflicts between navigation tools and egocentric visual evidence and that the results are consistent with the failure mode RouteGround is designed to address. We address each of your comments below.

W1: Closed-loop success is confounded by the definition of perturbation difficulty (2DGS results are non-monotonic: 26.0% under low vs. 32.0% under medium errors) and by route projection, which may keep an erroneous action from becoming an off-route failure; please report how often projection is triggered in successful rollouts, recompute success when the first projection counts as failure, and report how often each perturbation level changes the next turn or route branch.

Route projection does affect success; without it RouteGround's success decreases but remains the only non-zero result, and projection mirrors the safety mechanisms of real deployment. Thank you for pointing out this issue.

When an action would leave the walkable area, such as stepping onto the vehicle road, the simulator blocks it with a geofence and projects the agent onto the nearest point of the route. We evaluated two alternatives:

2DGS success (%)NoneLowMed.High
With projection (paper)46.026.032.018.0
Invalid move → stay in place28.020.016.010.0
No geofence (free movement)36.022.020.08.0

All other models reach 0% under both alternatives.

We keep projection because it reflects real deployment. Pedestrians do not walk onto the vehicle road, and deployed robots enforce the same constraint with a geofence. When a robot drifts, an operator or safety layer moves it back to the nearest walkable point on its route, which is what projection does; in our robot trials, operators made similar corrections (12%, Table S9). Projection therefore keeps the simulator in the same distribution as physical deployment rather than idealizing it.

Higher levels apply perturbations more often, especially within turn windows, where they can change the next turn (Appendix D.2, lines 958–966; Table S8, lines 925–930). The stability of the low- and medium-error results across perturbation seeds is reported in our response to Q2.

W2: The closed-loop evidence is statistically narrow (50 fixed routes in the single matrixcity_street_clean scene, one deterministic perturbation plan per condition, no interval estimates), the cross-model comparison is not fully matched (minimap images vs. route text with heterogeneous native-output adapters), the common action-budget unit is unclear ("identical action budgets" vs. d_route/1.2 m in Table S2), and the curriculum results support the added data composition rather than the necessity of the sequential curriculum.

RouteGround's closed-loop advantage holds across perturbation seeds and under matched inputs. We appreciate these constructive comments and address them in turn. The statistical scope is addressed in Q2 (three perturbation seeds with intervals: RouteGround ranks first at every level, and the low/medium inversion is within seed variation) and Q3 (the single scene). The budget equalizes expected total translation rather than decision count, and we have corrected "identical action budgets" (Q5). An order-controlled comparison further shows that the sequential curriculum itself is effective (Q4).

On the cross-model comparison, the navigation specialists are indeed adapted cross-domain references, as stated in Sec. 5.1. All adapters convert each model's native output into the same action space, (x,y,Δθ)(x, y, \Delta\theta) or Stop, and an invalid output becomes a zero-displacement move rather than a Stop (Appendix C, Table S5). To remove the input mismatch, we have also evaluated every model with the same route text; RouteGround still outperforms every compared model at every perturbation level:

Text inputKP NoneKP LowKP Med.KP High2DGS None2DGS Low2DGS Med.2DGS High
GPT-5.6-sol58.337.213.614.54.02.000
Claude-Sonnet-547.726.611.016.12.0000
Gemini-3.6-Flash39.820.616.513.82.0000
Qwen3-VL-4B31.118.514.113.64.0000
Qwen3-VL-8B35.024.925.122.46.02.000
NaVid25.326.724.926.40000
NaVILA25.325.324.925.60000
StreamVLN25.323.424.226.00000
OmniVLA-edge26.026.026.026.00000
LeLaN20.520.120.520.50000
RouteGround (Text)73.349.949.127.558.028.026.016.0

We have added the full matched-input table to Appendix G (Table S11, lines 1044–1057).

Q1: "Please report the episode-level perturbation trigger rate, displacement magnitude, and route-projection rate, together with a sensitivity analysis that separates rollouts with and without projection."

RouteGround triggers projection least often of all models; the sensitivity analysis without projection is in our response to W1. Share of steps that trigger projection:

ModelRouteGroundQwen3-VL-4BQwen3-VL-8BGPT-5.6-solClaude-Sonnet-5Gemini-3.6-FlashNaVidNaVILAStreamVLNOmniVLA-edgeLeLaN
Projection rate (%)8.915.517.615.018.718.916.812.815.612.617.1

The application probability of each perturbation level is given in Appendix D.2, and the displacement magnitudes in Table S8.

Q2: "The closed-loop results use a fixed set of 50 routes and one deterministic perturbation plan. Please repeat the evaluation with multiple perturbation seeds and report route-level confidence intervals or bootstrap intervals. In particular, please clarify whether the 26.0% result under low errors, the 32.0% result under medium errors, and the main model ranking are stable."

Across three perturbation seeds, RouteGround ranks first at every level, and the low/medium inversion in the original run is within seed-to-seed variation. Following your suggestion, we have repeated the 2DGS evaluation with two additional perturbation seeds. The table reports the mean ± standard deviation over the three seeds and the Wilson 95% interval over all 150 episodes:

2DGS success (%)NoneLowMed.High
RouteGround47.3 ± 2.3 [39.5, 55.3]29.3 ± 3.1 [22.6, 37.1]30.0 ± 2.0 [23.2, 37.8]19.3 ± 2.3 [13.8, 26.4]
Best baseline3.3 ± 1.2 [1.4, 7.6]0 [0, 2.5]0 [0, 2.5]0 [0, 2.5]

Low and medium errors give nearly equal means (29.3% vs. 30.0%), and low exceeds medium in two of the three seeds, so the original 26.0% vs. 32.0% gap reflects seed variation rather than a systematic inversion. No baseline exceeds 4.0% in any seed.

Q3: "The 2DGS evaluation uses only the matrixcity_street_clean scene. If feasible, please reproduce the experiment in another 2DGS scene or an independent city block to determine whether RouteGround's closed-loop advantage depends on the road structure or rendering characteristics of the current scene."

We acknowledge that a single scene limits the closed-loop evidence, but RouteGround's advantage does not appear tied to this scene's rendering or road structure.

Request-to-arrival evaluation needs a city-scale scene in which an agent can walk kilometer-scale routes. When we built CityLoop, we found no open benchmark that offered this, and MatrixCity was the most suitable open-source city-scale scene we found. Rendering alone is not enough: the loop also needs POIs, user requests, a road graph, and a route service, none of which exist for this scene, so we built all of them. To our knowledge, this is the first closed loop from request to arrival in a reconstructed city (see also Table 1).

On rendering, RouteGround is trained only on real Street View images, which differ substantially in appearance from the 2DGS renderings, and no 2DGS data are used at any training stage, so the closed-loop result is already a zero-shot transfer. On road structure, the keypoint evaluation spans 43 cities and 121 Street View regions, where RouteGround leads at every error level, and the physical robot completes routes on real streets.

Q4: "Each row in Table 3 changes the data type, sample count, and training process. Is the intended claim that the added data composition is effective, or that the sequential curriculum itself is necessary? Without an order-controlled comparison, the text should limit the conclusion to the gain from adding conflict and full-route data."

Thank you for raising this point; we find it highly meaningful. Following your suggestion, we have run order-controlled comparisons that keep the training data fixed and change only the training order. Because understanding the navigation context is the basis of every later capability, Stage I always comes first; we then swapped Stages II and III, and separately Stages III and IV. Both swapped orders perform worse than the original sequence (swapping II and III: keypoint accuracy 61.4%/42.0% under clean/medium errors; swapping III and IV: 2DGS success 14.0%/8.0%), consistent with our expectation that the sequential curriculum itself is effective, beyond the added data composition. As in human learning, these capabilities are acquired more effectively step by step, each building on the previous one.

Reviewer_sv83OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CityLoop, a unified sandbox for tool-grounded urban navigation that connects online navigation services, 2D Gaussian Splatting (2DGS) simulation, and physical robots through a shared policy interface. Building on this platform, RouteGround uses a four-stage training curriculum to improve route-context understanding, visual grounding, conflict-aware decision making, and long-horizon execution. Experiments show substantial improvements in navigation decision accuracy and simulated route completion under localization errors, alongside preliminary zero-shot deployment on a physical robot.

Critical strengths
  1. Unified Navigation Framework. CityLoop integrates destination resolution, global route guidance, visual perception, closed-loop execution, and arrival detection, enabling evaluation across multiple deployment settings.

  2. Controllable Localization Errors. The framework introduces controlled position and heading perturbations while preserving physical observations, supporting systematic evaluation of navigation robustness.

  3. Geographically Grounded Supervision. Map and street-view data provide aligned navigation states and verifiable action labels, offering a practical alternative to relying exclusively on manually annotated trajectories or videos.

  4. Structured Training Strategy. RouteGround combines context understanding, visual grounding, counterfactual tool–vision conflicts, and complete-route supervision. Ablations provide evidence for the contributions of conflict training and full-route data.

  5. Strong Quantitative Improvements. Keypoint decision accuracy improves from 47.6% to 78.4% under clean localization and from 26.0% to 59.0% under medium errors. Simulated route success also substantially exceeds the reported baselines.

  6. Cross-Domain Deployment. Evaluation spans online navigation, 2DGS simulation, and a physical Unitree G1 robot, providing evidence beyond isolated decision benchmarks.

Critical weaknesses
  1. Unclear Incremental Novelty. CityLoop integrates existing navigation services, simulation, and robot interfaces, while RouteGround combines established supervised fine-tuning techniques with counterfactual data and trajectory supervision. The paper could more clearly distinguish its methodological advances from systems integration and curriculum design.

  2. Limited Closed-Loop Evaluation Scale. Although the benchmark covers 43 cities, 22 countries or regions, and 1,000 evaluation routes, closed-loop simulation uses only 50 routes from a single MatrixCity 2DGS scene. The extent to which the results generalize across diverse urban layouts remains uncertain.

  3. Imperfect Baseline Comparability. Navigation-specialized baselines receive route text when they cannot process minimaps, whereas RouteGround uses visual map inputs. Although the paper acknowledges these adaptations, the comparisons do not fully isolate the benefits of the proposed training framework from differences in model interfaces and input representations.

  4. Limited Robustness Coverage. The main robustness analysis focuses on controlled position and heading perturbations. Other practical challenges, such as dynamic obstacles, changing road conditions, unreliable navigation services, and localization errors beyond the tested settings, remain insufficiently explored.

  5. Weak Real-World Validation. The physical evaluation reports only 6 successful trials out of 20, with no matched baseline to isolate RouteGround's contribution. These results demonstrate preliminary deployment feasibility but provide limited evidence of reliable real-world navigation.

  6. Restricted Model Generalizability. RouteGround is fine-tuned on a single Qwen3-VL-4B-Instruct backbone using 4-bit LoRA. It remains unclear whether the proposed training strategy generalizes to larger models, different architectures, or policies with different navigation interfaces. More extensive cross-model evaluation would strengthen the claims.

Questions for authors
  1. Novelty and Contributions. How does CityLoop differ from existing tool-grounded navigation frameworks, and what are the key methodological contributions of RouteGround beyond combining supervised fine-tuning, counterfactual training data, and full-route supervision?

  2. Generalization Across Environments. Why is closed-loop simulation evaluated on only 50 routes from a single MatrixCity 2DGS scene? Can the authors provide results across multiple scenes or geographically diverse environments to demonstrate generalization beyond the current setting?

  3. Baseline Fairness. How much of RouteGround's performance gain comes from its training strategy rather than differences in input representations and interfaces, given that some navigation-specialized baselines receive textual routes while RouteGround uses visual map inputs? Can the authors provide comparisons under matched input conditions?

  4. Robustness Beyond Localization Errors. How does RouteGround perform under other practical challenges, such as dynamic obstacles, changing road conditions, or unreliable navigation services? Can the authors evaluate these factors to better characterize the system's robustness in realistic urban environments?

  5. Real-World Reliability. Given that the physical robot succeeds in only 6 out of 20 trials, what are the primary causes of failure? Can the authors provide a matched baseline evaluation, additional physical trials, or a failure analysis to clarify the practical benefits and limitations of RouteGround?

  6. Model Generalizability. Does the proposed training curriculum generalize to different model backbones, larger vision-language models, or alternative navigation interfaces? Can the authors provide cross-model experiments to establish whether the observed improvements are specific to Qwen3-VL-4B-Instruct or reflect a more general training strategy?

5 comments

We thank Reviewer sv83 for the constructive feedback and for recognizing the unified navigation framework from destination resolution to arrival detection, the controllable localization errors that preserve physical observations, and the geographically grounded supervision with verifiable action labels. We address each of your comments below.

W1 Unclear Incremental Novelty & Q1 Novelty and Contributions

CityLoop and RouteGround do build on existing services and established techniques; our contribution is a request-to-arrival closed-loop evaluation that existing work lacks, and a training design that solves the failures this evaluation reveals. We would like to further clarify these two contributions.

CityLoop's contribution is an experimental formulation and infrastructure for studying localization-induced tool–vision conflicts. It separates the physical state from the pose reported to navigation tools, allowing navigation guidance to change while the visual observations and oracle action remain fixed. Its shared interface connects destination resolution, local decisions, and closed-loop execution, enabling both controlled diagnosis and evaluation of downstream navigation consequences.

RouteGround contributes a supervision design and capability-structured training framework for tool–vision conflicts. Navigation context is generated from a perturbed pose, while the action target remains grounded in the true physical state. This breaks the shortcut of simply copying tool guidance and encourages the policy to use visual evidence to judge whether a navigation cue is currently executable. Training progresses through context understanding, visual grounding, and conflict handling, followed by full-route data to broaden sequential-state coverage. The contribution lies in the supervision design and training organization, rather than a new optimization.

W2 Limited Closed-Loop Evaluation Scale & Q2 Generalization Across Environments

We agree that a single 2DGS scene limits how far the closed-loop results generalize; the scene serves to test whether request-to-arrival navigation works in a closed loop under localization noise.

Request-to-arrival evaluation needs a city-scale scene in which an agent can walk kilometer-scale routes. When we built CityLoop, we found no open benchmark that offered this, and MatrixCity was the most suitable open-source city-scale scene we found. Rendering alone is not enough: the loop also needs POIs, user requests, a road graph, and a route service, none of which exist for this scene, so we built all of them. To our knowledge, this is the first closed loop from request to arrival in a reconstructed city (see also Table 1).

The 2DGS renderings differ substantially in appearance from the Street View images used for training, and no 2DGS data are used at any training stage, so the closed-loop result is a zero-shot test. The scene covers varied road layouts, and the 50 routes are drawn from a pool of 102, stratified by task type and sampled at approximately even distance intervals. Geographic diversity is covered by the keypoint evaluation (43 cities, 121 Street View regions) and by the physical robot in a real city.

We evaluated these 50 routes with three perturbation seeds. The Wilson 95% intervals of RouteGround and the best baseline do not overlap at any error level (mean ± standard deviation over the seeds; intervals over all 150 episodes):

2DGS success (%)NoneLowMed.High
RouteGround47.3 ± 2.3 [39.5, 55.3]29.3 ± 3.1 [22.6, 37.1]30.0 ± 2.0 [23.2, 37.8]19.3 ± 2.3 [13.8, 26.4]
Best baseline3.3 ± 1.2 [1.4, 7.6]0 [0, 2.5]0 [0, 2.5]0 [0, 2.5]

We have added the intervals to Appendix H (Table S12, lines 1081–1088).

W3 Imperfect Baseline Comparability & Q3 Baseline Fairness

The gain comes from the training strategy itself rather than from differences in input representations or interfaces. We appreciate this suggestion and have now evaluated every model with the same route text as well. With either text or image input, RouteGround outperforms every compared model at every perturbation level:

Text inputKP NoneKP LowKP Med.KP High2DGS None2DGS Low2DGS Med.2DGS High
GPT-5.6-sol58.337.213.614.54.02.000
Claude-Sonnet-547.726.611.016.12.0000
Gemini-3.6-Flash39.820.616.513.82.0000
Qwen3-VL-4B31.118.514.113.64.0000
Qwen3-VL-8B35.024.925.122.46.02.000
NaVid25.326.724.926.40000
NaVILA25.325.324.925.60000
StreamVLN25.323.424.226.00000
OmniVLA-edge26.026.026.026.00000
LeLaN20.520.120.520.50000
RouteGround (Text)73.349.949.127.558.028.026.016.0

RouteGround is trained from Qwen3-VL-4B, and with either text or image input it clearly outperforms the original Qwen3-VL-4B, which shows that the training strategy is effective. We have added the full matched-input table to Appendix G (Table S11, lines 1044–1057).

W4 Limited Robustness Coverage & Q4 Robustness Beyond Localization Errors

RouteGround is a high-level planner; paired with a low-level obstacle-avoidance policy, it still completes routes with added obstacles.

Many navigation systems separate high-level planning from low-level locomotion (e.g., NaVILA). RouteGround works at the high level: it decides where to go from the request, the route guidance, and the views, and our contribution advances this level. Its (x,y,Δθ)(x, y, \Delta\theta) output can be passed to a low-level policy that handles walking and obstacle avoidance.

To test this, we inserted obstacles such as boxes, trash bins, roadblocks, and moving pedestrians into the 2DGS scene, on average one every 100 m. Each obstacle is built as a 2DGS asset, placed at a given position, orientation, and scale, and rendered jointly with the scene from the same camera. OmniVLA-edge serves as the low-level obstacle-avoidance policy:

2DGS success (%)NoneLowMed.High
Without obstacles46.026.032.018.0
With obstacles + OmniVLA-edge40.022.022.012.0

Success decreases, and 13% of the failures come from the agent getting stuck in front of an obstacle without progress.

W5 Weak Real-World Validation & Q5 Real-World Reliability

We acknowledge that 6/20 does not establish reliable real-world navigation, and the paper presents these as preliminary trials (abstract, Sec. 5.3). The trials are meant to show two things: CityLoop connects directly to real physical scenes, so the same request-to-arrival pipeline runs on a real robot, and RouteGround, trained only on Street View data, transfers zero-shot to real routes.

Per-route results:

RouteLength (km)SuccessInterventionsPremature stopNo progressInvalid actionStep limit
R10.42/411%1001
R20.62/47%0002
R31.11/417%0102
R41.31/48%1200
R51.60/416%0004
All1.06/2012%2309

The successes are spread over four of the five routes. Most failures come from exhausting the step limit (9 of 14); the rest are 3 no-progress cases and 2 premature stops, and no invalid action occurs.

Due to time constraints, we have so far run matched robot baselines for only part of the compared models: four general multimodal models on the same five routes, with four trials per route.

ModelR1R2R3R4R5Total
GPT-5.6-sol1/41/40/40/40/42/20
Claude-Sonnet-50/40/40/40/40/40/20
Gemini-3.6-Flash0/40/40/40/40/40/20
Qwen3-VL-4B0/40/40/40/40/40/20
RouteGround2/42/41/41/40/46/20

RouteGround succeeds in 6/20 trials, versus at most 2/20 for the baselines, and most baseline failures are no-progress cases. We have added the per-route RouteGround table to Appendix H (Table S13, lines 1091–1100).

W6 Restricted Model Generalizability & Q6 Model Generalizability

The training strategy transfers to a larger backbone, and the gain comes mainly from the data and training rather than from model size. Following your suggestion, we have trained RouteGround on Qwen3-VL-8B with the same recipe:

ModelKP NoneKP LowKP Med.KP High2DGS None2DGS Low2DGS Med.2DGS High
Qwen3-VL-4B (base)15.417.212.815.40000
Qwen3-VL-8B (base)23.125.623.423.84.0000
RouteGround (4B)78.460.159.044.046.026.032.018.0
RouteGround (8B)81.861.160.443.850.032.030.016.0

Moving from 4B to 8B changes the results only slightly: higher at lower error levels and comparable at higher ones. Training, by contrast, lifts both backbones far above their base models. Model size therefore matters less than the data and training method, which drive the improvement.

We hope these responses address your concerns. We are happy to discuss further.

Comments on the paper

7 comments

General Response to All Reviewers and AC

We thank the reviewers for their constructive feedback, and we are encouraged that they recognized the unified request-to-arrival framework (DhwU, sv83, PXz9), the controllable tool–vision conflicts (sv83, PXz9), the well-motivated counterfactual supervision (DhwU), and the preliminary physical transfer (sv83, PXz9).

CityLoop establishes a unified request-to-arrival benchmark for controlled tool–vision conflicts, while RouteGround leverages these conflicts as supervision for visually grounded navigation. The new evidence addresses the main concerns as follows:

  • Not an artifact of input format (DhwU, sv83, PXz9): with the same route text for every model, RouteGround still leads.
  • Not seed noise or projection assistance (sv83, PXz9): the advantage holds over three perturbation seeds with non-overlapping intervals, and without route projection.
  • Not just an assembly of known techniques (DhwU, sv83): what is new is a supervision relation designed around urban navigation: guidance from the reported pose, labels from the true state, which discourages blind tool-following.
  • Both the training data and the training order matter (DhwU, PXz9): Table 3 shows that each added data group improves the capability it targets; with the data fixed, order-controlled comparisons further show that the training sequence itself also matters.
  • No train–test leakage (DhwU): training and evaluation share no images, and 2DGS is a zero-shot test in a synthetic city never used in training.
  • Not tied to one backbone or to obstacle-free scenes (sv83): the recipe carries over to Qwen3-VL-8B, and with obstacles added in simulation, RouteGround still completes routes when paired with a low-level obstacle-avoidance controller.
  • Real-world transfer, scoped as preliminary (DhwU, sv83): in real-world trials, RouteGround still outperforms the baselines; we present this as a proof of concept, not deployment-ready reliability.

Overall, our work bridges a controlled evaluation setting with targeted supervision, delivering measurable improvements from local conflict resolution to end-to-end navigation.

Additional Experiments and Revisions

We have run the following new experiments; the individual responses give the full results.

  • Matched-input baselines: all models evaluated with the same route-text input.
  • Multiple perturbation seeds: closed-loop evaluation repeated over three perturbation seeds.
  • Projection sensitivity: route projection compared with staying in place after invalid moves and with free movement.
  • Training-order controls: the grounding/conflict stages and the conflict/full-route stages swapped.
  • Larger backbone: the same training recipe applied to Qwen3-VL-8B.
  • Obstacle-augmented simulation navigation: RouteGround paired with OmniVLA-edge for low-level obstacle avoidance.
  • Robot baselines: additional models tested on the same physical routes.

We have uploaded a revised version of the paper, where all major changes are highlighted in blue.

We thank the reviewers again for their suggestions and are happy to discuss further.

We thank Reviewer DhwU for the constructive feedback and for recognizing that the request-to-arrival formulation connects usually separate urban-navigation components through one interface and that counterfactual conflicts are a well-motivated training signal. We address each of your comments below.

W1: "The benchmark compares models with different input interfaces: proprietary/open-source multimodal models receive images, while released navigation models receive route text. The comparison should be framed as a reference rather than a controlled leaderboard, with matched-input baselines."

Under matched text input, RouteGround still outperforms every compared model at every perturbation level. Navigation specialists cannot take a map image, so they receive the same route as text through their native interface, while the general multimodal models already receive exactly the inputs of RouteGround. Thank you for suggesting matched-input baselines; we have now also evaluated every model with the same route text:

Text inputKP NoneKP LowKP Med.KP High2DGS None2DGS Low2DGS Med.2DGS High
GPT-5.6-sol58.337.213.614.54.02.000
Claude-Sonnet-547.726.611.016.12.0000
Gemini-3.6-Flash39.820.616.513.82.0000
Qwen3-VL-4B31.118.514.113.64.0000
Qwen3-VL-8B35.024.925.122.46.02.000
NaVid25.326.724.926.40000
NaVILA25.325.324.925.60000
StreamVLN25.323.424.226.00000
OmniVLA-edge26.026.026.026.00000
LeLaN20.520.120.520.50000
RouteGround (Text)73.349.949.127.558.028.026.016.0

We have added the full matched-input table to Appendix G (Table S11, lines 1044–1057).

W2: "Dataset construction, route counts, scene/city splits, and contamination controls are not sufficiently clear. Since the model is trained on CityLoop-derived state–action data and evaluated in CityLoop, please establish strict route/city disjointness and report all training/evaluation overlap checks."

Training and evaluation share no images, and the 2DGS evaluation is strictly city-disjoint. Sorry for the confusion; these details were insufficient in the original submission.

Construction. We align Street View panoramas and the road graph in a common latitude–longitude frame and construct decision states along routes. Each sample contains the current and four preceding egocentric frames, plus a minimap or route text with heading. The ground-truth action is computed from the true position, the route, and the geometry of the current view. Counterfactual samples keep this true visual state unchanged and perturb only the navigation observation.

Scale. The keypoint set contains 1,000 tasks from 43 cities and 121 Street View regions. Full-route training data come from 200 complete routes, and per-group sample counts are listed in Table S3. The 2DGS task pool has 102 routes, of which the current evaluation uses 50, stratified by task type and sampled at approximately even distance intervals.

Overlap. The keypoint set shares cities with the training data, but a city spans many regions that differ substantially in layout and appearance. We also deduplicate by panorama ID, so no image is shared between tasks or between training and evaluation, which rules out image-level leakage. The 2DGS evaluation is further strictly city-disjoint: it runs in a synthetic city, no 2DGS data are used at any training stage, and RouteGround still clearly outperforms all baselines in this entirely unseen environment.

W3: "The closed-loop evaluation is small (the main text refers to 50 routes), and the physical evaluation has only 20 trials with 6 successes. Confidence intervals, per-route outcomes, interventions, and failure categories are needed before making claims about urban navigation."

RouteGround's 2DGS gains are statistically clear: across three perturbation seeds, its Wilson 95% intervals do not overlap with those of the best baseline at any error level. The table reports the mean ± standard deviation over the seeds and the interval over all 150 episodes:

2DGS success (%)NoneLowMed.High
RouteGround47.3 ± 2.3 [39.5, 55.3]29.3 ± 3.1 [22.6, 37.1]30.0 ± 2.0 [23.2, 37.8]19.3 ± 2.3 [13.8, 26.4]
Best baseline3.3 ± 1.2 [1.4, 7.6]0 [0, 2.5]0 [0, 2.5]0 [0, 2.5]

For the robot, 6/20 has a Wilson 95% interval of [14.5, 51.9]. Per-route results:

RouteLength (km)SuccessInterventionsPremature stopNo progressInvalid actionStep limit
R10.42/411%1001
R20.62/47%0002
R31.11/417%0102
R41.31/48%1200
R51.60/416%0004
All1.06/2012%2309

The 14 failures comprise 9 step-limit exhaustions, 3 no-progress cases, and 2 premature stops; no invalid action occurs. We have added the intervals and this robot table to Appendix H (lines 1007–1013; Tables S12–S13, lines 1081–1100).

W4: "The keypoint decision and 2DGS metrics may not measure request resolution, route acquisition, and arrival equally. Please report stage-wise failure rates and clarify whether POI resolution, route generation, and policy execution use any shared information or tuning."

We agree that the two metrics cover different stages, so we now report failures stage by stage; POI resolution, route generation, and policy execution share no ground-truth information.

What each metric measures. Keypoint accuracy measures only single-step execution decisions. Request resolution and route acquisition are measured by POI accuracy (500 requests, Table 2). Arrival is measured only by 2DGS and robot success, which cover the full process from the user request to arrival at the destination.

Stage-wise failures. The table gives the successes and the distribution of failure causes over the four stages:

EpisodesSuccessRequestRouteExecutionArrival
2DGS, clean (50)2310260
2DGS, medium (50)1600340
Robot (20)600122

Execution failures leave the route, make no progress, output invalid actions, or exhaust the step budget; arrival failures are early stops, more than 5 m from the destination.

Shared information and tuning. One RouteGround model performs POI resolution and policy execution, but the stages communicate only through the tool interface. POI resolution passes only the resolved POI ID to get_route; route generation is done by the map service (Google Routes API, or a simulated service with the same formats in 2DGS); policy execution sees only the returned route and its own egocentric views. No stage receives the ground-truth destination, route, or pose.

We have added this breakdown and the information flow to Appendix I (lines 1060–1072; Table S14, lines 1103–1109), and Table S2 now describes the simulated service.

W5: "The methodological novelty is difficult to separate from a careful assembly of route following, visual grounding, counterfactual augmentation, and full-episode supervision. The paper needs a stronger comparison against existing urban navigation or tool-arbitration training frameworks to show that RouteGround is a new learning principle rather than a domain-specific training recipe."

As you note, each component builds on established techniques; our contribution is to design them around the properties of urban navigation and make them work together. An urban agent follows guidance from map tools, must match it to the visible street at turns and intersections, receives wrong guidance under localization errors, and executes long closed-loop routes. Each of our four data groups targets one of these properties (Sec. 4.2).

As shown in Table 3, adding each data group improves the capability it targets: counterfactual conflicts markedly raise accuracy under localization errors, and full-route data raise closed-loop success. This shows that the training paradigm is effective for urban navigation. The closest works, UrbanVLA and NavTrust, neither perturb the pose behind the guidance nor train on the resulting conflict.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.