- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces CityLoop, a request-to-arrival sandbox for tool-grounded urban navigation, along with RouteGround, a training framework built on Qwen3-VL-4B. CityLoop runs the same high-level policy interface in Google-based online navigation, a MatrixCity 2DGS simulator, and a Unitree G1 deployment. A particularly useful part of the setup is its matched perturbation design: the reported position and heading are changed, and the navigation context changes with them, while the physical state, egocentric observation, and oracle action stay fixed. This allows the authors to test how a policy behaves when tool guidance disagrees with what the agent sees. RouteGround is trained using navigation-context, visual-grounding, counterfactual-conflict, and full-route data. It reports 78.4% and 59.0% Keypoint Decision Accuracy under clean and medium-error guidance, with corresponding 2DGS Success Accuracy of 46.0% and 32.0%, and succeeds in 6 of 20 preliminary robot trials. I see CityLoop as the more valuable contribution. My remaining concern is how much confidence to place in the closed-loop findings, given that they come from a single 2DGS scene and that the cross-model comparison is not fully matched.
Critical strengths
- CityLoop provides a tightly controlled setting for evaluating conflicts between navigation tools and egocentric visual evidence. The matched perturbation protocol preserves the physical state, egocentric observation, task identity, and oracle action while changing only the navigation context derived from the perturbed pose. It therefore directly tests whether a policy can arbitrate between route guidance and the current visual scene. The shared request-to-arrival interface across online navigation, 2DGS simulation, and physical deployment also supports diagnosis of failures at different stages of the navigation pipeline.
- The main results are consistent with the failure mode that RouteGround is designed to address. In Table 3, adding counterfactual conflicts raises Keypoint Decision Accuracy under medium localization errors from 16.6% to 61.0%; adding full-route supervision then raises 2DGS Success Accuracy under medium errors from 20.0% to 32.0%. In Table 4, removing the current egocentric view or the visual history reduces medium-error closed-loop success from 32.0% to 0.0% or 12.0%, while performance remains relatively high under clean guidance. These results indicate that the model uses visual evidence when the navigation guidance is wrong. The physical trials also provide preliminary evidence of cross-platform transfer.
Critical weaknesses
- CityLoop's closed-loop success rate is jointly affected by the definition of perturbation difficulty and the route-projection mechanism. The low, medium, and high conditions are defined by pose-offset ranges, but the 2DGS results do not vary monotonically with these labels: RouteGround obtains 32.0% under medium errors and 26.0% under low errors. Table S2 also states that invalid moves are projected to the active route. The current results therefore do not separate the effect of decision-conflict intensity from the sampled perturbation locations or the simulator's route correction, and route projection may prevent an erroneous action from developing into an actual off-route failure. The paper should report how often projection is triggered in successful rollouts, recompute success when the first projection is counted as failure, and report how often each perturbation level changes the next turn or route branch.
- The closed-loop evidence is statistically narrow and the cross-model comparison is not fully matched, so the results do not support broad generalization. The 2DGS evaluation uses a fixed set of 50 routes from the single matrixcity_street_clean scene and one deterministic perturbation plan per condition, without repeated perturbations or interval estimates. In Table 2, general multimodal models receive a minimap image, whereas navigation specialists receive the route as text and execute through heterogeneous native-output adapters. In addition, the main text states that all methods use identical action budgets, while Table S2 defines the 2DGS episode budget as droute/1.2 m high-level decisions; it is unclear whether this is a shared budget for all methods or a model-specific conversion, and what quantity is held constant—decision count, physical distance, or expected route coverage. These choices leave the stability of the non-monotonic 26.0% (low-error) versus 32.0% (medium-error) results unclear and do not establish whether the closed-loop advantage transfers across scenes. As the paper itself notes, the navigation specialists are adapted cross-domain references rather than fully matched baselines, so Table 2 does not support a strict model ranking. Multiple perturbation seeds, route-level uncertainty estimates, an additional scene, and a clear definition of the common budget unit would address these concerns. Finally, the curriculum results should be described as evidence for the added data composition rather than proof that the sequential curriculum itself is necessary.
Questions for authors
- Please report the episode-level perturbation trigger rate, displacement magnitude, and route-projection rate, together with a sensitivity analysis that separates rollouts with and without projection.
- The closed-loop results use a fixed set of 50 routes and one deterministic perturbation plan. Please repeat the evaluation with multiple perturbation seeds and report route-level confidence intervals or bootstrap intervals. In particular, please clarify whether the 26.0% result under low errors, the 32.0% result under medium errors, and the main model ranking are stable.
- The 2DGS evaluation uses only the matrixcity_street_clean scene. If feasible, please reproduce the experiment in another 2DGS scene or an independent city block to determine whether RouteGround's closed-loop advantage depends on the road structure or rendering characteristics of the current scene.
- Each row in Table 3 changes the data type, sample count, and training process. Is the intended claim that the added data composition is effective, or that the sequential curriculum itself is necessary? Without an order-controlled comparison, the text should limit the conclusion to the gain from adding conflict and full-route data.
- Please clarify the common budget unit across models. Is the fixed quantity the number of high-level decisions, the physical distance, or expected route coverage? If the episode budget is droute/1.2 m high-level decisions for all methods, please state so explicitly; if any model-specific conversion based on mean per-step displacement is used, please give the formula and per-model values.
7 comments
W2: The closed-loop evidence is statistically narrow (50 fixed routes in the single matrixcity_street_clean scene, one deterministic perturbation plan per condition, no interval estimates), the cross-model comparison is not fully matched (minimap images vs. route text with heterogeneous native-output adapters), the common action-budget unit is unclear ("identical action budgets" vs. d_route/1.2 m in Table S2), and the curriculum results support the added data composition rather than the necessity of the sequential curriculum.
RouteGround's closed-loop advantage holds across perturbation seeds and under matched inputs. We appreciate these constructive comments and address them in turn. The statistical scope is addressed in Q2 (three perturbation seeds with intervals: RouteGround ranks first at every level, and the low/medium inversion is within seed variation) and Q3 (the single scene). The budget equalizes expected total translation rather than decision count, and we have corrected "identical action budgets" (Q5). An order-controlled comparison further shows that the sequential curriculum itself is effective (Q4).
On the cross-model comparison, the navigation specialists are indeed adapted cross-domain references, as stated in Sec. 5.1. All adapters convert each model's native output into the same action space, or Stop, and an invalid output becomes a zero-displacement move rather than a Stop (Appendix C, Table S5). To remove the input mismatch, we have also evaluated every model with the same route text; RouteGround still outperforms every compared model at every perturbation level:
| Text input | KP None | KP Low | KP Med. | KP High | 2DGS None | 2DGS Low | 2DGS Med. | 2DGS High |
|---|---|---|---|---|---|---|---|---|
| GPT-5.6-sol | 58.3 | 37.2 | 13.6 | 14.5 | 4.0 | 2.0 | 0 | 0 |
| Claude-Sonnet-5 | 47.7 | 26.6 | 11.0 | 16.1 | 2.0 | 0 | 0 | 0 |
| Gemini-3.6-Flash | 39.8 | 20.6 | 16.5 | 13.8 | 2.0 | 0 | 0 | 0 |
| Qwen3-VL-4B | 31.1 | 18.5 | 14.1 | 13.6 | 4.0 | 0 | 0 | 0 |
| Qwen3-VL-8B | 35.0 | 24.9 | 25.1 | 22.4 | 6.0 | 2.0 | 0 | 0 |
| NaVid | 25.3 | 26.7 | 24.9 | 26.4 | 0 | 0 | 0 | 0 |
| NaVILA | 25.3 | 25.3 | 24.9 | 25.6 | 0 | 0 | 0 | 0 |
| StreamVLN | 25.3 | 23.4 | 24.2 | 26.0 | 0 | 0 | 0 | 0 |
| OmniVLA-edge | 26.0 | 26.0 | 26.0 | 26.0 | 0 | 0 | 0 | 0 |
| LeLaN | 20.5 | 20.1 | 20.5 | 20.5 | 0 | 0 | 0 | 0 |
| RouteGround (Text) | 73.3 | 49.9 | 49.1 | 27.5 | 58.0 | 28.0 | 26.0 | 16.0 |
We have added the full matched-input table to Appendix G (Table S11, lines 1044–1057).
Q1: "Please report the episode-level perturbation trigger rate, displacement magnitude, and route-projection rate, together with a sensitivity analysis that separates rollouts with and without projection."
RouteGround triggers projection least often of all models; the sensitivity analysis without projection is in our response to W1. Share of steps that trigger projection:
| Model | RouteGround | Qwen3-VL-4B | Qwen3-VL-8B | GPT-5.6-sol | Claude-Sonnet-5 | Gemini-3.6-Flash | NaVid | NaVILA | StreamVLN | OmniVLA-edge | LeLaN |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Projection rate (%) | 8.9 | 15.5 | 17.6 | 15.0 | 18.7 | 18.9 | 16.8 | 12.8 | 15.6 | 12.6 | 17.1 |
The application probability of each perturbation level is given in Appendix D.2, and the displacement magnitudes in Table S8.
Q2: "The closed-loop results use a fixed set of 50 routes and one deterministic perturbation plan. Please repeat the evaluation with multiple perturbation seeds and report route-level confidence intervals or bootstrap intervals. In particular, please clarify whether the 26.0% result under low errors, the 32.0% result under medium errors, and the main model ranking are stable."
Across three perturbation seeds, RouteGround ranks first at every level, and the low/medium inversion in the original run is within seed-to-seed variation. Following your suggestion, we have repeated the 2DGS evaluation with two additional perturbation seeds. The table reports the mean ± standard deviation over the three seeds and the Wilson 95% interval over all 150 episodes:
| 2DGS success (%) | None | Low | Med. | High |
|---|---|---|---|---|
| RouteGround | 47.3 ± 2.3 [39.5, 55.3] | 29.3 ± 3.1 [22.6, 37.1] | 30.0 ± 2.0 [23.2, 37.8] | 19.3 ± 2.3 [13.8, 26.4] |
| Best baseline | 3.3 ± 1.2 [1.4, 7.6] | 0 [0, 2.5] | 0 [0, 2.5] | 0 [0, 2.5] |
Low and medium errors give nearly equal means (29.3% vs. 30.0%), and low exceeds medium in two of the three seeds, so the original 26.0% vs. 32.0% gap reflects seed variation rather than a systematic inversion. No baseline exceeds 4.0% in any seed.
Q3: "The 2DGS evaluation uses only the matrixcity_street_clean scene. If feasible, please reproduce the experiment in another 2DGS scene or an independent city block to determine whether RouteGround's closed-loop advantage depends on the road structure or rendering characteristics of the current scene."
We acknowledge that a single scene limits the closed-loop evidence, but RouteGround's advantage does not appear tied to this scene's rendering or road structure.
Request-to-arrival evaluation needs a city-scale scene in which an agent can walk kilometer-scale routes. When we built CityLoop, we found no open benchmark that offered this, and MatrixCity was the most suitable open-source city-scale scene we found. Rendering alone is not enough: the loop also needs POIs, user requests, a road graph, and a route service, none of which exist for this scene, so we built all of them. To our knowledge, this is the first closed loop from request to arrival in a reconstructed city (see also Table 1).
On rendering, RouteGround is trained only on real Street View images, which differ substantially in appearance from the 2DGS renderings, and no 2DGS data are used at any training stage, so the closed-loop result is already a zero-shot transfer. On road structure, the keypoint evaluation spans 43 cities and 121 Street View regions, where RouteGround leads at every error level, and the physical robot completes routes on real streets.
Q4: "Each row in Table 3 changes the data type, sample count, and training process. Is the intended claim that the added data composition is effective, or that the sequential curriculum itself is necessary? Without an order-controlled comparison, the text should limit the conclusion to the gain from adding conflict and full-route data."
Thank you for raising this point; we find it highly meaningful. Following your suggestion, we have run order-controlled comparisons that keep the training data fixed and change only the training order. Because understanding the navigation context is the basis of every later capability, Stage I always comes first; we then swapped Stages II and III, and separately Stages III and IV. Both swapped orders perform worse than the original sequence (swapping II and III: keypoint accuracy 61.4%/42.0% under clean/medium errors; swapping III and IV: 2DGS success 14.0%/8.0%), consistent with our expectation that the sequential curriculum itself is effective, beyond the added data composition. As in human learning, these capabilities are acquired more effectively step by step, each building on the previous one.
We thank Reviewer PXz9 for the constructive feedback and for recognizing that CityLoop provides a tightly controlled setting for evaluating conflicts between navigation tools and egocentric visual evidence and that the results are consistent with the failure mode RouteGround is designed to address. We address each of your comments below.
Route projection does affect success; without it RouteGround's success decreases but remains the only non-zero result, and projection mirrors the safety mechanisms of real deployment. Thank you for pointing out this issue.
When an action would leave the walkable area, such as stepping onto the vehicle road, the simulator blocks it with a geofence and projects the agent onto the nearest point of the route. We evaluated two alternatives:
All other models reach 0% under both alternatives.
We keep projection because it reflects real deployment. Pedestrians do not walk onto the vehicle road, and deployed robots enforce the same constraint with a geofence. When a robot drifts, an operator or safety layer moves it back to the nearest walkable point on its route, which is what projection does; in our robot trials, operators made similar corrections (12%, Table S9). Projection therefore keeps the simulator in the same distribution as physical deployment rather than idealizing it.
Higher levels apply perturbations more often, especially within turn windows, where they can change the next turn (Appendix D.2, lines 958–966; Table S8, lines 925–930). The stability of the low- and medium-error results across perturbation seeds is reported in our response to Q2.