- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CityGRASP is an agent framework for open-vocabulary urban 3D grounding that dynamically adapts search and geometric code generation through execution feedback, using verification, revision, and selective recovery to correct earlier decisions and reuse valid computations. Experiments on three benchmarks report higher grounding accuracy than the evaluated baselines, while component ablations and restart comparisons support the effectiveness of its main designs.
Critical strengths
S1. The integration of search, geometric computation, and error correction is well motivated. The system uses execution feedback to adjust where to search, how to perform geometric computations, and whether earlier decisions need revision, while reusing computations that remain valid. This design fits the challenges of uncertain reference objects and complex spatial relationships in urban grounding. S2. The experiments provide substantial evidence that the components are useful. With the backbone models and initial candidates held fixed, removing progressive program generation reduces accuracy by approximately 28 and 34 percentage points in the two CityRefer settings. Selective recovery also reduces average runtime from 43.5 to 24.1 seconds compared with a full restart. These results support the practical value of the proposed designs.
Critical weaknesses
W1. The additional benefit of the specialized workflow over a general agent has not been directly established. The ablations mainly compare CityGRASP with reduced variants of its own system. They do not include a general agent that uses the same models and tools but autonomously organizes its planning and retries. Thus, the effectiveness of individual components is supported, but how much the complete specialized workflow adds over a general agent remains unclear. W2. Some key execution and comparison details are missing. These include how the system terminates and produces a prediction when verification remains unsuccessful, as well as the exact model and image processing procedure used for CitySTAR’s visual verification. These details affect the reproducibility of accuracy, computational cost, and baseline comparisons. This is currently a documentation and verifiability gap; there is no evidence that the reported results are incorrect.
Questions for authors
Q1. How much does the specialized workflow improve over a general agent under comparable conditions? Could you compare accuracy and actual computational cost while holding the language and vision models, basic tools, initial information, and budget limits fixed, and allowing a general agent to autonomously organize search, code execution, and retries? Q2. Could you clarify the visual model used in your CitySTAR baseline and provide a comparison with the same visual backbone? In your CitySTAR implementation, Qwen3-Reranker-4B is listed as the visual verification model, although Qwen’s official documentation identifies it as a text reranking model. Please clarify the exact checkpoint and how images are processed in your implementation. Since CitySTAR and CityGRASP already share Qwen3.5-9B and SAM3-based candidate construction, why were their visual backbones not aligned? Could you provide an additional comparison using the same visual model, such as Qwen3-VL-4B-Instruct, while retaining each system’s reasoning workflow? This would help distinguish improvements from the proposed workflow from those due to differences in visual model capabilities. Q3. When does the system stop, and what happens if verification remains unsuccessful? What are the inference and retry limits for each query? If the acceptance criteria remain unmet, how is the final prediction produced? What proportion of queries fall into this category, and what is their accuracy?
5 comments
Response to Reviewer q1Sm (2/3): Visual Model Clarification and Matched-Backbone Comparisons
2. Visual Model Clarification and Matched-Backbone Comparisons
We thank the reviewer for carefully identifying this error. After checking the model-loading code and inference calls, we confirm that our CitySTAR baseline uses Qwen3-VL-Reranker-8B, which we mistakenly reported as the text-only Qwen3-Reranker-4B. We sincerely apologize for the confusion and will correct the model name throughout the manuscript.
We would also like to explain why we originally selected different visual models. For CitySTAR, we use the 8B reranker to score candidate-centered renderings against the language query, then combine these relevance scores with topology scores for final selection. For CityGRASP, we deliberately selected Qwen3-VL-4B-Instruct to evaluate intermediate visual constraints during agent-based coding and spatial reasoning. Its instruction-following capability also supports our broader goal of extending the agent to urban reasoning tasks beyond grounding.
Nevertheless, we understand and share the reviewer’s concern about backbone differences. We have therefore added CitySTAR–4B and CityGRASP–8B experiments to compare both methods under each shared visual backbone. We retain their respective reasoning workflows and keep Qwen3.5-9B, SAM3-based candidate construction, scene inputs, rendering, and preprocessing fixed.
For CitySTAR–4B, we normalize the Yes/No answer-token logits into a visual relevance score and retain the original topology fusion. For CityGRASP–8B, we adapt vlm_verify to calibrate the reranker’s per-view relevance scores using backbone-specific parameters. We then apply the original multi-view aggregation to obtain mean support and empirical bounds.
| Method | Visual model | CityRefer NO [email protected] (%) | CityRefer ND [email protected] (%) | Time/query (s) |
|---|---|---|---|---|
| CitySTAR | Qwen3-VL-Reranker-8B | 71.68 | 80.68 | 11.6 |
| CityGRASP | Qwen3-VL-Reranker-8B | 75.13 | 85.46 | 27.9 |
| CitySTAR | Qwen3-VL-4B-Instruct | 66.84 | 71.31 | 9.69 |
| CityGRASP | Qwen3-VL-4B-Instruct | 74.61 | 84.26 | 24.1 |
CityGRASP retains its accuracy advantage on both splits under either shared backbone. Replacing CitySTAR’s original 8B model with 4B yields net losses of 28 NO and 47 ND correct predictions, reducing pooled accuracy by 6.94 percentage points. In comparison, replacing CityGRASP’s original 4B model with 8B yields net gains of 3 NO and 6 ND correct predictions, increasing pooled accuracy by 0.83 percentage points. These results show that CityGRASP is less sensitive to this visual-model substitution.
The runtime results also support our original model choices. Replacing CitySTAR’s original 8B reranker with the 4B instruction model reduces mean runtime from 11.6 to 9.69 seconds (16.5%), alongside a substantial accuracy loss. Replacing CityGRASP’s original 4B instruction model with the 8B reranker increases runtime from 24.1 to 27.9 seconds (15.8%), with a comparatively modest accuracy gain. CitySTAR therefore benefits substantially from the 8B reranker, while the 4B instruction model provides CityGRASP with a favorable balance of accuracy, efficiency, and flexibility.
Response to Reviewer q1Sm (3/3): Stopping Criteria and Unverified Predictions
3. Stopping Criteria and Unverified Predictions
We thank the reviewer for raising this important question. We understand the concern about how the system concludes when verification remains unsuccessful, and we agree that the stopping conditions and output rules should be stated explicitly.
CityGRASP accepts a prediction when the criteria in Section 3.5 are satisfied. These criteria require support for all necessary constraints under consistent reference bindings, every represented search branch to be explored or closed by evidence, and the applicable separation condition in Eqs. (4)–(5). Otherwise, the agent gathers additional evidence, revises references or search scopes, or repairs the program while preserving valid computations.
We impose a per-query execution budget and allow up to five recovery attempts. Each logical model request permits at most two retries after the initial attempt. Once the five recovery attempts are exhausted, the agent can continue ordinary search within the remaining execution budget. We terminate inference when that budget is exhausted, no admissible action remains, or an unrecoverable error occurs.
When inference ends without successful verification, we return a Top-1 prediction marked as unverified. We retain the current provisional prediction if available; otherwise, we select the candidate with the highest recorded support. If no usable score exists, we attempt visual scoring within the remaining budget and use deterministic candidate-ID ordering as the final fallback. We record the verification status and stopping reason.
To quantify these outcomes, we examined all 1081 CityRefer validation queries. The table reports the proportion of non-accepted queries and their [email protected], calculated within each non-accepted subset.
| Setting | Total queries | Non-accepted queries | Non-accepted proportion (%) | Correct non-accepted predictions | Non-accepted [email protected] (%) |
|---|---|---|---|---|---|
| NO | 579 | 128 | 22.11 | 89 | 69.53 |
| ND | 502 | 58 | 11.55 | 43 | 74.14 |
| Overall | 1,081 | 186 | 17.21 | 132 | 70.97 |
We interpret the higher non-acceptance rate in NO as reflecting the greater difficulty of establishing target–context correspondences for unseen instances. NO places greater demands on reference matching and spatial disambiguation, whereas ND evaluates new descriptions. Nevertheless, our returned predictions correctly localize 69.53% of non-accepted NO queries and 74.14% of non-accepted ND queries. A prediction can therefore identify the correct target even when the available evidence does not satisfy every verification condition.
Response to Reviewer q1Sm (1/3): Controlled Comparison with a General Agent
1. Controlled Comparison with a General Agent
We thank the reviewer for this suggestion and agree that a general-agent comparison is important for assessing the contribution of our specialized workflow. We have therefore added a comparison between CityGRASP and a general coding agent, reporting both grounding accuracy and actual computational cost.
The baseline follows a CodeAct-style interaction loop and uses the same Qwen3.5-9B reasoning model and Qwen3-VL-4B-Instruct visual model as CityGRASP. It invokes the same
vlm_verifyprimitive and accesses the same entities, bounding boxes, maps and projected images. The agent autonomously organizes search, executes Python code, and retries or revises its actions using execution feedback, without following CityGRASP’s state machine.In our initial runs, unrestricted exploration led to excessive runtime and occasional non-termination, while the agent made limited use of visual evidence. We therefore introduced landmark-first search with local inspection restricted to the 20 nearest candidates around the selected anchor, and provided projected candidate observations for appearance-based reasoning. We also added intermediate prediction saving, enforced termination under execution limits, and bounded request retries. Within these constraints, the agent determines its own sequence of searches, code executions and revisions. The general agent is limited to 16 outer model requests per query.
We evaluate both methods on 1081 CityRefer validation queries, comprising 579 NO and 502 ND queries, using the same 3D IoU protocol. We also report average runtime, model requests and token consumption per query.
CityGRASP improves [email protected] by approximately 55.4 percentage points on NO and 67.5 percentage points on ND. It reduces average runtime by 80.2% with similar model-call counts and 1.6% lower token consumption. These results demonstrate a substantial advantage over the general coding agent and support the value of task-specific organization of agent-based coding and spatial reasoning for city-scale 3D grounding.