- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper present RaCE, an inference framework for urban spatial multi-hop reasoning that predicts a complete route and its distance category from navigation instructions, visual observations, and a road graph. Its central motivation is that early location errors change subsequent spatial contexts, allowing incorrect routes to remain locally plausible. The main contributions are history-dependent spatial state modeling, comparable visual–geometric evidence accumulated over complete routes, and adaptive visual evidence acquisition. RaCE retains alternative routes, updates each branch’s spatial reference, and combines independently assessed visual matches with geometric penalties. It selects the highest-scoring route while separately aggregating route evidence for answer prediction. Adaptive acquisition uses bounds on both route and answer scores to avoid unnecessary visual queries while preserving exhaustive inference outputs under the same scoring and tie-breaking rules. The framework requires no additional backbone fine-tuning. Validation uses approximately 10,000 OSM-based instances across five cities, four general-purpose VLMs, and two spatially post-trained models. RaCE achieves 77.7% mean route accuracy and 92.1% mean answer accuracy across the four general-purpose backbones. On Qwen3.5, adaptive acquisition reduces average visual queries from 21.0 to 3.3. Ablations, ambiguity analyses, cross-city transfer, and scaling experiments provide further validation, with the strongest ablation evidence supporting geometry, spatial-reference updates, and cumulative scoring. Evaluation remains primarily limited to rendered map scenes.
Critical strengths
Strong evidence for the value of explicit, history-dependent spatial grounding. RaCE updates each branch’s spatial reference and separately evaluates visual semantic matches and geometric constraints, directly addressing the changing context of multi-hop spatial reasoning. In Table 2, removing geometry or fixing the spatial reference reduces Route ACC from 91.1% to 10.1% and 14.7%, respectively. The complete framework also achieves mean route and answer accuracies of 77.7% and 92.1% across four general purpose backbones. These results substantiate the usefulness of the spatial grounding mechanism and demonstrate benefits across multiple models, supporting the framework’s practical value within the evaluated setting.
Adaptive visual acquisition with a well-defined output-preservation guarantee. The method propagates bounds on unqueried evidence and separately certifies the best route and answer, correctly accounting for the distinction between route selection and answer aggregation. Under identical scoring and tie-breaking rules, the procedure preserves exhaustive inference outputs and terminates after finitely many queries. On Qwen3.5, it reduces average visual-query events from 21.0 to 3.3. This combination of a clear theoretical guarantee and substantial measured query savings is a substantive technical strength: it offers a principled way to reduce expensive perception while preserving the decisions of the full scoring framework.
Critical weaknesses
The distinctive benefit of preserving competing routes is not convincingly established. In Table 2, the w/o Branch variant already achieves 89.6% Route ACC, compared with 91.1% for full RaCE, whereas removing geometric evidence reduces accuracy to 10.1%. These results support the importance of geometric scoring and evidence accumulation, but leave unclear how much of the large improvement over existing baselines comes from preserving alternative histories and reassessing earlier decisions. Since Beam 4 uses a different scoring scheme, a comparison using the same local scores and state updates, together with a geometry-only control, is needed to isolate this contribution. This uncertainty substantially weakens the evidence for the claimed methodological advance.
Then, the bench construction and partitioning are insufficiently specified to assess the validity of the empirical conclusions.Although the paper reports dataset size and split ratios, it does not adequately describe instruction and reference-route generation, distractor selection, or geographic overlap across calibration, validation, and test sets. These details are consequential for OSM-derived instances that may share POIs, routes, or map regions, and for understanding how much ambiguity the candidate selection procedure introduces. The cross city evaluation protocol also needs a clearer account of sample selection and parameter fitting. Because the main empirical claims depend on this benchmark, these omissions limit reproducibility and prevent a confident assessment of the reported reasoning gains and generalization.
Questions for authors
-
Contribution of branch preservation. Does the w/o Branch variant retain the same candidate construction, visual calibration, geometric scoring, and spatial-reference updates as full RaCE? Given its 89.6% Route ACC versus 91.1% for the full method in Table 2, how many test instances are corrected or made incorrect by preserving alternative branches, particularly within the high-ambiguity subset?
-
Benchmark construction and candidate ambiguity. How are the navigation instructions, valid reference routes, and four candidate choices generated and verified? To what extent do reference-label generation and distractor selection use the same geometric rules as RaCE’s scorer, and how frequently do multiple candidates satisfy both the direction and distance constraints?
-
Geographic independence and cross-city evaluation. What geographic overlap exists across the calibration, validation, and test sets at the map-region, POI, and route levels? For cross-city evaluation, which data are used to fit calibration parameters and select prompts and the geometric weight, and is the target city excluded from all these choices?
-
End-to-end computational cost. Appendix C measures latency from the first visual query, whereas Algorithm 1 constructs the route space and geometric states beforehand. Do the reported latencies include this initialization? What is the full end-to-end inference time, and how do the reported logical visual-query counts correspond to actual model forward passes?
-
Differences between the main and scaling experiments. What differences in evaluation subsets or protocols explain SpatialReasoner-R1’s 68.5% Route ACC in Table 1 versus 29.0% in Table 4 under the nominally same three-hop, four-candidate setting? Please clarify the sample sizes and any changes in dataset construction, sampling, prompts, or model configuration.
-
Answer aggregation and output consistency. How do the authors explain the increase in Answer ACC from 95.7% to 96.8% when answer pooling is removed in Table 2? How often does the predicted answer differ from the distance category implied by the selected route, and what proportion of instances have both a correct route and a correct answer?
-
Treatment of uncertain visual evidence. Why is the unknown probability excluded when visual evidence is computed as ? Identical support-to-contradiction ratios produce identical visual scores even when the unknown probabilities differ substantially. How is this uncertainty accounted for, and how do high-unknown-probability instances perform in the reported evaluation?
Minor:The text in several figures, particularly Figures 1, 2, and 5, is difficult to read at normal viewing size. Please enlarge labels, legends, and equations, and simplify crowded annotations to improve readability.
I am open to revising my score if the authors can provide satisfactory clarifications that address the concerns raised above.
update
The authors have clarified the role of branch retention, the benchmark setup, computational costs, and the differences between the main and scaling experiments. They have also explained why answer pooling can reduce accuracy and assessed the impact of excluding unknown probabilities from visual scoring. Thanks for the responses and my main concerns have been addressed. I revised my score accrodingly.
5 comments
Author Response to Reviewer WhY1 (Weakness 2)
Responce to W2:
We thank the reviewer again for pointing out the insufficient details regarding benchmark construction and data partitioning. To improve experimental reproducibility, we have added three subsections to page 17 of the revised manuscript's appendix: Instruction and Route Construction, Data Partitioning and Geographic Overlap, and Cross-City Evaluation and Parameter Calibration. The details are provided below.
(1) Instruction, Reference Route, and Distractor Candidate Construction.
The benchmark constructs three-hop route reasoning tasks based on the OSM road network. First, three target locations are sequentially selected to form a reference route based on road reachability, distance, and POI category constraints. Navigation instructions are then generated according to the target categories, travel directions, and approximate block-level distances at each hop. Given the limitations of POI categories and their spatial distributions in OSM, we require adjacent hops to have different target categories while allowing the first and third hops to share the same category. Next, four candidate POIs are assigned to each hop, with three distractors selected from nearby road-reachable locations, prioritizing those with similar categories, directions, or distances. Alternative routes that also satisfy the complete instructions are excluded.
To quantify candidate ambiguity, we counted the number of candidates satisfying both directional and distance constraints at each hop in the test set. The results are presented below.
\hline \mathbf{Decision\\;Hop} & \mathbf{1\\;Candidate} & \mathbf{2\\;Candidates} & \mathbf{3\\;Candidates} & \mathbf{Ambiguous\\;(\geq2)} \\\\ \hline \text{Hop-1} & 326 & 451 & 223 & 67.4\\% \\\\ \text{Hop-2} & 357 & 433 & 210 & 64.3\\% \\\\ \text{Hop-3} & 393 & 419 & 188 & 60.7\\% \\\\ \text{Total} & 1{,}076 & 1{,}303 & 621 & 64.1\\% \\\\ \hline \end{array}$$ (2) Data Partitioning and Geographic Overlap. The dataset consists of 6,000 training instances, 2,000 calibration instances, 1,000 validation instances, and 1,000 test instances. The splits are partitioned by task source to prevent duplicate questions and similar tasks generated from the same source from appearing across different splits. The calibration set is used for fitting scoring parameters. Since different navigation tasks within the same city may share the road network, we do not enforce strict geographic separation between splits, thereby preserving road network continuity and spatial coverage of candidate POIs. The overlap of roads, POIs, and reference routes across different splits is reported below. $$\begin{array}{lrrrr} \hline \mathbf{Comparison} & \mathbf{Shared\\;Nodes} & \mathbf{Shared\\;Edges} & \mathbf{Shared\\;POIs} & \mathbf{Shared\\;Reference\\;Routes} \\\\ \hline \text{Test / Train} & 24.60 & 21.40 & 16.70 & 0.90 \\\\ \text{Train / Test} & 11.43 & 2.70 & 5.85 & 0.18 \\\\ \text{Test / Validation} & 15.80 & 10.60 & 9.80 & 0.40 \\\\ \text{Validation / Test} & 13.10 & 6.40 & 8.70 & 0.50 \\\\ \text{Test / Calibration} & 8.10 & 3.90 & 4.30 & 0.20 \\\\ \hline \end{array}$$ (3) Cross-City Sample Selection and Parameter Calibration. Cross-city evaluation follows a Leave-One-City-Out protocol. In each round, all test instances from the target city are selected from the frozen Test1000 set, without additional random sampling, difficulty-based filtering, or selection based on prediction results. Model parameters are determined using data from the remaining four source cities only. Specifically, the visual calibration parameters $T$ and $b$ are fitted using L-BFGS-B by minimizing the candidate-level, equally weighted binary negative log-likelihood (NLL) on the source-city calibration sets. The geometric weight $\lambda_{\mathrm{geo}}$ is selected from ${1,2,4,8,16,32\}$ based on the equally weighted macro-average Route ACC across source cities on the validation set. The dataset sizes and parameter configurations for each city are presented below. $$\begin{array}{lrrrrrr} \hline \mathbf{Held\text{-}out\\;City} & \mathbf{Source\\;Calib.} & \mathbf{Source\\;Val.} & \mathbf{Target\\;Test} & \mathbf{T} & \mathbf{b} & \lambda_{\mathrm{geo}} \\\\ \hline \text{New York City} & 92 & 728 & 96 & 1.17 & 1.09 & 16 \\\\ \text{Berlin} & 111 & 936 & 148 & 1.14 & 1.09 & 16 \\\\ \text{Singapore} & 68 & 640 & 59 & 1.10 & 0.94 & 16 \\\\ \text{Sydney} & 106 & 936 & 48 & 1.13 & 1.10 & 16 \\\\ \text{Tokyo} & 83 & 760 & 649 & 1.16 & 1.17 & 16 \\\\ \hline \end{array}$$Author Response to Reviewer WhY1 (Question 1-2)
Responce to Q1:
We thank the reviewer again for their attention to this issue. We first clarify that the w/o Branch variant in Table 2 uses exactly the same candidate construction, visual calibration, visual and geometric scoring, spatial reference frame updates, and cross-hop evidence accumulation as the full RaCE model. The only difference is whether multiple candidate branches are retained. The w/o Branch variant retains only the highest-scoring branch at each hop, whereas RaCE maintains multiple route hypotheses and uses subsequent evidence to reassess them at the complete-route level. Therefore, this ablation study directly measures the additional contribution of the multi-branch retention mechanism.
To further analyze the impact of the multi-branch mechanism on route decisions, we grouped the full test set by candidate ambiguity level and conducted a case-level analysis to count the complete routes corrected and the new errors introduced by RaCE compared with the w/o Branch variant.
\hline \mathbf{Ambiguity} & \mathbf{Samples} & \mathbf{w/o\\;Branch\\;Route\\;ACC\\;(\\%)} & \mathbf{RaCE\\;Route\\;ACC\\;(\\%)} & \mathbf{Corrected} & \mathbf{New\\;Errors} \\\\ \hline \text{Low} & 299 & 94.0 & 94.0 & 1 & 1 \\\\ \text{Medium} & 269 & \mathbf{94.1} & 92.9 & 2 & 5 \\\\ \text{High} & 432 & 83.8 & \mathbf{88.0} & 30 & 12 \\\\ \hline \end{array}$$ The results show that the benefits of multi-branch retention are primarily concentrated in high-ambiguity scenarios, rather than consistently improving performance across all samples. In the low-ambiguity group, both methods achieve a Route ACC of 94.0%. In the medium-ambiguity group, RaCE underperforms by 1.2 percentage points, mainly because some incorrect branches receive higher scores from subsequent visual evidence, causing previously correct routes to be replaced. In contrast, in the high-ambiguity group, RaCE successfully corrects 30 complete-route errors, improving Route ACC by 4.2 percentage points. It is important to note that branch retention inevitably allows incorrect routes to receive higher scores due to misleading later evidence. Because local visual and geometric scoring cannot fully resolve candidate ambiguity, multi-branch reasoning does not guarantee that every reassessment improves the decision. However, sequential decision-making methods such as CoT and Spatial-Agent, as well as limited-width search methods such as Beam-4, may discard correct routes through early commitments or intermediate pruning, preventing later evidence from correcting these errors. In contrast, RaCE retains all candidate branches and uses subsequent evidence to reassess earlier choices. Thus, the main value of branch retention is not to eliminate all local decision errors, but to avoid prematurely discarding correct routes when early evidence is insufficient, preserving opportunities for later correction. **Responce to Q2:** We thank the reviewer again for the suggestion. We have addressed the construction of navigation instructions, reference routes, and candidates, as well as candidate ambiguity, in our response to **Weakness 2**. Here, we further clarify the relationship between reference label generation and RaCE's geometric scoring. The reference routes are determined in advance based on road reachability, distance, and POI category constraints, rather than selected using RaCE's fused scores. Distractor selection and RaCE's scoring function use the same definitions of direction and distance: the geometric penalty is zero when the candidate's bearing deviation from the current anchor does not exceed 50°, and its block count, obtained by dividing the Euclidean distance by 180 meters per block and rounding up, satisfies the instruction. Thus, both share the same geometric criteria, but the reference labels are not determined by RaCE's fused scores. Since multiple candidates may satisfy these constraints, geometric scoring alone cannot uniquely identify the correct route; POI semantics and subsequent spatial relationships are also needed.
Author Response to Reviewer WhY1 (Weakness 1)
Responce to W1:
We thank the reviewer for their attention to the multi-branch mechanism. We agree that the original ablation study demonstrates the contributions of individual components to overall performance but lacks a direct comparison with Beam Search under the same scoring mechanism. To address this, we have added two sets of controlled experiments in the Contribution of Branch Retention and Route-Level Comparison subsection on page 16 of the revised manuscript's appendix.
Specifically, Matched Greedy, Matched Beam-4, and RaCE use the same local visual–geometric scoring, spatial reference frame updates, and cross-hop evidence accumulation, but differ in their search strategies: step-wise greedy selection, fixed-width Beam Search, and complete-route comparison, respectively. Geometry-only retains RaCE's search strategy while fixing the visual evidence factor to 1, allowing us to isolate the contribution of geometric evidence.
\hline \mathbf{Method} & \mathbf{Search\\;Strategy} & \mathbf{Route\\;ACC\\;(\\%)} \\\\ \hline \text{Matched Greedy (w/o Branch)} & \text{Greedy (top-1)} & 89.6 \\\\ \text{Matched Beam-4} & \text{Beam search (width 4)} & 90.4 \\\\ \text{Full RaCE} & \text{Route-level comparison} & \mathbf{91.1} \\\\ \text{Geometry-only} & \text{Route-level comparison} & 49.8 \\\\ \hline \end{array}$$ The results show that under the same scoring and state update mechanisms, RaCE improves Route ACC by 1.5 and 0.7 percentage points over Matched Greedy and Matched Beam-4, respectively. Multi-branch retention prevents premature route selection by step-wise greedy decisions, while RaCE's additional advantage over fixed-width Beam Search lies in reducing the loss of candidates caused by intermediate pruning. When the correct route falls outside the top four in cumulative score at a given hop, Matched Beam-4 prunes it prematurely, making recovery impossible even if subsequent evidence supports that route. In contrast, RaCE retains more competing route hypotheses, allowing initially lower-scoring candidates to be reassessed using subsequent evidence. Combined with the ambiguity-group analysis in RQ3, where RaCE outperforms Greedy by 4.2 percentage points in the high-ambiguity group, these findings further indicate that the benefits of branch retention are primarily concentrated in scenarios where early candidates are difficult to distinguish.