- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper argues that in tool-augmented urban spatial reasoning, a correct final answer does not imply a faithful derivation. It proposes a five-layer verifier/certificate used both to evaluate and to train, through three stages: Tool-SFT, certificate-guided Process-SFT, and Conditional Dual-GRPO. Experiments use about 6,000 OSM-based tasks across four task types and five cities, and report large gains in faithful accuracy (FA) and in the faithful share of correct answers (FC) over baselines.
Critical strengths
- The problem is well-motivated. Decoupling answer correctness from derivation faithfulness matters for urban spatial reasoning.
- The verifier is interpretable. Its layers map to distinct reasoning stages. The layer-wise failure analysis shows where derivations break.
- The work covers a broad range of evaluations such as four backbones, cross-city transfer, staged ablations and so on.
Critical weaknesses
- The verification and scoring procedure are underspecified and unvalidated. It is unclear how the inputs to the checks are obtained from model’s raw output; whether the extraction is rule based or uses an LLM. The same gap applies to answer correctness. The paper does not say how the answer is compared with the reference, strict match or something else? A correct answer with any unresolved layer is counted as invalid, and "insufficient evidence" is never defined. These matters for the reliability of metrics. The verifier's reliability is never assessed, for example, through agreement with human judgment or error rates.
- The baseline comparison is confounded by output format. A trajectory is only faithful if it explicitly reports every required intermediate value in a parseable form. Only UrbanSDF is trained to produce this format, while C-GRPO and Tree-GRPO outputs were "converted to our common trajectory representation.” It is not described how it was converted. Therefore, part of the reported gap may reflect format compliance rather than reasoning ability.
- Training and evaluation share the same verifier. UrbanSDF is directly optimized on the certificate. Large gains over methods that never saw this certificate are expected by construction. No independent evidence of faithfulness is given.
- The experiments are small. The training has an effective batch size of 2. GRPO makes roughly one pass over 660 tasks, with one optimizer step. Test set has about 325 tasks for the main evaluation, across four task types and three cities. Results come with no confidence intervals to exclude noise.
Questions for authors
- Please report exact counts of tasks for each stage (Tool-SFT, Process-SFT, GRPO, dev, test), broken down by city and task type, and reconcile them with the stated ratios. Which test set does each table use?
- Please report the full GRPO configuration. Some key setups are missing, for example, learning rate, temperature, maximum length, LoRA settings…
- How were baselines trained and run in this environment? What budgets and hyperparameters did they use?
9 comments
Author Response to Reviewer eunD (Weakness 2)
Resonce to W2:
We thank the reviewer for highlighting potential formatting bias in the evaluation of external baselines. Upon re-examining the original trace conversion and certificate verification procedures, we found that overly strict requirements on tool-call ordering and field representations incorrectly rejected some valid reasoning traces with different formats. Consequently, the original FA results underestimated the reasoning faithfulness of certain external baselines. To address this issue, we revised the unified trace parsing and verification rules and conducted a new evaluation. Detailed results and analysis are provided in RQ1 (p. 7) of the revised manuscript.
(1) Unified Trace Parsing and Verification. For all methods, we use the same rule-based parser, combining JSON parsing, regular expressions, and predefined alias mappings to convert raw outputs into a unified interaction trace representation. This includes four types of information: 1) Tool Calls, extracting tool names, parameters, result variables, and their references; 2) Tool Observations, linking outputs directly to actual GIS execution results; 3) Spatial Reasoning, extracting model-generated distance calculations, spatial comparisons, and intermediate conclusions; and 4) Final Answers, extracting the model's submitted option. Details are provided in the Appendix (p. 18, Baseline Adaptation and Trace Normalization Section). The revised verifier accepts valid tool-call reorderings, field aliases, and equivalent computational expressions, evaluating actual tool dependencies rather than enforcing fixed reference templates. The conversion process neither modifies original model calls or GIS outputs nor adds missing reasoning steps. All methods remain subject to the same L1–L5 evidence constraints.
(2) Case Study and Re-evaluation of External Baselines. For example, in a route selection task, the reference derivation queries and analyzes candidate routes sequentially, whereas Tree-GRPO first queries all four routes and then analyzes them separately. Although the execution orders differ, all tool calls and their dependencies in the Tree-GRPO trajectory are correct, and the model reaches the correct final answer. However, the original certificate incorrectly rejects this trajectory because it strictly matches the reference execution order. After revising the verification rules, the trajectory passes all five layers (L1–L5) without modifying the original model output, GIS execution records, or adding any reasoning steps. This confirms that the previous rejection resulted from overly restrictive verification of the derivation format rather than an actual spatial reasoning error. We further re-evaluate C-GRPO and Tree-GRPO under the unified experimental configuration using the revised verification rules. Selected results are presented below.
\hline \mathbf{Method} & \mathbf{Setting} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{C-GRPO} & \text{controlled} & 61.4 & 34.6 & 56.4 \\\\ & \text{Full} & 51.6 & 16.8 & 32.6 \\\\ \text{Tree-GRPO} & \text{controlled} & 61.2 & 42.5 & 69.4 \\\\ & \text{Full} & 51.2 & 16.3 & 31.8 \\\\ \hline \end{array}$$ After re-evaluation, both baselines achieve non-zero FA, compared with the original 0%, indicating that the previous formatting constraints introduced substantial evaluation bias. Importantly, the revision only relaxes trace representation requirements without weakening the verification of tool dependencies, parameter provenance, or spatial computation correctness. Thus, the updated FA more fairly reflects each method's ability to produce verifiable spatial reasoning traces, rather than merely conforming to a fixed output template.Author Response to Reviewer eunD (Weakness 3)
Resonce to W3:
We appreciate the reviewer’s suggestion. We agree that comparisons solely with baselines lacking five-layer certificates cannot fully distinguish the contributions of certificate-based supervision and training mechanisms. To address this, we add controlled experiments using the same five-layer certificate but different training strategies in the Appendix (p. 16, Additional Ablation of Certificate-Guided Supervision and Optimization Section).
Specifically, we first establish a baseline using the full five-layer certificate with full-trajectory Process-SFT and standard Weighted-Reward GRPO, without targeted supervision or Conditional-Dual GRPO, to assess the effectiveness of certificate feedback under conventional training. Next, we introduce targeted Process-SFT while keeping the certificate and GRPO unchanged to evaluate the benefits of supervising erroneous decision segments. Finally, we replace standard Weighted-Reward GRPO with Conditional-Dual GRPO to further assess the contributions of local credit assignment and conditional gating. The results are presented below.
\hline \mathbf{Method} & \mathbf{OA\\;\uparrow} & \mathbf{FA\\;\uparrow} & \mathbf{FC\\;\uparrow} \\\\ \hline \text{Full Certificate + Full-trajectory Process-SFT + Weighted-Reward GRPO} & 60.3 & 25.8 & 42.8 \\\\ \text{Full Certificate + Targeted Process-SFT + Weighted-Reward GRPO} & 63.9 & 28.5 & 44.6 \\\\ \text{UrbanSDF (Ours)} & 66.3 & 33.4 & 50.3 \\\\ \hline \end{array}$$ Compared with full-trajectory supervision, targeted Process-SFT improves FA by 2.7 percentage points, indicating that focusing supervision on erroneous decision segments is more effective than repeatedly learning entire reasoning traces. Introducing Conditional-Dual GRPO further improves FA by 4.9 percentage points, suggesting that separately optimizing outcome and local process rewards, together with conditional gating, enables more effective use of certificate feedback. Overall, UrbanSDF improves FA by 7.6 percentage points over the baseline using conventional certificate-based supervision and standard GRPO, demonstrating that its faithfulness gains arise not only from the five-layer certificate but also from targeted supervision and optimization of certificate feedback.
Author Response to Reviewer eunD (Weakness 1)
Responce to W1:
We appreciate the reviewer’s suggestion. To address the unclear verification and scoring rules and the lack of independent reliability assessment in the original manuscript, we have clarified the procedures for trace parsing, answer matching, and certificate verification, and added an independent blind review by human experts and an LLM judge. Details are provided in the Appendix (p. 17, Trace Parsing and Answer Matching Section; p. 16, Verifier Reliability Assessment Section).
(1) Verification and Scoring Procedure.
UrbanSDF separately evaluates answer correctness and reasoning faithfulness. First, JSON parsing, regular expressions, and predefined alias mappings are used to extract entity grounding, tool calls, parameter references, intermediate computations, and final answers from raw model outputs. Tool inputs and outputs are obtained from actual GIS execution logs. The entire extraction process is rule-based and requires no additional LLM.
Answer correctness is determined by exact matching: the model's final answer is normalized to a task-defined option ID and compared with the ground-truth answer. Reasoning faithfulness is verified using the five-layer certificate (L1–L5), with spatial computations independently recalculated from actual GIS results. Notably, L5 checks whether the final answer is supported by the model's own reasoning conclusions, rather than comparing it again with the ground truth.
Certificate verification distinguishes three states: Pass, Fail, and Unresolved. Pass indicates that verification requirements are satisfied; Fail denotes confirmed errors based on the trace and execution logs, including clearly identifiable missing required operations or fields; and Unresolved (Insufficient Evidence) indicates insufficient prior tool observations, parameter provenance, or intermediate reasoning evidence to determine validity. Inapplicable checks are excluded from scoring. Only traces with correct answers and all applicable checks passed are classified as faithful (F), while traces containing unresolved checks are classified as indeterminate (I), separately from confirmed reasoning failures.
(2) Agreement Between Automated Certificates and Independent Reviews.
To assess the reliability of automated certificate verification, we sample 25 real model traces from each of the four GIS tasks and conduct independent blind reviews by human experts and an LLM judge. Model identities and automated certificate results are concealed. Reviewers assess answer correctness and compliance with each certificate layer (L1–L5) based solely on task images, original reasoning traces, and replayed GIS execution logs. We then compare the cross-checked review judgments with automated certificate results and calculate agreement rates for cases with definitive judgments. The results are presented below.
\hline \mathbf{Verification\\;Item} & \mathbf{Compared\\;Traces} & \mathbf{Matching\\;Judgments} & \mathbf{Agreement\\;(\\%)} \\\\ \hline \text{Answer Correctness} & 98 & 92 & 93.9 \\\\ \text{L1 Entity Grounding} & 81 & 76 & 93.8 \\\\ \text{L2 Tool Dependencies} & 97 & 88 & 90.7 \\\\ \text{L3 Parameter Provenance} & 93 & 82 & 88.1 \\\\ \text{L4 Spatial Derivation} & 62 & 57 & 91.9 \\\\ \text{L5 Answer Entailment} & 75 & 70 & 93.3 \\\\ \hline \end{array}$$ The results show 93.9% agreement on answer correctness and 88.2%–93.8% agreement across the five certificate layers, with four layers exceeding 90.0%. These findings indicate high consistency between automated certificate verification and independent blind reviews on most assessable samples, providing evidence for the reliability of automated evaluation of answer correctness and reasoning faithfulness.