acceptodds
Under review as a conference paper at ICLR 2027

Beyond Answer Correctness: Executable Derivation Faithfulness for Urban Spatial Reasoning

Abstract

Urban spatial reasoning typically relies on external tools to acquire spatial evidence and on multi-step derivations to reach final decisions. However, answer correctness does not imply derivation faithfulness: models may produce correct answers despite misusing spatial evidence or making errors in intermediate derivations. Outcome-only evaluation and training cannot distinguish these trajectories from faithful ones that yield the same answer. To address this issue, we propose UrbanSDF (Urban Spatial Derivation Faithfulness), a framework for derivation faithfulness in urban spatial reasoning. UrbanSDF explicitly models how entity bindings determine tool arguments, how returned observations support spatial computations, and how these computations determine final decisions. To verify these dependencies, we construct executable derivation certificates that assess derivation faithfulness separately from outcome correctness. The resulting feedback guides a progressive training strategy: after initial tool-protocol supervision, we address detected errors through targeted process supervision with faithful demonstrations, improving the model's ability to generate faithful trajectories under limited sampling. Building on this supervision, we employ Conditional-Dual GRPO optimization during online interaction, using separate outcome and process feedback to favor faithful derivations among answer-correct trajectories. Experiments on four types of spatial reasoning tasks show that UrbanSDF maintains task accuracy while substantially increasing the proportion of faithful trajectories among those yielding correct answers, with an average improvement of 45.6% over the strongest baseline. Our code and data are available at https://anonymous.4open.science/r/UrbanSDF-5A02.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Urban Spatial Reasoning, Urban Intelligence, Derivation Faithfulness, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eunDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper argues that in tool-augmented urban spatial reasoning, a correct final answer does not imply a faithful derivation. It proposes a five-layer verifier/certificate used both to evaluate and to train, through three stages: Tool-SFT, certificate-guided Process-SFT, and Conditional Dual-GRPO. Experiments use about 6,000 OSM-based tasks across four task types and five cities, and report large gains in faithful accuracy (FA) and in the faithful share of correct answers (FC) over baselines.

Critical strengths
  1. The problem is well-motivated. Decoupling answer correctness from derivation faithfulness matters for urban spatial reasoning.
  2. The verifier is interpretable. Its layers map to distinct reasoning stages. The layer-wise failure analysis shows where derivations break.
  3. The work covers a broad range of evaluations such as four backbones, cross-city transfer, staged ablations and so on.
Critical weaknesses
  1. The verification and scoring procedure are underspecified and unvalidated. It is unclear how the inputs to the checks are obtained from model’s raw output; whether the extraction is rule based or uses an LLM. The same gap applies to answer correctness. The paper does not say how the answer is compared with the reference, strict match or something else? A correct answer with any unresolved layer is counted as invalid, and "insufficient evidence" is never defined. These matters for the reliability of metrics. The verifier's reliability is never assessed, for example, through agreement with human judgment or error rates.
  2. The baseline comparison is confounded by output format. A trajectory is only faithful if it explicitly reports every required intermediate value in a parseable form. Only UrbanSDF is trained to produce this format, while C-GRPO and Tree-GRPO outputs were "converted to our common trajectory representation.” It is not described how it was converted. Therefore, part of the reported gap may reflect format compliance rather than reasoning ability.
  3. Training and evaluation share the same verifier. UrbanSDF is directly optimized on the certificate. Large gains over methods that never saw this certificate are expected by construction. No independent evidence of faithfulness is given.
  4. The experiments are small. The training has an effective batch size of 2. GRPO makes roughly one pass over 660 tasks, with one optimizer step. Test set has about 325 tasks for the main evaluation, across four task types and three cities. Results come with no confidence intervals to exclude noise.
Questions for authors
  1. Please report exact counts of tasks for each stage (Tool-SFT, Process-SFT, GRPO, dev, test), broken down by city and task type, and reconcile them with the stated ratios. Which test set does each table use?
  2. Please report the full GRPO configuration. Some key setups are missing, for example, learning rate, temperature, maximum length, LoRA settings…
  3. How were baselines trained and run in this environment? What budgets and hyperparameters did they use?
9 comments
AuthorsOpenReview ↗

Author Response to Reviewer eunD (Weakness 1)

Responce to W1:

We appreciate the reviewer’s suggestion. To address the unclear verification and scoring rules and the lack of independent reliability assessment in the original manuscript, we have clarified the procedures for trace parsing, answer matching, and certificate verification, and added an independent blind review by human experts and an LLM judge. Details are provided in the Appendix (p. 17, Trace Parsing and Answer Matching Section; p. 16, Verifier Reliability Assessment Section).

(1) Verification and Scoring Procedure.

UrbanSDF separately evaluates answer correctness and reasoning faithfulness. First, JSON parsing, regular expressions, and predefined alias mappings are used to extract entity grounding, tool calls, parameter references, intermediate computations, and final answers from raw model outputs. Tool inputs and outputs are obtained from actual GIS execution logs. The entire extraction process is rule-based and requires no additional LLM.

Answer correctness is determined by exact matching: the model's final answer is normalized to a task-defined option ID and compared with the ground-truth answer. Reasoning faithfulness is verified using the five-layer certificate (L1–L5), with spatial computations independently recalculated from actual GIS results. Notably, L5 checks whether the final answer is supported by the model's own reasoning conclusions, rather than comparing it again with the ground truth.

Certificate verification distinguishes three states: Pass, Fail, and Unresolved. Pass indicates that verification requirements are satisfied; Fail denotes confirmed errors based on the trace and execution logs, including clearly identifiable missing required operations or fields; and Unresolved (Insufficient Evidence) indicates insufficient prior tool observations, parameter provenance, or intermediate reasoning evidence to determine validity. Inapplicable checks are excluded from scoring. Only traces with correct answers and all applicable checks passed are classified as faithful (F), while traces containing unresolved checks are classified as indeterminate (I), separately from confirmed reasoning failures.

(2) Agreement Between Automated Certificates and Independent Reviews.

To assess the reliability of automated certificate verification, we sample 25 real model traces from each of the four GIS tasks and conduct independent blind reviews by human experts and an LLM judge. Model identities and automated certificate results are concealed. Reviewers assess answer correctness and compliance with each certificate layer (L1–L5) based solely on task images, original reasoning traces, and replayed GIS execution logs. We then compare the cross-checked review judgments with automated certificate results and calculate agreement rates for cases with definitive judgments. The results are presented below.

\hline \mathbf{Verification\\;Item} & \mathbf{Compared\\;Traces} & \mathbf{Matching\\;Judgments} & \mathbf{Agreement\\;(\\%)} \\\\ \hline \text{Answer Correctness} & 98 & 92 & 93.9 \\\\ \text{L1 Entity Grounding} & 81 & 76 & 93.8 \\\\ \text{L2 Tool Dependencies} & 97 & 88 & 90.7 \\\\ \text{L3 Parameter Provenance} & 93 & 82 & 88.1 \\\\ \text{L4 Spatial Derivation} & 62 & 57 & 91.9 \\\\ \text{L5 Answer Entailment} & 75 & 70 & 93.3 \\\\ \hline \end{array}$$ The results show 93.9% agreement on answer correctness and 88.2%–93.8% agreement across the five certificate layers, with four layers exceeding 90.0%. These findings indicate high consistency between automated certificate verification and independent blind reviews on most assessable samples, providing evidence for the reliability of automated evaluation of answer correctness and reasoning faithfulness.
AuthorsOpenReview ↗

Author Response to Reviewer eunD (Weakness 2)

Resonce to W2:

We thank the reviewer for highlighting potential formatting bias in the evaluation of external baselines. Upon re-examining the original trace conversion and certificate verification procedures, we found that overly strict requirements on tool-call ordering and field representations incorrectly rejected some valid reasoning traces with different formats. Consequently, the original FA results underestimated the reasoning faithfulness of certain external baselines. To address this issue, we revised the unified trace parsing and verification rules and conducted a new evaluation. Detailed results and analysis are provided in RQ1 (p. 7) of the revised manuscript.

(1) Unified Trace Parsing and Verification. For all methods, we use the same rule-based parser, combining JSON parsing, regular expressions, and predefined alias mappings to convert raw outputs into a unified interaction trace representation. This includes four types of information: 1) Tool Calls, extracting tool names, parameters, result variables, and their references; 2) Tool Observations, linking outputs directly to actual GIS execution results; 3) Spatial Reasoning, extracting model-generated distance calculations, spatial comparisons, and intermediate conclusions; and 4) Final Answers, extracting the model's submitted option. Details are provided in the Appendix (p. 18, Baseline Adaptation and Trace Normalization Section). The revised verifier accepts valid tool-call reorderings, field aliases, and equivalent computational expressions, evaluating actual tool dependencies rather than enforcing fixed reference templates. The conversion process neither modifies original model calls or GIS outputs nor adds missing reasoning steps. All methods remain subject to the same L1–L5 evidence constraints.

(2) Case Study and Re-evaluation of External Baselines. For example, in a route selection task, the reference derivation queries and analyzes candidate routes sequentially, whereas Tree-GRPO first queries all four routes and then analyzes them separately. Although the execution orders differ, all tool calls and their dependencies in the Tree-GRPO trajectory are correct, and the model reaches the correct final answer. However, the original certificate incorrectly rejects this trajectory because it strictly matches the reference execution order. After revising the verification rules, the trajectory passes all five layers (L1–L5) without modifying the original model output, GIS execution records, or adding any reasoning steps. This confirms that the previous rejection resulted from overly restrictive verification of the derivation format rather than an actual spatial reasoning error. We further re-evaluate C-GRPO and Tree-GRPO under the unified experimental configuration using the revised verification rules. Selected results are presented below.

\hline \mathbf{Method} & \mathbf{Setting} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{C-GRPO} & \text{controlled} & 61.4 & 34.6 & 56.4 \\\\ & \text{Full} & 51.6 & 16.8 & 32.6 \\\\ \text{Tree-GRPO} & \text{controlled} & 61.2 & 42.5 & 69.4 \\\\ & \text{Full} & 51.2 & 16.3 & 31.8 \\\\ \hline \end{array}$$ After re-evaluation, both baselines achieve non-zero FA, compared with the original 0%, indicating that the previous formatting constraints introduced substantial evaluation bias. Importantly, the revision only relaxes trace representation requirements without weakening the verification of tool dependencies, parameter provenance, or spatial computation correctness. Thus, the updated FA more fairly reflects each method's ability to produce verifiable spatial reasoning traces, rather than merely conforming to a fixed output template.
AuthorsOpenReview ↗

Author Response to Reviewer eunD (Weakness 3)

Resonce to W3:

We appreciate the reviewer’s suggestion. We agree that comparisons solely with baselines lacking five-layer certificates cannot fully distinguish the contributions of certificate-based supervision and training mechanisms. To address this, we add controlled experiments using the same five-layer certificate but different training strategies in the Appendix (p. 16, Additional Ablation of Certificate-Guided Supervision and Optimization Section).

Specifically, we first establish a baseline using the full five-layer certificate with full-trajectory Process-SFT and standard Weighted-Reward GRPO, without targeted supervision or Conditional-Dual GRPO, to assess the effectiveness of certificate feedback under conventional training. Next, we introduce targeted Process-SFT while keeping the certificate and GRPO unchanged to evaluate the benefits of supervising erroneous decision segments. Finally, we replace standard Weighted-Reward GRPO with Conditional-Dual GRPO to further assess the contributions of local credit assignment and conditional gating. The results are presented below.

\hline \mathbf{Method} & \mathbf{OA\\;\uparrow} & \mathbf{FA\\;\uparrow} & \mathbf{FC\\;\uparrow} \\\\ \hline \text{Full Certificate + Full-trajectory Process-SFT + Weighted-Reward GRPO} & 60.3 & 25.8 & 42.8 \\\\ \text{Full Certificate + Targeted Process-SFT + Weighted-Reward GRPO} & 63.9 & 28.5 & 44.6 \\\\ \text{UrbanSDF (Ours)} & 66.3 & 33.4 & 50.3 \\\\ \hline \end{array}$$ Compared with full-trajectory supervision, targeted Process-SFT improves FA by 2.7 percentage points, indicating that focusing supervision on erroneous decision segments is more effective than repeatedly learning entire reasoning traces. Introducing Conditional-Dual GRPO further improves FA by 4.9 percentage points, suggesting that separately optimizing outcome and local process rewards, together with conditional gating, enables more effective use of certificate feedback. Overall, UrbanSDF improves FA by 7.6 percentage points over the baseline using conventional certificate-based supervision and standard GRPO, demonstrating that its faithfulness gains arise not only from the five-layer certificate but also from targeted supervision and optimization of certificate feedback.
Reviewer_m4ooOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

UrbanSDF separates final-answer correctness from derivation faithfulness in urban spatial reasoning. UrbanSDF explicitly models how entity bindings determine tool arguments, how returned observations support spatial computations, and how these computations determine final decisions. Its three-stage training uses Tool-SFT, certificate-targeted Process-SFT, and Conditional-Dual GRPO. Roughly 6,000 OpenStreetMap-derived tasks span four task types, UrbanSDF maintains task accuracy while substantially increasing the proportion of faithful trajectories among those yielding correct answers, with an average improvement of 45.6% over the strongest baseline.

Critical strengths

1、Specific process metric. L1–L5 audits tie model statements to actual tool outputs and spatial recomputation, a final correct option will not hide process errors like wrong coordinates or fabricated intermediate values.

2、Staged ablation and multiple contexts. The Process-SFT and dual-GRPO ablations demonstrate gains beyond tool supervision, and cross-backbone plus held-out-Sydney experiments test the approach beyond one checkpoint.

3、Direct check on a consequential failure mode. This work highlights the distinction between answer correctness and valid evidence propagation, then operationalizes it as five linked checks. The separation is particularly useful for map reasoning where an accidentally correct discrete choice can hide wrong route computations. (Fig. 2)

Critical weaknesses

1、Possible benchmark–verifier coupling limits the interpretation. Training demonstrations, local process feedback, and final evaluation use the same task-specific specification family Ωx and deterministic checks. The large differences in faithful accuracy may reflect learning the evaluator’s structured protocol, not necessarily broader semantic faithfulness in genuinely messy urban scenes.

2、The motivation may beyond the results. Although framed as urban spatial reasoning faithfulness, the benchmark consists of generated OpenStreetMap map images and four task templates with prescribed GIS operations. The verifier and agent both operate in a designed symbolic environment, so the method possibly can detect or improves faithfulness under genuine scene or other similar scenes.

3、The decisive new ingredient over existing process supervision is under-isolated. Prior tool-aware rewards, evidence-consistency checks and process supervision are explicitly described in Related Work. UrbanSDF adds cross-step executable provenance and conditional two-branch optimization, but the gap may partly reflect baseline adaptation to a new verifier-specific output schema. A comparison to a generic execution-trace verifier using identical supervision and valid-trace affordances is needed to isolate the new ingredient.

Questions for authors

1、Please check weakness3.

2、Does cross-city performance persist when road-network topology, tool API, map style, is shifted rather than only holding out a city under the same generator?

3、Against a baseline that uses the same five checks solely for process rewards or rejection sampling, but no Conditional-Dual GRPO and no targeted segments, how much improvement comes from the certificate's cross-step dependency verification rather than the proposed optimization machinery?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer m4oo (Weakness 1-2)

Responce to W1:

We appreciate the reviewer’s valuable suggestion. We acknowledge that sharing task specifications between training and evaluation may lead models to rely on specific certificate protocols. To examine this concern, we re-evaluate Tool-SFT, Process-SFT, and UrbanSDF using Qwen3.5-9B under the Full setting, varying tool descriptions, parameter aliases, and evidence representations while keeping task content, GIS data, and ground-truth answers unchanged. We then conduct cross-checked blind reviews of the resulting traces to assess whether UrbanSDF retains its faithfulness advantage. Detailed experimental settings are provided in the Appendix (p. 14, Robustness to Tool Protocol Variations).

\hline \mathbf{Method} & \mathbf{Original\\;OA\\;(\\%)} & \mathbf{Perturbed\\;OA\\;(\\%)} & \mathbf{Original\\;FA\\;(\\%)} & \mathbf{Perturbed\\;FA\\;(\\%)} & \mathbf{Human\\;FA\\;(\\%)} & \mathbf{LLM\\;Judge\\;FA\\;(\\%)} \\\\ \hline \text{Tool-SFT} & 49.0 & 43.0 & 11.0 & 8.0 & 5.0 & 6.0 \\\\ \text{Process-SFT} & 57.0 & 48.0 & 25.0 & 19.0 & 16.0 & 18.0 \\\\ \text{UrbanSDF} & 60.0 & 54.0 & 32.0 & 28.0 & 27.0 & 26.0 \\\\ \hline \end{array}$$ The results show that protocol perturbations reduce FA across all three methods, with Process-SFT experiencing the largest decline. This is because Process-SFT primarily learns tool-call patterns and evidence representations from fixed demonstrations, limiting its ability to adapt to unseen protocol variations. In contrast, UrbanSDF's FA decreases by only 4.0 percentage points, indicating its ability to maintain faithful spatial reasoning under protocol changes. Independent blind reviews by human experts and an LLM judge further show that UrbanSDF achieves 26.5% FA, outperforming Process-SFT and Tool-SFT by 9.5 and 21.0 percentage points, respectively. These findings further demonstrate that UrbanSDF maintains a robust advantage in reasoning faithfulness under tool protocol variations. **Responce to W2:** We sincerely thank the reviewer for their continued interest in our work. We acknowledge that the current benchmark, primarily based on OSM maps and four predefined GIS tasks, does not fully capture the complex visual observations and spatial changes in real-world urban environments. To examine whether UrbanSDF relies on fixed map representations or road network structures, we conduct two controlled perturbation experiments involving map appearance and road topology, evaluating its reasoning faithfulness under visual and network variations. Detailed experimental settings and results are provided in the **Appendix (p. 15, Robustness to Map Appearance and Road Topology Variations section).** $$\begin{array}{llrrr} \hline \mathbf{Perturbation} & \mathbf{Method} & \mathbf{Original\\;FA\\;(\\%)} & \mathbf{Perturbed\\;FA\\;(\\%)} & \mathbf{\Delta FA\\;(pp)} \\\\ \hline \text{Map Style} & \text{Process-SFT} & 25.0 & 22.0 & -3.0 \\\\ & \text{UrbanSDF} & 32.0 & 30.0 & -2.0 \\\\ \text{Road Topology} & \text{Process-SFT} & 25.0 & 24.0 & -1.0 \\\\ & \text{UrbanSDF} & 32.0 & 30.0 & -2.0 \\\\ \hline \end{array}$$ The results show that UrbanSDF's FA decreases by an average of only 2.0 percentage points across both perturbations, demonstrating good robustness. This suggests that UrbanSDF relies on actual GIS observations and cross-step evidence dependencies rather than fixed map appearances or route results. Under map style changes, visual differences may affect entity grounding, but subsequent GIS queries and spatial computations remain valid once the target entities are correctly identified. Under road topology changes, GIS tools return updated routes and distances, allowing the model to adapt its subsequent computations and decisions based on the new evidence rather than relying on the original routes.
AuthorsOpenReview ↗

Author Response to Reviewer m4oo (Weakness 3, Question 1-3)

Responce to W3:

We thank the reviewer for pointing out that the original experiments did not sufficiently distinguish the contribution of cross-step evidence verification from conventional tool execution checks. To address this, we add a controlled comparison between the Generic Verifier and the full five-layer certificate in the Appendix (p. 16, “Additional Ablation of Certificate-Guided Supervision and Optimization”) to isolate the contribution of cross-step dependency verification.

To isolate the contribution of cross-step evidence verification from targeted Process-SFT and Conditional-Dual GRPO, both experiments start from the same full-trajectory Process-SFT checkpoint and use standard Weighted-Reward GRPO, with identical training tasks, sampling budgets, and trace parsing rules. The only difference lies in the verification mechanism used for process rewards: the Generic Verifier checks individual tool executions and computational results, whereas the full five-layer certificate additionally verifies cross-step parameter provenance and tool dependencies. This comparison isolates the contribution of cross-step evidence verification to reasoning faithfulness without introducing targeted supervision or dual-branch optimization.

\hline \mathbf{Method} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{Generic Verifier + Full-trajectory Process-SFT + Weighted-Reward GRPO} & 58.8 & 21.7 & 36.9 \\\\ \text{Full Certificate + Full-trajectory Process-SFT + Weighted-Reward GRPO} & 60.3 & 25.8 & 42.8 \\\\ \hline \end{array}$$ The results show that, under identical training conditions, the full five-layer certificate improves FA by 4.1 percentage points over the Generic Verifier. While single-step execution checks can verify the correctness of tool outputs, they struggle to detect incorrect references to prior spatial data in subsequent calls. In contrast, the full certificate tracks cross-step parameter provenance and tool dependencies, enabling the detection of such errors and the provision of corresponding process feedback. **Responce to Q1:** We sincerely thank the reviewer again for the suggestion. We have addressed this issue in detail in our response to Weakness 3. **Responce to Q2:** We thank the reviewer again for highlighting this issue. We have addressed it in our response to Weakness 2. **Responce to Q3:** We appreciate the reviewer’s suggestion. To distinguish the contributions of the five-layer certificate itself from those of subsequent training mechanisms, we add controlled experiments using the same five-layer certificate but different training strategies in the revised **Appendix (p. 16, “Additional Ablation of Certificate-Guided Supervision and Optimization”).** Specifically, the first setting uses the full certificate with full-trajectory Process-SFT and standard Weighted-Reward GRPO, without targeted supervision or Conditional-Dual GRPO. The second replaces full-trajectory supervision with targeted Process-SFT while keeping the certificate and GRPO unchanged. The third further introduces Conditional-Dual GRPO. All settings use identical training tasks, GIS tool environments, and sampling budgets. $$\begin{array}{lccc} \hline \mathbf{Method} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{Full Certificate + Full-trajectory Process-SFT + Weighted-Reward GRPO} & 60.3 & 25.8 & 42.8 \\\\ \text{Full Certificate + Targeted Process-SFT + Weighted-Reward GRPO} & 63.9 & 28.5 & 44.6 \\\\ \text{UrbanSDF (Ours)} & 66.3 & 33.4 & 50.3 \\\\ \hline \end{array}$$ The results show that, with the same five-layer certificate, targeted Process-SFT improves FA by 2.7 percentage points. Although full-trajectory supervision enables learning from complete correct reasoning traces, its training loss is distributed across all steps, repeatedly supervising already correct operations and diluting the learning signal for critical errors. For example, when a model incorrectly references coordinates from a previous tool output, full-trajectory supervision covers the entire tool-use and reasoning process, whereas targeted Process-SFT uses certificate feedback to identify the parameter error and focuses supervision on correct parameter selection while preserving the necessary context. Further introducing Conditional-Dual GRPO improves FA by another 4.9 percentage points, indicating that separate local credit assignment and conditional gating enable more effective use of certificate feedback.

Thank you for the added content, I will keep the rating of this work.

Reviewer_pyoSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UrbanSDF to distinguish answer correctness from derivation faithfulness in urban spatial reasoning. It introduces five-layer executable certificates for entity grounding, tool dependencies, argument provenance, spatial derivation, and answer entailment. These certificates guide Process-SFT and Conditional-Dual GRPO. Experiments across four GIS tasks, multiple backbones, and cross-city transfer show improvements in faithful accuracy.

Critical strengths
  1. The executable certificates provide a concrete way to identify incorrect intermediate derivations even when final answers are correct. Separating tool interactions, spatial computations, and answer entailment is valuable for evaluating tool-using agents.

  2. UrbanSDF improves both answer accuracy and faithful accuracy. The staged ablations, multiple backbones, and Controlled/Full settings provide useful evidence, while cross-city evaluation further supports generalization.

Critical weaknesses
  1. The paper does not sufficiently justify the choice and granularity of the five verification layers or validate whether alternative correct derivations are accepted, particularly for external baselines.

  2. Most improvements come from SFT, while Conditional-Dual GRPO provides a relatively modest additional gain. Comparisons with simpler reward combinations and ablations of Conditional gating are missing.

  3. Table 3 reports higher L2–L4 failure rates after GRPO, but these rates are calculated only among correct-but-unfaithful trajectories. Without failure rates over all applicable trajectories, it is unclear whether GRPO actually reduces errors at each layer.

  4. Related studies such as VERITAS and Faithful GRPO are not sufficiently discussed, leaving the methodological distinctions unclear.

Questions for authors
  1. Why were these five layers selected? Would finer-grained verification of distance, direction, and topological reasoning provide more informative feedback?

  2. Since tool specifications are already provided to the model, how does Tool-SFT compare with few-shot tool-use prompting? What explains the near-zero faithful accuracy of the untrained Base model despite its nontrivial answer accuracy?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer pyoS (Weakness 1)

Responce to W1:

(1) Design Rationale and Verification Granularity of the Five-Layer Certificate.

The five-layer certificate is designed to capture multi-stage evidence propagation across tools in urban spatial reasoning. Such tasks require grounding visual map objects in geographic entities, retrieving spatial information through multiple GIS tool calls, and integrating tool outputs for computation and decision-making. Thus, reasoning faithfulness requires not only correct computations and answers but also consistent and traceable evidence dependencies.

Accordingly, we define five verification layers, with granularity determined by evidence units that can be independently verified and used to localize errors: L1 Entity Grounding verifies geographic entities against their task roles; L2 Tool Dependencies checks required operations and their dependencies rather than a fixed execution order; L3 Parameter Provenance verifies the source and value of each tool parameter; L4 Spatial Reasoning checks independently reproducible distance, direction, route changes, and related decisions; and L5 Answer Entailment verifies consistency between the final answer and the model's reasoning conclusions.

(2) Robustness to Reasoning Trace Variations.

To assess whether the certificates depend on fixed reference traces, we keep the task specifications and verifier unchanged and apply tool-call reordering, equivalent computation transformations, and parameter reference substitution. These tests evaluate the certificates' ability to accept valid alternative reasoning traces while enforcing evidence provenance constraints. Detailed settings and analysis are provided in Appendix (p. 13, Table 6).

\hline \text{Transformation Type} & \text{Number of Traces} & \text{Accepted Traces} & \text{Acceptance Rate} \\\\ \hline \text{Tool-Call Reordering} & 100 & 100 & 100\\% \\\\ \text{Equivalent Computation} & 100 & 100 & 100\\% \\\\ \text{Parameter Reference Substitution} & 100 & 75 & 75\\% \\\\ \hline \end{array}$$ The results show that both tool-call reordering and equivalent computation achieve 100% acceptance rates, indicating that the certificates are independent of fixed execution orders or computational expressions. For parameter reference substitution, all 75 traces with valid prior evidence are accepted, while 25 unsupported references are rejected. These results confirm that the certificates accommodate valid alternative reasoning traces while enforcing evidence provenance constraints. (3) **Validation of Alternative Reasoning Traces from External Baselines.** To address the reviewer's concerns about external baselines, we re-examined the original reasoning traces of Spatial-Agent, GeoFlow, C-GRPO, and Tree-GRPO. We found that the previous certificate implementation imposed overly strict constraints on tool-call ordering and field representations, incorrectly rejecting some traces with valid evidence dependencies but different reasoning structures. We therefore revised the unified trace parsing and verification rules to validate actual tool dependencies rather than enforce fixed reference templates. For example, in a route selection task, the reference trace queries and analyzes candidate routes sequentially, whereas Tree-GRPO first queries all four candidate routes before analyzing them individually. Although the tool-call orders differ, all calls and their dependencies in the Tree-GRPO trace are correct. However, the original certificate incorrectly rejected the trace due to strict matching against the reference execution order. After revision, verification relies on actual tool dependencies. Without modifying the original model outputs, GIS execution logs, or adding reasoning steps, the trace passes all five certificate layers (L1–L5). We further re-evaluated the external baselines under a unified budget using the revised verification rules. The results are summarized below (**full results in RQ1, p. 7, Table 1**). $$\begin{array}{llccc} \hline \mathbf{Method} & \mathbf{Setting} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{Spatial-Agent} & \text{controlled} & 61.0 & 40.3 & 66.1 \\\\ & \text{Full} & 48.7 & 16.6 & 34.0 \\\\ \text{GeoFlow} & \text{controlled} & 60.6 & 40.8 & 67.4 \\\\ & \text{Full} & 32.1 & 4.1 & 12.9 \\\\ \text{C-GRPO} & \text{controlled} & 61.4 & 34.6 & 56.4 \\\\ & \text{Full} & 51.6 & 16.8 & 32.6 \\\\ \text{Tree-GRPO} & \text{controlled} & 61.2 & 42.5 & 69.4 \\\\ & \text{Full} & 51.2 & 16.3 & 31.8 \\\\ \hline \end{array}$$ After re-evaluation, all external baselines achieve non-zero FA, indicating that prior rules rejected some valid traces. The revision removes rigid ordering and field-format constraints while preserving checks on provenance, tool dependencies, and spatial correctness. These results confirm acceptance of diverse valid reasoning traces.
AuthorsOpenReview ↗

Author Response to Reviewer pyoS (Weakness 2-3)

Responce to W2:

We sincerely thank the reviewer again for their valuable suggestions.We acknowledge that Process-SFT accounts for most of the performance gains, while Conditional-Dual GRPO serves a complementary role. Process-SFT establishes foundational spatial evidence reasoning capabilities through targeted supervision from correct demonstrations, whereas Conditional-Dual GRPO addresses decision errors arising during online GIS interactions by separately assigning credit based on outcome rewards and local certificate feedback, while using conditional gating to reduce low-information updates and further improve reasoning faithfulness.

To further examine their performance differences, we compare four strategies: (1) Outcome-only GRPO, using only final answer rewards; (2) Weighted-Reward GRPO, combining outcome and certificate-based process rewards within standard GRPO; (3) Dual-GRPO w/o Conditional Gating, computing separate outcome and local process advantages without gating; and (4) Conditional-Dual GRPO, incorporating both dual-branch optimization and conditional gating. Detailed settings and analysis are provided in the Appendix (p. 14, Additional Ablation Studies on GRPO Optimization).

\hline \mathbf{Method} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{Process-sft} & 62.6 & 28.6 & 45.6 \\\\ \text{+ Outcome-only GRPO} & 63.0 & 28.4 & 45.1 \\\\ \text{+ Weighted-Reward GRPO} & 63.9 & 28.5 & 44.6 \\\\ \text{+ Dual-GRPO w/o Conditional Gating} & 65.2 & 31.4 & 48.2 \\\\ \text{+ Conditional-Dual GRPO (Ours)} & 66.3 & 33.4 & 50.3 \\\\ \hline \end{array}$$ The results show that although Weighted-Reward GRPO incorporates certificate-based process rewards, combining them with outcome rewards into a single advantage signal limits fine-grained credit assignment for local reasoning errors, resulting in a 0.3% relative decrease in FA compared with Process-SFT. In contrast, UrbanSDF separately computes outcome advantages and certificate-guided local process advantages, achieving 3.8% and 17.2% relative improvements in OA and FA, respectively, over Weighted-Reward GRPO. Incorporating reward-variance-based conditional gating further improves OA and FA by 1.7% and 6.4%, respectively, compared with Dual-GRPO without gating, suggesting that gating helps reduce low-information updates. Overall, these results indicate that local credit assignment and conditional gating further improve spatial reasoning faithfulness beyond Process-SFT. **Responce to W3:** We thank the reviewer for the suggestion. To better assess how GRPO reduces errors across verification layers, we recalculate failure rates using all applicable traces at each layer as the denominator. Using Task 4 (Target Search Direction) as an example, we analyze errors in target retrieval, directional statistics, and road-network distance comparisons. Detailed settings and results are presented in **RQ4 (p. 8, Fig. 3).** $$\begin{array}{lccc} \hline \mathbf{Layers} & \mathbf{Process\text{-}SFT} & \mathbf{+\\;Dual\text{-}GRPO} & \mathbf{change} \\\\ \hline \text{L1} & 20.96\\% & 15.63\\% & -5.33\\,\text{pp} \\\\ \text{L2} & 3.38\\% & 1.81\\% & -1.57\\,\text{pp} \\\\ \text{L3} & 12.08\\% & 8.88\\% & -3.20\\,\text{pp} \\\\ \text{L4} & 32.04\\% & 24.81\\% & -7.23\\,\text{pp} \\\\ \text{L5} & 4.44\\% & 0.75\\% & -3.69\\,\text{pp} \\\\ \hline \end{array}$$ The results show that Dual-GRPO reduces failure rates across all five verification layers, with the largest reduction of 7.23 percentage points at L4. This indicates that GRPO not only changes the distribution of errors among remaining failed traces but also reduces verification failures at each layer.
AuthorsOpenReview ↗

Author Response to Reviewer pyoS (Weakness 4, Question 1-2)

Responce to W4:

We appreciate the reviewer’s insightful feedback. We have added discussions of VERITAS and Faithful GRPO in the Related Work section (p. 3, Paragraph 1) and further clarified their methodological differences from UrbanSDF.

Specifically, VERITAS primarily verifies semantic consistency among reasoning, search actions, and retrieved evidence in multi-turn retrieval, while Faithful GRPO focuses on logical consistency and visual evidence alignment in visual spatial reasoning. Both provide effective mechanisms for evaluating and improving reasoning faithfulness, but neither establishes executable verification of cross-step data provenance, information flow, and tool execution dependencies in GIS interactions.

This distinction is particularly important for tool-augmented urban spatial reasoning. Models must not only align their reasoning with observations but also correctly propagate and use spatial data from prior GIS tool calls. For example, even if route distances and subsequent comparisons are correct, the reasoning remains unfaithful if the route query starts from an entrance other than the one specified in the task. Such errors require verification of tool parameter provenance and cross-step computational dependencies, rather than relying solely on textual or visual evidence consistency.

To address this issue, UrbanSDF employs five-layer executable certificates to verify entity grounding, tool dependencies, parameter provenance, spatial reasoning, and answer entailment. It further leverages local certificate feedback for targeted Process-SFT and Conditional-Dual GRPO, enabling executable verification of cross-step spatial evidence dependencies and localized decision optimization.

Responce to Q1:

We sincerely thank the reviewer for raising this important question. The design rationale for the five-layer certificates has been addressed in Weakness 1. We further clarify that the five layers are organized around the flow of spatial evidence from entity grounding to final decision-making, rather than by spatial reasoning types such as distance, direction, or topology.

For finer-grained verification, UrbanSDF incorporates independent checks for different types of spatial reasoning within L4. Specifically, T1 verifies segment distance aggregation and route comparisons; T3 checks post-closure reachability, route changes, and impact classification; and T4 verifies directional partitioning, target counts, and road-network distance statistics. Each check produces an independent verification result and local score, with feedback assigned to the corresponding decision segment.

Responce to Q2:

We thank the reviewer for the valuable suggestion. To compare Tool-SFT with few-shot tool-use prompting, we evaluate Base Zero-shot, Base 2-shot, and the corresponding Tool-SFT checkpoint on 100 test instances using Qwen3.5-9B under the Full setting. Detailed analysis is provided in the Appendix (p. 14, Effectiveness of Tool-SFT in Faithful Reasoning).

\hline \mathbf{Method} & \mathbf{OA\\;(\\%)\\;\uparrow} & \mathbf{FA\\;(\\%)\\;\uparrow} & \mathbf{FC\\;(\\%)\\;\uparrow} \\\\ \hline \text{Base Zero-shot} & 32.0 & 0.0 & 0.0 \\\\ \text{Base 2-shot} & 49.0 & 1.0 & 2.0 \\\\ \text{Tool-SFT} & 49.0 & \mathbf{11.0} & \mathbf{22.4} \\\\ \hline \end{array}$$ The results show that 2-shot prompting improves OA by 17.0 percentage points, but FA remains at only 1.0%, indicating limited gains in reasoning faithfulness. In contrast, Tool-SFT improves FA by 10.0 percentage points and FC by 20.4 percentage points at the same OA. Further analysis reveals that although tool specifications and few-shot examples can guide tool use, they do not ensure the correct propagation and use of spatial evidence across steps. None of the 32 correctly answered Base Zero-shot tasks pass the full certificate, while only 1 of the 49 correctly answered 2-shot tasks passes. For example, in a road-closure impact assessment task, the tool returns identical pre- and post-closure route distances of 551.826 m, implying zero distance change. However, the Base model incorrectly classifies the route as unreachable while still selecting the correct final answer, No Impact. Moreover, both Base settings achieve 100% output parsing success, confirming that the low FA reflects deficiencies in multi-step spatial evidence utilization and reasoning rather than parsing failures.

Thank you for the detailed rebuttal.

My remaining concern is about the significance and novelty. The challenges mainly stem from the process verification of general agents, rather than unsolved problems inherent to spatial reasoning. The method is closer to an adaptation of process-verification ideas for GIS tool settings.

Nevertheless, the authors have adequately addressed my other concerns. I am willing to raise my score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.