acceptodds
Under review as a conference paper at ICLR 2027

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

Abstract

Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.

Primary area:
datasets and benchmarks
Keywords:
Geospatial Reasoning, Spatial Relations, Multimodal Large Language Models, Spatial Editing

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_kjDbOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This is a benchmark paper evaluating multimodal LLMs (close+open) on their ability to reason ofer GIS data and imagery. Further, they evaluate its ability to modify GIS data to satisfy a condition and show that even when the models can reconize geospatial conditions,they cannot reliably make edits that satisdy them.

Critical strengths

For a paper like this -- I think that the contributionhinges on whether the benchmark exposes an existing flaw, and the value is increased when it shows a flaw that you may not have expected. This is exposing a flaw that I have also experienced in other domains. First, a general difficulty in metric reasoning, and second a frustrating inability to suggest edits that satisfy them. It is good to see this in a benchmark. Overall there was not much in a 9 page paper that I could find to complain about. Some minor comments below. It does not analyze the causes or suggest solutions but it does find an issue and provide a benchmark to attack when trying to solve it, which I suppose is the goal of this type of paper.

Critical weaknesses

The most critcial concerns are:

  1. This is a benchmark paper. It identifies a weakness and provides a dataset and metrics to evaluate performance on it. This in principle could be used to improve models. However it does not speculate on the cause of the issues—why metric performance struggles, why editing is often so much worse than recognizing. It does not propose a solution. I am unsure what the bar should be in this paper; however, it would certainly be stronger if these ingredients were present.

  2. It is based on OpenStreetMap data and is trying to measure quantities that I think OSM is not reliable for. For example, in my experience, many roads are "drawn through" in OpenStreetMap. Many annotations are at different levels of completeness. It is unclear to me how much of an impact that itself has on the benchmark. Since they are not measuring whether the maps match, e.g., satellite imagery (this is a major problem with OSM) perhaps the error rate is acceptable, but I think some sort of visual inspection sites to estimate the prevalence of topological issues in the "truth" data may be necessary.

** Post Rebuttal** I feel like the authors addressed my concerns and will raise my score to clear-accept.

  1. The project the geojsons to a metric CRS—okay [human em dash!]. What? UTM zones? And then project the imager to WGS84. Why not the same metric system? This seems like a flaw to me. Bearings and vertical distances will be different. And the magnitudes of the numbers could be way off.... much simpler to reproject both, right? How can you compare distances and bearings when each representation is in a different CRS? I mean, it's possible but seems both error-prone on the human side and unnecessarily challenging to the LLM side. Also the validity of comparing edits to a GeoJSON vs. an image .... again, how can it be expected to separate linestrings? It has to—what? Invent the GeoJSON itself? This was not very clear to me and seems like a different (more challenging) task. I think that if you are giving it a map image, you would need to somehow encode features in it very carefully.

PS: If I were using an LLM -- I would have asked it those questions and possibly accepted its answer rather than put them here.

Questions for authors

You specify that for the 70 edits, the presence of one solution is used to validate. Were these manually done? How are the images used produced? Are the from OSM tile server? Did you render them yourself? How are attributes represented in the images?

What prompts were used? Do you have examples? Is Fig 6 a duplicate of Table 4?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kjDb, Critical Weakness 1 (Part 1), Causes of Weak Metric Performance and the Recognition–Editing Gap

Response

Thank you for encouraging us to explain these difficulties and discuss possible solutions. We regard them as two related challenges: computing precise geometric quantities and using spatial information to generate edits that satisfy the target constraints. Recent studies provide support for this interpretation and suggest concrete directions for improvement.

Metric Measurement and Geometric Computation

We suggest that metric tasks are harder than other spatial tasks because they require more precise geometric information and additional numerical computation. For POI Density in GeoSpatialBench, the model must identify the points covered by a region, count them, compute the region’s area, and calculate the density. These steps add numerical operations beyond relation recognition.

Recent research shows that arithmetic in the studied LLM models relies on heuristics tied to numerical patterns. Some errors occur when these heuristics provide insufficient support for the correct answer (Nikankin et al., 2025). The additional operations can make metric tasks more sensitive to unreliable numerical computation. This also helps explain why precise measurement can remain difficult even when GeoJSON provides explicit coordinates.

For image inputs, models must also recover the geometric detail needed for measurement. Geometric errors may leave a relation category unchanged while pushing a numerical answer beyond the allowed tolerance. In 3D measurement experiments, SD-VLM performs better with depth positional encoding than with depth supplied as additional images or tokens (Chen et al., 2025). This shows that how spatial information is represented matters for quantitative measurement. Together, these findings motivate our interpretation: compared with other spatial relations, metric tasks place greater demands on both the precision of spatial representations and the reliability of numerical computation.

From Relation Recognition to Spatial Editing

Editing requires spatial information to guide the generation of operation parameters or new geometry. A possible source of the gap is that information available for recognition does not consistently influence this generation process. Recent research using hidden-state interventions shows that encoded spatial information affects model outputs differently across layers, token positions, and prompt formats. Visually grounded prompts and text-based reasoning prompts can change how this information reaches the answer (Wang & Xu, 2026). This motivates our interpretation that spatial editing depends on how effectively spatial representations guide the output, beyond whether the relevant information has been encoded.

Recent action-selection research also provides complementary evidence. Reading Right, Acting Wrong finds that models can read facts and apply rules separately, yet fail when they must combine these steps. In controlled tasks, composing the models’ own factual answers through code improves decisions (Li et al., 2026). This points to another possible difficulty in spatial editing: combining recognized relations with target conditions to select an appropriate geometric change.

Manuscript Revision

We expand section 4.4, currently on pp. 8–9 (lines 436–443 and 449–462), to connect metric measurement and computation with geometric representation and numerical reliability. We also explain the recognition–editing gap in terms of how spatial information guides generation and how models combine target constraints.

References

Chen, P., Lou, Y., Cao, S., Guo, J., Fan, L., Wu, Y., Yang, L. F., Ma, L., & Ye, J. (2025). SD-VLM: Spatial measuring and understanding with depth-encoded vision-language models. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, & N. Chen (Eds.), Advances in Neural Information Processing Systems (Vol. 38, pp. 33938–33968). Curran Associates. https://doi.org/10.52202/085713-1140

Li, Y., Krishnan, R., & Padman, R. (2026). Reading right, acting wrong: Diagnosing decision failures of System One models in agent loops [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2610.18579

Nikankin, Y., Reusch, A., Mueller, A., & Belinkov, Y. (2025). Arithmetic without algorithms: Language models solve math with a bag of heuristics. In Y. Yue, A. Garg, N. Peng, F. Sha, & R. Yu (Eds.), International Conference on Learning Representations (pp. 55939–55965). https://proceedings.iclr.cc/paper_files/paper/2025/hash/8c5f30296296d2ae402ebbd09aaa9c12-Abstract-Conference.html

Wang, Z., & Xu, X. (2026). Knowing isn’t always saying: When do spatial encodings reach answers in vision-language models? In H.-Y. Lee, S. Dev, J. J. Li, & D. Elliott (Eds.), Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 52683–52705). Association for Computational Linguistics. https://doi.org/10.18653/2026.emnlp-main.2319

AuthorsOpenReview ↗

Response to Reviewer kjDb, Critical Weakness 1 (Part 2), Possible Solutions

Response (Continued)

Possible Solutions

These explanations suggest two complementary directions.

First, spatial representations could preserve geometric and relational structure more explicitly. Section 4.4 already identifies SPOK as one example. SPOK encodes relative location and connectivity through spatial tokens (F. Huang et al., 2025). Extending such representations to metric computation and constrained editing is a concrete direction to explore. The aim would be to keep task-relevant spatial structure available throughout recognition and generation.

Second, editing could combine model predictions with explicit geometric constraint solving. FirePlace translates language instructions into constraints on object surfaces and uses a solver to produce feasible placements. Its ablations show that replacing constraint solving with model selection among random placements reduces placement quality (I. Huang et al., 2025).

For GeoSpatialBench, this suggests a pipeline that expresses the target object and requested relations as explicit constraints, uses geometric operations to compute measurements and candidate edits, and checks the resulting geometry against those constraints. The existing executor and constraint checker provide a basis for evaluating this approach. Explicit representations would support the use of spatial information, while geometric solvers would support the construction of valid edits.

Manuscript Revision

We add the cited studies and extend the existing “Toward explicit spatial representations” discussion (section 4.4, lines 473–485). This revision clarified the role of SPOK and introduce constraint solving, supported by FirePlace, as a complementary direction.

References

Huang, F., Lv, J.-R., Li, G.-L., & Yue, Y. (2025). SPOK: Tokenizing geographic space for enhanced spatial reasoning in GeoAI. International Journal of Geographical Information Science, 39(12), 2768–2808. https://doi.org/10.1080/13658816.2025.2497810

Huang, I., Bao, Y., Truong, K., Zhou, H., Schmid, C., Guibas, L., & Fathi, A. (2025). FirePlace: Geometric refinements of LLM common sense reasoning for 3D object placement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13466–13476). https://openaccess.thecvf.com/content/CVPR2025/html/Huang_FirePlace_Geometric_Refinements_of_LLM_Common_Sense_Reasoning_for_3D_CVPR_2025_paper.html

AuthorsOpenReview ↗

Response to Reviewer kjDb, Critical Weakness 2, OSM Data Quality and Its Impact on Evaluation

Response

Thank you for highlighting the importance of source data quality and topology checks. We chose OSM for its broad coverage across cities. A shared source allows us to use a common processing pipeline and helps limit differences introduced by combining separate datasets. Within this pipeline, reference answers are computed through GIS operations on the geometry and attributes supplied in each task. For example, POI Density is calculated only from the provided POIs, so its answer does not depend on a complete inventory of real-world facilities.

We use reusable relation-specific templates to construct tasks consistently across cities. Each template assigns roles to the participating objects and defines how the reference answer is computed. City-specific settings, such as field mappings and filtering thresholds, are configured separately. As described in Section 3.2, candidate features are filtered by geometry type and length or area thresholds. Scenes are then exported as GeoJSON with anonymous feature IDs, while records of their origin and construction are retained for auditing. When suitable candidates are insufficient, the pipeline selects other scenes or uses documented controlled constructions.

Each resulting instance undergoes further validation. For QA, we check the geometry and recompute the reference answer. We also verify that the answer options are distinct. For editing, we reject candidates whose initial scenes already satisfy the target constraints. A reference edit must then demonstrate that a valid solution exists.

We complement these automated checks with manual visual inspection of drawing and topological issues. Following the revisions to map rendering, we conducted an additional inspection of the same sampled instances.

Previous inspection: We inspected 300 instances across six cities, comprising 150 QA scenes and 150 Editing tasks. Each city contributed 50 instances, and the sample covered all 28 relations. The inspection identified 17 issues. Eleven concerned drawing and three concerned display; these were judged not to affect the reference answer or edit feasibility and were recorded for revision. The remaining three were Image QA instances that lacked the required layer information.

Additional inspection: Following these findings, we revised the rendering process and included the required layer attributes in the map legends. We then inspected the same 300 instances using the updated metric OSM maps. This inspection identified nine drawing issues, all judged not to affect the tasks. It also identified five input issues: three GeoJSON QA instances lacked the required layer attributes, while two editing prompts contained a specific layer value in an example. We revised the affected prompts to address these issues and verified the changes in the final review.

Neither inspection identified inconsistencies in the task-defined topological relations. We also manually reviewed the 150 reference edits in the sample and confirmed their validity. Appendix C reports the findings from both inspections and the revision status of the identified issues.

Manuscript Revision

We expand Section 3.2 (lines 191–230) to explain this quality-control process, from source selection and template-based construction to automatic checks and manual inspection. The appendix (Appendix C, lines 866–883) additionally reports the inspection criteria, sample coverage, findings, and any corrections.

Reviewer_ZpfrOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces GeoSpatialBench, a benchmark designed to evaluate geospatial reasoning of multimodal large language models (MLLMs) through spatial QAs and geometric editing. The benchmark consists of 8400 questions (4200 QA, 4200 editing questions) derived from OpenStreetMap data of 6 cities, categorized into 28 spatial relations from four relation families (topology, direction, metric, ordering). Evaluating on nine MLLM models lead to the conclusion that (1) GeoJSON produces stronger performance than map images and combining GeoJSON data with the map image doesn't lead to performance improvement, (2) models still lack when it comes to metric relations, and (3) there are performance differences when comparing QA performance and editing success. Based on these, the paper concludes that recognizing spatial relations (in the form of QA) doesn't imply the ability to construct geometries satisfying those relationships.

Critical strengths
  1. Paper is detailed and well-organized. Section 3 clearly describes the process of creating the benchmark, including the data source and tools, along with the evaluation protocol. This makes it easy to follow how the benchmark was developed and evaluated.

  2. The idea of adding spatial editing as an additional way to evaluate models is interesting. It extends beyond traditional question answering by examining whether models can apply spatial relationships to modify geometric configurations.

  3. The benchmark uses real-world geographic sources (OSM) across six cities and deterministic GIS operations for evaluation. This reduces the ambiguity in the ground-truth answers and provides a reproducible evaluation framework.

Critical weaknesses
  1. Some of the paper's conclusions do not appear to be novel. For example, unlike map images, which require models to extract spatial information through visual evidence, GeoJSON provides explicit geometric coordinates and attributes that can be used to compute answers deterministically. Therefore, it is reasonable to expect models to perform better with GeoJSON than with map images (Conclusion 3), even without experimentation. Also, the finding that metric reasoning remains a persistent bottleneck (Conclusion 2) has already been reported in prior work. It is unclear what additional insights these findings provide beyond existing research.

  2. On a similar note, several components of the benchmark have already been explored independently in prior spatial QA and GeoJSON spatial editing work. While combining these two tasks into one unified benchmark is an interesting direction, it is unclear what new insights can be obtained from evaluating them together that cannot be derived independently from the two lines of prior work. The paper would benefit from a clearer explanation of the unique contribution that this combination provides.

  3. It is unclear how editing performance directly becomes complementary to spatial QA. The paper states that successful QA does not necessarily indicate the model understands underlying spatial relations, but it doesn't really tell why and how adding the spatial editing task fills in this gap. In addition, the performance of editing might be dependent on the model's ability to generate valid transformations rather than lacking understanding of these spatial concepts.

Questions for authors
  1. Please specify why one meter for distance, one degree bearing for direction, and 0.001 density was used for evaluating open-ended metric questions. What is the evidence or method justification that supports these thresholds.

  2. Please clarify whether the underlying scenes and target spatial relations used for the 4200 QA and 4200 editings are matched. If not, can the authors provide a paired evaluation using identical scenes and spatial relations to see how often the model correctly answer a spatial-relation question but fails to perform valid spatial editing on the same scene and relation? This analysis will help support the claim that spatial editing adds another dimension in which QAs cannot provide.

  3. Please clarify how the editing tasks acts as a complementary way that supports understanding model's spatial relation understanding.

5 comments

Response to Reviewer Zpfr (1/4): Summary and W1

Dear Reviewer Zpfr,

Thank you for your careful review and for asking what spatial editing adds to spatial QA. Your valuable and constructive Question 2 led directly to the main new experiment of this revision. In short, we made four changes:

(1) Same-scene evaluation. We ran QA and editing on the same 180 scenes with all nine models, using both GeoJSON and the original OSM maps. A correct answer does not guarantee a successful edit. Every model fails some edits after answering correctly: 454 of 1,271 cases (35.7%) with GeoJSON and 687 of 906 (75.8%) with maps (Q2, Q3).

(2) Tolerance check. We rescored all open-ended answers at 0.5×, 1×, 2× and 5× the thresholds. The family order and the model ranking do not change (Q1).

(3) Narrower claims. We describe each task by what it measures, not as "an escalating progression of spatial cognition", "a direct test" or a check of whether models "truly comprehend" relations. We also report where the expected GeoJSON advantage does not hold (W1–W3).

(4) Failure checks and analysis. We classified every failed GeoJSON edit. Section 3 now defines the bearing convention, the density unit, both editing interfaces and the retry rule (W3, Q1).

All changes are marked in blue in the revised PDF. Our point-by-point replies follow each comment. Appendix F of the paper and Supplement Tables S1–S9 give the full numbers.

W1

Some of the paper's conclusions do not appear to be novel. For example, unlike map images, which require models to extract spatial information through visual evidence, GeoJSON provides explicit geometric coordinates and attributes that can be used to compute answers deterministically. Therefore, it is reasonable to expect models to perform better with GeoJSON than with map images (Conclusion 3), even without experimentation. Also, the finding that metric reasoning remains a persistent bottleneck (Conclusion 2) has already been reported in prior work. It is unclear what additional insights these findings provide beyond existing research.

Response. We agree that a GeoJSON advantage and the difficulty of metric reasoning are expected. Our contribution is a benchmark detailed enough to test whether they hold for each relation, model and task.

Coverage and design. GeoSpatialBench covers 28 formally defined relations in four families (topological, directional, metric and ordering) across six cities and nine models. Each QA item is posed with GeoJSON and with a map rendered from the same geometry, in open-ended and multiple-choice form, and editing tests each relation as a constraint. Prior geospatial benchmarks mostly use one representation and QA: map images, imagery with coordinates or raw geometry. Few compare GeoJSON with a map of the same geometry over a broad relation set (Section 2.2). With matched items, an accuracy difference reflects the representation or the relation, not item difficulty.

Findings that the overall trends do not predict.

(a) The GeoJSON advantage depends on the model. On 4,050 matched open-QA items per model, GeoJSON leads maps by +37.6 points for GPT-6 Astra but only +2.1 for InternVL3-8B, and the gap grows with GeoJSON accuracy (Spearman ρ = 0.98; Table S6). Coordinates help mainly the models that can compute with them.

(b) It reverses for one relation. For axial direction (road orientation), maps beat GeoJSON on open QA (56.0% vs. 46.7%) and MCQ-4 (56.8% vs. 43.9%) (Table S7). Orientation is easier to read off a map than to derive from coordinate sequences.

(c) Adding the map makes editing worse. With the output format fixed, adding the map to GeoJSON lowers editing success for all five models with complete results under both inputs, by 1.4 to 10.1 points (McNemar p < 0.01 each; Table 8).

(d) Metric errors are not small imprecisions. For GeoJSON minimum distance, the 25th-percentile error is 0.14 m but the median is 52.6 m (Appendix F.6). Near-exact answers and large failures coexist. Prior work reports that metric reasoning is hard; we locate the failure in the computation, not in access to coordinates.

An overall accuracy score would hide all four patterns.

Response to Reviewer Zpfr (2/4): W2 and W3

W2

On a similar note, several components of the benchmark have already been explored independently in prior spatial QA and GeoJSON spatial editing work. While combining these two tasks into one unified benchmark is an interesting direction, it is unclear what new insights can be obtained from evaluating them together that cannot be derived independently from the two lines of prior work. The paper would benefit from a clearer explanation of the unique contribution that this combination provides.

Response. Separate QA and editing scores cannot tell whether the same model, on the same scene, answers correctly but fails to edit. Evaluating both together can. We now state this explicitly as the contribution.

(a) Across models, QA and editing agree; across relations, they do not. QA and editing rank the nine models almost identically: Spearman ρ = 0.95 on the full benchmark (Table 1) and 0.93 on the 180 paired scenes. Across the 23 non-direction relations, open and multiple-choice QA agree (ρ = 0.922), but open QA and editing do not (ρ = −0.042) (Appendix F.2). QA shows which model is stronger, not which relations it can edit.

(b) On the same scene, a correct answer is often followed by a failed edit. This happens for all nine models (Q2). Touching is the clearest case. With GeoJSON, 99.1% of Touching answers are correct, yet 61.7% of those correct answers are followed by a failed edit.

(c) A correct answer helps but does not guarantee the edit. With GeoJSON, edits succeed after 64.3% of correct answers and 16.9% of incorrect ones (maps: 24.2% vs. 10.9%). Success after an incorrect answer is rare (59 GeoJSON and 78 map edits; Table S8). Answering correctly is closer to a prerequisite for editing than a substitute for it.

Changes in the paper. Sections 1 and 2.3 now state the contribution as a relation-level comparison: how well models identify or measure a relation, and how well they construct geometry that satisfies it. This replaces "a stronger test", "a direct test" and "actually understand". The contribution list in Section 1 no longer says that editing verifies "genuine spatial reasoning abilities". Appendix F.1 and F.2 report the joint outcomes.

W3

It is unclear how editing performance directly becomes complementary to spatial QA. The paper states that successful QA does not necessarily indicate the model understands underlying spatial relations, but it doesn't really tell why and how adding the spatial editing task fills in this gap. In addition, the performance of editing might be dependent on the model's ability to generate valid transformations rather than lacking understanding of these spatial concepts.

Response. We checked whether editing fails mainly because models cannot produce a valid transformation. They usually can: almost all failed edits are valid, changed geometry that misses the target relation.

(a) Failures are valid edits that miss the goal. In the paired test, 447 of the 454 GeoJSON edits that failed after a correct answer (98.5%) produced valid, changed geometry that violated a requested constraint. In a six-model audit of 25,200 original edits, 12,070 of 12,714 failures (94.9%) were of this kind. Only 0.5% were unusable and 4.5% left the geometry unchanged. 96.4% of failed edits used the operation named in the prompt (Table 7).

(b) Example. In Hong Kong scene HKG-ED-M01-018, GPT-5.6 Sol measures the distance between two features as 102.031 m (reference 102.017 m), which is correct. When asked to move one feature so that the distance becomes 8–22 m, it returns an executable translation, but the features end up only 0.010 m apart (Appendix F.1). It measures the relation correctly but does not produce it.

Editing therefore measures one additional step: turning a relation into a geometric change that satisfies it. We do not claim to identify the internal cause of a failure, such as choosing the wrong target or computing the parameters wrongly. The evaluator checks only the outcome. Section 4.4 discusses possible explanations (Wang & Xu, 2026; Li et al., 2026).

Changes in the paper. Section 3.1 replaces "evaluate whether the models truly comprehend spatial relations" with "tests whether a model can use a relation as a constraint when changing a scene". Section 3.3 now explains what each editing interface asks the model to produce. It also explains how the Together condition isolates the visual input while the output format stays fixed. Section 3.4 states the retry rule: at most one corrective retry for an invalid or unchanged output, with no constraint feedback. Section 4.4 adds the discussion above, and Appendix F.3 gives the failure breakdown.

Response to Reviewer Zpfr (3/4): Q1 and Q2

Q1

Please specify why one meter for distance, one degree bearing for direction, and 0.001 density was used for evaluating open-ended metric questions. What is the evidence or method justification that supports these thresholds.

Response. These thresholds are benchmark scoring conventions. They are not taken from an external standard. We now say so and show that our conclusions do not depend on them.

(a) Definitions. An answer is correct if it is within 1 m of the reference for distance and perimeter. Bearings must be within 1° (circular difference), measured clockwise from grid north. Density must be within 0.001 POIs/km², in all six cities. Density is the number of supplied POIs inside or on the polygon divided by its projected area in km². The 0.001 tolerance is one unit in the last decimal of the three-decimal reference.

(b) Sensitivity. Following the multi-threshold evaluation of VSI-Bench (Yang et al., 2025), we rescored the same predictions at 0.5×, 1×, 2× and 5× the thresholds. GeoJSON metric accuracy is 37.0%, 38.5%, 40.6% and 44.2%. Metric remains the hardest of the four families at every setting. The model ranking is stable: Kendall τ with the nominal ranking is 1.00, 0.99 and 0.94, and 0.94 under a 5% relative-error criterion (Table 9).

(c) Maps. On a map, 1 m can be smaller than one pixel, so we also used relative error. On 1,200 matched minimum-distance items, 68.2% of GeoJSON answers and 25.8% of map answers are within 25% of the reference (Appendix F.6). The gap is not an artefact of the threshold.

Changes in the paper. Section 3.4 gives the units and the bearing convention, and points to Appendix F.6, which reports the sensitivity analysis (Table 9).

Q2

Please clarify whether the underlying scenes and target spatial relations used for the 4200 QA and 4200 editings are matched. If not, can the authors provide a paired evaluation using identical scenes and spatial relations to see how often the model correctly answer a spatial-relation question but fails to perform valid spatial editing on the same scene and relation? This analysis will help support the claim that spatial editing adds another dimension in which QAs cannot provide.

Response. They were not matched. The original 4,200 QA and 4,200 editing items share the 28 relations but were built separately (Section 3.2). Following your suggestion, we ran the paired evaluation.

Design. We took 180 existing editing scenes with a fixed seed: two per city and relation, across six cities and the 15 relations where QA and editing test the same relation or quantity. Appendix F.1 explains how all 28 relations were screened. For each scene we wrote an open-ended question about the original geometry. Each model answered the question and attempted the edit in separate calls, once with GeoJSON and once with the original annotated OSM map. All nine models completed both tasks in both formats. Prompts and scoring were fixed after a 46-scene pilot.

Result. The table counts failed edits among the scenes where the same model answered the question correctly.

ModelGeoJSON: edit failed after correct QAOSM map: edit failed after correct QA
GPT-6 Astra1/180 (0.6%)64/139 (46.0%)
KIMI K324/174 (13.8%)73/132 (55.3%)
GPT-5.6 Sol40/154 (26.0%)96/124 (77.4%)
MiMo-V2.6-Pro60/151 (39.7%)101/118 (85.6%)
DeepSeek-V4.1-Flash36/132 (27.3%)103/118 (87.3%)
Qwen3.5-9B49/101 (48.5%)88/89 (98.9%)
Qwen3.8-27B35/130 (26.9%)103/120 (85.8%)
Qwen3-VL-8B-Instruct103/126 (81.7%)37/40 (92.5%)
InternVL3-8B106/123 (86.2%)22/26 (84.6%)
All nine models454/1,271 (35.7%)687/906 (75.8%)

Map editing uses a 4-pixel allowance. Under strict map scoring, 701/906 (77.4%) edits fail (Appendix F.5).

Every model fails some edits after answering correctly. With GeoJSON the rate ranges from 0.6% (GPT-6 Astra) to 86.2% (InternVL3-8B). With maps it ranges from 46.0% to 98.9%. The pattern holds in every relation family and every city: in each city, 31–39% of correct answers are followed by a failed edit with GeoJSON, and 72–80% with maps. A correct answer raises the chance of a successful edit but does not ensure it (W2c). The gap narrows for stronger models: with GeoJSON, GPT-6 Astra fails only 1 of 180 edits after a correct answer, but 46.0% with maps.

Changes in the paper. Section 4.4 points to the new experiment. Appendix F.1 (Table 6) and F.5 describe the paired design, the relation screening and the map scoring. The paired prompts, model responses, reference answers and scorer are added to the reproduction package.

Reviewer_7776OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoSpatialBench, a benchmark for geospatial question answering and constrained geometric editing over 28 relations in four families: topology, direction, metric, and ordering. It contains 4,200 QA and 4,200 separate editing instances, built from OpenStreetMap features and controlled geometric constructions across six cities. QA is presented open-ended and as four-option multiple choice; editing asks for a change to a designated feature F0 while the rest of the scene is held fixed. Inputs are GeoJSON, an annotated map rendered from the same geometry, or both. Nine models are scored against GIS-derived references: editing by constraint satisfaction (CSR) for GeoJSON and Together, and by a raster-tolerant criterion for map images. GeoJSON generally yields stronger QA performance than map images while adding images offers no reliable benefit; the editing comparison additionally changes the output interface and scoring rule. Relation-level results show substantially different difficulty profiles for recognition and editing, particularly for touching, ordering, and endpoint connectivity.

Critical strengths
  1. Constrained editing is a useful complement to spatial QA. Models must specify geometric or attribute changes that an executor applies to a designated feature, and the result is checked against explicit relation constraints rather than a described relation. Feasibility is verified with a reference edit, and under CSR any alternative edit satisfying the constraints is accepted, so scoring is not tied to that reference. (§3.1–3.4)

  2. The shared scenes support informative representation comparisons. QA uses the same scene, question, and reference answer across all three conditions, and for editing GeoJSON and Together share the structured interface and differ only in the added map. Representation effects are therefore separable from item content. (§3.3–3.4; Table 1)

  3. Relation-level reporting exposes heterogeneity that aggregates average away. GeoJSON multiple-choice error ranges from 1.63% (T01) to 62.44% (M18), and GeoJSON editing error from 0.07% (O06, attribute-dependent) to 78.07% (M01), so failures concentrated in individual relations remain visible instead of being absorbed into a family mean. (Tables 1–4, Figure 6)

Critical weaknesses
  1. Image-editing success can include outputs that violate the requested constraints under strict checking. §3.4 accepts either metric-space constraint satisfaction or Hausdorff distance ≤ 4 px from the reference, subject to matching required attributes. Because this is a disjunction it is a strict superset of CSR, so the effect is one-sided leniency, and its magnitude is what matters: the ground-distance equivalent of the threshold is not reported, and geometric proximity need not preserve topological predicates. Report strict CSR on the same image predictions, the fraction accepted only through the proximity branch, and the pixel-to-ground conversion.

  2. The QA–editing comparison does not isolate recognition-to-editing transfer. §2.3 calls it a direct test, but the pools are unpaired and editing instances are selected to initially violate the requested constraints. The within-pool contrast between Equality (3.26% editing error) and Touching (71.19%) is suggestive but is not a transfer test. Report final-attempt failures by invalid output, unchanged geometric edit, and constraint violation, and either add a recognition probe on editing scenes with editing success conditioned on correct recognition, or temper the transfer claim.

  3. Missing evaluation specifications limit independent auditability. Beyond the CSR definition, the QA tolerances, and the retry rule, the paper does not give the content of C_i: per-relation predicates and editing constraints, the numerical treatment of touching and endpoint connectivity, full prompts, transformation schemas, decoding settings, and an evaluator release (anonymized, consistent with double-blind). Without these, it is unclear how much of the construction the executor performs and which edits count as successes. Also clarify how the 28 relations were selected, since their IDs are nonconsecutive.

Questions for authors
  1. Image-editing scoring. On the same image-editing predictions, what is the strict constraint-satisfaction rate, and what fraction of accepted outputs pass only the Hausdorff ≤ 4 px branch while violating the requested constraints? Does this branch enforce any relation-specific condition for touching, equality, or endpoint connectivity, and what ground-distance range does four pixels represent across scenes?
  2. Evidence for recognition-to-editing transfer. Given that QA and editing use unpaired pools and editing scenes are selected to initially violate the requested constraints, what evidence in the current evaluation supports interpreting their performance gap as a failure of recognition-to-editing transfer? In particular, was recognition evaluated on the editing scenes, and are final editing failures separated into invalid outputs, unchanged geometric edits, and valid edits that violate constraints?
  3. Evaluation specification and availability. Are the complete per-relation predicates and editing constraints, numerical tolerances, task prompts, transformation schemas, decoding settings, and evaluator implementation available for review? In particular, how are touching and endpoint connectivity checked, and does the structured executor perform any geometric solving beyond applying the model’s predicted transformation parameters?
4 comments

Response to Reviewer 7776

Thank you for the constructive assessment. We have checked the same image-editing predictions under strict constraints, evaluated recognition on the editing scenes, and assembled anonymous evaluation materials organized as Q1, Q2, and Q3. These additions quantify the scoring allowance and provide direct, task-matched evidence about recognition and editing. Below we address each question and identify focused manuscript clarifications.

Q1 Editing scoring across input formats

We report both inputs on the same eight models with complete six-city coverage, comprising 33,600 final predictions per input. GeoJSON CSR aggregates the recorded final constraint-based scores. For Image, we independently recompute strict constraint-satisfaction rate (CSR) and success with the four-pixel proximity allowance on the same saved predictions:

MeasureCountResult
GeoJSON CSR17,085 / 33,60050.85%
Image strict CSR6,334 / 33,60018.85%
Image success with proximity6,447 / 33,60019.19%
Accepted only through proximity113 / 6,4471.75%
Increase over strict CSR113 / 33,6000.34 pp

The proximity branch applies only to Image. It checks Hausdorff distance and required attributes, without independently enforcing touching, equality, or endpoint-connectivity predicates. We therefore report strict Image CSR separately from success including proximity. Removing this branch leaves the Image model ranking unchanged.

Using each scene’s coordinate transform, four pixels correspond to approximately 0.053–108.72 m along the horizontal or vertical image axis at the map center. This conversion varies across scenes and is not a bound over the entire image. Q1 provides the scene-level conversions, prediction records, and scoring code needed to reproduce these results.

Q2 Recognition on editing scenes

We agree that the original unpaired QA and editing pools alone do not isolate recognition-to-editing transfer. We added parallel GeoJSON and Image recognition probes on all 700 Stockholm editing tasks, covering all 28 relations, with Qwen3.5-9B, Qwen3.8-27B, Qwen3-VL-8B-Instruct, and InternVL3-8B. Each probe is matched to the same model’s final editing predictions in the corresponding input format.

Each task produces two independent YES/NO queries: one on the initial scene and one on a feasible reference-edited scene. The correct answers are NO and YES, respectively. For Cardinal 4, Cardinal 8, Bearing, and Observer side, the query includes initial and current geometry or images to evaluate observed displacement and movement limits. Other relations use the current scene. Each input format yields 1,400 queries, 5,600 responses, and 2,800 model–task pairs.

The Image probe retains the original editing-input canvases and renders reference states in the same coordinate frame. Models receive images, the stated condition, and rendering scale, with layer attributes supplied for Layer order; no geometry coordinates or gold answers are provided. We recompute strict CSR from the saved final Image Editing predictions using Q1’s evaluator, excluding proximity acceptance.

InputRecognition groupPairsSuccessful editsEditing CSR
GeoJSONBoth states correct45031570.00%
GeoJSONAt least one state incorrect235064227.32%
GeoJSONAll pairs280095734.18%
ImageBoth states correct230125.22%
ImageAt least one state incorrect257025810.04%
ImageAll pairs28002709.64%

Even when both scene states are recognized correctly, editing still fails in 135 of 450 GeoJSON pairs and 218 of 230 Image pairs, showing that correct recognition does not guarantee successful editing on the same task. These are subsets of the 2,800 pairs evaluated for each input format. Requiring both answers to be correct excludes constant YES or NO responses; for example, Qwen3.8-27B answered NO to all 1,400 Image queries and therefore contributed no pairs to the both-correct subset.

The paired experiments show that correctly recognizing both scene states does not guarantee successful editing. Comparisons within the same model and relation remain inconclusive, so we interpret these results as evidence of recognition–editing discrepancies rather than proof of a causal transfer effect.

We also classify final failures for both inputs on the same eight models and six cities, with 33,600 final outputs per input. Image failures use strict scoring:

Final failure categoryGeoJSONImage
Invalid or unparseable output695,278
Numerically unchanged geometry5910
Valid output failing constraints15,85521,988
All failed outputs16,51527,266

We classify failed edits as invalid outputs, unchanged geometry, or valid edits that violate the constraints. Attribute-only edits are exempt from the unchanged-geometry check. Q2 provides the analyses, raw responses, and paired records for both input formats. We will describe the results as a task-matched comparison of recognition and editing, rather than “a direct test” of transfer.

Q3 Evaluation specification and availability

The anonymous Q3 supplement provides evaluation specifications and implementation for all 28 relations, with both GeoJSON and Image input paths:

Requested componentMaterial provided in Q3
Predicates and constraintsShared QA predicates and metric Editing constraints for all 4,200 six-city tasks; both input paths.
Numerical tolerancesQA gold tolerances; Editing thresholds applied to metric geometry after execution or inverse projection.
Task promptsFor each input: 700 Stockholm QA Open, 700 QA MCQ, and 700 Editing prompts, covering all 28 relations.
Transformation schemasGeoJSON transformation JSON and executor; Image pixel-geometry schema and matched coordinate transforms.
Decoding settingsStockholm settings indexed by model, task and input, with source evidence; paired-probe configurations in Q2.
Evaluator and reproductionScoring modules, regression tests, and complete 700-task replay examples for GeoJSON and Image.

Touches. QA uses the standard GIS topological touching predicate. Editing applies the task’s declared 1 m contact tolerance: exact contact, a positive gap of at most 1 m, or shallow overlap whose intersection becomes empty after a 1 m inward buffer. Equality and containment are excluded. This explicitly defines the tolerated contact condition used by the editing evaluator.

Connectivity. Editing accepts topological touching between lines, an edited endpoint within 1 m of the reference line, or nonintersecting lines separated by at most 1 m. Equal lines are excluded. Endpoint-to-interior contact is permitted; interior crossings alone do not establish success. Touches and Connectivity distance comparisons include a numerical slack of 0.000001 m. The criterion is connectivity within tolerance, rather than exact endpoint-to-endpoint coincidence.

Structured executor. The executor applies the model’s predicted parameters directly. Translation uses the predicted offsets; rotation and scaling are centered on F0’s centroid; attribute editing applies the predicted value; and snap copies the specified reference geometry. Only F0 is changed. The executor performs no constraint-solving search or parameter optimization. Constraint satisfaction is evaluated afterward and is not returned as corrective feedback. The structured retry checks undecodable, invalid, or unchanged outputs; it does not solve an unsatisfied spatial condition. For Image, the model supplies the edited pixel geometry; the evaluator only inverse-projects it into the metric task frame before constraint checking.

Relation selection and identifiers. The 28 relations draw on established GIS and spatial-database concepts, spanning topology, direction, metric, and order over points, lines, and polygons. Each has an explicit geometric or attribute-based definition, enabling GIS-derived QA answers and verifiable editing constraints. Registry identifiers are retained, explaining the nonconsecutive numbering. Q3 contains all 28 definitions and implementations.

Reproducibility. Q3 provides the relation predicates, editing constraints, numerical tolerances, task prompts, transformation schemas, available decoding settings, and evaluator implementation for both GeoJSON and Image inputs. It also documents the touching and endpoint-connectivity checks and the executor’s behavior, with runnable examples for reproducing the evaluation.

Thank you for the detailed and thoughtful rebuttal. I appreciate the time you took to address each of my concerns.

  • Q1: the strict-CSR breakdown and the pixel-to-ground conversion confirm that the proximity allowance does not affect rankings, which resolves my concern about one-sided leniency.
  • Q2: the added recognition probes on the editing scenes and broke down editing failures into invalid outputs, unchanged geometric edits, and valid edits that violate constraints, and they softened the "direct test" claim to "recognition–editing discrepancies"; this fully explained what I asked for.
  • Q3: the authors provided the complete per-relation predicates and editing constraints, numerical tolerances, touching and connectivity definitions, full prompts, transformation schemas, decoding settings, and the executor/evaluator implementation (which applies predicted parameters without constraint-solving search), all of which resolves my auditability concern.

Overall, the authors have satisfactorily addressed the issues I raised. My concerns have been resolved, and I will keep my overall score unchanged as 3 (weak accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.