- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This is a benchmark paper evaluating multimodal LLMs (close+open) on their ability to reason ofer GIS data and imagery. Further, they evaluate its ability to modify GIS data to satisfy a condition and show that even when the models can reconize geospatial conditions,they cannot reliably make edits that satisdy them.
Critical strengths
For a paper like this -- I think that the contributionhinges on whether the benchmark exposes an existing flaw, and the value is increased when it shows a flaw that you may not have expected. This is exposing a flaw that I have also experienced in other domains. First, a general difficulty in metric reasoning, and second a frustrating inability to suggest edits that satisfy them. It is good to see this in a benchmark. Overall there was not much in a 9 page paper that I could find to complain about. Some minor comments below. It does not analyze the causes or suggest solutions but it does find an issue and provide a benchmark to attack when trying to solve it, which I suppose is the goal of this type of paper.
Critical weaknesses
The most critcial concerns are:
-
This is a benchmark paper. It identifies a weakness and provides a dataset and metrics to evaluate performance on it. This in principle could be used to improve models. However it does not speculate on the cause of the issues—why metric performance struggles, why editing is often so much worse than recognizing. It does not propose a solution. I am unsure what the bar should be in this paper; however, it would certainly be stronger if these ingredients were present.
-
It is based on OpenStreetMap data and is trying to measure quantities that I think OSM is not reliable for. For example, in my experience, many roads are "drawn through" in OpenStreetMap. Many annotations are at different levels of completeness. It is unclear to me how much of an impact that itself has on the benchmark. Since they are not measuring whether the maps match, e.g., satellite imagery (this is a major problem with OSM) perhaps the error rate is acceptable, but I think some sort of visual inspection sites to estimate the prevalence of topological issues in the "truth" data may be necessary.
** Post Rebuttal** I feel like the authors addressed my concerns and will raise my score to clear-accept.
- The project the geojsons to a metric CRS—okay [human em dash!]. What? UTM zones? And then project the imager to WGS84. Why not the same metric system? This seems like a flaw to me. Bearings and vertical distances will be different. And the magnitudes of the numbers could be way off.... much simpler to reproject both, right? How can you compare distances and bearings when each representation is in a different CRS? I mean, it's possible but seems both error-prone on the human side and unnecessarily challenging to the LLM side. Also the validity of comparing edits to a GeoJSON vs. an image .... again, how can it be expected to separate linestrings? It has to—what? Invent the GeoJSON itself? This was not very clear to me and seems like a different (more challenging) task. I think that if you are giving it a map image, you would need to somehow encode features in it very carefully.
PS: If I were using an LLM -- I would have asked it those questions and possibly accepted its answer rather than put them here.
Questions for authors
You specify that for the 70 edits, the presence of one solution is used to validate. Were these manually done? How are the images used produced? Are the from OSM tile server? Did you render them yourself? How are attributes represented in the images?
What prompts were used? Do you have examples? Is Fig 6 a duplicate of Table 4?
5 comments
Response to Reviewer kjDb, Critical Weakness 1 (Part 2), Possible Solutions
Response (Continued)
Possible Solutions
These explanations suggest two complementary directions.
First, spatial representations could preserve geometric and relational structure more explicitly. Section 4.4 already identifies SPOK as one example. SPOK encodes relative location and connectivity through spatial tokens (F. Huang et al., 2025). Extending such representations to metric computation and constrained editing is a concrete direction to explore. The aim would be to keep task-relevant spatial structure available throughout recognition and generation.
Second, editing could combine model predictions with explicit geometric constraint solving. FirePlace translates language instructions into constraints on object surfaces and uses a solver to produce feasible placements. Its ablations show that replacing constraint solving with model selection among random placements reduces placement quality (I. Huang et al., 2025).
For GeoSpatialBench, this suggests a pipeline that expresses the target object and requested relations as explicit constraints, uses geometric operations to compute measurements and candidate edits, and checks the resulting geometry against those constraints. The existing executor and constraint checker provide a basis for evaluating this approach. Explicit representations would support the use of spatial information, while geometric solvers would support the construction of valid edits.
Manuscript Revision
We add the cited studies and extend the existing “Toward explicit spatial representations” discussion (section 4.4, lines 473–485). This revision clarified the role of SPOK and introduce constraint solving, supported by FirePlace, as a complementary direction.
References
Huang, F., Lv, J.-R., Li, G.-L., & Yue, Y. (2025). SPOK: Tokenizing geographic space for enhanced spatial reasoning in GeoAI. International Journal of Geographical Information Science, 39(12), 2768–2808. https://doi.org/10.1080/13658816.2025.2497810
Huang, I., Bao, Y., Truong, K., Zhou, H., Schmid, C., Guibas, L., & Fathi, A. (2025). FirePlace: Geometric refinements of LLM common sense reasoning for 3D object placement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13466–13476). https://openaccess.thecvf.com/content/CVPR2025/html/Huang_FirePlace_Geometric_Refinements_of_LLM_Common_Sense_Reasoning_for_3D_CVPR_2025_paper.html
Response to Reviewer kjDb, Critical Weakness 2, OSM Data Quality and Its Impact on Evaluation
Response
Thank you for highlighting the importance of source data quality and topology checks. We chose OSM for its broad coverage across cities. A shared source allows us to use a common processing pipeline and helps limit differences introduced by combining separate datasets. Within this pipeline, reference answers are computed through GIS operations on the geometry and attributes supplied in each task. For example, POI Density is calculated only from the provided POIs, so its answer does not depend on a complete inventory of real-world facilities.
We use reusable relation-specific templates to construct tasks consistently across cities. Each template assigns roles to the participating objects and defines how the reference answer is computed. City-specific settings, such as field mappings and filtering thresholds, are configured separately. As described in Section 3.2, candidate features are filtered by geometry type and length or area thresholds. Scenes are then exported as GeoJSON with anonymous feature IDs, while records of their origin and construction are retained for auditing. When suitable candidates are insufficient, the pipeline selects other scenes or uses documented controlled constructions.
Each resulting instance undergoes further validation. For QA, we check the geometry and recompute the reference answer. We also verify that the answer options are distinct. For editing, we reject candidates whose initial scenes already satisfy the target constraints. A reference edit must then demonstrate that a valid solution exists.
We complement these automated checks with manual visual inspection of drawing and topological issues. Following the revisions to map rendering, we conducted an additional inspection of the same sampled instances.
Previous inspection: We inspected 300 instances across six cities, comprising 150 QA scenes and 150 Editing tasks. Each city contributed 50 instances, and the sample covered all 28 relations. The inspection identified 17 issues. Eleven concerned drawing and three concerned display; these were judged not to affect the reference answer or edit feasibility and were recorded for revision. The remaining three were Image QA instances that lacked the required layer information.
Additional inspection: Following these findings, we revised the rendering process and included the required layer attributes in the map legends. We then inspected the same 300 instances using the updated metric OSM maps. This inspection identified nine drawing issues, all judged not to affect the tasks. It also identified five input issues: three GeoJSON QA instances lacked the required layer attributes, while two editing prompts contained a specific layer value in an example. We revised the affected prompts to address these issues and verified the changes in the final review.
Neither inspection identified inconsistencies in the task-defined topological relations. We also manually reviewed the 150 reference edits in the sample and confirmed their validity. Appendix C reports the findings from both inspections and the revision status of the identified issues.
Manuscript Revision
We expand Section 3.2 (lines 191–230) to explain this quality-control process, from source selection and template-based construction to automatic checks and manual inspection. The appendix (Appendix C, lines 866–883) additionally reports the inspection criteria, sample coverage, findings, and any corrections.
Response to Reviewer kjDb, Critical Weakness 1 (Part 1), Causes of Weak Metric Performance and the Recognition–Editing Gap
Response
Thank you for encouraging us to explain these difficulties and discuss possible solutions. We regard them as two related challenges: computing precise geometric quantities and using spatial information to generate edits that satisfy the target constraints. Recent studies provide support for this interpretation and suggest concrete directions for improvement.
Metric Measurement and Geometric Computation
We suggest that metric tasks are harder than other spatial tasks because they require more precise geometric information and additional numerical computation. For POI Density in GeoSpatialBench, the model must identify the points covered by a region, count them, compute the region’s area, and calculate the density. These steps add numerical operations beyond relation recognition.
Recent research shows that arithmetic in the studied LLM models relies on heuristics tied to numerical patterns. Some errors occur when these heuristics provide insufficient support for the correct answer (Nikankin et al., 2025). The additional operations can make metric tasks more sensitive to unreliable numerical computation. This also helps explain why precise measurement can remain difficult even when GeoJSON provides explicit coordinates.
For image inputs, models must also recover the geometric detail needed for measurement. Geometric errors may leave a relation category unchanged while pushing a numerical answer beyond the allowed tolerance. In 3D measurement experiments, SD-VLM performs better with depth positional encoding than with depth supplied as additional images or tokens (Chen et al., 2025). This shows that how spatial information is represented matters for quantitative measurement. Together, these findings motivate our interpretation: compared with other spatial relations, metric tasks place greater demands on both the precision of spatial representations and the reliability of numerical computation.
From Relation Recognition to Spatial Editing
Editing requires spatial information to guide the generation of operation parameters or new geometry. A possible source of the gap is that information available for recognition does not consistently influence this generation process. Recent research using hidden-state interventions shows that encoded spatial information affects model outputs differently across layers, token positions, and prompt formats. Visually grounded prompts and text-based reasoning prompts can change how this information reaches the answer (Wang & Xu, 2026). This motivates our interpretation that spatial editing depends on how effectively spatial representations guide the output, beyond whether the relevant information has been encoded.
Recent action-selection research also provides complementary evidence. Reading Right, Acting Wrong finds that models can read facts and apply rules separately, yet fail when they must combine these steps. In controlled tasks, composing the models’ own factual answers through code improves decisions (Li et al., 2026). This points to another possible difficulty in spatial editing: combining recognized relations with target conditions to select an appropriate geometric change.
Manuscript Revision
We expand section 4.4, currently on pp. 8–9 (lines 436–443 and 449–462), to connect metric measurement and computation with geometric representation and numerical reliability. We also explain the recognition–editing gap in terms of how spatial information guides generation and how models combine target constraints.
References
Chen, P., Lou, Y., Cao, S., Guo, J., Fan, L., Wu, Y., Yang, L. F., Ma, L., & Ye, J. (2025). SD-VLM: Spatial measuring and understanding with depth-encoded vision-language models. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, & N. Chen (Eds.), Advances in Neural Information Processing Systems (Vol. 38, pp. 33938–33968). Curran Associates. https://doi.org/10.52202/085713-1140
Li, Y., Krishnan, R., & Padman, R. (2026). Reading right, acting wrong: Diagnosing decision failures of System One models in agent loops [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2610.18579
Nikankin, Y., Reusch, A., Mueller, A., & Belinkov, Y. (2025). Arithmetic without algorithms: Language models solve math with a bag of heuristics. In Y. Yue, A. Garg, N. Peng, F. Sha, & R. Yu (Eds.), International Conference on Learning Representations (pp. 55939–55965). https://proceedings.iclr.cc/paper_files/paper/2025/hash/8c5f30296296d2ae402ebbd09aaa9c12-Abstract-Conference.html
Wang, Z., & Xu, X. (2026). Knowing isn’t always saying: When do spatial encodings reach answers in vision-language models? In H.-Y. Lee, S. Dev, J. J. Li, & D. Elliott (Eds.), Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 52683–52705). Association for Computational Linguistics. https://doi.org/10.18653/2026.emnlp-main.2319