acceptodds
Under review as a conference paper at ICLR 2027

GeoMapBench: Cognition-Aware Characterization of Geospatial Capabilities in Vision-Language Models

13 views

Abstract

Vision-language models (VLMs) are increasingly applied to maps, remote-sensing imagery, and other geospatial data, yet their geospatial capabilities remain difficult to evaluate systematically because existing benchmarks typically cover only subsets of cartographic understanding, spatial reasoning, geographic knowledge, and structured geospatial problem solving. We develop a comprehensive taxonomy that decomposes geospatial capability into 23 fine-grained tasks organized into four domains: Visual Perception and Interpretation, Spatial Reasoning and Geometric Processing, Geospatial Language and Semantic Reasoning, and Applied Geospatial and Earth-System Reasoning. Guided by this taxonomy, we construct GeoMapBench, a benchmark containing 2,300 examples with 100 instances per task. Each example is additionally assigned a level of the revised Bloom taxonomy, enabling analysis across both geospatial capability and cognitive complexity. We also construct GeoMapRAGCorpus, a benchmark-separated multimodal retrieval corpus containing 180,344 records from seven public geospatial source families. The corpus covers structured geographic data, text, map imagery, and geocoded geographic images. We benchmark five contemporary VLMs on all 2,300 GeoMapBench examples and observe substantial variation across tasks, with no single model consistently dominating the full task space. Finally, to demonstrate the downstream utility of the released resources, we evaluate GeoMapPlus, a task-aware retrieval- and tool-augmented inference pipeline built using GeoMapRAGCorpus. GeoMapPlus improves both task-aware and strict evaluation performance over its underlying VLM, demonstrating that the corpus can support measurable improvements in geospatial reasoning.ب

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Multimodal Geospatial Reasoning, Geospatial Benchmarking, Map Understanding, Spatial Reasoning, Retrieval-Augmented Generation, Multimodal Retrieval, Benchmark Datasets

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oZDwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoMapBench, a benchmark for evaluating geospatial capabilities in vision-language models. It contains 2,300 examples evenly distributed across 23 tasks in four domains: visual perception and interpretation, spatial reasoning and geometric processing, geospatial language and semantic reasoning, and applied geospatial and Earth-system reasoning. The main contributions are a unified capability taxonomy, cognitive annotations based on the revised Bloom taxonomy, strict and task-aware scoring adapted to each task, and GeoMapRAGCorpus, a retrieval corpus containing 180,344 records from seven source families with explicit separation from the benchmark. The authors evaluate five VLMs and identify substantial differences in their performance across individual tasks. They also evaluate GeoMapPlus, a pipeline that combines retrieval and geographic tools with Claude Sonnet 5. It improves the task-aware score from 47.90% to 52.24% and strict accuracy from 37.83% to 45.26%, although the benefits vary considerably across tasks.

Critical strengths

Broad coverage with evaluation appropriate to each task. The benchmark brings together cartographic understanding, remote sensing, geometric computation, geographic language, and applied geographic knowledge within a common framework. Importantly, it preserves heterogeneous outputs, including coordinates, masks, graphs, routes, and polygons, rather than reducing every problem to generic classification. Equal sampling across the 23 tasks and the use of both strict and task-aware scores provide a useful basis for examining specific strengths and failures. This is consequential because the observed changes in model rankings across tasks demonstrate that a single aggregate score would conceal meaningful differences in capability.

An evidence resource with explicit safeguards against benchmark overlap. GeoMapRAGCorpus is designed as a reusable evidence collection rather than a resource tied to one particular retrieval algorithm. Provenance for individual records, licensing information for each source, and explicit exclusion checks make its construction more inspectable. These checks cover source identifiers, normalized text matches, exact observations defined by country, indicator, and year, and geographically nearby records when location could reveal the answer. The authors also appropriately distinguish these safeguards from stronger claims about semantic independence or the absence of pretraining contamination. These design choices strengthen the resource’s potential value for controlled research on geospatial retrieval and tool use.

Critical weaknesses

The distinctive cognitive dimension lacks sufficient validation. Even accepting that the Bloom labels describe cognitive operations rather than an ordered difficulty scale, the paper does not adequately establish that its variants capture meaningfully distinct operations. In Table 4, Remember asks for the relation from polygon A to polygon B, whereas Apply asks the model to determine the relation between the polygons. These appear operationally similar based on the provided descriptions. Likewise, Create requests a graph with two nodes containing the forward and inverse relations already requested by Analyze. This raises the possibility that the distinction primarily concerns output representation rather than a different cognitive operation. Each source example is assigned only one Bloom level, and comparisons across levels mix different tasks and response structures. The authors acknowledge these confounding factors but do not report a matched analysis that resolves them. The automated check establishes structural integrity rather than independently validating annotation quality. Consequently, the additional diagnostic value of the cognitive dimension beyond task identity and output format remains unclear. This directly affects my assessment because the characterization of cognitive operations is a defining contribution of the paper.

The augmentation experiment does not establish the retrieval corpus’s incremental utility. GeoMapPlus simultaneously introduces retrieval, geographic tools, evidence selection, and answer verification. The authors explicitly acknowledge that the comparison does not isolate individual components, but this limitation leaves the empirical validation of GeoMapRAGCorpus incomplete. In particular, coordinate transformation improves by 45.9 percentage points and metric distance computation by 35.8 points. With equal weighting across the 23 tasks, these two tasks alone contribute approximately 3.55 percentage points to the reported overall improvement of 4.34 percentage points, accounting for about 82% of the net gain. This calculation follows from the displayed results and does not establish which component caused the improvements. Since these operations are precisely those targeted by dedicated geographic tools, the results do not yet demonstrate how much benefit the corpus provides beyond exact computation and additional inference stages. Comparisons using retrieval alone and tools alone, with comparable answering and verification budgets, are important for evaluating this central resource contribution. The declines on ten tasks further highlight the need to distinguish useful retrieved evidence from errors introduced by augmentation.

Questions for authors

Cognitive label validity (clarification and supporting evidence): What operational criteria distinguish Remember from Apply, and Analyze from Create, in the polygon relation example? Is there independent human validation of the Bloom assignments, or a matched analysis controlling for geospatial content and response format, that demonstrates what the cognitive labels diagnose beyond differences in task and formatting?

Evaluation protocol (clarification): For masks, graphs, routes, and polygons, what exact output representations, matching rules, numerical tolerances, and acceptance criteria are used for strict and task-aware scoring? Can you report how often failures arise from invalid serialization or parsing rather than incorrect geospatial content? This distinction is important for interpreting the benchmark as a measure of geospatial capability.

Corpus contribution (evidence for the current claim): Can you provide results using retrieval alone and tools alone, with the same final VLM and comparable answering and verification budgets? In particular, does adding corpus retrieval improve over a system that uses only tools, and how much of that improvement remains outside coordinate transformation and metric distance computation?

Routing and information access (clarification): Does the router receive the ground truth task identifier, or does it infer the task solely from the question and image? What information is available to the geographic tools and verification stage, and how is reference information reserved for evaluation excluded? Please also clarify which development data were used to select routing rules and thresholds.

I am open to revising my score if the authors can provide satisfactory clarifications that address the concerns raised above.

Reviewer_DGiUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper proposes GeoMapBench, which is a benchmark that evaluates the geospatial capabilities of vision-language models (VLMs). The benchmark consists of 2300 examples across 23 fine-grained tasks (i.e., 100 questions per task), organized into four domains (Visual Perception and Interpretation, Spatial Reasoning and Geometric Processing, Geospatial Language and Semantic Reasoning, Applied Geospatial and Earth-System Reasoning). Each example is assigned a level of the Bloom taxonomy, which enables analysis for both geospatial capability and cognitive complexity. In addition, the paper introduces GeoMapRAGCorpus, which is a separate retrieval dataset curated from seven data sources (GeoNames, Wikidata, OSM, World Bank, EPSG/PROJ, Wikipedia, Wikimedia Commons). Evaluation on current VLMs reveals substantial variation in performance across different evaluated geospatial capabilities, and GeoMapPlus, which is a task-aware retrieval (using the GeoMapRAGCorpus) and tool-augmented inference pipeline, shows improved performance, illustrating the potential of incorporating external geographic evidence and computation tools.

Critical strengths
  1. The paper covers questions from 23 different tasks ranging from relatively simple ones such as Cartographic Symbol Recognition to more complex ones such as Geographic Fact Reasoning. In addition to the diversity in the covered subtasks, the benchmark consists of heterogeneous input and output types, further showing the broad coverage and breadth covered by the benchmark.

  2. The paper introduces GeoMapPlus, which utilizes the retrieval corpus, GeoMapRAGCorpus, to evaluate whether using retrieved geographic information and external tools can improve the performance of VLMs. Testing and evaluating on such tasks is valuable, as some geospatial tasks require domain-specific knowledge or precise numerical computation that might not be reliably addressed by generalized VLMs. The performed evaluation helps us understand which types of geospatial reasoning questions may benefit from these retrieval and tool-type external resources.

Critical weaknesses
  1. The GeoMapPlus framework combines various components, including retrieval from the GeoMapRAGCorpus, but also uses external tools. Though the result shows improvement in performance, it is unclear how each component contributes to the improvement; therefore, it is unclear how beneficial this corpus is. An ablation study comparing retrieval only, tool only, and GeoMapPlus (which does the routing) will help clarify the contribution of each component.

  2. While the incorporation of the Bloom taxonomy is highlighted as one of the main contributions of the paper, it is unclear what complementary analysis these levels provide beyond the 23 geospatial task taxonomy. While the paper does state that Bloom annotations are to be interpreted diagnostically rather than as a calibrated scale of difficulty, the paper doesn't establish the value of the Bloom-based evaluation beyond the task-level analysis. A controlled within-task (i.e., share the same task but only differ in Bloom level) comparison will help strengthen the paper's claim regarding the need for this additional taxonomy level.

Questions for authors
  1. What does the Bloom level introduce to the benchmark? How do the Bloom categories relate to the reasoning capabilities in VLMs?Aren't there cases where two different Bloom levels potentially share the same spatial reasoning requirement?

  2. Please clarify what additional insights the Bloom taxonomy provides beyond the 23 task capability taxonomy. Are there any evaluation results where the task and spatial content stay static but only the Bloom level is modified?

Reviewer_MjbSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GeoMapBench, a 2,300-example benchmark covering 23 geospatial tasks across four domains, with revised Bloom taxonomy annotations and task-specific scoring. It also constructs GeoMapRAGCorpus, a benchmark-separated collection of 180,344 evidence records, and evaluates five VLMs alongside GeoMapPlus, a retrieval- and tool-augmented pipeline. Results reveal substantial task-level variation and show that augmentation improves aggregate task-aware and strict scores by 4.34 and 7.43 percentage points, respectively, while degrading performance on some tasks.

Critical strengths

GeoMapBench brings cartographic perception, remote sensing, geometric computation, geographic language, and applied knowledge into a balanced evaluation with structured outputs rather than only multiple-choice answers. This breadth makes the resource useful for identifying capability gaps that narrower benchmarks or aggregate scores can obscure.

Critical weaknesses
  1. The interpretation is insufficiently validated: Table 4’s Remember and Apply variants both request a polygon relation, while Create largely reformats relations into a two-node graph. Although the authors appropriately disclaim a universal difficulty ordering, the absence of reported expert validation or controlled comparisons leaves unclear whether the Bloom axis captures distinct cognitive operations beyond prompt and output-format differences.
  2. Benchmark validity is difficult to assess because Table 5 names broad metrics without specifying numerical tolerances, strict-correctness thresholds, structured-output parsing, or how text-generating VLMs produce dense masks and polygons. Appendix D explicitly limits its audit to structural integrity, leaving no reported independent assessment of adapted-question correctness or evaluator agreement; these omissions make it difficult to distinguish geospatial errors from annotation or formatting failures.
Questions for authors
  1. What annotation guidelines distinguish Remember from Apply and Analyze from Create in examples such as Table 4? Was expert agreement or another independent validation of these assignments measured?
  2. Please provide representative complete prompts, reference answers, and scoring rules. How are outputs serialized and parsed, and what tolerances determine strict correctness?
  3. Are existing within-task analyses available that account for answer format when comparing Bloom levels? Since each source example receives only one level, what evidence supports interpreting these profiles as cognitive characterization rather than differences in sampled content and response structure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.