acceptodds
Withdrawn from ICLR 2027

CARTA: A Traceable Benchmark for Cross-Representation Map Understanding in Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The same place can be shown as satellite imagery, a street map, a land-cover map, a contour sheet, or a weather chart. Reading across these visual languages is a basic requirement for models that navigate, monitor, or respond in the physical world, yet existing benchmarks rarely change the representation while holding the geographic referent fixed. We introduce CARTA, a benchmark that asks what breaks when the place stays the same but its representation changes. CARTA derives images, questions, answers, and distractors from shared geospatial state, so each of its 16,408 items is traceable from source data through rendering, audit, and frozen image bytes. The items form matched comparison families for representation profiling, continuous-field reading, and cross-representation correspondence, together with breadth and grounding tracks. Evaluating twelve MLLMs against a matched human reference reveals failures at three successive stages. Evidence: human accuracy on forest and cropland/bare targets falls to 15–45% on street maps, and an objective purity screen shrinks the human satellite–minimalist gap from 20.0 to 8.6 pp. Correspondence: models match categories more readily than places, as cross-class distractors inflate correspondence accuracy by up to 22 pp (Qwen3-VL-8B: 34.7% to 56.9%), and every model loses 3.5–14.4 pp when the two views are not aligned. Geometry: on point grounding, open-weight models do no better than copying the source mark's coordinates (17.6% versus 17.5%), and even GPT-5.5 reaches only 28.9%, against 88.3% for readers on a matched subset. On matched multiple-choice items GPT-5.5 scores 74.4% against the readers' 83.3%. Reasoning-enabled serving lifts two open-weight models from about 61% to 71% on MCQs but provides no reliable benefit for geometric grounding. CARTA thus provides a matched test bed for distinguishing representation-specific success from transferable geographic understanding. Data and code will be released after acceptance.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Map Understanding, Spatial Reasoning, Visual Reasoning, Multimodal Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3iTgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Existing benchmarks do not adequately test models when the visual representation of a fixed geographic location changes. To address this, the paper introduces CARTA, a benchmark containing over 16k generated question-answer pairs designed to evaluate how MLLMs handle identical locations rendered across completely different map styles. The authors sourced raw data from multiple platforms: vector geometry and street networks from OpenStreetMap, satellite imagery from Esri World Imagery, and land cover, terrain, and weather fields from ESA WorldCover, SRTM, and NOAA GFS. They utilized a two-LLM "generate-audit-seal" pipeline, where a DATA agent synthesizes the items and an EVAL agent strictly audits them. The final benchmark consists of over 20k verified images, validated by human annotators who reviewed a subset of 358 items.

The benchmark is structured around three core tasks: C1 tests how map style affects the visibility of land-cover evidence, C2 evaluates the ability to read continuous weather and terrain data regardless of visual encoding, and C3 assesses spatial matching across different map types. Evaluating ten open-source and two proprietary MLLMs, the authors found that GPT-5.5, Gemini 3 Pro, and Qwen 35B perform the best, though they still lag behind the human baseline. Ultimately, the study identifies three distinct failure modes. First, models fail when a specific map representation omits necessary visual evidence. Second, in the C3 task, models rely heavily on semantic shortcuts (e.g., matching visual categories instead of spatial locations). Finally, models fail at precise geometric grounding, often resorting to naively copying source coordinates instead of executing true spatial transfer.

Critical strengths
  1. The paper has an interesting approach to assessing MLLM's capabilities on map data. Instead of using a static benchmark, they test the model's ability to understand the geographical information across different visual representations.
  2. The benchmark is useful with data pulled from many sources.
  3. The failure analysis effectively moves beyond aggregate metrics to expose specific model shortcuts. For instance, the authors clearly demonstrate how models rely on matching semantic categories rather than spatial locations, and how they resort to naively copying coordinates instead of executing true geometric grounding.
Critical weaknesses
  1. The paper's structural organization and writing are very difficult to follow. Critical results and evaluation numbers are densely packed into the prose rather than being organized into tables. Additionally, the writing occasionally lacks natural flow, suggesting a heavy reliance on AI writing tools. There should be better organization in tables.
  2. In the C2 continuous-field task, the visual encoding is entangled with the underlying physical field being measured. Because these variables are changed simultaneously, the results are hard to interpret. The authors must ablate these changes separately to prove whether a model is struggling with the visual style or the actual map content.
  3. There are insufficient technical details regarding the generation pipeline. While the benchmark synthesizes items from a shared geospatial state, the exact programmatic rules, rendering libraries, and LLM agent prompts used to create the datasets are missing.
Questions for authors
  1. How exactly were the different map representations generated for the specific tasks? Please provide the detailed programmatic parameters, rendering tools, or EVAL/DATA agent prompts used to reliably synthesize the exact same coordinate across distinct idioms.
Reviewer_XqZ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the CARTA benchmark to assess location understanding when the representation (map view) changes across satellite images, vector street maps, and land-cover color maps. CARTA consists of 16,408 question answering pairs containing multiple choices and point/bounding box answers. The paper evaluates 12 multimodal large language models (MLLMs) and shows that performance varies across representations, and point grounding is not easily achievable with MLLMs.

Critical strengths

S1. CARTA includes map question-answering tasks for reading terrain and weather fields and matching the same geographic location across different map views. These tasks provide a useful evaluation of map understanding and the connection between distinct representations of the same place.

Critical weaknesses

W1. The paper lacks a concrete explanation of how matching map views are obtained and at which spatial scales they are constructed. Section 3.1 describes a "four-idiom point group" and a "four-cell field–encoding group," but these descriptions are very abstract. More explanation is needed of how the geographic extent, scale, and alignment are determined for each view. This makes it difficult to understand how the matched comparisons are constructed.

Although the C2 task family provides motivating examples for map understanding, the notation and purpose of the "conventional interaction" in Equation 1 are difficult to follow. The meanings of the T/P and I/C indices are not explicitly introduced in the main text. Although S_swap and S_std are defined mathematically, their meanings need a clearer explanation. This makes it difficult to understand what the comparison measures and how its results should be interpreted.

W3. The construction audit relies on a single reviewer to identify defects in the 300 candidate sample, without an independent agreement check. This procedure lacks multiple human reviewers and agreement scores. The reliability of these manual judgments needs stronger validation.

Questions for authors

Q1. How are the matching map views obtained? What geographic extent, spatial scale, and image resolution are used, and how is alignment checked across the views?

Reviewer_RjmxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CARTA, a benchmark of 16,408 items across 89 regions that tests whether multimodal models can maintain geographic understanding across satellite imagery and different map representations. It distinguishes failures in reading visual evidence, matching geographic objects, and transferring geometry across views. Evaluations reveal that missing visual evidence limits both humans and models, while cross-category distractors can inflate apparent matching ability. All models perform worse on non-aligned views, with particularly weak point and box localization, exposing a substantial gap between answering map questions and grounding geographic objects across representations.

Critical strengths
  1. The main idea is to split cross-representation ability into three stages: whether the evidence is visible, whether the same object is matched, and whether the geometry transfers precisely. Tests use same-place paired renders, aligned and non-aligned twins, and anonymous grounding.
  2. Place is fixed and only the visual representation changes, which localizes the cause better than comparing accuracies across datasets.
  3. Several controls support the main claims, including a 2x2 of translation versus scale or context, a resolution sweep, name restoration, and reasoning mode.
Critical weaknesses
  1. Representation is not a pure causal variable. Map types also change texture, symbols, labels, and what is actually visible, so one representation cannot be called generally better.
  2. The human reference is six Chinese-speaking participants and 400 items. It shows answerability more than a stable human ceiling, and it does not measure cross-language or cross-cultural map literacy.
Questions for authors
  1. Is it possible to use factorial renders that vary texture, labels, symbols, evidence visibility, and color separately?
  2. Should enlarge the human study across languages. Report agreement, reaction time, and learning effects.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.