- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Existing benchmarks do not adequately test models when the visual representation of a fixed geographic location changes. To address this, the paper introduces CARTA, a benchmark containing over 16k generated question-answer pairs designed to evaluate how MLLMs handle identical locations rendered across completely different map styles. The authors sourced raw data from multiple platforms: vector geometry and street networks from OpenStreetMap, satellite imagery from Esri World Imagery, and land cover, terrain, and weather fields from ESA WorldCover, SRTM, and NOAA GFS. They utilized a two-LLM "generate-audit-seal" pipeline, where a DATA agent synthesizes the items and an EVAL agent strictly audits them. The final benchmark consists of over 20k verified images, validated by human annotators who reviewed a subset of 358 items.
The benchmark is structured around three core tasks: C1 tests how map style affects the visibility of land-cover evidence, C2 evaluates the ability to read continuous weather and terrain data regardless of visual encoding, and C3 assesses spatial matching across different map types. Evaluating ten open-source and two proprietary MLLMs, the authors found that GPT-5.5, Gemini 3 Pro, and Qwen 35B perform the best, though they still lag behind the human baseline. Ultimately, the study identifies three distinct failure modes. First, models fail when a specific map representation omits necessary visual evidence. Second, in the C3 task, models rely heavily on semantic shortcuts (e.g., matching visual categories instead of spatial locations). Finally, models fail at precise geometric grounding, often resorting to naively copying source coordinates instead of executing true spatial transfer.
Critical strengths
- The paper has an interesting approach to assessing MLLM's capabilities on map data. Instead of using a static benchmark, they test the model's ability to understand the geographical information across different visual representations.
- The benchmark is useful with data pulled from many sources.
- The failure analysis effectively moves beyond aggregate metrics to expose specific model shortcuts. For instance, the authors clearly demonstrate how models rely on matching semantic categories rather than spatial locations, and how they resort to naively copying coordinates instead of executing true geometric grounding.
Critical weaknesses
- The paper's structural organization and writing are very difficult to follow. Critical results and evaluation numbers are densely packed into the prose rather than being organized into tables. Additionally, the writing occasionally lacks natural flow, suggesting a heavy reliance on AI writing tools. There should be better organization in tables.
- In the C2 continuous-field task, the visual encoding is entangled with the underlying physical field being measured. Because these variables are changed simultaneously, the results are hard to interpret. The authors must ablate these changes separately to prove whether a model is struggling with the visual style or the actual map content.
- There are insufficient technical details regarding the generation pipeline. While the benchmark synthesizes items from a shared geospatial state, the exact programmatic rules, rendering libraries, and LLM agent prompts used to create the datasets are missing.
Questions for authors
- How exactly were the different map representations generated for the specific tasks? Please provide the detailed programmatic parameters, rendering tools, or EVAL/DATA agent prompts used to reliably synthesize the exact same coordinate across distinct idioms.