acceptodds
Under review as a conference paper at ICLR 2027

MapSpatial: A Benchmark for Representation-Robust Spatial Reasoning over Real Maps

Abstract

Reliable map use requires stable answers across views and orientations, and revised answers when roads or buildings change; single-image accuracy cannot measure both. We introduce MapSpatial, a geographic information system (GIS)-grounded benchmark of 77,800 instances spanning 14 sub-tasks in four families: qualitative direction, metric comparison, route-relative building identification and counting, and road-network path reasoning. Because every scene is stored as GIS geometry, each answer can be recomputed exactly when the same question is posed across three registered views (satellite imagery, a text-free structural map, and a conventional text-bearing map), under rotations and mirror flips, and, where applicable, after solver-verified structural edits paired with answer-preserving shams. We evaluate 25 multimodal models. Among six representative models, direct accuracy is 35.2–48.3%, but 33.8–60.7% of registered-view triplets yield different semantic predictions across views that share an answer but differ in visual cues. After a single declared geometric action, 21.1–34.3% of base-correct answers become wrong; the stricter base-plus-seven joint score is 11.2–22.8%. After an answer-changing edit, 9.6–26.2% of base-correct pairs update correctly, while 50.4–84.3% retain the pre-edit answer. Gemini-3-Flash correctly updates 50.4% of its base-correct edit pairs. Controlled visual aids show task-dependent effects. MapSpatial quantifies cross-view disagreement, transform sensitivity, and limited correct updating through paired, GIS-verifiable views and edits.

Primary area:
datasets and benchmarks
Keywords:
spatial reasoning, map understanding, multimodal large language models, geospatial reasoning, representation robustness, multimodal benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rxYxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces MapSpatial, which consists of 14 subtasks in four task families, including qualitative direction, metric comparison, route-relative building identification and counting, and road-network path reasoning. MapSpatial is constructed on three distinct views, including satellite imagery and maps with/without text, and with rotations or mirror flips. The evaluation shows that 25 multimodal large language models perform inconsistently across visual views for the same question-answer pairs and lack robustness across transformations.

Critical strengths

S1. The paper proposes an evaluation design that varies map views (satellite, map with/without text), applies geometric transformations (rotations), and edits geographic entities on the map. This design can show performance inconsistencies in multimodal LLMs' map understanding that evaluating a single map may miss.

Critical weaknesses

W1. The paper lacks clear introductions to terminology and the 14 subtasks, making it very difficult to follow. The intended meaning often has to be inferred from the figures. For example, T3/T4 appear before their task definitions. Explanations of the subtasks and 'oracle-conditioned evidence' are spread across sections. The method section also starts with abstract definitions without providing a complete example of how the source data is used to construct questions, images, and answers. These difficulties make it hard to understand in detail how the benchmark is constructed and to assess whether the experiments support the conclusions.

W2. The evaluation is restricted to single-choice and multiple-choice questions. This tests whether models can select correct answers from provided candidates, but leaves out capabilities related to route-relative building identification and road-network path reasoning. Thus, the results offer limited insight into practical map understanding capabilities, and the benchmark's practical applications remain unclear.

Questions for authors

Q1. Which practical applications does this benchmark support? How do results on single-choice and multi-selection questions demonstrate models’ ability to perform route-relative building identification and road-network path reasoning?

Q2. How are the geographic extent, image resolution, and map scale matched across the three map views? What checks ensure that task-relevant entities are aligned and visible in each view?

Q3. How are geographic scenes and entities, such as roads, buildings, and points of interest, selected? What sampling methods are used?

7 comments
AuthorsOpenReview ↗

Overview: Contribution and Scope

We thank the reviewer for noting that our design "can show performance inconsistencies in multimodal LLMs' map understanding that evaluating a single map may miss" (S1). This is the central contribution, and the paper quantifies it. Each base question is linked to registered satellite, structural, and conventional views, to seven D4D_4 geometric actions with recomputed answers, and, for eligible T3/T4 cases, to a solver-verified answer-changing edit paired with an answer-preserving sham. Across the six anchor models, direct accuracy is 35.2–48.3%, yet 33.8–60.7% of registered-view triplets yield different answers, joint correctness on the base and all seven actions is only 11.2–22.8%, and only 9.6–26.2% of base-correct edit pairs are updated correctly (ScfS_{\mathrm{cf}} 2.1–8.8%). Existing single-rendering map benchmarks do not measure these quantities. Below we address the specific concerns.

AuthorsOpenReview ↗

W1: Task Definitions, Evidence, and Construction Example

Thank you for pointing out these readability issues. We have revised the paper as follows.

  • T1–T4 defined at first use. The Introduction now defines the four task families when they are first mentioned (p. 2): qualitative direction (T1), metric comparison (T2), route-relative building identification and counting (T3), and road-network path reasoning (T4).

  • Operational definitions of the 14 sub-tasks. Section 3.1 (pp. 4–5) now states in one sentence what each sub-task asks the model to determine, so the task meanings no longer have to be inferred from Figure 1. The full rules remain in Appendix A.4, and the Task Matrix in Appendix E.3 summarizes each family's question, evidence, and answer forms. For the other points raised, the relevant material is located as follows.

  • Evidence terminology. Question-conditioned and oracle-conditioned evidence are defined where they first appear in Section 3.1 (p. 5). Oracle-conditioned evidence additionally uses the answer to select the correct counted set or route-relevant state for T3/T4, and serves as an answer-relevant readout reference rather than a deployable baseline or upper bound.

  • Construction pipeline and a worked example. Section 4 and Appendix B describe the construction pipeline from GIS records to registered views, geometry-generated questions and distractors, and deterministic label checks. A worked evaluation example from the released benchmark is given in Appendix K.1. Its four options consist of a base route and a natural alternative with the same endpoints, both taken from the GIS road network, plus one checkpoint reordering of each route as a distractor; option order is then randomized. All images share one crop and coordinate frame. In this case, the base route and the alternative appear as options D and C, so the solver-verified answers are D before the edit, D under the sham, and C after the closure.

AuthorsOpenReview ↗

W2/Q1: Option-based Evaluation and Practical Applications

Thank you for highlighting the distinction between controlled option-based evaluation and practical map use. MapSpatial supports model development and pre-deployment diagnostics rather than serving as a navigation system. Concretely, it supports (i) reliability and regression audits of map/GIS assistants across satellite, structural, and conventional renderings and the benchmark's discrete rotations and reflections; (ii) natural-language spatial question answering over satellite imagery and maps, including direction, distance comparison, and route-relative building identification and counting; and (iii) verification of supplied candidate routes after localized road closures, including checking road connectivity, remaining traversability, and the shortest feasible candidate.

T3 and T4 target the two capabilities named in the review, in controlled discriminative form. For T3, the correct count requires identifying eligible first-row buildings relative to a directed segment or multi-segment route, applying the queried side, updated after each turn for multi-segment routes, and avoiding duplicate counts. T4 requires determining candidate-route validity, waypoint order, the set of routes still traversable after a closure, and the shortest traversable alternative; the options do not supply the road connectivity needed for these determinations. Thus, the tasks evaluate the named map-dependent operations while standardizing the answer space.

The matched text-only control provides an additional check: Qwen2.5-VL-7B-Instruct scores 35.6% without images versus 45.7% with satellite images on the same 1,400 base questions; for T3 and T4, the comparisons are 25.5% versus 38.0% and 31.3% versus 40.5%. This does not reveal the model's internal reasoning process or eliminate guessing, but shows that the maps provide useful information beyond the questions and options. The option-based format is a deliberate design choice: it isolates map reading from generation fluency and makes every reference answer solver-verifiable, which is what a reliability audit requires. The benchmark is also not purely selection-based: in the RQ4 generated-evidence conditions, unified multimodal models must produce new visual artifacts, which we audit for failure modes such as omitted evidence and hallucinated text (Appendix L). We agree that open-ended building enumeration, route generation, and multi-step re-planning remain outside the current scope, as the revised Limitations state; because all questions, labels, and edits are computed from the underlying GIS geometry and solver, these open-ended variants are natural extensions of the released pipeline rather than a redesign.

Reviewer_ZE6bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MapSpatial benchmark, which tests spatial reasoning with satellite images, structural maps, and conventional maps of the same scenes. It contains 1,400 base questions from 585 scenes in China. Rotations, reflections, map edits, and other evaluation conditions expand the release to 77,800 instances across 14 sub-tasks.

The paper evaluates many models on the proposed benchmark. Six representative open-source models score about 35-48% in direct accuracy. Their answers differ across the three views on 34-61% of question groups. When a map edit changes the correct answer, they update correctly on only very few of cases where the original answer was correct. Proprietary model performs much better.

Critical strengths

Each real map edit is paired with a similar edit that keeps the same answer. A model that understands the change should update only when the answer changes.

The same question is shown on three map views, then again after rotation and flipping. This checks whether the answer stays consistent across these versions.

Critical weaknesses

The evaluation has limited coverage. All scenes are from China, and human accuracy is not reported. Only two proprietary models are tested: Gemini-3-Flash and Qwen3.5-Plus.

Questions for authors

For T4 edits, how many questions can be answered by spotting a visible roadblock, and how many require checking which roads are still connected?

4 comments
AuthorsOpenReview ↗

T4 Edits: Roadblock Elimination and Connectivity Reasoning

Thank you for this question. To answer it directly: under the spotting-only elimination rule reported in Table 8, which is deliberately favorable because it also uses the checkpoint pair adjacent to the closed link—more information than merely spotting the red cross—only 2 of the 200 T4 edited questions yield a unique option. The remaining 198 are unresolved by this rule; under the intended solution, distinguishing the surviving options requires additional map-based reasoning, such as checking which candidate sequence follows connected roads. This analysis does not establish that every possible roadblock-spotting strategy fails.

Setup. Each edited T4 question offers four checkpoint sequences: the base route, a natural alternative route with the same start and destination, and one reordering of each route. Routes are not drawn: the waypoint image shows only colored checkpoints. The edit places a red cross on a road link used only by the base route, so the base route is blocked and the alternative remains open; after the closure, the open alternative is the unique correct option among the four in both detour sub-tasks. A matched sham places a corresponding red cross at least 30 m from both routes without changing the answer, so reacting merely to the presence of a red cross is penalized.

Spotting-only elimination rule. Given the checkpoint pair on either side of the closed link, we append the destination to each option and discard any option that lists these two checkpoints consecutively, in either order.

Sub-taskEdits3 options remain2 options remain1 option remains
Detour shortest10052471
Detour traversability10052471
Total200104942

Scope. For the surviving options, the intended solution uses the same map and route information required on the unedited map; the edit adds a localized step of locating the closure relative to the candidate routes. It does not require multi-step re-planning, because the alternative route is guaranteed to remain open. The revision reports this rule and table in the appendix (Table 8, p. 20), states this scope where T4 edits are introduced, and notes in Limitations that the current edits do not require multi-step re-planning.

AuthorsOpenReview ↗

Geographic Scope and Human Answerability

Thank you for raising these two points. We have clarified the geographic scope of MapSpatial and conducted an additional human-answerability study.

MapSpatial currently contains 585 scenes from China, and the revised paper states this geographic scope explicitly. The task definitions and answer computation are not region-specific: reference answers are computed from road, building, and route geometry, and answering the questions does not require reading Chinese labels. Thus, although the current evaluation sample is limited to China, the tasks themselves are not tied to Chinese-language annotations. This distinction does not by itself establish cross-country generalization, which we do not claim.

The original submission included a post-screening answerability check by the two original annotators, but not a separate human study. We therefore asked an independent third annotator, who had not participated in the original annotation, to answer retained questions without access to the reference answers. We sampled 10 retained base questions per sub-task, giving 140 distinct base questions across all 14 sub-tasks, and presented each in the satellite, structural, and conventional views with the same questions, options, and images used for model evaluation.

The annotator answered all 140 questions correctly in each view (Table 9, p. 23), including exact full-set matches for the multi-select sub-task. Because the same annotator answered the same 140 questions in all three views, the 420 responses are repeated question–view responses rather than independent observations, and cross-view memory effects are possible. We therefore present this strictly as a single-annotator sampled answerability study: it supports the human answerability of the sampled retained questions, not a benchmark-wide estimate of human performance or a claim about excluded candidates.

AuthorsOpenReview ↗

Additional API-accessed Models

The original submission included two proprietary/API-accessed models. We have now evaluated three additional API-accessed systems: Qwen3.6-Plus, Qwen3.7-Plus, and GPT-5.6-Luna. All three use the full 585-scene cohort, and each now has 77,800 condition-specific evaluation records. The Direct results below use the base/direct questions in the satellite, structural, and conventional views; the marker-only analysis separately uses the blank base/direct records.

Three-view Direct accuracies (%) and T1–T4 macro-averages. Brackets denote 95% scene-bootstrap intervals for the macro-average.

ModelT1T2T3T4Avg [95% CI]
Qwen3.6-Plus76.270.943.879.067.5 [64.99, 69.84]
Qwen3.7-Plus73.870.946.875.466.8 [64.45, 68.98]
GPT-5.6-Luna80.081.266.757.971.5 [67.77, 74.81]

For comparison, Qwen3.5-Plus obtains a 64.9% Direct macro-average. The added models retain substantial task-family difficulty: T3 remains the weakest family for Qwen3.6-Plus and Qwen3.7-Plus (43.8% and 46.8%), whereas T4 is the weakest family for GPT-5.6-Luna (57.9%).

The revised Direct, marker-only, per-view, and paired-view tables (Tables 16, 17, 20, and 21) include all three models, with scene-bootstrap intervals where applicable. The Direct results therefore cover 28 models; the original six-anchor RQ2/RQ3 paired analyses remain unchanged.

Thank you for the detailed rebuttal and additional experiments. The T4 roadblock analysis clarifies the extent to which connectivity reasoning is required, while the human-answerability study and additional model evaluations address my concerns about the benchmark's evaluation coverage. I also appreciate the clarification of the geographic limitations. Overall, my concerns have been adequately addressed, and I maintain my original rating of 3 (Weak Acceptance).

Reviewer_dz2eOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MapSpatial, a GIS-grounded benchmark for representation-robust spatial reasoning over real maps. It evaluates whether models maintain consistent semantic predictions across registered map views, follow task-aware answer transformations under rotations and reflections, and correctly update their answers after changes to task-relevant structure.

The benchmark comprises 1,400 base questions across 585 geographic scenes in China, covering 14 sub-tasks in four families: qualitative direction, metric comparison, route-relative building identification and counting, and road-network path reasoning. Three registered views, geometric transformations, solver-verified structural edits with matched answer-preserving shams, and visual-evidence conditions yield 77,800 condition-specific evaluation records. The authors evaluate 25 multimodal models, with primary paired diagnostics using six category anchors and two scale references. The results show that single-image accuracy does not fully capture reliability: models exhibit substantial cross-view disagreement and transform sensitivity, and the six anchors frequently retain pre-edit answers on initially correct cases after answer-changing edits. Visual-evidence effects vary by task, evidence condition, and model interface.

Critical strengths
  1. The paired design is the main contribution. The same latent scene, question, and answer are rendered into three registered views, seven D4D_4 actions, and, for eligible T3/T4 cases, solver-verified edits with matched shams. This exposes what marginal accuracy misses: Direct is 35.2–48.3%, but Base+7 is 11.2–22.8%, ScfS_{cf} is 2.1–8.8%, and 33.8–60.7% of view triplets disagree.

  2. Labels are recomputed from GIS geometry. Base, transformed, and edited answers come from structured geometry and a solver, not from relabeling options; 400 answer-preserving shams provide a control against mistaking any prediction flip for correct updating. This improves label auditability.

  3. The statistics are unusually explicit. The 77,800 instances are described as correlated variants of 1,400 base questions; intervals use 10,000 scene-level bootstrap replicates; and the paper separates tri-view disagreement, joint correctness, transform retention, and conditional update rates.

  4. The results go beyond a leaderboard. Table 4 shows Qwen3-VL-32B with higher Direct accuracy (57.4%) but ScfS_{cf} of only 4.8% and 83.6% stale, versus Gemini-3-Flash at 37.5% ScfS_{cf} and 35.1% stale. The conditional update rate (10.1% vs 50.4%) points the same way, though it is conditioned on different base-correct subsets.

Critical weaknesses
  1. The marker-only ablation is insufficiently specified for the network-distance subtask. T2 includes composite road-network distance comparison, which requires road connectivity and link lengths. However, Appendix G.1 describes marker-only inputs as retaining point markers while removing map structure, and Table 14 reports only aggregate T2 accuracy. It is therefore unclear whether the network-distance subtask is excluded, modified, or evaluated without necessary information. This ambiguity limits the ablation’s ability to establish which tasks require map structure. A subtask-level breakdown and clarification of the marker-only construction are needed. (Appendices A.4 and G.1; Table 14)

  2. The unusually stringent filtering of T3 candidates is insufficiently characterized. Approximately 12% of candidate images are retained, but the paper does not provide an exclusion breakdown distinguishing rendering defects, ambiguity in applying the counting rules, and difficult scene configurations. These causes have different implications for benchmark validity: removing defective renderings improves quality, whereas systematically excluding particular layouts may substantially narrow the task distribution. Post-screening answer agreement does not resolve this concern. Reporting exclusion categories and relevant scene characteristics before and after filtering would clarify what population the T3 results represent. (§4; Appendix B)

Questions for authors
  1. Marker-only ablation: T2 includes composite road-network distance comparison, but Appendix G.1 states that marker-only inputs remove map structure. Is this subtask excluded, modified, or retained in Table 14? If retained, what information allows the network-distance answer to be determined without the road network?

  2. T3 filtering: What are the main exclusion categories behind the approximately 12% retention rate for T3? In particular, were candidates primarily rejected for rendering defects, ambiguity in applying the counting rules, or difficult scene configurations, and did filtering substantially change the scene distribution?

3 comments
AuthorsOpenReview ↗

W1/Q1: Marker-only Ablation and Composite Network Distance

Thank you for identifying this ambiguity. The composite network-distance subtask was retained unchanged in the original Table 14 T2 aggregate: it accounts for 100 of the 400 blank-view T2 base questions in the standard per-model slice. The marker-only image retains point markers but not the road connectivity or link lengths required to compute shortest network paths. Consequently, the network-distance answer cannot be determined from this input alone; the reference label is computed from the underlying GIS road network, which is not provided in this view. The network-distance blank score is therefore an information-removal control, not evidence that a model can solve network distance from point markers alone. The revised paper distinguishes it from the three road-independent T2 subtasks and clarifies the ablation description.

We checked the four subtask scores for all 28 models in the revised analysis. For the six primary paired-analysis anchors, the numbers of correct answers out of 100 questions per subtask are:

ModelNearestDirectional nearestComposite EuclideanComposite network
GLM-4.6V-Flash58676155
Qwen2.5-VL-7B-Instruct53484746
MiMo-Embodied-7B66446658
Spatial-MLLM48406758
Bagel-7B-MoT59662841
SenseNova-U1-8B-MoT42386862

The revised Table 17 shows T1 and the four T2 subtask columns, with the three road-independent subtasks grouped separately from the network-distance information-removal control. It no longer pools the four T2 subtasks or reports a T1/T2 average. The primary RQ1 results use the three canonical map views rather than blank images, so this clarification does not change those results or the benchmark instance count.

AuthorsOpenReview ↗

W2/Q2: T3 Screening and the Evaluable Population

Thank you for requesting a breakdown of T3 screening. We screened 1,666 candidate cases (554 segment and 1,112 route) and retained 200 (100 of each; 12.0% overall). Of the 1,466 exclusions, 326 involved adjacent buildings appearing as one, 248 involved connected parts of one building appearing separate, 739 involved a mismatch in the number of task-relevant buildings visible in the map and satellite views, and 153 involved tall buildings obscuring the road so that the rendered route appeared to pass through a building. The first two categories concern visual identification of countable building units, rather than inconsistent counting rules. We deliberately excluded cross-view building-count mismatches: for every retained T3 case, annotators checked that the task-specific building count, and hence the answer, agrees across the conventional, structural, and satellite views.

Candidates were inspected in random order until 100 segment and 100 route cases passed, so 12% is the pass rate among the 1,666 inspected candidates, and the equal segment/route split is set by this quota. Screening was designed to ensure that every retained question is answerable from all three supplied views, and it deliberately narrows the evaluable scene distribution: retained T3 cases are restricted to scenes in which individual buildings are visually separable, task-relevant building counts agree across the conventional, structural, and satellite views, and the route is not occluded by tall buildings. Consequently, the retained set may under-represent scenes with hard-to-separate building boundaries, including some dense or irregular layouts. T3 results should therefore be read as performance on visually unambiguous route-relative counting, not on the full candidate pool. The revision states this target population where T3 is introduced and includes the exclusion table in the appendix (Table 7, p. 20). The main-text scope statement appears in Section 4.

Thank you for the clarifications. W1/Q1 has been resolved. The detailed breakdown of the T3 exclusion, along with the clarification of the screening procedure, has adequately addressed my concerns. I particularly appreciate the explicit acknowledgment that filtering narrows the scene distribution, thereby limiting the scope of the claims to visually unambiguous, route-relative counting. I consider this a clearly delineated scope limitation rather than a significant weakness.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Guide to the Revised PDF

We thank the reviewers and the AC for their constructive feedback. Point-by-point responses, including new analyses, are provided under each review. Below we summarize the manuscript revisions.

RevisionLocation
Task definitions. T1–T4 are defined at first use, and the name-only sub-task list is replaced by operational definitions of all 14 sub-tasks.Introduction, p. 2; Section 3.1, pp. 4–5
T2 marker-only ablation. Composite network distance is reported separately as a missing-road-information control, and all four T2 sub-tasks are reported individually.Appendix G.1; Table 17, p. 29
T3 screening. Inspected, retained, and excluded cases by sub-task and exclusion category, the segment/route quota, and the scope of the retained set.Section 4, pp. 5–6; new Table 7, p. 20
T4 roadblock analysis. A permissive spotting-only elimination rule (given the checkpoints adjacent to the closure) with option counts; the edits are clarified as candidate-route reasoning without multi-step re-planning.New Table 8, p. 20; Limitations, pp. 9–10
Human answerability. An independent third annotator answered 140 retained base questions across 14 sub-tasks in all three views.Section 4, pp. 5–6; Appendix D and new Table 9, pp. 22–23
Additional API models. Qwen3.6-Plus, Qwen3.7-Plus, and GPT-5.6-Luna, each on the full 585-scene cohort.Tables 16, 17, 20, 21
Scene selection. The sampling frame, random-order AOI (Area of Interest) enumeration, AOI and route eligibility thresholds, and the non-probability nature of the scene set.Appendix B, pp. 17–18

Table renumbering. Tables 7–9 are new; original Tables 7–25 are now Tables 10–28 (e.g., original Table 14 is now Table 17). Our responses use the revised numbering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.