- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces MapSpatial, which consists of 14 subtasks in four task families, including qualitative direction, metric comparison, route-relative building identification and counting, and road-network path reasoning. MapSpatial is constructed on three distinct views, including satellite imagery and maps with/without text, and with rotations or mirror flips. The evaluation shows that 25 multimodal large language models perform inconsistently across visual views for the same question-answer pairs and lack robustness across transformations.
Critical strengths
S1. The paper proposes an evaluation design that varies map views (satellite, map with/without text), applies geometric transformations (rotations), and edits geographic entities on the map. This design can show performance inconsistencies in multimodal LLMs' map understanding that evaluating a single map may miss.
Critical weaknesses
W1. The paper lacks clear introductions to terminology and the 14 subtasks, making it very difficult to follow. The intended meaning often has to be inferred from the figures. For example, T3/T4 appear before their task definitions. Explanations of the subtasks and 'oracle-conditioned evidence' are spread across sections. The method section also starts with abstract definitions without providing a complete example of how the source data is used to construct questions, images, and answers. These difficulties make it hard to understand in detail how the benchmark is constructed and to assess whether the experiments support the conclusions.
W2. The evaluation is restricted to single-choice and multiple-choice questions. This tests whether models can select correct answers from provided candidates, but leaves out capabilities related to route-relative building identification and road-network path reasoning. Thus, the results offer limited insight into practical map understanding capabilities, and the benchmark's practical applications remain unclear.
Questions for authors
Q1. Which practical applications does this benchmark support? How do results on single-choice and multi-selection questions demonstrate models’ ability to perform route-relative building identification and road-network path reasoning?
Q2. How are the geographic extent, image resolution, and map scale matched across the three map views? What checks ensure that task-relevant entities are aligned and visible in each view?
Q3. How are geographic scenes and entities, such as roads, buildings, and points of interest, selected? What sampling methods are used?
7 comments
W1: Task Definitions, Evidence, and Construction Example
Thank you for pointing out these readability issues. We have revised the paper as follows.
-
T1–T4 defined at first use. The Introduction now defines the four task families when they are first mentioned (p. 2): qualitative direction (T1), metric comparison (T2), route-relative building identification and counting (T3), and road-network path reasoning (T4).
-
Operational definitions of the 14 sub-tasks. Section 3.1 (pp. 4–5) now states in one sentence what each sub-task asks the model to determine, so the task meanings no longer have to be inferred from Figure 1. The full rules remain in Appendix A.4, and the Task Matrix in Appendix E.3 summarizes each family's question, evidence, and answer forms. For the other points raised, the relevant material is located as follows.
-
Evidence terminology. Question-conditioned and oracle-conditioned evidence are defined where they first appear in Section 3.1 (p. 5). Oracle-conditioned evidence additionally uses the answer to select the correct counted set or route-relevant state for T3/T4, and serves as an answer-relevant readout reference rather than a deployable baseline or upper bound.
-
Construction pipeline and a worked example. Section 4 and Appendix B describe the construction pipeline from GIS records to registered views, geometry-generated questions and distractors, and deterministic label checks. A worked evaluation example from the released benchmark is given in Appendix K.1. Its four options consist of a base route and a natural alternative with the same endpoints, both taken from the GIS road network, plus one checkpoint reordering of each route as a distractor; option order is then randomized. All images share one crop and coordinate frame. In this case, the base route and the alternative appear as options D and C, so the solver-verified answers are D before the edit, D under the sham, and C after the closure.
W2/Q1: Option-based Evaluation and Practical Applications
Thank you for highlighting the distinction between controlled option-based evaluation and practical map use. MapSpatial supports model development and pre-deployment diagnostics rather than serving as a navigation system. Concretely, it supports (i) reliability and regression audits of map/GIS assistants across satellite, structural, and conventional renderings and the benchmark's discrete rotations and reflections; (ii) natural-language spatial question answering over satellite imagery and maps, including direction, distance comparison, and route-relative building identification and counting; and (iii) verification of supplied candidate routes after localized road closures, including checking road connectivity, remaining traversability, and the shortest feasible candidate.
T3 and T4 target the two capabilities named in the review, in controlled discriminative form. For T3, the correct count requires identifying eligible first-row buildings relative to a directed segment or multi-segment route, applying the queried side, updated after each turn for multi-segment routes, and avoiding duplicate counts. T4 requires determining candidate-route validity, waypoint order, the set of routes still traversable after a closure, and the shortest traversable alternative; the options do not supply the road connectivity needed for these determinations. Thus, the tasks evaluate the named map-dependent operations while standardizing the answer space.
The matched text-only control provides an additional check: Qwen2.5-VL-7B-Instruct scores 35.6% without images versus 45.7% with satellite images on the same 1,400 base questions; for T3 and T4, the comparisons are 25.5% versus 38.0% and 31.3% versus 40.5%. This does not reveal the model's internal reasoning process or eliminate guessing, but shows that the maps provide useful information beyond the questions and options. The option-based format is a deliberate design choice: it isolates map reading from generation fluency and makes every reference answer solver-verifiable, which is what a reliability audit requires. The benchmark is also not purely selection-based: in the RQ4 generated-evidence conditions, unified multimodal models must produce new visual artifacts, which we audit for failure modes such as omitted evidence and hallucinated text (Appendix L). We agree that open-ended building enumeration, route generation, and multi-step re-planning remain outside the current scope, as the revised Limitations state; because all questions, labels, and edits are computed from the underlying GIS geometry and solver, these open-ended variants are natural extensions of the released pipeline rather than a redesign.
Overview: Contribution and Scope
We thank the reviewer for noting that our design "can show performance inconsistencies in multimodal LLMs' map understanding that evaluating a single map may miss" (S1). This is the central contribution, and the paper quantifies it. Each base question is linked to registered satellite, structural, and conventional views, to seven D4 geometric actions with recomputed answers, and, for eligible T3/T4 cases, to a solver-verified answer-changing edit paired with an answer-preserving sham. Across the six anchor models, direct accuracy is 35.2–48.3%, yet 33.8–60.7% of registered-view triplets yield different answers, joint correctness on the base and all seven actions is only 11.2–22.8%, and only 9.6–26.2% of base-correct edit pairs are updated correctly (Scf 2.1–8.8%). Existing single-rendering map benchmarks do not measure these quantities. Below we address the specific concerns.