acceptodds
Under review as a conference paper at ICLR 2027

UAV-MapVQA: Evaluating Foundation Models on Instance-to-Entity Grounding Question Answering across UAV Imagery and Vector Maps

Abstract

Current aerial perception benchmarks largely evaluate class-level recognition within the image plane, such as detection, counting, and scene understanding, but do not test whether models can ground visual instances to their corresponding real-world entities in a vector map. This capability is critical for UAV applications that rely on vector maps as compact, structured world knowledge for linking observations to semantics, attributes, and spatial context. We introduce UAV-MapVQA, the first benchmark for cross-view instance-level grounding and geospatial reasoning between UAV images and vector maps. The benchmark contains 336K multiple-choice QA pairs generated from 14,000 real-world UAV images collected across 3 countries (USA, China, and Germany) and more than 1,000 km of flights, using an automatic registration and formally verified QA generation pipeline with human validation. It spans eight tasks across four levels, from cross-view instance–entity grounding and change detection to viewpoint-aware localization and object-centric geospatial reasoning. Each question is provided with two alternative representations of the same vector map—a rendered raster map and a structured entity table—enabling controlled study of how map representation affects cross-view grounding and geospatial reasoning. Evaluating 13 foundation models (FMs), spanning proprietary and open-weight general-purpose models such as GPT-5.5 and Claude Opus 4.7 as well as domain-specific geospatial FM GeoChat, we find that no zero-shot model exceeds 50% overall accuracy, against 94.8% for humans. Supervised fine-tuning on UAV-MapVQA raises Qwen2.5-VL-32B from 23.0% to 57.2%, surpassing all zero-shot models, which confirms that the dataset provides an effective training signal; yet a large gap to humans remains, and aligning the map to the UAV heading shows that orientation errors propagate into downstream reasoning but explain only part of this gap. These findings expose major gaps in cross-view instance grounding and map-conditioned geospatial reasoning, establishing UAV-MapVQA as a new evaluation and training resource for map-aware multimodal intelligence at the intersection of aerial robotics, multimodal learning, and geospatial AI. An interactive demo of the benchmark is available at an anonymous URL: https://storage.googleapis.com/uav_map_vqa/index.html.

Primary area:
datasets and benchmarks
Keywords:
UAV Imagery, Aerial Robotics, Foundation Model, Geospatial AI, Multimodal, Instance-level Recognition, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bGWTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces UAV-MapVQA, the first benchmark for instance-level grounding and map-based geospatial reasoning between UAV images and vector maps. It contains 336K multiple-choice QA pairs derived from 14,000 real UAV images collected across 3 countries and over 1,000 km of flights. The benchmark covers eight tasks at four levels: bidirectional instance–entity grounding, change detection, egocentric heading estimation, and object-centric vehicle-to-map reasoning. Each question is paired with two synchronized map representations: a rendered raster map (Visual) and a structured entity table (Tabular). The paper evaluates 13 proprietary, open-weight, and remote-sensing foundation models, showing that all perform far below human accuracy (94.8%), and that fine-tuning on the proposed training split improves performance but does not close the gap. Heading-alignment experiments reveal that orientation errors are a measurable but partial bottleneck. The main contributions are: (1) formulating instance-to-entity grounding between UAV imagery and vector maps; (2) providing a large-scale, human-verified benchmark with two map encodings; and (3) demonstrating its utility as both an evaluation and training resource.

Critical strengths
  1. UAV-MapVQA is the first benchmark to require instance-level correspondence between UAV images and vector maps, going beyond class-level or image-space grounding. This is consequential because it evaluates a capability that is central to autonomous UAV navigation, map-aware reasoning, and geospatial AI, yet was previously unmeasured.
  2. The benchmark spans 14,000 real UAV images from 3 countries, 8 tasks, and 336K QA pairs, with both raster and tabular map encodings. The extensive human verification (registration filtering, correspondence verification, QA rating) and the scene-disjoint training split ensure high-quality supervision. The finding that fine-tuning on this data substantially improves performance (e.g., Qwen2.5-VL-32B from 23.0% to 57.2% on Visual) demonstrates its value as a training resource.
Critical weaknesses
  1. The benchmark inherits OSM incompleteness and residual registration error (median 1.9 m, P95 5.4 m). Although the authors exclude small buildings and require decision margins to exceed three times the per-image error, this still limits the inclusion of small objects and fine spatial distinctions. The impact of these limitations on the difficulty and validity of the benchmark could be further analyzed.
  2. The multiple-choice format enables reproducible evaluation, but it may not fully capture free-form reasoning or complex multi-step geospatial inference. Additionally, fine-tuning is only performed on Qwen2.5-VL-7B/32B, so the training utility of the benchmark for other model families remains untested. The paper acknowledges these points but could provide more discussion on how the benchmark could be extended.
Questions for authors
  1. How do you ensure that the multiple-choice questions cannot be answered by exploiting biases such as option position, answer letter frequency, or visual prompt color? You mention randomization and balancing, but can you provide quantitative evidence that a text-only or map-only model performs at chance?
  2. For dense scenes, the tabular map can require 2.7× more tokens than sparse scenes. How do you handle cases where the serialized table exceeds the model’s context window? Do you filter out such questions, or do you truncate? What fraction of the Tabular questions are affected?
  3. The heading-alignment experiment uses the registered heading as an upper bound. In a real deployment, the heading would need to be estimated from the UAV image itself. How sensitive is downstream reasoning to errors in estimated heading? Can you provide a simulation where realistic estimation errors perturb heading?
  4. The human reference is obtained from six college students answering 240 Visual questions each. Were they given any training or examples? How do you ensure their performance represents expert human performance, especially for tasks like T4 (heading estimation) and T8 (compass direction)?
  5. The dataset covers 3 countries, but OSM tags may be in different languages (e.g., German, Chinese). Did you translate or normalize tags for the Tabular version? How does language variation affect model performance?
  6. The paper compares with several remote-sensing and map VQA benchmarks in Table 1, but does not include a direct quantitative comparison on the same tasks. Could you clarify why this comparison isn't possible and how UAV-MapVQA complements existing benchmarks?
2 comments
AuthorsOpenReview ↗

Clarifications and Additional Experiments Strengthening Benchmark Validity and Generalizability

We thank the reviewer for feedback that improved our benchmark and evaluation.

W1: OSM Map Incompleteness and Registration Errors' Impact on Tiny Buildings

Our pipeline (Fig. 2) combines OSM geometry with independent UAV imagery. We estimate residual offsets between projected OSM footprints and SAM3 building masks, retaining high-confidence one-to-one matches. We manually checked fine-tuning/evaluation correspondences; an independent audit found 98% correspondence accuracy. Distance questions require margins ≥3× the estimated per-image error. Coverage depends on OSM completeness and image resolvability.

We require closed-way building targets and distractors to be ≥100 m²: smaller roofs are hard to distinguish in high-altitude imagery, even for humans. Median area rises from 157 m² (OSM buildings in study footprints) to 246 m² (verified pairs). On Visual Test-3.2K, selecting the largest candidate yields 23.8%, near 22.8% chance. We leave smaller buildings and finer distinctions to future work.

W2: Multiple-Choice Design and Cross-Model Generalization

Multiple choice tests cross-modal grounding and geospatial reasoning reproducibly, without precise coordinate generation (still hard for VLMs). It mirrors UAV pipelines where detection/segmentation provides reasoning candidates.

We fine-tuned Gemma 3 12B/27B on the scene-disjoint 12.8K-question split used for Qwen2.5-VL. Both improved across all eight tasks and both representations on Test-3.2K (setup: Sec. 4.1; results: Secs. 4.2–4.3, Tables 2–3).

ModelVisual accuracyTabular accuracy
Gemma 3 12B24.5% → 44.6% (+20.1 pp)25.5% → 42.9% (+17.4 pp)
Gemma 3 27B26.9% → 50.8% (+23.9 pp)30.1% → 48.5% (+18.4 pp)

Q1: Shortcut Biases in Multiple-Choice Questions

UAV-only and map-only baselines are near chance (App. D.5, Table 15). We balance answer positions per task; randomize colors, shapes, line widths, and option order (Sec. 3.4); and stratify distractors by size, centrality, road class, and compass direction. All heuristics are within 1.5 pp of chance on Visual Test-3.2K (Sec. 4.2, Table 4).

HeuristicOverall (%)
Chance22.8
Fixed letter (A–D)21.9–24.0
Fixed color (best: green)22.0–23.4
Largest candidate23.8
Most central candidate24.3

Q2: Context Window Limits for Tabular Maps

We neither truncate (risking lost information) nor filter (excluding dense scenes the Tabular variant aims to test). The longest serialized map has 107K tokens; all questions fit within ≥128K windows (GPT, Claude, Qwen-VL at maximum). At 32K (Qwen2.5-VL-7B/32B defaults), 6.1% exceed the limit, only in feature-dense scenes (AirLock urban areas and high-altitude VETRA tiles spanning larger regions). We score these incorrect, conservatively estimating performance.

Q3: Sensitivity to Heading Estimation Errors

We perturb the registered heading with Gaussian noise ϵ∼N(0,σ2)\epsilon \sim \mathcal{N}(0,\sigma^2) and re-evaluate the same questions (excluding T4).

Heading noise (σ)0°10°20°45°90°Random
GPT-5.5 (Visual)55.054.052.549.547.045.6
Qwen2.5-VL-32B FT (Visual)35.034.834.433.833.233.0

The trend supports our expectation of declining accuracy with noise; Qwen is less sensitive, consistent with its limited gain from perfect alignment. This suggests heading mismatch is a measurable but partial bottleneck.

Q4: Human Evaluation Protocol

Six college students uninvolved in dataset construction participated (App. B.3). After a task/interface briefing (link: p. 6), they practiced on excluded items; no feedback was given during scoring. Human results show non-experts can answer without GIS expertise, including orientation tasks (T4: 86.7%; T8: 93.3%). We added clarification in App. B.3.

Q5: Language Variations in OSM Tags

We do not translate tags. OSM semantic tags (e.g., building=residential, highway=primary, landuse=industrial) use standardized, language-independent terms across all three countries. Only free-text fields (e.g., name) may be in local languages. Questions and options use semantic categories, object IDs, or coordinates; English-speaking participants can answer without knowing German or Chinese.

Q6: Comparison with Existing Benchmarks

Direct quantitative comparison is infeasible: prior benchmarks lack paired UAV imagery, aligned vector maps, and instance-level correspondences. Remote-sensing VQA (RSVQA, GeoChat) uses single images; UAV benchmarks (UAVBench, AerialVG) omit maps; Map-VQA (MapEval, MapIQ) omits imagery. UAV-MapVQA complements them by testing grounding of observed objects to map entities, then geospatial reasoning. GeoChat's below-chance 20.9% accuracy here underscores this distinction.

AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer bGWT,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #6622

Reviewer_LpVjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces UAV-MapVQA, a benchmark for grounding visual instances in UAV imagery to corresponding entities in vector maps and answering map-conditioned spatial questions. It contains 336K multiple-choice questions derived from 14,000 image–map pairs across three countries, covering eight tasks involving building correspondence, unmapped-building detection, heading estimation, and vehicle-centric spatial relationships. Each question has raster-map and tabular-map versions. The construction pipeline combines image registration, segmentation-assisted correspondence filtering, geometric answer generation, and human verification. Evaluation includes 13 foundation models, supervised fine-tuning, a human reference, single-view ablations, and heading-alignment experiments. Results reveal substantial performance gaps and identify relative orientation as one contributing bottleneck.

Critical strengths

1.The benchmark addresses a meaningful and well-defined gap. Linking a drone-observed instance to a specific mapped entity is practically important and goes beyond recognizing object categories or grounding descriptions within a single image. The task suite provides a useful evaluation setting connecting aerial perception, cross-view correspondence, and map-conditioned spatial reasoning. The paired raster and tabular interfaces further broaden its potential value as an evaluation and training resource.

2.The experiments provide substantive validation of the benchmark and its motivation. The independent human reference supports the answerability of the Visual tasks, while single-view ablations test the need for both inputs. Fine-tuning produces substantial improvements, supporting the dataset’s utility as supervision. Particularly useful is the heading-alignment experiment with a random-rotation control, which provides direct evidence that relative orientation affects grounding and downstream performance. These analyses offer more insight than model rankings alone.

Critical weaknesses

Some dataset documentation could be more precise. The definition of “scene-disjoint” splits and the coverage of human verification would benefit from clarification, particularly regarding geographic overlap, shared OSM entities, and the handling of questions requiring correction.

Questions for authors

How is “scene-disjoint” defined for the training/test split? Are overlapping geographic areas or shared OSM entities allowed across splits? Please clarify the separation criterion, as this affects the interpretation of the fine-tuning gains as generalization to unseen scenes.

2 comments
AuthorsOpenReview ↗

Geographically Disjoint Train/Test Splits with Human-Verified Benchmark Quality

Thank you for taking the time to review our work and for acknowledging the importance of linking UAV observations to specific map entities.


Q1. Train/test scene separation and shared OSM entities

Within each UAV data source, 0% of test UAV footprints overlap training footprints, and the minimum edge-to-edge gap is 1 km. The original submission described the split as scene-disjoint. Revised Sec. 4.1 and App. B.1 (Table 9) define the split and report the measured gaps.

SourceMin. gap (km)Median gap (km)Test footprints overlapping train
VETRA1.6120.50%
AirLock1.001.40%
UAV-VisLoc3.426.20%
USGS-RSCC1.7916.50%

We group frames into scenes by linking footprints less than 1 km apart or frames from the same flight, then assign each connected component to one split. For USGS-RSCC we group by distance alone so that surveys spanning hundreds of kilometres can contribute distinct scenes. Although map windows extend beyond the camera view to preserve context (Sec. 3.3), potentially including the same long OSM roads across splits, the 1 km separation makes overlap in observed buildings unlikely. The fine-tuning gains measure generalization to geographically held-out UAV imagery (Secs. 4.2–4.3, Tables 2–3).

W1. Dataset documentation

We expanded App. B.1 (Tables 8–10) with per-source details on imagery acquisition, flights and surveys, frame sampling and retention, altitude, ground resolution, and footprint area. App. C (Table 13) now summarizes how we construct questions for all eight tasks, derive correct answers, and select plausible distractors. Example questions show what the model sees with a raster map (Visual) or a table of map features (Tabular).

W2. Human verification and QA corrections

We designed the pipeline with human checks at each major construction stage (Sec. 3.4 and App. B.3).

  1. Registration. Three coauthors replayed the alignment overlays in flight order for all four sources and removed visibly misregistered frames. Separately, six annotators placed ground control points on a stratified sample of 1,440 retained images. The median registration error was 1.9 m (App. B.4, Table 12).
  2. Building correspondences. The coauthors checked the OSM–SAM3 matches used to generate the 12.8K training questions and Test-3.2K, discarding shifted or ambiguous matches. A separate end-to-end audit of 20% of the images used in these splits found 98% correctly matched (App. B.3 and B.5).
  3. Question quality. The coauthors audited 1,200 questions from the initial test set for legibility, unique grounding, and prompt and answer correctness. Before correction, ratings were 92.73% accept, 3.32% minor fix, 1.58% major fix, and 2.37% reject. We corrected legibility issues by adjusting mask opacity and contrast. We replaced rejected questions, typically cases where parallax or similar building layouts made grounding ambiguous, with newly sampled questions. The audit found the question templates clear and identified no answer-key errors among the cases we corrected (Sec. 3.4 and App. B.3).
QA RatingReasonAction
Minor fix (3.32%) or major fix (1.58%)Low marker contrast, e.g., a green mask blending into grass or a label blending into the imageAdjusted mask opacity and CIELAB contrast settings
Reject (2.37%)Ambiguous matches from tall-building parallax in low-altitude urban AirLock scenes, similar building shapes, or symmetric layoutsReplaced with newly sampled QA pairs

We use the resulting Test-3.2K set as our quality-controlled evaluation subset. In a separate answerability study, six students uninvolved in construction achieved 94.8% on 1,440 Visual questions with the answer key hidden (App. B.3, Table 11).

We invite you to explore the registration and correspondence visualizations on the interactive project site and inspect all 3,200 Visual questions in the dataset viewer and quality inspector. We hope these clarifications address your concerns, and we would be happy to answer any further questions. Thank you again for your thoughtful feedback!

Thank you for your detailed response and the additional clarifications. I appreciate the effort to improve the manuscript and have no further questions.

Reviewer_PHSnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UAV-MapVQA, a benchmark for evaluating instance-to-entity grounding and geospatial reasoning between UAV imagery and vector maps. The dataset contains 336K automatically generated and human-validated multiple-choice questions from 14,000 real-world UAV images collected across three countries and over 1,000 km of flights. It covers eight tasks across four levels, including cross-view building grounding, map-change detection, heading estimation, and vehicle-to-map reasoning. Each question is provided with two synchronized map representations: a rendered raster map and a structured tabular representation derived from OpenStreetMap. The authors evaluate 13 foundation models and human participants, finding that no zero-shot model exceeds 50% overall accuracy, whereas humans achieve 94.8% on the visual setting. Fine-tuning Qwen2.5-VL-32B improves performance from 23.0% to 57.2%, demonstrating the benchmark’s value as a training resource. Additional experiments show that aligning map orientation with the UAV heading improves performance, although substantial gaps to human reasoning remain.

Critical strengths
  1. This paper formulates cross-view instance-to-entity grounding between UAV imagery and vector maps, addressing an important capability that existing aerial and geospatial benchmarks largely overlook. This makes the benchmark relevant to real UAV applications requiring map-based identification and reasoning.
  2. UAV-MapVQA combines substantial real-world data, eight task types, two synchronized map representations, automated geometric verification, and human quality audits. Evaluations with 13 foundation models, humans, fine-tuning, orientation alignment, and single-view ablations convincingly establish the benchmark’s difficulty and help identify specific model limitations.
Critical weaknesses
  1. This paper targets cross-view instance-to-entity grounding between arbitrary-view UAV imagery and structured vector maps, a capability largely missing from existing aerial and geospatial benchmarks. This directly reflects practical UAV applications where recognizing an object is insufficient without linking it to a persistent mapped entity and its attributes.
  2. UAV-MapVQA combines 14,000 real-world images, 336K questions, eight task types, and synchronized raster/tabular map representations. The registration, correspondence filtering, formal QA generation, and human auditing provide stronger supervision reliability than purely synthetic benchmarks. Evaluations across 13 foundation models, human participants, fine-tuning, orientation alignment, and single-view ablations demonstrate both the benchmark’s difficulty and its usefulness for diagnosing and improving model capabilities.
Questions for authors
  1. Please clarify whether images from the same UAV flight, nearby geographic regions, or temporally adjacent frames can appear across training, validation, and test sets. This is important for determining whether the reported fine-tuning gains reflect generalization rather than trajectory or scene memorization.
  2. Only 1,200 QA pairs were manually audited, with 92.73% accepted. How were the remaining minor, major, and rejected cases corrected or filtered, and how sensitive are the benchmark results to potential residual QA errors?
  3. Please report retention rates at each filtering stage and performance broken down by registration quality or residual error. The strict filtering may improve label reliability but could also bias the benchmark toward easier, larger, or less occluded buildings.
  4. Beyond the single-view ablations, have you tested models on questions with alternative marker styles, map renderers, option orderings, or independently regenerated distractors? Such tests would clarify whether the benchmark measures cross-view reasoning rather than artifacts of the QA-generation pipeline.
  5. Please provide details on participant training, question assignment, inter-annotator agreement, and confidence intervals. Since humans answered only Visual questions and models were evaluated under chain-of-thought prompting, the comparability of the 94.8% human reference should be discussed more precisely.
  6. The proprietary models use Test-0.8K, whereas open-weight models use Test-3.2K, and the models may differ in context limits, image preprocessing, reasoning budgets, and API versions. Please provide paired significance tests or uncertainty estimates for the main rankings and clarify whether these protocol differences could change the conclusions.
5 comments
AuthorsOpenReview ↗

Q1-Q3: We hold out scenes geographically and check data quality at each stage

Thank you for the constructive review.

Q1: Train/test scene separation

Within each source, train and test footprints have 0% overlap and a minimum edge-to-edge gap of 1 km (revised Sec. 4.1, App. B.1, Table 9).

SourceMin. gap (km)Median gap (km)Test overlap
VETRA1.6120.50%
AirLock1.001.40%
UAV-VisLoc3.426.20%
USGS-RSCC1.7916.50%

We hold out entire groups of frames linked by <1 km footprint gaps or a shared flight. For USGS-RSCC, we use distance alone so long surveys can span distinct scenes. Buffered maps can share long OSM roads (Sec. 3.3).

Q2: QA corrections and residual errors

On 1,200 audited questions, 92.73% of pre-correction ratings were accept.

RatingReasonAction
Minor (3.32%) or major fix (1.58%)Set-of-Mark legibility, e.g., a green mask blending into grassAdjusted opacity and CIELAB contrast
Reject (2.37%)Ambiguous matches from tall-building parallax in low-altitude AirLock scenes, similar shapes, or symmetric layoutsSampled replacement questions

Geometric rules generate the keys after registration and correspondence checks. The audit found clear templates and no answer-key errors in the corrected cases (Sec. 3.4, App. B.3).

With predictions and items fixed, correcting up to 5% of keys in each scored set could shift accuracy by at most 5 percentage points (pp) and gaps by 10 pp. Qwen2.5-VL-32B’s 34.2 pp Visual fine-tuning gain stays ≥24.2 pp and its 37.6 pp gap to humans stays ≥27.6 pp under this scenario. Close rankings and the 50% zero-shot Tabular threshold are more sensitive.

Explore the registration and matching demo and all 3,200 Visual questions.

Q3: Filtering, registration quality, and selection bias

Frame retention. Video subsampling and USGS-RSCC screening explain most initial losses. 91.6% of candidates yield valid footprints, and we sample 3,500 frames per source for balanced coverage (App. B.1, Table 8).

Filtering stageFramesRetained from prior stage
Raw frames858,560—
Registration candidates28,7353.3%
Successfully registered28,04497.6%
Valid image footprints26,31993.8%
Sampling pool25,58397.2%
Final benchmark14,00054.7%

Registration quality. Model accuracy by initial registration error remains unmeasured. To reduce label errors, we correct offsets between independent SAM3 building masks and projected OSM polygons, then match them and retain unambiguous, human-verified pairs (Sec. 3.3, App. B.3–B.5).

Building-size bias. We set a 100 m² minimum for closed-way targets and distractors so roofs remain distinguishable at coarse resolution. Median area rises from 157 m² for OSM buildings in the study footprints to 246 m² for verified pairs. We balance options by area rank (Sec. 3.3, App. C). Picking the largest building scores 23.8% against 22.8% chance. The best fine-tuned model reaches 57.2% on Visual and humans reach 94.8% (Tables 2 and 4). Occlusion effects remain unmeasured.

AuthorsOpenReview ↗

Q4-Q6: Visual questions are answerable by non-experts and remain challenging for models

Q4: Markers, rendering, and distractors

We have taken several measures to mitigate potential shortcuts and biases in the QA-generation pipeline.

1. Single-view baselines. Text-only and map-only baselines are reported in Appendix D.5 (Table 15), with performance near chance level, suggesting that neither modality alone provides sufficient information to solve the tasks.

2. Marker randomization and distractor design. To prevent shortcut learning, we balance answer positions within each task and randomize visual prompt styles, including marker colors, shapes, line widths, and option orderings (Sec. 3.4). Distractors are also stratified by task-relevant attributes, such as building size, centrality, road class, and compass direction, to reduce systematic biases.

3. Heuristic-based bias analysis. As reported in our response to Q3, we evaluate several answer-selection heuristics on Test-3.2K, including fixed answer positions, fixed marker colors, largest-building selection, and most-central-building selection. All perform near chance level (21.9–24.3%, versus 22.8% expected by random guessing), providing evidence against obvious answer-position, color-based, and spatial-selection shortcuts.

4. Map rendering. We adopt the standard OpenStreetMap Carto (OSM Carto) style rather than a custom task-specific renderer. This widely used cartographic style provides familiar, human-interpretable visual conventions, encoding semantic information through colors, line widths, and patterns to distinguish buildings, road categories, parking areas, water bodies, and green spaces. Using a standardized renderer avoids introducing arbitrary task-specific visual conventions while preserving semantic cues essential for cross-view reasoning.

Q5: Human study protocol and uncertainty

Participants achieved 94.8% accuracy on timed Visual questions (nominal 95% Wilson CI 93.5–95.8%). The interval describes response-level precision for these six participants under an independent-response approximation.

Training and assignment. We recruited six college students uninvolved in dataset construction to estimate average accuracy for non-experts using ordinary map-reading skills. They received instructions on the eight tasks and the browser interface, then completed an unscored practice run. Each answered 240 questions (30 per task), with one minute per question, hidden answer keys, and no feedback. We assigned distinct question sets to give each task broad coverage within that workload, yielding 1,440 unique questions, 180 per task. The resulting score is the mean accuracy of the six participants against the answer key. Estimating inter-participant agreement would require multiple responses to the same questions.

Comparison with models. We chose Visual so participants could apply ordinary map-reading skills, such as matching landmarks and interpreting compass directions. Both human and model scores measure accuracy against the answer key. Humans selected options in the timed viewer, and models generated answers under zero-shot chain-of-thought prompting. The 94.8% serves as an answerability reference for Visual under these evaluation conditions. Revised App. B.3 gives the study protocol, task-level intervals (Table 11), and viewer screenshot (Fig. 6). Sec. 4.1 describes the model protocol.

Q6: Shared-test rankings

Thank you for raising this. As described in Sec. 4.1, Test-0.8K is a nested subset of Test-3.2K, with 100 questions per task. We compare proprietary and open-weight/API models on the same 800 question IDs. On this shared subset, GPT-5.5 scores 44.0% versus MiniMax-01’s 33.5% on Visual (+10.5 percentage points), and GPT-5 scores 49.3% versus MiniMax-01’s 36.3% on Tabular (+13.0 points). Across the eight models reported on both splits, overall accuracy changes by at most 2.6 points (Appendix C).

The nominal 95% binomial confidence half-widths for individual accuracies are at most 9.8 points per task and 3.5 overall on Test-0.8K, and 4.9 per task and 1.7 overall on Test-3.2K. These quantify uncertainty in individual scores. We therefore describe close results, such as the 0.7-point Tabular gap between GPT-5 and GPT-5.5, as observed score differences and interpret their ordering cautiously.

Appendix C and D documents the shared prompting, client-side resizing, and scoring, as well as model-specific inference settings. Context limits, provider-side image processing, reasoning budgets, and API versions could change the ranking. In particular, the shared-subset comparison controls for question selection, while the sensitivity of the cross-model gaps to those inference settings remains unmeasured. Our conclusions are scoped to the reported configurations.

AuthorsOpenReview ↗

Thank you for your time and feedback - happy to discuss further

Dear Reviewer PHSn,

Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.

Thank you again for your time and consideration.

Best regards, Authors of Submission #6622

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Authors' Responses

We sincerely thank all reviewers for their valuable comments and constructive suggestions.

To facilitate discussion among the reviewers and the Area Chair, we summarize the key strengths and weaknesses following the ICLR review format in the table below. We address Reviewer PHSn's six questions separately in the subsequent discussion.

Core contributions (Key Strengths acknowledged by reviewers)bGWTLpVjPHSn
Task formulation: First benchmark for cross-view instance-to-entity grounding between UAV imagery and vector mapsS1S1S1
Benchmark scale and quality: 336K questions, 14K aerial images, eight tasks, and paired Visual/Tabular map representationsS2S1S2
Evaluation and training utility: Fine-tuning improves Qwen2.5-VL and Gemma 3 across model sizes and map formats; Qwen2.5-VL-32B surpasses zero-shot baselines on a shared subset; heading alignment measures orientation effectsS2S2S2
Critical Weakness raisedbGWTLpVjPHSnHow we addressed it
Unclear scene split rules—W1—Clarified split rules; verified 0% overlap and ≥1 km separation
Unclear human verification procedures—W1—Detailed verification procedures; confirmed 98% accuracy and audited 1,200 test questions
Impact of residual registration error on small buildingsW1——Clarified alignment corrections and justified the 100 m² threshold
Limited free-form and multi-step evaluationW2——Clarified evaluation scope and rationale for multiple-choice questions
Fine-tuning limited to QwenW2——Added Gemma 3 12B/27B, improving all eight tasks across both map formats
Figure readability——Follow-upImproved figure readability and presentation
Initial rating332—

Additional reviewer questions

  • Shortcuts (bGWT, PHSn). We balance answer positions and randomize markers; single-view baselines are near chance. Position and marker color/size/centrality heuristics score 21.9–24.3% versus 22.8% chance.

  • Heading (bGWT). With Gaussian heading noise (standard deviation 10°/20°), GPT-5.5 scores 54.0%/52.5% on seven downstream Visual tasks, versus 55.0% with exact alignment.

  • Context/language (bGWT). All questions are retained; overflows count as incorrect. 6.1% of Tabular prompts exceed 32K tokens (Qwen2.5-VL tokenizer). Tasks rely on standardized OSM tags, not local-language place names.

  • Human evaluation (bGWT, PHSn). The 94.8% accuracy is a timed Visual answerability reference for non-experts (nominal 95% Wilson confidence interval: 93.5–95.8%). We detail training and question assignment.

  • Comparisons (PHSn). Cross-group comparisons use the same 800 test questions. For models tested on both sets, overall scores differ by ≤2.6 percentage points; interpret close rankings descriptively given model-specific inference settings.

  • Prior work (bGWT). We clarify benchmark differences; direct task-level comparison requires paired UAV imagery and vector maps with instance-level correspondences.

We sincerely thank all reviewers and the Area Chair for their time, thoughtful feedback, and constructive suggestions, which have helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.