- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces UAV-MapVQA, the first benchmark for instance-level grounding and map-based geospatial reasoning between UAV images and vector maps. It contains 336K multiple-choice QA pairs derived from 14,000 real UAV images collected across 3 countries and over 1,000 km of flights. The benchmark covers eight tasks at four levels: bidirectional instance–entity grounding, change detection, egocentric heading estimation, and object-centric vehicle-to-map reasoning. Each question is paired with two synchronized map representations: a rendered raster map (Visual) and a structured entity table (Tabular). The paper evaluates 13 proprietary, open-weight, and remote-sensing foundation models, showing that all perform far below human accuracy (94.8%), and that fine-tuning on the proposed training split improves performance but does not close the gap. Heading-alignment experiments reveal that orientation errors are a measurable but partial bottleneck. The main contributions are: (1) formulating instance-to-entity grounding between UAV imagery and vector maps; (2) providing a large-scale, human-verified benchmark with two map encodings; and (3) demonstrating its utility as both an evaluation and training resource.
Critical strengths
- UAV-MapVQA is the first benchmark to require instance-level correspondence between UAV images and vector maps, going beyond class-level or image-space grounding. This is consequential because it evaluates a capability that is central to autonomous UAV navigation, map-aware reasoning, and geospatial AI, yet was previously unmeasured.
- The benchmark spans 14,000 real UAV images from 3 countries, 8 tasks, and 336K QA pairs, with both raster and tabular map encodings. The extensive human verification (registration filtering, correspondence verification, QA rating) and the scene-disjoint training split ensure high-quality supervision. The finding that fine-tuning on this data substantially improves performance (e.g., Qwen2.5-VL-32B from 23.0% to 57.2% on Visual) demonstrates its value as a training resource.
Critical weaknesses
- The benchmark inherits OSM incompleteness and residual registration error (median 1.9 m, P95 5.4 m). Although the authors exclude small buildings and require decision margins to exceed three times the per-image error, this still limits the inclusion of small objects and fine spatial distinctions. The impact of these limitations on the difficulty and validity of the benchmark could be further analyzed.
- The multiple-choice format enables reproducible evaluation, but it may not fully capture free-form reasoning or complex multi-step geospatial inference. Additionally, fine-tuning is only performed on Qwen2.5-VL-7B/32B, so the training utility of the benchmark for other model families remains untested. The paper acknowledges these points but could provide more discussion on how the benchmark could be extended.
Questions for authors
- How do you ensure that the multiple-choice questions cannot be answered by exploiting biases such as option position, answer letter frequency, or visual prompt color? You mention randomization and balancing, but can you provide quantitative evidence that a text-only or map-only model performs at chance?
- For dense scenes, the tabular map can require 2.7× more tokens than sparse scenes. How do you handle cases where the serialized table exceeds the model’s context window? Do you filter out such questions, or do you truncate? What fraction of the Tabular questions are affected?
- The heading-alignment experiment uses the registered heading as an upper bound. In a real deployment, the heading would need to be estimated from the UAV image itself. How sensitive is downstream reasoning to errors in estimated heading? Can you provide a simulation where realistic estimation errors perturb heading?
- The human reference is obtained from six college students answering 240 Visual questions each. Were they given any training or examples? How do you ensure their performance represents expert human performance, especially for tasks like T4 (heading estimation) and T8 (compass direction)?
- The dataset covers 3 countries, but OSM tags may be in different languages (e.g., German, Chinese). Did you translate or normalize tags for the Tabular version? How does language variation affect model performance?
- The paper compares with several remote-sensing and map VQA benchmarks in Table 1, but does not include a direct quantitative comparison on the same tasks. Could you clarify why this comparison isn't possible and how UAV-MapVQA complements existing benchmarks?
2 comments
Thank you for your time and feedback - happy to discuss further
Dear Reviewer bGWT,
Thank you very much for your time and thoughtful feedback on our work. As the discussion period is nearing its end, we would be grateful if you could take a look at our rebuttal when you have a chance. Please let us know if there is anything we can clarify or elaborate on; we would be happy to discuss further.
Thank you again for your time and consideration.
Best regards, Authors of Submission #6622
Clarifications and Additional Experiments Strengthening Benchmark Validity and Generalizability
We thank the reviewer for feedback that improved our benchmark and evaluation.
Our pipeline (Fig. 2) combines OSM geometry with independent UAV imagery. We estimate residual offsets between projected OSM footprints and SAM3 building masks, retaining high-confidence one-to-one matches. We manually checked fine-tuning/evaluation correspondences; an independent audit found 98% correspondence accuracy. Distance questions require margins ≥3× the estimated per-image error. Coverage depends on OSM completeness and image resolvability.
We require closed-way building targets and distractors to be ≥100 m²: smaller roofs are hard to distinguish in high-altitude imagery, even for humans. Median area rises from 157 m² (OSM buildings in study footprints) to 246 m² (verified pairs). On Visual Test-3.2K, selecting the largest candidate yields 23.8%, near 22.8% chance. We leave smaller buildings and finer distinctions to future work.
Multiple choice tests cross-modal grounding and geospatial reasoning reproducibly, without precise coordinate generation (still hard for VLMs). It mirrors UAV pipelines where detection/segmentation provides reasoning candidates.
We fine-tuned Gemma 3 12B/27B on the scene-disjoint 12.8K-question split used for Qwen2.5-VL. Both improved across all eight tasks and both representations on Test-3.2K (setup: Sec. 4.1; results: Secs. 4.2–4.3, Tables 2–3).
UAV-only and map-only baselines are near chance (App. D.5, Table 15). We balance answer positions per task; randomize colors, shapes, line widths, and option order (Sec. 3.4); and stratify distractors by size, centrality, road class, and compass direction. All heuristics are within 1.5 pp of chance on Visual Test-3.2K (Sec. 4.2, Table 4).
We neither truncate (risking lost information) nor filter (excluding dense scenes the Tabular variant aims to test). The longest serialized map has 107K tokens; all questions fit within ≥128K windows (GPT, Claude, Qwen-VL at maximum). At 32K (Qwen2.5-VL-7B/32B defaults), 6.1% exceed the limit, only in feature-dense scenes (AirLock urban areas and high-altitude VETRA tiles spanning larger regions). We score these incorrect, conservatively estimating performance.
We perturb the registered heading with Gaussian noise ϵ∼N(0,σ2) and re-evaluate the same questions (excluding T4).
The trend supports our expectation of declining accuracy with noise; Qwen is less sensitive, consistent with its limited gain from perfect alignment. This suggests heading mismatch is a measurable but partial bottleneck.
Six college students uninvolved in dataset construction participated (App. B.3). After a task/interface briefing (link: p. 6), they practiced on excluded items; no feedback was given during scoring. Human results show non-experts can answer without GIS expertise, including orientation tasks (T4: 86.7%; T8: 93.3%). We added clarification in App. B.3.
We do not translate tags. OSM semantic tags (e.g.,
building=residential,highway=primary,landuse=industrial) use standardized, language-independent terms across all three countries. Only free-text fields (e.g.,name) may be in local languages. Questions and options use semantic categories, object IDs, or coordinates; English-speaking participants can answer without knowing German or Chinese.Direct quantitative comparison is infeasible: prior benchmarks lack paired UAV imagery, aligned vector maps, and instance-level correspondences. Remote-sensing VQA (RSVQA, GeoChat) uses single images; UAV benchmarks (UAVBench, AerialVG) omit maps; Map-VQA (MapEval, MapIQ) omits imagery. UAV-MapVQA complements them by testing grounding of observed objects to map entities, then geospatial reasoning. GeoChat's below-chance 20.9% accuracy here underscores this distinction.