- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces SentinelGeo, a benchmark for evaluating geographic grounding in VLMs from single overhead satellite images. SentinelGeo contains 11,210 globally distributed Sentinel-2 images and evaluates both coordinate-level geolocation and the reliability of model-generated rationales. The authors evaluate 18 VLMs and further study a task-adapted Qwen2.5-VL-7B baseline, including on a geographically held-out region. The results show that current VLMs exhibit non-random but coarse geolocation ability, while their rationales provide limited support for the predicted locations. Task adaptation improves coordinate accuracy without comparable improvements in rationale quality.
Critical strengths
None
Critical weaknesses
- My first and bigger concern is the task setup. Each overhead image covers roughly 110 km on the ground, yet its label is a single coordinate at the tile center. Such an image depicts a broad geographic region, and there is often no principled reason why its center should be treated as the precise answer. The evaluation compensates for this ambiguity by subtracting 77.64 km from every geodesic error and using additional footprint-aware rules for city and country matches. While reasonable given the large footprint, these choices make the actual localization precision difficult to interpret. The reporting scale is also too limited. Beyond city- and country-level agreement, the benchmark lacks region- and continent-level evaluation. Given the coarse spatial scale of the task, adding these broader geographic levels would provide a more informative characterization of model performance.
- My second concern is the overall presentation. The manuscript is poorly organized and often difficult to follow. Moreover, the figures convey little information beyond showing what the benchmark images look like, with almost no visual explanation of the benchmark design or evaluation setup.
Questions for authors
- Could the authors further clarify the motivation and validity of formulating geolocation from such large satellite-image footprints as a single-coordinate prediction task? In particular, what does successful performance on SentinelGeo demonstrate about geographic grounding beyond coarse regional recognition?
- Could the authors clarify how the different components of SentinelGeo fit together into a coherent evaluation framework? The current manuscript is difficult to follow, and the figures mainly show example satellite images without clearly communicating the benchmark construction, task formulation, or evaluation pipeline. Both the writing and visual presentation would require substantial revision. More broadly, the current presentation does not yet demonstrate a level of technical depth and completeness that I would expect for an ICLR paper.
- Minor issue: The paper appears to misuse \cite{} in places where \citep{} is intended under the ICLR template, resulting in incorrect citation formatting throughout the manuscript.