acceptodds
Withdrawn from ICLR 2027

SentinelGeo: Evaluating Geographic Grounding in Vision-Language Models from Satellite Imagery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models encode substantial geographic knowledge, yet it remains unclear how reliably they can apply it to a new observation and state a location claim that can be checked. We introduce **SentinelGeo**, a benchmark that turns single-image overhead geolocation into a falsifiable test of geographic grounding. It pairs 11,210 globally distributed, coordinate-labeled Sentinel-2 observations with an image-only protocol in which a model sees one satellite image, with no metadata, retrieval candidates, or tools, and must return a latitude-longitude prediction together with the evidence it relies on. Unlike existing overhead benchmarks, SentinelGeo scores both, auditing each rationale against the image and the true region. Across 18 systems, predictions depart clearly from random but remain coarse, with the best model missing by a median of 1,151 km, while audited rationales score poorly for narrowing the search toward the target region. A calibrated open baseline shows that lightweight task adaptation improves coordinate accuracy, including on a geographically held-out region, without a comparable change in rationale scores. SentinelGeo thus separates what models can place from what they can justify. The dataset and evaluation code will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Spatial Grounding, Vision-Language Models, Geolocation, Overhead Imagery, Evaluation Framework, Reasoning Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TXHBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SentinelGeo, a benchmark for evaluating geographic grounding in VLMs from single overhead satellite images. SentinelGeo contains 11,210 globally distributed Sentinel-2 images and evaluates both coordinate-level geolocation and the reliability of model-generated rationales. The authors evaluate 18 VLMs and further study a task-adapted Qwen2.5-VL-7B baseline, including on a geographically held-out region. The results show that current VLMs exhibit non-random but coarse geolocation ability, while their rationales provide limited support for the predicted locations. Task adaptation improves coordinate accuracy without comparable improvements in rationale quality.

Critical strengths

None

Critical weaknesses
  • My first and bigger concern is the task setup. Each overhead image covers roughly 110 km on the ground, yet its label is a single coordinate at the tile center. Such an image depicts a broad geographic region, and there is often no principled reason why its center should be treated as the precise answer. The evaluation compensates for this ambiguity by subtracting 77.64 km from every geodesic error and using additional footprint-aware rules for city and country matches. While reasonable given the large footprint, these choices make the actual localization precision difficult to interpret. The reporting scale is also too limited. Beyond city- and country-level agreement, the benchmark lacks region- and continent-level evaluation. Given the coarse spatial scale of the task, adding these broader geographic levels would provide a more informative characterization of model performance.
  • My second concern is the overall presentation. The manuscript is poorly organized and often difficult to follow. Moreover, the figures convey little information beyond showing what the benchmark images look like, with almost no visual explanation of the benchmark design or evaluation setup.
Questions for authors
  1. Could the authors further clarify the motivation and validity of formulating geolocation from such large satellite-image footprints as a single-coordinate prediction task? In particular, what does successful performance on SentinelGeo demonstrate about geographic grounding beyond coarse regional recognition?
  2. Could the authors clarify how the different components of SentinelGeo fit together into a coherent evaluation framework? The current manuscript is difficult to follow, and the figures mainly show example satellite images without clearly communicating the benchmark construction, task formulation, or evaluation pipeline. Both the writing and visual presentation would require substantial revision. More broadly, the current presentation does not yet demonstrate a level of technical depth and completeness that I would expect for an ICLR paper.
  3. Minor issue: The paper appears to misuse \cite{} in places where \citep{} is intended under the ICLR template, resulting in incorrect citation formatting throughout the manuscript.
Reviewer_Szr3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces SentinelGeo, a benchmark asking whether vision-language models can identify where a satellite image was taken and explain why. It contains 11,210 Sentinel-2 images, with a 1,000-image main test set and a 421-image South Asian regional holdout. Models receive only the image (no metadata, candidates or tools) and output a rationale plus integer latitude/longitude. Predictions are scored by geodesic distance and country/city matching; rationales are scored by a GPT-5.4 judge on visual consistency, geographic informativeness, ground-truth consistency and geographic support. Eighteen systems and a fine-tuned Qwen2.5-VL-7B baseline are tested. Main findings: the best system still has a median error above 1,000 km, rationales are often descriptive rather than localizing, and calibration effects persist on the held-out region only coarsely.

Critical strengths
  1. The research question is interesting, and the benchmark is potentially useful: it pairs location prediction with an evaluation of whether the explanation is supported by visual evidence, rather than accuracy alone.
  2. The task is clearly defined and the dataset is of reasonable size, with a regional holdout.
Critical weaknesses
  1. The explanation evaluation is not solid. Rationales are judged by GPT-5.4 as an oracle without human validation, so the explanation results, a central part of the contribution, cannot be trusted as they stand.
  2. The claim is broader than the evidence, and the paper is hard to follow. The paper frames a big question (whether large models can understand images) but studies one narrow task, coordinate prediction from satellite images. The introduction is unclear and the logical organization is weak, which makes the novelty and importance difficult to judge.
Questions for authors
  1. How well does the GPT-5.4 judge agree with human annotators on the four rationale criteria?
  2. How does this single geolocation task support the broader claim about whether VLMs understand images?
Reviewer_TaMmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

A benchmark for evaluating the geographic knowledge of VLMs by evaluating both their predicted coordinates and reasoning (through an LLM judge). Composed of 10k sentinel images semi-automatically filtered. They evaluate several LLMs and propose to calibrate a model using LLm generated reasonings, as well as a spatially held out test set to evaluate sptial inductive reasoning

Critical strengths
  • the paper is clear and well written, with all the info the readers need and a clear appendix
  • the task is interesting and the evaluation protocol is sound (with some reserve on the judge)
  • the data will be released
  • many models are evaluated, including some EO baselines
  • the held out test set is interesting
Critical weaknesses
  • The main weakness are design choices around the LLM judge of geographic reasoning (see questions for sugestions on how to address these concerns):

    • as the judge is chatgpt5.4, which is a private model, the benchmark becomes unusable the day the model is discontinued by OpenAI. Using an open weight model would have been a much better choice.
    • the prompt for the evaluation is not precise enough, as it just tell hte LLM to use 0-5 but doesn't calibrate it: what constitute a 0 or a 5 or a 2 or a 3. As such it may be very arbitrary and fickle.
    • there is no human qualification of the LLM judge: how well rated are humans experts or novies? How do the LLM ratings align with human's? See questions for sugestions on how to address that.
    • there is no control on the VLM output format, for example verbosity. It could be that the LLM judge prefers verose answers regardless or their actual insights.
  • No details on the spatial split / buffer

  • parameter efficient fine-tuning of models (like LoRA) is missing as an adaptation baseline

Minor Weaknesses:

  • In 3.2 the breakdown is difficult to follow, a table would do a better job

  • Why use the Gaussian weighted score when the GeoScore exists (from geoGuessr)? Established in image geoloc (PIGEON CVPR 2024, OpenStreetView-5M CVPR 2024) and more readable then GWPS

  • Random coordinate should be restricted to landmass

  • Figure 5-8: give the ratings of the LLM-judge

Overall:

The reviewer really appreciate the paper. It's interesting and well-written, and the evaluation and dataset are sound. The main concern is that the LLM-judge for reasoning (the most important contribution of the paper) have several deep design flaw, in terms of reproducibility, calibration, and qualification. If these issues are not solved, the value/impact of the benchmark would be greatly diminished. Thus the reviewer thinks the paper should NOT be accepted as is as it would be a waste of a potentially very interesting benchmark. The reviewer suggests some ways to address these issues in the section below. If the authors can address them, the reviewer will increase their rating, but the scope of required change may be too large for a rebuttal.

Questions for authors
  • Human evaluations (on a limited amount of images, as it will be quite expensive)

    • have humans (expert or novice) produce reasonings. What is the human performance for reasoning?
    • have humans rate LLM geographic reasonings along your axis. How do human correlate to the judge's?
  • Regarding the Closedness of the Judge:

    • Use one or several open models and use that instead, or show how well do they correlate together / with the closed Judge
  • Regarding the calibration of the Judge:

    • control for verbosity etc, does word count correlate with quality?
    • either be more precise in your prompt, or show that different LLMs produce the same ratings given the same prompt

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.