acceptodds
Under review as a conference paper at ICLR 2027

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning

Abstract

We introduce GeoRefer-Bench, a benchmark for evaluating whether vision-language models identify the objects denoted by spatial expressions in remote sensing imagery. Pixel overlap measures segmentation quality but does not directly assess whether the predicted object set satisfies a spatial query. GeoRefer-Bench links natural-language expressions to executable logical forms over annotated scene graphs, providing explicit query semantics and traceable answer sets. It comprises 1,710 scenes from georeferenced UAV imagery and three re-instrumented public datasets, with 50,488 objects and 49,512 executable queries covering five query structures, from entity selection to two-hop relations. Unanswerable queries, counterfactual pairs and paraphrases enable targeted tests of spatial reference resolution. We complement pixel overlap with Exact Query Success (EQS), assessing object selection through a shared mask-to-object read-out that uses ground-truth instance masks at evaluation time. Relation-blind baselines expose a discrepancy between overlap and correct selection: on answerable UAV relation queries, predicting all instances of the named category achieves 36.9 mIoU but only 9.2% EQS. Evaluation of fifteen models on 1,881 UAV test queries reveals substantial performance gaps between entity and compositional queries: the strongest overall model reaches 98.9 at entity selection but only 60.5 at two-hop composition. Stratified analyses also associate greater same-category ambiguity with lower success, even within answer-area strata. Together, these results motivate evaluating spatial referring segmentation through both pixel accuracy and query-consistent object selection.

Primary area:
datasets and benchmarks
Keywords:
remote sensing, referring segmentation, geospatial reasoning, compositional reasoning, spatial grounding, vision-language segmentation, structured prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QaThOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoRefer-Bench, a benchmark for evaluating object selection in remote sensing referring segmentation. It pairs natural-language queries with executable scene-graph programs across five levels of spatial reasoning and proposes Exact Query Success (EQS) to measure whether models select the correct object sets. The benchmark contains 1,710 scenes and 49,512 executable queries across UAV and public imagery, and evaluates a wide range of models. Diagnostic experiments, including relation-blind controls, show that models can achieve reasonable mask overlap while selecting incorrect objects.

Critical strengths
  1. I think the paper identifies a real issue with existing referring-segmentation benchmarks. The relation-blind result is particularly convincing: selecting every object in the requested category reaches high mIoU on L3 UAV queries, but low EQS. That is a striking gap, and it makes the case for checking whether the correct objects were selected.

  2. The executable queries are a good design choice. Having a scene graph and a logical form for each question makes the intended answer much less ambiguous, and it allows the authors to study empty answers, relation changes, and paraphrases systematically. I also appreciate that the spatial relations use projected geometry and physical distances when the metadata allow it. Those details give the benchmark a meaningful connection to geospatial imagery. The evaluation is also thorough.

Critical weaknesses
  1. My main concern is how much the results depend on the annotated object inventory. For a query like “the closest building,” one missed instance can change the correct answer, even if the query program itself runs perfectly. The authors describe expert checks and I would still like to see some numbers from an independent inventory audit. I am also curious how sensitive EQS is to the 0.5 object-coverage threshold used to read predicted masks back into object sets.

  2. The benchmark covers a carefully controlled kind of spatial language. This makes the evaluation precise, but it also limits what the results say about more open-ended referring expressions. I would also like a clearer comparison with executable referring-expression datasets such as CLEVR-Ref+.

Questions for authors
  1. What quantitative checks were performed on the completeness of the instance inventory and the validity of the resolved target sets? If an independently checked subset exists, please provide its size and summarize the observed errors, particularly for superlative and relational queries. Evidence of systematic omissions that change answer sets would affect my confidence in the benchmark.

  2. How do the main EQS findings change when the coverage threshold in Eq. 3 is varied within a reasonable range? I am particularly interested in whether the relation-blind diagnostic and broad model-family comparisons are stable.

Reviewer_Nf3FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose GeoRefer_Bench as a new benchmark to combine languages and queries on annotated scene graphs. It contain ~1.7k scenesand ~49k logical queries on five structures. The experiments on 15 models show that pixel overlap has relation with incorrect object selection.

Critical strengths
  1. The experiments have 15 models on both UAV imagery and three re-instrumented public datasets.
  2. Executable queries make the intended target sets explicit.
Critical weaknesses
  1. The paper's main claimed contribution is a new benchmark, but it didn't release the data/code, which lower the value of the paper.
  2. The novelty is limited since TerraScope[1] already address closely related spatial reasoning and grounding evaluation. The difference should be disucssed more clearly. [1] Shu, Yan, et al. "Terrascope: Pixel-grounded visual reasoning for earth observation." arXiv preprint arXiv:2603.19039 (2026).
  3. Some queries need distances in mettres but the paper claimed that most models didn't use the geospatial metadata. If a model lack the necessary scale information, related errors cannot be fully attributed to spatial reasoning capability. So the inputs of each model should be clearly specified and such queries should be controlled separately.
Questions for authors

Please see the Weaknesses section and consider each point as a question to address.

Reviewer_gsyNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoRefer-Bench, a benchmark for evaluating spatial referring segmentation in remote sensing imagery through both object selection and pixel delineation. It links natural-language expressions to executable logical forms over annotated scene graphs, covering five query structures: entity selection, superlatives, single relations, conjunctions, and two-hop composition. The benchmark contains 1,710 scenes, 50,488 objects, and 49,512 logical queries across a georeferenced UAV track and a public-data track constructed from three existing remote sensing datasets. Its evaluation combines Exact Query Success (EQS), using a shared read-out based on ground-truth instance masks at evaluation time, with pixel overlap, and includes unanswerable queries, counterfactual pairs, and paraphrases. Validation includes expert review, automated consistency audits, relation-blind controls, and evaluation of fifteen models on 1,881 UAV test queries and 7,879 public-track test queries. The results demonstrate that category-level success does not reliably extend to compositional references. For example, SegEarth-R2 achieves 98.9% EQS on answerable entity-selection queries but 60.5% on two-hop queries, with similar performance declines observed across the three public sources.

Critical strengths

The benchmark makes a useful and empirically supported distinction between selecting the correct objects and delineating their boundaries. Executable logical forms make the intended answer sets inspectable, while baselines that ignore spatial relations demonstrate why overlap alone is insufficient. On answerable UAV L3 queries, selecting every instance of the named category obtains 36.9 mIoU but only 9.2% EQS. Importantly, the paper treats these metrics as complementary rather than claiming that exact object selection can replace segmentation quality. This distinction is consequential because it provides a concrete evaluation mechanism for identifying errors in reference resolution that an aggregate overlap score does not explicitly characterize.

The empirical evaluation is broad and accompanied by appropriately cautious statistical analysis. The comparison includes fifteen models from four families, and evaluation on the public track checks whether the decline on compositional queries persists beyond the UAV imagery. The paper also uses paired bootstrapping with scenes as the sampling unit and correction for multiple comparisons, explicitly acknowledging that the ordering of the top three systems is unresolved. These analyses strengthen confidence in the broad performance patterns without overstating small differences between models, increasing the benchmark’s value as a diagnostic resource.

Critical weaknesses

The reliability of the ground truth is described procedurally but remains insufficiently quantified. Section 5 and Appendix G report repeated expert review and a subset with manually traced boundaries. However, they do not provide the number of independent annotators, the size and sampling strategy of that subset, quantitative agreement on query answers, or residual inventory error rates after review. This matters because both the executable answers and the reported EQS depend on the annotated object inventory. Executing the same resolver again establishes internal consistency, but it cannot establish that relevant objects were not missed or that the expressions match human interpretations of spatial relations. The paper acknowledges this distinction, yet provides insufficient quantitative evidence for independent human validation. Consequently, it is difficult to assess how much annotation error might affect exact object selection and the labels assigned to unanswerable queries.

The semantic necessity of the more complex query programs is not sufficiently established. Section 4 states that a discrete reference must be the sole instance of its category, whereas L5 is intended to select an intermediate anchor through another relation. Unless intermediate anchors qualified by a relation are exempt from this uniqueness rule, the inner relation may be redundant. For example, selecting the building closest to the pond could already produce the answer to a query about the building closest to the pond that is east of the road. As written, the interaction between these rules is unclear, and the paper does not quantify how often removing a constraint changes the answer. The existing controls, which select all instances of a category, the largest instance, or a random instance, do not test this potential shortcut. This uncertainty is consequential because the value of L5 as a diagnostic of composition depends on whether its queries actually require both relations, rather than merely expressing a simpler selection problem with a longer sentence.

Questions for authors

(1) Annotation validity. Clarification request: How many independent annotators participated in the validation, and how many scenes, objects, and queries were independently checked? Please specify the size and sampling strategy of the subset with manually traced boundaries and report any available agreement or error statistics for inventory completeness, spatial interpretation, and exact answer sets. In particular, were any queries answered independently without showing annotators the proposed answer IDs? (2) Necessity of two-hop composition. Clarification and targeted validation request: Does the requirement that an anchor be the only instance of its category apply to L5 intermediate anchors before relational filtering, or only after the inner relation has identified them? What fraction of L5 queries changes its answer when the inner relation is removed? How well does a baseline perform when it selects the nearest target to the named anchor category while ignoring the relation that qualifies the anchor? These results would help establish whether L5 genuinely requires both relations. (3) Robustness of the scoring protocol. Targeted validation request: How sensitive are EQS scores and model rankings to the coverage threshold in Equation 3, for example at eta = 0.3, 0.5, and 0.7? For systems with native instance outputs, how do the results compare with those obtained using Equation 2? Since foreground outside annotated objects may not affect the shared mapping procedure, quantifying disagreement between these criteria would clarify how strongly the conclusions depend on the evaluation adapter. (4) Availability of geospatial inputs. Clarification request: Which evaluated models actually received and used ground sampling distance and information about the direction of north, and through what input format? For queries involving metric distances, such as WITHIN(d), can the authors report performance separately according to whether the required metadata was available to the model? This would clarify whether some failures reflect missing task information rather than unsuccessful spatial reference resolution.

I am open to revising my score if the authors can provide satisfactory clarifications that address the concerns raised above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.