- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoRefer-Bench, a benchmark for evaluating object selection in remote sensing referring segmentation. It pairs natural-language queries with executable scene-graph programs across five levels of spatial reasoning and proposes Exact Query Success (EQS) to measure whether models select the correct object sets. The benchmark contains 1,710 scenes and 49,512 executable queries across UAV and public imagery, and evaluates a wide range of models. Diagnostic experiments, including relation-blind controls, show that models can achieve reasonable mask overlap while selecting incorrect objects.
Critical strengths
-
I think the paper identifies a real issue with existing referring-segmentation benchmarks. The relation-blind result is particularly convincing: selecting every object in the requested category reaches high mIoU on L3 UAV queries, but low EQS. That is a striking gap, and it makes the case for checking whether the correct objects were selected.
-
The executable queries are a good design choice. Having a scene graph and a logical form for each question makes the intended answer much less ambiguous, and it allows the authors to study empty answers, relation changes, and paraphrases systematically. I also appreciate that the spatial relations use projected geometry and physical distances when the metadata allow it. Those details give the benchmark a meaningful connection to geospatial imagery. The evaluation is also thorough.
Critical weaknesses
-
My main concern is how much the results depend on the annotated object inventory. For a query like “the closest building,” one missed instance can change the correct answer, even if the query program itself runs perfectly. The authors describe expert checks and I would still like to see some numbers from an independent inventory audit. I am also curious how sensitive EQS is to the 0.5 object-coverage threshold used to read predicted masks back into object sets.
-
The benchmark covers a carefully controlled kind of spatial language. This makes the evaluation precise, but it also limits what the results say about more open-ended referring expressions. I would also like a clearer comparison with executable referring-expression datasets such as CLEVR-Ref+.
Questions for authors
-
What quantitative checks were performed on the completeness of the instance inventory and the validity of the resolved target sets? If an independently checked subset exists, please provide its size and summarize the observed errors, particularly for superlative and relational queries. Evidence of systematic omissions that change answer sets would affect my confidence in the benchmark.
-
How do the main EQS findings change when the coverage threshold in Eq. 3 is varied within a reasonable range? I am particularly interested in whether the relation-blind diagnostic and broad model-family comparisons are stable.