acceptodds
Under review as a conference paper at ICLR 2027

RS-Ref: Benchmarking Reasoning in Remote Sensing Referring Expression Tasks

Abstract

Referring Expression Comprehension (REC) bridges natural language descriptions with specific image regions, serving as a fundamental capability for Vision-Language Models (VLMs). However, existing REC benchmarks in Remote Sensing (RS) frequently rely on simplistic expressions and lack sufficient distractors. This allows models to exploit single-attribute shortcuts rather than performing visual-language reasoning. In this paper, we investigate reasoning-intensive visual grounding in RS imagery, focusing on scenarios that demand multi-constraint logical reasoning amidst abundant same-category distractors. To this end, we introduce RS-Ref, a new benchmark specifically designed for multi-step remote sensing referring expression grounding. All images and expressions are strictly curated to ensure the presence of multiple same-category distractors and require compositional descriptions involving multiple constraints. Furthermore, we propose GeoREC, a two-stage visual chain-of-thought framework for referring expression. GeoREC introduces a grounded chain-of-thought mechanism to enable multi-step spatial reasoning, and subsequently employs GRPO-based reinforcement learning to optimize these reasoning trajectories. Extensive evaluations of 20 representative VLMs on RS-Ref, alongside validation on seven existing RS visual grounding benchmarks, demonstrate both the rigorous difficulty of RS-Ref and the state-of-the-art performance of our proposed framework.

Primary area:
datasets and benchmarks
Keywords:
Remote Sensing, Referring Expression Comprehension, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6fYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets reasoning-intensive referring expression comprehension (REC) in remote sensing imagery, where existing benchmarks can be solved by shallow category or position biases. The authors introduce RS-Ref, a distractor-controlled benchmark of 1,560 image-expression pairs with same-category distractors and multi-constraint descriptions, constructed through a filtering, LLM-assisted generation, and two-round human verification pipeline. They further propose GeoREC, a two-stage grounded visual reasoning framework that combines supervised fine-tuning on grounded chain-of-thought trajectories with GRPO-based refinement using outcome and trajectory-level rewards. Experiments evaluate 20 VLMs on RS-Ref and GeoREC on seven external remote sensing grounding benchmarks, reporting strong performance and diagnostic robustness under controlled perturbations.

Critical strengths
  1. RS-Ref explicitly targets shortcut-based grounding by enforcing same-category distractors and multi-constraint expressions, and the perturbation analyses provide useful diagnostic evidence that the benchmark depends more on complete referring expressions than existing datasets.
  2. The construction pipeline includes image filtering, LLM-assisted candidate generation, annotation rules, and two rounds of human verification, with explicit efforts to avoid overlap and data leakage. These details make the benchmark contribution more credible.
  3. The two-stage framework achieves the best performance on RS-Ref among the evaluated models and generalizes well to five of seven external grounding benchmarks. The ablation further supports the contributions of grounded trajectory SFT and trajectory-aware GRPO.
Critical weaknesses
  1. Many citations in the main text are not properly formatted or integrated into the sentences, which disrupts the reading flow. The authors should carefully check and correct the citation formatting throughout the paper.
  2. The proposed GeoREC framework largely follows a conventional two-stage training pipeline consisting of SFT and GRPO-based reinforcement learning. Although grounded reasoning trajectories and a trajectory-level reward are introduced, the methodological novelty beyond standard SFT + RL remains insufficiently justified.
  3. While the paper introduces RS-Ref and evaluates 20 representative VLMs, some benchmark construction details remain unclear. Moreover, the analysis of frontier models primarily focuses on quantitative comparisons, with limited discussion of why different models exhibit substantial performance differences and what these results reveal about their reasoning limitations.
Questions for authors
  1. The proposed framework appears to primarily combine SFT on grounded reasoning trajectories with GRPO-based reinforcement learning. Could the authors clarify what distinguishes GeoREC from a naive SFT + GRPO pipeline? In particular, it would be helpful to compare against a standard CoT-based SFT + GRPO baseline under comparable settings to better isolate the benefits of the proposed grounded reasoning formulation.
  2. Could the authors elaborate on the benchmark construction process, particularly the image filtering criteria, hard distractor selection, and distribution of reasoning constraints? Furthermore, Table 5 reveals substantial performance differences among frontier models. For example, InternVL3.5-8B significantly outperforms several larger general-purpose VLMs. Could the authors provide a more detailed analysis of these observations, including representative failure cases and the underlying reasoning or grounding limitations revealed by RS-Ref?
Reviewer_d9SpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper targets referring expression comprehension (REC) in remote sensing, with two main contributions plus an evaluation study.

RS-Ref (benchmark). Motivated by the claim that existing RS referring-expression benchmarks are too easy, short, and often templated expressions with few same-category distractors. Built from object detection datasets (DIOR, DOTA, HRSC2016, FAIR1M, LEVIR), keeping images with ≥3 same-category instances; Gemini-3 drafts multi-constraint expressions under rules banning single-attribute shortcuts; two rounds of human verification yield 1,560 expressions from 9,249 candidates (16.9% keep rate, κ = 0.86 for second phase of human annotations).

GeoREC (model). Grounded Chain-of-Thought (G-CoT) represents localization as an ordered trajectory, pairing a rationale with a bounding box at each step. It starts from GLM-4.1V-9B-Base. In Stage I SFT on G-CoT-Instruct (35,967 trajectories built from VRSBench, generated by GPT-4o and verified by a second GPT-4o pass); then in Stage II GRPO on DIOR-RSVG + G-CoT-Instruct (70,711 samples, box-only supervision).

Evaluation. 20 VLMs on RS-Ref (Table 5), all zero-shot with default generation configs; seven external RS grounding benchmarks (Table 6); a stage-wise ablation (Table 7); and three perturbation-based robustness tests (§5.3).

Critical strengths
  • S1) The benchmark construction pipeline is reasonable. Building from detection datasets rather than re-annotating existing REC benchmarks is a sensible choice that avoids inheriting template bias, and the explicit exclusion of DIOR-RSVG images shows the authors considered overlap. The annotation rules (requiring 2–3 constraints and ruling out single-salient-attribute shortcuts) are a reasonable mechanism for forcing multi-step disambiguation rather than hoping it emerges, and Table 3 indicates this had the intended effect on expression length and distractor density.

  • S2) The human verification is adequately rigorous and honestly reported. Two independent rounds, 10 trained annotators, three defined intervention types, and a reported κ of 0.86. Reporting a 16.9% keep rate is a useful signal about how much of the LLM-generated data was unusable, and I appreciate that it was disclosed rather than buried.

Critical weaknesses

These are offered constructively — several may be addressable with clarification rather than new experiments.

  • W1) The "existing benchmarks are too easy" claim is partly undercut by Table 6: RSVG and GeoBench-VLM appear at least as hard as RS-Ref: baselines score in the single digits (2.0–6.0 and 1.5–8.5), and GeoREC itself reaches only 22.5 and 13.5, below its 37.0 on RS-Ref. Table 3 does show RS-Ref leads on same-category distractors (5.1 vs. 0.6), but "harder overall" doesn't follow. Without additional results, a softer claim of more control would be better supported by the evidence.

  • W2) There is circular sourcing between the benchmark and the training data: RS-Ref draws from DIOR, DOTA, HRSC2016, FAIR1M, and LEVIR (L161); G-CoT-Instruct is built on VRSBench (LL 754-755), which per its official repository is "built based on DOTA-v2 and DIOR" (https://github.com/lx709/VRSBench); and DIOR-RSVG is also built on DIOR (assuming from the name). So both training sets derive from the same image pool RS-Ref samples from. The paper does assert no leakage in §4.2 and §5.1, and extending that to filtered-out candidates is a thoughtful detail, but no deduplication procedure is described (e.g. image-file level, source image ID, perceptual hash, or instance level). Without these it would be difficult to draw conclusion based on the results.

  • W3) The novelty of G-CoT could be positioned more clearly: Using an LLM annotator to generate reasoning chains and then fine-tuning on them is a fairly common recipe in other domains, so as I understand it the contribution depends solely on using this in remote sensing. However, a quick search turned up Wu et al. (arXiv:2503.12799, Mar 2025), who attach grounding coordinates to intermediate reasoning steps and show the approach generalizes to REC (based on my quick read of abstract). So some discussion of how G-CoT is different would help clarify the contribution.

  • W4) Table 6 is framed as generalization but includes two training distributions: VRSBench is the Stage I SFT source and DIOR-RSVG is in the Stage II GRPO set, yet both are reported alongside zero-shot baselines under "to evaluate generalization beyond RS-Ref". So the reported +14.0 on DIOR-RSVG (87.8 vs. 73.8) is an in-distribution result. The gains on CHOICE (+31.0) and RSVG (+16.5) stand on their own (I did not have the bandwidth to check all the sources so I accept them at face value), whereas on VRSBench the margin is only +0.6 (76.3 vs. 75.7).

  • W5) The base model may already be near state-of-the-art on RS-Ref: Table 7 reports GLM-4.1V-9B-Base at 33.6 [email protected], which would rank first in Table 5 (above RSThinker's 29.4) before any of the paper's contributions are applied; GeoREC finishes at 37.0. However, surprisingly, the base model does not appear in Table 5 (GLM-4.6V-Flash-10B is listed instead, at 10.1).

Minor Points:

  • W6) It is unclear how much of the gain comes from the G-CoT formulation versus simply fine-tuning on ~70k in-domain grounding samples: The ablation in Table 7 varies the training stages (Base → +SFT → +GRPO w/o R_traj → full), but every row is trained on G-CoT-Instruct, so the trajectory formulation itself is never ablated. The natural control is missing: fine-tuning the same base on the same images and expressions with plain expression→box supervision, without interleaved trajectories.

  • W7) The "RSVG" column in Table 6 is undefined and uncited: Based on the reference section, I assumed it is, Zhan et al. (2023), but that one is already used for the DIOR-RSVG column.

Questions for authors
  • Q1) What is GLM-4.1V-9B-Base's full performance on RS-Ref across all metrics and distractor bins, and could it be added to Table 5?

  • Q2) Have any of the compared baselines been trained on VRSBench or DIOR-RSVG in their original work?

Reviewer_DTYkOpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RS-Ref, a remote-sensing referring-expression comprehension benchmark intended to test multi-constraint grounding among similar objects. It contains 1,560 image-expression pairs spanning 48 categories, assembled from five object-detection datasets using Gemini-assisted generation and two rounds of human verification. The paper also introduces GeoREC, initialized from GLM-4.1V-9B-Base, which learns interleaved textual reasoning and bounding-box trajectories through supervised fine-tuning and subsequent GRPO training. Evaluation covers 20 models including GeoREC, diagnostic language perturbations, training ablations, and seven additional grounding benchmarks.

Critical strengths
  1. The paper's analysis of the limitations in existing RSVG datasets and the introduction of same-category distractors are insightful. Furthermore, the proposed RS-Ref benchmark appears to have undergone rigorous human verification and filtering.

  2. Extensive experiments are conducted across multiple remote sensing visual grounding benchmarks.

Critical weaknesses
  1. Table 6 conflates within-dataset evaluation with cross-dataset generalization. Section 5.1 states that all models are evaluated zero-shot. However, GeoREC trains on VRSBench-derived data and DIOR-RSVG, both listed as external benchmarks in Table 6. The exact train/test splits are unspecified. Even with disjoint splits, these results demonstrate held-out performance within training datasets, not transfer to unseen datasets, the zero-shot needs a precise definition.

  2. Table 5 obscures the strength of the underlying base model. GLM-4.1V-9B-Base is absent from Table 5, although Table 7 reports 33.6 [email protected], which is already above all 19 comparison models. GeoREC reaches 37.0, a 3.4-point improvement. A matched direct-answer training baseline is needed to distinguish the benefit of grounded reasoning from backbone strength and additional supervision.

  3. The implemented reward contradicts its stated purpose. Section 4.2 says Eq. (2) penalizes degraded intermediate predictions, but Appendix C sets λ = 0, removing that penalty. With this setting, IoU trajectories (0,0,0,1) and (0,1,0,1) receive trajectory rewards of 1/3 and 2/3, respectively. Thus, the reward can favor oscillation over monotonic refinement, weakening the claimed process-consistency interpretation.

  4. Multiple constraints are not shown to be necessary. Removing all non-category descriptors reduces [email protected] only from 28.0 to 21.6 (Section 5.3), retaining 77% of the aggregate score. This does not establish that the remaining descriptors must be combined. A per-constraint ambiguity audit would better distinguish compositional reasoning from single-cue matching and localization difficulty.

  5. Important related work is insufficiently addressed. Ref-Adv is cited but its methodological relationship to RS-Ref is barely discussed. CoT Referring[1] and DVGBench[2] are absent, while Geo-R1 is cited but not evaluated. These works are closely relevant to distractor-controlled benchmarking, intermediate grounding, or remote-sensing reasoning; the incremental contribution needs clearer positioning and appropriately matched comparisons.

  6. The promised prompt templates are missing. Appendix B ends with "We provide the prompt templates below", but immediately proceeds to Appendix C. The GPT-4o generation and verification prompts are absent, preventing reproduction of the trajectory-construction pipeline from the supplied manuscript.

[1] Dong Q, Figueroa L, Zhao H, et al. Cot referring: Improving referring expression tasks with grounded reasoning[J]. arXiv preprint arXiv:2510.06243, 2025.

[2] Zhou Y, Chen J, Zhang Z, et al. DVGBench: Implicit-to-explicit visual grounding benchmark in UAV imagery with large vision–language models[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2026, 232: 831-847.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.