acceptodds
Withdrawn from ICLR 2027

When to Ground, Clarify, or Abstain: Evidence-Aware Selection and Grounding in Remote Sensing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have recently advanced remote sensing visual grounding (RSVG). However, existing RSVG methods typically assume that every query refers to one or more identifiable targets present in the image. This assumption forces models to produce a localization even when the intended target lacks sufficient visual support or the query does not contain enough information to distinguish it from similar objects. Existing work either handles these cases separately or maps both to abstention, rather than selecting between clarification and abstention according to the available evidence. To address this limitation, we formulate RSVG as a three-way selective grounding problem and propose EASG, an evidence-aware selection and grounding framework that requires the model to select Ground when one or more intended targets are sufficiently supported, Clarify when multiple candidates remain plausible, or Abstain when visual evidence is insufficient. Specifically, EASG uses a detector to extract coarse-grained candidate evidence and presents it to the MLLM as non-binding guidance, enabling the model to complete selection and grounding according to the available evidence. To support this task, we construct RS-EviBench, a benchmark comprising 68,328 samples that cover reliable grounding, referential ambiguity, and insufficient visual evidence. On the RS-EviBench test set, EASG achieves 96.89% decision accuracy, 58.83% mIoU, and 55.04% [email protected], improving mIoU and [email protected] by 8.01 and 13.36 percentage points, respectively, over the baseline, while experiments on VRSBench-VG and DIOR-RSVG further demonstrate its generalization to grounding-only RSVG benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing Visual Grounding, Evidence-Aware Grounding, Multimodal Reasoning, Ambiguity Detection, Selective Prediction, Visual Abstention, Vision-Language Models, Remote Sensing Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_cS8kOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether a remote-sensing grounding model should localize a target, request clarification, or abstain. It constructs RS-EviBench by keeping the original image–query pair, removing distinguishing attributes from the query, or masking the target. The proposed EASG combines a frozen LAE-DINO detector with Qwen2.5-VL, using marked images and candidate coordinates as prompts, followed by LoRA fine-tuning and an additional decision loss.

Critical strengths

Distinguishing an ambiguous query from insufficient visual evidence is practically useful. These situations can reasonably call for different responses.

Critical weaknesses
  1. The benchmark does not yet establish the intended evidence-aware decision ability beyond its construction process. The three labels closely follow three data-generation operations: the original query supports Ground, an attribute-deleted query produces Clarify, and a target-masked image produces Abstain. Adding auxiliary masks to Ground and Clarify is a useful control, but the mask-area, location, and aspect-ratio slices in Table 6 remain within this same construction process. They do not rule out other cues introduced by query editing or the masking procedure.
  2. The external benchmarks only evaluate localization, so neither tests whether Clarify and Abstain transfer to naturally ambiguous queries, missing targets, or different forms of visual degradation. Consequently, the high decision accuracy does not yet show that the model can make the intended distinction in realistic cases.
  3. The method’s contribution to the central decision task is limited, and the localization comparisons do not fully isolate its technical value. Ordinary Qwen2.5-VL SFT already reaches 96.44% decision accuracy, compared with 96.89% for EASG. Clarify F1 changes from 98.81 to 98.84. These results suggest that ordinary fine-tuning already solves most of the constructed decision task, while the added detector evidence mainly improves localization. Given the variation across EASG seeds in Table 4, the small decision gain also needs matched-seed comparisons with the baseline.
  4. The localization improvement is useful, but the method largely combines detector-generated visual prompts, coordinate prompts, and standard fine-tuning losses. A stronger comparison would use the same LAE-DINO evidence in a conventional proposal-ranking or refinement system. The oracle-proposal and output-dependence analyses show that EASG can go beyond copying a proposal, but do not establish its advantage over such an inference-time baseline.
Questions for authors
  1. Do the three-way decisions still work when ambiguity and insufficient evidence are created differently—for example, through independently written ambiguous queries, naturally missing targets, or different occlusion patterns? A text-only decision baseline would also help check whether Clarify can be recognized from the edited query alone.
  2. How does EASG compare with a grounding model that uses the same LAE-DINO proposals for ranking or refinement? Could the SFT baseline and EASG also be compared across the same training seeds to assess the small decision-accuracy difference?
  3. How were the data splits formed: by referring expression, image, or original source image? Since VRSBench includes DIOR imagery, please clarify whether the RS-EviBench training images overlap with DIOR-RSVG test images.
  4. Which LAE-DINO checkpoint was used, and what data was used to train it? This matters for interpreting the DIOR-RSVG results as cross-dataset generalization of the complete system.
Reviewer_iiGQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies selective visual grounding in remote sensing by dividing the conventional localization task into three responses: Ground when the available information supports localization, Clarify when the referring expression matches multiple plausible targets, and Abstain when visual evidence is insufficient. The proposed EASG framework uses a frozen LAE-DINO detector to generate candidate regions, then provides both image overlays and coordinate text to a LoRA-fine-tuned Qwen2.5-VL-3B model. The candidate boxes serve only as references, so the model may produce a final box outside the proposal set. The training objective combines autoregressive response supervision with an additional action-classification loss.

The authors also construct RS-EviBench from the Non-Unique subset of VRSBench-VG, yielding 22,776 triplets and 68,328 samples. Each triplet preserves sufficient evidence for Ground, weakens the referring expression for Clarify, and masks the target for Abstain. Most Ground and Clarify samples also include auxiliary masks to reduce the chance that the model predicts the response from the presence of a mask alone. On RS-EviBench, the paper reports 96.89% decision accuracy, 58.83% mIoU, and 55.04% [email protected], along with component ablations, proposal-dependence analysis, and experiments on VRSBench-VG and DIOR-RSVG.

Critical strengths

A clear separation between cases that need user clarification and cases that lack visual evidence. The paper distinguishes inputs for which the user must provide more information from inputs for which the image does not provide enough evidence for localization, instead of treating both as abstention. These two failure modes call for different follow-up actions, which motivates evaluating response selection alongside localization accuracy.

Critical weaknesses
  1. The benchmark needs further validation, and the dataset split and construction details are not sufficiently documented. Clarify samples are created by deleting distinguishing attributes, and Abstain samples by masking the target, so models may exploit regularities in the generated descriptions and masks rather than assess the available evidence. Plain SFT already reaches 96.44% decision accuracy, yet the paper offers neither a text-only baseline nor a baseline using simple construction cues to examine this concern. The auxiliary masks reduce the obvious mask-presence shortcut, but the reported slices by mask size and location still do not show whether construction cues alone can predict the response. The human review reports neither correction rates nor an estimate of remaining label errors. The paper also does not state which original VRSBench split supplied the 22,776 source samples, or whether samples were grouped by source image. I therefore cannot verify whether RS-EviBench training overlaps with the original VRSBench test set, or whether its own training and test sets contain overlapping images. These omissions weaken confidence in the benchmark and make the transfer results on VRSBench-VG and the other evaluation datasets harder to interpret.
  2. The method offers limited technical novelty, and the comparisons do not establish a clear advantage over existing approaches. Region-based visual prompting has precedents in SoM and in EarthMarker for remote sensing. LLM-Optic already combines query processing, open-vocabulary detection, candidate marking, and MLLM-based target selection. In remote sensing visual grounding, MB-ORES likewise uses detector proposals, cross-modal object reasoning, and box refinement, although its reasoner is not an MLLM. EASG adds three response labels, allows boxes beyond the proposals, and assigns extra weight to action classification, but its technical advance over a direct adaptation of an existing framework remains unclear. Most RS-EviBench baselines receive no task-specific fine-tuning while EASG does, and GeoViS and RSGround-R1 are absent from this evaluation. Table 2(a) also omits the original Qwen2.5-VL and a standard grounding-only SFT baseline trained on VRSBench. On both VRSBench-VG and DIOR-RSVG, EASG trails GeoViS at both IoU thresholds, and on DIOR-RSVG it also falls clearly below RSGround-R1 at both thresholds.

[1] Jianwei Yang et al. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441, 2023.

[2] Haoyu Zhao, Wenhang Ge, and Ying-cong Chen. LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding. arXiv:2405.17104, 2024.

[3] Wei Zhang et al. EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensing. arXiv:2407.13596, 2024.

[4] Karim Radouane, Hanane Azzag, and Mustapha Lebbah. MB-ORES: A Multi-Branch Object Reasoner for Visual Grounding in Remote Sensing. arXiv:2503.24219, 2025.

Questions for authors
  1. Which original VRSBench split or splits supplied the 22,776 source samples? When forming the RS-EviBench splits, were samples grouped by source image or by the original scene? Please report any shared source images between the RS-EviBench training and test sets, as well as between RS-EviBench training and the original VRSBench test set.
  2. What fraction of cases accepted by the Qwen verifier were corrected by human reviewers? Did you evaluate a text-only baseline for identifying Clarify samples?
  3. Using the same source training data, how does EASG compare with a Qwen2.5-VL-3B baseline trained only for grounding?
  4. Why is the original Qwen2.5-VL-3B result missing from Table 2(a), and why are GeoViS and RSGround-R1 not evaluated on RS-EviBench?
  5. On VRSBench-VG and DIOR-RSVG, does EASG retain the Ground/Clarify/Abstain response protocol, or is it prompted to always output a box? How are abstentions, invalid outputs, and missing boxes counted in the localization metrics?
Reviewer_TGvxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of forced localization in remote sensing visual localization when the target is missing or the representation is unclear. It proposes a three-way selective localization task, dividing the model output into Ground, Clarify, and Abstain. The authors introduce RS-EviBench, a dataset comprising 68,328 samples generated by manipulating original data through the weakening of referring expressions and the occlusion of target regions. Furthermore, the paper presents EASG, a framework that leverages a frozen object detector to provide candidate bounding boxes and coordinate prompts, while employing LoRA-based fine-tuning of a multimodal language model to jointly learn both the decision-making process and target localization. The approach is evaluated on RS-EviBench, VRSBench-VG, and DIOR-RSVG, with assessments covering three-way decision accuracy, localization precision, and cross-dataset generalization.

Critical strengths
  1. This paper points out that traditional RSVG assumes that the target exists and has a clear referent, which cannot reasonably handle insufficient visual evidence and ambiguity in linguistic referent. By distinguishing between clarification and abstention, the study offers a valuable task-oriented perspective for developing more reliable visual grounding systems.

  2. The authors organize various decision-making conditions using triplet-based data and conduct comprehensive evaluations, including component ablation studies, experiments with multiple random seeds, and an analysis of dependency on candidate bounding boxes. As shown in Table 1, the proposed EASG model improves the mIoU from 50.82% to 58.83% while maintaining high decision accuracy, demonstrating that the candidate evidence provided by the detector effectively facilitates the grounding process.

Critical weaknesses
  1. In Section 4.1, decision labels are generated through three fixed operations: retaining the original image (corresponding to the "Ground" label), removing discriminative attributes ("Clarify"), and occluding the target region ("Abstain"). Consequently, these three decision categories are strongly linked to the specific manual modifications applied to the input; Therefore, the three decisions are strongly correlated with the manual modification of the input, and the model may partially rely on text changes or occlusion patterns to identify the decision category. Although the authors attempt to rule out such shortcuts through auxiliary occlusion and attribute-masking analyses, these experiments do not sufficiently verify whether the model has acquired decision-making capabilities that are truly independent of the data construction process. Furthermore, the external evaluations on VRSBench-VG and DIOR-RSVG are limited to localization tasks, which precludes an independent assessment of the model's generalization ability regarding the "Ground," "Clarify," and "Abstain" behaviors.

  2. EASG primarily integrates existing object detectors, candidate region prompts, LoRA fine-tuning, and a decision-specific loss function. As shown in Table 1, a standard SFT of Qwen2.5-VL already achieves a 96.44% accuracy on the three-way decision task, while EASG improves this figure only marginally to 96.89%. When compared to an SFT baseline that already incorporates the decision loss, the improvement shrinks further to just 0.32 percentage points. In contrast, the primary performance gains are concentrated in localization accuracy. The ablation results in Table 3 show that candidate evidence and decision loss are somewhat complementary, but they do not fully demonstrate that EASG provides a substantial improvement in the ability to judge the sufficiency of evidence, which is different from routine task fine-tuning.

Questions for authors
  1. Are the training, validation, and test sets for RS-EviBench strictly partitioned based on the original images? Do these samples overlap with the original VRSBench-VG test set in terms of images or pointer representation?

  2. Regarding potential shortcuts in data construction, the authors have analyzed the location, area, and shape of the auxiliary occlusions. Are there any existing results that can further distinguish between a model’s recognition of occlusion patterns and its assessment of whether the visual evidence for a target is sufficient?

  3. Standard SFT already achieves a high accuracy for three-way decision-making. Could the authors further elaborate on the independent gains provided by EASG at the decision-making level? Additionally, why does the inclusion of candidate evidence lead to a significant improvement in localization accuracy, yet only a marginal increase in decision-making accuracy?

Reviewer_iKv6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EASG, an evidence-aware framework for remote sensing visual grounding that selects among Ground, Clarify, and Abstain. It combines detector-generated candidate evidence with MLLM fine-tuning and introduces RS-EviBench to evaluate three-way decision-making. Experimental results demonstrate improvements in localization accuracy and response selection.

Critical strengths
  1. The paper addresses a meaningful problem by extending visual grounding to clarification and abstention, improving the reliability of remote sensing applications.
  2. RS-EviBench provides a structured benchmark for evaluating different levels of visual and linguistic evidence.
  3. The proposed framework achieves promising localization improvements, supported by ablation studies and evaluations on multiple benchmarks.
Critical weaknesses
  1. Dataset Reliability. The benchmark relies on attribute removal and target masking, raising concerns about potential shortcuts. More natural ambiguous and visually unsupported cases are needed.
  2. Limited Methodological Novelty. The framework mainly combines existing detector prompting and MLLM fine-tuning. The contribution of evidence-aware reasoning beyond detector-assisted localization remains unclear.
  3. Insufficient Generalization. Cross-dataset experiments only evaluate localization, without validating three-way decision generalization. Potential data overlap with VRSBench-VG should also be clarified.
Questions for authors
  1. How can the authors ensure that the model learns genuine evidence reasoning rather than shortcuts from attribute removal and target masking?
  2. Since standard SFT already achieves 96.44% decision accuracy, what is the key methodological contribution beyond detector-assisted localization?
  3. Can the authors evaluate three-way decision generalization on unseen datasets or more realistic ambiguous scenarios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.