acceptodds
Withdrawn from ICLR 2027

GeoSeg: Training-Free Reasoning-Driven Segmentation in Remote Sensing Imagery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in multimodal large language models (MLLMs) have reframed segmentation from fixed-category prediction toward open-ended, instruction-grounded localization. While reasoning-based segmentation has progressed rapidly in natural scenes, extending these advances to remote sensing remains an open challenge, owing to the scarcity and high annotation cost of reasoning-oriented training data, as well as domain-specific obstacles such as overhead viewpoints and extreme scale variation. We present GeoSeg, a training-free framework for reasoning-driven remote sensing segmentation that circumvents the supervision bottleneck by composing pretrained vision–language models for zero-shot inference. GeoSeg bridges MLLM reasoning and pixel-level localization through two key components: (i) bias-aware coordinate refinement, which corrects the systematic spatial offset introduced when MLLMs process overhead imagery, and (ii) dual-route prompting, which fuses visual keypoint cues with semantic text prompts for robust mask prediction. We also introduce GeoSeg-Bench, a diagnostic benchmark comprising 810 annotated image–query pairs across four scene domains and three hierarchical difficulty levels, designed to support standardized, zero-shot evaluation of reasoning-driven segmentation methods in remote sensing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Remote Sensing Segmentation, Zero-shot Segmentation, Reasoning-driven Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nJ2KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary: This paper addresses instruction-guided reasoning segmentation in remote sensing imagery, where overhead perspectives and scarce reasoning-level annotations hinder the application of supervised natural-scene models. The authors present GeoSeg, a training-free framework that connects an off-the-shelf MLLM with CLIP Surgery and SAM3 without parameter fine-tuning. To handle systematic bottom-right coordinate drift caused by overhead viewpoints, the method calibrates initial bounding boxes via an empirically determined asymmetric expansion. Within the adjusted region of interest, target masks are generated through two parallel routes—visual keypoint prompting and semantic text prompting—which are merged using an intersection-first consensus strategy to filter out background noise.
Contributions:

  1. Problem Formulation & Grounding Drift Characterization: Formalizes the task of instruction-driven segmentation in remote sensing and quantifies systematic spatial drift in off-the-shelf MLLM bounding boxes under overhead views.
  2. Training-Free Dual-Route Architecture: Proposes a zero-shot inference pipeline combining bias-aware coordinate refinement (α=0.2,β=0.1\alpha=0.2, \beta=0.1) with dual-route prompt segmentation and conservative consensus fusion.
  3. GeoSeg-Bench: Builds a diagnostic benchmark of 810 curated image-query-mask triplets across four scene categories (Urban, Rural, Traffic, Nature) and three hierarchical reasoning tiers (Basic, Description, Reasoning) under a standardized zero-shot protocol.
    Validation: The framework is evaluated on GeoSeg-Bench and SegEarth-R2 against 13 baselines spanning generalist segmenters, supervised reasoning models, and open-source MLLMs under a strict zero-shot protocol. GeoSeg demonstrates superior mask accuracy, outperforming both zero-shot generalist baselines and domain-trained reasoning models across primary pixel-level metrics while maintaining competitive inference latency. Systematic ablations confirm that coordinate calibration, visual keypoint prompts, and semantic text prompts are mutually indispensable, with the omission of any component leading to clear performance drops. In addition, semantic alignment evaluations via MLLM judges and a blind user study show that the proposed method consistently ranks highest in instruction faithfulness, boundary delineation, and distractor rejection.
Critical strengths

The paper demonstrates strong empirical performance under a zero-shot protocol without any model fine-tuning. On GeoSeg-Bench, the proposed pipeline achieves 56.4% IoU and 64.2% Dice, showing marked improvements over naive zero-shot combinations and remaining competitive against supervised alternatives like LISA-7B.
Why it is consequential: High-quality instruction-mask pairs are labor-intensive to annotate in remote sensing. Demonstrating that domain-aware coordinate refinement paired with dual-route prompting can produce viable segmentation accuracy in a training-free manner offers solid practical and engineering value.

Critical weaknesses
  1. Limited Methodological Novelty Centered on Heuristic Engineering The proposed framework relies entirely on cascading existing off-the-shelf foundation models (L=\mathcal{L}= Qwen-VL-32B, C=\mathcal{C}= CLIP Surgery, and S=\mathcal{S}= SAM3) without any novel representation learning, parameter adaptation, or architectural design. The core methodological contribution is essentially a combination of empirical heuristics: static bounding box expansion (α=0.2,β=0.1\alpha=0.2, \beta=0.1) and simple intersection-first mask filtering with an area ratio threshold (γ=0.01\gamma=0.01). This leans heavily toward system-level pipeline assembly and prompt engineering rather than a fundamental visual computing innovation.
  2. Questionable Soundness and Brittleness of the "Systematic Drift" Assumption The premise that overhead imagery inherently induces a systematic bottom-right coordinate shift lacks sound geometric justification, as remote sensing images are fundamentally rotation-invariant and do not exhibit canonical orientations. Rather than being an intrinsic property of the domain, this observed shift appears to be an empirical artifact tied to the specific MLLM backbone evaluated. Relying on fixed, hardcoded expansion parameters (α=0.2,β=0.1\alpha=0.2, \beta=0.1) calibrated on a single setup is inherently brittle, raising concerns about whether this heuristic generalizes across different base models, sensor resolutions, or diverse object scales.
  3. Inappropriate Baseline Choices and Substantial Gap on Complex Benchmarks The comparative evaluation has notable fairness issues. Several MLLM baselines in Table 1 (such as OmniGen2, UniWorld, and HunyuanImage 3.0) are unified generative or image-editing models that are not designed for dense pixel-level segmentation, serving as weak strawmen that artificially pull down baseline averages. Furthermore, on the larger and more diverse SegEarth-R2 benchmark, GeoSeg's performance drops sharply (17.4% IoU and 22.0% Dice), lagging significantly behind the supervised LISA-7B baseline (26.0% IoU and 32.9% Dice). This performance gap undermines the paper's central premise that an uncalibrated training-free pipeline can practically substitute domain-specific supervision in complex remote sensing scenarios.
Questions for authors

Drift Universality & Rotation Invariance: Does the bottom-right drift persist across other MLLM backbones, or is it specific to Qwen-VL-32B? Furthermore, if an overhead image is rotated by 90°, 180°, or 270°, does the drift rotate with the objects or stay anchored to the image coordinate system?
Calibration Overlap: Given that the calibration set and GeoSeg-Bench share identical source datasets (LoveDA, NWPU-VHR-10, and DIOR), how well do the fixed parameters (α=0.2,β=0.1\alpha=0.2, \beta=0.1) generalize to entirely unseen remote sensing sensors or domains?
Small Sample Size for Reasoning: The Level 3 (implicit reasoning) subset contains only 83 samples (10.2% of the benchmark). How do the authors ensure the reported gains on reasoning queries reflect genuine reasoning capability rather than small-sample variance or noise?

Reviewer_r3z5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

To address the task of reasoning-driven segmentation in remote sensing imagery, this paper proposes GeoSeg, a training-free zero-shot reasoning framework. By orchestrating frozen pre-trained models, the framework decouples the reasoning process into three stages:

(1) leveraging a MLLM to parse complex queries and generate coarse bounding boxes with referential phrases;

(2) introducing bias-aware coordinate refinement, which mitigates spatial grounding drift in overhead views via asymmetric bounding box expansion; and

(3) employing dual-route prompting with consensus-driven fusion, which combines local visual keypoint prompts (Route A) and global semantic text prompts (Route B) within the refined crop, generating the final pixel-level mask through an "intersection-first and ordered fallback" strategy.

Furthermore, the paper constructs GeoSeg-R810, a diagnostic benchmark comprising 810 samples across four scene domains and three difficulty levels, validating the effectiveness of each component.

Critical strengths
  1. Clear Training-Free Decoupled Architecture: The paper elegantly decouples RS reasoning segmentation into three distinct stages, effectively bridging high-level semantic reasoning and low-level pixel segmentation without heavy instruction tuning.
  2. Empirical Analysis and Geometric Compensation for Overhead Grounding Drift: Using Kernel Density Estimation (KDE), the paper quantifies the bounding box prediction bias of MLLMs in overhead RS imagery, theoretically clarifying the upper-bound constraint that cropping regions impose on final recall.
  3. Complementary Dual-Route Prompting and Anti-Interference Fusion: Within the shared refined crop, the method synergizes salient local point prompts mined by CLIP Surgery with global text prompts from SAM3, enhancing robustness against background clutter.
Critical weaknesses
  1. Disconnect Between Technical Contributions and Motivation: While the paper emphasizes reasoning-driven segmentation, its innovations primarily focus on spatial grounding correction and segmentation refinement rather than reasoning capabilities per se. Consequently, there is a misalignment between the proposed method and the complex reasoning challenges highlighted in the motivation.
  2. Insufficient Justification of Research Gaps: The paper cites the scarcity of training data and standardized benchmarks as core motivations. However, existing resources such as GeoSeg-1M, LaSeRS, EarthReason, and UniGeoSeg's GeoSeg-Bench already provide substantial data and evaluation support for reasoning segmentation, undermining this claim.
  3. Inadequate Validation of Core Assumptions: The premise of "systematic spatial grounding drift" relies solely on empirical observations from a single model and limited calibration data. It lacks cross-model, cross-dataset validation and rigorous statistical analysis, failing to prove the universality or RS-specificity of this bias.
  4. Absence of Comparisons with Strong Domain-Specific Baselines: Despite focusing on RS reasoning segmentation, the experiments omit representative domain-specific methods like GeoPixel, SegEarth-R1/R2, and UniGeoSeg. This omission prevents a fair assessment of GeoSeg’s true competitiveness against existing RS technologies and weakens the persuasiveness of its performance claims.
  5. Limited Benchmark Scale and Insufficient Validation of Complex Reasoning: GeoSeg-R810 contains only 810 samples, which is relatively small compared to existing RS reasoning benchmarks. More critically, the highest difficulty level (L3) comprises merely 83 samples (10.2%). Since the paper reports only aggregate metrics without breaking down L1/L2/L3 results, it remains unclear whether performance gains stem from improved complex semantic reasoning or simply more accurate segmentation of simple attributes and spatial localization.
  6. Significant Performance Degradation on External Datasets Raises Generalizability Concerns: GeoSeg achieves 56.4% IoU on its self-built GeoSeg-R810 but drops to 17.4% on an external LaSeRS subset, reducing its margin over the strongest baseline from 31.7 to 2.9 percentage points. This stark discrepancy suggests high sensitivity to data distribution shifts, rendering current experiments insufficient to support claims of strong cross-dataset generalization.
  7. Lack of Cost-Benefit Analysis for Dual-Route Design: The dual-route strategy requires separate segmentation passes for keypoint and text prompts, potentially introducing significant inference overhead. However, the paper fails to report directly comparable metrics—such as inference latency, GPU memory footprint, or stage-wise computational costs—against single-path alternatives.
Questions for authors

Please see weaknesses. I will adjust my score based on the authors' response.

Reviewer_YjQbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GeoSeg, a training-free (frozen-weight) pipeline for referring and reasoning segmentation in overhead imagery, together with GeoSeg-R810, a manually annotated benchmark of 810 image-query-mask pairs. GeoSeg asks Qwen3-VL-32B for a box and referential phrase, asymmetrically expands the box using empirically calibrated margins, and runs SAM3 in parallel with CLIP-Surgery-derived point prompts and a text prompt before applying an intersection-first fallback rule. It reports 56.4 IoU on GeoSeg-R810, versus 24.7 for CLIP Surgery+SAM3, and 17.4 IoU on 10,000 LaSeRS training instances used as an external subset. The results indicate that this particular composition is effective on the curated benchmark, but the methodological novelty is incremental, the calibration and evaluation protocol leave important independence questions, and the evidence does not yet establish broad reasoning robustness or state-of-the-art performance.

Critical strengths
  1. The paper identifies a meaningful failure mode in overhead imagery: grounding errors constrain the spatial support available to a downstream segmenter. It makes this constraint explicit and connects crop coverage to a bound on recoverable recall.

  2. The pipeline is specified with concrete interfaces and fixed parameters, and the paper reports component ablations on both the curated set and LaSeRS. On GeoSeg-R810, removing box refinement, Route A, or Route B reduces IoU by 5.3, 3.5, or 13.2 points, respectively.

Critical weaknesses
  1. The paper does not sufficiently establish what is technically new relative to prior grounding + SAM and reasoning-segmentation systems.

  2. GeoSeg-R810 is small and skewed toward easier queries: 480/810 pairs are L1, while only 83/810 are L3.

  3. The external result is limited and not a clean held-out generalization test. It uses 10,000 public LaSeRS training instances, explicitly not an official test split, and the authors acknowledge possible upstream pretraining exposure.

  4. The ablations establish that the full pipeline is better than its component removals, but do not isolate the source of the large 31.7-point gap over CLIP Surgery+SAM3.

  5. Figure 2 appears to be generated directly by an image-generation model.

Questions for authors

please see Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.