acceptodds
Under review as a conference paper at ICLR 2027

DISP: Decoupled Image-Semantic Prompting with Multi-scale Visual Alignment for Remote Sensing Referring Segmentation

Abstract

In remote sensing referring segmentation, models must localize socially defined regions—such as parks, schools, and commercial zones—whose boundaries depend not only on visual appearance but also on functional and contextual semantics. This makes socio-semantic segmentation especially challenging, since visually similar regions may correspond to different social meanings and require fine-grained cross-modal reasoning. Existing vision-language segmentation paradigms still face two key limitations: single-token methods entangle semantic reasoning and spatial localization within one representation, while multi-stage VLM-to-SAM pipelines compress rich socio-semantic understanding into coarse geometric prompts; moreover, they insufficiently model multi-scale regional visual statistics needed for precise masks. To address these challenges, we propose DISP (Decoupled Image-Semantic Prompting with Multi-scale Visual Alignment), a unified framework that decouples the language-semantic role and visual-spatial role into two dedicated tokens, [SEG] and [IMG]. These tokens are projected through separate pathways and used as independent sparse prompts for the SAM2 decoder under a split-path dual-image architecture, enabling collaborative yet disentangled semantic-spatial segmentation. We further introduce Multi-scale Masked Feature Matching (MS-MFM), a training-only supervision module that aligns the [IMG] token with attention-pooled target-region representations from multiple ViT layers, allowing it to capture hierarchical visual cues with zero inference overhead. Experiments on socio-semantic segmentation benchmarks demonstrate that DISP achieves consistent SOTA performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object Detection, Segmentation & Scene Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zhDuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DISP, a framework that employs two separate tokens to model semantic understanding and visual localization, respectively, and introduces Multi-scale Masked Feature Matching to enhance visual representations. Experiments on the SocioSeg dataset demonstrate that the proposed method outperforms existing baselines, while ablation studies validate the contributions of the two components. The paper addresses a practically relevant problem with a relatively simple framework and achieves certain performance improvements. However, the core technical novelty is limited, the existing ablation studies do not sufficiently justify the necessity of the decoupled design and multi-scale supervision, and evaluation on a single dataset is insufficient to establish the broader effectiveness of the method.

Critical strengths
  1. Practically relevant task. The paper addresses socio-semantic segmentation in remote sensing, where identifying target regions requires functional and contextual information, making the task practically meaningful.

  2. Relatively simple framework. The dual-token prompting mechanism and multi-scale feature supervision can be integrated into existing MLLM-SAM2 frameworks and yield measurable performance improvements.

Critical weaknesses
  1. Limited novelty and insufficient methodological motivation. The dual-token decoupling mechanism and multi-scale feature alignment are conceptually similar to common designs in existing vision-language segmentation methods, offering limited methodological novelty. Furthermore, the scope and definition of socio-semantic segmentation remain insufficiently clarified, and the paper lacks a convincing justification of how this task fundamentally differs from conventional referring segmentation.

  2. Insufficient experimental validation. Experiments are conducted exclusively on the SocioSeg dataset, without cross-dataset evaluations to assess generalization. The existing ablation studies are also relatively limited. Additionally, the figures are overly schematic, making it difficult to adequately assess the quality of segmentation boundaries.

Questions for authors
  1. How is socio-semantic segmentation rigorously defined? What unique challenges does it present compared with conventional remote sensing referring segmentation? Is DISP equally effective on standard segmentation datasets?

  2. Do the performance improvements primarily result from semantic-spatial decoupling, or from the additional tokens, projection parameters, and auxiliary supervision?

  3. Could the authors provide cross-dataset experiments to further evaluate the generalization capability of the proposed method?

Reviewer_H6QJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies socio-semantic segmentation from paired satellite imagery, digital maps, and text queries. It introduces two separately projected tokens, [SEG] and [IMG], to provide semantic and visual prompts to SAM2, together with a training-only multi-scale masked feature-matching objective. Evaluation on the existing SocioSeg benchmark reports improvements across the Socio-name, Socio-class, and Socio-function tasks, with the largest gains on Socio-function. The paper also presents ablations of the image-oriented token and the auxiliary matching objective.

Critical strengths
  1. A focused prompt-learning design. The two separately projected tokens provide a concrete way to explore complementary semantic and visual prompting for mask prediction. The region-level feature-matching objective offers an explicit training signal for the image-oriented token, while the auxiliary supervision is confined to training.
  2. Potentially useful task-level gains. The reported improvements are strongest on Socio-function, where the target is defined by social use rather than appearance alone. If verified, these results would support the practical value of the proposed design for segmenting functional urban regions.
Critical weaknesses
  1. The provenance of the baseline results requires clarification. Table 1 states that all baselines were re-trained on SocioSeg. However, comparing Table 1 of this submission (p. 8) with Table 1 in Wang et al. [1] (p. 9) reveals a highly regular pattern across the ten shared methods and twelve metrics: all 116 non-zero entries differ by exactly 0.1 percentage points, with 57 increases and 59 decreases; the remaining four zero entries are unchanged. This consistent pattern warrants a clear explanation of how the baseline results were obtained and reported.
  2. The evidence does not yet isolate the claimed semantic–spatial decoupling mechanism. Table 2 compares the full model only with variants removing [IMG] or MS-MFM. Removing [IMG] also removes the representation supervised by MS-MFM, and the treatment of that loss is unspecified. These comparisons therefore do not isolate role-specific decoupling from additional prompt capacity or auxiliary supervision. An equal-capacity two-token control and separate ablations of the multi-scale, attention-pooling, and covariance components would be informative. The full model’s overall advantage over the variant without MS-MFM is 0.9/0.9/0.6 points, without repeated-run variability being reported. Equations (15)–(16) compute covariance across samples’ pooled region descriptors, rather than across patches within an individual region. They directly support cross-sample distribution alignment, but do not establish the claimed modeling of within-region texture or spatial structure. DISP specifies InternVL3-2B, whereas the cited baseline implementation uses Qwen2.5-VL-3B; the configurations of the re-trained baselines are not sufficiently documented to isolate the proposed design. Key training details, including parameter-freezing choices, batch size, learning rate, training duration, and EMA/gradient handling, are also needed for reproducibility.

[1] Wang, Y., Wang, Y., Dai, R., Wang, Y., Liu, K., Chu, X., & Li, Y. (2026). Urban socio-semantic segmentation with vision-language reasoning. In International Conference on Learning Representations (ICLR).

Questions for authors
  1. Clarify the provenance of the baseline results How were the baseline results in Table 1 obtained, and how do you explain the numerical pattern described above? Please identify quoted, re-evaluated, and re-trained results and provide the corresponding configurations, seeds, run records, and evaluation outputs.
  2. Clarify input routing and feature supervision Which images enter the MLLM during training and inference? Figure 2 and Equations (3), (7)–(8) suggest separate map/MLLM and satellite/SAM2 paths, while the algorithm and prompt templates suggest joint use of both images. Which encoder supplies the MS-MFM features, and how are freezing, stop-gradient operations, and EMA updates implemented?
  3. Add controlled experiments Please add an equal-capacity two-token control and separate the effects of multi-scale feature matching and the covariance term. Clarify which losses remain active when [IMG] is removed. For the main comparison, use a matched VLM backbone and comparable training budget, and report means and variability across repeated runs.
  4. Substantiate the interpretation of covariance matching What evidence links the cross-sample covariance objective to within-region structural modeling? Please either provide a direct analysis supporting this interpretation or narrow the claim to what the objective and experiments establish.
  5. Release code and reproducibility materials Please release the training and inference code, preprocessing and evaluation scripts, and complete configurations, including dataset splits and random seeds, together with checkpoints or instructions for obtaining them.
  6. Expand qualitative comparisons and failure analysis Please extend Figure 3 with representative examples from all three task levels, showing satellite images, digital maps, queries, ground-truth masks, and predictions from the full model, key ablations, and the strongest baseline. Include failure cases involving semantic confusion, incomplete coverage, and boundary errors, and explain what these comparisons reveal about the roles of [SEG], [IMG], and MS-MFM.
Reviewer_eGx7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses socio‑semantic referring segmentation for remote‑sensing data, where targets are defined by social‑functional semantics rather than visual appearance. Built on MLLM‑SAM2, the proposed DISP introduces two disentangled special tokens for semantic reasoning and spatial localization, alongside a training‑only multi‑scale masked feature‑matching loss (MS‑MFM) with zero inference overhead. Evaluated on the SocioSeg benchmark, it outperforms prior methods, especially for high‑level functional segmentation tasks.

Critical strengths
  1. The practical conflict between semantic reasoning and spatial localization in single‑token prompting is well‑identified. Splitting roles into [SEG] and [IMG] yields consistent empirical gains under end‑to‑end training.
  2. This multi‑order feature regularization works purely in training and adds no test‑time computation cost, which is appealing for practical deployment.
Critical weaknesses
  1. The core claim that two tokens specialize in semantics vs. spatial cues is only supported by final metrics, without probing experiments. Improvements may stem merely from larger prompt capacity.
  2. MS‑MFM’s first‑/second‑order terms are not ablated separately. All main experiments rely solely on SocioSeg; quantitative robustness tests for noisy/out‑dated map inputs are missing.
Questions for authors
  1. Add ablation to quantify individual contributions of first‑order and second‑order terms within MS‑MFM.
  2. Report quantitative results under noisy or outdated map inputs to test real‑world robustness.
  3. Discuss obstacles for evaluating on extra remote‑sensing benchmarks beyond SocioSeg.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.