acceptodds
Withdrawn from ICLR 2027

UniEvo-RS: Omni-Prompt Unified Remote Sensing Segmentation with Representative Exemplar-Driven Prototype Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prompt-driven vision-language models (VLMs) hold immense promise for accelerating dense remote sensing (RS) annotation, but static models suffer from severe performance degradation when deployed on novel scenes, unseen categories, or visually confusing backgrounds. Moreover, existing unified paradigms primarily rely on intra-image specific prompts, lacking flexible task routing to adapt to multi-intent operational workflows. In practical batch mapping, annotators typically refine a small set of representative samples before processing large datasets. Motivated by this practice, we propose UniEvo-RS, an omni-prompt unified RS segmentation framework equipped with representative exemplar-driven prototype evolution. First, we construct a multi-instruction prompt dataset that unifies text-driven and visual-driven prompts within a single architecture, establishing a dynamic task-routing mechanism for highly diverse RS annotation scenarios. Second, we introduce a representative feedback-driven, training-free prototype evolution mechanism. By contrasting manual annotations with initial predictions on exemplars, UniEvo-RS distills prediction errors into positive and negative prototypes. These prototypes enhance LLM query recall and suppress spatial background noise under a fixed-budget clustering memory. Extensive experiments show that UniEvo-RS unifies diverse prompting tasks, achieving state-of-the-art performance across most settings. Crucially, with minimal interaction on a few exemplars, it enables training-free, progressive accuracy enhancement on unseen categories during batch annotation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
remote sensing segmentation, vision-language models, multimodal prompts, unified segmentation, cross-image segmentation, prototype evolution, training-free adaptation, batch annotation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rk5sOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has two parts. The first is a unified remote sensing segmentation model that handles five prompt settings in one network (generic, referring, interactive, intra-image one-to-many and cross-image one-to-many). It is initialised from PSALM and trained on a multi-instruction dataset reorganised from SIOR, iSAID, NWPU-Refer and RRSIS-D. The second is a training-free "prototype evolution" for batch annotation. After an annotator verifies a few predictions, the true-positive, false-negative and false-positive regions are pooled into prototypes and compressed with K-Means under a fixed budget. Positive prototypes are injected into the LLM input and negative ones are used to suppress mask logits. Table 1 compares the unified model with general and RS-specific baselines. Table 2 evaluates prototype evolution on the Vehicle category of SIOR.

Critical strengths
  1. The batch-annotation setting is realistic. Reusing what an annotator has already corrected, without updating a large model, matches how mapping work is done, and splitting the verified result into TP, FN and FP regions that feed separate positive and negative paths is a clean design. I am not aware of prior work doing exactly this. The observations that FN prototypes matter more than TP ones (Table 4) and that K-Means is much better than other compression rules (Table 8) are useful.

  2. One model covers five prompt settings and is compared with a current set of baselines, both as released and fine-tuned on the same data. The gains on generic segmentation (PQ 62.60 vs 42.81) and on the one-to-many settings are large.

Critical weaknesses
  1. Prototype evolution is in the title, but the evidence for it is thin. The only quantitative result in the main text is Table 2, which covers one category, has no variance, and does not say how many exemplars were verified or how they were picked. The gain is +4.81 gIoU and +4.54 mIoU. The abstract promises "progressive" improvement, yet in Table 7 mIoU is 77.06, 76.67, 77.71 and 77.12 as the sequence grows from 50 to 1000. If sequence length counts verified images, more feedback does not help.

  2. The method also looks fragile. On Vehicle, a 7:1 allocation gives 32.25 gIoU (Table 4), and importance-based selection gives 18.46 in Table 8, which appears to use the same setting. Both are below the no-memory baseline of 36.43 in Table 2. On Storage Tank the 4:4 allocation drops to 40.95 while 1:7 and 6:2 give 56.54 and 54.53. This does not fit the statement in A.2.5 that the method does not depend on a highly specific parameter value, and these settings appear to be chosen on the evaluated categories.

  3. The unified model looks like PSALM fine-tuned on reorganised RS data. A.3.1 says the model follows PSALM and starts from its weights, but PSALM appears only in the zero-shot block of Table 1, and there is no single-task versus multi-task ablation. How the baselines were fine-tuned is not described, and some entries look odd. RemoteSAM drops from 25.67 to 10.74 PQ after fine-tuning and YOLOE from 13.86 to 9.78 cIoU on cross-image. On referring segmentation the model is about 10 points behind RemoteSAM (55.72 / 69.83 vs 65.75 / 78.96).

Minor points:

  • Settings are missing in several tables. mIoU is never defined, Tables 5-7 do not say which category they use, and it is not stated whether Table 1 uses the prototype memory.
  • Citations are in textual form where parenthetical ones are needed throughout Sections 1 and 2, e.g. "The SAM series Kirillov et al. (2023); Ravi et al. (2024)". Spaces after periods are missing in several places, e.g. "context.These".
  • The same mechanism has several names, "representative feedback-driven" and "representative exemplar-driven" prototype evolution, "fixed-budget clustering memory" and "fixed-budget memory fusion".
Questions for authors
  1. In Table 2, how many exemplars are verified and how are they selected? Is the model the one trained with Vehicle held out, as in A.2.3? Please report mean and standard deviation over exemplar sets and K-Means seeds, and add more categories.
  2. What does "Seq. Len." count in Table 7? If it is the number of verified images, why is there no improvement from 50 to 1000, and what does the curve look like between 0 and 50?
  3. With the same verified exemplars, how does prototype evolution compare with a few steps of LoRA fine-tuning, in accuracy and in cost?
  4. What does PSALM get when fine-tuned on your training split with the same schedule? How were the baselines in the lower block of Table 1 fine-tuned, and why do RemoteSAM and YOLOE get worse?
  5. Why can a different allocation or compression rule push the result well below the no-memory baseline? Is there any safeguard against the memory hurting predictions?
Reviewer_s17QOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes UniEvo-RS, an omni-prompt unified remote sensing segmentation framework, which is designed to address the limited task adaptability of existing methods and their performance degradation on novel scenes and unseen categories.

First, the authors construct a multi-instruction dataset integrating textual and visual prompts and develop a unified vision-language segmentation architecture to support diverse segmentation tasks. Second, they introduce a representative exemplar-driven, training-free prototype evolution mechanism that extracts positive and negative prototypes by comparing human-verified annotations with initial model predictions.

Experimental results demonstrate competitive performance across multiple remote sensing segmentation tasks and show that the proposed method can improve segmentation accuracy on subsequent images using a small number of verified exemplars without updating model parameters.

Critical strengths
  • The paper presents a unified architecture supporting diverse text- and visual-prompted remote sensing segmentation tasks, achieving competitive performance across multiple benchmarks.

  • The proposed prototype evolution mechanism leverages verified exemplar feedback to improve segmentation performance without updating model parameters, offering practical value for large-scale remote sensing annotation.

Critical weaknesses
  • The novelty of this paper is a big issue. The proposed prototype evolution mechanism mainly combines existing techniques, including mask pooling, K-Means clustering, cross-attention, and logit suppression, without introducing substantial algorithmic innovation. The authors should clearly explain the significance of the proposed method.

  • The global negative spatial map is applied to all segmentation queries, which may inadvertently suppress valid foreground predictions and introduce cross-query interference. I hope the authors could explain how the global negative spatial map works.

  • The training or initialization of the negative prototype projection function is unclear, raising concerns about the reliability of prototype-query similarity estimation. The authors are encouraged to make this clear in the revision.

  • The claimed training-free adaptation to unseen categories is primarily supported by qualitative examples, without comprehensive quantitative evaluations across multiple held-out categories and exemplar budgets. Maybe, the author should provide more evidence to support this.

  • The number and selection of verified exemplars are insufficiently specified. Moreover, the reconstructed multi-task dataset lacks explicit image-level split verification, leaving potential data overlap concerns unresolved.

Questions for authors
  • Could the authors clarify whether all baselines use comparable training budgets and explain the performance degradation of several fine-tuned baselines? How is image-level overlap between training and testing prevented?

  • How are prototype memories managed when multiple categories are processed simultaneously? Could historical prototypes from different categories interfere with each other during batch annotation?

  • Does segmentation accuracy consistently improve as more exemplars are verified? Could the authors provide performance curves under increasing annotation budgets and different exemplar selection strategies?

Reviewer_JrJ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniEvo-RS, a unified remote sensing image segmentation framework based on PSALM architecture and omni-prompt. The experimental results show certain potential for application, but the overall innovation of the method is limited.

Critical strengths
  1. The problem has practical value: The paper addresses the practical pain points of high cost of dense annotation of remote sensing images, cross-scene distribution shift, and frequent switching of multiple intentions. Under a single architecture, it uniformly supports five types of segmentation tasks and achieves competitive results in multiple indicators.
  2. The error feedback reuse mechanism is intuitive and effective: The ablation experiments in Table 2 preliminarily show that compared to directly passing the previous-image mask or simply concatenating positive memory features, explicitly decoupling and reusing verified prediction errors can effectively improve cross-image segmentation performance.
Critical weaknesses

The architecture design of the model has not undergone any fundamental changes, mainly the direct migration of PSALM to remote sensing datasets; The core improvement belongs to the combination of engineering and incremental methods. This article mainly adds a K-Means-based prototype memory and two heuristic guidance paths during the inference stage. Lack of fair comparison of key baselines (Table 1): Table 1 only reports zero-shot results of PSALM, lacking fine-tuned PSALM results under the same dataset and training conditions; The baseline of the prototype evolution experiment is too weak (Table 2): Table 2 only compares simple variants and lacks strong baseline comparison using the same verified annotations as multi-exemplar visual prompting. It is necessary to input the verified examples to DINOv and OmniSegNet simultaneously for comparison.

Questions for authors

Compared to PSALM and OmniSegNet, what substantial innovations does the method proposed in this article have in terms of model architecture or learning paradigm? Can the fine-tuned PSALM benchmark results be added to Table 1? Meanwhile, in the prototype evolution experiment in Table 2, can we supplement the use of a strong multi-exemplar prompting baseline with the same verified annotation budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.