- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of text-prompt selection for text-conditioned remote sensing segmentation models (e.g., SAM 3, SegEarth-OV, CAT-Seg) under a limited annotation budget. The authors introduce PromptCoreset, a training-free calibration method that selects representative unlabeled target images based on a "prompt-behavior signature" combining foreground extent and pairwise candidate mask agreement. Using deterministic k-medoids, a small calibration subset is annotated to estimate cluster-weighted global IoU and pick a single reusable prompt for deployment. Evaluated across six remote-sensing datasets and three frozen segmentation systems (24 dataset-class units), PromptCoreset successfully reduces prompt regret and lowers prompt-conditioned inference calls by approximately 89.5% relative to exhaustive search.
Critical strengths
- A major strength of the paper is its entirely training-free pipeline, which avoids the heavy computational overhead of fine-tuning backbone segmentation models. By leveraging candidate-mask behavior, PromptCoreset successfully reduces prompt-conditioned inference calls by approximately 89.5% relative to exhaustive search. This provides a highly efficient, plug-and-play solution for resource-constrained remote-sensing deployments.
- Instead of relying solely on naive visual diversity or standard image coreset methods, the authors uniquely formulate prompt selection as a calibration-set design problem driven by "prompt-behavior signatures" (combining foreground extent and pairwise mask agreement). This shifts the focus from generic data representation to model-specific behavioral variance, offering a conceptually interesting perspective for prompt-selection tasks.
Critical weaknesses
- The paper addresses a very niche setting: frozen remote-sensing segmenters with a fixed prompt bank. In practical applications, the overhead of building a coreset, managing screening pools, and annotating target images for prompt selection is often unjustified. Practitioners typically rely on direct fine-tuning (e.g., LoRA), visual prompting, or manual curation of robust prompt templates. The problem formulation feels over-engineered for a marginal pipeline step, questioning its real-world utility.
- While the paper claims reduced prompt regret, the actual performance improvements are marginal and, in key cases, statistically unsupported. As noted by the authors, the paired-gain bootstrap confidence intervals for SegEarth-OV and CAT-Seg include zero (e.g., pp for SegEarth-OV), indicating that the gains over random calibration or unweighted baselines are statistically inconclusive. A method requiring complex coreset construction should demonstrate robust, statistically significant superiority across all evaluated architectures.
- The technical components—such as k-medoids subset selection, feature representation of predictions, and cluster-weighted estimation are straightforward adaptations of existing active model selection and core-set literature (e.g., active learning and active model selection). Specializing these techniques into a dense segmentation global-IoU ratio offers limited conceptual or algorithmic advancement over existing active selection paradigms.
Questions for authors
- Given that the performance gains on mainstream models like SegEarth-OV and CAT-Seg are statistically inconclusive, why should this complex selection pipeline be preferred over simple random or manual selection?
- How does the method scale when the prompt bank size or the candidate class diversity increases significantly, especially considering its heavy reliance on the fixed screening pool ()?
6 comments
Response to Reviewer 6fFh (Part 2/5: Q2)
Q2. How does PromptCoreset scale with the prompt bank and target classes?
Prompt-bank size. We agree that the submitted evaluation at left open how PromptCoreset responds to a larger or smaller candidate bank. In the revision, we evaluated and across ten screening draws, with images per pool and an eight-mask annotation budget. The table reports mean Prompt Regret in IoU percentage points (pp) relative to the Oracle for each bank. Class Name uses no annotations. Random Calibration and PromptCoreset each use eight class masks.
| Segmenter | Method | |||
|---|---|---|---|---|
| SAM 3 | Class Name | 3.62 | 5.36 | 7.25 |
| Random Calibration | 4.04 | 5.75 | 7.81 | |
| PromptCoreset | 2.39 | 2.72 | 5.21 | |
| SegEarth-OV | Class Name | 3.70 | 5.21 | 5.49 |
| Random Calibration | 1.84 | 2.53 | 3.27 | |
| PromptCoreset | 0.78 | 0.94 | 1.50 | |
| CAT-Seg | Class Name | 1.83 | 2.77 | 4.41 |
| Random Calibration | 2.02 | 2.82 | 3.93 | |
| PromptCoreset | 1.33 | 1.48 | 3.25 | |
| Mean | PromptCoreset | 1.50 | 1.71 | 3.32 |
PromptCoreset achieves lower mean regret than both baselines at every tested bank size on all three segmenters. Increasing from 10 to 20 raises Oracle F1 from 63.23% to 64.45%, showing that the expanded bank contains stronger candidates. It also makes the selection decision harder by introducing more competing prompts. In this comparison, selected-prompt F1 falls from 61.63% to 61.39%, and mean regret increases from 1.71 to 3.32 pp. We use as the default because it achieves the highest selected-prompt F1 among the tested bank sizes at half the screening cost of . Regret measures the gap to each bank's own Oracle, so we use selected-prompt F1 to compare the segmentation quality achieved across bank sizes. Figure 4(c) and Appendix Table 7 report the full ablation, including selected-prompt F1 and Oracle F1.
Screening efficiency. The table reports prompt-conditioned screening calls per dataset–class unit and segmenter, and mean Prompt Regret across the three segmenters. The annotation budget stays fixed across settings.
| Setting | Screening calls | Mean regret (pp) | ||
|---|---|---|---|---|
| Smaller prompt bank | 5 | 32 | 160 | 1.50 |
| Smaller screening pool | 10 | 16 | 160 | 2.56 |
| Default | 10 | 32 | 320 | 1.71 |
| Larger prompt bank | 20 | 32 | 640 | 3.32 |
| Larger screening pool | 10 | 64 | 640 | 1.84 |
Screening requires prompt-conditioned calls. The number of pairwise mask-agreement features grows as , and each labeled mask can be reused to score all prompts. At , achieves the lowest mean regret among the tested pool sizes. Expanding to improves rare-target coverage on these draws, while doubling screening calls and raising overall mean regret from 1.71 to 1.84 pp. Figure 4(b) and Appendix Table 9 give the per-segmenter results.
Extending to new classes. Our evaluation spans 24 dataset–class units, 14 semantic classes, and 276 unique prompt strings across the three bank sizes. The class name is the only class-specific input the user needs to provide to generate a new prompt bank. An LLM can generate candidate prompts from that name, and PromptCoreset then applies the same calibration procedure with a small batch of labeled target masks. This lets users extend PromptCoreset to new categories without handcrafting prompts or retraining the segmenter. Appendix Table 14 shows the ten-prompt bank for building as an example.
Response to Reviewer 6fFh (Part 3/5: W1)
W1. Practical value of training-free prompt calibration. PromptCoreset targets deployments where a frozen open-vocabulary segmentation (OVS) model must map a class across a collection of target images. Recent advances such as SAM 3 [1] and RSKT-Seg [2] expand what these models can segment from text. This makes prompt choice a practical form of adaptation. If the model can already segment the target, selecting a suitable input can improve the deployed masks without updating the model.
-
Fine-tuning. LoRA and ViT-Adapter are established approaches for adapting pretrained models [3, 4]. They still require task-specific optimization, with associated training time and compute that can grow as more parameters are updated. Prior work on zero-shot models also documents a trade-off between improving target-domain accuracy through fine-tuning and retaining robustness under distribution shift [5]. PromptCoreset keeps the model frozen and adapts the text input, avoiding a task-specific training run. This is particularly useful when the pretrained model already has the required segmentation capability.
-
Visual prompting. Points and boxes are effective for specifying individual objects in SAM-style models [6]. In a manually prompted workflow, however, each new image requires spatial guidance, and point or box placement can affect which object or region the model returns. Work such as RSPrompter addresses this burden by learning to generate geometric prompts automatically for remote-sensing imagery [7]. For class-level mapping, a single text prompt provides a consistent class-level instruction across the entire collection. Once selected, it can be reused without placing new points or boxes on each image.
-
Manual text selection. A practitioner might naturally enter
buildingto map buildings, yet another phrasing can produce masks that better match the target dataset's annotation conventions. The same wording can also behave differently across frozen segmenters. Manual trial and error can therefore require running several candidates and inspecting their masks against labeled examples. PromptCoreset bases this decision on the target imagery, the frozen model's mask responses, and the dataset's annotation conventions. It uses candidate-mask behavior to choose which images to annotate, then uses pooled IoU against those annotations to select the deployment prompt. With eight class masks per task, this improves mean global IoU over the zero-label Class Name baseline by 2.64, 4.27, and 1.29 percentage points on SAM 3, SegEarth-OV, and CAT-Seg, respectively (Table 2).
Deployment workflow. The user starts with the target image collection and class name. PromptCoreset samples the screening pool, constructs the coreset, and identifies the eight images to annotate. The user can provide those masks in parallel in a single round. PromptCoreset confines multi-prompt inference to the screening pool, then applies the selected prompt across the target collection. On our benchmark, screening and deployment together use 89.5% fewer prompt-conditioned inference calls than exhaustive prompt search.
Applications. The selected prompt can be reused to map a target class across an aerial survey or to generate initial masks for human review and correction when building an annotated dataset. As OVS models extend to more classes and mapping tasks, this workflow gives practitioners a way to reuse those capabilities with a small, one-time calibration effort.
References
[1] Nicolas Carion et al. SAM 3: Segment Anything with Concepts. arXiv:2511.16719, 2025.
[2] Bingyu Li et al. Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing. AAAI, 2026.
[3] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
[4] Zhe Chen et al. Vision Transformer Adapter for Dense Predictions. ICLR, 2023.
[5] Mitchell Wortsman et al. Robust Fine-Tuning of Zero-Shot Models. CVPR, 2022.
[6] Alexander Kirillov et al. Segment Anything. ICCV, 2023.
[7] Keyan Chen et al. RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Response to Reviewer 6fFh (Part 1/5: Revision Notes & Q1)
Thank you for recognizing the value of training-free calibration and of using candidate-mask behavior to decide which images to annotate. Your review helped us sharpen two practical questions: when does prompt selection justify the annotation cost, and how sensitive is the method to the size of the prompt bank and screening pool? We revised the method and expanded the evaluation to address these questions.
Changes in the revised manuscript
We expanded the main comparison across ten screening draws. Table 2 averages results over ten screening draws, each generated by uniform sampling without replacement. The comparison covers six datasets, 24 dataset–class units, and three frozen segmenters, and includes the canonical class name, Random Calibration, Model Picker, Model Selector, and CODA. Evaluating multiple draws lets us assess how the results vary with pool composition. PromptCoreset achieves the lowest mean Prompt Regret among the compared methods on all three segmenters. It selects all eight images before receiving any labels, allowing their masks to be annotated in parallel in one round.
We revised image acquisition and prompt scoring. PromptCoreset now weights image acquisition by disagreement among candidate masks and pools true-positive and union counts directly across the labeled images to estimate global IoU. Tables 4 and 5 isolate the effects of these two changes under matched annotation budgets.
We added prompt-bank and screening-pool size ablations. Figure 4 reports results for 5, 10, and 20 candidate prompts and for screening pools of 16, 32, and 64 images, with the annotation budget fixed at eight class masks. The prompt-bank ablation also tracks Oracle F1 alongside regret, showing how a larger bank changes both the best achievable result and the difficulty of selecting a prompt from limited annotations.
Q1. Why prefer PromptCoreset to random or manual selection? In practical open-vocabulary segmentation, a practitioner might enter
buildingto map buildings. In one of our experiments,building from aboveproduced better segmentation on the target dataset. Both prompts describe the intended class, yet their wording offers little guidance about which one will yield masks that better match the dataset's annotation conventions.This observation motivated PromptCoreset. It connects the visual content of the target images, the frozen segmenter's responses to candidate prompts, and the dataset's annotation conventions. Candidate masks show how a particular segmenter interprets each prompt on the target imagery. PromptCoreset uses those responses to select eight images for annotation, then uses pooled IoU on the annotated masks to choose a deployment prompt. The same procedure applies across datasets and segmenters, while the selected prompt can vary with the imagery, the model, and the annotation conventions.
Comparison with Class Name and Random Calibration. Class Name represents a practical manual default that requires no target annotations. Random Calibration samples images uniformly for annotation and uses the same eight-mask budget as PromptCoreset. The following Table 2 results report mean Prompt Regret across ten reporting draws in IoU percentage points (pp). Lower is better.
With eight annotated masks, PromptCoreset reduces mean regret by 1.29–4.27 pp relative to Class Name. SegEarth-OV illustrates the benefit of choosing which images to annotate. Random Calibration lowers regret from 5.21 to 2.53 pp, and selecting the same number of images based on candidate-mask behavior lowers it further to 0.94 pp.
Paired comparison with Random Calibration. To address the uncertainty in the submitted results, we evaluated revised PromptCoreset against matched Random Calibration on ten randomly sampled screening pools. We computed paired 95% confidence intervals by resampling screening draws and dataset clusters. Positive reductions favor PromptCoreset.
All three paired 95% confidence intervals exclude zero. At the same eight-mask budget, revised PromptCoreset significantly reduces mean Prompt Regret relative to Random Calibration on all three evaluated segmenters, including SegEarth-OV and CAT-Seg, where the submitted results were inconclusive.