acceptodds
Under review as a conference paper at ICLR 2027

PromptCoreset: Behavior-Aware Text-Prompt Selection for Remote-Sensing Segmentation

12 views

Abstract

Text-conditioned segmentation is sensitive to prompt wording, but exhaustive prompt selection requires dense annotations and repeated inference over the target dataset. We study calibration-set design: which target images should be labeled to select a prompt under a limited annotation budget? We introduce PromptCoreset, a training-free method that represents unlabeled images by foreground extent and pairwise agreement of candidate masks. Deterministic \(k\)-medoids selects representative images for annotation, and cluster-weighted true-positive and union counts estimate global-IoU utility. Unlike purely visual sampling, the chosen images represent variation in foreground extent and mask agreement among prompts. The method selects one deployment prompt without changing the segmentation model or prompt bank. Across six remote-sensing datasets and three frozen segmentation systems, PromptCoreset improves mean segmentation IoU and F1 and lowers mean Prompt Regret relative to random calibration on each system using only a small labeled calibration set. Prompt-conditioned calls decrease by approximately 90% relative to exhaustive search. These results show that behavior-aware calibration can make text-prompt selection more annotation- and compute-efficient without retraining the segmentation model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Promptable Segmentation, Remote Sensing, Prompt Selection, Vision Foundation Models, Compute-Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6fFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of text-prompt selection for text-conditioned remote sensing segmentation models (e.g., SAM 3, SegEarth-OV, CAT-Seg) under a limited annotation budget. The authors introduce PromptCoreset, a training-free calibration method that selects representative unlabeled target images based on a "prompt-behavior signature" combining foreground extent and pairwise candidate mask agreement. Using deterministic k-medoids, a small calibration subset is annotated to estimate cluster-weighted global IoU and pick a single reusable prompt for deployment. Evaluated across six remote-sensing datasets and three frozen segmentation systems (24 dataset-class units), PromptCoreset successfully reduces prompt regret and lowers prompt-conditioned inference calls by approximately 89.5% relative to exhaustive search.

Critical strengths
  1. A major strength of the paper is its entirely training-free pipeline, which avoids the heavy computational overhead of fine-tuning backbone segmentation models. By leveraging candidate-mask behavior, PromptCoreset successfully reduces prompt-conditioned inference calls by approximately 89.5% relative to exhaustive search. This provides a highly efficient, plug-and-play solution for resource-constrained remote-sensing deployments.
  2. Instead of relying solely on naive visual diversity or standard image coreset methods, the authors uniquely formulate prompt selection as a calibration-set design problem driven by "prompt-behavior signatures" (combining foreground extent and pairwise mask agreement). This shifts the focus from generic data representation to model-specific behavioral variance, offering a conceptually interesting perspective for prompt-selection tasks.
Critical weaknesses
  1. The paper addresses a very niche setting: frozen remote-sensing segmenters with a fixed prompt bank. In practical applications, the overhead of building a coreset, managing screening pools, and annotating target images for prompt selection is often unjustified. Practitioners typically rely on direct fine-tuning (e.g., LoRA), visual prompting, or manual curation of robust prompt templates. The problem formulation feels over-engineered for a marginal pipeline step, questioning its real-world utility.
  2. While the paper claims reduced prompt regret, the actual performance improvements are marginal and, in key cases, statistically unsupported. As noted by the authors, the paired-gain bootstrap confidence intervals for SegEarth-OV and CAT-Seg include zero (e.g., [−0.96,1.79][-0.96, 1.79] pp for SegEarth-OV), indicating that the gains over random calibration or unweighted baselines are statistically inconclusive. A method requiring complex coreset construction should demonstrate robust, statistically significant superiority across all evaluated architectures.
  3. The technical components—such as k-medoids subset selection, feature representation of predictions, and cluster-weighted estimation are straightforward adaptations of existing active model selection and core-set literature (e.g., active learning and active model selection). Specializing these techniques into a dense segmentation global-IoU ratio offers limited conceptual or algorithmic advancement over existing active selection paradigms.
Questions for authors
  1. Given that the performance gains on mainstream models like SegEarth-OV and CAT-Seg are statistically inconclusive, why should this complex selection pipeline be preferred over simple random or manual selection?
  2. How does the method scale when the prompt bank size MM or the candidate class diversity increases significantly, especially considering its heavy reliance on the fixed screening pool (K=32K=32)?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6fFh (Part 1/5: Revision Notes & Q1)

Thank you for recognizing the value of training-free calibration and of using candidate-mask behavior to decide which images to annotate. Your review helped us sharpen two practical questions: when does prompt selection justify the annotation cost, and how sensitive is the method to the size of the prompt bank and screening pool? We revised the method and expanded the evaluation to address these questions.

Changes in the revised manuscript

  1. We expanded the main comparison across ten screening draws. Table 2 averages results over ten screening draws, each generated by uniform sampling without replacement. The comparison covers six datasets, 24 dataset–class units, and three frozen segmenters, and includes the canonical class name, Random Calibration, Model Picker, Model Selector, and CODA. Evaluating multiple draws lets us assess how the results vary with pool composition. PromptCoreset achieves the lowest mean Prompt Regret among the compared methods on all three segmenters. It selects all eight images before receiving any labels, allowing their masks to be annotated in parallel in one round.

  2. We revised image acquisition and prompt scoring. PromptCoreset now weights image acquisition by disagreement among candidate masks and pools true-positive and union counts directly across the labeled images to estimate global IoU. Tables 4 and 5 isolate the effects of these two changes under matched annotation budgets.

  3. We added prompt-bank and screening-pool size ablations. Figure 4 reports results for 5, 10, and 20 candidate prompts and for screening pools of 16, 32, and 64 images, with the annotation budget fixed at eight class masks. The prompt-bank ablation also tracks Oracle F1 alongside regret, showing how a larger bank changes both the best achievable result and the difficulty of selecting a prompt from limited annotations.

Q1. Why prefer PromptCoreset to random or manual selection? In practical open-vocabulary segmentation, a practitioner might enter building to map buildings. In one of our experiments, building from above produced better segmentation on the target dataset. Both prompts describe the intended class, yet their wording offers little guidance about which one will yield masks that better match the dataset's annotation conventions.

This observation motivated PromptCoreset. It connects the visual content of the target images, the frozen segmenter's responses to candidate prompts, and the dataset's annotation conventions. Candidate masks show how a particular segmenter interprets each prompt on the target imagery. PromptCoreset uses those responses to select eight images for annotation, then uses pooled IoU on the annotated masks to choose a deployment prompt. The same procedure applies across datasets and segmenters, while the selected prompt can vary with the imagery, the model, and the annotation conventions.

Comparison with Class Name and Random Calibration. Class Name represents a practical manual default that requires no target annotations. Random Calibration samples images uniformly for annotation and uses the same eight-mask budget as PromptCoreset. The following Table 2 results report mean Prompt Regret across ten reporting draws in IoU percentage points (pp). Lower is better.

SegmenterClass NameRandom CalibrationPromptCoresetGain vs. Class Name
SAM 35.365.752.722.64
SegEarth-OV5.212.530.944.27
CAT-Seg2.772.821.481.29

With eight annotated masks, PromptCoreset reduces mean regret by 1.29–4.27 pp relative to Class Name. SegEarth-OV illustrates the benefit of choosing which images to annotate. Random Calibration lowers regret from 5.21 to 2.53 pp, and selecting the same number of images based on candidate-mask behavior lowers it further to 0.94 pp.

Paired comparison with Random Calibration. To address the uncertainty in the submitted results, we evaluated revised PromptCoreset against matched Random Calibration on ten randomly sampled screening pools. We computed paired 95% confidence intervals by resampling screening draws and dataset clusters. Positive reductions favor PromptCoreset.

SegmenterGain vs. RandomPaired [95% CI]
SAM 33.03[1.41, 4.93]
SegEarth-OV1.59[0.93, 2.23]
CAT-Seg1.33[0.61, 2.12]

All three paired 95% confidence intervals exclude zero. At the same eight-mask budget, revised PromptCoreset significantly reduces mean Prompt Regret relative to Random Calibration on all three evaluated segmenters, including SegEarth-OV and CAT-Seg, where the submitted results were inconclusive.

AuthorsOpenReview ↗

Response to Reviewer 6fFh (Part 2/5: Q2)

Q2. How does PromptCoreset scale with the prompt bank and target classes?

Prompt-bank size. We agree that the submitted evaluation at M=10M = 10 left open how PromptCoreset responds to a larger or smaller candidate bank. In the revision, we evaluated M=5,10,M = 5, 10, and 2020 across ten screening draws, with K=32K = 32 images per pool and an eight-mask annotation budget. The table reports mean Prompt Regret in IoU percentage points (pp) relative to the Oracle for each bank. Class Name uses no annotations. Random Calibration and PromptCoreset each use eight class masks.

SegmenterMethodM=5M = 5M=10M = 10M=20M = 20
SAM 3Class Name3.625.367.25
Random Calibration4.045.757.81
PromptCoreset2.392.725.21
SegEarth-OVClass Name3.705.215.49
Random Calibration1.842.533.27
PromptCoreset0.780.941.50
CAT-SegClass Name1.832.774.41
Random Calibration2.022.823.93
PromptCoreset1.331.483.25
MeanPromptCoreset1.501.713.32

PromptCoreset achieves lower mean regret than both baselines at every tested bank size on all three segmenters. Increasing MM from 10 to 20 raises Oracle F1 from 63.23% to 64.45%, showing that the expanded bank contains stronger candidates. It also makes the selection decision harder by introducing more competing prompts. In this comparison, selected-prompt F1 falls from 61.63% to 61.39%, and mean regret increases from 1.71 to 3.32 pp. We use M=10M = 10 as the default because it achieves the highest selected-prompt F1 among the tested bank sizes at half the screening cost of M=20M = 20. Regret measures the gap to each bank's own Oracle, so we use selected-prompt F1 to compare the segmentation quality achieved across bank sizes. Figure 4(c) and Appendix Table 7 report the full ablation, including selected-prompt F1 and Oracle F1.

Screening efficiency. The table reports prompt-conditioned screening calls per dataset–class unit and segmenter, and mean Prompt Regret across the three segmenters. The annotation budget stays fixed across settings.

SettingMMKKScreening callsMean regret (pp)
Smaller prompt bank5321601.50
Smaller screening pool10161602.56
Default10323201.71
Larger prompt bank20326403.32
Larger screening pool10646401.84

Screening requires M×KM \times K prompt-conditioned calls. The number of pairwise mask-agreement features grows as M(M−1)/2M(M-1)/2, and each labeled mask can be reused to score all MM prompts. At M=10M = 10, K=32K = 32 achieves the lowest mean regret among the tested pool sizes. Expanding to K=64K = 64 improves rare-target coverage on these draws, while doubling screening calls and raising overall mean regret from 1.71 to 1.84 pp. Figure 4(b) and Appendix Table 9 give the per-segmenter results.

Extending to new classes. Our evaluation spans 24 dataset–class units, 14 semantic classes, and 276 unique prompt strings across the three bank sizes. The class name is the only class-specific input the user needs to provide to generate a new prompt bank. An LLM can generate candidate prompts from that name, and PromptCoreset then applies the same calibration procedure with a small batch of labeled target masks. This lets users extend PromptCoreset to new categories without handcrafting prompts or retraining the segmenter. Appendix Table 14 shows the ten-prompt bank for building as an example.

AuthorsOpenReview ↗

Response to Reviewer 6fFh (Part 3/5: W1)

W1. Practical value of training-free prompt calibration. PromptCoreset targets deployments where a frozen open-vocabulary segmentation (OVS) model must map a class across a collection of target images. Recent advances such as SAM 3 [1] and RSKT-Seg [2] expand what these models can segment from text. This makes prompt choice a practical form of adaptation. If the model can already segment the target, selecting a suitable input can improve the deployed masks without updating the model.

  1. Fine-tuning. LoRA and ViT-Adapter are established approaches for adapting pretrained models [3, 4]. They still require task-specific optimization, with associated training time and compute that can grow as more parameters are updated. Prior work on zero-shot models also documents a trade-off between improving target-domain accuracy through fine-tuning and retaining robustness under distribution shift [5]. PromptCoreset keeps the model frozen and adapts the text input, avoiding a task-specific training run. This is particularly useful when the pretrained model already has the required segmentation capability.

  2. Visual prompting. Points and boxes are effective for specifying individual objects in SAM-style models [6]. In a manually prompted workflow, however, each new image requires spatial guidance, and point or box placement can affect which object or region the model returns. Work such as RSPrompter addresses this burden by learning to generate geometric prompts automatically for remote-sensing imagery [7]. For class-level mapping, a single text prompt provides a consistent class-level instruction across the entire collection. Once selected, it can be reused without placing new points or boxes on each image.

  3. Manual text selection. A practitioner might naturally enter building to map buildings, yet another phrasing can produce masks that better match the target dataset's annotation conventions. The same wording can also behave differently across frozen segmenters. Manual trial and error can therefore require running several candidates and inspecting their masks against labeled examples. PromptCoreset bases this decision on the target imagery, the frozen model's mask responses, and the dataset's annotation conventions. It uses candidate-mask behavior to choose which images to annotate, then uses pooled IoU against those annotations to select the deployment prompt. With eight class masks per task, this improves mean global IoU over the zero-label Class Name baseline by 2.64, 4.27, and 1.29 percentage points on SAM 3, SegEarth-OV, and CAT-Seg, respectively (Table 2).

Deployment workflow. The user starts with the target image collection and class name. PromptCoreset samples the screening pool, constructs the coreset, and identifies the eight images to annotate. The user can provide those masks in parallel in a single round. PromptCoreset confines multi-prompt inference to the screening pool, then applies the selected prompt across the target collection. On our benchmark, screening and deployment together use 89.5% fewer prompt-conditioned inference calls than exhaustive prompt search.

Applications. The selected prompt can be reused to map a target class across an aerial survey or to generate initial masks for human review and correction when building an annotated dataset. As OVS models extend to more classes and mapping tasks, this workflow gives practitioners a way to reuse those capabilities with a small, one-time calibration effort.

References

[1] Nicolas Carion et al. SAM 3: Segment Anything with Concepts. arXiv:2511.16719, 2025.

[2] Bingyu Li et al. Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing. AAAI, 2026.

[3] Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.

[4] Zhe Chen et al. Vision Transformer Adapter for Dense Predictions. ICLR, 2023.

[5] Mitchell Wortsman et al. Robust Fine-Tuning of Zero-Shot Models. CVPR, 2022.

[6] Alexander Kirillov et al. Segment Anything. ICCV, 2023.

[7] Keyan Chen et al. RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model. IEEE Transactions on Geoscience and Remote Sensing, 2024.

Reviewer_UNwSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies selection of one text prompt from a fixed bank for a frozen remote-sensing segmentation model when only a small number of target masks can be annotated. PromptCoreset first runs all prompts on a K=32 unlabeled screening pool, represents each image by per-prompt foreground areas and pairwise mask IoUs, chooses B medoids with deterministic k-medoids, and estimates each prompt's population global IoU by cluster-weighted sums of TP and union counts on the labeled medoids. On six datasets (24 dataset-class units) and SAM 3, SegEarth-OV, and CAT-Seg, the method reports lower mean regret than random calibration at B=8 and an 89.5% reduction in prompt-conditioned calls. The problem is useful and the workflow is clearly specified, but the core algorithm combines standard representation, medoid selection, and weighted ratio estimation; the evidence for generality is limited by the fixed 32-image screening pool, six dataset clusters, transductive evaluation, non-uniform evaluation splits, and uncertainty of the smaller gains.

Critical strengths
  1. The paper formulates a practically relevant decision problem, selecting among fixed prompts for a frozen dense predictor under annotation and inference budgets, and uses global IoU regret rather than an unrelated per-image surrogate.

  2. The method is simple and reproducible at the conceptual level. The signature, block normalization, deterministic medoid selection, cluster assignment, weighted TP/union estimator, tie rule, and frozen-before-label-access protocol are described explicitly.

Critical weaknesses
  1. This paper is interesting, but overall it looks like a simple engineering optimization.

  2. The metrics for different datasets are not reported separately, making it difficult to perform a rough comparison with other studies.

  3. The reported iSAID storage-tank case shows that all ten prompt utilities can be zero in screening while full-population regret is 16.48 pp; this is a direct failure of the proposed selection signal. There is no systematic curve over K, alternative pool construction, or adaptive mechanism to detect and repair missing behavior modes.

  4. At B=8 on CAT-Seg, unweighted PromptCoreset has 1.09 pp regret versus 1.24 pp for the full method, and Near-Oracle@1 is 75.0% versus 70.8%. Table 5 likewise shows weighting improves SAM 3 and SegEarth-OV but increases CAT-Seg regret. This undermines any general claim that cluster weighting is a reliable contribution; the paper needs a principled criterion for when weighting should be used.

Questions for authors

please see Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer UNwS (Part 1/4: W1)

Thank you for the thoughtful review and for recognizing the practical value of prompt selection under annotation and inference budgets. After submitting the original manuscript, we expanded our evaluation across more screening pools. In these follow-up experiments, we found that cluster-size weighting did not consistently improve prompt selection across datasets. This finding prompted us to move weighting from prompt scoring to image acquisition. The revised method weights acquisition by disagreement among candidate masks and pools true-positive and union counts directly when scoring prompts. We appreciate that your comment in W4 identifies the same issue. With the annotation budget and scoring rule held fixed, the revised acquisition rule lowers mean Prompt Regret relative to uniform medoids on all three segmenters (Table 4).

Changes in the revised manuscript

  1. Weighting and scoring. The submitted method used uniform kk-medoids for acquisition and cluster-size weighting for scoring. The revision weights the kk-medoids objective by mask disagreement and directly pools true-positive and union counts across labeled images. Tables 4 and 5 evaluate these changes separately.
  2. Matched-budget comparisons and dataset-level results. Table 2 adds Model Picker, Model Selector, and CODA. Table 3 reports results by dataset, and Appendix Table 16 breaks down the main comparison by dataset and segmenter.
  3. Screening-pool sensitivity. Across ten screening draws, we compared nested pools of 16, 32, and 64 images at the same eight-mask budget. Screening images were sampled uniformly without replacement within each draw. Figure 4(b) and Appendix Table 9 report the results. We also revisited the iSAID storage-tank case to examine how coverage and regret vary with pool size.

W1. Methodological contribution. To our knowledge, we are the first to frame deployment prompt selection for remote-sensing open-vocabulary segmentation as a single-round annotation design problem. PromptCoreset selects the full batch of eight images before any labels are available, so the masks can be annotated in parallel. It then uses the resulting masks to choose one prompt for the entire target dataset.

Model Picker [1] uses annotation feedback to update its query decisions as samples arrive. Model Selector [2] and CODA [3] use that feedback to choose the next sample from an unlabeled pool. Prior work on few-shot remote-sensing segmentation [4] studies learnable prompts. PromptCoreset uses the masks produced by candidate prompts to decide which images are worth labeling. It builds on feature representation and kk-medoids to keep calibration training-free and tailors the representation, acquisition objective, and scoring rule to the deployment decision.

  1. Prompt-behavior signature. For MM candidate prompts, we concatenate MM foreground-area features and M(M−1)/2M(M-1)/2 pairwise mask-IoU features. Area captures how much each prompt segments, while pairwise IoU distinguishes equally large masks at different locations. Together, these features let the selector cover distinct patterns of prompt behavior.

  2. Disagreement-weighted acquisition. We weight the kk-medoids objective by pixel-level disagreement among candidate masks. This gives higher priority to images whose annotations can help distinguish prompts while preserving coverage of diverse behaviors. With the annotation budget and scorer fixed, disagreement weighting reduces mean regret by 0.81 pp relative to uniform medoids (95% CI [0.42, 1.25]). Covering diverse prompt behaviors also helps. PromptCoreset reduces mean regret by 0.57 pp relative to Top-XOR, which selects images by disagreement alone (95% CI [0.09, 1.34], Table 4).

  3. Pooled-IoU prompt estimation. We sum true-positive and union counts across the labeled images before taking their ratio, matching the aggregation used to evaluate the deployment prompt. With the same eight labeled images, pooled IoU yields the lowest mean regret on all three segmenters. The mean reductions are 0.63 pp relative to cluster-size-weighted pooling (95% CI [0.27, 0.96]) and 1.51 and 2.05 pp relative to mean and median per-image IoU, respectively (Table 5).

Across the ten reporting draws in Table 2, PromptCoreset achieves the lowest mean Prompt Regret among the compared methods on all three segmenters.

References

[1] Mohammad Reza Karimi et al. Online Active Model Selection for Pre-trained Classifiers. AISTATS, 2021.

[2] Patrik Okanovic et al. All Models Are Wrong, Some Are Useful: Model Selection with Limited Labels. AISTATS, 2025.

[3] Justin Kay et al. Consensus-Driven Active Model Selection. ICCV, 2025.

[4] Steve Andreas Immanuel and Hagai Raja Sinulingga. Learnable Prompt for Few-Shot Semantic Segmentation in Remote Sensing Domain. CVPR Workshops, 2024.

AuthorsOpenReview ↗

Response to Reviewer UNwS (Part 2/4: W2)

W2. Dataset-level results. We added Table 3 to report results separately for all six datasets. The table below summarizes mean Prompt Regret in IoU percentage points (lower is better), averaged over the evaluated classes and three segmenters.

DatasetRandomModel PickerModel SelectorCODAPromptCoreset
iSAID5.555.273.433.423.29
LoveDA2.742.331.441.701.27
Potsdam5.424.443.243.931.99
UAVid1.671.241.081.111.01
UDD54.473.402.002.351.55
OpenEarthMap2.332.001.341.401.17

PromptCoreset achieves the lowest mean regret among the methods shown on all six datasets. Its largest margin over the next-best method is 1.25 pp on Potsdam. Table 3 also reports PromptCoreset's IoU, F1, and Near-Oracle rate for each dataset. Appendix Table 16 gives the full dataset-by-segmenter breakdown.

Evaluation splits and classes (Appendix Table 12). For comparison with other studies, we list the evaluation population and target classes for each dataset. Tile counts refer to valid 512 × 512 evaluation tiles. Numbers in parentheses are the class label IDs.

DatasetEvaluation populationTilesClasses (label IDs)
iSAIDOfficial labeled validation split6,363plane (14), ship (1), storage tank (2), small vehicle (9)
LoveDALocally available labeled images10,088building (2), road (3), water (4), agriculture (7)
PotsdamRepository validation split (14 source tiles)7,406building (2), impervious surface (1), tree (4), car (5)
UAVidLocally available labeled images10,800building (1), road (2), tree (4), static car (3)
UDD5Official labeled validation split1,824building (1), road (2), vegetation (0), vehicle (3)
OpenEarthMapPaired official validation subset with available RGB images1,515building (8), road (4), water (6), agriculture (7)
Total37,99624 dataset–class units
AuthorsOpenReview ↗

Response to Reviewer UNwS (Part 3/4: W3)

W3. Screening-pool sensitivity and rare-target coverage. Thank you for highlighting the iSAID storage-tank failure. We evaluated how often the screening pool misses this class, how regret changes with pool size, and whether selective expansion can recover coverage while preserving a single round of annotation.

Pool-size sensitivity. We compared nested pools of 16, 32, and 64 images on the same ten reporting draws, with the annotation budget fixed at eight class masks. We measured positive-tile coverage using ground truth after selection. Regret is averaged across the three segmenters and reported in IoU percentage points (pp).

Pool sizeOverall regretStorage-tank regretZero-positive storage-tank pools
K=16K = 162.566.846/10
K=32K = 321.717.753/10
K=64K = 641.845.590/10

At K=32K = 32, iSAID storage tanks were the only one of the 24 dataset–class units with any zero-positive pool across these draws. On the missed draws, fixed-pool PromptCoreset and Random Calibration both selected the class-name prompt, reproducing the 16.48 pp regret on SAM 3. Among pools containing at least one positive storage-tank tile, PromptCoreset achieved lower mean regret than matched Random Calibration at all three pool sizes: 4.52 vs. 6.47 pp at K=16K = 16, 7.48 vs. 8.74 pp at K=32K = 32, and 5.59 vs. 8.42 pp at K=64K = 64. At K=16K = 16, each of the four positive pools contained just one positive tile. Figure 4(b) and Appendix Table 9 report the full pool-size ablation.

Exploratory adaptive expansion. We added this analysis to Appendix D (Table 17). The policy starts at K=32K = 32 and expands to the nested K=64K = 64 pool if the mean pairwise mask disagreement on the eight provisional medoids is below 1%. The trigger uses only predicted masks. The final eight images are selected before annotation, preserving a single annotation round. The missed subset comprises the three zero-positive storage-tank draws at K=32K = 32 evaluated across three segmenters.

Screening policyOverall regret (pp)Missed-subset regret (pp)Coverage recoveredScreening calls
Fixed K=32K = 321.718.380/9230,400
Adaptive K=32K = 32 to 64641.674.186/9240,320
Fixed K=64K = 641.843.319/9460,800

The adaptive policy expanded 31 of 720 model–unit–draw combinations. It restored coverage in 6 of 9 missed cases and reduced their mean regret from 8.38 to 4.18 pp, with 4.3% more screening calls than fixed K=32K = 32. On SAM 3, it reduced regret from 16.48 to 0 pp on two of the three missed draws. The three CAT-Seg misses did not trigger expansion.

Expanding every pool doubled the screening calls and raised overall mean regret from 1.71 to 1.84 pp, so we retain K=32K = 32 as the default. This retrospective experiment shows that selective expansion can recover some missed cases at modest extra cost. Validating the trigger on fresh pools and deciding reliably when to expand before any labels are available remain open questions for single-round annotation.

Reviewer_5UrcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PromptCoreset, a training-free method that represents unlabeled images by foreground extent and pairwise agreement of candidate masks. The method is evaluated on six remote-sensing datasets and three frozen segmentation models.t bank. Across six remote-sensing datasets and three frozen segmentation systems, PromptCoreset improves mean segmentation IoU and F1 and lowers mean Prompt Regret relative to random calibration on each system using only a small labeled calibration set.

Critical strengths

1.The paper proposes a practically meaningful and well-defined prompt-calibration problem. The paper cleanly formulates this as selecting representative target images for annotation under a limited budget and evaluates the resulting decision through Prompt Regret. 2.The method is simple, training-free, and yields cost-quality trade-off. Across three frozen segmentation models, the method consistently improves prompt-ranking fidelity.

Critical weaknesses

1.The experiments are not yet strong enough to establish that PromptCoreset provides a robust advantage across models and datasets. Although It achieves lower mean Prompt Regret than Random Calibration on all three models, the paired 95% confidence interval excludes zero only for SAM 3, while the gains on SegEarth-OV and CAT-Seg remain statistically inconclusive. 2.The comparison does not include the directly relevant active model-selection methods discussed in the paper itself. Model Picker, Model Selector, and CODA are closely related approaches that use limited labels and unlabeled candidate predictions to select among frozen predictors.

Questions for authors

1.The authors may provide matched-budget results for one or more adapted active model-selection baselines using the same screening pool, annotation budget, and prompt-conditioned inference budget. 2. The iSAID storage-tank example shows that a screening pool can completely miss behavior that is important in the full target population. How sensitive is PromptCoreset to the particular choice of the K=32 screening pool?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5Urc (Part 1/2: Q1 W2)

Thank you for highlighting the need for matched-budget comparisons with active model-selection methods and for asking how sensitive PromptCoreset is to the screening pool.

What changed in PromptCoreset. In our follow-up experiments, the cluster-size weighting used in the submitted version did not consistently improve prompt selection. We made two changes while keeping the prompt-behavior representation and annotation budget fixed:

  1. Image acquisition. The submitted method used uniform kk-medoids. The revised method weights the kk-medoids objective by pairwise mask disagreement. This preserves coverage of diverse prompt behaviors while giving more weight to images on which the candidate prompts produce different masks.
  2. Prompt scoring. The submitted method weighted each medoid's pixel counts by its cluster size. The revised method pools true-positive and union counts directly across the labeled images to estimate global IoU. We use this configuration across all datasets and segmenters, with no separate decision about whether to enable cluster-size weighting.

Batch acquisition. PromptCoreset selects all BB images jointly from unlabeled predictions before requesting any annotations. Sequential active selectors can use labels from earlier queries to decide which image to query next. PromptCoreset's selected images can therefore be annotated in parallel. Under the matched annotation budget in Table 2, revised PromptCoreset also achieves lower mean Prompt Regret than Model Picker, Model Selector, and CODA on all three segmenters across ten random sampling runs.

Q1. Matched-budget active model-selection baselines. We added Model Picker, Model Selector, and CODA to Table 2. On each of the ten random draws, all methods used the same K=32K=32 screening images, the same cached predictions for all ten candidate prompts, and a budget of at most B=8B=8 class masks. The active baselines retained their original acquisition rules, native scoring rules, and default hyperparameters throughout image querying and final prompt selection. We also updated Table 2 in the main text to include the matched-budget results for Model Picker, Model Selector, and CODA. Please see the revised Table 2 for the full main comparison.

SegmenterRandom CalibrationModel PickerModel SelectorCODAUniform MedoidsOriginal PCRevised PC
SAM 35.754.773.303.733.573.332.72
SegEarth-OV2.532.161.171.151.992.280.94
CAT-Seg2.822.411.802.072.022.161.48

Values are mean Prompt Regret in IoU percentage points (pp), with lower values indicating better prompt selection. Revised PromptCoreset achieves the lowest mean regret among the compared methods on all three segmenters. Uniform Medoids uses the same pooled-IoU scorer as revised PromptCoreset, isolating the effect of disagreement weighting during acquisition. The mean gains are 0.86, 1.05, and 0.53 pp for SAM 3, SegEarth-OV, and CAT-Seg, respectively (Table 4). The CAT-Seg confidence interval includes zero.

AuthorsOpenReview ↗

Response to Reviewer 5Urc (Part 2/2: Q2 W1)

Q2. Sensitivity to the screening pool. We repeated the matched comparison across the same ten reporting draws as Table 2 to assess whether PromptCoreset's advantage persists as pool composition varies. Each K=32K = 32 pool was generated by uniformly sampling 32 images per dataset without replacement. Within each draw, K=16K = 16 uses the first 16 images of that pool, and K=64K = 64 appends 32 images sampled without replacement from the remaining tiles. The annotation budget stays fixed at eight class masks. Each draw-level result averages Prompt Regret over the same 24 dataset–class units.

The PC columns report mean regret [min, max] across draws. Random, Wins, and Gain refer to K=32K = 32. Gain is the paired regret reduction relative to Random Calibration, with a 95% bootstrap CI obtained by resampling draws and dataset clusters within the reporting set. Regret and gains are in IoU percentage points (pp).

SegmenterPC (K=16K=16)Original PC (K=32K=32)PC (K=32K=32)PC (K=64K=64)RandomWinsGain [95% CI]
SAM 34.24 [2.36, 6.68]3.33 [2.00, 4.98]2.72 [1.66, 4.28]2.83 [2.04, 3.79]5.7510/103.03 [1.41, 4.93]
SegEarth-OV1.45 [0.85, 2.32]2.28 [1.06, 3.77]0.94 [0.60, 1.41]1.00 [0.49, 1.71]2.5310/101.59 [0.93, 2.23]
CAT-Seg1.98 [0.78, 2.88]2.16 [1.05, 3.12]1.48 [0.80, 2.21]1.68 [1.17, 2.37]2.8210/101.33 [0.61, 2.12]

At K=32K = 32, revised PromptCoreset achieves the lowest mean regret among the methods compared in Table 2 on all three segmenters. Its advantage over matched Random Calibration holds on every reported draw for each segmenter, and all three paired intervals exclude zero within this reporting set. Across the three segmenters, K=32K = 32 gives the lowest mean regret among the tested pool sizes (2.56, 1.71, and 1.84 pp at K=16,32,K = 16, 32, and 6464, respectively). Appendix Table 9 of the revised manuscript provides the full per-segmenter results for the screening-pool size ablation.

PromptCoreset's gains on a rare target. We examined the iSAID storage-tank class using ground truth after prompt selection. The first two numerical columns summarize coverage across all ten reporting draws. The regret columns compare PromptCoreset and Random Calibration on the same pools containing at least one positive tile, averaging over those draws and all three segmenters.

Pool sizeZero-positive poolsPositive tiles, mean [range]PC regret (pp)Random regret (pp)
K=16K = 166/100.4 [0, 1]4.526.47
K=32K = 323/101.2 [0, 3]7.488.74
K=64K = 640/102.4 [1, 4]5.598.42

PromptCoreset achieves lower mean storage-tank regret than Random Calibration at all three pool sizes when positive tiles are available to query. The gain is visible even at K=16K = 16, where each of the four positive pools contains just one positive tile and PromptCoreset lowers mean regret from 6.47 to 4.52 pp under the same eight-mask budget. Expanding from K=32K = 32 to K=64K = 64 restores coverage in all three previously zero-positive storage-tank pools, and PromptCoreset's regret decreases on each of those draws. Across all ten draws, its mean storage-tank regret falls from 7.75 to 5.59 pp. This case shows that behavior-aware acquisition can improve prompt selection over random calibration even when positive examples are scarce in the screening pool.

Dear Reviewer 5Urc,

We have posted our detailed response and a summary of revisions addressing your comments. We understand you may be busy and would appreciate any brief feedback on remaining concerns. Thank you for your time.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes from the Original Submission

We thank all three reviewers for their constructive feedback. The revision changes where and how PromptCoreset applies weighting, expands the matched-budget comparisons, and adds experiments on screening-pool and prompt-bank sensitivity.

  1. We moved weighting from prompt scoring to image acquisition. After submitting the original manuscript, we evaluated more screening pools and found that cluster-size weighting did not consistently improve prompt selection. Reviewer UNwS raised the same concern in W4. In the revision, we weight the kk-medoids acquisition objective by disagreement among candidate masks and score each prompt by directly pooling intersection and union counts across annotated images. With the annotation budget and scoring rule held fixed, disagreement weighting lowered mean Prompt Regret relative to uniform medoids on all three segmenters. Tables 4 and 5 evaluate the acquisition and scoring changes separately. PromptCoreset continues to select all eight images before receiving any labels, allowing the masks to be annotated in parallel in one round.

  2. We expanded the matched-budget comparisons and statistical analysis. Table 2 adds Model Picker, Model Selector, and CODA and reports averages over ten screening draws. Revised PromptCoreset achieves the lowest mean Prompt Regret among the compared methods on all three segmenters. In the paired comparisons against Random Calibration, Image k-Center, and Model Picker, all nine 95% confidence intervals for regret reduction lie above zero, compared with five of nine for the submitted version under the same protocol. Table 3 reports results by dataset, and Appendix Table 16 provides the dataset-by-segmenter breakdown.

  3. We tested sensitivity to pool composition, pool size, and prompt-bank size. Across ten screening draws, we evaluated nested screening pools of K=16K=16, 3232, and 6464 images and prompt banks of M=5M=5, 1010, and 2020 candidates under a fixed eight-mask budget. At K=32K=32, revised PromptCoreset achieved lower regret than matched Random Calibration on every reported draw for each segmenter. Figure 4 and Appendix Tables 7 and 9 report the size ablations. We also revisited the iSAID storage-tank case and report an exploratory adaptive pool expansion experiment in Appendix Table 17.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.