PromptCoreset: Behavior-Aware Text-Prompt Selection for Remote-Sensing Segmentation

Text-conditioned segmentation is sensitive to prompt wording, but exhaustive prompt selection requires dense annotations and repeated inference over the target dataset. We study calibration-set design: which target images should be labeled to select a prompt under a limited annotation budget? We introduce PromptCoreset, a training-free method that represents unlabeled images by foreground extent and pairwise agreement of candidate masks. Deterministic \(k\)-medoids selects representative images for annotation, and cluster-weighted true-positive and union counts estimate global-IoU utility. Unlike purely visual sampling, the chosen images represent variation in foreground extent and mask agreement among prompts. The method selects one deployment prompt without changing the segmentation model or prompt bank. Across six remote-sensing datasets and three frozen segmentation systems, PromptCoreset improves mean segmentation IoU and F1 and lowers mean Prompt Regret relative to random calibration on each system using only a small labeled calibration set. Prompt-conditioned calls decrease by approximately 90% relative to exhaustive search. These results show that behavior-aware calibration can make text-prompt selection more annotation- and compute-efficient without retraining the segmentation model.