Beyond Prompt Prediction: Response Preference Learning for Automatic SAM Prompting

Automatic prompting is essential for extending the Segment Anything Model (SAM) to fully automated segmentation pipelines. Existing approaches typically formulate this problem as direct prompt prediction, where a model learns to generate prompts expected to produce accurate SAM segmentation results. However, SAM responses can vary substantially even for nearby prompts, making it challenging to determine desirable prompts without observing the responses they actually induce. This motivates us to investigate whether prompts can be determined after observing their induced SAM responses. We therefore decouple automatic prompting into candidate generation and response-based prompt determination, and find that even random exploration can discover candidate prompts with strong segmentation potential. This finding indicates that the central challenge of automatic prompting may not be generating a desirable prompt directly, but determining which candidate is preferable based on its realized SAM response. Based on this insight, we propose Response Preference Prompting (RePP), which performs automatic SAM prompting through response preference learning. Given a current prompt, RePP explores a set of candidate prompt updates and learns a preference model to determine which update to retain based on its induced SAM response. To improve exploration efficiency, we further introduce a lightweight multi-point generator that guides a more focused search over candidate updates than random sampling does. Across VOC, COCO, and NWPU, RePP exceeds the strongest compared automatic prompting methods by 4.2–10.7 AP points. Applying its learned response preference model to SAC-Prompt, Matcher, and PPD further improves AP by 15.4, 5.3, and 4.0 points, respectively.