acceptodds
Under review as a conference paper at ICLR 2027

Beyond Prompt Prediction: Response Preference Learning for Automatic SAM Prompting

Abstract

Automatic prompting is essential for extending the Segment Anything Model (SAM) to fully automated segmentation pipelines. Existing approaches typically formulate this problem as direct prompt prediction, where a model learns to generate prompts expected to produce accurate SAM segmentation results. However, SAM responses can vary substantially even for nearby prompts, making it challenging to determine desirable prompts without observing the responses they actually induce. This motivates us to investigate whether prompts can be determined after observing their induced SAM responses. We therefore decouple automatic prompting into candidate generation and response-based prompt determination, and find that even random exploration can discover candidate prompts with strong segmentation potential. This finding indicates that the central challenge of automatic prompting may not be generating a desirable prompt directly, but determining which candidate is preferable based on its realized SAM response. Based on this insight, we propose Response Preference Prompting (RePP), which performs automatic SAM prompting through response preference learning. Given a current prompt, RePP explores a set of candidate prompt updates and learns a preference model to determine which update to retain based on its induced SAM response. To improve exploration efficiency, we further introduce a lightweight multi-point generator that guides a more focused search over candidate updates than random sampling does. Across VOC, COCO, and NWPU, RePP exceeds the strongest compared automatic prompting methods by 4.2–10.7 AP points. Applying its learned response preference model to SAC-Prompt, Matcher, and PPD further improves AP by 15.4, 5.3, and 4.0 points, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segment Anything Model, automatic prompting, response preference learning, prompt optimization, segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bwqaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Response Preference Prompting (RePP), which reframes automatic SAM prompting from direct prompt prediction into response-based candidate selection.

At each refinement step, RePP generates several candidate prompt updates, evaluates their induced masks with a frozen SAM, and uses a learned pairwise preference model to retain the preferable response.

The paper further introduces a multi-point generator that jointly proposes several point edits to reduce the number of sequential SAM interactions.

Experiments on VOC, COCO, and NWPU show substantial gains over several adapted automatic point-prompting methods, and the preference module also improves SAC-Prompt, Matcher, and PPD when applied as a plug-in.

Critical strengths
  1. The central response-selection hypothesis is supported by unusually clean ablations.

The strongest evidence is Table 1: learned response preference improves AP from 23.7 with the native SAM decoder score to 59.8, approaching the 65.6 AP oracle that selects candidates using ground-truth IoU.

Table 3 further separates per-step candidate selection from final trajectory selection and shows that both contribute substantially. These results make a convincing case that evaluating realized SAM responses provides information that conventional prompt-side criteria fail to capture.

  1. The proposed preference mechanism appears reasonably generator-agnostic rather than being tied to the authors' candidate generator. Adding the same response preference mechanism to SAC-Prompt, Matcher, and PPD improves AP by 15.4, 5.3, and 4.0 points, respectively.

Together with the results across natural-image and remote-sensing benchmarks, this provides useful evidence that response-based determination is a reusable component rather than an isolated improvement from the multi-point generator.

Critical weaknesses
  1. The pairwise preference formulation does not guarantee a globally consistent ranking, although inference relies on sequential tournaments.

The antisymmetric construction in Eq. (4) guarantees that swapping a pair reverses its preference, but it does not impose transitivity across three or more responses. Consequently, the selected candidate can in principle depend on tournament ordering.

This matters (theoretically only) because candidate determination is the central component of RePP, yet the paper reports only pairwise preference accuracy and does not characterize preference cycles or ordering sensitivity.

I would view this primarily as an unresolved robustness concern rather than evidence of an actual failure.

Questions for authors

Generalization of response preference across refinement steps.

In Section 3.2 and Appendix A.1, the response preference model is trained on pairs of SAM masks generated by applying different candidate prompt updates to the same current prompt state. In other words, the model learns to compare candidate responses within a single refinement step.

However, during final selection (Section 3.4, Eq. 10), the same preference model is used to compare masks retained at different refinement steps. Unlike the training pairs, these masks are generated from different prompt states along the refinement trajectory. It is therefore unclear whether the preference model trained on within-step comparisons can reliably generalize to cross-step comparisons.

Although Table 3 shows that preference-based final selection improves segmentation performance, it does not directly establish how accurately the model ranks responses across different refinement steps.

Could the authors clarify whether the preference model maintains its ranking accuracy when comparing responses from different refinement steps? For example, reporting preference accuracy separately for within-step and cross-step response pairs, using ground-truth IoU ordering as the reference, would help assess whether the training procedure adequately supports the final selection mechanism.

Reviewer_srr7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work targets automatic prompting for the Segment Anything Model (SAM). The authors observe that SAM responses can vary substantially even for nearby prompts, making it difficult to determine desirable prompts (previous method). Instead, this work proposes Response Preference Prompting (RePP), a framework that decouples automatic prompting into candidate generation and response-based prompt selection. RePP implements this idea with two components: a pairwise response preference model that selects the best candidate by comparing induced SAM outputs, and a lightweight multi-point generator that improves exploration efficiency. Experiments on three benchmarks (VOC, COCO, NWPU) show that RePP outperforms state-of-the-art methods by 4.2–10.7 AP points. Furthermore, the preference module can be plugged into existing prompting methods to bring consistent improvements.

Critical strengths
  1. The paper reframes automatic SAM prompting from a forward prediction problem to a feedback-based preference learning problem. The diagnostic experiment is convincing (Fig. 1b).

  2. RePP achieves consistent and significant improvements across three diverse datasets. And, the proposed response preference module can be plugged into existing prompting methods to bring consistent improvements.

Critical weaknesses
  1. The method is exclusively designed and evaluated for point prompts. While point prompting is an important setting, SAM natively supports multiple prompt types including bounding boxes, coarse masks, and text prompts. The adaptability of the RePP framework, especially the multi-point candidate generator, to other prompt modalities is not explored, which restricts the broader impact of the work.

  2. The iterative refinement paradigm requires multiple SAM forward passes to evaluate candidate prompts at each step. Even with the multi-point generator reducing interaction counts, RePP still incurs noticeably higher latency than single-step prompt prediction methods. This may limit its applicability in latency-sensitive real-world systems.

Questions for authors

I will raise my score if the authors are able to address these weaknesses.

Reviewer_fwMaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that predicting best prompt without inferencing SAM is not ideal. They propose a response preference based method which refines the prompt according to the SAM predicted mask. Experiments on several datasets show improvements over existing baseline methods.

Critical strengths

The finding is very interesting and the random prompt exploration experiment is also inspiring. The paper also highlights the sensitivity of SAM to different prompts and solve this problem through a new aspect.

Critical weaknesses
  1. The motivation for the needs of seeing the response to predict the prompt is not fully convincing. I agree that SAM is sensitive to prompt locations. However, the prompt sensitivity does not necessarily mena that SAM responses are unpredictable. Since SAM is a deterministic mapping function from prompts to segmentation masks for a given image, a trained prompt predictor could potentially learn to generate effective prompts directly. I'm not convinced that repeatedly quering SAM and selecting among its reponses is necessary, at the cost of extra inference overhead.

  2. As mentioned above, I think this method may be effective but is inefficient. The inference overhead is huge compared to exsiting baselines which directly predict the prompt without seeing SAM's mask. Even with the multi-point generator, it evaluates nine candidate responses per instance, in addition to the initial prompt. This overhead could become substantial for images containing many objects. Although the authors report inference latency, I would like to see a more detailed analysis of scalability and whether the additional computation is justified by the substantial performance improvements. We are trading inference latency with performance improvement here, and I'm not sure whether this trade-off is valuable. Also, for the main results, I think reporting mean IOU would provide a more direct evaluation because the preference model is explicitly trained using IOU.

Questions for authors

see the weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.