acceptodds
Withdrawn from ICLR 2027

Tiny Clicks Trigger Large Consequences: Exploring Sensitivity to Spatial-Prompt Novelty in SAM-Based Promptable Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Click-based point prompts provide a simple and intuitive interface for promptable segmentation, where additional correct clicks are generally expected to refine the prediction. However, the number of clicks does not necessarily reflect how much new spatial evidence they provide. We term this property Spatial-Prompt Novelty and ask a fundamental question: how sensitive are SAM-based segmentation models to different levels of spatial evidence novelty in additional clicks? We investigate this question across three progressively relaxed regimes: (1) At the Zero-Novelty limit, exact repetition adds no new spatial location yet can reduce mean IoU by up to 22.1 points. (2) Under Controlled Low-Novelty, the failure persists even when all added clicks are distinct but closely spaced, while performance consistently recovers as their spatial dispersion increases. (3) Finally, under Naturalistic Low-Novelty, 89,924 valid human interaction events show that nearby click additions occur in recorded interaction and are associated with smaller immediate gains. Further experiments reveal that seemingly minor differences in the spatial novelty of correct clicks can trigger disproportionately large changes in model behavior, exposing spatial-prompt novelty as an important but overlooked factor in SAM-based promptable segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segment Anything Model, promptable segmentation, interactive segmentation, point prompts, Spatial-Prompt Novelty, prompt robustness, Duplicate Consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WwecOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of SAM-based promptable segmentation models to the spatial novelty of additional point prompts. The authors introduce the concept of Spatial-Prompt Novelty to distinguish the number of clicks from the amount of new spatial evidence they provide. The study examines this phenomenon under three settings: Zero-Novelty, where identical clicks are repeated; Controlled Low-Novelty, where distinct but spatially clustered clicks are compared with more dispersed configurations; and Naturalistic Low-Novelty, which analyzes recorded human interaction trajectories. Experiments with SAM-based models on COCO and VOC2012 reveal that repeating correct clicks can substantially degrade segmentation accuracy, with mean IoU reductions of up to 22.1 percentage points. The authors further show that closely spaced but distinct clicks can also lead to performance degradation, while greater spatial dispersion generally improves segmentation utility. An analysis of 89,924 valid human interaction events provides additional evidence that nearby click additions are associated with smaller immediate improvements. Overall, the paper highlights spatial-prompt novelty as an important factor in understanding the robustness and behavior of promptable segmentation models.

Critical strengths
  1. An interesting and practically relevant perspective on promptable segmentation. Rather than proposing another segmentation architecture, this paper investigates a fundamental but relatively underexplored issue: whether additional correct clicks necessarily provide useful information and improve segmentation performance. The distinction between click count and spatial-prompt novelty offers a meaningful perspective for understanding the behavior and robustness of SAM-based segmentation models. The findings may provide useful insights for future prompt encoding, click selection, and interactive segmentation strategies.

  2. Systematic experimental investigation with compelling observations. The paper investigates spatial-prompt novelty through three complementary settings, progressing from exact duplicate clicks to distinct but closely spaced clicks and finally to recorded human interactions. This experimental design helps demonstrate that the observed sensitivity is not limited to exact duplicate coordinates. The substantial performance degradation observed in some SAM-based models is particularly noteworthy. Furthermore, the analysis of a large collection of human interaction events strengthens the practical relevance of the study, while the controlled experiments provide useful evidence for understanding the effects of spatial prompt configurations.

Critical weaknesses
  1. Limited mechanistic understanding of the observed sensitivity. Although the paper reveals substantial performance degradation caused by redundant or closely spaced clicks, it provides limited insight into the underlying mechanisms responsible for this behavior. The analysis of prompt representations and duplicate consistency helps explain why repeated clicks can produce different outputs, but does not sufficiently explain why these changes sometimes lead to severe segmentation failures. A deeper investigation of the prompt encoder, attention mechanisms, and mask decoder would strengthen the technical contribution and provide more actionable insights for improving model robustness.
  2. Limited evidence of practical significance in realistic interactive segmentation. The most striking performance degradation is observed when identical positive clicks are repeated multiple times, which is an uncommon scenario in practical interactive segmentation. Although the authors extend their analysis to distinct nearby clicks and human interaction trajectories, the real-world experiments mainly establish an association between click proximity and smaller immediate improvements, rather than demonstrating comparable performance degradation. Therefore, the extent to which the severe failures observed under controlled settings translate into practical interactive segmentation remains insufficiently established.
  3. Limited generalizability across segmentation models and interaction settings. The main experimental findings focus on the SAM family, while different interactive segmentation architectures may encode and integrate spatial prompts in substantially different ways. The paper includes a comparison with SimpleClick, but this primarily illustrates differences in duplicate handling and interaction state. A broader evaluation of representative interactive segmentation methods, including different prompt encoding and iterative refinement strategies, would help determine whether spatial-prompt novelty is a general challenge or mainly an architectural characteristic of SAM-based models.
  4. Limited actionable validation of the proposed findings. While the paper identifies an interesting sensitivity phenomenon, it does not sufficiently demonstrate how the findings can be translated into improved segmentation robustness or interaction efficiency. Beyond removing exact duplicate clicks, evaluating simple novelty-aware strategies, such as distance-based prompt filtering or spatially diverse click selection, would help quantify the practical benefits of addressing this issue and strengthen the overall significance of the study.
Questions for authors
  1. Mechanistic explanation of performance degradation. The experiments reveal substantial differences in sensitivity across SAM generations, particularly between SAM1 and SAM2.1/SAM3. Could the authors provide a more detailed analysis of which components of the prompt encoding and mask prediction pipeline are primarily responsible for the observed performance degradation? Such an analysis would help clarify the underlying mechanism rather than merely documenting the phenomenon.

  2. Isolating the effect of spatial-prompt novelty. In the controlled low-novelty experiments, changes in click dispersion may also affect spatial coverage and the relative positions of clicks within the target object. Could the authors provide additional controlled experiments that better isolate spatial-prompt novelty from these confounding factors? This would help establish whether the observed performance differences are specifically attributable to spatial novelty.

  3. Practical impact under realistic interaction. Although the analysis of human interaction trajectories shows that nearby clicks are associated with smaller immediate improvements, this does not necessarily indicate performance degradation. Could the authors further quantify how frequently low-novelty clicks actually reduce segmentation accuracy in realistic interaction sequences, and how substantial these reductions are? This would help establish the practical significance of the identified phenomenon.

Reviewer_sK6pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how additional correct clicks affect SAM segmentation when little new spatial evidence is added. Experiments compare repeated and clustered clicks across SAM models, followed by sequential interaction and human click analysis. Repeated or clustered clicks can reduce segmentation quality, while nearby human clicks are associated with smaller immediate gains.

Critical strengths
  1. The research question is interesting. Separating click count from the new spatial evidence supplied by each click offers a useful perspective on interactive segmentation.

  2. The findings challenge the assumption that more correct clicks improve segmentation, highlighting the importance of new spatial evidence.

  3. The paper is easy to follow, and the tables and figures are informative.

Critical weaknesses
  1. Later interactions often focus on boundary refinement after the initial clicks establish the object. The controlled studies use interior foreground points without a dedicated analysis of boundary corrections or fine structures. The relevance of the findings to this refinement stage remains unclear.

  2. The practical benefit remains unclear. Users choose clicks to express the intended correction. Turning the findings into click guidance requires identifying useful regions while preserving that intent, and the benefit over automatically adding such clicks is untested. Beyond exact duplicate removal, the paper does not demonstrate how the analysis improves the annotation workflow.

  3. The evaluation covers the SAM family and the earlier SimpleClick model. Recent methods developed specifically for interactive segmentation are discussed but not analyzed.

Questions for authors

The research direction is interesting, but the relevance to boundary refinement and recent interactive segmentation methods remains unclear. The practical benefit also needs further evidence. For the specific points, please refer to Weaknesses.

Reviewer_LSSsOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how SAM-family models react when additional positive clicks add little or no new spatial information. Its clearest experiment keeps the image, click position, label, and interaction state fixed while repeating the same click. SAM2.1 and SAM3 can perform much worse in this setting, while SAM1 is more stable on the paper's official/native multimask path. The paper also studies tightly clustered but distinct clicks and human interaction data. The results show that click count and new spatial evidence should be treated separately.

Critical strengths
  1. The exact-repeat experiment is clean and useful. It isolates click multiplicity from click position and other state changes. The large COCO and VOC evaluations, paired comparisons, and image-level bootstrap make this core result convincing.

  2. The paper includes helpful controls. Oracle-best analysis shows that candidate selection is not the whole explanation. The SimpleClick and input-deduplication tests also show that prompt handling strongly affects the result. The implementation and reproducibility details are unusually thorough.

Critical weaknesses
  1. The claim about distinct but clustered clicks depends on the inference setup. Exact-repeat experiments use the official multi-mask path, while controlled low-novelty experiments use a forced single-mask path, with post-processing disabled for SAM2.1 in part of the study. C1-C6 uses 194 feasible targets and D0-D6 uses 195, with a bias toward larger objects. The paper does not measure prompt-embedding similarity, so it remains unclear whether C1-C3 behave like near-duplicates. Mechanistic explanations such as attention multiplicity are still hypotheses, not established findings.

  2. The real-interaction evidence is much weaker than the controlled result. In PointPrompt, the adjusted difference is about 0.0097 raw IoU, or roughly one percentage point, and remains correlational, while the model and inference settings are not fully known. The paper appropriately describes this analysis as observational. Earlier work has also reported sensitivity to nearby clicks and over-prompting. The clearest novelty is the fixed-state duplicate-prompt diagnosis, not a general causal claim that nearby or additional clicks hurt.

Questions for authors
  1. Do C1-C6 and D0-D6 show the same pattern under each model's released default inference path, including post-processing for SAM2.1?

  2. Does the decline track prompt-token embedding similarity or effective multiplicity? What do the failed masks look like in terms of area ratio, precision, and recall?

  3. How are the PointPrompt scores, model version, inference settings, IoU bins, and bootstrap clusters defined? Which details cannot be recovered from the dataset?

  4. How does this work differ from prior studies of click-position sensitivity, over-prompting, and prompt redundancy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.