acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Interactive Image Segmentation: A Prototype Selection View

Abstract

Interactive image segmentation aims to extract target objects based on the user guidance, and recent click-based approaches gain popularity for their simplicity and efficiency. However, existing deep interactive methods often struggle to retain sparse spatial prompts, capture intra-class variance, and leverage label semantics beyond predefined channels. In this paper, we rethink interactive segmentation from a prototype learning perspective, where each user click defines a prototype comprising both spatial and label attributes. We propose a novel framework that learns spatial prototypes to guide feature flow and progressive label prototypes to model object characteristics, enhancing the embedding of user intention and enabling parallel segmentation querying without fixed label constraints. Instead of direct selecting an object, the joint representation of multiple prototypes helps to capture the intra-instance variance. Additionally, we introduce difficulty-aware and similarity-driven training mechanisms to enhance focus on challenging regions and improve inter-class discrimination. Vast experiments demonstrate that our method outperforms current state-of-the-art methods across diverse benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interactive Image Segmentation, Prototype Selection, Label Query, Temporal Click-aware Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1wCzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a prototype-based interactive image segmentation framework, which considers not only the spatial but also semantic information provided by the user clicks. The main experiments were conducted in several mainstream IS benchmarks.

Critical strengths

The paper clearly presents the motivation for the prototype learning enhanced interative image segmentation.

Critical weaknesses

There are a lot of incorrect symbols and claims in this paper. In addition, this paper lacks a complete literature review and the insufficient experiments cannot support the contributions. I believe the current version is not ready for the submission to ICLR 2027. Here follow the detais.

  1. What does the 'leverage label semantics beyond pre-defined channels' mean?
  2. Actually, interactive image segmentation is a class-agnostic task. While the authors state 'without fixed label constraints' in the abstract, I do not fully get the contribution of this paper.
  3. The related work section is quite limited. The authors should provide a complete literature review for interactive segmentation, built on model-designed and SAM-based methods. There are quite a lot papers published in recent CVPR, ICLR, NeurIPS, ICCV venues.
  4. The citation format should be carefully checked before submission, e.g., Line 107.
  5. How did the authors determine the four discrete neighbors?
  6. While I noticed there is clear difference between stuff and things in Figure 2, I then couldn't find any related clarification on this point. In addition, there are a lot incorrect formatting issues for the variables.
  7. In the experimental section, the claimed comparison with 'state-of-the-art' interactive segmentation methods is not true. There are a lot recent works in this active field, while the compared methods were mostly published before 2023. The incomplete comparisons cannot support the effectiveness of the proposed method.
  8. Again, in figure 5, SimpleClick is a transformer-based baseline in 2023 ICCV. Only comparing the results with it for a work submitted in 2026 is not convincing.
  9. The inference experiments are missing.
Questions for authors

See the weaknesses above.

Reviewer_XF5ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on interactive image segmentation, focusing on the challenges of sparse spatial prompts, intra-class variance, and leveraging label semantics beyond predefined channels. The authors propose a prototype learning framework, where the prototype aggregates spatial and label attributes, helping to capture intra-instance variance. The proposed difficulty-aware and similarity-driven training strategies further improve performance in challenging regions.

Critical strengths
  • The consideration of wrong predictions that emerge in later interactions is thoughtful.

  • The contrastive loss constructs positive and negative samples to encourage more discriminative representations.

Critical weaknesses
  1. While applying prototype learning to interactive segmentation is interesting, the proposed approach appears to be an extension, or even a simplified task adaptation, of the CVPR'22 work[1] to this task.

  2. An efficiency comparison is missing. For each interaction round, does the encoder need to be run again?

  3. The baselines are relatively old, with most comparison methods published in 2023 or earlier. Comparisons with more recent methods would strengthen the evaluation.

  4. Although the ablation studies demonstrate the effectiveness of each component, the performance gains from LPL come from multiple factors, including additional attention layers, progressive updates, increased parameters, and multiple prototypes. Removing the entire LPL module does not establish which factor contributes most to the improvement, especially multiple prototypes.

Also, this paper claims that multiple prototypes help capture intra-instance variance, which is not verified. An experiment would compare single and multiple prototypes at the current interaction round, given the same image, click history, and previous mask. To avoid for the influence of model-specific interaction histories, both configurations should be evaluated: single-prototype model generated click/mask history + single and multiple prototypes, multiple-prototype model generated click/mask history + single and multiple prototypes

  1. I'm interested in the robustness of the prototype representation to incorrect clicks. If a user provides an incorrect click during an interaction round, does the representation remain robust? A robustness experiment similar to MM'23 [2] would help clarify this issue.

[1] Rethinking semantic segmentation: A prototype view. CVPR'22

[2] Feature decoupling-recycling network for fast interactive segmentation. MM'23

Questions for authors
  1. Conduct the experiment on efficiency comparison.

  2. Compare with more recent methods.

  3. Ablation studies.

  4. Robustness evaluation.

Please refer to the weaknesses for details.

Reviewer_cHYzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a prototype-based framework for click-based interactive image segmentation. Click features initialize prototypes, while pooled spatial prototypes mediate click-conditioned encoder-to-decoder feature transfer. A progressive label-prototype module refines representations across decoder scales and predicts each object by taking the maximum response over its associated click prototypes. Training combines normalized focal loss, a temporally weighted hard-region loss, and a prototype-guided contrastive objective. Experiments cover single-object segmentation on GrabCut, Berkeley, DAVIS, and SBD, and multi-object segmentation on SBD, COCO, and DAVIS, with additional SBD-only training results and qualitative prototype visualizations. The results show useful improvements over several baselines, although the advantage is not consistent across datasets and metrics.

Critical strengths
  1. The click-conditioned feature-transfer design has useful empirical support across multiple backbone families. In Table 1, the proposed ResNet-34 model reduces DAVIS NoC@90 from RITM's 9.07 to 5.40 and SBD NoC@90 from 6.57 to 5.74. The HRNet and ViT variants also improve these two metrics over the corresponding RITM and SimpleClick baselines. This breadth makes the approach more convincing than a gain demonstrated on only one architecture.

  2. The paper provides evidence beyond its main comparison table. Table 2 shows that removing SPL, LPL, or the combined click-aware supervision reduces performance relative to the complete ResNet-34 model. Appendix A adds a smaller training-data setting, while Figure 7 illustrates complementary responses from different label prototypes. These results support the usefulness of the overall framework, even though they do not yet isolate all of its claimed mechanisms.

Critical weaknesses
  1. The distinctive contribution and the role of the proposed training mechanisms are insufficiently established. DynaMITe already initializes queries from features at click locations, includes learned background queries, and aggregates multiple query masks per object by a pixelwise maximum followed by an argmax across objects. Consequently, multiple click-derived representations, flexible object counts, and the aggregation in Eq. (8) do not by themselves establish a new capability. Although SPL and progressive label learning may provide a meaningful advance, the paper needs to distinguish that advance more clearly from the existing formulation.

  2. Table 2 bundles the temporal hard-region and contrastive losses into CAS, leaving their individual contributions unresolved. The stated contrastive objective also treats same-object pixels assigned to different click prototypes as negatives, without establishing whether this component separation benefits segmentation.

  3. The comparative results do not substantiate the breadth of the performance claims. The claim of best overall multi-object performance across all datasets (lines 409-411) is stronger than Table 1 supports. On DAVIS, FocalClick achieves both lower NCI (5.20 vs. 5.26) and higher IoU (82.40 vs. 81.63). On COCO, DynaMITe requires fewer clicks (NCI 7.59 vs. 7.82), while the proposed method achieves higher IoU, indicating a trade-off. In the ViT-B single-object comparison, the proposed method wins three of six columns, with CCF-BC winning the other three. These results demonstrate useful but mixed gains.

Questions for authors
  1. What are the individual effects of the temporal hard-region loss and the prototype-guided contrastive loss? Is treating same-object pixels assigned to different click prototypes as negatives intentional, and what evidence shows that this separation improves the final segmentation?

  2. How should the overall performance claim be qualified in light of the mixed results in Table 1, and what evidence establishes the stability of the smaller reported gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.