acceptodds
Withdrawn from ICLR 2027

Beyond Representation: Self-supervised Image Encoders to Segment Anything

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Segment Anything Model (SAM) has been the state-of-the-art interactive image segmentation paradigm, which enables users to extract target objects with simple prompts. Despite its generalizability to unseen objects, it relies on large-scale annotated images for training. Meanwhile, Self-Supervised Learning (SSL) has yielded image encoders with strong visual representations. We therefore ask whether self-supervised image encoders alone can support “segment anything” without fully supervised training. We present SSL-SAM, a training-free framework based on on-the-shelf self-supervised vision encoders for interactive segmentation from points, boxes, masks, and scribbles. SSL-SAM converts each type of prompt into a support set of foreground and background respectively, and segments the target object by prototype-consistency scoring in the feature space of a frozen encoder. To deal with very sparse prompt (e.g., a single foreground click), we also propose an automatic support expansion strategy to achieve high performance while reducing the user's effort. Across 20 datasets with 5000+ classes, SSL-SAM outperforms the SAM family (SAM, SAM2, and SAM3) across multiple prompt modes, and achieves a 13.49% gain in mean Intersection over Union (mIoU) in box-only mode. These results show that self-supervised vision encoders can be adapted without retraining for promptable segmentation, which avoids the high cost of large-scale annotation and supervised training in the SAM models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Segment anything, Interactive segmentation, Self-supervised learning, Foundation model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CeTHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SSL-SAM, demonstrating that on-the-shelf self-supervised pre-trained vision encoders already contains enough semantic representations to support promptable segmentation. SSL-SAM consists of several important components, including PGSE, FSG, and SCT. On multiple benchmarks, it achieves strong performance.

Critical strengths
  1. SSL-SAM is training-free and demonstrates promising performance compared with models in the SAM family.
  2. Extensive experiments are conducted to assess the effectiveness of the proposed methods.
Critical weaknesses
  1. Prototype-based segmentation built on DINO models is a well-established and extensively studied paradigm. The primary contribution of this paper lies in converting different types of prompts into prototypes. From this perspective, the paper’s overall contribution appears limited.
  2. Although SSL-SAM, unlike SAM, does not require training on large-scale annotated datasets, its performance still depends fundamentally on DINOv3, which has undergone large-scale unsupervised pretraining. Moreover, while the paper claims that SSL-SAM outperforms SAM across several prompt modes, SSL-SAM currently lacks support for some of SAM’s capabilities, such as text-based referring segmentation.
  3. In the experiments, why is evaluation limited to 200 randomly selected images from each dataset rather than the full dataset? Are SSL-SAM and SAM evaluated at the same image resolution? How does SSL-SAM ensure temporal consistency when applied to video object segmentation?
Questions for authors

Please refer to Weaknesses.

Reviewer_DB5aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether a frozen self-supervised image encoder can support prompt-based segmentation without task-specific training. SSL-SAM uses foreground and background prototypes derived from point, box, mask or scribble prompts to identify the target region, with additional support expansion for sparse prompts. Experiments across multiple datasets and prompt types show improvements over the reported SAM-family baselines in most prompt settings. The evaluation uses prompts generated from GT masks.

Critical strengths
  1. The technical choice is clear. Prompt prototypes let a frozen encoder replace a trained decoder, and the ablations separate the sparse-prompt expansion from that basic score.

  2. The evaluation covers many datasets and settings. Those comparisons support the reported gain over the SAM baselines in most settings.

  3. The paper is easy to follow, and the tables and figures are informative.

Critical weaknesses
  1. The evaluation uses one prompt and does not test a corrective click after the first mask. The automatic everything mode is also untested. The broad segment-anything framing therefore lacks support from the experiments.

  2. Using a frozen pretrained model for this kind of segmentation was already explored. [1] does training-free point-prompt segmentation with Stable Diffusion and counts clicks. [2] and [3] also segment from diffusion features. SSL-SAM mainly changes the pretrained model.

  3. The paper reports no runtime or FLOPs. The inference cost relative to the SAM baselines remains unclear.

  4. Section 2 and the reference list do not mention [1], [2] or [3]. The paper also misses interactive-segmentation work from 2025 onward, so the research focus is not up to date.

[1] Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation. CVPR, 2025.

[2] Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion. CVPR, 2024.

[3] DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut. NeurIPS, 2024.

Questions for authors

The main concern is that the novelty is weakened by earlier work. For the specific points, please refer to Weaknesses.

Reviewer_hpG4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SSL-SAM, a training-free framework that uses frozen self-supervised vision-transformer features for interactive binary image segmentation. User points, boxes, masks, or scribbles are converted into foreground and background support sets, whose mean features define prototypes. Predictions are obtained from foreground-minus-background prototype similarity followed by Otsu thresholding. For sparse prompts, the method introduces prototype-guided support expansion (PGSE); box prompts additionally use foreground seed generation (FSG); and size/confidence-aware thresholding (SCT) restricts predictions for difficult cases. The evaluation samples 200 images from each of 20 datasets and considers six prompt modes. The paper reports average mIoUs of 66.48, 69.80, 72.93, 74.82, 78.30, and 70.44 across these modes, with substantial reported gains over SAM-family models in box, mask, and scribble settings.

Critical strengths
  1. The paper directly investigates whether frozen self-supervised representations can replace task-specific mask-supervised training for promptable segmentation. This is practically and scientifically relevant given the annotation and training cost of SAM-like systems.

  2. The study covers 20 datasets from natural, salient, specialized, challenging, video, and medical domains and evaluates six prompt configurations.

Critical weaknesses
  1. Section 4.1, p. 6 states only that prompts are generated from ground truth “with some perturbations.” It omits click placement, box jitter, mask corruption, scribble construction, prompt repetitions, and random seeds. Exact SAM/SAM2/SAM3 checkpoints, preprocessing, multimask selection, and thresholding are also absent. Scribbles are converted to SAM point prompts without a defined conversion. SAM natively supports dense mask prompts, but the conversion of the paper’s perturbed binary masks into SAM’s expected low-resolution mask input is not explained. TokenCut is described as designed for box prompts, although its standard formulation is unsupervised object discovery rather than an external-box-prompt method.

  2. The closest cited work, INSID3, already uses frozen DINOv3 features, prototypes, positional debiasing, and training-free support expansion. SSL-SAM’s clearer increment is same-image multi-prompt interaction with explicit background supports and PGSE/FSG/SCT. Equation (3) itself is a two-prototype linear classifier. M2N2 (CVPR 2025), a training-free point-interactive segmentation method evaluated on SBD and DAVIS, is not discussed. FocalClick and SimpleClick are cited but not evaluated.

  3. Figure 5(c) compares encoders with different architectures, capacities, patch sizes, and pretraining data. Section 4.3 calls BEiT supervised, although §2.2 itself cites BEiT as masked self-supervised pretraining. Therefore, the experiment cannot establish that self-supervision generally outperforms supervision or that Gram anchoring causes DINOv3’s advantage. Sections 1 and 5 claim OOD flexibility, “zero human annotation cost,” and a new paradigm without a controlled domain-shift or pretraining-overlap analysis.

  4. Several values and references are inconsistent.The abstract’s 13.49% box gain matches none of the §4.2 margins of 14.06, 14.83, and 11.81 percentage points; their rounded mean is 13.57 points. Equation (2) incorrectly writes. Section 5 refers to Figure 5(e) for encoder compatibility, apparently intending Figure 5(c).

Questions for authors
  1. What are the exact deterministic procedures and parameters for generating every prompt type, including all perturbations and random seeds?
  2. Which SAM-family checkpoints and inference settings were used, and how were mask and scribble prompts encoded, multimask outputs selected, and classical baselines initialized?
  3. Can the authors provide sampled image/instance IDs, split information, target-selection and aggregation formulas, realized category coverage, and paired uncertainty estimates?
  4. How does SSL-SAM compare with INSID3 and M2N2 under suitable common protocols, and with standard iterative interactive methods under NoC85/NoC90?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.