acceptodds
Withdrawn from ICLR 2027

Taming SAM3 for Few-shot Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Few-shot semantic segmentation (FSS) requires segmenting target objects using a few reference images and masks. Conventional methods learn cross-image correspondences through episodic training, incurring training costs and posing challenges for generalizing to novel categories. A recent vision foundation model (VFM), SAM3, invites a different view: once a visual prompt identifies a concept, the model can propagate it to corresponding instances. This behavior, also known as promptable concept segmentation (PCS), mirrors FSS yet exposes a spatial mismatch: the support visual evidence and the geometry prompts inhabit different image coordinates. To this end, we introduce Visual Concept Transportation for SAM3 (VCT-SAM3), which brings the support concept into the query coordinate system to exploit SAM3’s native concept propagation. Rather than transferring the entire support scene, VCT-SAM3 extracts the annotated support object and places it into the query image. The inserted object then serves as a visual prompt, enabling SAM3 to discover and segment corresponding instances. Moreover, we explore a position-debiased semantic prior to protect potential target regions through the visual concept transportation. Our VCT-SAM3 uses a frozen SAM3, without auxiliary models or additional training. Experiments indicate that VCT-SAM3 achieves state-of-the-art (SOTA) performance (e.g., 7.9% mIoU gain on the COCO-20i benchmark) with lower latency and memory consumption. Our codes will be released to foster future research.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segment anything model 3, Few-shot segmentation, Promptable concept segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_reL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Visual Concept Transportation for SAM3 (VCT-SAM3), a training-free approach to few-shot semantic segmentation. The key observation is that SAM3 can propagate a concept from a visual prompt to other instances within the same image, whereas standard FSS provides the exemplar and target in separate support and query images. Instead of learning cross-image correspondence or concatenating the full support and query scenes, VCT-SAM3 extracts the annotated support foreground and inserts it into the query image. The inserted object and its tight bounding box then serve as an in-context visual prompt, allowing frozen SAM3 to discover and segment query instances of the same concept.

To choose a safe insertion location, the method constructs a semantic prior from SAM3 features. The authors observe positional bias in these features and introduce Position-Debiased Semantic Matching (PDSM), which estimates a position-related feature subspace from a content-free image and removes it before matching. The resulting prior protects likely target regions during support-object placement. The method requires no training and no auxiliary vision model.

Critical strengths
  1. Transporting only the annotated object into the query turns a cross-image problem into within-image concept propagation, enabling SAM3 to perform both target discovery and segmentation without training or an external matching model. This is conceptually clean and practically attractive.

  2. The foreground-cutout comparison isolates the effect of irrelevant support context, while random-subspace and content-free-input controls provide evidence that PDSM is doing more than arbitrary feature suppression. The paper also analyzes thresholds, rank, feature sources, placement feasibility, efficiency, and fold-level behavior, which substantially strengthens the empirical case.

Critical weaknesses
  1. A pasted support object can differ from the query in scale, resolution, illumination, style, and boundary statistics. SAM3 may respond partly to copy-paste artifacts or contextual discontinuities rather than only to concept identity. The paper does not compare blending strategies, quantify artifact sensitivity, or test whether the approach remains reliable under strong support-query domain shifts. This issue is central because input manipulation is the mechanism of the method.

  2. Experiments use only PASCAL-5i and COCO-20i with clean ground-truth support masks. The approach directly depends on accurate foreground extraction and available placement space, and multi-shot insertion can increasingly alter the query. Results under mask noise, crowded scenes, fine-grained concepts, parts, and cross-domain datasets are missing. The fold-level PASCAL 5-shot results also show that adding more support evidence does not consistently outperform existing methods.

Questions for authors
  1. How sensitive is VCT-SAM3 to copy-paste artifacts? Please compare hard cutout insertion with alpha blending, boundary smoothing, color/statistics matching, and a control that pastes an unrelated object or the correct object with a deliberately artificial border. These experiments would help determine whether SAM3 follows the intended concept rather than synthetic-image cues.

  2. How does the method perform with imperfect support masks? Please evaluate erosion/dilation, boundary corruption, missing object parts, and masks predicted by a realistic upstream segmenter. Since foreground isolation is a core advantage over tight-box crops, this directly affects the practical validity of the method.

  3. Can the authors provide results on at least one cross-domain or specialized FSS benchmark and one fine-grained or part-segmentation benchmark? This would test whether PDSM and object insertion remain effective when support and query styles differ substantially or when semantic distinctions are subtle.

  4. What explains the mixed PASCAL-5i 5-shot fold results? Please report performance as a function of the number of inserted support objects and compare inserting all objects with selecting one or a subset. An analysis of prompt interference would materially affect the assessment of the claimed few-shot scalability.

  5. Beyond performance after subspace removal, what evidence shows that the estimated low-rank subspace specifically encodes position rather than other low-level or semantic statistics? Spatial probing or visualization across different content-free inputs and model layers would strengthen the interpretation of PDSM.

Reviewer_t4JdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VCT-SAM3, a training-free approach to few-shot semantic segmentation using a frozen SAM3. Rather than translating support annotations into query-side localization prompts or concatenating complete support and query images, the method extracts the support foreground and inserts it into the query image. The inserted object’s bounding box becomes a visual concept prompt, allowing SAM3 to segment matching instances in the composite image. The inserted foreground is subsequently removed from the prediction.

To reduce the risk of covering query targets, the method constructs a support–query similarity prior using SAM3 features. Position-Debiased Semantic Matching estimates a feature subspace from a constant image and projects out its leading directions before computing similarities. Positive and negative regions then guide placement and scale selection. Multiple support objects are inserted sequentially, with a padding fallback when placement is infeasible.

Experiments on PASCAL-5i and COCO-20i show competitive performance without task-specific training, text prompts, or an auxiliary feature extractor. The most substantial result is on COCO-20i, where the method improves over the text-free FSS-SAM3 comparator by 7.9 and 4.8 percentage points in 1-shot and 5-shot mIoU.

Critical strengths

1. The paper introduces a simple and effective way to use SAM3’s concept-segmentation interface, with a meaningful same-model comparison.

Transporting only the annotated support foreground is a practical alternative to concatenating complete support scenes. It allows a frozen model to perform both concept discovery and segmentation without task-specific training or additional models. The comparison with text-free FSS-SAM3 is particularly relevant because it reduces the confounding effect of different foundation models. The COCO-20i improvement is substantial, and Appendix B.2 shows that the 1-shot gain occurs across all four folds rather than being driven by a single favorable split.

The method’s simplicity is a strength: the paper identifies a useful input construction for an existing model rather than introducing a complicated trainable architecture whose benefits are difficult to isolate.

2. The experimental analysis goes beyond component removal and examines several plausible alternative explanations.

The PDSM analysis includes a same-rank random-subspace control, two content-free inputs, rank sensitivity, and measurements of positive-region quality. These experiments support the usefulness of the estimated subspace more convincingly than qualitative feature visualizations alone. For example, black-input PDSM improves final mIoU by 2.7 points on PASCAL and 2.3 points on COCO relative to raw SAM3 features, while arbitrary rank-64 removal produces much smaller gains.

The appendix also compares foreground cutouts with tight-box crops under shared object scale, placement, and prompts, and explicitly documents the padding fallback. Although that content comparison does not isolate every causal factor, it provides useful evidence for foreground-only transportation and acknowledges practical cases where the default placement procedure cannot be used.

Critical weaknesses

1. The experiments do not yet establish which part of the transportation strategy is essential.

The comparison against full-scene composition changes several factors simultaneously: the amount of support background, the composite dimensions, the effective representation of the support object, and its position relative to the query. Appendix B.1 improves on this by comparing crops and cutouts at the same location and scale. However, the crop also overwrites more query pixels than the cutout, so the difference reflects both support-context contamination and preservation of query content.

Two additional controls would substantially clarify the contribution. First, the foreground cutout could always be placed in a compact padded strip outside the original query, using the fallback already implemented by the authors. This would retain foreground-only prompting while avoiding query occlusion and the need for semantic placement. Second, the PDSM prior could be used directly to generate query-side geometric prompts for the same frozen SAM3. This would test whether transportation offers a benefit beyond the improved localization prior.

These controls matter because the method still depends on cross-image semantic matching to decide which query regions to protect. The demonstrated contribution is therefore not the elimination of correspondence, but its use for composition rather than direct segmentation prompting. That distinction should be made explicit.

2. The evaluation supports strong COCO performance, but some broader accuracy and efficiency claims need tighter qualification.

The paper evaluates two natural-image benchmarks and reports sensitivity studies on COCO-20i, with the selected rank, thresholds, and scale range also being the best-performing settings in those studies. It is unclear whether these choices were fixed using independent validation data before evaluating the reported test episodes. The sensitivity analyses are useful, but they do not substitute for an independent model-selection protocol. No uncertainty estimates across episode samples are reported, which is particularly relevant to the smaller PASCAL improvements of 1.2 and 0.6 points over FSS-SAM3.

The state-of-the-art claim is also setting-dependent. On PASCAL-5i, FSSAM obtains 81.0%/85.4%, compared with 80.8%/81.8% for VCT-SAM3, and GF-SAM is higher in the 5-shot setting. The training-free, single-model trade-off remains attractive, but it should not be presented as uniform superiority.

Finally, Table 4 compares against full-support composition whose input grows with the number of shots and is not resized to a fixed resolution. Appendix A.4 clearly discloses this, so the comparison is informative for that configuration. Nevertheless, it does not establish an efficiency advantage at matched input-token budgets or comparable accuracy. An accuracy–latency–memory comparison across composition resolutions would support a more general efficiency claim.

Questions for authors

1. Is insertion inside the original query necessary?
Please evaluate an always-padding variant that places foreground-only support cutouts in a compact external strip and prompts SAM3 with their boxes. This uses the fallback already described in Appendix B.3. How does it compare with prior-guided insertion in accuracy, latency, and memory, especially for five support images? This would distinguish the value of foreground-only evidence from the value of placement inside the query scene.

2. What does transportation add beyond the PDSM localization prior?
Can the same PDSM prior be converted into query-side boxes or points for frozen SAM3, without pasting a support object? Please compare under the same feature extraction and output-processing settings. Also clarify how the subspace-removal procedure differs from the position-debiasing approaches cited in the motivation, particularly INSID3 and Denoising Vision Transformers.

3. How were hyperparameters and evaluation episodes selected?
Were rank 64, thresholds 0.7/0.3, and the scale range selected on a separate validation set? Were the same fixed episodes used for VCT-SAM3 and the reproduced SAM3 baseline? Please provide paired uncertainty estimates or results across additional episode samples, particularly for the relatively small PASCAL gains. Which comparative results were reproduced, and which were taken from prior papers?

4. How robust is placement when the semantic prior misses targets?
Equation (12) removes all pasted pixels from the final mask, so any true query foreground covered by a support cutout is necessarily lost. Please report the fraction of query ground-truth foreground overwritten by transported masks, including its distribution in 1-shot and 5-shot evaluation. How are empty positive regions, multiple positive clusters, and cases with no negative cluster handled? For multi-shot insertion, are query features computed once from the original image, and how is support-object order determined?

5. What is the efficiency trade-off under controlled resolutions?
Please specify the SAM3 checkpoint and feature layer, actual encoder input sizes, hardware, precision, batching, and timing protocol. Appendix A.4 states that timings include the placement pipeline, which is helpful. Can the authors additionally report accuracy and cost for full-scene composition resized to matched token budgets, and for compact foreground-only padding? These comparisons would clarify how much of the advantage comes from the proposed concept interface versus the chosen input sizes.

Reviewer_LTFfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses a frozen SAM3 for training-free few-shot segmentation. Instead of placing the full support and query images side by side, as FSS-SAM3 does, it cuts out the annotated support object, pastes it into the query image, and uses the pasted object as a visual prompt. A position-debiased semantic matching (PDSM) chooses a paste location that does not cover likely targets. 1-shot mIoU reaches 80.8 on PASCAL-5i and 74.5 on COCO-20i, which is 7.9 points above FSS-SAM3 on COCO.

Critical strengths

Simple method with a consistent gain on COCO. It needs only one frozen SAM3, with no extra model, text prompt, or training. On COCO-20i it beats FSS-SAM3 on all four 1-shot folds by 6.6 to 9.1 mIoU, so the gain does not depend on one favorable split. It also uses much less 5-shot memory than side-by-side composition (6.6 GB vs. 18.3 GB).

Critical weaknesses

It is unclear what the paper adds beyond prior work. PDSM closely follows the debiasing step of INSID3 (CVPR 2026): both run SVD on features of a content-free input and remove the top singular directions, differing mainly in backbone and input image. And the effect of pasting itself is also not isolated. The evaluation does not support the generalization claim. Only PASCAL-5i and COCO-20i are used, and SAM3 has very likely seen these categories during training, so the "novel" classes are not novel to it.

Questions for authors

How does PDSM differ from the debiasing in INSID3,? What are the results of the two missing baselines: prompting with the box of R+ on the original query, and blank-strip pasting for all episodes?

Comments on the paper

1 comment

Dear Authors,

We are approaching the end of the discussion period, and I note that no response has been provided to the reviewers' comments.

Please confirm if you intend to submit a rebuttal. If you have decided to withdraw this submission, please officially withdraw it through the system. If you are no longer able to respond, I will proceed with the evaluation based on the current reviews on file.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.