acceptodds
Under review as a conference paper at ICLR 2027

AwSAM: One-Shot Vocabulary-Free Segmentation of Anything with SAM-3

12 views

Abstract

Promptable vision foundation models (VFMs) such as SAM-3 can segment targets from text or geometric prompts, but each input creates distinct limitations for domain-specific applications. Text prompts fail for concepts outside the model’s learned semantic space, while geometric prompts incorporating points and boxes require specifying the target in every query image. Few-shot semantic segmentation (FSS) addresses this challenge by specifying a novel category at inference from a small set of annotated support examples. Existing FSS methods largely follow one of two routes: (i) converting support examples into geometric prompts by matching support and query patch features in a frozen VFM, which improves generalization but underperforms on fine-grained targets due to coarse patch features; or (ii) learning an explicit concept representation through task-specific training, tens of labeled images, or learned text-token conditioning, which improves performance but limits generalization. Given these limitations, we revisit FSS and SAM-3 prompt conditioning approaches and ask whether a single annotated example can be used to define a transferable concept directly, without training, a target dataset, or a class name to improve generalization? In our study, we demonstrate that a frozen SAM-3 can hold such a concept in its prompt space, a shared token sequence into which its text, geometric, and visual-exemplar encoders write. Fitting concept tokens in this space bypasses the text encoder entirely, and thus the language prior that limits generalization. We introduce AwSAM, a vocabulary-free FSS framework that learns a set of concept tokens from a single annotated example. We further study the contribution of external localization cues by pairing the fitted tokens with point prompts derived from a simple closed-form discriminant on VFM features. AwSAM sets a new state-of-the-art in one-shot vocabulary-free FSS across 12 diverse benchmarks, outperforming the strongest prior method by 11.3 mIoU while providing 2.9× faster query inference.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Few-shot segmentation, Vision foundation models, Promptable segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2qkTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AwSAM addresses vocabulary-free few-shot semantic segmentation by fitting reusable prompt tokens directly in the shared prompt space of a frozen SAM-3. Its Concept Branch performs Visual Prompt Inversion: two sets of 16 tokens are optimized separately to reconstruct annotated support masks, using scene-preserving augmentations for one set and additional foreground-background composites for the other. An optional Localization Branch fits a regularized linear foreground-background discriminant on frozen C-RADIOv4 support features, transfers it to query features, and converts its predictions into positive and negative point prompts. The fitted concept/scene tokens and query-specific localization tokens are concatenated for segmentation. Across 12 benchmarks, the method reports mean mIoU of 59.78 and 68.07 at one and five shots, improving over the strongest prior method at each shot count by 11.33 and 7.36 points, respectively.

Critical strengths
  1. Fitting tokens directly in SAM-3's shared prompt space offers a practical alternative to language-derived initialization or relying entirely on support-query feature correspondences.
  2. Substantial gains under a broad, shared evaluation protocol. The evaluation covers natural objects, object parts, medical images, remote sensing, and defects. AwSAM outperforms the best prior result on 11 of 12 benchmarks at both shot counts. The gains are sufficiently large for further investigation of the method.
Critical weaknesses

Major Weakness:

  1. For Sec. 3.4, the description is hard to follow. From my understanding, the authors intend to leverage C-RADIOv4 features to directly fit a linear classifier on samples consisting of support images and masks, and then apply this classifier directly to query image features to obtain positive and negative points. I believe both parameters of this discriminator rely on support images. Therefore, it may learn representations specific to the support images rather than target rules.
  2. Building on Question 1, the authors do not report the accuracy and target coverage of positive and negative prompt points. And why introduce C-RADIOv4? Can the SAM3 backbone be used on its own to achieve this goal? It would not introduce an additional vision foundation model.
  3. The calculation of amortized time relies on a reuse assumption, but the authors do not explain why they chose this number, nor do they specify whether the evaluation actually reuses the same support set for 25 query images (varies across different datasets and deployment scenarios). This choice affects efficiency conclusions: AwSAM requires approximately 25 query reuses to achieve an overall runtime advantage. The amortized advantage observed under 25 queries might not generalize as an efficiency advantage in practical few-shot scenarios.
  4. Why does the concept branch require additional training on unrelated composite images? If a complete support object is composited onto a background image with consistent overall style, what is the difference from synthesizing a new support-specific context? Since the unrelated images from ImageNet are selected randomly instead of through specific selection, have the authors considered this potential challenge? In addition, according to the experimental results in the supplement, the maximum performance gap between concept + scene and concept + concept / scene + scene is at most 1 point, indicating a minor performance difference. The authors are encouraged to discuss the impact of such potential false negative samples.

Minor weakness:

  1. The authors' claim in Sec.3.3 that PsP_s can be reused across all query images is overclaimed.
Questions for authors
  1. The authors claim that the concept branch distinguishes what, while the Localization branch distinguishes where. Yet the concept branch alone can output spatial masks and can evidently drive localization, in other words, where, as well.
  2. The paper claims in the abstract that the method works without training, while training is actually required, so consistency needs to be maintained. This is essentially a work similar to test-time prompt learning, which learns support prototypes into learnable prompts.
  3. If reuse of support materials is needed, is it necessary to retrieve reusable query-support pairs for dataset episodes in advance?
2 comments
AuthorsOpenReview ↗

Support-to-query transfer, point accuracy, runtime and composites

We thank the reviewer for the detailed feedback. We have uploaded the revised manuscript and supplement.

Support-to-query transfer and point accuracy. Both discriminant parameters are fitted on support features and applied to frozen query features, as correctly interpreted by the reviewer. Similar feature matching underlies PerSAM, Matcher, GF-SAM, and INSID3. We evaluated discriminant transferability and point accuracy across all 50 folds (Supp. Sec. F.2.5, Table S20).

Measure1 shot5 shots
Region IoU on support0.840.81
Region IoU on query0.530.64
Query IoU with unrelated class support0.130.14
Positive points on target0.670.71
Negative points outside target0.960.97
Ground-truth component coverage0.800.86

The unrelated-class control indicates that the discriminant transfers class-specific information despite lower query IoU. We further evaluated oracle points to examine the influence of point prediction errors (Supp. Tables S21 and S22).

Setting (one shot)Predicted pointsOracle points
Points only (12 benchmarks)46.0761.25
AwSAM (PASCAL-5i)86.7991.41
AwSAM (ISIC)64.9270.62

Choice of C-RADIOv4. Following the suggestion, we evaluated alternative localization encoders, including SAM-3's visual backbone (Supp. Table S18).

EncoderPoints onlyGain over AwSAM-lite
SAM-3 trunk40.95+1.26
DINOv3-L45.44+3.32
C-RADIOv446.07+4.02

C-RADIOv4 provides the largest improvement and is selected as the default. AwSAM-lite achieves 56.93 mIoU without an additional encoder.

Runtime and support reuse. We agree that amortization over 25 queries limits the interpretation of runtime comparisons. We have replaced this metric with total runtime as a function of query count (Fig. 4). For a single query, AwSAM requires 5.60 / 8.06 s compared to 0.47 / 0.67 s for INSID3 at one / five shots. AwSAM becomes faster than INSID3 at approximately 20 / 37 queries, respectively. All reported mIoU values use fresh support optimization for each query. The fitted prompt can be reused for subsequent queries of the same target concept without advance retrieval or pairing.

Composite views. Composites preserve the target foreground while varying the background to reduce dependence on support-specific context. We evaluated background diversity and composite count across five ablation folds (Supp. Table S14).

Composites (backgrounds)mIoU
None (scene only)62.30
4 (1 background)63.79
4 (2 backgrounds)63.96
4 (4 backgrounds)64.23
4 (random from 24, default)64.27
8 (random from 24)64.28

Greater background diversity improves accuracy, with negligible gains beyond four composites. We further evaluated the token decomposition across all 12 benchmarks with matched seeds (Supp. Table S12).

Prompt (32 tokens)With pointsWithout points
Scene + scene59.5055.86
Concept + concept59.7055.19
Single scene fit59.0555.98
Concept + scene59.8456.02

The decomposition improves AwSAM by 0.79 mIoU over the single scene fit (95% CI [0.08, 1.42]). Without localization points, both configurations achieve comparable accuracy.

Unlabeled positives. Although the ImageNet background pool has no image overlap with the benchmarks, name-based SAM-3 detects COCO classes in 15 of the 24 backgrounds. We replaced backgrounds containing detected target-class instances and refitted prompts for 7,450 affected episodes (Supp. Sec. B.3.1, Table S1).

BenchmarkDefaultFilteredChange [95% CI]
PASCAL-5i87.9788.26+0.28 [-0.25, +0.84]
COCO-20i59.8259.81-0.01 [-0.82, +1.16]
LVIS-92i53.1553.23+0.08 [-0.29, +0.48]
PASCAL-Part53.3253.26-0.05 [-0.61, +0.46]
PACO-Part42.4642.68+0.22 [-0.14, +0.52]

All changes are within 0.28 mIoU, with confidence intervals including zero.

Branch functionality and optimization. We agree that the Concept Branch also performs spatial localization. The supplement distinguishes the query-independent concept prompt from optional query-specific points (Supp. Sec. F.2.6). We revised the abstract, introduction, and conclusion to describe AwSAM as support-only test-time prompt optimization with frozen backbone weights.

Reviewer_2qkTOpenReview ↗

Response from 2qkT

Thanks for author's detailed response. It addresses most part of my concerns, including choice of extra VFM, e.g. C-RADIOv4, composite view and estimated point accuracy. But some concerns still maintain.

I'm afraid that the rebuttal answering about discriminant parameter over support samples is not quite convincing. Prior methods like INISID3 and other similar-based works, utilize cross-image similarity map (or prototype similarity) to find the positive and negative localization point prompt. This work instead propose to fit a set of parameters on the support image and then employ it on the query image. If the scene context shows difference between them, would this parameter still work? And what is the advantage of this method compared to similarity-based approaches?

The result in rebuttal also shows the Positive points accuracy is not satisfied. The Negative points accuracy is also easy to understand, since there are much broader regions for non-target.

Moreover, the runtime is also a concern. As shown in new Table 4, this work performs better when 'queries per support' is large and performs worse when 'queries per support' is small. I believe it's better than previous version but I'm not sure about its application in real practice. Thus, the score to this point is only slightly raised.

Based on current version, My rating is only slightly raised but not sufficient. So I maintain the rating.

Reviewer_y8UVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new method for few-shot segmentation, in which a model is given 1-or-more annotated examples of a visual concept, and it is tasked with segmenting the same category in novel query images. The paper proposes to exploit SAM3, without its text encoder, extending its native exemplar prompting mechanism to work across independent images. Additionally the paper explores using an external encoder for feature matching, specifically C-RADIOv4.

Critical strengths
  • Clarity. The proposed method is simple and easy to understand. The paper is well written.
  • Motivation. SAM3 natively supports only intra-image exemplar prompts. Therefore, extending it to work across image with visual prompts is an interesting and useful direction, that allows to exploit the pre-trained knowledge of the model for few-shot segmentation tasks
  • Generality. By relying on visual prompting, rather than text alone, the method can generalize to OOD and fine-grained concept examples for which a textual description is non-trivial to obtain or ambiguous. Experiments show that the text-based SAM3 baseline is outperformed on these kind of data.
Critical weaknesses
  • Motivation. The abstract, intro and related works present incoherent claims. The abstract says that a limitation of prior work is that it requires task-specific training, or tens of labeled images. This is exactly the case also of the presented method. Similarly in the intro, (L77) this method does require training, and target-specific data. In the RW (L136), this method does require optimization on labeled data. Most of the literature the paper cites is based on meta-learning for few-shot segmentation, which aims at testing generalization on unseen data. Therefore requiring training is not a weakness of these methods, and while training-free baselines are appreciable (GF-SAM, INSID3), the purpose of this whole staple of literature is to test generalization to unseen categories. Therefore this argument, that we should avoid any training whatsoever, would hold if the paper proposed a training free approach; which is not the case.
  • Framing and fairness of comparison. The paper frames itself in the literature of few-shot segmentation. However, few shot segmentation entails a specific protocol, well established in literature: train the model on a set of base classes, and evaluate on unseen ones in a few-shot manner; and is thus done per dataset with k-fold protocol, typically on COCO-20i and Pascal-5i. A generalization of this setting are in-context methods, that use a single generalist model that can be evaluated across benchmarks (SegIC, GF-SAM, SANSA, INSID3). Although not explicitly stressed, the proposed method requires to optimize the concept prototypes on the support examples at test time. This is not explicitly discussed, although one of the selling points of the teaser is that the method is 3x times faster than previous SoTa. This comparison excludes the training time on the reference, hiding the overall cost. The paper reports the ‘amortized’ cost over 25 queries; but the choice of dividing the cost of the reference by 25 is arbitrary. I know that in principle it is possible to train on the references and re-use it, however this does not cover all use cases. This brings me to my second point, the fairness of comparison. The method is not directly comparable to frozen or pre-trained methods; it does not fall in the definition of few-shot because it does not adopt a k-fold protocol, and does not fall in the generalist in-context because there is not a single set of weights used for all evals. The version only with points (no concept) would be the closest analogous to the relevant baselines Matcher, GF-SAM, INSID3. Additional generalist models should be in the table (SANSA, SegIC). Lastly, the experiments or related works do not consider prior works that, similar to this, trains on the support examples at test time [A, B, C, D].
  • More on fairness. The usage of C-RADIOv4 as encoder should be ablated: all previous methods use DINOv2/3 for feature matching. Do we need the RADIO? What improvement does it bring? Could it be done with SAM3 features alone? Note that RADIOv4 is distilled from SAM3 and DINOv3 as a teacher.
  • Questions. (1) Do we need to train the exemplars at test-time on the support? Could the prompts inversion be learned through meta-learning on a separate training set? I would imagine some MLP could be added and learn to derive the prompt tokens from such training set. There might be a trade-off with generalization, and it should be explored, as it would make the method more general, clean, and directly comparable.(2) SAM3 in its exemplar prompting mechanism also provides instance separation. Can this be supported for few shot instance segmentation?
  • Minor. The considered baselines, while correctly cited in the related works, are reported in the tables and in the discussion of experiments without citations. This makes it hard for the reader to easily navigate references

[A] Boudiaf et al. Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need? CVPR 2021

[B] Hajimiri et al. A Strong Baseline for Generalized Few-Shot Semantic Segmentation. CVPR 2023.

[C] Lu et al. Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021

[D] Johnander et al. Dense Gaussian Processes for Few-Shot Segmentation. ECCV2022

Questions for authors

The paper is well written, and presents a methodology to support visual prompts for semantic segmentation in SAM3, that achieves strong results outperforming the considered baselines across wide benchmarks. My main concern is in the framing of the paper. The authors should consider better positioning the paper wrt the few-shot, in-context and transductive inference literature, and present in the main table a comparable baseline (DINOv3/SAM3 encoder for matching, no concept prompts). Only then the claims and motivations outlined in the abstract, intro, RW would make sense and be coherent. The training time should be more explicitly discussed, and the ‘amortized’ metric which is completely arbitrary should be removed, and perhaps replaced with a plot in function of the number of queries for which a support image can be reused, with an x axis that clearly shows the trade-off (i.e. for less than 20 queries the method would be much slower than baselines). The efficiency claims in general should be more thorough: do all methods use the same setup? (toch.compile,floating precision,ecc). I expect the authors to acknowledge these points, and I would be inclined to raise my rating.

3 comments
AuthorsOpenReview ↗

Framing, positioning, points-only baseline, trained generalists and efficiency

We thank the reviewer for the detailed feedback. We have revised the framing, included additional experiments, and uploaded the revised manuscript and supplement.

Framing and evaluation protocol. We agree that base-class training is standard in few-shot segmentation and that our earlier use of "training-free" was imprecise. We have revised the abstract, introduction, and related work to position AwSAM as a support-only test-time optimization approach. AwSAM optimizes 32 prompt tokens using the annotated support while keeping all backbone weights frozen. The benchmark folds define evaluation episodes without additional base-class training.

Positioning. Sec. 2 now discusses the suggested methods and distinguishes their test-time adaptation protocols.

MethodTask-specific trainingTest-time fittingData for fitting
SANSA, SegICYesNoneN/A
GF-SAM, INSID3NoNoneN/A
RePRI [A], DIaM [B]YesClassifierSupport + unlabeled query
CWT [C]YesClassifierSupport
DGPNet [D]YesGaussian processSupport
M2CNoText-initialized embeddingSupport
AwSAMNoNoise-initialized prompt tokensSupport

Localization encoder and points-only baseline. Following the suggestion, Table 2 now reports points-only results across all 12 benchmarks using three encoders. We also report the improvement over AwSAM-lite when incorporating localization across seven benchmarks (Supp. Table S18).

EncoderPoints only (1 / 5 shots)Gain over AwSAM-lite
SAM-3 trunk40.95 / 49.48+1.26
DINOv3-L45.44 / 52.67+3.32
C-RADIOv446.07 / 53.37+4.02
INSID348.45 / 54.66N/A

C-RADIOv4 provides the largest localization gains, while SAM-3 features alone also improve segmentation. AwSAM-lite operates with fitted concept tokens alone and achieves 56.93 / 65.51 mIoU compared to 45.45 / 60.71 for M2C. Both methods optimize concept prompts within frozen SAM-3, providing a direct comparison of support-based concept learning.

Comparison with trained generalists. We evaluated the released SANSA and SegIC checkpoints on the same episodes (Table 1).

MethodMean (1 / 5 shots)COCO-20iLVIS-92iISICRSDD
SANSA51.10 / 57.1675.6350.3942.4426.82
SegIC41.36 / N/A70.8339.9727.5124.92
AwSAM59.78 / 68.0768.9556.0364.7646.72

Both methods outperform AwSAM only on COCO-20i, where all 80 classes and 74% (SANSA) or 87% (SegIC) of query images overlap with their training data (Supp. D.2.1).

Efficiency. We have removed the amortized runtime metric and report total runtime, including the support optimization cost, as a function of the number of queries (Fig. 4, Supp. Tables S8 and S9).

MethodSetup timePer queryTotal (1 query)AwSAM faster from
GF-SAMNone471 ms0.47 s17 queries
INSID352 ms417 ms0.47 s20 queries
SegICNone333 ms0.33 s29 queries
SANSANone208 ms0.21 s84 queries
AwSAM5456 ms143 ms5.60 sN/A

AwSAM incurs a higher total runtime than INSID3 for fewer than 20 queries per support. All runtimes were measured using the same L40S GPU model, batch size, fp16 precision, and supports. Reducing the number of optimization steps further decreases fitting time at the cost of segmentation accuracy (Supp. Table S11).

Adam steps102550100
Fit time (s)1.522.173.275.46
mIoU60.1463.6263.9464.27

Learned prompt prediction. Following the suggestion, we trained a 3.3M-parameter encoder on COCO to predict concept tokens in a single forward pass (Supp. Table S16).

PromptMean (1 / 5 shots)COCO-20iRSDD
Learned predictor47.81 / 51.1273.1722.50
AwSAM-lite56.93 / 65.5166.1244.70

The learned predictor outperforms AwSAM-lite on COCO-20i and one-shot PASCAL-5i. These results indicate a trade-off between learning a prompt predictor on predefined categories and the generalization achieved through support-only optimization.

Instance segmentation. We further evaluated the fitted prompts using SAM-3's instance segmentation head (Supp. Table S24).

Decoding strategymIoU
Semantic head (default)60.25
Instance masks (union)57.62
Prompt fitted on instance head58.81

The fitted prompts support individual instance mask generation. The reported results evaluate their combined semantic segmentation accuracy. Instance-level performance requires further investigation.

Citations. We have added citations for each baseline where introduced in Sec. 4.

I thank the authors for their detailed reply. I appreciate the authors concretely acting on my feedback, and I believe the manuscript has been improved. The most important changes were the correct reframing of the paper from few shot to support test time optimization. The training-free denomination was also misleading previously. In terms of positioning the literature study is now more complete with relevant methods. Adding the generalist models was another important change for fair comparison. Overall, i appreciate also the improved discussion on inference cost, ablation vs prompt prediction, SAM3 instance head, and backbone.
My last concern is on the main table, where, as already mentioned in my previous review, I strongly recommend the authors to emphasize clearly the proper baseline for comparison to prior methods, i.e. (DINOv3/SAM3 encoder for matching, no concept prompts).

If this happens I woud happy to raise my rating, as overall I appreciate the content of the paper

AuthorsOpenReview ↗

Table 1 updated with the points-only baseline

Thank you for the follow-up and the clear recommendation. We have updated Table 1 accordingly.

Table 1 now reports the points-only variant of AwSAM at one and five shots as shaded rows: feature matching with a frozen SAM-3 trunk or DINOv3-L encoder and no fitted concept prompt. The caption identifies these rows as the like-for-like baseline for the feature-matching priors. With DINOv3-L, this baseline scores 45.44 / 52.67 mean mIoU at one / five shots, compared with 48.45 / 54.66 for INSID3 and 43.40 / 49.60 for GF-SAM. With SAM-3's trunk it scores 40.95 / 49.48.

We hope this resolves your remaining concern.

Reviewer_nWP4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets one-shot segmentation from an example, with no class name. Given one support image and its mask, AwSAM keeps SAM-3 frozen and uses gradient descent to optimize 32 prompt tokens directly in SAM-3's internal prompt space, until SAM-3 reproduces the support mask. It never uses the text encoder. A second, optional branch fits a linear classifier on C-RADIOv4 features of the support image, applies it to the query image, and turns the result into positive and negative point prompts. Across 12 benchmarks it reports +11.33 mIoU (one-shot) over the strongest prior method.

Critical strengths
  1. Extensive validation across diverse benchmarks and state-of-the-art methods.

  2. AwSAM improves the mean one-shot mIoU over the strongest prior (INSID3) by 11.33 (59.78 vs. 48.45) and the mean five-shot mIoU over the strongest prior (M2C) by 7.36 (68.07 vs. 60.71).

Critical weaknesses
  1. The speed claim leaves out the fitting time. Fitting the tokens takes about 5.5 s per reference, which is 104× slower than INSID3's setup (Table 2).

  2. Most of the gain comes from SAM-3's prompt space, not from localization, and backbone differences complicate the comparisons.

  • The concept branch drives the improvement. It adds 13.70 mIoU at one shot, while the localization branch adds only 2.85 (Table 3). Localization even hurts on PASCAL-5i and ISIC, and it roughly doubles the per-query cost.
  • AwSAM relies on two strong frozen backbones (SAM-3 at 1008×1008 and C-RADIOv4 at 1152×1152), whereas the older baselines (PerSAM, Matcher, GF-SAM, PR-MaGIC) build on SAM and/or DINOv2. Part of the margin over them may therefore reflect backbone strength.
  • These are the SAM-3-based methods, FSS-SAM3 and M2C. AwSAM does outperform them by a wide margin under the vocabulary-free protocol. However, M2C is evaluated with SAM-3's generic "visual" prompt rather than its intended class-name initialization, and with the class name it comes within 3.47 mIoU of AwSAM on known-vocabulary benchmarks (Table 4).
Questions for authors

The concept/scene token decomposition is a central design choice of Visual Prompt Inversion, but it is not ablated or analyzed. Both sets are fit from the same noise initialization, and the concept-view set contains all scene views plus only four composites, so it is unclear how different the two sets become. Moreover, combining them doubles the token count (32 vs. 16), confounding the decomposition with added capacity or ensembling. Could the authors report (i) scene-only and concept-only tokens, (ii) two independently fit 32-token sets without composites, and (iii) a single 32-token fit on the concept views, ideally broken down by natural, part, and cross-domain benchmarks? Visualizing the masks or decoder attention produced by each set alone would also clarify what each captures.

1 comment
AuthorsOpenReview ↗

Fitting cost, localization, backbone strength, class names and token controls

We thank the reviewer for the constructive feedback. We have addressed the concerns through additional experiments and uploaded the revised manuscript and supplementary material.

Fitting time. We agree that the fitting time should be accounted for in the runtime comparison. We have removed the speed claim and reported total runtime across varying numbers of queries per support (Sec. 4.1, Fig. 4). For one shot and one query, AwSAM requires 5.60 s compared to 0.47 s for INSID3 and 11.32 s for M2C. AwSAM becomes faster than INSID3 at approximately 20 and 37 queries per support for one-shot and five-shot settings, respectively.

Contribution of localization. While the Concept Branch accounts for most of the performance gains, the Localization Branch provides complementary improvements, particularly for object parts (Supp. Table S21).

SettingNaturalPartCross-domainMean
Only Points48.7641.8245.7046.07
Only Concept62.8043.5057.4956.93
AwSAM65.8850.7958.7159.78
Gain from localization+3.08+7.29+1.22+2.85
Gain from concept+17.12+8.97+13.00+13.70

Localization increases per-query inference time from 76 to 143 ms. To investigate the performance degradation on PASCAL-5i and ISIC, we evaluated AwSAM with oracle points. Oracle points improve the concept-only results by 3.48 and 4.51 mIoU, respectively, indicating that inaccurate point predictions limit performance on these benchmarks (Supp. Table S22).

Influence of backbone strength. With the same SAM-3 backbone and without an additional encoder, AwSAM-lite achieves 56.93 mIoU compared to 45.45 for M2C and 43.69 for FSS-SAM3. We further evaluated different localization encoders across five ablation folds (Supp. Tables S17 and S18).

Localization encodermIoUGain over INSID3
None (AwSAM-lite)61.80+8.71
DINOv2-L62.30+9.21
DINOv3-L63.37+10.28
C-RADIOv464.27+11.18

AwSAM retains a substantial performance margin with DINOv2-L, the encoder employed by Matcher and GF-SAM. The comparison with SAM-3-based methods and the encoder ablations substantiate the contribution of the proposed approach beyond backbone strength.

M2C and class-name initialization. The vocabulary-free evaluation aims to isolate the contribution of visual exemplars from explicit class-name information. We additionally evaluate M2C and AwSAM with class names to examine the influence of SAM-3's pretrained vocabulary (Sec. 4.3, Table 3).

VocabularySAM-3 (name)M2C (exemplar)AwSAM (exemplar)M2C (+name)AwSAM (+name)
Known80.1752.0870.5971.5975.07
Unknown30.7132.8649.4340.5050.07

For known vocabulary, introducing class names improves M2C by 19.51 mIoU and AwSAM by 4.48 mIoU. Furthermore, name-only SAM-3 outperforms M2C with both the exemplar and class name. For unknown vocabulary, AwSAM outperforms M2C by 16.57 and 9.57 mIoU without and with class names, respectively. These results indicate that AwSAM derives more information from visual exemplars and exhibits reduced dependence on class-name initialization.

Scene and concept token decomposition. Based on the reviewer's suggestion, we conducted controlled experiments across all 12 benchmarks with matched initialization seeds to examine the influence of support views, token count, and independent optimizations (Supp. Sec. F.1.3, Table S12).

Prompt (total tokens)NaturalPartCross-domainMeanMean w/o points
Scene only (16)62.0046.4859.5458.1854.64
Concept only (16)62.6645.8259.5158.2853.92
Two scene fits (64)64.3844.8561.6459.7556.42
Single concept fit (32)64.3445.5360.5159.2954.35
Single scene fit (32)64.3745.5160.0159.0555.98
Scene + scene (32)62.9447.0961.3559.5055.86
Concept + concept (32)63.8446.7661.2659.7055.19
Concept + scene (32)64.2347.0361.1759.8456.02

The proposed decomposition achieves the highest mean among the 32-token configurations. Compared to a single 32-token scene fit, the decomposition improves AwSAM by 0.79 mIoU (95% CI [0.08, 1.42]), particularly on part (+1.52) and cross-domain (+1.16) benchmarks. Without localization points, both configurations achieve comparable performance. The benefits of the decomposition are therefore more pronounced when combined with localization.

SettingmIoU
Single concept set (32 tokens)63.26
Concept + concept63.61
Scene + scene63.43
8 / 32 tokens per set61.41 / 64.26
Concept + scene (16 + 16 tokens)64.27

These experiments support the selection of complementary support views with 16 tokens per set.

Comments on the paper

1 comment

Dear Reviewers nWP4 and y8UV,

The authors have posted detailed responses with new experiments and a revised manuscript. The discussion period ends in less than a day. Please reply soon: say which concerns are resolved or remain, and update your rating if needed.

Reviewer 2qkT, thank you for engaging.

Best regards,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.