- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
AwSAM addresses vocabulary-free few-shot semantic segmentation by fitting reusable prompt tokens directly in the shared prompt space of a frozen SAM-3. Its Concept Branch performs Visual Prompt Inversion: two sets of 16 tokens are optimized separately to reconstruct annotated support masks, using scene-preserving augmentations for one set and additional foreground-background composites for the other. An optional Localization Branch fits a regularized linear foreground-background discriminant on frozen C-RADIOv4 support features, transfers it to query features, and converts its predictions into positive and negative point prompts. The fitted concept/scene tokens and query-specific localization tokens are concatenated for segmentation. Across 12 benchmarks, the method reports mean mIoU of 59.78 and 68.07 at one and five shots, improving over the strongest prior method at each shot count by 11.33 and 7.36 points, respectively.
Critical strengths
- Fitting tokens directly in SAM-3's shared prompt space offers a practical alternative to language-derived initialization or relying entirely on support-query feature correspondences.
- Substantial gains under a broad, shared evaluation protocol. The evaluation covers natural objects, object parts, medical images, remote sensing, and defects. AwSAM outperforms the best prior result on 11 of 12 benchmarks at both shot counts. The gains are sufficiently large for further investigation of the method.
Critical weaknesses
Major Weakness:
- For Sec. 3.4, the description is hard to follow. From my understanding, the authors intend to leverage C-RADIOv4 features to directly fit a linear classifier on samples consisting of support images and masks, and then apply this classifier directly to query image features to obtain positive and negative points. I believe both parameters of this discriminator rely on support images. Therefore, it may learn representations specific to the support images rather than target rules.
- Building on Question 1, the authors do not report the accuracy and target coverage of positive and negative prompt points. And why introduce C-RADIOv4? Can the SAM3 backbone be used on its own to achieve this goal? It would not introduce an additional vision foundation model.
- The calculation of amortized time relies on a reuse assumption, but the authors do not explain why they chose this number, nor do they specify whether the evaluation actually reuses the same support set for 25 query images (varies across different datasets and deployment scenarios). This choice affects efficiency conclusions: AwSAM requires approximately 25 query reuses to achieve an overall runtime advantage. The amortized advantage observed under 25 queries might not generalize as an efficiency advantage in practical few-shot scenarios.
- Why does the concept branch require additional training on unrelated composite images? If a complete support object is composited onto a background image with consistent overall style, what is the difference from synthesizing a new support-specific context? Since the unrelated images from ImageNet are selected randomly instead of through specific selection, have the authors considered this potential challenge? In addition, according to the experimental results in the supplement, the maximum performance gap between concept + scene and concept + concept / scene + scene is at most 1 point, indicating a minor performance difference. The authors are encouraged to discuss the impact of such potential false negative samples.
Minor weakness:
- The authors' claim in Sec.3.3 that can be reused across all query images is overclaimed.
Questions for authors
- The authors claim that the concept branch distinguishes
what, while the Localization branch distinguisheswhere. Yet the concept branch alone can output spatial masks and can evidently drive localization, in other words,where, as well. - The paper claims in the abstract that the method works without training, while training is actually required, so consistency needs to be maintained. This is essentially a work similar to test-time prompt learning, which learns support prototypes into learnable prompts.
- If reuse of support materials is needed, is it necessary to retrieve reusable query-support pairs for dataset episodes in advance?
2 comments
Response from 2qkT
Thanks for author's detailed response. It addresses most part of my concerns, including choice of extra VFM, e.g. C-RADIOv4, composite view and estimated point accuracy. But some concerns still maintain.
I'm afraid that the rebuttal answering about discriminant parameter over support samples is not quite convincing. Prior methods like INISID3 and other similar-based works, utilize cross-image similarity map (or prototype similarity) to find the positive and negative localization point prompt. This work instead propose to fit a set of parameters on the support image and then employ it on the query image. If the scene context shows difference between them, would this parameter still work? And what is the advantage of this method compared to similarity-based approaches?
The result in rebuttal also shows the Positive points accuracy is not satisfied. The Negative points accuracy is also easy to understand, since there are much broader regions for non-target.
Moreover, the runtime is also a concern. As shown in new Table 4, this work performs better when 'queries per support' is large and performs worse when 'queries per support' is small. I believe it's better than previous version but I'm not sure about its application in real practice. Thus, the score to this point is only slightly raised.
Based on current version, My rating is only slightly raised but not sufficient. So I maintain the rating.
Support-to-query transfer, point accuracy, runtime and composites
We thank the reviewer for the detailed feedback. We have uploaded the revised manuscript and supplement.
Support-to-query transfer and point accuracy. Both discriminant parameters are fitted on support features and applied to frozen query features, as correctly interpreted by the reviewer. Similar feature matching underlies PerSAM, Matcher, GF-SAM, and INSID3. We evaluated discriminant transferability and point accuracy across all 50 folds (Supp. Sec. F.2.5, Table S20).
The unrelated-class control indicates that the discriminant transfers class-specific information despite lower query IoU. We further evaluated oracle points to examine the influence of point prediction errors (Supp. Tables S21 and S22).
Choice of C-RADIOv4. Following the suggestion, we evaluated alternative localization encoders, including SAM-3's visual backbone (Supp. Table S18).
C-RADIOv4 provides the largest improvement and is selected as the default. AwSAM-lite achieves 56.93 mIoU without an additional encoder.
Runtime and support reuse. We agree that amortization over 25 queries limits the interpretation of runtime comparisons. We have replaced this metric with total runtime as a function of query count (Fig. 4). For a single query, AwSAM requires 5.60 / 8.06 s compared to 0.47 / 0.67 s for INSID3 at one / five shots. AwSAM becomes faster than INSID3 at approximately 20 / 37 queries, respectively. All reported mIoU values use fresh support optimization for each query. The fitted prompt can be reused for subsequent queries of the same target concept without advance retrieval or pairing.
Composite views. Composites preserve the target foreground while varying the background to reduce dependence on support-specific context. We evaluated background diversity and composite count across five ablation folds (Supp. Table S14).
Greater background diversity improves accuracy, with negligible gains beyond four composites. We further evaluated the token decomposition across all 12 benchmarks with matched seeds (Supp. Table S12).
The decomposition improves AwSAM by 0.79 mIoU over the single scene fit (95% CI [0.08, 1.42]). Without localization points, both configurations achieve comparable accuracy.
Unlabeled positives. Although the ImageNet background pool has no image overlap with the benchmarks, name-based SAM-3 detects COCO classes in 15 of the 24 backgrounds. We replaced backgrounds containing detected target-class instances and refitted prompts for 7,450 affected episodes (Supp. Sec. B.3.1, Table S1).
All changes are within 0.28 mIoU, with confidence intervals including zero.
Branch functionality and optimization. We agree that the Concept Branch also performs spatial localization. The supplement distinguishes the query-independent concept prompt from optional query-specific points (Supp. Sec. F.2.6). We revised the abstract, introduction, and conclusion to describe AwSAM as support-only test-time prompt optimization with frozen backbone weights.