- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies cross-domain few-shot segmentation (CD-FSS) and identifies a stability-adaptability trade-off in existing approaches. Pairwise reference-target visual matching can adapt to the appearance of an individual target but is sensitive to the selected reference and becomes unreliable under large appearance differences. A category prototype is more stable across target images but may fail to represent their diverse appearances.
To address this trade-off, the paper proposes D2-SAM3, a lightweight framework built on a frozen SAM3. Category Prototype Distillation (CPD) uses predictions produced from ground-truth box prompts on the annotated references as teachers for learning a reusable category prototype. Visual Residual Distillation (VRD) learns category-specific calibration of visual correspondence and expresses the visual prediction as a residual relative to the prototype prediction. Reliability-aware Fusion (RAF) modulates this residual using correspondence reliability and uncertainty in the prototype-based prediction. Only approximately 8.2K parameters are optimized per category.
Critical strengths
-
The distinction between stable but incomplete category prototypes and adaptive but unreliable pairwise correspondence is supported by per-target IoU distributions and reference-selection experiments. Modeling the correspondence signal as a residual rather than as the primary prediction is both technically sensible and well matched to this diagnosis.
-
D2-SAM3 adapts only about 8.2K parameters per category while keeping SAM3 frozen, and it performs strongly across four substantially different target domains. The component ablations, reference sensitivity analysis, efficiency measurements, task-transfer experiments, and backbone-controlled comparisons collectively provide credible evidence that the gains are not merely due to one favorable benchmark or a larger trainable model.
Critical weaknesses
-
The reusable prototype is defined in SAM3’s concept-prompt space, and the method learns a new prototype and calibration parameters for every category. The paper does not establish whether the main principle transfers to other promptable models or how adaptation/storage scale to hundreds or thousands of categories. This limits the generality of the claimed framework despite the small per-category parameter count.
-
CPD relies on a teacher produced by a ground-truth reference box, while VRD and RAF use correspondence reliability and prototype uncertainty to determine the correction. The paper does not test noisy masks/boxes or demonstrate that the reliability and uncertainty scores are calibrated predictors of actual error. These omissions are important because errors in the few available references could be amplified into a shared category prototype.
Questions for authors
-
How does D2-SAM3 behave when reference masks or their derived boxes are noisy? Please report controlled mask erosion/dilation, boundary noise, box jitter, and missing-part experiments. Because CPD distills a teacher derived from these annotations, this result would materially affect my confidence in the method’s robustness.
-
Can the authors quantify whether the correspondence reliability and prototype uncertainty used by RAF are calibrated? For example, please report their correlation with per-target visual/prototype prediction errors, reliability diagrams, or performance when either score is replaced by a learned or random ranking. This would validate the interpretation underlying RAF rather than only its aggregate mIoU benefit.
-
In the 5-shot setting, why are only the two references with the highest leave-one-out utility retained? Please compare against using all references, random selection of two references, and a simple learned or similarity-weighted aggregation. Please also clarify the computational scaling for larger values of K.
-
What is the total adaptation time and storage requirement when the number of categories becomes large? Can prototypes be amortized, initialized from text, or shared across related categories? This affects the practical meaning of the “lightweight” claim.
4 comments
Polite Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your constructive comments and valuable evaluation of our work.
We would like to kindly invite you to take another look at our responses and reconsider your assessment in light of our clarifications. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns, or if there are any remaining questions that we could further clarify.
Thank you again for your time and consideration.
Thank you for the detailed responses and additional experiments, which have addressed my main concerns. I will maintain my original score of 3 (weak acceptance).
Thank you very much for taking the time to read our rebuttal and provide further feedback. We sincerely appreciate your constructive comments and thoughtful consideration of our work. Your feedback has been very helpful in improving our paper.
Response to Reviewer sqH1
We thank the reviewer for recognizing our analysis of the stability–adaptability trade-off, the rationale for visual residual correction, and the extensive empirical evidence. We appreciate the constructive suggestions and address the concerns below.
Weakness 1a [Choice of SAM3 and transferability] Our motivation requires a segmentation model that combines a reusable semantic prompt space with visual correspondence. SAM3 is particularly well suited to this formulation: its promptable concept segmentation supports text and box-specified image exemplars within one model. Its large-scale concept supervision includes approximately 4 million distinct phrases and 52 million masks in SA-Co/HQ, alongside extensive synthetic training data. These capabilities allow us to distill box-prompted predictions into reusable category prompts while keeping the backbone frozen. SAM1/2 do not provide the same native text-conditioned concept-segmentation interface, so our implementation cannot be transferred directly.
Weakness 1b & Question 4 [Adaptation cost and scalability] The approximately 2-minute preparation time in Tab.6 covers three ISIC categories, with approximately 35 seconds per category (refer to revised Sec.5.6). Even we have hundreds of categories in practical scenarios, each new category prototype can be initialized from its textual embedding, but the learned prototype is category-specific and is not directly reused across categories. Each category can be incorporated through a very short 35-second adaptation, supporting rapid extension. In the scope of CD-FSS, four widely adopted benchmarks contain at most six categories.
Weakness 2 & Question 1 [Robustness to reference annotation noise] Standard CD-FSS protocols provide ground-truth reference masks and do not contain annotation errors. Nevertheless, we agree that robustness to imperfect annotations is relevant. Following your suggestion, we evaluated two perturbation settings:
The decreases are limited to 0.3–0.8 mIoU, supporting robustness to the tested perturbations.
Question 2 [Validity of correspondence reliability] Following your suggestion, we analyzed the correlation between correspondence reliability and per-target visual prediction error, defined as 1−IoUvis, using 600 evaluation episodes per dataset in the 1-shot setting.
On these datasets, higher reliability is associated with lower visual prediction error, supporting its use in weighting visual corrections. This analysis validates the reliability score’s predictive association in these settings. We add this analysis in the revised Appendix Sec.A4.4*.
Question 3a [Reference selection] We retain the two references with the highest leave-one-out utility to limit inference cost while prioritizing useful visual evidence. All five references still participate in category adaptation. Following your suggestion, we compared alternative inference strategies:
The default performs comparably to using all references or similarity-weighted aggregation, while random selection reduces mIoU by 0.7 and 0.8 mIoU. These results suggest that performance is not strongly dependent on the exact aggregation strategy; top-2 selection offers a practical accuracy–cost trade-off. We add these discussions to Appendix Sec.A3.6.
Question 3b [Scaling with K] For larger K, reference-averaged adaptation scales approximately linearly per optimization step, while leave-one-out selection evaluates K(K-1) directed reference pairs. This preparation is performed once per reference set. Retaining two references bounds the number of visual comparisons per target at two, whereas using all references requires K.