- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new benchmark for Few Shot segmentation covering a large number of categories and scenarios. The authors collected 104K images spanning 2400 classes from natural scenes to specialized domains like biology, agriculture etc. Additionally, the authors presented a MLLM assisted Few Shot segmentation approach that takes advantage of the visual understanding capabilities of the MLLMs and the segmentation strength of SAM 3. The proposed approach overlays the GT mask on the support image and feeds it as input into the MLLM with a fixed text instruction and a set of learnable tokens. They use the GT mask to restrict the learnable tokens to attend to only foreground visual tokens and then use an adaptive token gate to reweigh them. These learnable and reweighed tokens form target aware prompts for SAM 3 predictor. Additionally, they inject a support-query similarity prior into the query features using SAM 3's memory encoder and memory attention. The proposed approach outperforms state-of-the-art FSS methods on the proposed benchmark and generalizes well to established FSS benchmarks.
Critical strengths
The paper is well written and easy to follow. It addresses a key issue with FSS benchmarks, most of which only cover a handful of common categories (like Coco-20i and Pascal-5i) or for single specialized domains (like medical or remote sensing). From that perspective having a FSS benchmark containing 2400 categories and more than 100K images is a major step up. The authors also document in details all the classes along with the annotation pipeline, data sources, statistics, an ethics section, and a maintenance plan. Additionally, the authors proposed a new approach, which combines an MLLM with SAM 3 and outperforms recent state-of-the-art methods on the proposed benchmark.
Critical weaknesses
In my opinion below are the key weaknesses of the work:
-
The annotation quality is not quantified by the authors. It is stated that categories were reviewed by graduate students in related fields and that masks went through multiple rounds of verification by two to three reviewers. However, there is no quantitative evidence like inter-annotator agreement. An independent quality study (from external users) at least on some elected subset, especially the ones on which the masks are manually delineated could have strengthened their benchmark. Alternatively, the authors could try automated approaches such as using frontier MLLMs as a judge on the annotation quality or an IoU between manually delineated masks and SAM3 output given the ground truth bounding boxes.
-
The construction of the folds are unclear and has potential of taxonomy overlap and contamination. The category list contains synonyms like (e.g., bag / bag, wallet; tie/necktie), superclass-subclass relationships (e.g., insect -> ant, bee; flower -> rose, daisy) and ambiguous or abstract categories (e.g. bell-clapping). The authors mention that the categories are divided into four disjoint folds but don't describe how the folds were determined (random or stratified by domains or in a different manner). If similar or related classes fall in a separate fold, then the test categories may not be truly unseen and unbiased, which would overestimate generalization. I feel a clear delineation of synonyms and class hierarchy and taxonomy is required.
-
I feel that the zero-shot transfer claims to standard FSS dataset are of limited value since many of the categories in these benchmark would inevitable appear in VastFSS.
-
The authors do not report standard deviation but only report the mean mIoU. One shot results are known to be sensitive to the choice of support image. Standard deviation or variance over the episodes are needed. Another concern I have is that they evaluated on 2000 randomly sampled episodes. It is not clear if the authors took care to ensure that there was at least one episode for each category.
-
There seems to be limited gain from 1 shot to 5 shot for their proposed approach. MA-FSS improves by only 2.7 mIoU from 1-shot to 5-shot, whereas the approaches gain considerably more (e.g., FSSAM +10.9, GF-SAM +7.7, DiffewS +6.7). There is a lack of discussion from the authors about why additional shots help so little. I believe, experiment with more shots (like 10-shot) would help understand whether the proposed approach scales to increased shots.
-
Even though the authors gathered about 109 samples per category, Fig 2b shows long-tail distribution and that some categories have very few samples (less than 10). I am not sure whether evaluation on these categories are reliable or valuable and how they affect sampling of the episodes, especially in the 5-shot setting since only a handful query images remain for evaluation.
-
Several relevant few-shot segmentation works are not cited or discussed, including DIaM (Hajimiri et al., CVPR 2023), visual prompting for generalized FSS (Hossain et al., CVPR 2024), Segment From Attention (Hossain et al., TMLR 2026), and Retrieve and Segment (Aravanis et al., CVPR 2026). Particularly, the last two approaches use MLLMs for few-shot settings. They should be discussed in the related work and potentially as baselines to clarify the contributions of their proposed approach.
Questions for authors
Please check the weakness section. I would appreciate if the authors could particularly answer the following:
- How were the categories divided into the 4 folds? Was any measure taken to prevent related classes (like synonyms, subclass-superclass) from leaking into the test folds?
- How did the authors ensure the quality of manually annotated mask? Can any quantitative evaluation be done?
- How were the 2000 test episodes sample? Did the authors ensure that each category was evaluated at least once?
7 comments
Response to Q2: Taxonomy Overlap and Fold Construction
Q2: Taxonomy overlap and fold construction. How were categories assigned to the four folds? How were synonyms, parent–child relationships, and ambiguous labels handled to prevent semantic overlap between training and test categories?
A2: Thank you for highlighting the importance of semantic consistency in constructing the evaluation folds. We agree that category-ID disjointness alone does not guarantee complete semantic disjointness. We clarify our taxonomy-guided fold construction and category handling policies below.
(i) Taxonomy-guided fold construction. For the 2,057 V3Det-derived categories, we use the V3Det taxonomy to guide fold assignment. Specifically, synonymous or closely related categories, such as bag/bag, wallet and tie/necktie, are assigned to the same fold to avoid separating equivalent concepts between training and testing. Similarly, hierarchically related categories, such as insect/ant/bee and flower/rose/daisy, are grouped within the same fold to reduce semantic overlap across folds. The resulting dataset is organized into four predefined folds, each containing 600 category entries. We will further clarify the fold assignment procedure and provide the complete category-to-fold mapping in the revised manuscript.
(ii) Cross-source category handling. For categories collected from different source datasets, we retain separate entries when they correspond to distinct application scenarios, even if they share the same underlying semantic concept. Such entries are not necessarily assigned to the same fold, as they are intended to capture differences in visual appearance, imaging conditions , and segmentation contexts. We acknowledge that these cases do not represent strictly unseen semantic concepts, and will explicitly document cross-source semantic overlaps.
(iii) Category ambiguity. For potentially ambiguous category labels, such as bell-clapping, we provide category-specific textual descriptions inherited from the V3Det taxonomy to clarify their semantic meanings and distinguish them from related concepts. These descriptions provide additional context beyond category names and help ensure consistent interpretation of the segmentation targets. We will make these descriptions available as part of the released dataset metadata to improve clarity and reproducibility.
To improve transparency, we will provide the complete category-to-source-to-fold mapping and explicitly document synonym groups, hierarchical relationships, and cross-source semantic overlaps in the revised appendix.
We appreciate the reviewer's constructive feedback, which will help us clarify the fold construction protocol and the intended scope of generalization evaluated by VastFSS.
Response to Q3: Transfer Generalization and Evaluation Reliability
Q3: Interpretation and reliability of transfer results. (a) Given category overlap with VastFSS, what do the external transfer results establish? (b) How sensitive are the results to support-image and episode sampling? (c) Do the 2,000 test episodes cover every test category?
Tab. B: Robustness of MA-FSS to support–query episode sampling on VastFSS under the 1-shot setting. Results are reported as mIoU (%) across three independently sampled episode lists with fixed model checkpoints.
| Seed | Fold-0 | Fold-1 | Fold-2 | Fold-3 | Mean |
|---|---|---|---|---|---|
| 42 | 54.0 | 56.9 | 54.4 | 55.8 | 55.3 |
| 43 | 54.7 | 56.6 | 54.5 | 56.5 | 55.6 |
| 44 | 55.1 | 56.5 | 54.7 | 57.0 | 55.8 |
| Mean | 54.6 | 56.7 | 54.5 | 56.4 | 55.6 |
| Std | 0.56 | 0.21 | 0.15 | 0.60 | 0.28 |
A3: Thank you for raising these important concerns. We clarify the interpretation of cross-dataset transfer results and provide additional evidence regarding evaluation reliability below.
To sub-question (a): We acknowledge that some category overlap exists between VastFSS and existing FSS benchmarks. However, our category-name overlap analysis indicates that such overlap is relatively limited, with only 19/80 (23.8%) COCO categories, 20/240 (8.3%) FSS-1000 test categories, and 159/920 (17.3%) LVIS- evaluation categories matching the training vocabulary of VastFSS. Thus, the majority of evaluation categories have no direct category matches in the VastFSS training vocabulary.
Furthermore, Chest X-ray provides an important example of transfer to an unseen target concept, because lung segmentation is absent from the VastFSS training categories, and no images were directly collected from the Chest X-ray benchmark. Nevertheless, MA-FSS achieves 87.7% mIoU on this dataset, further supporting its cross-dataset generalization capability.
Overall, these results provide evidence of MA-FSS's ability to generalize across different datasets and visual domains without target-dataset fine-tuning. Nevertheless, we agree that zero-shot transfer may not accurately characterize this evaluation setting, given the partial category overlap. We will therefore adopt the more precise term cross-dataset transfer in the revised manuscript.
To sub-question (b): To assess sensitivity to episode sampling, we conducted three repeated evaluations on VastFSS using independently sampled episode lists with random seeds 42, 43, and 44. The trained checkpoints remained fixed across evaluations, allowing us to isolate variability arising from support–query sampling. As shown in Tab. B, MA-FSS achieves 55.6 ± 0.28% mean mIoU across the three evaluations, with relatively small variations across the four folds. These results indicate that its performance on VastFSS is relatively stable under different support–query episode samplings.
To sub-question (c): We sample 2,000 episodes per fold, resulting in 8,000 episodes across four folds. Episode sampling is stratified by target category, ensuring that all 600 test categories in each fold are evaluated at least once. The remaining episodes are distributed approximately evenly across categories. We compute the average IoU for each category across its sampled episodes, and then average the category-level scores to obtain the fold-level mIoU. Thus, no test category is excluded from evaluation. All methods use identical episode lists for fair comparisons.
We appreciate the reviewer's constructive feedback. We will incorporate these additional experimental results and clarifications into the revised manuscript to provide a more precise interpretation of cross-dataset transfer and improve the transparency and reliability of our evaluation protocol.
Response to Q1: Annotation Quality and Consistency
Thanks for the thoughtful and detailed feedback. We sincerely appreciate your recognition of the manuscript’s clarity, VastFSS’s broad category coverage and detailed documentation, and MA-FSS’s strong performance. Below, we provide point-by-point responses to the concerns raised.
Tab. A: Annotation consistency evaluation on a subset of manually annotated masks from VastFSS. We report mean IoU and Dice similarity between the two annotators and the original ground truth (GT), as well as inter-annotator agreement.
A1: Thank you for highlighting the importance of quantitatively assessing annotation quality. Following the reviewer's suggestion, we conducted an independent annotation consistency study on 200 randomly sampled images covering 168 categories from VastFSS. We recruited two annotators with computer science backgrounds who were not involved in the original annotation process. Each annotator was provided with the target category description and independently delineated the corresponding segmentation masks without access to the original ground-truth annotations. We evaluated annotation agreement using Intersection over Union (IoU) and Dice similarity.
As shown in Tab. A, the two annotators achieve 87.0% and 85.2% mean IoU against the original ground-truth masks, with corresponding Dice scores of 90.5% and 89.8%, respectively. Furthermore, the inter-annotator agreement reaches 81.9% mean IoU and 88.0% mean Dice. These results indicate substantial consistency between independently produced annotations and provide quantitative evidence supporting the reliability of our manually delineated masks.
We will include the sampling procedure, annotation protocol, and quantitative agreement results in the revised manuscript. We appreciate the reviewer's constructive suggestion, which has helped strengthen the annotation quality assessment of VastFSS.