acceptodds
Under review as a conference paper at ICLR 2027

VastFSS: Vast Category Few-Shot Segmentation

Abstract

In this paper, we propose a novel benchmark, dubbed **VastFSS**, aiming to facilitate the development of more generalizable few-shot segmentation (FSS) by covering a vast number of categories across diverse scenarios. Specifically, VastFSS contains 104K images from 2,400 classes, spanning a wide range of application scenarios, from everyday natural scenes to specialized domains, such as biology, agriculture, medical imaging, remote sensing, aerospace science, and other scientific imagery. Compared to existing datasets, our VastFSS provides a broader and more diverse platform for training and evaluating generalizable FSS models, thereby facilitating their real-world applications. In addition, to encourage future research, we present the ***M****ultimodal large language model (MLLM)*-***A****ssisted* ***FSS***, termed **MA-FSS**, a novel framework which leverages the visual understanding capabilities of MLLMs for improving FSS. The key idea is to introduce a set of learnable object tokens to extract target-relevant representations from the support sample via an MLLM and employ them as additional prompts for a segmentation model to predict the query mask. To enhance the representation, we present mask-guided attention within the MLLM to direct the object tokens towards foreground target in the support image, together with a simple adaptive token gate to dynamically adjust their importance, leading to better performance. In extensive experiments on VastFSS, we show that MA-FSS achieves promising performance, outperforming other FSS methods, and generalizes well in a zero-shot manner to existing FSS benchmarks, demonstrating both the effectiveness of MA-FSS and benefits of the broad category and scenario coverage in VastFSS. Our benchmark and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Few-Shot Segmentation, Large-Vocabulary Benchmark, Multi-Domain Dataset, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JPeqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new benchmark for Few Shot segmentation covering a large number of categories and scenarios. The authors collected 104K images spanning 2400 classes from natural scenes to specialized domains like biology, agriculture etc. Additionally, the authors presented a MLLM assisted Few Shot segmentation approach that takes advantage of the visual understanding capabilities of the MLLMs and the segmentation strength of SAM 3. The proposed approach overlays the GT mask on the support image and feeds it as input into the MLLM with a fixed text instruction and a set of learnable tokens. They use the GT mask to restrict the learnable tokens to attend to only foreground visual tokens and then use an adaptive token gate to reweigh them. These learnable and reweighed tokens form target aware prompts for SAM 3 predictor. Additionally, they inject a support-query similarity prior into the query features using SAM 3's memory encoder and memory attention. The proposed approach outperforms state-of-the-art FSS methods on the proposed benchmark and generalizes well to established FSS benchmarks.

Critical strengths

The paper is well written and easy to follow. It addresses a key issue with FSS benchmarks, most of which only cover a handful of common categories (like Coco-20i and Pascal-5i) or for single specialized domains (like medical or remote sensing). From that perspective having a FSS benchmark containing 2400 categories and more than 100K images is a major step up. The authors also document in details all the classes along with the annotation pipeline, data sources, statistics, an ethics section, and a maintenance plan. Additionally, the authors proposed a new approach, which combines an MLLM with SAM 3 and outperforms recent state-of-the-art methods on the proposed benchmark.

Critical weaknesses

In my opinion below are the key weaknesses of the work:

  • The annotation quality is not quantified by the authors. It is stated that categories were reviewed by graduate students in related fields and that masks went through multiple rounds of verification by two to three reviewers. However, there is no quantitative evidence like inter-annotator agreement. An independent quality study (from external users) at least on some elected subset, especially the ones on which the masks are manually delineated could have strengthened their benchmark. Alternatively, the authors could try automated approaches such as using frontier MLLMs as a judge on the annotation quality or an IoU between manually delineated masks and SAM3 output given the ground truth bounding boxes.

  • The construction of the folds are unclear and has potential of taxonomy overlap and contamination. The category list contains synonyms like (e.g., bag / bag, wallet; tie/necktie), superclass-subclass relationships (e.g., insect -> ant, bee; flower -> rose, daisy) and ambiguous or abstract categories (e.g. bell-clapping). The authors mention that the categories are divided into four disjoint folds but don't describe how the folds were determined (random or stratified by domains or in a different manner). If similar or related classes fall in a separate fold, then the test categories may not be truly unseen and unbiased, which would overestimate generalization. I feel a clear delineation of synonyms and class hierarchy and taxonomy is required.

  • I feel that the zero-shot transfer claims to standard FSS dataset are of limited value since many of the categories in these benchmark would inevitable appear in VastFSS.

  • The authors do not report standard deviation but only report the mean mIoU. One shot results are known to be sensitive to the choice of support image. Standard deviation or variance over the episodes are needed. Another concern I have is that they evaluated on 2000 randomly sampled episodes. It is not clear if the authors took care to ensure that there was at least one episode for each category.

  • There seems to be limited gain from 1 shot to 5 shot for their proposed approach. MA-FSS improves by only 2.7 mIoU from 1-shot to 5-shot, whereas the approaches gain considerably more (e.g., FSSAM +10.9, GF-SAM +7.7, DiffewS +6.7). There is a lack of discussion from the authors about why additional shots help so little. I believe, experiment with more shots (like 10-shot) would help understand whether the proposed approach scales to increased shots.

  • Even though the authors gathered about 109 samples per category, Fig 2b shows long-tail distribution and that some categories have very few samples (less than 10). I am not sure whether evaluation on these categories are reliable or valuable and how they affect sampling of the episodes, especially in the 5-shot setting since only a handful query images remain for evaluation.

  • Several relevant few-shot segmentation works are not cited or discussed, including DIaM (Hajimiri et al., CVPR 2023), visual prompting for generalized FSS (Hossain et al., CVPR 2024), Segment From Attention (Hossain et al., TMLR 2026), and Retrieve and Segment (Aravanis et al., CVPR 2026). Particularly, the last two approaches use MLLMs for few-shot settings. They should be discussed in the related work and potentially as baselines to clarify the contributions of their proposed approach.

Questions for authors

Please check the weakness section. I would appreciate if the authors could particularly answer the following:

  • How were the categories divided into the 4 folds? Was any measure taken to prevent related classes (like synonyms, subclass-superclass) from leaking into the test folds?
  • How did the authors ensure the quality of manually annotated mask? Can any quantitative evaluation be done?
  • How were the 2000 test episodes sample? Did the authors ensure that each category was evaluated at least once?
7 comments
AuthorsOpenReview ↗

Response to Q1: Annotation Quality and Consistency

Thanks for the thoughtful and detailed feedback. We sincerely appreciate your recognition of the manuscript’s clarity, VastFSS’s broad category coverage and detailed documentation, and MA-FSS’s strong performance. Below, we provide point-by-point responses to the concerns raised.

Q1: Quantitative assessment of annotation quality. Can the authors provide an independent quality assessment of the segmentation masks, particularly the manually annotated subset, through inter-annotator agreement or another quantitative evaluation?

Tab. A: Annotation consistency evaluation on a subset of manually annotated masks from VastFSS. We report mean IoU and Dice similarity between the two annotators and the original ground truth (GT), as well as inter-annotator agreement.

ComparisonMean IoU (%) ↑Mean Dice (%) ↑
Annotator 1 vs. GT87.090.5
Annotator 2 vs. GT85.289.8
Annotator 1 vs. Annotator 281.988.0

A1: Thank you for highlighting the importance of quantitatively assessing annotation quality. Following the reviewer's suggestion, we conducted an independent annotation consistency study on 200 randomly sampled images covering 168 categories from VastFSS. We recruited two annotators with computer science backgrounds who were not involved in the original annotation process. Each annotator was provided with the target category description and independently delineated the corresponding segmentation masks without access to the original ground-truth annotations. We evaluated annotation agreement using Intersection over Union (IoU) and Dice similarity.

As shown in Tab. A, the two annotators achieve 87.0% and 85.2% mean IoU against the original ground-truth masks, with corresponding Dice scores of 90.5% and 89.8%, respectively. Furthermore, the inter-annotator agreement reaches 81.9% mean IoU and 88.0% mean Dice. These results indicate substantial consistency between independently produced annotations and provide quantitative evidence supporting the reliability of our manually delineated masks.

We will include the sampling procedure, annotation protocol, and quantitative agreement results in the revised manuscript. We appreciate the reviewer's constructive suggestion, which has helped strengthen the annotation quality assessment of VastFSS.

AuthorsOpenReview ↗

Response to Q2: Taxonomy Overlap and Fold Construction

Q2: Taxonomy overlap and fold construction. How were categories assigned to the four folds? How were synonyms, parent–child relationships, and ambiguous labels handled to prevent semantic overlap between training and test categories?

A2: Thank you for highlighting the importance of semantic consistency in constructing the evaluation folds. We agree that category-ID disjointness alone does not guarantee complete semantic disjointness. We clarify our taxonomy-guided fold construction and category handling policies below.

(i) Taxonomy-guided fold construction. For the 2,057 V3Det-derived categories, we use the V3Det taxonomy to guide fold assignment. Specifically, synonymous or closely related categories, such as bag/bag, wallet and tie/necktie, are assigned to the same fold to avoid separating equivalent concepts between training and testing. Similarly, hierarchically related categories, such as insect/ant/bee and flower/rose/daisy, are grouped within the same fold to reduce semantic overlap across folds. The resulting dataset is organized into four predefined folds, each containing 600 category entries. We will further clarify the fold assignment procedure and provide the complete category-to-fold mapping in the revised manuscript.

(ii) Cross-source category handling. For categories collected from different source datasets, we retain separate entries when they correspond to distinct application scenarios, even if they share the same underlying semantic concept. Such entries are not necessarily assigned to the same fold, as they are intended to capture differences in visual appearance, imaging conditions , and segmentation contexts. We acknowledge that these cases do not represent strictly unseen semantic concepts, and will explicitly document cross-source semantic overlaps.

(iii) Category ambiguity. For potentially ambiguous category labels, such as bell-clapping, we provide category-specific textual descriptions inherited from the V3Det taxonomy to clarify their semantic meanings and distinguish them from related concepts. These descriptions provide additional context beyond category names and help ensure consistent interpretation of the segmentation targets. We will make these descriptions available as part of the released dataset metadata to improve clarity and reproducibility.

To improve transparency, we will provide the complete category-to-source-to-fold mapping and explicitly document synonym groups, hierarchical relationships, and cross-source semantic overlaps in the revised appendix.

We appreciate the reviewer's constructive feedback, which will help us clarify the fold construction protocol and the intended scope of generalization evaluated by VastFSS.

AuthorsOpenReview ↗

Response to Q3: Transfer Generalization and Evaluation Reliability

Q3: Interpretation and reliability of transfer results. (a) Given category overlap with VastFSS, what do the external transfer results establish? (b) How sensitive are the results to support-image and episode sampling? (c) Do the 2,000 test episodes cover every test category?

Tab. B: Robustness of MA-FSS to support–query episode sampling on VastFSS under the 1-shot setting. Results are reported as mIoU (%) across three independently sampled episode lists with fixed model checkpoints.

SeedFold-0Fold-1Fold-2Fold-3Mean
4254.056.954.455.855.3
4354.756.654.556.555.6
4455.156.554.757.055.8
Mean54.656.754.556.455.6
Std0.560.210.150.600.28

A3: Thank you for raising these important concerns. We clarify the interpretation of cross-dataset transfer results and provide additional evidence regarding evaluation reliability below.

To sub-question (a): We acknowledge that some category overlap exists between VastFSS and existing FSS benchmarks. However, our category-name overlap analysis indicates that such overlap is relatively limited, with only 19/80 (23.8%) COCO categories, 20/240 (8.3%) FSS-1000 test categories, and 159/920 (17.3%) LVIS-92i92^i evaluation categories matching the training vocabulary of VastFSS. Thus, the majority of evaluation categories have no direct category matches in the VastFSS training vocabulary.

Furthermore, Chest X-ray provides an important example of transfer to an unseen target concept, because lung segmentation is absent from the VastFSS training categories, and no images were directly collected from the Chest X-ray benchmark. Nevertheless, MA-FSS achieves 87.7% mIoU on this dataset, further supporting its cross-dataset generalization capability.

Overall, these results provide evidence of MA-FSS's ability to generalize across different datasets and visual domains without target-dataset fine-tuning. Nevertheless, we agree that zero-shot transfer may not accurately characterize this evaluation setting, given the partial category overlap. We will therefore adopt the more precise term cross-dataset transfer in the revised manuscript.

To sub-question (b): To assess sensitivity to episode sampling, we conducted three repeated evaluations on VastFSS using independently sampled episode lists with random seeds 42, 43, and 44. The trained checkpoints remained fixed across evaluations, allowing us to isolate variability arising from support–query sampling. As shown in Tab. B, MA-FSS achieves 55.6 ± 0.28% mean mIoU across the three evaluations, with relatively small variations across the four folds. These results indicate that its performance on VastFSS is relatively stable under different support–query episode samplings.

To sub-question (c): We sample 2,000 episodes per fold, resulting in 8,000 episodes across four folds. Episode sampling is stratified by target category, ensuring that all 600 test categories in each fold are evaluated at least once. The remaining episodes are distributed approximately evenly across categories. We compute the average IoU for each category across its sampled episodes, and then average the category-level scores to obtain the fold-level mIoU. Thus, no test category is excluded from evaluation. All methods use identical episode lists for fair comparisons.

We appreciate the reviewer's constructive feedback. We will incorporate these additional experimental results and clarifications into the revised manuscript to provide a more precise interpretation of cross-dataset transfer and improve the transparency and reliability of our evaluation protocol.

Reviewer_pKV8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VastFSS, a large-scale few-shot segmentation dataset containing 2,400 categories and 104K images collected from multiple existing datasets, including V3Det, FIVES, FathomNet, and LoveDA. The dataset aims to address the limited category coverage of existing FSS benchmarks. The authors further propose MA-FSS, an MLLM-based few-shot segmentation framework incorporating mask-guided attention and an adaptive token gate. The MLLM processes a support image with the target region highlighted in red and uses learnable tokens to extract target-aware representations. Additionally, DINOv2 features are used to construct a spatial prior from support and query images, which is incorporated through SAM 3's memory modules to improve query segmentation. Experiments on VastFSS demonstrate improvements over several existing FSS methods, while zero-shot transfer evaluations on other benchmarks show promising generalization performance.

Critical strengths
  1. The paper introduces VastFSS, a large-scale few-shot segmentation dataset consisting of 2,400 categories and 104K images, addressing the limited category coverage of existing FSS benchmarks. The broader category coverage across different domains can be beneficial for developing more generalizable few-shot segmentation models.

  2. The proposed MA-FSS also demonstrates improved performance compared to several previous SOTA methods on VastFSS. Furthermore, training on the introduced dataset leads to promising zero-shot transfer performance on existing FSS benchmarks, demonstrating the potential benefit of training with a larger and more diverse set of categories.

Critical weaknesses
  1. The VastFSS dataset introduces 2,400 categories, but it is unclear what specific benefit it provides over combining existing datasets such as FSS-1000 (1,000 categories), LVIS-92i (920 categories), and other specialized FSS datasets. Since VastFSS is itself constructed from existing datasets, the authors should clarify what new or previously underrepresented categories it introduces and what specific limitations of existing datasets it addresses. Merely increasing the number of categories does not sufficiently establish the need for a new benchmark.

  2. The paper does not sufficiently distinguish the contribution of the proposed dataset from that of the MA-FSS method. The authors should report the performance of MA-FSS when trained directly on existing FSS datasets, rather than using VastFSS-trained weights, and compare these results against training on VastFSS. Additionally, evaluation on PASCAL-5i would strengthen the experimental comparison. Although the paper reports performance when trained on COCO-20i, it is unclear whether the model is trained directly on COCO-20i or fine-tuned from VastFSS-trained weights. Clarifying this distinction and providing the corresponding comparisons would help establish whether the improvements arise from the proposed methodology or the larger training dataset.

  3. The proposed MA-FSS framework is based on an MLLM, but the paper does not provide comparisons with existing MLLM-based few-shot segmentation approaches such as DSV-LFS [2]. Since leveraging MLLM-derived representations is a central aspect of the proposed method, comparison with such approaches is important to establish the effectiveness and novelty of MA-FSS relative to prior MLLM-based methods.

  4. The proposed MA-FSS framework closely resembles [1] in its use of a support-guided spatial prior incorporated through the SAM memory encoder. In particular, MA-FSS uses DINO features extracted from support and query images to construct the spatial prior using feature similarities, similar to [1]. This raises concerns about the methodological novelty of this component. Furthermore, since MA-FSS is primarily motivated as an MLLM-based approach, the necessity and significance of using a separate DINO feature extractor remain unclear. The authors should better justify why DINO features are required instead of relying on the visual representations available from the MLLM and clarify how the proposed approach differs from [1].

[1] Unlocking the Power of SAM 2 for Few-Shot Segmentation, ICML 2025.

[2] DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentation, CVPR 2025.

Questions for authors

Please see weakness

6 comments
AuthorsOpenReview ↗

Response to Q1: Contributions of VastFSS Beyond Existing FSS Benchmarks

We sincerely thank the reviewer for all the detailed and insightful comments. We truly appreciate your recognition of VastFSS’s broad category coverage, MA-FSS’s competitive performance, and the potential benefits of VastFSS training for cross-dataset transfer. Below, we provide point-by-point responses to all the concerns.

Q1: Value beyond combining existing FSS datasets. What does VastFSS provide beyond combining FSS-1000, LVIS-92i92^i, and specialized FSS datasets? Which categories are new or underrepresented, and what limitations does the benchmark address beyond increasing the category count?

A1: Thank you for raising this question. We agree that simply increasing the number of categories is insufficient to justify a new benchmark. The contribution of VastFSS lies not only in its category scale, but also in additional pixel-level annotations, broader coverage of segmentation targets, and a unified evaluation protocol.

(i) Additional segmentation annotations. VastFSS is not simply a combination of existing segmentation datasets. Specifically, its 2,057 V3Det-derived categories originate from a detection dataset that originally provides bounding-box annotations. We manually delineate target masks using these bounding boxes as references, introducing additional pixel-level segmentation annotations rather than merely reorganizing existing segmentation data.

(ii) Broader target coverage. The remaining 343 category entries are collected from 59 datasets spanning specialized applications, including medical imaging, biological microscopy, remote sensing, and solar physics. These introduce diverse segmentation targets, such as nuclei, thoracic aorta, and sunspots, with substantially different visual characteristics and imaging conditions. By integrating these specialized targets with natural-image categories, VastFSS provides broader coverage of segmentation scenarios than benchmarks focused primarily on natural images or individual specialized domains.

(iii) Unified evaluation protocol. Simply combining existing datasets does not automatically provide a consistent FSS evaluation setting. VastFSS unifies segmentation annotations and adopts a common episode-based evaluation protocol with four predefined category folds, each containing 600 held-out category entries. This enables systematic evaluation of FSS methods across diverse target categories and application scenarios under a shared protocol.

We do not claim that all 2,400 category entries represent previously unseen semantic concepts. We acknowledge that some categories overlap with existing FSS benchmarks. Specifically, using normalized category-name matching, we identify 29/80 categories shared with COCO-20i20^i, 141/1000 with FSS-1000, and 220/920 with LVIS-92i92^i. However, such overlap does not diminish the contribution of the newly annotated categories and the unified evaluation setting.

To further clarify the benchmark's contribution, we will provide a category-level comparison with existing FSS benchmarks in the revised appendix.

We appreciate the reviewer's constructive suggestion, which will help us better articulate the distinct contributions and intended scope of VastFSS.

AuthorsOpenReview ↗

Response to Q2: Contributions of MA-FSS and VastFSS

Q2: Separating the contributions of MA-FSS and VastFSS. (a) Was the COCO-20i20^i model trained directly on COCO, or fine-tuned from VastFSS-trained weights? (b) How does MA-FSS perform when trained directly on existing FSS datasets compared with training on VastFSS? (c) Can the authors provide results on PASCAL-5i5^i?

Tab. A: Comparison of MA-FSS trained on different datasets, evaluated on VastFSS under the 1-shot setting. All results are reported as mIoU (%).

Training datasetFold-0Fold-1Fold-2Fold-3Mean
FSS-100032.432.831.632.232.3
COCO-20i20^i37.735.135.236.336.1
LVIS-92i92^i47.748.748.247.548.0
VastFSS54.056.954.455.855.3

Tab. B: Comparisons on PASCAL-5i5^i under the 1-shot setting. All results are reported as mIoU (%).

Method505^0515^1525^2525^2Mean
GF-SAM (NeurIPS 24)71.175.769.273.372.1
PAHNet (ICCV 25)72.176.471.066.671.5
DeFSS (ICCV 25)73.776.670.965.871.7
FSSAM (ICML 25)81.684.981.676.081.0
DSV-LFS (CVPR 25)71.482.071.275.075.0
MA-FSS (Ours)84.190.586.184.986.4

A2: Thank you for highlighting the importance of disentangling the contributions of MA-FSS and VastFSS. We clarify the training settings and provide additional controlled comparisons below.

To sub-question (a): The result labeled “trained on COCO-20i20^i” was obtained by training MA-FSS directly on the corresponding COCO training folds, starting from the pretrained foundation-model weights rather than VastFSS-trained weights. It achieves 66.3% mean mIoU under the 1-shot setting. Separately, the VastFSS-trained model achieves 64.2% on COCO-20i20^i without target-dataset fine-tuning. We will make the initialization and training data explicit in the table and implementation details.

To sub-question (b): To examine the contribution of VastFSS with the model architecture held fixed, we trained MA-FSS separately on the full FSS-1000 dataset, COCO-20i20^i, and LVIS-92i92^i, starting from the same pretrained foundation-model weights rather than VastFSS-trained weights. We then compared these models with the VastFSS-trained model under the same evaluation protocol on VastFSS.

As shown in Tab. A, the models trained on FSS-1000, COCO-20i20^i, and LVIS-92i92^i achieve 32.3%, 36.1%, and 48.0% mIoU, respectively. In comparison, MA-FSS trained on VastFSS achieves 55.3% mIoU, outperforming the strongest alternative training dataset, LVIS-92i92^i, by 7.3 percentage points. Since the architecture is held fixed, these results demonstrate the effectiveness of VastFSS as a training resource for MA-FSS.

Together with the competitive directly COCO-trained results discussed in sub-question (a), these comparisons provide complementary evidence that MA-FSS achieves competitive performance without relying on VastFSS pretraining, while training on VastFSS yields stronger generalization to its held-out evaluation categories.

To sub-question (c): Following the reviewer's suggestion, we additionally evaluated MA-FSS on PASCAL-5i5^i under the 1-shot setting. Specifically, we initialized MA-FSS from the VastFSS-pretrained checkpoint and fine-tuned it on the corresponding PASCAL-5i5^i training folds for only two epochs.

As shown in Tab. B, MA-FSS achieves 86.4% mean mIoU, outperforming FSSAM (81.0%) by 5.4 percentage points, with improvements across all four folds. These results demonstrate that MA-FSS can achieve strong performance on an established FSS benchmark with only two epochs of target-dataset fine-tuning, highlighting its adaptability following VastFSS pretraining. We will explicitly clarify this training protocol and results in the revised manuscript.

We sincerely appreciate the reviewer's constructive suggestions, which have helped us provide a clearer distinction between the contributions of MA-FSS and VastFSS and strengthen the experimental evaluation.

AuthorsOpenReview ↗

Response to Q3: Comparison with MLLM-Based FSS Method

Q3: Comparison with existing MLLM-based FSS methods. Can the authors compare MA-FSS with existing MLLM-based approaches, particularly DSV-LFS, to establish its effectiveness relative to prior work?

Tab. C: Comparison with DSV-LFS on VastFSS under the 1-shot setting. Results are mIoU (%).

MethodFold-0Fold-1Fold-2Fold-3Mean
DSV-LFS (adapted)30.847.433.541.538.3
MA-FSS54.056.954.455.855.3

A3: Thank you for this valuable suggestion. We agree that comparing MA-FSS with existing MLLM-based FSS approaches would provide a more comprehensive evaluation.

DSV-LFS employs GPT-generated category descriptions to construct semantic prompts, whereas MA-FSS derives target-aware representations directly from annotated support images without requiring explicit category names or textual descriptions. To provide an empirical comparison, we adapted DSV-LFS to VastFSS by using category names to generate semantic prompts during both training and inference. We denote this implementation as DSV-LFS (adapted) to distinguish it from the original description-based setting.

As shown in Tab. C, MA-FSS achieves 55.3% mean mIoU, outperforming the adapted DSV-LFS (38.3%) by 17.0 percentage points, with improvements across all four folds. These results demonstrate the effectiveness of MA-FSS on VastFSS relative to this MLLM-based baseline, without requiring explicit category-level textual guidance at inference time.

We appreciate the reviewer's constructive suggestion and will include this comparison and the corresponding implementation details in the revised manuscript.

Reviewer_3V7COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VastFSS, a large-vocabulary, multi-domain benchmark for few-shot segmentation containing approximately 104K images and 2,400 categories. It combines 2,057 categories derived from V3Det with 343 categories from 59 specialized datasets, and establishes a four-fold, category-disjoint evaluation protocol.

The authors also propose MA-FSS, which extracts target-aware learnable tokens from annotated support images using Qwen3-VL. Mask-guided attention and an adaptive token gate refine these representations, which are combined with a DINOv2-based spatial prior to guide a SAM 3 segmentation framework. MA-FSS achieves 55.3% and 58.0% mIoU in the 1-shot and 5-shot settings on VastFSS, respectively. The paper additionally reports transfer without target-dataset fine-tuning to COCO-20i, Chest X-ray, FSS-1000, and LVIS-92i.

The main contribution is the broader category and application coverage of the benchmark, accompanied by a competitive foundation-model-based baseline.

Critical strengths

1. The benchmark addresses a meaningful limitation of existing FSS evaluation and could provide substantial community value.

VastFSS combines a large category vocabulary with applications spanning medical imaging, remote sensing, biology, and other specialized settings. This is a useful direction because adaptation across diverse target types and imaging conditions is not adequately characterized by benchmarks containing only a small number of natural-image categories. The construction involves both new segmentation annotations for V3Det-derived images and integration of existing specialized annotations. The category inventory, source documentation, and multi-round annotation-review procedure are positive features. With a well-validated taxonomy and evaluation protocol, the dataset itself could constitute a meaningful contribution.

2. MA-FSS is a coherent and competitive baseline, with evidence that its two guidance sources are complementary.

At inference time, MA-FSS specifies the target through support images and masks rather than requiring category names or target-specific descriptions. Table 2 reports improvements over SANSA, the strongest comparator by mean performance, of 3.5 and 1.8 percentage points in the 1-shot and 5-shot settings. Importantly, Appendix G already evaluates the contributions of the token and spatial-prior branches: token-only, prior-only, and combined variants obtain 54.0%, 44.4%, and 55.3% mIoU, respectively. These results support the usefulness of target-aware representations and the complementary contribution of spatial guidance.

Critical weaknesses

1. The semantic validity and independence of the benchmark’s evaluation splits are insufficiently established.

The paper explicitly ensures image-disjoint training and testing within each split, which is a positive step. However, category-ID disjointness does not necessarily imply semantic category disjointness. The appendix contains repeated category names across sources, such as antenna, bridge, plane, and building, as well as potentially synonymous labels such as inferior vena cava and IVC. It is unclear whether these are merged, treated as domain-specific categories, or constrained to remain in the same fold. Consequently, the interpretation of “2,400 categories” and “unseen-category generalization” requires a clearer canonical taxonomy and an explanation of how synonymous and hierarchical concepts are handled.

Furthermore, image-level separation alone may not eliminate correlations arising from the same patient, scan volume, video sequence, or geographic scene. The manuscript does not sufficiently document group-level separation or cross-source near-duplicate auditing. External transfer experiments similarly need an explicit audit of image/source/category overlap with VastFSS and a description of the training checkpoint used. Evaluation without target-dataset fine-tuning is meaningful, but it does not by itself establish transfer to unseen concepts or unseen domains. These issues directly affect the validity of the benchmark’s central claims.

2. The experiments do not adequately disentangle benchmark diversity, foundation-model strength, and method-specific contributions.

Table 2 does not sufficiently document which baselines were retrained on VastFSS, which used public checkpoints, and how their training budgets and model-selection procedures were determined. Since MA-FSS combines Qwen3-VL, DINOv2, and SAM 3, comparisons across different backbones and training regimes establish practical competitiveness but do not cleanly attribute improvements to the proposed design. A same-backbone baseline using non-language visual support representations would help establish the specific benefit of the MLLM-based representation pathway.

More importantly for a datasets-and-benchmarks submission, the paper attributes transfer performance to VastFSS’s broad category and domain coverage without controlled experiments separating diversity from image count and training compute. The current evidence establishes that MA-FSS trained on VastFSS transfers, but not that increasing category or domain coverage is responsible for the gains. Domain-level reporting is also important: approximately 85.7% of the categories are derived from V3Det, so overall category-averaged mIoU may obscure performance on specialized applications. Per-domain results and a targeted, matched-budget data-diversity comparison would substantially strengthen the contribution.

Questions for authors

The following questions concern issues that could materially change my assessment.

1. Category semantics and fold construction — clarification and audit. How are repeated names and synonymous concepts handled across sources, particularly antenna, bridge, building, and inferior vena cava/IVC? Are these merged or retained as distinct domain-specific labels, and can semantically equivalent concepts occur in different folds? Please provide or describe a canonical category-to-source-to-fold mapping and the policy for parent–child category relationships.

2. Data independence and external transfer — clarification and audit. Are splits and support–query sampling constrained at the patient, scan-volume, video-sequence, or geographic-scene level where applicable? What exact and near-duplicate checks were performed across sources? For the external benchmarks, what image/source/category overlaps exist with VastFSS, and which VastFSS checkpoint is used for transfer? Were target-benchmark results used in checkpoint or hyperparameter selection?

3. Baseline comparability and evidence for dataset diversity — clarification and additional evidence. Which methods in Table 2 were retrained on VastFSS, and what training data, backbones, budgets, and validation procedures were used? Can the authors provide domain-level results and a controlled comparison isolating category or domain diversity from image count and training compute? Such evidence would directly support the claimed benefit of VastFSS beyond its aggregate size.

4. Evaluation coverage and uncertainty — clarification and robustness evidence. Does “2,000 randomly sampled episodes” refer to each fold or the entire evaluation? Is sampling stratified to cover all 600 test categories per fold, and how are unsampled categories handled in category-averaged mIoU? Do all methods use identical episode lists? Please report variability across repeated episode sampling or training seeds, particularly for the smaller MA/ATG improvements. What validation split is used to select the number of learnable tokens and other hyperparameters?

7 comments
AuthorsOpenReview ↗

Response to Q1(a–b): Category Semantics and Data Independence

We thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of VastFSS’s potential value to the FSS community through its broad coverage of categories and applications, as well as your assessment of MA-FSS as a coherent and competitive baseline. We address your concerns point by point below.

Q1: Split validity and external transfer evaluation. (a) Category semantics and fold assignment. How are repeated category names, synonyms, and parent–child relationships handled when counting the 2,400 categories and assigning them to folds? (b) Data independence within VastFSS. Beyond image-level separation, what safeguards, if any, prevent images from the same patient, scan volume, video sequence, or geographic scene from appearing in both training and test sets? What checks were performed for group-level separation across sources? (c) External transfer evaluation. For each external benchmark, which VastFSS checkpoint was used? To what extent do its training data overlap with the target benchmark in images, data sources, or categories?

A1: Thank you for highlighting the distinction between category-ID separation and semantic separation, as well as the distinction between image-level and group-level independence. We clarify our construction protocol below.

To sub-question (a): Thank you for raising this point. For categories derived from V3Det, we use its taxonomy to guide fold assignment. In particular, synonymous and hierarchically related categories, including parent–child relationships, are assigned to the same fold to reduce semantic overlap between training and testing.

Across different source datasets, we retain separate category entries when they correspond to distinct segmentation settings. For example, antenna refers to a rooftop antenna in one dataset and a receiving antenna mounted on a spacecraft in another. Similarly, IVC and inferior vena cava denote the same anatomical structure but are annotated in surgical imagery and MRI slices, respectively. These entries are retained to capture differences in visual appearance and imaging conditions, rather than to treat them as semantically distinct concepts.

We clarify that the 2,400 categories in VastFSS represent segmentation categories defined by both semantic concepts and application-specific visual settings, rather than 2,400 entirely distinct semantic concepts. This design is intentional, as our goal is to evaluate the generalization of FSS models across both diverse object categories and real-world application scenarios. For categories sharing the same underlying semantic concept across different sources, the distinction lies in their imaging conditions, visual appearances, and segmentation contexts. Therefore, VastFSS provides a unified benchmark for evaluating generalization to novel semantic concepts as well as adaptation to different visual settings. To improve transparency, we will provide the complete category list for each fold, together with the corresponding source datasets in the revised appendix.

To sub-question (b): Thank you for asking us to clarify how we addressed potential correlations among samples. For datasets from which we directly sampled MRI slices or video frames, we selected at most one slice per patient or one frame per video sequence to minimize within-group correlations. For datasets containing multiple crops potentially originating from the same underlying scene, such as microscopy and remote-sensing imagery, we preferentially selected visually distinct samples to reduce redundancy. However, some source datasets provide only pre-extracted images without patient, scan-volume, sequence, or scene identifiers, making strict group-level independence difficult to verify. For these sources, we ensure image-disjoint training and testing within each evaluation split but cannot guarantee complete group-level separation.

During dataset construction, we also screened for potential duplicates using filenames and original image identifiers provided by the source datasets, supplemented by manual visual inspection to identify redundant or highly similar samples. While these procedures help reduce duplication, they cannot guarantee the detection of all cross-source near-duplicates, particularly when source identifiers are unavailable or inconsistent. We will clarify these screening procedures and their limitations in the revised appendix.

AuthorsOpenReview ↗

Response to Q1(c): External Transfer Evaluation and Category Overlap

Tab. A: Cross-dataset transfer comparison on COCO-20i20^i without target-dataset fine-tuning.

Training DatasetOverlap with COCOCOCO-20i20^i mIoU (%)
FSS-100037 / 8054.4
VastFSS (Folds 0, 2, 3)19 / 8064.2

To sub-question (c): Thank you for raising this question. For cross-dataset evaluation, we used the MA-FSS checkpoint trained on VastFSS Folds 0, 2, and 3, with Fold 1 held out. The model was evaluated on external benchmarks without any target-dataset fine-tuning.

For Chest X-ray, lung segmentation was not included as a target category in the VastFSS training data, and no images were collected from the Chest X-ray benchmark. MA-FSS achieves 87.7% mIoU under the 1-shot setting, demonstrating its ability to transfer to a target concept absent from its training vocabulary.

For COCO-20i20^i, FSS-1000, and LVIS-92i92^i, none was directly used as a source dataset during VastFSS construction. Nevertheless, we acknowledge that their category vocabularies partially overlap with VastFSS. Using normalized category-name matching, we identified 19/80 (23.8%) COCO categories, 20/240 (8.3%) FSS-1000 test categories, and 159/920 (17.3%) LVIS-92i92^i evaluation categories that overlap with the VastFSS training folds.

To further investigate whether category overlap accounts for the transfer performance, we additionally trained MA-FSS on all 1,000 FSS-1000 categories using the same architecture and evaluated it on COCO-20i20^i without target-dataset fine-tuning. As shown in Tab. A, FSS-1000 shares 37 categories with COCO-20i20^i, nearly twice the 19 matches found in the VastFSS training folds. Nevertheless, the VastFSS-trained model achieves 64.2% mIoU, outperforming the FSS-1000-trained model (54.4%) by 9.8 percentage points. These results suggest that the superior cross-dataset transfer performance cannot be explained solely by category-name overlap, providing further evidence of the benefits of VastFSS as a training benchmark.

We agree that the term "zero-shot transfer" may be misleading because some target categories overlap with the training vocabulary. We will therefore adopt the more precise term "cross-dataset transfer" in the revised manuscript and clarify the scope of the generalization claims.

We thank the reviewer again for the constructive feedback, which helps improve the transparency and reproducibility of VastFSS.

AuthorsOpenReview ↗

Response to Q2(a–b): Baseline Comparability and MLLM-Derived Representations

Q2: Separating the contributions of the model and the training data. (a) Baseline comparability . Which methods in Table 2 were retrained on VastFSS, which were evaluated using public checkpoints, and how were training budgets and checkpoints selected? (b) MLLM-derived representations. With the segmentation backbone held fixed, how do MLLM-derived target tokens compare with purely visual support representations? (c) Training-data diversity. Does broader category or application coverage improve transfer when the number of training images and the training budget are held fixed? (d) Performance by application area. How does performance vary across application areas, particularly those less represented in VastFSS?

Tab. B: Comparison of visual-only and MLLM-derived support representations on VastFSS under the 1-shot setting.

Support representationFold-0Fold-1Fold-2Fold-3Mean mIoU
Visual-only target tokens50.553.351.753.052.1
MLLM-derived target tokens (MA-FSS)54.056.954.455.855.3

A2: Thank you for these insightful comments. We agree that distinguishing the contributions of the model architecture and training-data diversity is important. We clarify the baseline evaluation settings and provide additional controlled experiments and domain-level analyses below.

To sub-question (a): Baseline comparability. All trainable baselines in Tab. 2 were retrained on the corresponding VastFSS training folds for five epochs. Each baseline was evaluated after every epoch, and the checkpoint with the highest mIoU was selected for reporting. Training-free methods were evaluated directly without parameter updates. MA-FSS was trained for 80,000 iterations with an effective batch size of four, corresponding to approximately 2.5 epochs under our training sampler.

Under these settings, MA-FSS achieves 55.3% mean mIoU, outperforming SANSA, the strongest competing baseline, by 3.5 percentage points in the 1-shot setting. We will clarify the training schedules, initialization settings, and checkpoint selection procedures in the revised manuscript.

To sub-question (b): MLLM-derived representations. Following the reviewer's suggestion, we conducted a controlled comparison by replacing the MLLM-derived target tokens with purely visual support representations. Specifically, we extracted features from the Qwen3-VL visual encoder, selected foreground features using the support mask, and projected them into target tokens without passing them through the language model. We retained the same DINOv2 spatial prior and SAM 3 segmentation components, using identical training settings and evaluation episodes for both variants.

As shown in Tab. B, MA-FSS achieves 55.3% mean mIoU, outperforming the visual-only variant (52.1%) by 3.2 percentage points, with consistent improvements across all four folds. With the spatial prior and segmentation backbone held fixed, these results demonstrate the benefit of MLLM-derived target-aware representations over directly extracted visual support features.

Comments on the paper

1 comment

Dear Reviewers (3V7C and pKV8) and Authors,

I would like to commend the authors for their exceptionally detailed and structured point-by-point responses to all reviewers' questions. I also thank Reviewer JPeq for engaging in a constructive follow-up discussion.

To Reviewers 3V7C and pKV8: As we approach the end of the discussion period, the authors have provided specific responses to your concerns. I kindly request that you take a moment to review their responses and leave a brief note on OpenReview. A simple confirmation of whether your concerns have been addressed is completely sufficient.

If I do not receive a follow-up before the discussion period closes, I will proceed with my evaluation based on the original reviews and the authors' written responses.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.