acceptodds
Under review as a conference paper at ICLR 2027

Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation

Abstract

Few-shot segmentation (FSS) aims to segment unseen object categories with a few (e.g., one or five) labeled examples, enabling efficient adaptation to novel classes. Conventional models typically rely on appearance-based visual matching between support and query images for segmentation. While straightforward, these methods often struggle to handle significant appearance discrepancies and occlusions in the query image due to insufficient target knowledge. To mitigate this, we introduce a novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS. Specifically, building on SAM 2, our method, named MK-FSS, exploits two forms of complementary knowledge derived from a query image by an MLLM for FSS, including spatial knowledge, which provides a spatial prior indicating the potential target location, and semantic knowledge, which describes the target using text. The spatial knowledge is first encoded into a memory representation, and then resulting memory is integrated with the support-guided memory feature from query image through a carefully designed dual-memory debate-fusion (DMDF) module, yielding a more robust target memory feature. In parallel, the semantic knowledge is encoded into the textual feature, which is fused with multi-scale query features via a progressive cross-modal prompt generator (PCPG), producing a target-aware multimodal prompt for segmentation. Working together, the dual-memory feature and the multimodal prompt provide a comprehensive representation of the target, enabling more robust segmentation under complicated scenarios. In extensive experiments on multiple datasets, MK-FSS shows state-of-the-art results and largely surpasses existing approaches. Our code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Few-shot segmentation, Multimodal Large Language Models (MLLMs)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4ct3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MK-FSS, a few-shot segmentation framework that supplements support-query visual matching with target-specific knowledge generated by a multimodal large language model (MLLM). Given a query image and its class label, the MLLM produces two complementary signals: spatial knowledge in the form of candidate target boxes and semantic knowledge describing the visible attributes of the target. The spatial prior is encoded as a memory representation and combined with a support-guided memory through a Dual-Memory Debate-Fusion (DMDF) module. In parallel, the semantic description is encoded and progressively fused with multi-scale query features through a Progressive Cross-Modal Prompt Generator (PCPG).

Critical strengths
  1. A few support examples cannot represent the full range of query appearances, particularly under occlusion or in the presence of similar distractors. Generating query-specific spatial and semantic cues is therefore consequential, and the proposed two-path design uses these cues more substantively than treating an MLLM output as a final box or mask.

  2. The authors provide component ablations, random-seed results, episode-count analysis, multiple MLLM backbones, and a direct comparison between class-specific and target-specific descriptions. These experiments make the core empirical claim considerably more credible than a result based only on two headline benchmark tables.

Critical weaknesses
  1. MK-FSS appears to require per-query MLLM inference in addition to DINOv2, SAM 2, and the proposed modules. The paper does not provide a complete comparison of latency, memory, FLOPs, throughput, or monetary/API cost against the principal baselines. This is consequential because the main accuracy gains may come with a substantially different inference budget, limiting both practical relevance and fairness of comparison.

  2. The model conditions the MLLM on the query image and ground-truth class label, yet there are no controlled experiments with missing boxes, shifted boxes, incorrect descriptions, hallucinated attributes, ambiguous labels, or label synonyms. Since these signals are directly injected into the representation, error propagation is central to the method’s soundness and real-world applicability.

Questions for authors
  1. What are the end-to-end latency, peak memory, FLOPs, and, where applicable, API or inference cost per query for MK-FSS, including MLLM generation, DINOv2, text encoding, SAM 2, DMDF, and PCPG? Please compare these quantities with the strongest baselines under the same hardware and input resolution. This comparison would materially affect my assessment of the practical significance and fairness of the reported gains.

  2. How robust is MK-FSS to imperfect MLLM knowledge? In particular, can the authors report controlled experiments that perturb or remove predicted boxes and corrupt or replace the semantic descriptions? It would also be useful to report the frequency and impact of MLLM localization failures on the two benchmarks.

  3. How much does the method depend on access to a canonical ground-truth class name? Please evaluate label paraphrases, ambiguous/fine-grained labels, and a setting in which the target identity must be inferred only from the support image and mask. This determines whether the method addresses general FSS or a more restrictive label-assisted variant.

  4. Can the authors provide a comparison that more carefully controls for pretrained-model capacity and external knowledge, for example simpler combinations using the same MLLM, DINOv2, and SAM 2 without DMDF/PCPG? This would help attribute the improvement to the proposed mechanisms rather than primarily to the additional foundation models.

6 comments
AuthorsOpenReview ↗

Response to Q1: End-to-End Inference Cost and Computational Efficiency

We thank the reviewer for the detailed and constructive feedback, and for recognizing the value of query-specific spatial and semantic knowledge, our two-path integration design, and the breadth of our experimental evaluation. We appreciate the concerns regarding computational cost, robustness to imperfect guidance, and dependence on class names. Below, we address each concern and present additional experimental results.

Q1: End-to-end computational cost. Please report the complete inference cost of MK-FSS, including per-query MLLM generation, and compare latency, peak memory, FLOPs, throughput, or monetary/API cost where applicable with the principal baselines under matched evaluation conditions.

Tab. A: Computational cost with cached knowledge and with knowledge generation included.

MethodKnowledge SettingLatency (s/query) ↓Peak Memory (GB) ↓FLOPs (G/query) ↓Throughput (queries/s) ↑
DSV-LFSCached descriptions0.51821.3322,6421.930
DSV-LFSOnline description generation1.68662.226,1890.593
MK-FSSCached spatial and semantic knowledge0.1532.927456.541
MK-FSSOnline spatial and semantic knowledge generation2.01744.459,1730.496

A1: We thank the reviewer for highlighting the importance of complete inference-cost accounting. To assess whether MK-FSS’s accuracy gains are accompanied by a substantially larger computational budget, we conducted a supplementary comparison with the recent MLLM-based FSS method DSV-LFS, reporting both cached-knowledge inference and inference including online knowledge generation.

DSV-LFS uses GPT-generated class-level descriptions. Since the server-side memory usage and FLOPs of its original GPT service are not directly accessible, we use Qwen3-VL-8B-Instruct deployed through vLLM as a common generation backend. The online DSV-LFS configuration is therefore a proxy for estimating description-generation cost, rather than a measurement of its original GPT-based implementation.

The segmentation client and generation server run on separate NVIDIA A100-PCIE GPUs, each with 40 GB of memory. For online configurations, the reported memory is the sum of the client and server memory usages across the two devices, rather than the peak usage of a single GPU.

As shown in Tab. A, MK-FSS has lower latency, memory usage, and FLOPs, and higher throughput when knowledge is cached. Crucially, its computational advantages remain when online knowledge generation is included: MK-FSS requires 9,173 G FLOPs/query and 44.45 GB of aggregate GPU memory, compared with 26,189 G FLOPs/query and 62.2 GB for the DSV-LFS proxy configuration. Together with the higher segmentation accuracy reported in our evaluation, these results suggest that MK-FSS’s accuracy gains are not simply attributable to a larger FLOPs or memory budget.

We will include inference budget comparison in the revised manuscript, making the computational advantages and the online accuracy–latency trade-off explicit. Thank you again for this helpful suggestion.

AuthorsOpenReview ↗

Response to Q2: Robustness to Imperfect MLLM Guidance

Q2: Robustness to imperfect MLLM guidance. Please provide controlled experiments with missing or shifted boxes, incorrect descriptions, hallucinated attributes, ambiguous labels, and label synonyms to assess error propagation and the method’s robustness and practical applicability.

Tab. B: Robustness to imperfect MLLM guidance on PASCAL-5i5^i.

Evaluation ConditionPerturbation Setting505^0515^1525^2535^3Mean
Original guidanceUnmodified bounding-boxes and descriptions85.089.386.187.887.1
Missing boxesRemove 30% of generated bounding-boxes81.181.682.483.082.0
Shifted boxesShift bounding-boxes rightward by 10% of the image width82.985.582.782.283.3
Incorrect descriptionsReplace with descriptions of an incorrect class84.187.583.284.684.8
Hallucinated attributesReplace descriptions with those of another image from the same class84.288.683.886.185.7
Ambiguous labelsRegenerate knowledge using ambiguous target names, e.g., “aeroplane” → “vehicle” and “cow” → “animal”78.981.479.480.580.1
Label synonymsRegenerate knowledge using equivalent target names, e.g., “aeroplane” → “airplane” and “motorbike” → “motorcycle”85.588.485.686.886.6
Anonymized labelsInfer the target category from the support sample and generate spatial and semantic knowledge without providing canonical class name82.785.881.479.582.3

A2: We thank the reviewer for highlighting the potential error propagation from imperfect MLLM guidance. To investigate this issue, we conducted controlled test-time experiments on PASCAL-5i5^i using the same trained model and identical evaluation episodes. For spatial perturbations, we kept the original descriptions unchanged, while for semantic perturbations, we retained the original bounding boxes. For target-name variations, both spatial and semantic knowledge were regenerated. The results are reported in Tab. B and summarized below.

(i) Spatial guidance. Removing 30% of the generated bounding boxes reduces mIoU from 87.1 to 82.0, while shifting the boxes rightward by 10% of the image width yields 83.3. These results indicate that incomplete or displaced localization cues can affect segmentation accuracy. Nevertheless, MK-FSS maintains strong segmentation performance under both perturbations, suggesting that support-guided information can partially compensate for imperfect spatial guidance.

(ii) Semantic guidance. Replacing descriptions with those of an incorrect class yields 84.8 mIoU. Using descriptions from another image of the same class yields 85.7. This results shows that incorrected or hallucinated descriptions could reduce the segmentation performance of the model but do not dominate the prediction when the original spatial guidance and support information remain available.

(iii) Target-name variations. Equivalent category names yield 86.6 mIoU, only 0.5 points below the original setting. Ambiguous names reduce performance to 80.1 mIoU, as broader labels introduce uncertainty in target identification. Without canonical class names, the MLLM infers target identity from the support image and generates query-specific knowledge, achieving 82.3 mIoU.

Overall, these experiments demonstrate that MK-FSS maintains strong segmentation performance under various forms of imperfect MLLM guidance. Although spatial perturbations and ambiguous target names lead to relatively larger performance drops, the model still achieves favorable results across these challenging conditions, highlighting its robustness to inaccurate or incomplete guidance. We will incorporate these results and the detailed perturbation protocols into the revised manuscript. We thank the reviewer again for this constructive suggestion.

AuthorsOpenReview ↗

Response to Q3–Q4: End-to-End Inference Efficiency and Robustness to Localization Failures

Q3: Complete inference budget. Please report the end-to-end latency, GPU memory, FLOPs, and applicable inference costs of MK-FSS, and compare them with strong baseline under matched hardware and input resolution.

A3: Thank you for this question. As detailed in our response to Q1 and Tab. A, we evaluate both cached-knowledge inference and the complete online pipeline, including MLLM generation, DINOv2, text encoding, SAM 2, DMDF, and PCPG.

With knowledge cached, MK-FSS achieves higher segmentation accuracy than DSV-LFS while reducing latency from 0.518 to 0.153 s/query, GPU memory from 21.33 to 2.92 GB, and FLOPs from 22,642 to 745 G/query. Throughput increases from 1.930 to 6.541 queries/s. These results demonstrate the efficiency of our segmentation pipeline.

Including online knowledge generation, MK-FSS requires 2.017 s/query, 44.45 GB of aggregate GPU memory, and 9,173 GFLOPs/query. Under the common Qwen3-VL-8B-Instruct generation backend, the DSV-LFS proxy configuration requires 1.686 s/query, 62.20 GB, and 26,189 GFLOPs/query. MK-FSS therefore requires less aggregate memory and fewer FLOPs, with slightly higher latency when generating query-specific knowledge.

Overall, MK-FSS combines improved segmentation accuracy with an efficient pipeline. We will include these results and measurement details in the revised manuscript. Thank you again for this helpful comment.

Q4: Robustness to imperfect knowledge and localization failures. Please evaluate perturbed or missing boxes and corrupted semantic descriptions, and report the frequency and impact of MLLM localization failures on both benchmarks.

Tab. C: MLLM localization statistics on PASCAL-5i5^i and COCO-20i20^i.

DatasetLocalization Failure Rate ↓Mean GT Coverage ↑Mean mIoU ↑
PASCAL-5i5^i1.90%96.63%87.1%
COCO-20i20^i9.60%86.99%74.3%

A4: Thank you for this question. We address it through controlled perturbation experiments and an analysis of occurring localization failures.

(i) Controlled perturbations. As detailed in our response to Q2 and Tab. B, we evaluated the trained model under imperfect guidance on PASCAL-5i5^i. Removing 30% of the generated bounding-boxes and shifting bounding-boxes rightward by 10% of the image width yield 82.0 and 83.3 mIoU, respectively. Replacing descriptions with those of an incorrect class or another image from the same class yields 84.8 and 85.7 mIoU. These results demonstrate that MK-FSS maintains strong segmentation performance under the tested perturbations, highlighting its robustness to inaccurate or incomplete MLLM guidance.

(ii) Localization failures. We additionally analyzed the pre-generated MLLM bounding-boxes on both benchmarks. Tab. C reports the localization failure rate and mean ground-truth coverage.

The generated bounding boxes achieve higher target coverage and a lower failure rate on PASCAL-5i5^i than on COCO-20i20^i, indicating that localization remains less reliable in more challenging scenes. This difference in localization quality could further explain the segmentation performance gap between the two benchmarks, where MK-FSS achieves 87.1 and 74.3 mIoU, respectively.

We will include these additional experiments and the localization failure analysis in the revised manuscript. Thank you again for this helpful comment!

Reviewer_r4yMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MK-FSS addresses few-shot segmentation with access to target-class names by extracting bounding boxes and image-specific descriptions from a multimodal large language model. It incorporates the boxes into SAM 2 query memories through dual-memory debate-fusion and converts the descriptions into multimodal prompts through progressive cross-modal fusion. Evaluation covers one-shot and five-shot segmentation, component ablations, and supplementary analyses of evaluation seeds, episode counts, and alternative language models.

Critical strengths
  1. MK-FSS reports one-shot mIoU scores of 87.1% on PASCAL-5i and 74.3% on COCO-20i, with five-shot scores of 87.7% and 75.9%. The one-shot results exceed the reported 81.0% of FSSAM on PASCAL-5i and 71.3% of DSV-LFS on COCO-20i, providing a substantial system-level improvement under the explicitly stated setting with target-class names.
  2. The ablations demonstrate complementary contributions from the information sources within the proposed system. Adding the spatial-memory branch raises PASCAL-5i mIoU from 77.3% to 86.0%, and including the semantic branch raises it to 87.1%, while combining both memories improves over the 85.8% obtained with MLLM-derived memory alone.
Critical weaknesses
  1. The claims that the complete DMDF and progressive PCPG designs are crucial rest on relatively small differences, including 0.5 points over partial DMDF variants, 0.6 points over learnable memory addition, and 0.4 points over learnable multiscale addition. Appendix A.4 repeats only the complete model and therefore does not quantify uncertainty in these differences. The reported results favor the proposed designs, but their benefit is less firmly established than the larger benefit of introducing the spatial-knowledge branch.
  2. DMDF’s interpretation as a verification mechanism requires evidence that it handles misleading guidance. CMX [ref.1] already combines bilateral cross-modal exchange with pooled channel reweighting, although DMDF uses different inputs, attention formulation, and ordering. The accuracy ablations and qualitative examples do not measure behavior specifically when predicted boxes are incorrect or replaced by the full-image fallback. This limits support for the claimed verification benefit when the two memories provide conflicting guidance.

REF:

  • ref.1: Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023). CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers. IEEE Transactions on intelligent transportation systems, 24(12), 14679-14694.
Questions for authors
  1. With Qwen3-VL-8B-Instruct fixed as the generator for both class-level and query-specific descriptions, does the 0.7-point improvement in Table 13 persist?
  2. Do the DMDF and PCPG improvements over learnable addition persist across paired repetitions using identical evaluation episodes, with training randomness and episode-sampling randomness explicitly distinguished?
  3. On existing test episodes with inaccurate MLLM boxes or the full-image fallback, how does DMDF compare with learnable memory addition?
5 comments
AuthorsOpenReview ↗

Response to Q1: Statistical Reliability of DMDF and PCPG Improvements

We thank the reviewer for the detailed and constructive feedback, and for recognizing the substantial performance improvements and the complementary contributions of the spatial and semantic knowledge sources. Below, we provide point-by-point responses to all the concerns.

Q1: Statistical reliability of module improvements. The gains from DMDF and PCPG over partial variants and learnable addition are modest, while repeated evaluations cover only the complete model. Please quantify the uncertainty in these improvements through paired repetitions.

Tab. A: Paired multi-seed comparison of DMDF and learnable addition on PASCAL-5i5^i.

SeedLearnable AdditionDMDFGain
085.786.60.9
186.487.00.6
285.886.30.5
386.186.80.7
32186.587.10.6
Mean86.186.80.7

Tab. B: Paired multi-seed comparison of PCPG and learnable addition on PASCAL-5i5^i.

SeedLearnable AdditionPCPGGain
085.986.60.7
186.387.00.7
285.886.30.5
386.686.80.2
32186.787.10.4
Mean86.386.80.5

Tab. C: Ablation on the comparison of DMDF with learnable addition strategy on COCO-20i20^i.

Memory fusion20020^020120^120220^220320^3Mean mIoU (%)
Learnable addition71.074.875.367.672.2
DMDF73.276.675.971.674.3

Tab. D: Ablation on the comparison of PCPG with learnable addition strategy on COCO-20i20^i.

Multi-scale Cross-modal Fusion20020^020120^120220^220320^3Mean mIoU (%)
Learnable addition70.774.574.569.172.2
PCPG73.276.675.971.674.3

A1: Thank you for this important observation. We agree that repeating only the complete model does not quantify the uncertainty in the module-level improvements. We therefore extended the evaluation to the learnable addition baselines and conducted paired comparisons across five seeds on PASCAL-5i5^i. Within each seed, all configurations were evaluated on identical episodes using the same MLLM-generated knowledge, with only the corresponding fusion mechanism changed.

As shown in Tabs. A and B, DMDF outperforms learnable memory addition across all five seeds, with gains ranging from 0.5 to 0.9 percentage points. The mean paired improvement is 0.66 ± 0.15 points. Similarly, PCPG consistently outperforms learnable multi-scale addition, with gains ranging from 0.2 to 0.7 points and a mean paired improvement of 0.50 ± 0.21 points. Here, the standard deviations are computed across the five paired differences. These results show that the improvements are consistent across the tested seeds rather than arising from a single favorable evaluation.

To further assess the effectiveness of both modules, we conducted additional controlled comparisons on the more challenging COCO-20i20^i dataset. As shown in Tabs. C and D, DMDF and PCPG each improve mean mIoU from 72.2 to 74.3, outperforming their respective learnable addition baselines across all four folds. These comparisons complement the PASCAL-5i5^i results and provide further evidence for the benefits of the proposed fusion designs.

We will include the additional comparisons and explicitly describe the seed settings in the revised manuscript. Thank you again for this constructive comment.

AuthorsOpenReview ↗

Response to Q2: DMDF Design and Robustness under Misleading Guidance

Q2: Distinction from CMX and verification under misleading guidance. (a) Clarify how DMDF differs from CMX’s bilateral cross-modal exchange with channel reweighting, and (b) provide controlled comparisons under inaccurate MLLM boxes and full-image fallback to substantiate its verification capability.

Tab. E: Comparison of DMDF with CMX-style fusion on PASCAL-5i5^i.

Memory Fusion505^0515^1525^2535^3Mean mIoU (%)
CMX-style fusion84.589.082.986.885.8
DMDF85.089.386.187.887.1

Tab. F: Comparison of memory fusion strategies under imperfect spatial guidance on PASCAL-5i5^i. Results are mean mIoU (%).

Spatial GuidanceLearnable AdditionDMDFGain
Original boxes86.587.10.6
Shift bounding-boxes rightward by 10% of image width81.783.31.6
Shift bounding-boxes rightward by 20% of image width73.176.73.6
Full-image fallback71.873.31.5

A2: To sub-question (a): Thank you for highlighting this connection. While both methods use bilateral interaction and channel reweighting, DMDF first exchanges information between the two memories and then weights the updated representations before fusion, retaining both original memories through residual integration. This differs from CMX’s input-level channel–spatial rectification and subsequent fusion design.

To assess these design differences, we replaced DMDF with CMX-style fusion while retaining the same input memories, MLLM knowledge, and remaining framework. As shown in Tab. E, DMDF achieves 87.1 mIoU, compared with 85.8 for CMX-style fusion. This result supports the effectiveness of our design for the dual-memory fusion task. We will clarify these differences and include the comparison in the revised manuscript.

To sub-question (b): Thank you for this suggestion. We compared DMDF with learnable addition under identical bounding-box perturbations and full-image fallback, keeping the remaining inputs unchanged. As shown in Tab. F, DMDF improves mIoU by 1.6 and 3.6 percentage points under 10% and 20% rightward shifts, respectively, and by 1.5 points under full-image fallback. These gains exceed the 0.6-point improvement with unmodified bounding-boxes, and DMDF also exhibits smaller performance drops from its own unperturbed baseline.

These results support DMDF’s ability to better integrate the two memories when spatial guidance is inaccurate or uninformative. We clarify that “verification” denotes adaptive feature weighting rather than an explicit guarantee of localization correctness. We will include these results and the clarification in the revised manuscript.

AuthorsOpenReview ↗

Response to Q3–Q5: Description Specificity, Statistical Reliability, and Robustness

Q3: Description specificity with a fixed generator. Does the improvement from query-specific descriptions persist when both class-level and query-specific descriptions are generated by Qwen3-VL-8B-Instruct?

Tab. G: Performance comparison under different language description configurations on PASCAL-5i5^i.

Language Description505^0515^1525^2535^3Mean mIoU (%)
Class-specific (Qwen3-VL-8B-Instruct)84.788.284.586.786.0
Class-specific (ChatGPT-4.0)84.789.284.487.286.4
Target-specific (Qwen3-VL-8B-Instruct)85.089.386.187.887.1

A3: Thank you for identifying this potential confounding factor. We conducted an additional controlled experiment using Qwen3-VL-8B-Instruct to generate both class-specific and target-specific descriptions. For class-specific descriptions, we followed the prompt provided by DSV-LFS, while keeping the remaining experimental settings unchanged.

As shown in Tab. G, class-specific descriptions generated by Qwen achieve 86.0 mIoU, whereas target-specific descriptions generated by the same model achieve 87.1 mIoU. This represents a consistent improvement of 1.1 percentage points across all four folds, demonstrating that the advantage of target-specific descriptions persists when the description generator is held fixed.

We also observe that Qwen-generated class-specific descriptions perform slightly worse than the ChatGPT-generated counterparts (86.0 vs. 86.4 mIoU), potentially due to differences in their language generation capabilities. Nevertheless, target-specific descriptions outperform both class-specific alternatives, further supporting the effectiveness of incorporating query-specific semantic information, independent of the choice of description generator.

Thank you again for this helpful comment, we will include this controlled comparison in the revised manuscript.

Q4: Paired evaluation and sources of randomness. Do the improvements of DMDF and PCPG persist across paired repetitions, with training and episode-sampling randomness explicitly distinguished?

Tab. H: Mean mIoU and standard deviation across three independent training runs on PASCAL-5i5^i.

Memory FusionMulti-scale Cross-modal FusionMean mIoU (%)Std
Learnable additionPCPG85.90.5
DMDFLearnable addition86.50.2
DMDFPCPG87.00.3

A4: Thank you for this valuable question. As reported in our response to Q1 and Tabs. A–B, both DMDF and PCPG consistently outperform learnable addition across five paired evaluation seeds, with identical test episodes used for each comparison. The mean paired improvements are 0.66 ± 0.15 percentage points for DMDF and 0.50 ± 0.21 for PCPG, where the standard deviations are computed across paired performance differences.

To further evaluate training stability, we conducted three independent training runs for each configuration, while keeping the MLLM-generated knowledge and evaluation episodes fixed. As shown in Tab. H, the complete MK-FSS achieves 87.0 ± 0.3 mIoU, outperforming the variants without DMDF (85.9 ± 0.5) and without PCPG (86.5 ± 0.2) by 1.1 and 0.5 percentage points, respectively.

Together, these results demonstrate that the improvements introduced by DMDF and PCPG remain consistent under different evaluation episodes and independent training runs, providing further evidence of their effectiveness and stability. We will incorporate these results and clarify the corresponding evaluation protocols in the revised manuscript. We thank the reviewer again for this constructive suggestion.

Q5: Fusion under inaccurate localization and fallback. How does DMDF compare with learnable addition when MLLM boxes are inaccurate or replaced by the full-image fallback?

A5: Thank you for this question. Please refer to our response to Q2(b) and Tab. F. Under controlled rightward box shifts of 10% and 20% of the image width, DMDF outperforms learnable addition by 1.6 and 3.6 percentage points, respectively. With a full-image prior, the improvement is 1.5 points. These gains exceed the 0.6-point improvement under unmodified guidance, supporting the benefit of DMDF when spatial guidance is unreliable.

Once again, we deeply appreciate the reviewer's constructive feedback. We have taken all suggestions seriously and would revise the paper to improve clarity, rigor, and transparency.

Reviewer_rWZ7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MK-FSS, a few-shot segmentation framework built on SAM 2. Given a query image and the target class name, an MLLM produces bounding boxes and an image-specific textual description. The proposed DMDF module fuses a memory derived from those boxes with a support-guided memory; PCPG injects the textual description into multiscale query features. The method reports 87.1%/87.7% mIoU on PASCAL-5ᶦ and 74.3%/75.9% on COCO-20ᶦ in the one-shot/five-shot settings, respectively. Ablations, alternative MLLMs, and additional episode-sampling results are provided.

Critical strengths

(i) Strong results on two established benchmarks. The reported gains over prior methods are substantial, particularly in the one-shot setting. The paper also evaluates multiple MLLMs and provides fold-level results and ablations, making the empirical case broader than a single headline comparison.

(ii) A useful way to combine coarse localization with segmentation features. The method addresses a concrete limitation of support–query appearance matching. Table 3 shows that MLLM-derived spatial guidance provides a large gain, while the textual branch adds a further improvement.

Critical weaknesses

(i) The evaluation does not establish that the reported gains come from few-shot adaptation rather than class-name-conditioned MLLM recognition. At inference time, the MLLM receives both the query image and the exact target class name, and is explicitly asked to locate all instances and describe their visible attributes. Thus, the MLLM already performs most of the semantic identification and coarse localization before the support example is used. This concern is supported by Table 4: under that setting, the MLLM-derived memory alone reaches 85.8 mIoU, only 1.3 points below the complete model at 87.1. Moreover, increasing from 1-shot to 5-shot improves the full model by only 0.6 points on PASCAL-5i5^{i} and 1.6 points on COCO-20i20^{i}. The submission provides no support-free, incorrect-support, class-name-free, or anonymized-label control. Because PASCAL and COCO contain common categories that a large pretrained MLLM is already expected to recognize, the current results may primarily measure open-vocabulary query segmentation rather than robust transfer from a few annotated supports. This directly weakens the paper’s central FSS claim.

(ii) The evidence for the novelty and necessity of DMDF and PCPG is limited once the MLLM information is held fixed. The large gain in Table 3 from adding DMDF also introduces the MLLM-derived spatial prior, so it does not isolate the module from the extra information source. When identical memories are compared in Table 6, DMDF improves over learnable addition by only 0.6 mIoU (87.1 versus 86.5); similarly, the progressive PCPG fusion improves over learnable addition by only 0.4 mIoU in Table 7 (87.1 versus 86.7). Architecturally, DMDF consists of bidirectional cross-attention, channel-wise weighting, convolution, and addition, while PCPG uses standard text-vision cross-attention and hierarchical weighted fusion. Calling the former a “debate” mechanism does not itself create an iterative disagreement-resolution process analogous to multi-agent debate. Without stronger equal-information/equal-capacity baselines and statistical evidence on both datasets, it is unclear whether the proposed modules—not simply the strong Qwen3-VL priors—constitute the main technical contribution.

Questions for authors

In Table 4, does the �0��0�-only setting completely remove all information from the support image and mask? If so, the resulting 85.8 mIoU is only 1.3 points below the full model, suggesting limited dependence on few-shot support. Please clarify this setting and additionally evaluate a conflict condition in which each query is paired with a support image/mask from an incorrect class while retaining the correct target class name. The resulting performance change may help determine whether MK-FSS actually follows the few-shot support or mainly relies on class-conditioned MLLM localization.

3 comments
AuthorsOpenReview ↗

Response to Q1: Support Dependence and Class-Name-Free Evaluation

We sincerely thank the reviewer for the thoughtful and constructive feedback. We greatly appreciate the reviewer's recognition of the clarity of our manuscript and the soundness of our proposed framework. Below, we provide detailed responses to each of the concerns the reviewer raised.

Q1: Support dependence versus class-name-conditioned recognition. The current evaluation does not clearly establish whether the gains arise from support-based transfer or class-name-conditioned MLLM recognition. Please provide controlled comparisons under support-free, incorrect-support, and class-name-free conditions to assess the contribution of support information.

Tab. A: Performance under different support and class-name conditions on PASCAL-5i5^i.

Evaluation Condition505^0515^1525^2535^3Mean mIoU (%)
Full model85.089.386.187.887.1
Support-free72.880.973.571.374.6
Incorrect support81.383.174.277.779.1
Class-name-free82.785.881.479.582.3

A1: We thank the reviewer for this important concern. To assess the model’s dependence on support information, we conducted additional controlled evaluations on PASCAL-5i5^i using the trained full model and identical query episodes.

In the support-free condition, we removed support information while retaining the correct target class name and the original MLLM outputs. In the incorrect-support condition, we replaced the support with an image and its corresponding mask from an incorrect class, again retaining the correct class name and MLLM outputs. In the class-name-free condition, no canonical class name was provided. Instead, the MLLM inferred the target category from the support image and generated the corresponding spatial and semantic knowledge for the query image.

As shown in Tab. A, removing support information reduces mIoU by 12.5 percentage points, while incorrect support reduces it by 8.0 points. Both interventions degrade performance across all four folds, showing that our model depends on support information even when correct class-conditioned MLLM guidance remains available. Without canonical class names, the support-inferred setting still achieves 82.3 mIoU, demonstrating that the framework can operate when target identity is inferred from support rather than explicitly supplied.

Taken together, these results show that MK-FSS meaningfully uses support information even when correct class-conditioned MLLM guidance is available, and can operate without canonical class names by inferring target identity from support. This addresses the concern that MK-FSS merely performs class-name-conditioned query segmentation, while acknowledging the substantial contribution of MLLM-derived knowledge.

Thanks again for this helpful comment. We will include these clarifications and additional experimental results in the revision.

AuthorsOpenReview ↗

Response to Q2: DMDF and PCPG Contributions

Q2: Module contributions beyond MLLM priors. The current evidence does not sufficiently establish the benefits of the proposed modules independently of MLLM knowledge. (a) Validate DMDF against equal-information and capacity-matched baselines, (b) similarly establish the effectiveness of PCPG with statistical evidence, and (c) clarify how DMDF relates to multi-agent debate given the absence of explicit iterative disagreement resolution.

Tab. B: Ablation on the comparison of DMDF with capacity-matched learnable addition strategy on COCO-20i20^i.

Memory fusion20020^020120^120220^220320^3Mean mIoU (%)
Learnable addition71.074.875.367.672.2
DMDF73.276.675.971.674.3

Tab. C: Ablation on the comparison of PCPG with capacity-matched learnable addition strategy on COCO-20i20^i.

Multi-scale Cross-modal Fusion20020^020120^120220^220320^3Mean mIoU (%)
Learnable addition70.774.574.569.172.2
PCPG73.276.675.971.674.3

Tab. D: Ablation on the iterative number of DMDF on PASCAL-5i5^i.

Iterative number505^0515^1525^2535^3Mean mIoU (%)
185.089.386.187.887.1
284.888.785.887.086.6

A2: Thanks for this insightful comment.

To sub-question (a): To control for model capacity, we augment the learnable addition baseline with linear projections so that its trainable parameter count approximately matches that of DMDF. We further evaluate both configurations on the more challenging COCO-20i20^i dataset, using identical MLLM-generated knowledge and keeping the remaining experimental settings unchanged.

As shown in Tab. B, DMDF achieves 74.3 mIoU, outperforming the capacity-matched baseline by 2.1 percentage points, with improvements across all four folds. These results provide further evidence for the effectiveness of DMDF beyond the benefits of additional MLLM knowledge or increased parameter capacity.

To sub-question (b): We similarly compare PCPG with a capacity-matched learnable multi-scale addition baseline, retaining the same text-conditioned features and the remaining framework. As shown in Tab. C, PCPG improves mean mIoU from 72.2 to 74.3, with improvements across all four folds. This supports the benefit of progressive multi-scale integration over direct weighted aggregation.

To sub-question (c): DMDF is inspired by multi-agent debate, particularly the exchange of complementary information to refine individual representations. We translate this principle into bidirectional interaction between support-guided and MLLM-derived memories, followed by adaptive weighting and integration. The analogy concerns information exchange rather than an explicit multi-agent reasoning process.

We also explored repeated refinement by evaluating a two-round variant. As shown in Tab. D, increasing the number of rounds from one to two reduces mIoU from 87.1 to 86.6. Additional iteration therefore provides no benefit in this experiment, supporting the original single-round design. We will clarify this motivation and include the iteration ablation in the revised manuscript.

Thank you again for this constructive comment. We will incorporate these clarifications and additional experimental results into the revised manuscript.

AuthorsOpenReview ↗

Response to Q3: Support Information in Memory Ablation

Q3: Support information in the memory ablation. (a) Clarify whether the MLLM-memory-only variant in Table 4 completely removes all support information, and (b) evaluate incorrect-class support while retaining the correct target class name to assess the model’s dependence on support information.

A3: Thanks for this insightful comment and pointing out the ambiguity.

To sub-question (a): The “FmemMLLMF_{mem}^{MLLM}-only” setting in Table 4 does not completely remove all information derived from the support image and mask. It removes the support-derived memory branch at the DMDF fusion stage, while retaining the downstream memory-attention mechanism. In addition, support-derived auxiliary information, such as object-pointer tokens when enabled, remains available to the decoder. Therefore, this setting is a branch-level ablation rather than a fully support-free evaluation. We acknowledge that the term “FmemMLLMF_{mem}^{MLLM}-only” does not clearly convey this distinction.

To sub-question (b): Thanks for this careful comment. To assess the model’s dependence on support information, we evaluated the trained full model with support images and their corresponding masks from incorrect classes, while retaining the correct target class names and the original MLLM-generated knowledge. As reported in Tab. A, mIoU decreases from 87.1 to 79.1, with performance drops across all four folds. These results demonstrate that support-guided and MLLM-guided knowledge jointly contribute to the segmentation process.

Once again, we thank the reviewer for all the constructive feedback, which has significantly helped improve the clarity and rigor of our paper.

Comments on the paper

1 comment

Dear Reviewers rWZ7 and 4ct3,

The author–reviewer discussion closes in less than a day (November 18, AoE). The authors have posted detailed responses with new experiments addressing your reviews. Please reply now: say which concerns are resolved and which aren’t, and update your review or rating if your assessment changed.

Thank you,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.