- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MK-FSS, a few-shot segmentation framework that supplements support-query visual matching with target-specific knowledge generated by a multimodal large language model (MLLM). Given a query image and its class label, the MLLM produces two complementary signals: spatial knowledge in the form of candidate target boxes and semantic knowledge describing the visible attributes of the target. The spatial prior is encoded as a memory representation and combined with a support-guided memory through a Dual-Memory Debate-Fusion (DMDF) module. In parallel, the semantic description is encoded and progressively fused with multi-scale query features through a Progressive Cross-Modal Prompt Generator (PCPG).
Critical strengths
-
A few support examples cannot represent the full range of query appearances, particularly under occlusion or in the presence of similar distractors. Generating query-specific spatial and semantic cues is therefore consequential, and the proposed two-path design uses these cues more substantively than treating an MLLM output as a final box or mask.
-
The authors provide component ablations, random-seed results, episode-count analysis, multiple MLLM backbones, and a direct comparison between class-specific and target-specific descriptions. These experiments make the core empirical claim considerably more credible than a result based only on two headline benchmark tables.
Critical weaknesses
-
MK-FSS appears to require per-query MLLM inference in addition to DINOv2, SAM 2, and the proposed modules. The paper does not provide a complete comparison of latency, memory, FLOPs, throughput, or monetary/API cost against the principal baselines. This is consequential because the main accuracy gains may come with a substantially different inference budget, limiting both practical relevance and fairness of comparison.
-
The model conditions the MLLM on the query image and ground-truth class label, yet there are no controlled experiments with missing boxes, shifted boxes, incorrect descriptions, hallucinated attributes, ambiguous labels, or label synonyms. Since these signals are directly injected into the representation, error propagation is central to the method’s soundness and real-world applicability.
Questions for authors
-
What are the end-to-end latency, peak memory, FLOPs, and, where applicable, API or inference cost per query for MK-FSS, including MLLM generation, DINOv2, text encoding, SAM 2, DMDF, and PCPG? Please compare these quantities with the strongest baselines under the same hardware and input resolution. This comparison would materially affect my assessment of the practical significance and fairness of the reported gains.
-
How robust is MK-FSS to imperfect MLLM knowledge? In particular, can the authors report controlled experiments that perturb or remove predicted boxes and corrupt or replace the semantic descriptions? It would also be useful to report the frequency and impact of MLLM localization failures on the two benchmarks.
-
How much does the method depend on access to a canonical ground-truth class name? Please evaluate label paraphrases, ambiguous/fine-grained labels, and a setting in which the target identity must be inferred only from the support image and mask. This determines whether the method addresses general FSS or a more restrictive label-assisted variant.
-
Can the authors provide a comparison that more carefully controls for pretrained-model capacity and external knowledge, for example simpler combinations using the same MLLM, DINOv2, and SAM 2 without DMDF/PCPG? This would help attribute the improvement to the proposed mechanisms rather than primarily to the additional foundation models.
6 comments
Response to Q2: Robustness to Imperfect MLLM Guidance
Q2: Robustness to imperfect MLLM guidance. Please provide controlled experiments with missing or shifted boxes, incorrect descriptions, hallucinated attributes, ambiguous labels, and label synonyms to assess error propagation and the method’s robustness and practical applicability.
Tab. B: Robustness to imperfect MLLM guidance on PASCAL-.
| Evaluation Condition | Perturbation Setting | Mean | ||||
|---|---|---|---|---|---|---|
| Original guidance | Unmodified bounding-boxes and descriptions | 85.0 | 89.3 | 86.1 | 87.8 | 87.1 |
| Missing boxes | Remove 30% of generated bounding-boxes | 81.1 | 81.6 | 82.4 | 83.0 | 82.0 |
| Shifted boxes | Shift bounding-boxes rightward by 10% of the image width | 82.9 | 85.5 | 82.7 | 82.2 | 83.3 |
| Incorrect descriptions | Replace with descriptions of an incorrect class | 84.1 | 87.5 | 83.2 | 84.6 | 84.8 |
| Hallucinated attributes | Replace descriptions with those of another image from the same class | 84.2 | 88.6 | 83.8 | 86.1 | 85.7 |
| Ambiguous labels | Regenerate knowledge using ambiguous target names, e.g., “aeroplane” → “vehicle” and “cow” → “animal” | 78.9 | 81.4 | 79.4 | 80.5 | 80.1 |
| Label synonyms | Regenerate knowledge using equivalent target names, e.g., “aeroplane” → “airplane” and “motorbike” → “motorcycle” | 85.5 | 88.4 | 85.6 | 86.8 | 86.6 |
| Anonymized labels | Infer the target category from the support sample and generate spatial and semantic knowledge without providing canonical class name | 82.7 | 85.8 | 81.4 | 79.5 | 82.3 |
A2: We thank the reviewer for highlighting the potential error propagation from imperfect MLLM guidance. To investigate this issue, we conducted controlled test-time experiments on PASCAL- using the same trained model and identical evaluation episodes. For spatial perturbations, we kept the original descriptions unchanged, while for semantic perturbations, we retained the original bounding boxes. For target-name variations, both spatial and semantic knowledge were regenerated. The results are reported in Tab. B and summarized below.
(i) Spatial guidance. Removing 30% of the generated bounding boxes reduces mIoU from 87.1 to 82.0, while shifting the boxes rightward by 10% of the image width yields 83.3. These results indicate that incomplete or displaced localization cues can affect segmentation accuracy. Nevertheless, MK-FSS maintains strong segmentation performance under both perturbations, suggesting that support-guided information can partially compensate for imperfect spatial guidance.
(ii) Semantic guidance. Replacing descriptions with those of an incorrect class yields 84.8 mIoU. Using descriptions from another image of the same class yields 85.7. This results shows that incorrected or hallucinated descriptions could reduce the segmentation performance of the model but do not dominate the prediction when the original spatial guidance and support information remain available.
(iii) Target-name variations. Equivalent category names yield 86.6 mIoU, only 0.5 points below the original setting. Ambiguous names reduce performance to 80.1 mIoU, as broader labels introduce uncertainty in target identification. Without canonical class names, the MLLM infers target identity from the support image and generates query-specific knowledge, achieving 82.3 mIoU.
Overall, these experiments demonstrate that MK-FSS maintains strong segmentation performance under various forms of imperfect MLLM guidance. Although spatial perturbations and ambiguous target names lead to relatively larger performance drops, the model still achieves favorable results across these challenging conditions, highlighting its robustness to inaccurate or incomplete guidance. We will incorporate these results and the detailed perturbation protocols into the revised manuscript. We thank the reviewer again for this constructive suggestion.
Response to Q3–Q4: End-to-End Inference Efficiency and Robustness to Localization Failures
Q3: Complete inference budget. Please report the end-to-end latency, GPU memory, FLOPs, and applicable inference costs of MK-FSS, and compare them with strong baseline under matched hardware and input resolution.
A3: Thank you for this question. As detailed in our response to Q1 and Tab. A, we evaluate both cached-knowledge inference and the complete online pipeline, including MLLM generation, DINOv2, text encoding, SAM 2, DMDF, and PCPG.
With knowledge cached, MK-FSS achieves higher segmentation accuracy than DSV-LFS while reducing latency from 0.518 to 0.153 s/query, GPU memory from 21.33 to 2.92 GB, and FLOPs from 22,642 to 745 G/query. Throughput increases from 1.930 to 6.541 queries/s. These results demonstrate the efficiency of our segmentation pipeline.
Including online knowledge generation, MK-FSS requires 2.017 s/query, 44.45 GB of aggregate GPU memory, and 9,173 GFLOPs/query. Under the common Qwen3-VL-8B-Instruct generation backend, the DSV-LFS proxy configuration requires 1.686 s/query, 62.20 GB, and 26,189 GFLOPs/query. MK-FSS therefore requires less aggregate memory and fewer FLOPs, with slightly higher latency when generating query-specific knowledge.
Overall, MK-FSS combines improved segmentation accuracy with an efficient pipeline. We will include these results and measurement details in the revised manuscript. Thank you again for this helpful comment.
Q4: Robustness to imperfect knowledge and localization failures. Please evaluate perturbed or missing boxes and corrupted semantic descriptions, and report the frequency and impact of MLLM localization failures on both benchmarks.
Tab. C: MLLM localization statistics on PASCAL- and COCO-.
| Dataset | Localization Failure Rate ↓ | Mean GT Coverage ↑ | Mean mIoU ↑ |
|---|---|---|---|
| PASCAL- | 1.90% | 96.63% | 87.1% |
| COCO- | 9.60% | 86.99% | 74.3% |
A4: Thank you for this question. We address it through controlled perturbation experiments and an analysis of occurring localization failures.
(i) Controlled perturbations. As detailed in our response to Q2 and Tab. B, we evaluated the trained model under imperfect guidance on PASCAL-. Removing 30% of the generated bounding-boxes and shifting bounding-boxes rightward by 10% of the image width yield 82.0 and 83.3 mIoU, respectively. Replacing descriptions with those of an incorrect class or another image from the same class yields 84.8 and 85.7 mIoU. These results demonstrate that MK-FSS maintains strong segmentation performance under the tested perturbations, highlighting its robustness to inaccurate or incomplete MLLM guidance.
(ii) Localization failures. We additionally analyzed the pre-generated MLLM bounding-boxes on both benchmarks. Tab. C reports the localization failure rate and mean ground-truth coverage.
The generated bounding boxes achieve higher target coverage and a lower failure rate on PASCAL- than on COCO-, indicating that localization remains less reliable in more challenging scenes. This difference in localization quality could further explain the segmentation performance gap between the two benchmarks, where MK-FSS achieves 87.1 and 74.3 mIoU, respectively.
We will include these additional experiments and the localization failure analysis in the revised manuscript. Thank you again for this helpful comment!
Response to Q1: End-to-End Inference Cost and Computational Efficiency
We thank the reviewer for the detailed and constructive feedback, and for recognizing the value of query-specific spatial and semantic knowledge, our two-path integration design, and the breadth of our experimental evaluation. We appreciate the concerns regarding computational cost, robustness to imperfect guidance, and dependence on class names. Below, we address each concern and present additional experimental results.
Tab. A: Computational cost with cached knowledge and with knowledge generation included.
A1: We thank the reviewer for highlighting the importance of complete inference-cost accounting. To assess whether MK-FSS’s accuracy gains are accompanied by a substantially larger computational budget, we conducted a supplementary comparison with the recent MLLM-based FSS method DSV-LFS, reporting both cached-knowledge inference and inference including online knowledge generation.
DSV-LFS uses GPT-generated class-level descriptions. Since the server-side memory usage and FLOPs of its original GPT service are not directly accessible, we use Qwen3-VL-8B-Instruct deployed through vLLM as a common generation backend. The online DSV-LFS configuration is therefore a proxy for estimating description-generation cost, rather than a measurement of its original GPT-based implementation.
The segmentation client and generation server run on separate NVIDIA A100-PCIE GPUs, each with 40 GB of memory. For online configurations, the reported memory is the sum of the client and server memory usages across the two devices, rather than the peak usage of a single GPU.
As shown in Tab. A, MK-FSS has lower latency, memory usage, and FLOPs, and higher throughput when knowledge is cached. Crucially, its computational advantages remain when online knowledge generation is included: MK-FSS requires 9,173 G FLOPs/query and 44.45 GB of aggregate GPU memory, compared with 26,189 G FLOPs/query and 62.2 GB for the DSV-LFS proxy configuration. Together with the higher segmentation accuracy reported in our evaluation, these results suggest that MK-FSS’s accuracy gains are not simply attributable to a larger FLOPs or memory budget.
We will include inference budget comparison in the revised manuscript, making the computational advantages and the online accuracy–latency trade-off explicit. Thank you again for this helpful suggestion.