acceptodds
Under review as a conference paper at ICLR 2027

MAD-SP: Multi-Agent Dynamic Semantic Prompting Framework for Few-Shot Image Recognition

Abstract

Few-shot learning (FSL) recognizes novel classes from few labeled examples, but scarce support data yield unstable visual representations. Multimodal methods alleviate this limitation by incorporating textual semantics as prompts, but LLM-generated captions may be overly verbose and redundant. To address these issues, we propose Multi-Agent Dynamic Semantic Prompting (MAD-SP), which improves both the construction and utilization of semantic prompts. First, Multi-Agent Caption Refinement (MACR) coordinates generator, critic, and refiner agents under CLIP-based alignment and discriminative scoring to produce concise, visually grounded captions, thereby providing higher-quality semantic prompts for visual recognition. Building on these refined prompts, Dynamic Prompt Injection (DPI) enables their instance-adaptive utilization by employing a lightweight learnable decision network to estimate the compatibility between each semantic prompt and visual features at different blocks, thereby selecting an appropriate injection layer for each image. To ensure reliable alignment between semantic prompts and layer-specific visual representations during varying-depth injection, Contrastive Modality Alignment (CMA) further supports DPI by introducing a bottleneck alignment adapter after the text encoder and optimizing it with the InfoNCE loss. Together, MACR improves the quality of semantic prompts, while DPI and CMA enable their effective and adaptive integration into the visual encoder. Extensive experiments show that MAD-SP achieves state-of-the-art results in all eight settings on four standard benchmarks and in seven of eight settings across four cross-domain benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Learning, Semantic Prompting, Multi-Agent Collaboration, Dynamic Prompt Injection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CvVwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Objective: The paper address few-shot image classification by resolving the noise/verbosity of LLM-generated captions and overcoming the inflexibility of fixed-layer prompt injection in Vision Transformers.
  • Methodology: The paper introduces a framework using Visformer-T and CLIP text encoder:
    1. MACR (Multi-Agent Caption Refinement): Coordinates Generator, Critic, and Refiner LLM agents (powered by Qwen3-VL-8B) under CLIP-based image-text alignment and discriminative margin scoring to produce visually grounded prompts.
    2. DPI (Dynamic Prompt Injection): Predicts layer compatibility probabilities across Visformer blocks and uses a Straight-Through Estimator (STE) gate to dynamically inject prompts spatially and channel-wise per image.
    3. CMA (Contrastive Modality Alignment): Uses a bottleneck MLP adapter and class-level InfoNCE loss to align prompt prototypes with visual prototypes.
  • Key Findings & Empirical Validation: The proposed framework is evaluated across 4 standard FSL benchmarks (miniImageNet, tieredImageNet, CIFAR-FS, FC100) and 4 cross-domain benchmarks (CUB, Places, Plantae, ChestX) under 1-shot and 5-shot protocols over 2,000 test episodes. Sets new state-of-the-art results in 8/8 standard settings (+2.72% average gain) and 7/8 cross-domain settings (+1.97% average gain).
Critical strengths
  • The paper effectively bridges closed-loop multi-agent prompt evolution with instance-adaptive Transformer layer routing. DPI addresses visual scale abstraction by dynamically routing structural scenes to shallow layers and fine-grained targets to deep layers.
  • The paper shows exceptional performance across 16 benchmark settings with 2,000 test episodes and reported 95% confidence intervals. Comprehensive ablations validate the contribution of MACR (+7.01% 1-shot gain), DPI (+6.62% 1-shot gain), and CMA, supported by t-SNE clustering and attention map visualizations.
Critical weaknesses
  • Regarding the MACR module, it runs a 3-agent LLM debate loop, so this process is completed before meta-training begins. Referring to it as a "dynamic multi-agent system" is slightly misleading, it is essentially an offline prompt pre-processing step.
  • Evaluation is restricted to a lightweight Visformer-T backbone (10M parameters). There is no empirical proof that DPI and CMA generalize to standard ViT-S/16, ViT-B/16, or Swin Transformers. Furthermore, discrete STE gating can suffer from gradient variance and layer-collapse risks during meta-training.
Questions for authors
  1. What is the total wall-clock pre-processing time and GPU cost for MACR caption refinement?
  2. How does MAD-SP perform if MACR is replaced by a single-pass VLM captioner (e.g., LLaVA-1.5 or BLIP-2) or standard class-name prompts?
  3. Does DPI and CMA module generalize to standard vision backbones (e.g., ViT-S/16, ViT-B/16, or Swin-T)? Can you provide results on ViT-B/16?
  4. When evaluating novel cross-domain classes (e.g., CUB, ChestX), are MACR module captions generated online for support images at test time, or pre-cached? If generated online, how does LLM latency impact inference speed?
  5. What is the individual contribution of spatial token concatenation vs. channel-wise squeeze-and-excitation block interaction within DPI?
  6. How stable is the layer routing probability across training epochs, and did you observe any layer-collapse failure modes where routes all samples to a single index of layer?
6 comments
AuthorsOpenReview ↗

Response to Reviewer CvVw (1/4)

We sincerely thank the reviewer for the valuable and constructive feedback. We appreciate the reviewer’s recognition of how MACR’s multi-agent caption refinement complements DPI’s image-specific layer selection, as well as the strong results across 16 benchmark settings and the supporting ablation studies and visualizations. The reviewer’s questions about computational cost and generalization to other visual backbones helped guide our additional experiments. Our point by point responses are provided below.


[W1] MACR completes its three-agent refinement loop before meta-training, so describing it as a “dynamic multi-agent system” may be misleading.

[Response] We thank the reviewer for pointing out this potential ambiguity.

Exactly, MACR operates as an offline multi-agent refinement procedure rather than a dynamic multi-agent system that continuously interacts during meta-training. We would like to clarify that the term "dynamic" in MAD-SP primarily refers to the instance-adaptive prompt injection mechanism (DPI), rather than the execution of the multi-agent refinement process. Specifically, MACR employs three specialized agents (Generator, Critic, and Refiner) to iteratively improve semantic prompts under score-guided feedback before meta-training. The resulting prompts are cached for subsequent use, and the agents do not participate in gradient-based optimization, as described in Algorithm 1 (p. 17, lines 868–896).

In contrast, DPI dynamically determines the appropriate prompt injection layer for each support image during meta-training and episodic inference, based on its visual features and corresponding semantic prompt. Unlike fixed-layer injection, this decision is input-dependent and is performed by a learnable decision network.

Therefore, MAD-SP integrates offline multi-agent semantic refinement with dynamic, instance-adaptive prompt injection, rather than claiming that the multi-agent system itself operates dynamically throughout training. Following the reviewer's suggestion, we have added the clarification of the source of dynamic behavior in Section 1 (p. 2, lines 101–104). The separation between caption preparation and episodic optimization has been also specified in Appendix A.1 (p. 17, Algorithm 1, lines 872–888; discussion, p. 17, lines 891–896), with the preparation and caching protocol in Appendix A.2 (p. 19, lines 998–1008).


[Q1] What is the total wall-clock preprocessing time and GPU cost of MACR caption refinement?

[Response] We have reported the full framework’s computational cost and compared it with other methods. Measurements on miniImageNet show that MAD-SP has modest computational requirements: 10.7M downstream parameters, comprising a 9.94M visual backbone, 0.70M for CMA, and 0.05M for the DPI layer selection network. CMA and DPI add only 0.75M parameters, about 7.5% of the backbone’s parameter count. Qwen3-VL-8B’s fast responses keep semantic preprocessing, including initial caption generation and all refinement rounds, to a practical one-time cost of 2.0 hours. Training takes 34 minutes, within the baselines’ 32–57 minute range. MAD-SP has the fewest parameters, matches SP at the lowest computational cost of 1.3 GFLOPs, and has the lowest inference latency (51 ms versus 81–135 ms) among the compared methods. Following the reviewer’s suggestion, we have included these measurements in Section 5.5 (p. 10, Table 8, lines 486–494; discussion, p. 10, lines 513–521).

MethodParams / GFLOPsMemory (GB)Offline semantics (h)Training (min)Inference (ms)
SP [1]10.8M / 1.34.0—3493
SemFew [2]19.8M / 3.58.31.742112
ECER [3]26.5M / 5.510.50.857135
DVLA-RL [4]12.8M / 2.04.81.23281
MAD-SP10.7M / 1.34.12.03451

[1] Chen W, Si C, Zhang Z, et al. Semantic prompt for few-shot image recognition[J]. arXiv preprint arXiv:2303.14123, 2023.

[2] Zhang H, Xu J, Jiang S, et al. Simple semantic-aided few-shot learning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 28588-28597.

[3] Liu M, Wu F, Li B, et al. Envisioning class entity reasoning by large language models for few-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(18): 18906-18914.

[4] Li W, Meng X, Wang Q, et al. Dvla-rl: Dual-level vision-language alignment with reinforcement learning gating for few-shot learning[J]. arXiv preprint arXiv:2602.00795, 2026.

AuthorsOpenReview ↗

Response to Reviewer CvVw (2/4)

[Q2] How does MAD-SP perform if MACR is replaced by a single-pass VLM captioner, such as LLaVA-1.5 or BLIP-2, or by standard class-name prompts?

[Response] Full MACR outperforms both single-pass caption generation and standard class-name prompts in all four settings on miniImageNet and CIFAR-FS.

First, The Generator Only ablation in Table 11 of Appendix A.4 (p. 20, lines 1065–1077) generates one caption per support image using the same Qwen3-VL-8B model as Full MACR. With the downstream framework fixed, adding the score-guided critique and refinement loop improves accuracy by an average of 3.62 percentage points in 1-shot and 1.36 points in 5-shot classification. Table 5 (p. 8, lines 388–395) also evaluates standard class-name prompts under the same downstream framework. Full MACR improves over these prompts by 1.50–4.81 percentage points across the four settings.

Prompt variantminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
Class name (Table 5)81.1787.8984.7490.36
Generator Only (Table 11)82.3588.2785.7690.44
Full MACR85.9889.3989.3692.03

Second, we also evaluated the your suggested LLaVA-1.5-7B [1] in a separate foundation model ablation. For this comparison, we prioritized strong multimodal models with parameter counts close to Qwen3-VL-8B that could follow the image-conditioned instructions for generation, critique, and revision. We therefore selected LLaVA-1.5-7B, Janus-Pro-7B [2], and Qwen2.5-VL-7B [3]. The LLaVA results below use the complete MACR loop, while the Generator Only results above isolate the effect of replacing that loop with a single generation pass.

With CLIP ViT-B/32 fixed, the three alternative foundation models differ from Qwen3-VL-8B by 0.57 percentage points on average, taking the absolute differences across the four settings. Janus-Pro-7B achieves the highest mean accuracy on miniImageNet 1-shot and CIFAR-FS 5-shot. With Qwen3-VL-8B fixed, replacing the downstream CLIP text encoder with BERT [4] yields an average difference of 1.01 percentage points. These results show that the full framework maintains comparable accuracy across different foundation models and text encoders, supporting the generalizability of our method. Following the reviewer's suggestion, we have included these ablations and their analysis in Section 5.3 (p. 8, Table 6, lines 397–405; p. 9, lines 463–470).

Foundation modelText encoderminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
LLaVA-1.5-7B [1]CLIP ViT-B/3284.50 ± 0.3788.58 ± 0.3288.34 ± 0.3391.09 ± 0.28
Janus-Pro-7B [2]CLIP ViT-B/3286.26 ± 0.4489.36 ± 0.3888.83 ± 0.4592.17 ± 0.42
Qwen2.5-VL-7B [3]CLIP ViT-B/3285.44 ± 0.3589.03 ± 0.3088.91 ± 0.4191.82 ± 0.35
Qwen3-VL-8BBERT [4]84.71 ± 0.3588.19 ± 0.2888.16 ± 0.3091.65 ± 0.30
Qwen3-VL-8BCLIP ViT-B/3285.98 ± 0.3789.39 ± 0.2889.36 ± 0.3392.03 ± 0.24

[1] Liu H, Li C, Li Y, et al. Improved baselines with visual instruction tuning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 26286-26296.

[2] Chen X, Wu Z, Liu X, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling[J]. arXiv preprint arXiv:2501.17811, 2025.

[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025b. URL https://arxiv.org/abs/2502.13923.

[4] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.

AuthorsOpenReview ↗

Response to Reviewer CvVw (3/4)

[W2&Q3] Do DPI and CMA generalize to standard visual backbones such as ViT-S/16, ViT-B/16, or Swin-T? Can the authors provide results on ViT-B/16?

[Response] MAD-SP achieves comparable accuracy gains with ViT-S/16, Swin-T, and DeiT-Ti/16, demonstrating the generalization of our method across different backbones. On miniImageNet and CIFAR-FS, MAD-SP improves over the corresponding baseline with the same backbone in all 12 additional settings, with gains ranging from 3.89 to 12.82 percentage points. Averaged across both datasets and shot settings, the gains are 8.56 points with ViT-S/16, 7.45 points with Swin-T, and 8.22 points with DeiT-Ti/16, compared with 9.35 points with Visformer-T. All three alternative backbones also follow the same pattern as Visformer-T, with larger gains in the 1-shot setting on both datasets. These results demonstrate that DPI can be applied effectively within MAD-SP across the tested Transformer-based visual backbones. Following the reviewer's suggestion, we have included these experiments and their analysis in Appendix A.8 (p. 24, lines 1244–1268; Table 17, p. 24, lines 1249–1261).

MethodBackboneminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
BaselineViT-S/1673.4583.3577.8187.06
MAD-SPViT-S/1686.27 (+12.82)89.30 (+5.95)88.64 (+10.83)91.69 (+4.63)
BaselineSwin-T73.1383.7479.7088.22
MAD-SPSwin-T84.42 (+11.29)89.12 (+5.38)88.94 (+9.24)92.11 (+3.89)
BaselineDeiT-Ti/1671.1682.8077.0486.96
MAD-SPDeiT-Ti/1683.92 (+12.76)88.52 (+5.72)87.25 (+10.21)91.14 (+4.18)
BaselineVisformer-T73.4382.6277.1886.13
MAD-SPVisformer-T85.98 (+12.55)89.39 (+6.77)89.36 (+12.18)92.03 (+5.90)

[Q4] For novel cross-domain classes such as CUB and ChestX, are support captions generated online at test time or cached in advance? If generated online, how does LLM latency affect inference speed?

[Response] We thank the reviewer for raising this important question regarding the cross-domain evaluation protocol and inference efficiency.

We clarify that MACR is designed as a separate semantic prompt preparation stage rather than an online component of the recognition network. For novel cross-domain classes, such as those in CUB and ChestX, captions are generated from labeled support images and their corresponding class names. Once generated and refined, the resulting semantic prompts are encoded by the frozen CLIP text encoder and cached for subsequent use.

First, this process does not require retraining the LLM or accessing query labels. During recognition with cached prompts, the model directly retrieves the corresponding semantic embeddings, while DPI dynamically selects the injection layer for each support image. Query images are processed by the visual backbone without LLM involvement.

Second, regarding efficiency, we acknowledge that MACR introduces an additional one-time preprocessing cost, which should be distinguished from the inference latency of the recognition network. Since the refined prompts can be reused, repeated episodic evaluation does not require regenerating captions for previously processed support images. However, the initial caption preparation cost remains relevant when encountering new support samples.

Following the reviewer's suggestion, we have clarified this protocol and the distinction between preparation and recognition costs in Section 5.5 (p. 10, lines 520–521) and more details in Appendix A.2 (p. 19, lines 998–1008).

Reviewer_crmqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

MAD-SP enhances few-shot learning by generating concise, visually grounded semantic prompts and integrating them adaptively into the visual encoder. MACR refines LLM-generated captions, DPI determines image-specific injection layers, and CMA improves text–vision alignment across depths. Extensive evaluations on standard and cross-domain benchmarks demonstrate strong and generally state-of-the-art performance.

Critical strengths

A key strength is the coherent treatment of both semantic prompt quality and prompt utilization: MACR refines noisy LLM-generated captions, while DPI and CMA adaptively inject and align the resulting prompts with visual representations. This integrated design directly addresses two practical limitations of multimodal FSL and strengthens the methodological contribution. A second strength is the extensive evaluation across both standard and cross-domain benchmarks, where MAD-SP achieves state-of-the-art results in nearly all settings, providing strong evidence of effectiveness and generalization.

Critical weaknesses

I have several concerns towards this manuscript:

  1. The complementary roles of the three agents—generator, critic, and refiner—should be discussed in greater depth. In particular, the authors should clarify why each agent is necessary, what unique function it serves, and how their collaboration contributes to improving caption quality and the overall effectiveness of MACR.

  2. In DPI, only the most suitable layer is selected for prompt injection. It would be interesting to investigate whether selecting and jointly utilizing multiple layers, such as two or three layers, could further improve performance by exploiting complementary information across different representation depths.

  3. In the SOTA comparisons reported in Tables 1 and 2, only a limited number of methods published in 2025–2026 are included. More recent approaches, such as VT-FSL and GPS, should be added to better demonstrate the competitiveness and superiority of the proposed model against the latest advances.

  4. Why are Qwen3-VL-8B and CLIP ViT-B/32 selected as the shared foundation model and text encoder, respectively? The authors should provide a clearer justification for this heterogeneous backbone design and discuss whether using the same or more comparable architectures would lead to a fairer evaluation. In addition, comparisons with other representative LLM/MLLM backbones are encouraged to demonstrate the robustness and generalizability of the proposed method.

  5. The InfoNCE loss is used to align corresponding semantic prompts with visual prototypes while enhancing inter-class separation. However, this formulation appears largely standard and introduces limited technical novelty. The authors should clarify what specific improvement or new insight is provided beyond conventional contrastive alignment objectives.

Questions for authors

Please clarify several points that could materially affect my assessment: first, what are the distinct roles of the generator, critic, and refiner in MACR, why are all three necessary, and what unique contribution does each make; second, why are Qwen3-VL-8B and CLIP ViT-B/32 selected as the foundation model and text encoder, respectively, and how does this heterogeneous design affect comparison fairness and generalization to other LLM/MLLM backbones; third, what methodological novelty does the InfoNCE-based CMA provide beyond standard contrastive alignment between textual and visual representations; and fourth, how does MAD-SP compare with recent 2025–2026 methods such as VT-FSL and GPS, which are currently missing from the SOTA tables. As a suggestion for future work, it would also be interesting to investigate whether DPI could jointly inject prompts into multiple layers rather than selecting only a single layer.

7 comments
AuthorsOpenReview ↗

Response to Reviewer crmq (1/3)

We sincerely thank the reviewer for the valuable and constructive reviews. We appreciate the reviewer’s recognition that MACR, DPI, and CMA work together to improve both the quality of semantic prompts and how effectively they are used, along with the extensive evaluation and strong results across standard and cross-domain benchmarks. We are especially grateful for the reviewer’s insightful suggestion to explore joint prompt injection across multiple layers and look forward to investigating it in future work. Our point by point responses are provided below.


[W1&Q1] What are the distinct roles of the generator, critic, and refiner in MACR, why is each necessary, and what does each contribute?

[Response] MACR improves few-shot semantic prompts by assigning caption generation, diagnosis, and revision to three complementary agents. With only a few support images, the captions need to describe visible attributes that distinguish the target class.

  • Generator. The Generator takes the support image and its class name and produces an initial description of the image. Its contribution is to turn the visual evidence into a caption specific to that support example. This draft is necessary because it gives MACR a concrete description to evaluate and refine.
  • Critic. When a caption falls below the acceptance threshold, the Critic examines the image and caption alongside the scores and the closest competing class. It identifies specific problems, such as an attribute that is unsupported by the image, a visible detail that would help distinguish the target class, or redundant information that distracts from the subject. This role is necessary because the score indicates whether the caption needs revision, while the Critic explains what needs to change.
  • Refiner. The Refiner uses the original image, the current caption, and the Critic’s feedback to produce a revised description. It preserves accurate details while correcting unsupported statements and adding relevant attributes that are visible in the image. This role is necessary because the diagnosis must be translated into a revised caption before MACR can evaluate whether the changes improve it.

Together, these three roles cover all the agent functions needed to sustain the refinement loop. MACR first evaluates the Generator’s caption using CLIP-based image–text alignment and class discriminativeness scores. MACR accepts the initial caption if its combined score meets the threshold. Otherwise, the Critic diagnoses the problems, the Refiner revises the caption, and MACR scores the revision again using the same criteria. The loop continues until a caption meets the threshold or three refinement rounds are completed. In this way, each round uses explicit quality feedback to guide a targeted revision and then checks the revised caption against the same criteria.

Table 11 in Appendix A.4 evaluates the Critic and Refiner together because their functions are coupled. Adding a single round of critique and revision without scoring improves accuracy over Generator Only in all four settings. The full score-guided loop improves accuracy further and exceeds Generator Only by an average of 3.62 percentage points in 1-shot and 1.36 points in 5-shot classification across miniImageNet and CIFAR-FS. These results show that the captions produced by the complete loop provide more effective semantic guidance for few-shot recognition. Following the reviewer's suggestion, we have added this explanation to Appendix A.4 (p. 21, lines 1112–1123). We have also clarified the distinct roles and necessity of the three agents in Section 1 (p. 2, lines 95–100), with supporting ablations in Appendix A.4 (p. 20, Table 11 and discussion, lines 1065–1077).

VariantminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
Generator Only82.3588.2785.7690.44
Full MACR85.9889.3989.3692.03
AuthorsOpenReview ↗

Response to Reviewer crmq (2/3)

[W4&Q2] Why are Qwen3-VL-8B and CLIP ViT-B/32 selected as the foundation model and text encoder, respectively, and how does this heterogeneous design affect comparison fairness and generalization to other LLM/MLLM backbones?

[Response] We thank the reviewer for suggesting this important ablation on the choice of foundation model and text encoder. First, we must clarify that we do not present the choice of a particular LLM or multimodal model for either role as a key innovation. The contributions we emphasize are twofold:

  1. MACR builds on LLM-based caption generation by introducing interaction among multiple agents and refining captions over multiple rounds to improve semantic prompt quality. This refinement process can be used with different foundation models.
  2. DPI allows each semantic prompt to dynamically select the most suitable injection layer based on the characteristics of its corresponding image. This contribution lies in adaptive layer selection, regardless of which text encoder produces the semantic prompt tokens.

Second, we recognize that this ablation is necessary to validate the effectiveness of MAD-SP. We therefore evaluated alternative models within the same framework. For a fair comparison, we selected three multimodal models with similar parameter counts, LLaVA-1.5-7B [1], Janus-Pro-7B [2], and Qwen2.5-VL-7B [3], as alternative foundation models. We also evaluated BERT [4] as an alternative text encoder. The table below reports the results across four settings.

With CLIP ViT-B/32 fixed, the three alternative foundation models differ from Qwen3-VL-8B by an average of 0.57 percentage points in mean accuracy in any setting. Janus-Pro-7B achieves the highest mean accuracy on miniImageNet 1-shot and CIFAR-FS 5-shot. With Qwen3-VL-8B fixed, BERT’s mean accuracy differs from CLIP’s by an average of 1.01 percentage points across the four settings. The similar results indicate that MAD-SP’s performance is relatively insensitive to the choice of foundation model and text encoder among the tested configurations, supporting the generalizability of our method. Following the reviewer's suggestion, we have included these ablations and their analysis in Section 5.3 (p. 8, Table 6, lines 397–405; p. 9, lines 463–470).

LLMText encoderminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
LLaVA-1.5-7B [1]CLIP ViT-B/3284.50 ± 0.3788.58 ± 0.3288.34 ± 0.3391.09 ± 0.28
Janus-Pro-7B [2]CLIP ViT-B/3286.26 ± 0.4489.36 ± 0.3888.83 ± 0.4592.17 ± 0.42
Qwen2.5-VL-7B [3]CLIP ViT-B/3285.44 ± 0.3589.03 ± 0.3088.91 ± 0.4191.82 ± 0.35
Qwen3-VL-8BBERT [4]84.71 ± 0.3588.19 ± 0.2888.16 ± 0.3091.65 ± 0.30
Qwen3-VL-8BCLIP ViT-B/3285.98 ± 0.3789.39 ± 0.2889.36 ± 0.3392.03 ± 0.24

[1] Liu H, Li C, Li Y, et al. Improved baselines with visual instruction tuning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 26286-26296.

[2] Chen X, Wu Z, Liu X, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling[J]. arXiv preprint arXiv:2501.17811, 2025.

[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025b. URL https://arxiv.org/abs/2502.13923.

[4] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.

AuthorsOpenReview ↗

Response to Reviewer crmq (3/3)

[W5&Q3] What methodological novelty does the InfoNCE-based CMA offer beyond standard contrastive alignment between textual and visual representations?

[Response] CMA is designed to support dynamic prompt injection rather than to introduce a new contrastive objective. Different Transformer blocks encode visual representations with different levels of abstraction and feature statistics. DPI therefore requires semantic features to remain compatible with visual representations encountered at varying injection depths. CMA addresses this requirement through a nonlinear alignment adapter and contrastive alignment between semantic and visual class prototypes. Its contribution primarily lies in supporting dynamic injection, while InfoNCE provides a standard optimization objective.

Moreover, the adapter’s bottleneck structure is another notable feature naturally suited to few-shot learning, where limited data increase overfitting risk. Compressing the intermediate representation reduces trainable parameters while preserving nonlinear mapping capacity, balancing expressiveness and regularization for episodic training. Table 13 supports this design: the bottleneck adapter outperforms both the linear projector and the full MLP across all evaluated settings. Following the reviewer's suggestion, we have clarified CMA's supporting role in Section 1 (p. 2, lines 105–107) and expanded this explanation in Appendix A.6 (p. 23, lines 1194–1207).

AdapterminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
Linear Projector84.9888.9488.3291.71
Full-width MLP85.5189.1688.7391.85
Bottleneck MLP85.9889.3989.3692.03

[W3&Q4] How does MAD-SP compare with recent methods such as VT-FSL and GPS?

[Response] We thank the reviewer for highlighting these recent methods.

The comparison shows that MAD-SP substantially outperforms GPS and outperforms VT-FSL without generating additional support images. GPS introduces a visual matching metric without incorporating semantic information, and reports substantially lower accuracy across the four standard benchmarks. VT-FSL provides a closer methodological comparison: its CIP generates class descriptions through structured LLM reasoning, followed by image generation to supplement the support set. MAD-SP offers three distinct advantages: MACR iteratively refines instance captions using alignment and discriminativeness scores; DPI selects injection depth per image; and the framework avoids synthetic image generation and selection. With the same Visformer-T backbone, MAD-SP outperforms VT-FSL in both 1-shot and 5-shot settings across all four standard datasets, with an average gain of 1.12 percentage points. Following the reviewer's suggestion, we have included GPS and VT-FSL in the comparisons associated with Section 5.2: Table 1 (p. 7: GPS, line 341; VT-FSL, p. 7, lines 342–343) and Table 3 (p. 7: GPS, lines 372–373; VT-FSL, p. 7, line 374). The corresponding analysis appears on p. 7, lines 368–377, and p. 8, lines 407–409.

ModelBackboneminiImageNet 1-shotminiImageNet 5-shottieredImageNet 1-shottieredImageNet 5-shot
GPS [1]ResNet-1266.27 ± 0.3781.19 ± 0.4773.16 ± 0.4385.52 ± 0.48
VT-FSL [2]Visformer-T83.66 ± 0.3188.38 ± 0.2588.02 ± 0.3491.71 ± 0.27
MAD-SPVisformer-T85.98 ± 0.3789.39 ± 0.2889.51 ± 0.4092.27 ± 0.34

[1] Zhang Z, Lin F, Liu H, et al. Gps: A probabilistic distributional similarity with gumbel priors for set-to-set matching[C]//International Conference on Learning Representations. 2025, 2025: 39858-39872.

[2] Li W, Wang Q, Meng X, et al. Vt-fsl: Bridging vision and text with llms for few-shot learning[J]. Advances in Neural Information Processing Systems, 2026, 38: 113697-113727.


[W2&Q5] As a suggestion for future work, it would be interesting to investigate whether DPI could jointly inject prompts into multiple layers rather than selecting only a single layer.

[Response] We sincerely thank the reviewer for this thoughtful suggestion. We appreciate the insight that injecting prompts into multiple layers could make better use of complementary visual features at different depths. We look forward to exploring this idea in future work, including how to adjust the injection strength at each layer.

Reviewer_Ca8SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework (MAD-SP) that combines LLM-generated semantic prompts with visual representations to perform few shot image classification. The proposed method relies on multi agent system to generate and refine captions (MACR), a prompt injection mechanism (DPI) that dynamically chooses in which layer to incorporate information, and a contrastive modality alignment (CMA) mechanism, to align textual and visual representations.

Critical strengths

The paper is well written and easy to follow, even though it contains a lot of components.

The idea of dynamic prompt injection, seems particularly well motivated and combines well with the semantic prompts.

The experimental evaluation and ablations are extensive.

Critical weaknesses

The introduced framework mostly combines existing ideas (LLM based prompt generation, and contrastive alignment are widely used).

The complexity of the whole pipeline is pretty big, the multi agent system adds significant complexity and its computational cost and the number of required LLM calls are not thoroughly evaluated.

Although I appreciate the idea of dynamic layer selection, its mechanism remains mostly descriptive, with no real evidence explaining which particular layers are selected and for which classes/concept.

Questions for authors

Could the authors provide a measurement of the cost of the system, show how much does the multi agent part contributes to it, and compare it with other few shot pipelines?

Could the authors provide an analysis of why different visual layers are selected for different categories and whether these choices correlate with semantic/ local details, or visual complexity?

Finally, would the proposed framework provide similar improvements with a different visual backbone?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Ca8S (1/4)

We sincerely thank the reviewer for the valuable and constructive reviews. We appreciate the reviewer’s recognition that dynamic prompt injection is well motivated and complements semantic prompting. We also thank the reviewer for acknowledging the breadth of our experimental evaluation and strong results. Below is our feedback. All concerns have been carefully addressed point by point.


[W1] The framework mainly combines existing ideas, including widely used LLM-based prompt generation and contrastive alignment.

[Response] We thank the reviewer for the valuable comment. We must clarify that the novelty of MAD-SP does not lie in these individual components, but in the proposed mechanisms for addressing two important limitations of existing semantic prompting methods for few-shot learning: (1) the lack of explicit quality supervision for generated semantic prompts, and (2) the reliance on fixed-layer semantic prompt injection.

First, MACR goes beyond conventional LLM-based prompt generation. Rather than directly adopting LLM-generated descriptions, MACR introduces a score-guided multi-agent refinement mechanism that jointly considers image-text consistency and inter-class discriminativeness. The generator–critic–refiner collaboration selectively improves unreliable captions using explicit evaluation feedback. Importantly, Appendix Table 11 (p. 20, lines 1065–1071) demonstrates that this mechanism provides benefits beyond merely using an LLM: compared with direct generation, full MACR improves 1-shot accuracy by 3.63% on miniImageNet and 3.60% on CIFAR-FS. Removing the scoring mechanism also consistently reduces performance, validating the contribution of explicit quality supervision.

VariantminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
Generator Only82.3588.2785.7690.44
Full MACR85.9889.3989.3692.03

Second, DPI represents a key methodological distinction from existing semantic prompting approaches. While conventional methods inject semantic prompts at predefined layers, DPI learns to select the injection depth for each image by jointly considering its intermediate visual representation and semantic prompt. This enables instance-adaptive visual-semantic interaction rather than uniform semantic injection. As shown in Table 7 (p. 9, lines 432–445), DPI outperforms the best fixed-layer strategy by 5.23%, 4.73%, and 4.98% in the 1-shot settings on miniImageNet, CIFAR-FS, and CUB, respectively. Furthermore, the layer-selection distributions in Figure 3 (p. 9, lines 447–461) provide evidence that DPI adapts its decisions to different visual characteristics rather than degenerating into a fixed injection strategy.

InjectionminiImageNet 1-shotminiImageNet 5-shotCIFAR-FS 1-shotCIFAR-FS 5-shot
S3-L176.62 ± 0.4482.37 ± 0.4078.94 ± 0.3985.64 ± 0.35
S3-L277.31 ± 0.4582.66 ± 0.4080.10 ± 0.4086.01 ± 0.35
S3-L380.75 ± 0.4183.92 ± 0.3984.63 ± 0.3687.38 ± 0.34
S3-L478.13 ± 0.4483.58 ± 0.3983.02 ± 0.3987.34 ± 0.34
DPI (S3)85.98 ± 0.3789.39 ± 0.2889.36 ± 0.3392.03 ± 0.24

Third, CMA is designed to support dynamic prompt injection rather than to introduce a new contrastive objective. Specifically, it aligns semantic prompt prototypes with the corresponding visual prototypes under instance-adaptive injection, providing complementary cross-modal supervision. The ablation results in Table 4 (p. 8, lines 378–386) confirm its additional contribution.

Therefore, MAD-SP is not intended as a novel formulation of LLM generation or contrastive learning. Its primary contribution lies in the explicitly supervised construction and instance-adaptive utilization of semantic prompts for few-shot recognition. The consistent improvements across standard and cross-domain benchmarks further demonstrate the effectiveness of these task-specific mechanisms. Following the reviewer's suggestion, we have revised Section 1 (p. 2, lines 87–91) to highlight explicit quality supervision of prompt construction. We have also clarified CMA's supporting role in Section 1 (p. 2, lines 105–107). The explanation of the agents' complementary roles has been added to Appendix A.4 (p. 21, lines 1112–1123).

AuthorsOpenReview ↗

Response to Reviewer Ca8S (2/4)

[W2&Q1] Could the authors measure the overall system cost, quantify the contribution of the multi-agent component, and the number of required LLM calls?

[Response] We have reported the computational cost of the full framework, compared it with other methods, and provided statistics on the number of refinement rounds.

First, MACR generates and refines captions for all images in a one-time offline stage before training. During inference, MAD-SP retrieves the cached semantic prompts and makes zero LLM calls. The table below reproduces the refinement statistics from Figure 5 (p. 21, lines 1086–1099) of the paper in tabular form. MACR averages 1.33 refinement rounds per image, with a maximum of three. Each image requires one Generator call, and each refinement round adds one Critic call and one Refiner call. This amounts to 3.66 LLM calls per image on average, including 2.66 calls for refinement, with at most seven calls per image.

Classcliffdomeschool buslionmeerkatvaseladybugantOverall average
Average refinement rounds1.741.320.480.911.931.460.762.211.33

Second, our computational cost measurements on miniImageNet show that MAD-SP has modest computational requirements. MAD-SP has 10.7M downstream parameters. The visual backbone accounts for 9.94M, while CMA and the DPI layer selection network add 0.70M and 0.05M, respectively. Together, these two modules add only 0.75M parameters, about 7.5% of the backbone’s parameter count. Qwen3-VL-8B responds quickly in our setup, keeping the full semantic preprocessing stage on miniImageNet, including initial caption generation and all refinement rounds, to a practical one-time cost of 2.0 hours. MAD-SP takes 34 minutes to train, within the 32–57 minute range of the baselines. It has the fewest parameters, matches SP at the lowest computational cost of 1.3 GFLOPs, and achieves the lowest inference latency at 51 ms, compared with 81–135 ms for the baselines. Following the reviewer's suggestion, we have included the computational cost results in Section 5.5 (Table 8; p. 10, lines 486–494 and 513–521).

MethodParams / GFLOPsMemory (GB)Offline semantics (h)Training (min)Inference (ms)
SP [1]10.8M / 1.34.0—3493
SemFew [2]19.8M / 3.58.31.742112
ECER [3]26.5M / 5.510.50.857135
DVLA-RL [4]12.8M / 2.04.81.23281
MAD-SP10.7M / 1.34.12.03451

[1] Chen W, Si C, Zhang Z, et al. Semantic prompt for few-shot image recognition[J]. arXiv preprint arXiv:2303.14123, 2023.

[2] Zhang H, Xu J, Jiang S, et al. Simple semantic-aided few-shot learning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 28588-28597.

[3] Liu M, Wu F, Li B, et al. Envisioning class entity reasoning by large language models for few-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(18): 18906-18914.

[4] Li W, Meng X, Wang Q, et al. Dvla-rl: Dual-level vision-language alignment with reinforcement learning gating for few-shot learning[J]. arXiv preprint arXiv:2602.00795, 2026.

AuthorsOpenReview ↗

Response to Reviewer Ca8S (3/4)

[W3&Q2] Why does DPI select different layers for different categories, and how do these choices relate to semantic cues, local details, and visual complexity?

[Response] DPI selects injection depths according to target scale, texture complexity, and the semantic processing required for recognition. Large objects or simple textures provide salient cues that are readily extracted by shallower stage-3 blocks, whereas small objects or complex textures require deeper contextual and semantic processing. Category-level preferences reflect shared visual characteristics: ant images often contain small targets in complex natural backgrounds, while school bus images typically contain large objects with simpler textures. Accordingly, 81.3% of ant images select Layers 3–4, while 62.5% of school bus images select Layers 1–2. Figure 3 in Section 5.4 Model Analysis, under “Distribution of Instance-Level Injection Decisions,” (p. 9, lines 447–461 and 479–485) is reproduced in the table below. We selected eight representative miniImageNet classes and report how often each layer is selected for prompt injection. The final row reports the selection proportions across all images.

Across all images, 40.2% select Layer 3 for prompt injection, a substantially higher share than for any of the other three layers. This preference is consistent with the findings in [1]. We believe that higher-level semantic extraction becomes prominent at Layer 3, while the preceding layers primarily process and recognize texture cues for many images. Following the reviewer's suggestion, we have expanded this explanation in Appendix A.5 (p. 21, lines 1127–1133; p. 22, lines 1174–1180).

ClassLayer 1 (shallow)Layer 2Layer 3Layer 4 (deep)
cliff39.6%30.8%23.5%6.1%
dome34.2%31.4%27.1%7.3%
school bus29.9%32.6%30.0%7.5%
lion22.9%27.8%37.6%11.7%
vase20.7%25.9%40.4%13.0%
meerkat13.1%20.6%47.5%18.8%
ladybug7.5%15.6%51.0%25.9%
ant5.8%12.9%49.0%32.3%
All classes20.3%24.1%40.2%15.4%

[1] Chen W, Si C, Zhang Z, et al. Semantic prompt for few-shot image recognition[J]. arXiv preprint arXiv:2303.14123, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.