- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Objective: The paper address few-shot image classification by resolving the noise/verbosity of LLM-generated captions and overcoming the inflexibility of fixed-layer prompt injection in Vision Transformers.
- Methodology: The paper introduces a framework using Visformer-T and CLIP text encoder:
- MACR (Multi-Agent Caption Refinement): Coordinates Generator, Critic, and Refiner LLM agents (powered by Qwen3-VL-8B) under CLIP-based image-text alignment and discriminative margin scoring to produce visually grounded prompts.
- DPI (Dynamic Prompt Injection): Predicts layer compatibility probabilities across Visformer blocks and uses a Straight-Through Estimator (STE) gate to dynamically inject prompts spatially and channel-wise per image.
- CMA (Contrastive Modality Alignment): Uses a bottleneck MLP adapter and class-level InfoNCE loss to align prompt prototypes with visual prototypes.
- Key Findings & Empirical Validation: The proposed framework is evaluated across 4 standard FSL benchmarks (miniImageNet, tieredImageNet, CIFAR-FS, FC100) and 4 cross-domain benchmarks (CUB, Places, Plantae, ChestX) under 1-shot and 5-shot protocols over 2,000 test episodes. Sets new state-of-the-art results in 8/8 standard settings (+2.72% average gain) and 7/8 cross-domain settings (+1.97% average gain).
Critical strengths
- The paper effectively bridges closed-loop multi-agent prompt evolution with instance-adaptive Transformer layer routing. DPI addresses visual scale abstraction by dynamically routing structural scenes to shallow layers and fine-grained targets to deep layers.
- The paper shows exceptional performance across 16 benchmark settings with 2,000 test episodes and reported 95% confidence intervals. Comprehensive ablations validate the contribution of MACR (+7.01% 1-shot gain), DPI (+6.62% 1-shot gain), and CMA, supported by t-SNE clustering and attention map visualizations.
Critical weaknesses
- Regarding the MACR module, it runs a 3-agent LLM debate loop, so this process is completed before meta-training begins. Referring to it as a "dynamic multi-agent system" is slightly misleading, it is essentially an offline prompt pre-processing step.
- Evaluation is restricted to a lightweight Visformer-T backbone (10M parameters). There is no empirical proof that DPI and CMA generalize to standard ViT-S/16, ViT-B/16, or Swin Transformers. Furthermore, discrete STE gating can suffer from gradient variance and layer-collapse risks during meta-training.
Questions for authors
- What is the total wall-clock pre-processing time and GPU cost for MACR caption refinement?
- How does MAD-SP perform if MACR is replaced by a single-pass VLM captioner (e.g., LLaVA-1.5 or BLIP-2) or standard class-name prompts?
- Does DPI and CMA module generalize to standard vision backbones (e.g., ViT-S/16, ViT-B/16, or Swin-T)? Can you provide results on ViT-B/16?
- When evaluating novel cross-domain classes (e.g., CUB, ChestX), are MACR module captions generated online for support images at test time, or pre-cached? If generated online, how does LLM latency impact inference speed?
- What is the individual contribution of spatial token concatenation vs. channel-wise squeeze-and-excitation block interaction within DPI?
- How stable is the layer routing probability across training epochs, and did you observe any layer-collapse failure modes where routes all samples to a single index of layer?
6 comments
Response to Reviewer CvVw (2/4)
[Q2] How does MAD-SP perform if MACR is replaced by a single-pass VLM captioner, such as LLaVA-1.5 or BLIP-2, or by standard class-name prompts?
[Response] Full MACR outperforms both single-pass caption generation and standard class-name prompts in all four settings on miniImageNet and CIFAR-FS.
First, The Generator Only ablation in Table 11 of Appendix A.4 (p. 20, lines 1065–1077) generates one caption per support image using the same Qwen3-VL-8B model as Full MACR. With the downstream framework fixed, adding the score-guided critique and refinement loop improves accuracy by an average of 3.62 percentage points in 1-shot and 1.36 points in 5-shot classification. Table 5 (p. 8, lines 388–395) also evaluates standard class-name prompts under the same downstream framework. Full MACR improves over these prompts by 1.50–4.81 percentage points across the four settings.
| Prompt variant | miniImageNet 1-shot | miniImageNet 5-shot | CIFAR-FS 1-shot | CIFAR-FS 5-shot |
|---|---|---|---|---|
| Class name (Table 5) | 81.17 | 87.89 | 84.74 | 90.36 |
| Generator Only (Table 11) | 82.35 | 88.27 | 85.76 | 90.44 |
| Full MACR | 85.98 | 89.39 | 89.36 | 92.03 |
Second, we also evaluated the your suggested LLaVA-1.5-7B [1] in a separate foundation model ablation. For this comparison, we prioritized strong multimodal models with parameter counts close to Qwen3-VL-8B that could follow the image-conditioned instructions for generation, critique, and revision. We therefore selected LLaVA-1.5-7B, Janus-Pro-7B [2], and Qwen2.5-VL-7B [3]. The LLaVA results below use the complete MACR loop, while the Generator Only results above isolate the effect of replacing that loop with a single generation pass.
With CLIP ViT-B/32 fixed, the three alternative foundation models differ from Qwen3-VL-8B by 0.57 percentage points on average, taking the absolute differences across the four settings. Janus-Pro-7B achieves the highest mean accuracy on miniImageNet 1-shot and CIFAR-FS 5-shot. With Qwen3-VL-8B fixed, replacing the downstream CLIP text encoder with BERT [4] yields an average difference of 1.01 percentage points. These results show that the full framework maintains comparable accuracy across different foundation models and text encoders, supporting the generalizability of our method. Following the reviewer's suggestion, we have included these ablations and their analysis in Section 5.3 (p. 8, Table 6, lines 397–405; p. 9, lines 463–470).
| Foundation model | Text encoder | miniImageNet 1-shot | miniImageNet 5-shot | CIFAR-FS 1-shot | CIFAR-FS 5-shot |
|---|---|---|---|---|---|
| LLaVA-1.5-7B [1] | CLIP ViT-B/32 | 84.50 ± 0.37 | 88.58 ± 0.32 | 88.34 ± 0.33 | 91.09 ± 0.28 |
| Janus-Pro-7B [2] | CLIP ViT-B/32 | 86.26 ± 0.44 | 89.36 ± 0.38 | 88.83 ± 0.45 | 92.17 ± 0.42 |
| Qwen2.5-VL-7B [3] | CLIP ViT-B/32 | 85.44 ± 0.35 | 89.03 ± 0.30 | 88.91 ± 0.41 | 91.82 ± 0.35 |
| Qwen3-VL-8B | BERT [4] | 84.71 ± 0.35 | 88.19 ± 0.28 | 88.16 ± 0.30 | 91.65 ± 0.30 |
| Qwen3-VL-8B | CLIP ViT-B/32 | 85.98 ± 0.37 | 89.39 ± 0.28 | 89.36 ± 0.33 | 92.03 ± 0.24 |
[1] Liu H, Li C, Li Y, et al. Improved baselines with visual instruction tuning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 26286-26296.
[2] Chen X, Wu Z, Liu X, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling[J]. arXiv preprint arXiv:2501.17811, 2025.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025b. URL https://arxiv.org/abs/2502.13923.
[4] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.
Response to Reviewer CvVw (3/4)
[W2&Q3] Do DPI and CMA generalize to standard visual backbones such as ViT-S/16, ViT-B/16, or Swin-T? Can the authors provide results on ViT-B/16?
[Response] MAD-SP achieves comparable accuracy gains with ViT-S/16, Swin-T, and DeiT-Ti/16, demonstrating the generalization of our method across different backbones. On miniImageNet and CIFAR-FS, MAD-SP improves over the corresponding baseline with the same backbone in all 12 additional settings, with gains ranging from 3.89 to 12.82 percentage points. Averaged across both datasets and shot settings, the gains are 8.56 points with ViT-S/16, 7.45 points with Swin-T, and 8.22 points with DeiT-Ti/16, compared with 9.35 points with Visformer-T. All three alternative backbones also follow the same pattern as Visformer-T, with larger gains in the 1-shot setting on both datasets. These results demonstrate that DPI can be applied effectively within MAD-SP across the tested Transformer-based visual backbones. Following the reviewer's suggestion, we have included these experiments and their analysis in Appendix A.8 (p. 24, lines 1244–1268; Table 17, p. 24, lines 1249–1261).
| Method | Backbone | miniImageNet 1-shot | miniImageNet 5-shot | CIFAR-FS 1-shot | CIFAR-FS 5-shot |
|---|---|---|---|---|---|
| Baseline | ViT-S/16 | 73.45 | 83.35 | 77.81 | 87.06 |
| MAD-SP | ViT-S/16 | 86.27 (+12.82) | 89.30 (+5.95) | 88.64 (+10.83) | 91.69 (+4.63) |
| Baseline | Swin-T | 73.13 | 83.74 | 79.70 | 88.22 |
| MAD-SP | Swin-T | 84.42 (+11.29) | 89.12 (+5.38) | 88.94 (+9.24) | 92.11 (+3.89) |
| Baseline | DeiT-Ti/16 | 71.16 | 82.80 | 77.04 | 86.96 |
| MAD-SP | DeiT-Ti/16 | 83.92 (+12.76) | 88.52 (+5.72) | 87.25 (+10.21) | 91.14 (+4.18) |
| Baseline | Visformer-T | 73.43 | 82.62 | 77.18 | 86.13 |
| MAD-SP | Visformer-T | 85.98 (+12.55) | 89.39 (+6.77) | 89.36 (+12.18) | 92.03 (+5.90) |
[Q4] For novel cross-domain classes such as CUB and ChestX, are support captions generated online at test time or cached in advance? If generated online, how does LLM latency affect inference speed?
[Response] We thank the reviewer for raising this important question regarding the cross-domain evaluation protocol and inference efficiency.
We clarify that MACR is designed as a separate semantic prompt preparation stage rather than an online component of the recognition network. For novel cross-domain classes, such as those in CUB and ChestX, captions are generated from labeled support images and their corresponding class names. Once generated and refined, the resulting semantic prompts are encoded by the frozen CLIP text encoder and cached for subsequent use.
First, this process does not require retraining the LLM or accessing query labels. During recognition with cached prompts, the model directly retrieves the corresponding semantic embeddings, while DPI dynamically selects the injection layer for each support image. Query images are processed by the visual backbone without LLM involvement.
Second, regarding efficiency, we acknowledge that MACR introduces an additional one-time preprocessing cost, which should be distinguished from the inference latency of the recognition network. Since the refined prompts can be reused, repeated episodic evaluation does not require regenerating captions for previously processed support images. However, the initial caption preparation cost remains relevant when encountering new support samples.
Following the reviewer's suggestion, we have clarified this protocol and the distinction between preparation and recognition costs in Section 5.5 (p. 10, lines 520–521) and more details in Appendix A.2 (p. 19, lines 998–1008).
Response to Reviewer CvVw (1/4)
We sincerely thank the reviewer for the valuable and constructive feedback. We appreciate the reviewer’s recognition of how MACR’s multi-agent caption refinement complements DPI’s image-specific layer selection, as well as the strong results across 16 benchmark settings and the supporting ablation studies and visualizations. The reviewer’s questions about computational cost and generalization to other visual backbones helped guide our additional experiments. Our point by point responses are provided below.
[Response] We thank the reviewer for pointing out this potential ambiguity.
Exactly, MACR operates as an offline multi-agent refinement procedure rather than a dynamic multi-agent system that continuously interacts during meta-training. We would like to clarify that the term "dynamic" in MAD-SP primarily refers to the instance-adaptive prompt injection mechanism (DPI), rather than the execution of the multi-agent refinement process. Specifically, MACR employs three specialized agents (Generator, Critic, and Refiner) to iteratively improve semantic prompts under score-guided feedback before meta-training. The resulting prompts are cached for subsequent use, and the agents do not participate in gradient-based optimization, as described in Algorithm 1 (p. 17, lines 868–896).
In contrast, DPI dynamically determines the appropriate prompt injection layer for each support image during meta-training and episodic inference, based on its visual features and corresponding semantic prompt. Unlike fixed-layer injection, this decision is input-dependent and is performed by a learnable decision network.
Therefore, MAD-SP integrates offline multi-agent semantic refinement with dynamic, instance-adaptive prompt injection, rather than claiming that the multi-agent system itself operates dynamically throughout training. Following the reviewer's suggestion, we have added the clarification of the source of dynamic behavior in Section 1 (p. 2, lines 101–104). The separation between caption preparation and episodic optimization has been also specified in Appendix A.1 (p. 17, Algorithm 1, lines 872–888; discussion, p. 17, lines 891–896), with the preparation and caching protocol in Appendix A.2 (p. 19, lines 998–1008).
[Response] We have reported the full framework’s computational cost and compared it with other methods. Measurements on miniImageNet show that MAD-SP has modest computational requirements: 10.7M downstream parameters, comprising a 9.94M visual backbone, 0.70M for CMA, and 0.05M for the DPI layer selection network. CMA and DPI add only 0.75M parameters, about 7.5% of the backbone’s parameter count. Qwen3-VL-8B’s fast responses keep semantic preprocessing, including initial caption generation and all refinement rounds, to a practical one-time cost of 2.0 hours. Training takes 34 minutes, within the baselines’ 32–57 minute range. MAD-SP has the fewest parameters, matches SP at the lowest computational cost of 1.3 GFLOPs, and has the lowest inference latency (51 ms versus 81–135 ms) among the compared methods. Following the reviewer’s suggestion, we have included these measurements in Section 5.5 (p. 10, Table 8, lines 486–494; discussion, p. 10, lines 513–521).
[1] Chen W, Si C, Zhang Z, et al. Semantic prompt for few-shot image recognition[J]. arXiv preprint arXiv:2303.14123, 2023.
[2] Zhang H, Xu J, Jiang S, et al. Simple semantic-aided few-shot learning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 28588-28597.
[3] Liu M, Wu F, Li B, et al. Envisioning class entity reasoning by large language models for few-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(18): 18906-18914.
[4] Li W, Meng X, Wang Q, et al. Dvla-rl: Dual-level vision-language alignment with reinforcement learning gating for few-shot learning[J]. arXiv preprint arXiv:2602.00795, 2026.