- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a meta-promptable Mahalanobis matching framework for cross-domain few-shot learning (CD-FSL). The core idea is to replace isotropic point prototypes with prompt-conditioned anisotropic Gaussian class distributions, and to classify queries by Mahalanobis distance. Furthermore, a meta-learner is designed to estimate class-wise means and covariances through cross-modal fusion and low-rank factorization.
Critical strengths
Strength 1:Replacing point prototypes with Gaussian distributions is a valuable direction for few-shot cross domain learning. Using semantic descriptions to further enrich and characterize the prototypes can also help improve generalization.
Strength 2: Broad and unified evaluation. The paper evaluates the proposed framework on a wide range of cross-domain few-shot benchmarks covering segmentation, classification, and detection.
Critical weaknesses
Weaknesses 1: The paper's core idea "replacing isotropic prototypes with anisotropic Gaussian distributions and using Mahalanobis matching" is very similar to Simple CNAPS [1]. In my view, the only difference is that [1] shows covariances can be estimated from very few samples by feature design, while this paper uses a different mechanism, i.e., LLM CoT prompts and a meta-learner for covariance estimation. However, the paper does not cite [1], and also does not provide any evidence to show the necessity or advantage of using such mechanism for covariances estimation. The lack of novelty is the primary reason that I assigned the current score.
Weaknesses 2: The proposed meta-learner predicts the mean and covariance based on the prompt and image features. My concern is whether such a design can genuinely realize semantic injection for prototypes. As the authors state, "Fig. 4 shows that our method remains stable under noisy or ambiguous captions" and "The small performance variation indicates that our method does not depend on a specific VLM or LLM." This may indicate that the proposed method fails to properly inject semantics into the prototypes.
[1] Bateni P, Goyal R, Masrani V, et al. Improved few-shot visual classification[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 14481-14490.
Questions for authors
Question 1: :Can the authors compare with Simple CNAPS [1], or provide evidence that [1] is insufficient for covariance estimation, or explain the advantage of the proposed method over [1]?
Question 2:Can the authors justify the proposed method effectively injects semantics into the prototypes? For example, if the prompt is replaced, would the model output a significantly different prototype, and can this difference be verified? For instance, given two prompts, "wooden boat" and "iron boat," both added to a prototype "boat". Then, can the model generate two substantially different prototypes that genuinely enable it to distinguish between “wooden boats” and “iron boats”?
14 comments
Response to Reviewer pVn4 (Part-2)
Q2. Does the Meta-Learner effectively inject semantic information?
Response:
Yes. We first clarify that robustness to noisy or semantically similar descriptions is different from insensitivity to semantic content. In our framework, prompts condition both the mean and covariance of the estimated class distribution, rather than merely being appended to a fixed prototype.
Fig. 4 evaluates robustness to unreliable descriptions. When a caption contains ambiguous or visually unsupported words, text-image cross-attention assigns lower weights to tokens that are inconsistent with the support features. The residual visual pathway further prevents unreliable text from overriding the observed evidence. Therefore, robustness to caption noise indicates that the model selectively filters unsupported semantics, not that it ignores the prompt.
The comparison across different VLMs and LLMs has a similar interpretation. Although these models produce descriptions with different wording and levels of detail, the underlying category and domain semantics remain largely consistent. The small performance variation therefore demonstrates robustness to the choice of prompt generator rather than insensitivity to semantic content.
Quantitative evidence from the submitted experiments.
If the Meta-Learner ignored semantic information, removing or modifying the prompts should have little effect. Our ablations show the opposite.
| Variant | Average 1-shot mIoU | Average 5-shot mIoU | Drop from Ours |
|---|---|---|---|
| Ours | 76.6 | 80.5 | – |
| Ours w/o Prompt Generator | 64.6 | 71.0 | -12.0 / -9.5 |
| Ours w/o Domain-aware Prompt | 72.8 | 76.4 | -3.8 / -4.1 |
| Ours w/o Covariance Prompt | 72.3 | 76.1 | -4.3 / -4.4 |
| Ours w/o CoT | 73.6 | 77.2 | -3.0 / -3.3 |
These results support three conclusions:
-
Removing all semantic prompts causes substantial drops of 12.0 and 9.5 points, showing that the Meta-Learner does not rely on visual features alone.
-
Domain-aware and covariance prompts make distinct contributions. Removing either prompt consistently degrades performance, indicating that the model uses both target-domain context and category-specific variation cues.
-
Prompt quality also matters. Replacing structured CoT descriptions with direct prompting reduces performance by 3.0 and 3.3 points. If the textual information were ignored, changing its semantic completeness would not produce this consistent degradation.
Additional qualitative evidence.
Following the reviewer’s suggestion, we construct four paired few-shot cases from FSS-1000, each placing two semantically related categories in the same comparison setting. FSS-1000 is suitable for this analysis because it contains fine-grained categories that share a high-level concept but differ in language-describable attributes. The selected pairs are black swan versus swan, shakuhachi versus flute, school bus versus double-decker bus, and black stork versus white stork.
Despite their semantic similarity, the categories within each pair differ in color, material, or structural attributes. Our method accurately segments the corresponding targets while preserving these distinctions, rather than collapsing them into a shared coarse representation. This suggests that category-specific descriptions are integrated with visual evidence to produce discriminative distribution estimates.
These cases provide qualitative evidence rather than an isolated prompt-swapping experiment. The causal evidence that the model uses semantic information comes from the submitted ablations: removing the Prompt Generator, domain prompt, covariance prompt, or structured CoT descriptions consistently degrades performance. The paired cases complement these results by showing that fine-grained semantic distinctions are retained during inference. Since OpenReview does not support image insertion in the response box, we include the visualizations in Fig. 5 of the revised manuscript.
Together, the quantitative ablations and qualitative cases show that the model is sensitive to task-relevant semantic content while remaining robust to unsupported or semantically equivalent wording. Therefore, robustness to noisy captions or different foundation models does not imply that the Meta-Learner ignores semantic prompts.
Dear Reviewer, as the discussion deadline approaches, we would appreciate hearing whether you have any remaining questions or concerns about our paper or rebuttal. We would be happy to provide further clarification or additional information. Thank you again for your time and thoughtful feedback!
Response to Reviewer pVn4 (Part-1)
Q1. Relationship to Simple CNAPS.
Response:
We thank the reviewer for pointing out Simple CNAPS. We agree that it is closely related and will cite and discuss it in the revised paper. Simple CNAPS and our method share the general strategy of representing classes with Gaussian distributions and performing prediction using Mahalanobis distance. We clarify, however, that this decision rule itself is not the main technical contribution of our work. Modeling a class as a Gaussian distribution and using Mahalanobis distance is a natural extension of prototype matching. The central problem addressed in our work is how to estimate a reliable class-specific anisotropic distribution from extremely limited visual observations under domain shift. This presents two difficulties.
First, the source-domain representation does not necessarily preserve the target-domain class geometry. With only one or a few target examples, empirical covariance estimation is highly unstable and cannot adequately capture domain-specific appearance and intra-class variation. We therefore introduce semantic priors that explicitly describe category attributes, target-domain context, and plausible variation patterns. These textual priors complement the limited visual observations and guide target-domain distribution estimation.
Second, estimating a high-dimensional covariance matrix from few support samples is severely underdetermined. Our meta-learner addresses this issue through episodically amortized estimation, low-rank covariance factorization, and residual visual statistics. Rather than independently estimating a covariance matrix from each target support set, the model learns a transferable estimator across source-domain episodes and adapts it using target visual and semantic evidence.
Therefore, our contribution is not Mahalanobis matching itself, but a prompt-conditioned and data-efficient mechanism for estimating anisotropic class distributions under cross-domain few-shot constraints. The same estimator is further applied to segmentation pixels, classification images, and detection proposals, whereas Simple CNAPS focuses on image classification.
The existing ablations directly support the necessity of the proposed estimator.
The variant without Meta-Promptable Distribution Modeling directly computes means and covariances from visual feature statistics. Its substantial performance drop shows that direct few-shot covariance estimation is insufficient under domain shift. Removing the Prompt Generator causes a similarly large degradation, confirming that semantic priors provide information that cannot be recovered from scarce visual samples alone. The covariance-factorization and residual-strategy ablations further validate the proposed data-efficient estimation design.
Our framework also produces consistent improvements across fourteen benchmarks covering segmentation, classification, and detection. This indicates that the proposed estimator is not tied to a particular prediction unit or task.
Direct comparison with Simple CNAPS.
Following the reviewer’s suggestion, we additionally implement the Simple-CNAPS under the same CD-FSS protocol. All methods use the same backbone, and evaluation settings.
As shown, our method consistently and significantly outperforms Simple CNAPS across all datasets and settings. We will add the results, cite Simple CNAPS, and clarify the scope of our contribution in the revised manuscript.