acceptodds
Under review as a conference paper at ICLR 2027

Trust What You See or What You Know? Evidence-Complementary Prototype Prompting for Biomedical Vision-Language Models

Abstract

Few-shot adaptation of biomedical vision–language models can draw on two kinds of evidence: text descriptions that know each class in general terms, and labeled images that show how the classes look in the target task. We observe that the two are reliable in opposite regimes: the text prior helps most when labeled images are scarce, while class prototypes built from the labeled images overtake it as the images accumulate. Most existing prompt learners, however, keep the influence of the text fixed no matter how many labels are available. A second problem appears when prototypes are simply added to a prompt trained with distillation: the distillation loss pulls their weight toward zero, because the text teacher has never seen the labeled images. We therefore propose Evidence-Complementary Prototype Prompting (ECPP), which lets each kind of evidence contribute where it is reliable. Concretely, (1) an evidence handoff gradually shifts trust from the text prior to a prototype classifier as more labeled images become available, and (2) text-path distillation keeps the teacher's guidance on the text branch, so that the prototype weight is learned from the labels alone; we show analytically why this placement matters. ECPP adds only one trainable scalar to a frozen BiomedCLIP prompt learner. On eleven biomedical datasets spanning nine imaging modalities, ECPP improves over the state-of-the-art BiomedCoOp at every support size (with gains of up to 5.34% on breast ultrasound) and achieves the best base-to-novel harmonic mean among the compared methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
prompt learning, vision-language models, biomedical image classification, few-shot learning, parameter-efficient adaptation, knowledge distillation, prototype-based classification, base-to-novel generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MaAuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ECPP, a few-shot adaptation method for biomedical vision-language models that combines text priors with visual class prototypes. It introduces an evidence handoff mechanism that shifts reliance from text to prototypes as more labeled images become available, together with text-path distillation to prevent the teacher from suppressing visual evidence. Experiments on 11 biomedical datasets show consistent improvements over BiomedCoOp with minimal additional parameters.

Critical strengths
  1. The paper addresses an important problem in few-shot biomedical VLM adaptation: how to balance text priors and visual evidence under different support sizes.
  2. ECPP achieves consistent improvements over BiomedCoOp across 11 biomedical datasets while adding only one trainable scalar.
Critical weaknesses
  1. My main concern is the limited novelty. Several prior works already combine visual support evidence with CLIP text predictions, including Tip-Adapter, Proto-Adapter, and Proto-CLIP.
  2. Several design choices appear heuristic. In particular, a simple 0.5/K prior-weighting rule achieves 66.02% mean accuracy versus 66.04% for ECPP, raising questions about the necessity of the more complex uncertainty-aware weighting.
  3. The contribution of the text-path distillation is relatively limited. Although the proposed placement is well motivated, the ablation shows an average improvement of only 0.42 percentage points in few-shot accuracy, which makes its empirical impact modest compared with the other components.
Questions for authors
  1. Could the authors better justify the novelty of ECPP relative to existing prototype-based and text-visual fusion methods?
  2. Could the authors clarify the practical benefit of the proposed uncertainty-aware weighting, given that the simple 0.5/K rule achieves nearly identical performance in Table S8?
Reviewer_KTyEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence-Complementary Prototype Prompting (ECPP) for few-shot and base-to-novel adaptation of biomedical vision-language models (building directly on BiomedCoOp). The method combines three classifiers, a learned prompt (zθz_\theta), a frozen LLM description prior (z0z_0), and a support-set visual prototype classifier (v(X)v(X)), using shot-dependent weighting (rK=(K−1)/Kr_K = (K-1)/K for prototypes and 1/K1/K decay for the text prior), while keeping BiomedCoOp's knowledge distillation loss on the text branch rather than applying it to the final fused logits ("text-path distillation"). Experiments on 11 biomedical datasets show improvements over BiomedCoOp across KK values and in base-to-novel generalization.

Critical strengths

Clear Presentation and Visuals

The paper is well-written, and the figures clearly illustrate the empirical motivation and computational graph.

Consistent Empirical Numbers Across Datasets with Negligible Computational Overhead:

On the evaluated benchmark, the combined pipeline achieves solid gains over existing prompt-learning and adapter baselines. The method adds only a single learnable scalar β\beta and caches support prototypes once before training, keeping training and inference costs nearly identical to BiomedCoOp (Table 5).

Critical weaknesses

1. Limited Novelty

  • Prototype Mixing: The paper introduces a residual combination of visual prototype logits with textual (prior and learned) logits. However, adding scaled visual support-set prototype scores to text logits is already standard practice in methods such as Tip-Adapter (Zhang et al., 2022) and LP++ (Huang et al., 2024). Thus, the core mathematical operations of ECPP are combinations of well-established ensembling techniques.

  • Text-Path vs. Fused Distillation: The paper presents Text-Path Distillation as one of its two core contributions (highlighted in the Abstract, Figure 1b, Section 3.4, and Equation 6). However, "fused distillation" is neither a pre-existing method nor a known issue in the literature. Instead, it is a naive baseline constructed by the authors. BiomedCoOp,distills zθz_\theta, its only prediction, so ECPP's "text-path" placement is that same loss left where it was.. When augmenting BiomedCoOp with a visual prototype residual, keeping the zero-shot text teacher's supervision on the text branch is the obvious default. Forcing a visual support-prototype classifier to match a zero-shot text teacher (which has no access to visual support samples) is, to me, counterintuitive.

2. Missing Controls

  • Is Joint Training of β\beta Necessary? Because Text-Path Distillation isolates the distillation loss to the text path u(X)u(X) while the prototypes pcp_c and prior z0z_0 remain entirely frozen, it is unclear whether jointly training θ\theta and β\beta is necessary at all. What happens if we take an off-the-shelf trained BiomedCoOp model (zθz_\theta) and simply ensemble z0z_0 and the visual prototypes v(X)v(X) post-hoc at inference time (either using a fixed β\beta or tuning the mixing weight on the validation set, just as the authors already do for K=2K=2 prototype interpolation)? Without this baseline, it is impossible to disentangle the benefits of joint optimization from simple post-hoc ensembling.

  • Base-to-novel: novel accuracy measures the text prior, not the learned prompt. Training uses Eq. 3 (prior weight <0.0625< 0.0625 at K=16K=16), but novel classes are predicted with a fixed 0.1 zθ+0.9 z00.1\,z_\theta + 0.9\,z_0. Table S2 shows that $a = 1.0$, i.e. dropping the learned prompt entirely, gives the best HM for both ECPP (76.35) and BiomedCoOp (75.24), with identical novel accuracy (77.44). The learned prompt adds nothing on novel classes for either method, so the "best base-to-novel HM" claim reduces to higher base accuracy (78.00 vs 75.68) plus a zero-shot novel classifier. Report results under the training-time rule (Eq. 3) as well, and explain why the reproduced BiomedCoOp (75.68 / 71.31 without the rule) is below the published 76.26 / 73.92.

  • Regime-Specific Hyperparameters and Heuristics:

    • K=1K=1 rule: At K=1K=1, the visual residual is disabled (r1=0r_1 = 0) and the 1/K1/K schedule is replaced by a bespoke formula, α(X,1)=q2g(X)\alpha(X, 1) = q^2 g(X) with q=clip((C−1∑cpc⊤t0,c−0.30)/0.15,0,1)q = \text{clip}((C^{-1}\sum_c p_c^\top t_{0,c} - 0.30)/0.15, 0, 1). Both the quadratic exponent on qq and the hardcoded clipping thresholds (0.30,0.150.30, 0.15) are tailored specifically to the K=1K=1 setup.
    • K=2K=2 prototype refinement: At K=2K=2 only, the authors abandon the plain average prototype used at K≥4K \ge 4. Instead, they weight the two support images by their agreement with t0,ct_{0,c} and interpolate toward t0,ct_{0,c}. This is an ad-hoc fix strictly for K=2K=2.
Questions for authors

**Clarification requests **

  1. Post-hoc baseline: take a trained BiomedCoOp model and add the frozen prototypes and prior at inference, $\ell = (1-\alpha) z_\theta + \alpha z_0 + \beta r_K v$, with β\beta either fixed (e.g. the learned mean 0.79) or chosen on the validation images. How does this compare to ECPP at each KK? Without it I cannot tell whether joint training of θ\theta and β\beta matters.

  2. Appendix F reports that distilling zθz_\theta instead of uu changes accuracy by −0.03 points. Does this mean text-path distillation is equivalent to BiomedCoOp's original placement, and if so, what is the contribution claimed for it?

  3. K=2: what is ECPP's accuracy at K=2K=2 with the plain average prototype (no agreement weighting, no interpolation toward t0,ct_{0,c})? The +3.04 margin at K=2K=2 is the paper's largest and currently cannot be attributed between the residual and the K=2-only refinement.

  4. Where were the constants 0.30 and 0.15 in the K=1K=1 rule, the q2q^2 exponent, and the entropy-gate constants (0.5, 0.25) chosen? Appendix F mentions a five-dataset development panel drawn from the test benchmark. which design choices were made on it?

  5. Base-to-novel: what are the results under the training-time rule (Eq. 3 with K=16K=16) on novel classes? And why is the reproduced BiomedCoOp (75.68 base / 71.31 novel without the rule) below the published 76.26 / 73.92?

**Suggestions **

  • Cite and discuss WiSE-FT (Wortsman et al., CVPR 2022); the novel-class rule is weight interpolation of the text classifier head.
  • Reduce the K=2K=2 rule to match K>=4K>=4.
  • Report the HM at a=1.0a = 1.0 in the main text, since it is the best setting for both models.
Reviewer_K7oUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ECPP for biomedical few-shot classification, motivated by the changing reliability of textual and visual evidence as the number of support samples increases. It introduces shot-aware evidence weighting, visual prototypes, and text-path distillation to improve few-shot adaptation. Experiments across 11 biomedical datasets, together with extensive ablations and supplementary analyses, demonstrate consistent improvements over BiomedCoOp.

Critical strengths
  1. Meaningful motivation and analysis.

The paper identifies an interesting relationship between support-set size and the usefulness of textual and visual evidence. The analysis of distillation interference is particularly valuable and supported by theoretical and empirical evidence.

  1. Comprehensive experimental validation.

The evaluation covers multiple biomedical datasets and few-shot settings. The supplementary ablations and statistical analyses strengthen the reliability of the reported improvements.

Critical weaknesses
  1. Limited advantage of the adaptive weighting mechanism.

The proposed evidence handoff relies largely on heuristic weighting rules. More importantly, Table S8 shows that a simple (0.5/K) rule achieves nearly identical accuracy to ECPP (66.02% vs. 66.04%), raising questions about the necessity and contribution of the uncertainty-based adaptive weighting.

  1. Limited analysis of support-set composition.

Although three support seeds are evaluated, prototype reliability may depend substantially on sample diversity and representativeness rather than shot count alone. The robustness of the proposed weighting mechanism to different support-set compositions remains insufficiently investigated.

Questions for authors

Given the nearly identical performance of the simple (0.5/K) rule, what is the specific advantage of the proposed uncertainty-based adaptive weighting mechanism?

How sensitive is ECPP to support-set composition at a fixed shot count, particularly when the selected examples are redundant or poorly representative?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.