acceptodds
Withdrawn from ICLR 2027

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical Vision-Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable reasoning. However, a critical performance gap exists between their strong vision encoders and the full multimodal model: in dermatology, the MedSigLIP encoder outperforms MedGemma by an average of 10.26 percentage points even when both use zero target-task labels; few-shot linear probing provides further evidence of strong visual representations. This gap motivates an investigation of how visual information is used in end-to-end diagnosis and why plausible-sounding predictions can lack grounding in image evidence. Using dermatology as our primary testbed, we systematically investigate three hypotheses for this phenomenon. We further provide a mechanistic analysis of the model's internal attention patterns, showing that a simple describe-then-decide prompting strategy increases vision attention by 30–40% during generation. Task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in our evaluation. To address these challenges, we combine label-free prompting with low-label encoder-assisted reranking while keeping the VLM frozen. We validate the interventions across five VLM backbones in dermatology and provide supporting representation and attention analyses across additional medical modalities.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Medical LLM, Skin Disease

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Xy61OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the gap between medical VLMs and their vision encoders in skin disease classification. MedSigLIP outperforms MedGemma even when neither uses labels from the target task. The authors explore three possible causes: data mismatch, limited use of images, and different training goals for the vision and language components. The method uses a linear probe trained on a few labeled examples to shortlist diagnoses, then guides the frozen VLM with clinical descriptions and a prompt to describe the image before deciding. Experiments report accuracy gains across five VLMs and multiple skin disease datasets. The authors also analyze changes in visual attention.

Critical strengths
  1. The paper presents three hypotheses about the vision-language gap and examines them through targeted interventions. Clinical descriptions, DtD, and candidate selection offer simple, practical ways to improve classification while keeping the VLM frozen.

  2. Experiments cover five dermatology datasets and several VLMs, with additional representation and attention analyses in other medical modalities. Zero-label comparisons support the observed gap, while prompt controls and ablations help identify the sources of improvement.

Critical weaknesses
  1. The method mainly combines existing strategies, so its contribution lies largely in the empirical analysis. Higher visual attention alone does not establish that final predictions rely more on images. Description quality is also scored by the same model, without independent validation. These limitations weaken the evidence for the proposed mechanism.

  2. Tables 1 and 3 report different Top-5-to-Top-1 results, although both use an 8-shot linear probe. On Fitzpatrick17k, the result in Table 3 is even lower than the 8-shot linear probe alone, which contradicts the claim in Sec. 4.3. Table 3 also omits a linear-probe-only baseline. Since the full pipeline uses labeled data, this is the appropriate comparison. Other values are also inconsistent across tables, such as the SkinVL zero-shot accuracy on eSkinHealth (Table 4 vs. Table 21). These issues make the main results difficult to verify.

Questions for authors
  1. What is the main novelty of this work compared with existing prompting and candidate selection methods?

  2. Why do Tables 1 and 3 report reranking accuracies of 37.82% and 28.32% on Fitzpatrick17k? Both experiments use a linear probe trained with 8 shots. Are the data splits, support sets, and prompts identical?

  3. How is abstention prompted and scored in the masking experiment? Do DA and DtD have the same option to abstain? What are their abstention rates on the unmasked images?

  4. How does VAR differ between DA and DtD when computed only over the final diagnosis tokens? Since description tokens naturally attend to the image, the overall VAR increase may not reflect the decision step itself.

  5. How are candidates ordered during reranking? With the candidate set fixed, how does accuracy change when their order is shuffled or the input image is replaced?

Reviewer_BiHnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the gap between the vision encoder of a medical VLM and the full VLM on dermatology diagnosis. Using MedGemma-4B and its encoder MedSigLIP, it shows that the encoder (zero-shot image–text similarity or few-shot linear probing) outperforms the VLM's zero-shot answer on five dermatology datasets, with an average zero-label gap of 10.26 points. The paper proposes three hypotheses for this gap: (1) train–test distribution mismatch; (2) under-reliance on the vision branch; (3) objective misalignment between the discriminative encoder and the generative LLM. It pairs each hypothesis with an inference-time intervention: (1) class-aligned clinical descriptions in the prompt; (2) a describe-then-decide (DtD) prompt; (3) a "Top-5 to Top-1" pipeline, where a few-shot linear probe on the encoder shortlists candidates and the frozen VLM picks one. The combined pipeline improves Top-1 accuracy on the five datasets. Transfer is shown on SkinVL, InternVL3/3.5 and biomed-Qwen. A Vision Attention Ratio (VAR) analysis shows that DtD increases the attention to vision tokens by 30–40% across six medical modalities.

Critical strengths

The paper addresses a practically relevant problem for medical VLMs. Several of its controls are well designed. The zero-label MedSigLIP comparison (Table 1, Table 7) shows that the gap does not come only from fitting a classifier on target labels. The length-matched and shuffled description controls (Table 2) show that the gain from clinical descriptions comes from class-aligned content rather than prompt length. The interventions are simple, keep the VLM frozen, and are evaluated on several backbones.

Critical weaknesses

(1) The novelty is limited.

The core observation that a VLM classifies much worse than its own vision encoder has already been reported for general-domain VLMs. [1] showed that VLMs lag far behind CLIP on classification and attributed this mainly to class frequency in the training data, i.e., Hypothesis 1. [2] showed that VLMs underperform their vision encoders, follow the language priors of the LLM, and are bottlenecked by how the LLM uses visual features, overlapping with Hypotheses 2 and 3. Neither work is cited. The interventions are also close to existing techniques: class descriptions in the prompt [3, 4], describe-first prompting [5], and the low attention to visual tokens reported in [6] and Look-Back (Yang et al., 2025). For medical MLLMs, [7] already showed, against expert-annotated regions across eight models, that they often fail to ground predictions in clinically relevant regions, which overlaps with Hypothesis 2 and the heatmap analysis (Appendix E). The main contribution is therefore applying known findings to dermatology, and the paper should clarify its relationship to [1–7].

(2) The three hypotheses are not actually tested.

Each hypothesis is supported only by the gain of its paired intervention, which is confirmatory and cannot separate the hypotheses (e.g., clinical descriptions would also help under H2 or H3).

  • H1: no training-distribution analysis is given, and the evidence partly contradicts it: the model describes BCC and LE correctly (Table 25), and the gap persists on the in-domain PAD-UFES-20 (46.91 vs. 55.42).
  • H2: the abstention rate of DtD on unmasked images is not reported, so DtD may simply abstain more often rather than ground its decisions better.
  • H3: description quality is judged by the same model that generated it (Fig. 4c), which is circular; expert concept annotations such as SkinCon [8] could serve as ground truth.
  • VAR is averaged over all generated tokens, most of which are description tokens under DtD, so the higher VAR may reflect the longer, image-describing response rather than better grounding of the diagnosis.

(3) Several results and the evaluation protocol are inconsistent.

  • The same 8-shot Top-5-to-Top-1 setting differs across tables: Fitzpatrick17k 37.82 (Tables 1, 9) vs. 28.32 (Tables 3, 10), PAD-UFES-20 80.31 vs. 75.23, eSkinHealth 42.05 vs. 43.94. The zero-shot PAD-UFES-20 baseline is 46.91 (Table 1) vs. 46.22 (Tables 2, 3), and SkinVL zero-shot on eSkinHealth is 23.76 (Table 4) vs. 13.76 (Table 21).
  • LoRA: the learning-rate grid differs between Table 17 ({5e-5, 1e-4, 2e-4}, 1e-4 selected) and Table 18 ({5e-5, 3e-4, 4e-4}), whose best entry (47.32 at 3e-4) equals the test accuracy in Table 11, suggesting selection on the test set. The limited setup (4-bit QLoRA, rank at most 16, 10 epochs) also weakens the claim that fine-tuning underperforms LP.
  • InternVL3-4B (5.68%) and InternVL3.5-4B (7.21%) are at or below chance (7.14%) on the 14-class Derm7pt, suggesting answer-parsing failures; the rate of invalid outputs should be reported.
  • Top-5-to-Top-1 uses labels, so it should be compared with label-efficient encoder adaptation (e.g., Tip-Adapter [9], LP++ [10]) rather than the zero-shot VLM. Even against plain LP, it is lower on Fitzpatrick17k at 8 shots and on three of five datasets at 16 shots (Table 20).
  • The claim that MedSigLIP is ''comparable to, and at times superior to'' PanDerm does not hold in F1 on Derm7pt (30.07 vs. 57.93) and eSkinHealth (42.51 vs. 59.83), despite similar or higher accuracy, suggesting poor minority-class performance that should be discussed.

Reference

[1] Why are Visually-Grounded Language Models Bad at Image Classification? (Zhang et al., NeurIPS 2024)

[2] Hidden in Plain Sight: VLMs Overlook Their Visual Representations. (Fu et al., arXiv 2025)

[3] Visual Classification via Description from Large Language Models. (Menon & Vondrick, ICLR 2023)

[4] What Does a Platypus Look Like? Generating Customized Prompts for Zero-Shot Image Classification. (Pratt et al., ICCV 2023)

[5] Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs. (Ghosh et al., ICLR 2025)

[6] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. (Chen et al., ECCV 2024)

[7] How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images. (Liu et al., ICLR 2026)

[8] SkinCon: A Skin Disease Dataset Densely Annotated by Domain Experts for Fine-Grained Debugging and Analysis. (Daneshjou et al., NeurIPS 2022 Datasets and Benchmarks)

[9] Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification. (Zhang et al., ECCV 2022)

[10] LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP. (Huang et al., CVPR 2024)

Questions for authors

Apart from the main concerns in Weaknesses, here are the minor concerns:

The zero-shot VLM results are reported with standard deviations, and the same Derm7pt baseline (16.46) appears with ±0.01 (Table 9), ±0.02 (Table 11) and ±0.17 (Table 3). With greedy decoding, what is varied across runs?

Fig. 4a labels the y-axis as ''Vision Attention Ratio (%)'' with values up to 0.14, while Table 15 reports mean VAR of about 3% and Fig. 7 reports layer-17 VAR of about 7–13%. The values in Fig. 4a and Fig. 8 seem to be fractions rather than percentages; please clarify.

Reviewer_yKQNOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a notable paradox in medical vision-language models: their vision encoders are far stronger at diagnosis than the full end-to-end model that houses them. Using dermatology as the primary testbed and MedGemma-4B (with the MedSigLIP encoder) as the main backbone, the authors show that a few-shot linear probe on the frozen encoder—and even a label-free image–text similarity baseline—outperforms the VLM's zero-shot diagnosis by ~10 percentage points on average across five skin-lesion datasets, and that LoRA fine-tuning still fails to close this gap. They then examine three hypotheses for the gap (train/test distribution mismatch, over-reliance on the language branch, and a discriminative-vs-generative objective misalignment), and propose three fine-tuning-free inference interventions: class-aligned in-context clinical descriptions, a describe-then-decide (DtD) prompt, and an encoder-assisted "Top-5-to-Top-1" reranking pipeline. A mechanistic attention analysis (Vision Attention Ratio and attention entropy) is used to argue that DtD shifts 30–40% more generation-time attention onto visual tokens, and the interventions are validated across five VLM backbones and extended to five additional medical imaging modalities.

Critical strengths

The work tackles a genuinely important and timely reliability problem for clinical VLMs, and it is organized in a clear, hypothesis-driven manner that makes the narrative easy to follow.

Critical weaknesses

There are evaluation-protocol concerns (custom 0.5 case/class splits for several datasets that may not match official or prior protocols, SD-260 reduced to an overlapping subset, absolute accuracies that are low enough that some deltas are within noise), and the work offers no clinician study or inter-rater analysis to substantiate the "safety-critical / clinical reliability" framing that motivates it.

Questions for authors

There are evaluation-protocol concerns (custom 0.5 case/class splits for several datasets that may not match official or prior protocols, SD-260 reduced to an overlapping subset, absolute accuracies that are low enough that some deltas are within noise), and the work offers no clinician study or inter-rater analysis to substantiate the "safety-critical / clinical reliability" framing that motivates it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.