- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the gap between medical VLMs and their vision encoders in skin disease classification. MedSigLIP outperforms MedGemma even when neither uses labels from the target task. The authors explore three possible causes: data mismatch, limited use of images, and different training goals for the vision and language components. The method uses a linear probe trained on a few labeled examples to shortlist diagnoses, then guides the frozen VLM with clinical descriptions and a prompt to describe the image before deciding. Experiments report accuracy gains across five VLMs and multiple skin disease datasets. The authors also analyze changes in visual attention.
Critical strengths
-
The paper presents three hypotheses about the vision-language gap and examines them through targeted interventions. Clinical descriptions, DtD, and candidate selection offer simple, practical ways to improve classification while keeping the VLM frozen.
-
Experiments cover five dermatology datasets and several VLMs, with additional representation and attention analyses in other medical modalities. Zero-label comparisons support the observed gap, while prompt controls and ablations help identify the sources of improvement.
Critical weaknesses
-
The method mainly combines existing strategies, so its contribution lies largely in the empirical analysis. Higher visual attention alone does not establish that final predictions rely more on images. Description quality is also scored by the same model, without independent validation. These limitations weaken the evidence for the proposed mechanism.
-
Tables 1 and 3 report different Top-5-to-Top-1 results, although both use an 8-shot linear probe. On Fitzpatrick17k, the result in Table 3 is even lower than the 8-shot linear probe alone, which contradicts the claim in Sec. 4.3. Table 3 also omits a linear-probe-only baseline. Since the full pipeline uses labeled data, this is the appropriate comparison. Other values are also inconsistent across tables, such as the SkinVL zero-shot accuracy on eSkinHealth (Table 4 vs. Table 21). These issues make the main results difficult to verify.
Questions for authors
-
What is the main novelty of this work compared with existing prompting and candidate selection methods?
-
Why do Tables 1 and 3 report reranking accuracies of 37.82% and 28.32% on Fitzpatrick17k? Both experiments use a linear probe trained with 8 shots. Are the data splits, support sets, and prompts identical?
-
How is abstention prompted and scored in the masking experiment? Do DA and DtD have the same option to abstain? What are their abstention rates on the unmasked images?
-
How does VAR differ between DA and DtD when computed only over the final diagnosis tokens? Since description tokens naturally attend to the image, the overall VAR increase may not reflect the decision step itself.
-
How are candidates ordered during reranking? With the candidate set fixed, how does accuracy change when their order is shuffled or the input image is replaced?