acceptodds
Withdrawn from ICLR 2027

Is LLaVA-Med Visually Grounded?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical Vision-Language Models (Medical VLMs) offer a scalable pathway to support radiological workflows by automating report generation and clinical question answering. LLaVA-Med is one of the most widely adopted Medical VLMs. It is instruction-tuned on PubMed Central, enabling the original LLaVA model to adapt to the medical domain. However, we identify that LLaVA-Med is susceptible to a critical failure mode: rather than learning genuine visual-textual coherence, it often exploits biased language priors, effectively memorizing statistical patterns in clinical text and replaying them at inference time with limited regard to the actual image content. This undermines diagnostic reliability and poses a direct risk to patient safety in real-world clinical deployment. In this paper, we first propose a diagnostic framework to systematically identify and analyze the underlying causes of this failure mode. Building on these insights, we develop interventions that mitigate linguistic bias and encourage LLaVA-Med to generate responses that are more faithfully grounded in visual evidence. Results on in-distribution and out-of-distribution datasets like VQA-RAD, SLAKE and PathVQA support the claim.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language model, linguistic bias, medical, LLaVA-Med

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zD7wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether LLaVA-Med genuinely relies on visual evidence, using VFI, representation statistics, and downstream evaluation. It proposes LLaVA-MF, combining BiomedCLIP, Q-Former, and LoRA. The question is important, but the current evidence does not sufficiently support the main claims about improved visual grounding and its underlying mechanisms.

Critical strengths
  1. The paper addresses an important reliability question: whether medically plausible answers are actually supported by the input image.
  2. Combining image perturbations, encoder and projector representations, and generation analysis provides useful diagnostic perspectives beyond downstream accuracy alone.
  3. The reported SLAKE and PathVQA improvements suggest potential task-level benefits. The paper also acknowledges the VQA-RAD open-ended regression, making the limitations of these gains explicit.
Critical weaknesses
  1. VFI operates on raw logits rather than probabilities. Logits (2, 0) and (1, -1) yield identical softmax distributions but a token VFI of 0.5 (Section 3.3.1). Its grounding interpretation therefore requires validation with a probability-based measure and a calibrated threshold.
  2. The manuscript describes EM, whereas the imported open-ended baseline reports recall; actual scoring and target-dataset training exposure need clarification. Although both systems reportedly use the same 60K instruction data, a matched baseline retaining the original LLaVA-Med architecture with LoRA is needed to separate architectural gains from adaptation effects. Original LLaVA-Med, Section 5.2 and Table 4(b).
  3. Centroid similarity and real/null separation do not establish clinical information loss or correct decoder use (Tables 1-2). Declining conditional influence may reflect information carried by earlier tokens. Controlled real-image substitutions are needed.
  4. Although Table 4 includes a Q-Former ablation, architecture and additional alignment training remain confounded. GPT-4 composite scores also do not isolate grounding; matched training controls, a clear judging protocol, and grounding-specific scores are needed.
Questions for authors
  1. Do the grounding and temporal-decay findings persist with a probability-based metric, shared prefixes, and a calibrated threshold?
  2. Can controlled substitutions of real medical images, together with separate grounding evaluation, demonstrate clinically appropriate answer changes driven by visual evidence?
Reviewer_LNx9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates linguistic bias in LLaVA-Med, showing that it can generate clinically plausible responses even without meaningful visual inputs. The authors propose three diagnostics: VFI, cosine similarity in representation space, and ERQ—to quantify visual grounding and identify information bottlenecks. They further introduce LLaVA-MF, incorporating BiomedCLIP, Q-Former, and LoRA adaptation. Experiments on medical VQA benchmarks report improved performance.

Critical strengths

The paper studies whether medical VLMs genuinely rely on visual evidence when generating clinical responses, which is an important question for the reliability of AI-assisted medical applications.

Critical weaknesses

I have several concerns about this paper:

W1. Limited scope of analysis.

LLaVA-Med was first published in 2023 as a pioneering work in medical VLMs. Over the past three years, many more advanced medical VLMs have been released, e.g., HuatuoGPT-Vision and Lingshu. Furthermore, recent general-purpose VLMs (e.g., Qwen3-VL) have also demonstrated strong visual reasoning capabilities in medical imaging. Therefore, analyzing LLaVA-Med is reasonable as a baseline, but focusing exclusively on this relatively early model limits the significance of a research paper submitted to a 2027 venue. The current results do not establish whether the findings on LLaVA-Med generalize to other medical VLMs.

W2. The proposed diagnostics do not convincingly measure visual grounding.

Definition 1 determines whether a medical VLM is visually grounded by comparing its output distributions given a real medical image and a null image. However, this only measures whether the model responds differently to different visual inputs. A difference in the model's output does not necessarily indicate whether its predictions are grounded in clinically meaningful image content or the correct image regions.

VFI measures relative differences in raw token logits, which are sensitive to arbitrary additive shifts and therefore do not reliably characterize changes in output distributions. Greater sensitivity to image perturbations does not necessarily indicate better visual grounding. Similarly, cosine similarity and ERQ primarily measure differences between real and null-image representations, rather than whether the models produce clinically meaningful visual representations. Consequently, these metrics do not sufficiently support the claimed identification and localization of linguistic bias.

W3. Limited motivation and novelty of the proposed method.

The paper introduces multiple architectural changes, including BiomedCLIP and Q-Former. However, BiomedCLIP has already been explored as a visual encoder in the original LLaVA-Med work. Furthermore, the motivation for using Q-Former as the projector is insufficient. Although LLaVA-MF reduces the cosine similarity between real and null-image representations compared to the original LLaVA-Med, the similarity remains relatively high after projection. As discussed in W2, the proposed diagnostics, including cosine similarity and ERQ, only indicate whether informative and non-informative images are encoded differently, but do not establish whether the representations produce clinically meaningful information.

The experiments also do not adequately establish whether the improvements can be attributed to BiomedCLIP and Q-Former. In Table 5, the substantial performance gains from LoRA fine-tuning suggest that LLM adaptation plays a major role in the final performance. Without further controlled experiments, it remains unclear how much of the improvement is attributable to the proposed architectural changes.

W4. Limited scale of the diagnostic evaluation.

The diagnostic set is relatively small, consisting of only 193 prompts associated with 50 unique images. Given that the proposed diagnostics are central to the paper's contributions, a more extensive evaluation across diverse medical images, modalities, and models is needed to establish the reliability and generalizability of the findings.

Questions for authors

Please see the weakness.

Reviewer_XjxkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper specifically focuses on the LLaVA-Med model, which leverages its language priors and lacks grounding in the input image. The authors propose a diagnostic framework to understand the underlying causes of this failure model, and then develop interventions to mitigate linguistic bias.

The new model LLaVA-MedFormer entails replacing the encoder, from CLIP to BiomedCLIP, and introduce a Q-former based alignment module.

Critical strengths
  • The diagnostic framework is original and comprehensive, since it introduces the VFI metric. It also investigates cosine similarity of representations and the effective representation quality.
  • They evaluate the model on three VQA datasets.
Critical weaknesses
  • The identified problem is already well known as supported by the related work section of the paper. The term “linguistic bias” describes an existing phenomenon, which is lack of visual grounding and reliance on language priors.
  • My main concern is that the scope of the paper is too narrow, whereby it focuses on a single model only. I believe that this is a bit limited for the ICLR venue. It’s more like an empirical study.
Questions for authors

Can you expand the scope of the analysis to include other VLMs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.