- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether LLaVA-Med genuinely relies on visual evidence, using VFI, representation statistics, and downstream evaluation. It proposes LLaVA-MF, combining BiomedCLIP, Q-Former, and LoRA. The question is important, but the current evidence does not sufficiently support the main claims about improved visual grounding and its underlying mechanisms.
Critical strengths
- The paper addresses an important reliability question: whether medically plausible answers are actually supported by the input image.
- Combining image perturbations, encoder and projector representations, and generation analysis provides useful diagnostic perspectives beyond downstream accuracy alone.
- The reported SLAKE and PathVQA improvements suggest potential task-level benefits. The paper also acknowledges the VQA-RAD open-ended regression, making the limitations of these gains explicit.
Critical weaknesses
- VFI operates on raw logits rather than probabilities. Logits (2, 0) and (1, -1) yield identical softmax distributions but a token VFI of 0.5 (Section 3.3.1). Its grounding interpretation therefore requires validation with a probability-based measure and a calibrated threshold.
- The manuscript describes EM, whereas the imported open-ended baseline reports recall; actual scoring and target-dataset training exposure need clarification. Although both systems reportedly use the same 60K instruction data, a matched baseline retaining the original LLaVA-Med architecture with LoRA is needed to separate architectural gains from adaptation effects. Original LLaVA-Med, Section 5.2 and Table 4(b).
- Centroid similarity and real/null separation do not establish clinical information loss or correct decoder use (Tables 1-2). Declining conditional influence may reflect information carried by earlier tokens. Controlled real-image substitutions are needed.
- Although Table 4 includes a Q-Former ablation, architecture and additional alignment training remain confounded. GPT-4 composite scores also do not isolate grounding; matched training controls, a clear judging protocol, and grounding-specific scores are needed.
Questions for authors
- Do the grounding and temporal-decay findings persist with a probability-based metric, shared prefixes, and a calibrated threshold?
- Can controlled substitutions of real medical images, together with separate grounding evaluation, demonstrate clinically appropriate answer changes driven by visual evidence?