acceptodds
Withdrawn from ICLR 2027

The Retrieval Paradox: Why RAG Fails Medical Vision-Language Models Across Architectures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a widely adopted approach for adapting general-purpose vision-language models (VLMs) to specialized medical domains at inference time. However, its effectiveness and safety for medical visual question answering (VQA) remain critically understudied. We present a comprehensive empirical evaluation of five RAG configurations across three VLM architectures and three medical VQA benchmarks (VQA-RAD, SLAKE, PathVQA), using a retrieval bank of 50,000 medical image-QA pairs. We uncover the Retrieval Paradox: RAG systematically degrades performance across all tested architectures and benchmarks without exception, dropping LLaVA-1.5-7B from 45.0% to 35.7% on VQA-RAD and LLaVA-1.6 from 45.68% to 36.81%. Critically, BLIP-2, which achieves 96.0% at , suffers the largest degradation: from 96.0% to 55.65% on VQA-RAD (). Rather than preventing failure, Q-Former bottleneck alignment creates structural inflexibility that amplifies context collapse when retrieved text is injected at the decoder. A controlled Oracle experiment recovers accuracy from 24.0% to 96.0% with perfect context, and a random text ablation ( vs. for medical RAG) isolates semantic content as the driver of failure rather than token length. Population-level attention analysis across the full test set (, paired -test ) confirms that retrieved medical text displaces visual attention by 28.0% relative. A modern BGE cross-encoder reranker provides no meaningful improvement ( vs. flat FAISS), establishing the failure as architectural rather than retrieval-quality-dependent. None of the tested decoder-based VLMs under standard prompt-prefixing is RAG-immune, with direct implications for the safe deployment of multimodal AI in clinical settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, medical visual question answering, vision-language models, cross-modal attention, multimodal RAG, semantic overpowering, catastrophic context collapse, medical AI safety, VQA benchmarks, attention analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_iSB7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the effectiveness of RAG for medical VQA. The authors evaluate four vision-language models across three medical VQA benchmarks using five RAG configurations. The results show that RAG consistently degrades performance under the evaluated settings. The authors attribute this phenomenon, termed the "Retrieval Paradox," to "Semantic Overpowering," where retrieved text distracts models from visual information.

Critical strengths
  1. The paper studies a practically important problem concerning the reliability of RAG in medical VQA.

  2. The authors conduct several ablation experiments, including oracle retrieval, random-text controls, reranking, and attention analysis, to investigate the underlying failure mechanisms.

Critical weaknesses

W1. Poor presentation and insufficient methodological clarity.

The paper is difficult to follow, as many experimental settings are introduced without sufficient explanation or motivation. For example, the five RAG configurations and the reranker are presented without adequately explaining their design choices or why they are considered. Furthermore, almost ALL mathematical notations are introduced without proper definitions, and the results in Sec. 4.2 are poorly organized. Overall, the paper reads more like an experimental report than a systematic research study.

W2. Limitations of the experimental setting.

Although the paper investigates RAG for medical VQA, none of the evaluated models is a medical-domain VLM. This substantially limits the generalizability of the findings to medical VLMs.

**W3. The RAG pipeline is insufficiently justified. **

In particular, the retrieval corpus consists of medical image-QA pairs, but the retrieved snippets are injected only as plain-text prefixes ("Similar case: Q: [question] A: [answer]"), without including the corresponding retrieved images. The paper does not explain this design choice or investigate how excluding the retrieved visual information affects the results. Consequently, it is unclear whether the findings reflect general limitations of multimodal RAG or simply the particular retrieval strategy adopted.

W4. Concerns about the reliability of the experimental results.

The reported 96.0% zero-shot accuracy of BLIP-2 on VQA-RAD is unexpectedly high. For reference, the SoTA medical VLM Lingshu-7B reports 67.9% on VQA-RAD, although the evaluation protocols may differ. Such a substantial discrepancy requires further explanation.

Additionally, there are numerical inconsistencies in the reported results. For example, Table 7 reports 45.0% baseline accuracy and 40.0% random-text accuracy but assigns a 0.0 percentage-point difference. These issues raise concerns about the reliability of the evaluation and the conclusions drawn from the experimental results.

W5. Limited novelty relative to existing research.

The paper does not adequately distinguish its proposed Semantic Overpowering phenomenon from closely related prior research. In particular, Zhao et al., When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs (ICML 2026), already investigate how retrieved text suppresses visual attention and distracts models from question-relevant image regions. The authors should clarify what new insights or findings their work establishes beyond this existing study.

Questions for authors

Please see the weakness.

Reviewer_PLaYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether naive RAG helps medical VQA. They evaluated 4 VLMs (LLaVA-1.5, LLaVA-1.6, BLIP-2, InstructBLIP). The main finding is that RAG consistently hurts accuracy. The authors attribute this to retrieved text displacing visual attention ("Semantic Overpowering") and claim the failure is architectural rather than a retrieval quality issue.

Critical strengths
  1. The research question is interesting. Naive RAG is assumed to help, and a well-executed negative result for medical VLMs is valuable for the community.
  2. The study includes several controls (random text, reranker, image-priority prompt, oracle context) and runs significance tests on the main comparison.
Critical weaknesses
  1. The paper is poorly written. It seems like this is a complete AI generated submission.
  2. The experiment results are odd. For example, BLIP-2 achieves 96.0% zero-shot on VQA-RAD but only 35.06% on SLAKE and 30.41% on PathVQA. This gap strongly suggests there might exist some coding errors, could be substring matching. Overall, I feel the results are not reliable.
  3. This paper contains several inconsistent claims. For example, the oracle experiment shows accuracy jumping from 24% to 96% when ground-truth Q/A pairs are injected. This is strong evidence that retrieval quality is what matters, not architecture.
Questions for authors

Can the authors verify the BLIP-2 96.0% result?

Reviewer_Ve5fOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the role of Retrieval-Augmented Generation (RAG) in medical visual question answering, focusing on whether adding retrieved medical question-and-answer fragments as text prefixes to a vision-language model improves the model's responses. The authors constructed a retrieval database containing 50,000 medical image-question-and-answer fragments using SLAKE training data and PMC-VQA, and reported partial model-benchmark combinations of LLaVA-1.5, LLaVA-1.6, BLIP-2, and InstructBLIP on VQA-RAD, SLAKE, and PathVQA. The retrieved fragments were ultimately input into the model as question-and-answer texts from other cases.

Critical strengths
  1. The research question has practical value, and the reported negative results deserve careful verification. The authors examined an easily overlooked issue: question-and-answer retrieved from other medical case studies may not be suitable for the current image; directly adding this content to prompts may mislead the model. Table 1 reports performance degradation on multiple models and medical visual question answering benchmarks, indicating that this phenomenon deserves systematic investigation, rather than assuming that retrieval augmentation necessarily brings benefits.

  2. The authors arranged controls around several possible factors and attempted to provide statistical and internal model evidence. The paper examined the number of retrieved fragments, random non-medical text, image-priority instructions, reranking, and oracle context, and reported paired bootstrap confidence intervals and McNemar tests for the main accuracy difference in LLaVA-1.5. The authors also extended attention analysis to the test set level to examine whether the attentional share of visual tokens changed systematically after the addition of retrieval text.

Critical weaknesses
  1. The control experiments and charts supporting the core explanation lack consistent and verifiable evidence. The baseline for comparison in the random text experiment is unclear. Section 4.2 of the main text reports a random-text accuracy of 40.0%, which corresponds to a baseline of 40.0% without retrieval, thus showing no decrease. However, Table 7 uniformly lists the baseline without retrieval as 45.0%, while the random-text accuracy is still listed as 40.0%, with the difference labeled as 0.0 percentage points. If these results use the same set of questions, the numbers do not match; if they come from different subsets or settings, the corresponding results for no retrieval, random text, and medical retrieval should be reported separately.

The image-priority instructions experiment also has conflicting sample size descriptions: the main text explicitly states N=100, while the title of Table 7 uses N=451 to cover all configurations and ablation. Furthermore, the image attention annotation in Figure 7 shows 26.1%→23.6%, but the caption for the same figure states 96.0%→81.0%. If these figures use different statistical definitions, the authors need to clarify this; currently, readers cannot determine which set of values ​​supports the paper's interpretation.

Furthermore, the paper uses substring matching scores from normalized outputs. This rule needs to explain how negation, multiple candidate answers, and whether input prompts are excluded from the scoring text are handled. In particular, the 96.0% baseline of BLIP-2 on VQA-RAD should be further verified through original answers, rigorous answer matching, and manual spot-checking. High scores are not inherently erroneous evidence, but such crucial results require more transparent scoring criteria.

  1. Existing experiments do not adequately distinguish between retrieval quality, specific injection methods, and the architecture itself, thus failing to support strong causal conclusions at the architecture level. The BGE experiment only reranks the top 20 candidates from the same FAISS retrieval database and then selects the top three inputs to the model. If the candidates inherently lack evidence supporting the current image answer, reranking cannot solve this problem. Therefore, the lack of improvement in final accuracy only indicates that this reranking did not improve the current system; without measuring the relevance of evidence, the degree of support for answers, or the rate of contradiction, this cannot be used to rule out explanations related to retrieval quality.

The Oracle experiment also requires careful interpretation. The authors reported accuracies of 38%, 24%, and 96% for no context, FAISS retrieval, and Oracle on 50 questions, respectively. However, Oracle used one ideal fragment, while FAISS used five, with no matching between quantity and length. The manuscript also fails to adequately demonstrate whether the Oracle fragment directly contained the target answer. If the answer is already in the text, high scores may come from reading the text, rather than proving that the model has restored the use of images.

Attention results can provide clues, but adding text changes the number of tokens involved in attention normalization. A decrease in the proportion of visual attention does not automatically mean that visual information is ignored, nor can it alone prove that this is the cause of incorrect answers. The paper only measures attention at the last layer and the first generation position, and still needs to be analyzed in conjunction with text conditions matching the length, different answer results, and image dependencies.

Finally, the manuscript's explanation of the Q-Former model has conceptual issues. BLIP-2 inherently projects the visual output of the Q-Former as a visual prefix into the language model; therefore, the "pre-encoding of visual features" cannot be used to deduce that "the decoder cannot utilize visual representations."

The Q-Former of InstructBLIP can also receive instruction text. If this paper only injects the retrieved content into the decoder, this is an implementation choice that needs to be explicitly disclosed and cannot be directly generalized as an inherent limitation of the entire architecture. Different models also exhibit differences in language backbone, parameter size, and training methods; the current comparison does not separate these factors from the connector architecture.

Questions for authors
  1. Regarding the control experiment's sample and baseline: What questions, models, and prompts were used in the random text experiments? Why is the baseline for no retrieval in the main text 40.0% and Table 7 45.0%? How does N=100 in the image-priority experiment correspond to N=451 in the table? Are the in-figure numbers and captions in Figure 7 defined differently, and what are the correct results for each?

  2. Regarding scoring and statistical verification: Can you provide the original model's answer, the implementation of scoring only newly generated content, and the results of manual spot-checking? Does the 96.0% accuracy of BLIP-2 still hold after strict answer matching, negation, and multiple-answer processing?

  3. Regarding retrieval quality: Does BGE re-ranking actually improve the relevance of candidate evidence or increase support for the current image answer? If the retrieval quality has not been verified to have improved, why can the explanation of insufficient retrieval quality be ruled out based on the lack of improvement in final accuracy?

  4. Regarding text injection path and mechanism verification: Does the retrieved text enter the decoder, Q-Former, or both in InstructBLIP? Does the Oracle fragment directly contain the target answer? Under the same set of questions and similar token budgets, comparing random text, evidence supporting the answer, and conditions for removing or replacing images, do the changes in accuracy and visual attention still support the current mechanism explanation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.