acceptodds
Withdrawn from ICLR 2027

A Reliability-Aware Evidence Organization Framework for Retrieval-Augmented Medical Visual Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical large vision-language models (Med-LVLMs) have advanced medical visual question answering (VQA), but they can still produce answers unsupported by the image or clinical evidence. Retrieval-augmented generation (RAG) improves factual grounding by introducing radiology reports, yet visually similar examinations do not necessarily yield question-relevant or clinically consistent evidence. Report-level retrieval therefore suffers from two fundamental mismatches: retrieved reports may fail to cover the queried finding, and the retrieved content may contain irrelevant, uncertain, or conflicting statements. We propose \method, a novel factual grounding framework for medical VQA, which converts retrieved reports into question-specific clinical evidence. Without altering the image retriever, \method decomposes reports into structured evidence units encoding clinical findings, anatomical locations, and assertion status. It then uses a question-conditioned reliability estimator to jointly assess target relevance, uncertainty, and cross-evidence inconsistency. A risk-guided organizer subsequently selects and composes lower-risk, question-aligned evidence into a compact, clinically coherent context for the Med-LVLM. By separating evidence acquisition from evidence utilization, \method enables the Med-LVLM to reason over organized question-specific evidence rather than unfiltered reports. Its key contribution is not stronger retrieval, but a representation-level bridge that transforms retrieved reports into more reliable evidence for clinical reasoning. Under a unified evaluation protocol, \method reaches F1 scores of 88.60%, 88.89%, and 56.05% on IU-Xray, MIMIC-CXR, and PadChest-GR, with larger gains when retrieved reports already contain mixed but question-relevant evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Question Answering, Retrieval-augmented generation, Reliability-Aware, Evidence Organization, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PMCHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper observes that existing CXR RAG systems feed entire retrieved radiology reports directly into the model, even though these reports contain a variety of information that is irrelevant to the question or uncertain. To address this, the authors propose REO-RAG, which decomposes retrieved reports after retrieval into structured evidence units (finding, anatomy, assertion, polarity). For each unit, a question-conditioned risk is measured, and based on this, only the information necessary for answering is selected, reorganized, and provided to the Med-LVLM. Although REO-RAG keeps the retriever of existing RAG systems unchanged and modifies only how the retrieved information is utilized, it achieves the highest F1 score among the compared methods on the VQA tasks of IU-Xray, MIMIC-CXR, and PadChest-GR.

Critical strengths

Strength 1: Effective Information Extraction

Instead of inputting the entire retrieved reports, the approach of selecting and structuring only the key information essential for solving the VQA problem is a reasonable direction for improving the information-processing efficiency of the LLM.

Strength 2: Carefully designed risk-based filtering

The design of various textual risk factors for filtering and reorganizing evidence is particularly notable. Not only the overall structure of the proposed pipeline but also the individual risk factors (target relevance, uncertainty, polarity disagreement, etc.) give the impression of having been meticulously designed with deep consideration of the clinical context.

Critical weaknesses

Weakness 1: Insufficient evidence for the practical effectiveness of the evidence organization module

In the evaluation of this paper, the performance improvement is mainly observed from target relevance. The authors also acknowledge this, and in lines 430–431 they claim the contribution of the remaining factors from the perspective of structuring rather than substantial performance improvement on a specific task. There may be differing views on whether each factor must be practically meaningful, but in my view, if the authors claim the entire evidence organization module as a contribution of this paper, they should demonstrate within the paper, through other evaluation methods, that the other risk factors (uncertainty, polarity disagreement, etc.) also have an effect. In addition, I observed in Table 2 that a substantial portion of the performance improvement comes from AC-DPO rather than from evidence organization. I believe the fact that report-level RAG with AC-DPO already outperforms the existing models weakens the contribution of the proposed evidence organization. Furthermore, the authors claim that performance on PadChest-GR is degraded due to sparse target coverage. A lower overall performance is understandable, but I find it difficult to understand why applying REO-RAG leads to a performance drop. What is the performance of Report-level RAG and REO-RAG when restricted to cases where at least one piece of evidence related to the target finding exists?

Weakness 2: Concerns about part of evidence organization (conflict)

Assuming that the original radiologist reports are generally accurate, if the polarity of a specific finding conflicts among the retrieved reports, this suggests a high likelihood that some cases clinically unrelated to the current query image were incorrectly retrieved. A successful RAG system should retrieve images and reports with consistent findings similar to the query image in the first place, thereby minimizing such polarity conflicts. Of course, since REO-RAG keeps the retriever unchanged, it cannot address this issue. However, I think there is a problem with the approach in the paper of assigning a [Conflict] tag and retaining both sides when polarities conflict. I believe the authors should extract the cases where conflicts actually occurred and directly examine and report whether the cause is (1) the query image itself being clinically ambiguous so that divergent findings can reasonably arise, (2) reading/recording errors in the original data, or (3) the retriever misidentifying visual features and retrieving completely unrelated cases. If the overwhelming majority of conflict cases arise simply from retriever errors (irrelevant retrieval), it is logically difficult to understand how a situation in which such incorrect information (noisy evidence) is included and causes conflicts provides good evidence to the model. The authors should examine the ground-truth distribution of the actual images in the conflict cases and analyze how the conflict cases actually benefit the model.

Questions for authors

Please see the Weaknesses section.

Reviewer_vezYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REO-RAG for chest X-ray questions with Yes/No answers. It extracts structured findings from retrieved reports, scores their relevance and textual risk, and organizes supporting and conflicting statements for a medical VLM. The generator also uses preference training from IU-Xray. Reported F1 scores are 88.60%, 88.89%, and 56.05% on IU-Xray, MIMIC-CXR, and a constructed PadChest-GR subset.

Critical strengths
  • The distinction between missing target evidence and poor use of retrieved evidence is clear and useful. The method is interpretable and leaves retrieval unchanged.

  • Tables 2 and 4 provide useful matched generator, and representation. They help separate preference-training gains from organization gains.The paper explicitly acknowledges the substantial AC-DPO contribution and weaker PadChest transfer.

Critical weaknesses
  • The risk score does not check whether a statement is true for the current image. Retrieved reports describe other patients, while the choice between positive and negative evidence mainly uses word overlap. Keeping only the stronger side can repeat a bias in the retrieved cases. The paper correctly calls this textual utility, but tests against the current image are needed to support the reliability claim.

  • The full organizer adds limited benefit over simple settings. With the same trained generator, MIMIC F1 is 87.97% for images alone, 88.25% for the top five evidence units, and 88.89% for REO-RAG. On PadChest, the top five reach 59.44%, above REO-RAG’s 56.05%. A simple target-keyword filter is still needed to show why the complete organizer is necessary.

Questions for authors
  • How accurate are the extracted fields, and how often does selected evidence match the current image?

  • Answer formatting may explain part of the gain. The scorer rejects answers without a Yes/No opening; REO’s prompt explicitly requires it, and preference training uses short Yes/No templates. The report-level prompt is less strict. Please use identical format instructions and report formatting failures.

Reviewer_FRgnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REO-RAG, a framework for improving evidence utilization in retrieval-augmented medical visual question answering. Instead of modifying the retriever, the method converts retrieved radiology reports into structured evidence units containing clinical findings, anatomical locations, assertion status, and polarity. It then uses question-conditioned risk scores to select and organize relevant evidence before passing it to a medical vision-language model. The authors evaluate the method on three chest X-ray yes/no VQA datasets: IU-Xray, MIMIC-CXR, and PadChest-GR, reporting F1 scores of 88.60%, 88.89%, and 56.05%, respectively. They also provide controlled comparisons and ablation studies to examine the contributions of evidence organization and preference tuning, along with analyses of retrieval coverage and failure cases.

Critical strengths

1.The paper addresses a practical problem in medical RAG.The authors make a useful distinction between retrieving relevant reports and effectively using the information within them. Organizing evidence at the finding level, rather than directly feeding entire reports to the model, is a reasonable approach. The method is also interpretable and does not require modifying the retriever. 2.The experimental analysis is relatively thorough.The paper includes comparisons on three datasets, along with several ablation studies and retrieval diagnostics. I particularly appreciate the controlled experiments in Table 2, which help separate the effect of evidence organization from that of preference tuning. The authors also discuss cases where their method does not perform well, making the analysis more convincing.

Critical weaknesses

1.The novelty of the proposed evidence organization mechanism is somewhat limited. The framework mainly relies on rule-based extraction and manually designed risk factors. While the overall pipeline is reasonable, it is not clear how much the additional reliability factors contribute beyond simple question-relevant evidence selection. In Table 5, removing several risk factors has little effect on performance, with target relevance appearing to be the most important component. This makes it difficult to judge whether the complexity of the full framework is justified. 2.The effectiveness of the method is not consistent across datasets. Although REO-RAG performs well on IU-Xray and MIMIC-CXR, the results on PadChest-GR are less convincing. In particular, the simpler Top-5 evidence selection in Appendix Table 15 achieves a higher F1 score than the full method (59.44% vs. 56.05%). Moreover, the proposed reliability score is based on textual properties rather than direct image-evidence consistency. These issues raise concerns about how robustly the method can identify useful evidence across different settings.

Questions for authors

1.The ablation results suggest that target relevance is the main contributor to the performance gains. How does the full REO-RAG framework compare with a simpler baseline that only selects question-relevant evidence, under the same retrieval pool, generator, and evidence budget? 2.The reliability score is mainly based on textual features, without directly checking whether the retrieved evidence agrees with the query image. How do the authors assess whether the selected evidence is actually reliable for the current case, rather than just relevant to the question? 3.In Appendix Table 15, Top-5 evidence selection outperforms REO-RAG on PadChest-GR. What explains this result? Does it suggest that the additional risk filtering or polarity organization may sometimes remove useful evidence or introduce noise? 4.Section C.2.1 appears to be empty. Please check and complete this subsection.

Comments on the paper

1 comment

Thanks for the additional experiments. The Target-only results and the controlled Top-5 comparison help address my earlier concerns.

The gains over Target-only are meaningful. However, I'm still unsure how much the additional components of REO contribute, as several risk factors have little effect and the results on PadChest are nearly identical under matched settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.