- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper observes that existing CXR RAG systems feed entire retrieved radiology reports directly into the model, even though these reports contain a variety of information that is irrelevant to the question or uncertain. To address this, the authors propose REO-RAG, which decomposes retrieved reports after retrieval into structured evidence units (finding, anatomy, assertion, polarity). For each unit, a question-conditioned risk is measured, and based on this, only the information necessary for answering is selected, reorganized, and provided to the Med-LVLM. Although REO-RAG keeps the retriever of existing RAG systems unchanged and modifies only how the retrieved information is utilized, it achieves the highest F1 score among the compared methods on the VQA tasks of IU-Xray, MIMIC-CXR, and PadChest-GR.
Critical strengths
Strength 1: Effective Information Extraction
Instead of inputting the entire retrieved reports, the approach of selecting and structuring only the key information essential for solving the VQA problem is a reasonable direction for improving the information-processing efficiency of the LLM.
Strength 2: Carefully designed risk-based filtering
The design of various textual risk factors for filtering and reorganizing evidence is particularly notable. Not only the overall structure of the proposed pipeline but also the individual risk factors (target relevance, uncertainty, polarity disagreement, etc.) give the impression of having been meticulously designed with deep consideration of the clinical context.
Critical weaknesses
Weakness 1: Insufficient evidence for the practical effectiveness of the evidence organization module
In the evaluation of this paper, the performance improvement is mainly observed from target relevance. The authors also acknowledge this, and in lines 430–431 they claim the contribution of the remaining factors from the perspective of structuring rather than substantial performance improvement on a specific task. There may be differing views on whether each factor must be practically meaningful, but in my view, if the authors claim the entire evidence organization module as a contribution of this paper, they should demonstrate within the paper, through other evaluation methods, that the other risk factors (uncertainty, polarity disagreement, etc.) also have an effect. In addition, I observed in Table 2 that a substantial portion of the performance improvement comes from AC-DPO rather than from evidence organization. I believe the fact that report-level RAG with AC-DPO already outperforms the existing models weakens the contribution of the proposed evidence organization. Furthermore, the authors claim that performance on PadChest-GR is degraded due to sparse target coverage. A lower overall performance is understandable, but I find it difficult to understand why applying REO-RAG leads to a performance drop. What is the performance of Report-level RAG and REO-RAG when restricted to cases where at least one piece of evidence related to the target finding exists?
Weakness 2: Concerns about part of evidence organization (conflict)
Assuming that the original radiologist reports are generally accurate, if the polarity of a specific finding conflicts among the retrieved reports, this suggests a high likelihood that some cases clinically unrelated to the current query image were incorrectly retrieved. A successful RAG system should retrieve images and reports with consistent findings similar to the query image in the first place, thereby minimizing such polarity conflicts. Of course, since REO-RAG keeps the retriever unchanged, it cannot address this issue. However, I think there is a problem with the approach in the paper of assigning a [Conflict] tag and retaining both sides when polarities conflict. I believe the authors should extract the cases where conflicts actually occurred and directly examine and report whether the cause is (1) the query image itself being clinically ambiguous so that divergent findings can reasonably arise, (2) reading/recording errors in the original data, or (3) the retriever misidentifying visual features and retrieving completely unrelated cases. If the overwhelming majority of conflict cases arise simply from retriever errors (irrelevant retrieval), it is logically difficult to understand how a situation in which such incorrect information (noisy evidence) is included and causes conflicts provides good evidence to the model. The authors should examine the ground-truth distribution of the actual images in the conflict cases and analyze how the conflict cases actually benefit the model.
Questions for authors
Please see the Weaknesses section.
Thanks for the additional experiments. The Target-only results and the controlled Top-5 comparison help address my earlier concerns.
The gains over Target-only are meaningful. However, I'm still unsure how much the additional components of REO contribute, as several risk factors have little effect and the results on PadChest are nearly identical under matched settings.