- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AnatoRAG, an anatomy-guided retrieval-augmented framework for chest X-ray report generation, aiming to improve the clinical accuracy of generated reports. AnatoRAG employs an anatomy-guided retriever to select image-compatible evidence statements and an Anatomy Evidence Adapter to integrate the retrieved evidence into structured representations for report generation. Experiments on MIMIC-CXR and two external datasets demonstrate the effectiveness of the proposed framework in improving clinical evaluation metrics and cross-dataset generalization.
Critical strengths
The paper presents an interesting anatomy-guided retrieval-augmented framework that addresses the limitations of conventional report-level retrieval by selecting image-compatible evidence statements. The proposed Anatomy Evidence Adapter provides a structured way to integrate retrieved evidence into the generation process, potentially improving the clinical consistency of generated reports. Extensive experiments on MIMIC-CXR, along with evaluations on two external datasets, demonstrate promising improvements over existing methods across multiple clinical evaluation metrics.
Critical weaknesses
- The evaluation on CheXpert Plus includes only 62 samples, which is insufficient to convincingly demonstrate the model's generalization ability on a large-scale external dataset.
- The paper lacks a systematic analysis of clinical errors, such as hallucinated findings, missed abnormalities, and incorrect anatomical descriptions, making it difficult to assess the model's clinical reliability.
- The proposed framework integrates multiple models and processing stages, but the paper does not adequately analyze its computational overhead.
- The current ablation studies do not sufficiently isolate the contribution of anatomy-guided retrieval, leaving it unclear whether the improvements may be partly attributable to stronger disease classifiers and additional label supervision rather than anatomy guidance itself.
- Several baseline results are taken from previous publications rather than reproduced under a unified experimental setting, making direct performance comparisons less convincing.
Questions for authors
- CheXpert Plus is a large-scale dataset, yet only 62 samples were used for evaluation in this paper. The authors should clarify the data selection process and justify the limited sample size, as such a small evaluation set may not provide a reliable assessment of cross-dataset generalization.
- The authors should provide a detailed analysis of clinical error types and their frequencies, including hallucinated findings, missed abnormalities, and incorrect anatomical descriptions. In addition, the authors should clarify whether the model produces any potentially clinically consequential errors.
- The proposed method combines several components. The authors should report the computational costs of AnatoRAG, including inference latency, GPU memory usage, and the computational overhead introduced by its individual components.
- The authors should conduct additional controlled ablations, such as removing anatomical masks or replacing them with random masks, to better isolate the contribution of anatomy-guided retrieval.
- The authors should clarify whether the baseline methods were evaluated under identical settings and provide more controlled comparisons using consistent datasets, model inputs, and evaluation protocols.
4 comments
W4 / Q4: Isolating anatomical guidance with controlled mask ablations.
Correct anatomical alignment improves retrieval when the other settings are held fixed. Table S21 evaluates all 2,461 MIMIC-CXR test queries with model weights, admission, evidence bank, neighbour counts, and evidence-selection rules fixed:
| Retrieval condition | C14 micro | C14 macro | C5 micro | C5 macro |
|---|---|---|---|---|
| No explicit spatial guidance | 60.7 | 48.0 | 61.1 | 55.2 |
| Spatially misaligned masks | 59.2 | 46.2 | 59.6 | 54.1 |
| Correct spatial guidance | 63.0 | 50.3 | 64.9 | 58.7 |
The no-guidance condition removes spatial restrictions from query construction. Misaligned masks shift the permitted regions within the same image while preserving mask area. Correct guidance improves micro/macro F1 by 2.3/2.3 points over no guidance and 3.8/4.1 points over misaligned masks, demonstrating the benefit of correct spatial alignment under fixed retrieval settings.
W5 / Q5: Which comparisons use matched experimental conditions?
The comparisons used to attribute gains share the same BLIP-3 experimental setting. They use the same backbone initialization, training data, current/prior image inputs, decoding settings, and evaluation pipeline. Within each retrieval path, raw and adapted evidence share classifier guidance and the label/soft-target supervision. These comparisons test the evidence-adapter pathway with classifier guidance present in both branches. The main comparison tables also include published results; their provenance is identified in the captions. The label-only control removes that evidence pathway while retaining classifier-label guidance.
W5 / Q5: Dataset, input, and evaluation protocols.
The evidence bank contains training data only. Current/prior image associations are consistent across our branches: the prior is the most recent earlier examination of the same patient, excluding the current and future examinations. Appendix A and Table S1 document datasets and evaluation cohorts; Appendix C specifies the retrieval pipeline. LIBRA provides a second-backbone integration experiment. The captions distinguish our controlled experiments from published comparison results.
W3 / Q3: End-to-end inference latency relative to the backbone and RADAR.
AnatoRAG takes 23.27 s per case, compared with 15.05 s for the backbone and 40.34 s for RADAR, on the same NVIDIA RTX 4090 with beam width 5 (Table S24). RADAR's sketch, retrieval, and final-generation stages take 15.02, 2.77, and 22.55 s, respectively, summing to 40.34 s. The measured stage times and totals are:
| Method | Sketch generation | Retrieval | Final generation | Total |
|---|---|---|---|---|
| Backbone | — | — | 15.05 s | 15.05 s |
| RADAR | 15.02 s | 2.77 s | 22.55 s | 40.34 s |
| AnatoRAG | — | 4.04 s | 19.23 s | 23.27 s |
All runtime values in this table are our measurements.
W3 / Q3: Runtime breakdown and peak GPU memory.
For AnatoRAG, we measured mean online inference time over 100 samples on a single NVIDIA RTX 4090 after warm-up, excluding model and index initialization. Retrieval accounts for 4.04 s and covers input-image processing through final evidence selection; generation accounts for 19.23 s and includes the evidence adapter. Peak GPU memory is 20.04 GB. This is a retrieval/generation-stage breakdown; the adapter cost is included in generation. Offline evidence-bank construction is a one-time cost and is excluded from online inference.
Thanks to the authors for providing further clarification. They have addressed some of my concerns. I have decided to keep my rating.
We thank the reviewer for the follow-up. To avoid guessing, here is where each of the five points in your review stands after the discussion:
| Point from your review | Evidence added |
|---|---|
| 62-case CheXpert Plus | All 62 cases with Findings references in the public validation subset, none selected, same cohort used by RADAR and LIBRA (App. A) |
| Missing clinical error analysis | Expert audit of 100 generated reports with per-category frequencies and consequential examples (Table S23, Ethics Statement) |
| Computational overhead | 23.27 s per case vs 15.05 s backbone and 40.34 s RADAR, 20.04 GB peak, with retrieval/generation stage split (Table S24) |
| Anatomy ablation not isolated | Correct masks 63.0/50.3, no explicit spatial guidance 60.7/48.0, area-matched misaligned masks 59.2/46.2, with model weights, admission and evidence bank fixed (Table S21) |
| Baselines from prior papers | Controlled rows share backbone initialization, data, current/prior inputs, decoding and evaluation protocol; published numbers are marked as such in the captions |
Could you indicate which of the five is still open, or name the concern if it is not among them? A row number is enough. We will respond to that item alone and leave the rest of your time free.
We thank the reviewer for recognizing the motivation and clinical improvements of AnatoRAG. Your suggestions helped us strengthen the spatial controls, measure inference costs, and clarify the evaluation protocol.
W1 / Q1: Why were 62 cases evaluated, and what supports external generalization?
We did not select or filter the cases. Only 62 cases in the public CheXpert Plus validation subset have Findings references, and we evaluated all 62. The sample count follows the availability of Findings references in this subset. RADAR and LIBRA also use this Findings subset for external evaluation. Together with IU X-Ray, this complete Findings cohort provides the external evaluation.
W1 / Q1: Repeated-run evidence on the external cohorts.
For each dataset, we performed three independent generation runs with different random seeds using the same fixed model checkpoint, decoding settings, and evaluation cohort. Each cell reports the mean ± standard deviation across these generation runs (%):
We have just included the three-seed means and standard deviations in Table S25; Appendix A documents the evaluation cohorts.
W2 / Q2: Clinical error types and their frequencies.
The generated reports can contain clinically relevant errors; omissions or incorrect denials were the most frequent primary category in our audit. Human experts reviewed 100 randomly sampled MIMIC-CXR generated reports against reference Findings. We assess explicitly stated abnormalities and their attributes and allow equivalent wording. An omitted finding is counted when the reference clearly states its presence but the generated report fails to report it. Reference silence or an uncertain diagnosis alone does not establish a contradiction. Device assessment covers presence/absence. Each report with an identified error receives one primary error category:
This report-level analysis is separate from the expert audit of 562 retrieved statements, which found severity discrepancies in 11/562 (2.0%), laterality discrepancies in 4/562 (0.7%), and temporal-change discrepancies in 25/562 (4.4%), totaling 40/562 (7.1%). The two analyses examine errors before and after generation, respectively.
W2 / Q2: Potentially consequential clinical errors.
Some observed errors could affect clinical interpretation. For example, one report states a small right pleural effusion when its reference explicitly states no pleural effusion; another omits newly developed free intra-abdominal air explicitly documented in the reference. Such errors could respectively introduce an unsupported abnormality or conceal a finding relevant to further clinical assessment.
We have just expanded the Ethics Statement to explain the practical implications:
AnatoRAG is a research system for drafting chest X-ray Findings. Generated reports may contain unsupported findings, omit abnormalities, or misdescribe their location, severity, or temporal change. Label-level compatibility filtering does not establish the correctness of every clinical attribute. These errors could mislead clinical interpretation. Any clinical use would require qualified radiologists to review and correct generated reports against the source images and relevant clinical context, together with further validation in the intended clinical setting.