acceptodds
Under review as a conference paper at ICLR 2027

AnatoRAG: Anatomy-Guided Retrieval for Chest X-ray Report Generation

Abstract

Given radiographic images, chest X-ray report generation aims to produce clinically accurate textual findings. A common recipe is retrieval-augmented generation (RAG), which takes prior reports as reference. However, the reports in such a corpus come from other patients, so their findings are plausible yet unverified for the target image, and report-level retrieval cannot separate the compatible findings from the rest. Yet retrieving at statement level does not resolve this: the retrieved text remains unverified against the target image and largely redundant. We introduce AnatoRAG, which treats retrieved evidence as a hypothesis to be verified and compressed rather than text to be copied. First, AnatoRetriever uses coarse anatomical masks to guide statement retrieval, and a finding classifier discards statements that contradict the target image. Second, the Anatomy Evidence Adapter condenses the surviving statements and image-label predictions into compact anatomical representations, which a multimodal LLM fuses with visual features to generate the report. Extensive experiments on MIMIC-CXR show that AnatoRAG achieves 65.1 CheXbert-14 micro-F1, surpassing RADAR by 2.4 points. Trained only on MIMIC-CXR, AnatoRAG also transfers to the unseen CheXpert Plus and IU X-Ray datasets, yielding 60.4 and 54.6 CheXbert-14 micro-F1, respectively. The improvement is concentrated on findings whose evidence must be recomposed across anatomical regions, indicating that verification and compression, rather than retrieval alone, drive the gain.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
radiology report generation, retrieval-augmented generation, anatomy-guided retrieval, evidence compression, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hFF4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AnatoRAG, an anatomy-guided retrieval-augmented framework for chest X-ray report generation, aiming to improve the clinical accuracy of generated reports. AnatoRAG employs an anatomy-guided retriever to select image-compatible evidence statements and an Anatomy Evidence Adapter to integrate the retrieved evidence into structured representations for report generation. Experiments on MIMIC-CXR and two external datasets demonstrate the effectiveness of the proposed framework in improving clinical evaluation metrics and cross-dataset generalization.

Critical strengths

The paper presents an interesting anatomy-guided retrieval-augmented framework that addresses the limitations of conventional report-level retrieval by selecting image-compatible evidence statements. The proposed Anatomy Evidence Adapter provides a structured way to integrate retrieved evidence into the generation process, potentially improving the clinical consistency of generated reports. Extensive experiments on MIMIC-CXR, along with evaluations on two external datasets, demonstrate promising improvements over existing methods across multiple clinical evaluation metrics.

Critical weaknesses
  1. The evaluation on CheXpert Plus includes only 62 samples, which is insufficient to convincingly demonstrate the model's generalization ability on a large-scale external dataset.
  2. The paper lacks a systematic analysis of clinical errors, such as hallucinated findings, missed abnormalities, and incorrect anatomical descriptions, making it difficult to assess the model's clinical reliability.
  3. The proposed framework integrates multiple models and processing stages, but the paper does not adequately analyze its computational overhead.
  4. The current ablation studies do not sufficiently isolate the contribution of anatomy-guided retrieval, leaving it unclear whether the improvements may be partly attributable to stronger disease classifiers and additional label supervision rather than anatomy guidance itself.
  5. Several baseline results are taken from previous publications rather than reproduced under a unified experimental setting, making direct performance comparisons less convincing.
Questions for authors
  1. CheXpert Plus is a large-scale dataset, yet only 62 samples were used for evaluation in this paper. The authors should clarify the data selection process and justify the limited sample size, as such a small evaluation set may not provide a reliable assessment of cross-dataset generalization.
  2. The authors should provide a detailed analysis of clinical error types and their frequencies, including hallucinated findings, missed abnormalities, and incorrect anatomical descriptions. In addition, the authors should clarify whether the model produces any potentially clinically consequential errors.
  3. The proposed method combines several components. The authors should report the computational costs of AnatoRAG, including inference latency, GPU memory usage, and the computational overhead introduced by its individual components.
  4. The authors should conduct additional controlled ablations, such as removing anatomical masks or replacing them with random masks, to better isolate the contribution of anatomy-guided retrieval.
  5. The authors should clarify whether the baseline methods were evaluated under identical settings and provide more controlled comparisons using consistent datasets, model inputs, and evaluation protocols.
4 comments

We thank the reviewer for recognizing the motivation and clinical improvements of AnatoRAG. Your suggestions helped us strengthen the spatial controls, measure inference costs, and clarify the evaluation protocol.

W1 / Q1: Why were 62 cases evaluated, and what supports external generalization?

We did not select or filter the cases. Only 62 cases in the public CheXpert Plus validation subset have Findings references, and we evaluated all 62. The sample count follows the availability of Findings references in this subset. RADAR and LIBRA also use this Findings subset for external evaluation. Together with IU X-Ray, this complete Findings cohort provides the external evaluation.

W1 / Q1: Repeated-run evidence on the external cohorts.

For each dataset, we performed three independent generation runs with different random seeds using the same fixed model checkpoint, decoding settings, and evaluation cohort. Each cell reports the mean ± standard deviation across these generation runs (%):

DatasetC14 microC14 macroC5 microC5 macro
MIMIC-CXR65.16 ± 0.3750.94 ± 0.2867.47 ± 0.3359.83 ± 0.22
IU X-Ray57.45 ± 1.4233.53 ± 1.0145.70 ± 1.1333.52 ± 0.90
CheXpert Plus60.38 ± 1.7136.53 ± 0.5958.75 ± 0.7848.12 ± 0.80

We have just included the three-seed means and standard deviations in Table S25; Appendix A documents the evaluation cohorts.

W2 / Q2: Clinical error types and their frequencies.

The generated reports can contain clinically relevant errors; omissions or incorrect denials were the most frequent primary category in our audit. Human experts reviewed 100 randomly sampled MIMIC-CXR generated reports against reference Findings. We assess explicitly stated abnormalities and their attributes and allow equivalent wording. An omitted finding is counted when the reference clearly states its presence but the generated report fails to report it. Reference silence or an uncertain diagnosis alone does not establish a contradiction. Device assessment covers presence/absence. Each report with an identified error receives one primary error category:

Human expert review of generated reportsReports (%)
Hallucinated / unsupported findings6/100 (6%)
Omitted or incorrectly denied findings18/100 (18%)
Incorrect anatomical descriptions5/100 (5%)
Other clinical-description discrepancies: severity, change, diagnosis, or device presence11/100 (11%)
Total reports with identified errors40/100 (40%)

This report-level analysis is separate from the expert audit of 562 retrieved statements, which found severity discrepancies in 11/562 (2.0%), laterality discrepancies in 4/562 (0.7%), and temporal-change discrepancies in 25/562 (4.4%), totaling 40/562 (7.1%). The two analyses examine errors before and after generation, respectively.

W2 / Q2: Potentially consequential clinical errors.

Some observed errors could affect clinical interpretation. For example, one report states a small right pleural effusion when its reference explicitly states no pleural effusion; another omits newly developed free intra-abdominal air explicitly documented in the reference. Such errors could respectively introduce an unsupported abnormality or conceal a finding relevant to further clinical assessment.

We have just expanded the Ethics Statement to explain the practical implications:

AnatoRAG is a research system for drafting chest X-ray Findings. Generated reports may contain unsupported findings, omit abnormalities, or misdescribe their location, severity, or temporal change. Label-level compatibility filtering does not establish the correctness of every clinical attribute. These errors could mislead clinical interpretation. Any clinical use would require qualified radiologists to review and correct generated reports against the source images and relevant clinical context, together with further validation in the intended clinical setting.

W4 / Q4: Isolating anatomical guidance with controlled mask ablations.

Correct anatomical alignment improves retrieval when the other settings are held fixed. Table S21 evaluates all 2,461 MIMIC-CXR test queries with model weights, admission, evidence bank, neighbour counts, and evidence-selection rules fixed:

Retrieval conditionC14 microC14 macroC5 microC5 macro
No explicit spatial guidance60.748.061.155.2
Spatially misaligned masks59.246.259.654.1
Correct spatial guidance63.050.364.958.7

The no-guidance condition removes spatial restrictions from query construction. Misaligned masks shift the permitted regions within the same image while preserving mask area. Correct guidance improves micro/macro F1 by 2.3/2.3 points over no guidance and 3.8/4.1 points over misaligned masks, demonstrating the benefit of correct spatial alignment under fixed retrieval settings.

W5 / Q5: Which comparisons use matched experimental conditions?

The comparisons used to attribute gains share the same BLIP-3 experimental setting. They use the same backbone initialization, training data, current/prior image inputs, decoding settings, and evaluation pipeline. Within each retrieval path, raw and adapted evidence share classifier guidance and the label/soft-target supervision. These comparisons test the evidence-adapter pathway with classifier guidance present in both branches. The main comparison tables also include published results; their provenance is identified in the captions. The label-only control removes that evidence pathway while retaining classifier-label guidance.

W5 / Q5: Dataset, input, and evaluation protocols.

The evidence bank contains training data only. Current/prior image associations are consistent across our branches: the prior is the most recent earlier examination of the same patient, excluding the current and future examinations. Appendix A and Table S1 document datasets and evaluation cohorts; Appendix C specifies the retrieval pipeline. LIBRA provides a second-backbone integration experiment. The captions distinguish our controlled experiments from published comparison results.

W3 / Q3: End-to-end inference latency relative to the backbone and RADAR.

AnatoRAG takes 23.27 s per case, compared with 15.05 s for the backbone and 40.34 s for RADAR, on the same NVIDIA RTX 4090 with beam width 5 (Table S24). RADAR's sketch, retrieval, and final-generation stages take 15.02, 2.77, and 22.55 s, respectively, summing to 40.34 s. The measured stage times and totals are:

MethodSketch generationRetrievalFinal generationTotal
Backbone——15.05 s15.05 s
RADAR15.02 s2.77 s22.55 s40.34 s
AnatoRAG—4.04 s19.23 s23.27 s

All runtime values in this table are our measurements.

W3 / Q3: Runtime breakdown and peak GPU memory.

For AnatoRAG, we measured mean online inference time over 100 samples on a single NVIDIA RTX 4090 after warm-up, excluding model and index initialization. Retrieval accounts for 4.04 s and covers input-image processing through final evidence selection; generation accounts for 19.23 s and includes the evidence adapter. Peak GPU memory is 20.04 GB. This is a retrieval/generation-stage breakdown; the adapter cost is included in generation. Offline evidence-bank construction is a one-time cost and is excluded from online inference.

Thanks to the authors for providing further clarification. They have addressed some of my concerns. I have decided to keep my rating.

We thank the reviewer for the follow-up. To avoid guessing, here is where each of the five points in your review stands after the discussion:

Point from your reviewEvidence added
62-case CheXpert PlusAll 62 cases with Findings references in the public validation subset, none selected, same cohort used by RADAR and LIBRA (App. A)
Missing clinical error analysisExpert audit of 100 generated reports with per-category frequencies and consequential examples (Table S23, Ethics Statement)
Computational overhead23.27 s per case vs 15.05 s backbone and 40.34 s RADAR, 20.04 GB peak, with retrieval/generation stage split (Table S24)
Anatomy ablation not isolatedCorrect masks 63.0/50.3, no explicit spatial guidance 60.7/48.0, area-matched misaligned masks 59.2/46.2, with model weights, admission and evidence bank fixed (Table S21)
Baselines from prior papersControlled rows share backbone initialization, data, current/prior inputs, decoding and evaluation protocol; published numbers are marked as such in the captions

Could you indicate which of the five is still open, or name the concern if it is not among them? A row number is enough. We will respond to that item alone and leave the rest of your time free.

Reviewer_za4xOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

AnatoRAG generates chest X-ray reports by retrieving fine-grained radiological findings, filtering them based on consistency with image-predicted labels, and organizing the selected findings into anatomically structured tokens. The method improves BLIP-3's CheXbert-14 micro-F1 score from 56.5 to 65.1 and increases RadGraph F1 from 32.8 to 34.9.

Critical strengths
  • The motivation is well grounded and clinically meaningful. Retrieving entire reports can introduce findings that are irrelevant to the current patient, potentially propagating spurious or contradictory information into the generated report. By focusing on finding-level retrieval, the proposed approach directly addresses this limitation and promotes more patient-specific report generation.

  • The method demonstrates substantial improvements over the underlying BLIP-3 backbone. Significant gains are observed across multiple evaluation metrics, indicating that the proposed retrieval and anatomical organization strategy consistently enhances report quality beyond what can be achieved with the base generator alone.

  • The ablation studies provide useful evidence for the effectiveness of key design choices. In particular, the results show that the evidence adapter plays an important role in enabling the generator to effectively incorporate retrieved findings, leading to measurable performance improvements.

  • The paper includes informative analyses of both retrieval and generation performance. By evaluating these components separately, the authors provide valuable insight into where clinically relevant information is preserved or lost within the pipeline. This analysis is particularly useful on the IU X-Ray dataset, where it helps diagnose the limitations of the end-to-end system and identify opportunities for further improvement.

Critical weaknesses
  • The verification mechanism is relatively limited in scope. While the method checks the consistency of retrieved findings with predicted finding labels, it does not explicitly verify other clinically important attributes such as severity, anatomical extent, uncertainty, or temporal progression. As a result, the claim of “verifying each statement” may overstate the level of validation performed by the system.

  • The contribution of retrieval is not fully disentangled from that of classifier guidance. Although the overall framework yields strong improvements, the experiments do not include a classifier-guidance-only baseline without retrieved text. Such a comparison would help quantify how much of the gain arises from the retrieval component versus the label-based guidance mechanism.

  • The evaluation has several limitations that weaken the strength of the conclusions. The CheXpert Plus benchmark contains only 62 cases, raising questions about the statistical reliability of the reported results. In addition, the generation experiments do not report uncertainty measures, such as confidence intervals or results across multiple runs, and no expert radiologist evaluation is provided. Consequently, the clinical significance and robustness of the reported improvements remain difficult to assess.

Questions for authors

Please address the minor weakness points.

1 comment

We thank the reviewer for the positive assessment and constructive suggestions. Your comments helped us clarify verification, quantify gains beyond classifier guidance, and strengthen the evaluation.

W1: Does the verification claim exceed disease-label compatibility?

Our mechanism checks whether retrieved statements express disease labels compatible with image-predicted findings. We have just clarified this as “label-level compatibility filtering” in the revised manuscript.

Human experts reviewed 562 retrieved statements against their reference reports:

Attribute discrepancyStatements (%)
Severity11/562 (2.0%)
Laterality4/562 (0.7%)
Temporal change25/562 (4.4%)
Total40/562 (7.1%)

Temporal descriptions account for most observed conflicts, while severity and laterality discrepancies are less frequent. We have just included this analysis in Table S22.

W2: Separating retrieved evidence from classifier guidance.

Classifier guidance alone yields lower scores than the complete method. The label-only control isolates this question by retaining classifier-label guidance and removing retrieved evidence and the evidence-adapter pathway. The raw anatomy-linked row then separates using the retrieved text from adapting it into structured evidence slots:

ConfigurationClassifier guidanceReport-level evidenceAnatomy-linked evidenceEvidence adapterC14 microC14 macroC5 microC5 macro
Classifier-label guidance only✓✗✗✗58.942.762.854.7
Anatomy-linked, raw✓✗✓✗59.643.361.053.6
Full AnatoRAG✓✗✓✓65.150.867.559.8

Classifier guidance denotes the retained classifier-label input; evidence adapter denotes the pathway that encodes retrieved statements into evidence slots. The label-only row has no retrieved evidence or evidence-adapter pathway.

Full AnatoRAG improves over label-only generation by 6.2/8.1 C14 micro/macro points and 4.7/5.1 C5 micro/macro points.

All evidence-bearing Table 4 rows receive label guidance, and each raw/adapted pair shares the classifier prior and label/soft-target supervision. Adapting the same anatomy-linked evidence adds 5.5/7.5 C14 micro/macro points. We have just made these shared conditions explicit in Table 4 and Section 4.4; evidence reconstruction is used only with the evidence-adapter pathway.

W3: What does the 62-case external evaluation establish?

We did not select the CheXpert Plus cases. The public validation subset contains 62 cases with Findings references, and we evaluated all 62. RADAR and LIBRA also use this Findings subset for external evaluation. Together with IU X-Ray, this complete Findings cohort provides the external evaluation.

W3: Uncertainty and repeated runs.

For each dataset, we performed three independent generation runs with different random seeds using the same fixed model checkpoint, decoding settings, and evaluation cohort. Each cell reports the mean ± standard deviation across these generation runs (%):

DatasetC14 microC14 macroC5 microC5 macro
MIMIC-CXR65.16 ± 0.3750.94 ± 0.2867.47 ± 0.3359.83 ± 0.22
IU X-Ray57.45 ± 1.4233.53 ± 1.0145.70 ± 1.1333.52 ± 0.90
CheXpert Plus60.38 ± 1.7136.53 ± 0.5958.75 ± 0.7848.12 ± 0.80

W3: Expert evaluation and clinical significance.

The expert audit above examines retrieved evidence. Separately, human expert review of 100 generated reports randomly sampled from the MIMIC-CXR test set against reference Findings identified explicit discrepancies in 40/100 reports (40%). Table S23 reports the error categories. The revised ethics discussion states that generated reports require review and correction by qualified radiologists against the source images and clinical context, together with validation before clinical use.

Reviewer_quKeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AnatoRAG replaces whole-report retrieval with statement-level retrieval for chest X-ray Findings generation. An image-label classifier (initialized from RADAR’s auxiliary checkpoint) decides which CheXbert-14 labels are “admitted”; for each admitted label, statements are retrieved from a training-only bank of LLM-extracted atomic sentences organized into six anatomical families, with per-label queries whose attention is restricted by CXAS masks. Retrieved statements are kept only if their CheXbert labels are compatible with the admitted set, selected with a coverage bonus, then compressed by a cross-attention adapter into 72 slot tokens + 14 label tokens for a BLIP-3 (xGen-MM) generator fine-tuned with LoRA. On the standard 2,461-sample MIMIC-CXR Findings split it reports 65.1 / 50.8 CheXbert-14 micro/macro F1 (RADAR: 62.7 / 46.0), 34.9 RadGraph F1 and 2.61 RadCliQ-v0 (RADAR: 34.6 / 2.61), with transfer results on CheXpert Plus (62 cases) and IU X-Ray.

Critical strengths
  1. The compositional-sparsity analysis is the paper’s real contribution: it quantifies, with bootstrap CIs, that exact multi-label matches become rare in the training corpus as findings accumulate (1.6% exact match for top-1 report retrieval) while label-level evidence is almost always available (99.4%, median two statements). This cleanly explains why report-level RAG fails on multi-finding studies.

  2. Strong CheXbert results on the standard split against a same-backbone, same-classifier, same-input predecessor (RADAR): +2.4 micro / +4.8 macro, with the macro gain concentrated on rare labels. Separating retrieval-only from generation scores is good practice.

Critical weaknesses
  1. The source of the gain is not identified, and the most important baseline is missing. The admission classifier is RADAR’s checkpoint, and RADAR’s own ablation (Table 4) shows that injecting the classifier’s observation labels into the same BLIP-3 backbone with no retrieval already yields 62.4 micro / 44.2 macro. This paper never reports the admission set’s own CheXbert F1 and has no “backbone + 14 label tokens, no retrieval” row. The retriever’s 63.0 micro may simply be the classifier’s label F1 restated. Compounding this, the adapter-vs-raw comparison in Table 3 (+7.5 macro, attributed to “compression”) is confounded: the adapter output includes 14 label tokens Z_S from the calibrated classifier prior and is initialized with label-supervised losses (Eq. 7), while raw-text rows receive neither. The paper’s central claim—that verification and compression, not retrieval, drive the gain—is therefore unsupported.

  2. Claims exceed the mechanism, and the mechanism is incremental. “Verification against the target image” is only a CheXbert label-compatibility check; it cannot see severity, laterality or temporal change—the paper’s own Case I passes two mutually contradictory temporal statements (“worsening” / “minimally decreased”) as verified evidence. “Anatomy-guided” is the smallest contributor in the ablation (−1.1 macro for removing the anatomy residual, Table S11), and the six “anatomical families” are label groupings (Table S2). The core classify-then-label-filter design is RADAR’s (and LaB-RAG’s, uncited); region-level sentence retrieval exists in STAR3, AnatomiX and CPR-RAG (latter two uncited); LLM-extracted atomic banks in RA-RRG and AdaRAG-CT (uncited); classifier-gated fixed-slot evidence tokens in AM-MRG and EKAGen (uncited). Gains over RADAR appear only on CheXbert (the metric the pipeline optimizes by construction) and lexical metrics; RadGraph F1 (+0.3) and RadCliQ (tie) do not move, and MAIRA-X (64.1/47.2, uncited) is the nearest CheXbert competitor. Transfer claims rest on 62 CheXpert Plus cases and an IU X-Ray result that does not exceed RADAR (32.1/54.6 vs 32.5/54.6). All generation numbers are single runs without CIs.

Questions for authors
  1. Report the admission set A’s CheXbert-14 micro/macro F1 against reference labels, and add two ablations: (a) BLIP-3 + Z_S label tokens with no retrieved evidence; (b) raw anatomy-linked statements + Z_S label tokens. Do retrieval and the slot adapter still add ≥2 micro / ≥4 macro over (a)?
  2. In Table 3, do raw-text rows receive p̃ / Z_S? Do they receive any supervision equivalent to L_label / L_soft used to initialize the adapter?
  3. How were the per-label branch combinations, neighbour counts (4–256) and class-specific heads in Table S3 selected, and on which split?
  4. Why does the same retriever score 50.3/63.0 in Table 1 but 49.8/62.7 in Tables S5/S11?
  5. What fraction of “compatible” statements contradict the reference on severity, laterality or change? Case I suggests this is non-trivial.
5 comments

We thank the reviewer for the detailed comments, which helped us clarify the contribution, strengthen gain attribution, and improve the evaluation.

AnatoRAG addresses how findings retrieved independently from different reports can be used together by a report generator. It selects compatible statements with complementary finding coverage, then maps their content through anatomy-grouped cross-attention into evidence slots alongside label guidance and native image tokens. The contribution is this learned interface between compositional evidence selection and report generation.

Our original evidence-configuration ablation compares raw text with adapted evidence within both the report-level and anatomy-linked retrieval paths, with classifier guidance retained. It therefore tests the evidence-delivery pathway. The additional label-only baseline answers the reviewer's complementary question: whether classifier guidance alone explains the full system's performance. These two comparisons address different parts of gain attribution.

W1 / Q1(a): Missing label-only baseline and gains beyond classifier guidance.

Classifier-label guidance alone does not account for the full model’s gain. The label-only control retains the same image and clinical inputs and the classifier-label guidance branch. It removes retrieved text and the evidence-slot branch; the generator therefore receives label guidance without retrieved evidence tokens. Full AnatoRAG improves over this control by 6.2/8.1 C14 micro/macro points, above the 2/4-point margins raised in Q1. AnatoRAG also exceeds the published RADAR label-only result of 62.4/44.2 by 2.7/6.6 points.

W1 / Q1(b) / Q2: Are classifier priors and label supervision matched between raw and adapted evidence?

The raw-text rows already receive label guidance ZlabelZ_{\mathrm{label}}: removing the evidence-slot output does not remove the label-guidance branch. In the raw condition, selected statements enter the generator as text alongside label guidance; in the adapted condition, they enter through evidence slots alongside label guidance. Both retain the image-classifier prior and auxiliary label/soft-target supervision. Evidence reconstruction belongs to the evidence-adapter pathway. The comparison therefore measures the evidence-delivery pathway and its training, with classifier guidance present in both conditions. We have just made this explicit in Table 4 and Section 4.4. The original evidence configurations and the added label-only result are shown together below:

ConfigurationClassifier guidanceReport-level evidenceAnatomy-linked evidenceEvidence adapterC14 microC14 macroC5 microC5 macro
Classifier-label guidance only✓✗✗✗58.942.762.854.7
Report-level, raw✓✓✗✗58.642.160.252.4
Report-level, adapted✓✓✗✓63.146.465.156.7
Anatomy-linked, raw✓✗✓✗59.643.361.053.6
Full AnatoRAG✓✗✓✓65.150.867.559.8

W1: What do the controls establish about the contribution?

Three comparisons separate the questions raised in the review:

  1. Does label guidance suffice? Label-only reaches 58.9/42.7 C14 micro/macro, versus 65.1/50.8 for the full model. The complete evidence pathway adds 6.2/8.1 points beyond the retained label guidance.
  2. Does the way evidence is delivered matter? With the retrieved text and label guidance held fixed within each path, adaptation adds 4.5/4.3 points for report evidence and 5.5/7.5 for anatomy-linked statements. This evaluates the learned evidence-adapter pathway and its training.
  3. Does the anatomy-linked evidence source add value at the same delivery stage? Its advantage over report-level evidence is 1.0/1.2 points with raw text and 2.0/4.4 with adaptation. The larger advantage under adaptation supports the connection between complementary statement retrieval and structured evidence delivery.

Raw anatomy-linked statements add only 0.7/0.6 points over label-only, whereas adapting those statements adds a further 5.5/7.5 points. The result identifies the practical problem addressed by the adapter: making independently selected evidence useful to the generator. The spatial controls below separately examine correct image-region guidance.

The same evidence-delivery comparison also improves metrics beyond CheXbert: RadGraph F1 rises from 31.3 to 34.9 and RadCliQ-v0 falls from 2.78 to 2.61. The backbone scores are 32.8/2.74, while RADAR scores 34.6/2.61. These results support the adapter's contribution to report quality; relative to RADAR, the largest clinical gains are in finding-label agreement.

W1 / Q1: Admission-label performance versus retrieval-output performance.

Admission evaluates a binary label set, whereas retrieval evaluates labels extracted from the selected statements. Their separate scores are below. The label-only generation control above tests whether passing the classifier information to the generator suffices; its lower performance shows why the evidence contribution cannot be inferred from admission F1 alone.

Evaluated outputC14 microC14 macroC5 microC5 macro
Classifier60.943.262.154.5
Admission after neighbour fusion63.750.265.459.1
Full retrieval63.050.364.958.7

We have just added the classifier and admission-label results in Table S20; the final retrieval results appear in Table 3.

Q4: Why do Table 1 and S5/S11 give different retrieval scores?

The two reported scores came from two independently trained retrievers: the 63.0 result was obtained from training and retrieval on RTX PRO 6000, and 62.7 from an A6000-trained run. They are separate training runs. We have just unified the reported full-retriever results to C14 micro/macro 63.0/50.3 and C5 micro/macro 64.9/58.7. The three-run training summary is included at the end of Appendix J.

W2: How does the mechanism differ from the cited work?

The central distinction is what happens to the selected statement instances before generation. In AnatoRAG, masks guide image–statement matching; label compatibility and coverage select a complementary set across source reports; family-grouped cross-attention converts that set into 72 evidence tokens. These tokens retain the anatomical organization of the selected text at the generator interface, alongside 14 label-conditioning tokens and native image tokens.

RADAR and LaB-RAG provide precedents for classifier guidance; STAR3, AnatomiX, and CPR-RAG for regional or anatomical evidence; RA-RRG and AdaRAG-CT for fine-grained or selectively integrated evidence; and AM-MRG and EKAGen for disease-conditioned knowledge interfaces. Our mechanism links explicit statement selection to anatomy-structured adaptation. The raw/adapted comparisons test this interface on the same retrieved content, while the report-level/anatomy-linked comparisons test the two evidence-source systems at each delivery stage. This is why both axes of the original ablation matter to the contribution.

We have just added these references and the corresponding mechanism-level comparisons in Appendix I.1.

W2: Anatomical guidance beyond label grouping and the query-residual ablation.

The six families organize evidence and slots; the spatial contribution comes from masks restricting the image regions used for queries. The query-residual row in the revised Table S10 removes the learned query residual while retaining the base spatial query. With weights, admission, bank, neighbours, and selection fixed, the spatial controls give:

Retrieval conditionC14 microC14 macroC5 microC5 macro
No explicit spatial guidance60.748.061.155.2
Spatially misaligned masks59.246.259.654.1
Correct spatial guidance63.050.364.958.7

W2 / Q5: Scope of verification and reference-inconsistent clinical attributes.

In Case I, the selected evidence includes worsening congestive failure and pulmonary edema that may have decreased, while the reference describes worsening edema. The label-compatibility rule does not resolve these temporal differences. Human experts reviewed 562 retrieved statements against their reference reports:

Attribute discrepancyStatements (%)
Severity11/562 (2.0%)
Laterality4/562 (0.7%)
Temporal change25/562 (4.4%)
Total40/562 (7.1%)

Temporal change accounts for most observed attribute discrepancies. We have just clarified the wording as “label-level compatibility filtering” and included this review in Table S22.

Q3: How were branches, neighbours, and label-specific components chosen?

The revised Table S6 specifies which information sources contribute to admission for each disease. These branch combinations are predefined according to the type of evidence used to assess that disease, rather than selected by validation-set search. For example, the Fracture branch uses regional and osseous-subview information, while Support Devices combines global, regional, and device-subview information. The branches provide complementary scores for admission; they are distinct from the anatomy-conditioned queries used to retrieve statements. The admission classifier follows RADAR’s design. The current retriever uses shared retrieval parameters with label-conditioned queries, rather than separately trained class-specific classification heads. We have removed the optional classification-refinement heads from the configuration description, as they did not change the final retrieval results.

Neighbour counts within 4–256 and other numerical configurations were selected on the validation split using the combined CheXbert score. Fusion coefficients, normalization statistics, and admission thresholds are fitted on training data and frozen, as described in Appendix C. Admission neighbours contribute label scores, whereas the separate statement shortlist contains K=64K=64 candidate statements per label. We have just clarified these roles and selection procedures in Table S6 and Appendix C.

W2: External generalization and the 62-case CheXpert Plus cohort.

The external-cohort comparison in revised Table 2 reports comparable performance to RADAR on IU X-Ray; the updated three-seed results are reported below. CheXpert Plus provides an additional external cohort under the same Findings protocol. We evaluated all 62 available Findings cases in the public validation subset, without case selection. RADAR and LIBRA also use this Findings subset. Together with IU X-Ray, this complete Findings cohort provides the external evaluation. Appendix I.1 also compares MAIRA-X, whose 3.1-million-study Mayo Clinic training set differs from our MIMIC-CXR training protocol.

W2: Single-run results and statistical uncertainty.

For each dataset, we performed three independent generation runs with different random seeds using the same fixed model checkpoint, decoding settings, and evaluation cohort. Each cell reports the mean ± standard deviation across these generation runs (%). These repeated-run summaries are presented separately from the individual-run results in the main tables:

DatasetC14 microC14 macroC5 microC5 macro
MIMIC-CXR65.16 ± 0.3750.94 ± 0.2867.47 ± 0.3359.83 ± 0.22
IU X-Ray57.45 ± 1.4233.53 ± 1.0145.70 ± 1.1333.52 ± 0.90
CheXpert Plus60.38 ± 1.7136.53 ± 0.5958.75 ± 0.7848.12 ± 0.80

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers: Summary of Revisions and Clarifications

We thank all reviewers for their constructive feedback. The discussion helped us clarify the contribution of AnatoRAG, strengthen the controlled comparisons, and expand the evaluation.

AnatoRAG addresses how complementary findings retrieved from different reports can be assembled and used by a report generator. Its contribution connects anatomy-guided statement selection with a learned, anatomically structured evidence interface. Our comparisons examine both the evidence source and how that evidence reaches the generator.

1. Gains beyond complete classifier guidance.

We added the requested label-only generation control and reported the admission labels’ own performance. One terminology clarification is essential: admission, including neighbour fusion, belongs to the complete classifier pipeline. All classifier-guided generation configurations use this complete output.

Evaluated outputC14 microC14 macro
Complete classifier’s admitted labels63.750.2
Reports generated with complete classifier guidance only58.942.7
Raw anatomy-linked evidence + complete classifier guidance59.643.3
Full AnatoRAG with the same classifier guidance65.150.8

The label-only control already includes neighbour fusion. The matched generation comparison therefore shows +6.2 micro / +8.1 macro beyond complete classifier guidance. Within the anatomy-linked path, adapting the same evidence adds +5.5 / +7.5 over raw text. Raw and adapted branches retain label guidance and auxiliary label/soft-target supervision; evidence reconstruction belongs to the evidence-adapter pathway.

The report-level path provides a second controlled comparison: adaptation improves C14 micro/macro from 58.6/42.1 to 63.1/46.4. Together, these results support the contribution of the learned evidence-delivery pathway across both evidence sources.

2. Controlled spatial-guidance ablations.

With model weights, admission, evidence bank, neighbour counts, and selection rules fixed, correct spatial guidance achieves 63.0/50.3 C14 micro/macro, compared with 60.7/48.0 without explicit spatial guidance and 59.2/46.2 with area-matched misaligned masks. These controls examine the contribution of image-region alignment separately from classifier guidance.

3. Clinical error analysis and scope of verification.

We added human expert audits of 562 retrieved statements and 100 generated reports. The statement audit identified 40/562 attribute discrepancies (7.1%), chiefly temporal-change discrepancies. The report audit provides frequencies for unsupported findings, omissions or incorrect denials, anatomical errors, and other clinical-description discrepancies. We clarified the mechanism as label-level compatibility filtering and expanded the Ethics Statement to address consequential errors and the need for radiologist review.

4. External evaluation and repeated generation.

We evaluated all 62 available Findings cases in the public CheXpert Plus validation subset, without selecting cases; RADAR and LIBRA also use this Findings subset. We reported three-generation mean ± standard deviation on MIMIC-CXR, IU X-Ray, and CheXpert Plus using fixed checkpoints and consistent decoding settings.

For IU X-Ray, 54.6 remains the original individual-generation result; 57.45 ± 1.42 summarizes three additional generation runs. The latter is reported separately as generation variability.

5. Computational cost and comparison protocols.

On the same RTX 4090 with beam width 5, measured online latency is 23.27 s per case for AnatoRAG, versus 15.05 s for the backbone and 40.34 s for RADAR. AnatoRAG’s retrieval and generation stages take 4.04 s and 19.23 s, with peak GPU memory of 20.04 GB. We also clarified the shared settings of our controlled experiments, distinguished published comparison results, and expanded the discussion of the related methods raised by the reviewers.

We hope these controls and clarifications help the reviewers assess the contribution of anatomy-guided evidence selection and structured delivery beyond classifier guidance alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.