acceptodds
Under review as a conference paper at ICLR 2027

From Experience to Lessons: Organizing Visual Memory for Chest X-Ray Report Generation

Abstract

LLM agents have been found promising for completing complex tasks, where the episodes recording how an agent made decisions to obtain the outcomes can be maintained in a memory store for experience reuse to make better decisions. In this paper, we investigate the use of MLLM agents for chest X-ray report genera- tion. We first make use of a multimodal (vision-language) episodic memory store for maintaining the experience. We further propose to organize the episodes as a compact set of multimodal lessons which are expected to be more generalizable for reuse. We conduct experiments on MIMIC-CXR with 76 lessons learned for retrieval by the agent to enhance the report generation quality, resulting in im- provement on finding-level balanced accuracy by 2.90 points, more than retrieval over all 8,192 raw episodes (1.07 points). In addition, by allowing the agent to learn how to properly incorporate the retrieved lessons via rewarding corrected findings and penalizing those introducing errors, the gain over the agent without memory rises to 5.18 points and the introduced errors fall from 123 to 74. We also demonstrate that the lessons learned are transferable to different X-ray datasets, different agent frameworks, and different MLLM base models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal agents, agent memory, learning from experience, reinforcement learning, chest X-ray report generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ryGMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the use of multimodal large language model (MLLM) episodic memory towards enhancing report generation quality for chest X-rays (CXR), after initial organization into a compact set of multimodal lessons that are retrieved by case context (keys). A compact set of 76 lessons was found to improve finding-level balanced accuracy compared to using the original set of 8192 episodes for retrieval, on MIMIC-CXR. The lessons were moreover found to generalize to other datasets, agents and models, which increases its practical significance.

Critical strengths
  • Valuable insight into the potential of using consolidated lessons instead of raw cases as the retrieval target for MLLMs
Critical weaknesses
  • Design choices for constructing the lesson store are not well-described, despite the lesson store being the major product
  • Uncertain as to whether the increased performance is due to the lesson paradigm, or simply because more data is effectively retrieved with (consolidated) lessons than individual cases
Questions for authors
  1. In Section 3.2, a definition of each (visual) lesson is provided in Equation 2. It might be considered to indicate the variables LγL_\gamma, VγV_\gamma and AγA_\gamma with reference to concrete examples, such as the lesson traces in Appendix D Figure 3.

  2. In Section 3.2, AγA_\gamma is stated to store the lesson's writing cases and outcomes on its check cases. This might be further explained, especially regarding the outcomes - possibly again with concrete examples, and in terms of ξ\xi and other variables later referenced in Section 3.3.

  3. In Section 3.3, it is stated that "Matching the current case to the cases behind a lesson, rather than to the lesson's text, retrieves lessons by the situation in which the error occurred". However, the cases within a lesson appear to be generated from groups of 16 cases (8 writing cases; from Appendix A), with no particular guarantee as to the conditions and situations covered by the writing cases. Given that MIMIC-CXR contains 14 standard classes, it is unclear as to whether 76 lessons of three cases/images each would sufficiently or appropriately represent each class. This might be carefully discussed, also with respect to the major issue on the lesson construction protocol.

  4. In Section 4, five questions (RQ1 to RQ5) are declared and addressed through experiments. However, the optimal construction procedure of the lesson store SS from the original reports does not appear fully addressed in Section 3.2 (or Appendix A), which presents the procedure without either theoretical or empirical basis - and which does not appear to be reflected in the RQs either (Section 3.4 then describes the optimization of lesson usage, assuming a fixed set of lessons). In particular, it is not clear as to whether lessons are generated by uniform sampling over cases, by assigning a fixed number of lessons per context or other feature (in which case the number would be a relevant tunable parameter), etc.

  5. In Section 4.1, episodic memory is stated to retrieve the K=3K=3 most similar cases from the 8192 episodes with the same policy, prompt structure and acceptance rule as a lesson. However, each lesson appears to include far more independent episodes (3, constructed from a group of 16 cases from Appendix A), in which case it is unclear as to whether the performance gain is from lessons per se, or from the additional data (9 cases, compared to 3). This might be justified, along with whether a comparable amount of data is also retrieved for ExpeL and ViLoMem, since Appendix B.2 suggest that ViLoMem also retrieves 3 cases only.

  6. Related to the above, K=3K=3 for lesson/case retrieval is a pertinent parameter that might be ablated on, to quantify the value of additional cases for performance.

  7. In Section 4.2 relating to RQ1, some of the result descriptions do not appear readily referencable in Table 1. For example, it is stated that "Retrieval over all 8,192 source episodes raises FBA by 1.07 and CheXbert by 2.20 points and lowers RadGraph by 0.70", but it is not clear as to whether this refers to episodic memory, and might be clarified.

3 comments

We thank the reviewer for their positive assessment and constructive suggestions.

Summary of changes. We conducted additional experiments to evaluate (1) lessons per error type, (2) multi-case consolidation versus separate case records, and (3) the number of retrieved lessons. We also analyzed lesson coverage and clarified the lesson components, construction procedure, retrieval mechanism, and baseline comparisons. We will incorporate these clarifications and experimental results into the revised paper.

We address the reviewer's specific points below.

Q1-Q2

We clarify the components of a visual lesson, γ=(Lγ,Vγ,Aγ)\gamma=(L_\gamma,V_\gamma,A_\gamma), using the “add edema” example in Appendix D, Figure 3:

  • LγL_\gamma (lesson text): Contains the condition, decision, visual cue, and exception. The example suggests adding edema when the draft omits it and the image shows supporting findings, such as increased interstitial markings or peribronchial cuffing. The exception includes chronic interstitial disease or fibrosis.
  • VγV_\gamma (visual evidence): Contains one to three historical radiographs with marked anatomical regions. Figure 3 shows a radiograph from writing case 58365028.
  • AγA_\gamma (record): Links the lesson to its eight writing cases BγB_\gamma and stores its outcomes on eight held-out check cases HγH_\gamma, with no patient appearing in both sets.

For each check case, we apply the lesson to its saved context ξi=(xi,τi,di)\xi_i=(x_i,\tau_i,d_i) following Eq. 3. We compare the returned report with the initial draft did_i, using the reference yi∗y_i^* to count corrected findings and introduced errors (Eq. 4). For example, correctly adding a missing edema finding counts as a correction, while introducing an incorrect finding elsewhere counts as an introduced error.

During inference, the reporting policy receives LγL_\gamma and VγV_\gamma; AγA_\gamma is retained for inspection and traceability.

Q3

The lesson store is designed to address recurring reporting errors, rather than provide a fixed number of lessons for every diagnostic label. Errors are assessed across the finding vocabulary, and selected cases are grouped by target finding and correction direction. A finding being present in a case does not necessarily mean that the base agent reported it incorrectly.

Each lesson is written from eight writing cases, with eight additional cases reserved for checking. The writer consolidates the writing-case evidence into a condition, reporting decision, visual cue, and exception. The one to three retained historical images illustrate this guidance; they are not the only cases used to construct the lesson.

Table 1. Findings in writing cases and retained lesson targets.

FindingWriting cases with this labelLessons targeting this finding
Enlarged cardiomediastinum480
Cardiomegaly2360
Lung opacity2967
Lung lesion370
Edema1709
Consolidation598
Pneumonia5610
Atelectasis27210
Pneumothorax310
Pleural effusion29021
Pleural other260
Fracture280
Support devices2550
Vascular congestion10711

The table distinguishes findings present in historical cases from the findings targeted by retained lessons. For example, cardiomegaly appears in 236 writing cases, although no retained lesson explicitly targets it. Lesson targets depend on the selected reporting errors and available visual support. Consequently, a zero lesson count does not establish that the base agent makes no errors on that finding.

Retrieval ranks lessons by similarity to their writing-case contexts, rather than matching lesson text. The reporting policy then considers the lesson’s condition, visual cue, and exception against the current patient’s images to decide whether to apply it (§3.3). The construction procedure and the effect of retaining multiple lessons per error type are addressed in Q4.

Q4

Lesson construction. We identify reporting errors by comparing the findings in the base agent’s initial drafts with their corresponding reference reports across 8,192 MIMIC-CXR episodes. Each error specifies a target finding and correction direction: add when the reference reports a finding that the draft omits or denies, and remove when the draft asserts or hedges a finding that the reference doesn't contain. We then group cases requiring the same correction, such as add edema or remove pleural effusion (§3.2). Within each error type, we randomly select candidate cases using a fixed seed and form groups of 16 distinct patients: eight writing cases and eight reserved check cases, with no patient reused across groups. The number of candidate groups is limited separately for each error type. The writer consolidates the writing cases into textual guidance and one to three historical images. After visual-support and lesson-usability checks, 76 lessons are retained.

Lessons per error type. To address the reviewer’s question about lesson allocation, we evaluated how performance changes when retaining different numbers of lessons per finding–direction pair. We compared stores retaining at most one, two, or all available lessons per finding–direction pair, resulting in 8, 16, and 76 lessons, respectively. All settings use the same initial drafts, base policy, and retrieval budget (K=3).

Table 2. Effect of lessons per error type.

Lessons in storeRadGraph ↑CheXbert ↑GritLM ↑FBA ↑
817.2649.4165.0165.69
1617.1250.7965.8766.06
76 — original18.5653.7965.1467.69

Increasing the store from 8 to 16 and 76 lessons improves FBA from 65.69 to 66.06 and 67.69, respectively. The original store also achieves the highest CheXbert and RadGraph scores. These results support the benefit of retaining multiple lessons per error type.

Q5

We first clarify that each lesson is constructed from eight writing cases, while the other eight cases are used for checking. At inference, each revision call receives one lesson with one to three historical images. With K=3K=3, nine images is the maximum across three separate attempts, not the input to a single revision call.

To isolate the contribution of consolidation, we replaced each lesson's consolidated text with its eight writing-case grounding notes presented together, keeping the historical images and boxes, initial drafts, retrieval order, revision policy, and acceptance procedure unchanged.

Table 3. Effect of multi-case consolidation.

Metric ↑Consolidated lessonsEight writing caseDifference
RadGraph18.5616.1+2.46
CheXbert53.7950.59+3.2
GritLM65.1465.11+0.03
FBA67.6966.5+1.19

Consolidation improves FBA by 1.19 points, CheXbert by 3.20, and RadGraph by 2.46 over presenting the same underlying case records without consolidation. This demonstrates the benefit of organizing historical experience into lessons beyond providing additional case information.

For the other memory baselines, episodic memory retrieves three individual cases from the 8,192 episodes and attempts them separately. ExpeL provides six global rules and two retrieved reference reports, while ViLoMem retrieves three logical and three visual guidelines. Both ExpeL and ViLoMem are constructed from the same 608 writing cases as our lessons and use one revision call (Appendix B.2).

Q6

We evaluated the effect of the maximum number of retrieved lessons KK, keeping the lesson store, initial drafts, and base policy unchanged. Inference stops at the first accepted revision or after examining all KK candidates.

Table 4. Effect of retrieval budget KK.

KKRadGraph ↑CheXbert ↑GritLM ↑FBA ↑
118.3552.6065.0366.99
218.5853.3665.0967.13
3 — default18.5653.7965.1467.69
518.5254.3365.1968.18

Increasing KK from 1 to 3 improves FBA by 0.70 points and CheXbert by 1.19 points. Increasing KK further to 5 yields additional gains of 0.49 FBA and 0.54 CheXbert, while RadGraph remains similar. These results show that retrieving additional candidate lessons can further improve finding-level performance.

Q7

The statement refers to the “+ episodic memory” row in Table 1 of the submitted paper, compared with the “Agent, no memory” row. Retrieving the top (K=3) cases from the 8,192 source episodes increases FBA from 64.79 to 65.86 (+1.07) and CheXbert from 48.59 to 50.79 (+2.20), while RadGraph decreases from 16.76 to 16.06 (−0.70). We will clarify this baseline in §4.2.

We thank the authors for the clarifications.

Reviewer_34EKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a visual-language memory framework that turns past chest X-ray reporting errors into compact, reusable lessons. It builds 76 lessons from 8,192 MIMIC-CXR studies, each combining applicability guidance, annotated radiographs, and traceable source records. An agent retrieves relevant lessons to revise an initial report, while policy training uses repair and control cases to discourage harmful edits. On MIMIC-CXR, FBA increases from 64.79 to 67.69 with lessons and to 69.97 with policy training. The same lesson store also transfers to IU-Xray and other agent configurations. The results are encouraging, although individual lessons are not systematically checked against cases where their proposed correction would be wrong.

Critical strengths
  1. Compact, evidence-linked memory. Lessons combine explicit applicability conditions, visual examples, and source traces rather than retrieving isolated past cases. Removing lesson images lowers MIMIC-CXR FBA from 67.69 to 66.21 (Table 2), supporting the contribution of visual evidence.

  2. Broad and well-supported evaluation. Lessons outperform raw episodic memory, ExpeL, and the implemented ViLoMem baseline on the main finding-level measures. FBA improves by 2.90 points on MIMIC-CXR and 2.39 on IU-Xray; training further improves MIMIC-CXR FBA to 69.97 (Tables 1 and 3). Cross-framework experiments, paired bootstrap intervals, and both successful and failed case studies strengthen the evidence.

Critical weaknesses
  1. Lesson-level negative-case validation is insufficient. Lessons are checked on held-out historical errors with the same target finding and correction direction, rather than systematically on cases where the suggested change would introduce a wrong target-finding decision. Appendix D shows an effusion lesson that corrected no findings but introduced eight errors; all eight construction-stage check cases had effusion, so they could not expose this false-positive risk. Repair/control prompts during policy training reduce introduced errors overall (123 to 74), but do not independently validate or filter harmful lessons.

  2. Key component gains are not fully isolated. The ablations show that historical images matter, but do not separately test the value of region boxes, multi-case consolidation, or applicability conditions. Also, the trained model generates the initial draft (Appendix B.4), and retrieval depends on draft content (Appendix B.5); thus training may change drafting and retrieved lessons as well as revision decisions. This limits attribution of the RL gain specifically to better lesson applicability judgments.

Questions for authors
  1. Why are lesson-level check cases drawn only from the same error group? Evaluate each lesson on opposite-label cases and report its false-correction rate.

  2. Across all 76 lessons, how are corrections and introduced errors distributed? Does filtering consistently harmful lessons using held-out validation cases improve FBA and reduce new errors?

  3. Compare the base and trained revision policies on identical initial drafts with the same retrieved lessons to isolate the effect of policy training.

  4. Keep the same historical lesson images but remove their region boxes. How much of the gain comes from region annotations rather than the images alone?

2 comments

We thank the reviewer for their thoughtful feedback, which helps us clarify and strengthen our paper.

Summary of changes. We conducted additional experiments to address the reviewer's concerns:

  1. Lesson validation: Evaluated all 76 lessons on opposite-label cases, analyzed their corrections and introduced errors, and tested validation-based filtering.
  2. Policy training: Compared the base and trained reporting policies on identical initial drafts and retrieved lessons.
  3. Component ablations: Separately evaluated region bounding boxes, multi-case consolidation, and condition and exception fields.

We also clarify the lesson-construction procedure and will include the new experimental results in the revised paper.

We address the reviewer's specific points below.

W1+Q1

The original eight held-out check cases were drawn from the same error group to test whether a lesson could correct recurring errors. Following the reviewer's suggestion, we additionally evaluated all 76 lessons on opposite-label cases to assess the risk of introducing incorrect findings.

For an add lesson, we select cases where the reference finding is absent and the initial draft correctly indicates its absence. For a remove lesson, both the reference and initial draft indicate that the finding is present. We evaluated up to eight cases from distinct patients per lesson, yielding 578 lesson–case pairs across 236 distinct studies, using the base policy and original lesson-acceptance rule (§3.3). A false correction occurs when an initially correct target-finding decision becomes incorrect in the returned report; declined and unchanged attempts remain in the denominator.

Table 1. False-correction rates on opposite-label cases.

Lesson directionOpposite-label controlLessons evaluatedFalse corrections / trialsRate
Add findingReference absent, draft absent/unmentioned5770/42616.43%
Remove findingReference present, draft present191/1520.66%
OverallBoth directions7671/57812.28%

The false-correction rate is higher for add-finding lessons (16.43%) than remove-finding lessons (0.66%), identifying risks not captured by the original same-error checks. We further evaluated validation-based lesson filtering using randomly sampled held-out validation cases; the filtering procedure and test-set results are reported in Q2.

Q2

We analyzed the distribution of corrections and introduced errors across all 76 lessons using the base policy on the test set.

Table 2. Distribution of corrections and introduced errors across lessons.

Results per lessonNumber of lessonsCorrectionsIntroduced errors
More corrections than introduced errors5520478
More introduced errors than corrections14933
Equal corrections and introduced errors, both above zero61212
Zero corrections and introduced errors100
Total76225123

Of the 76 lessons, 55 produce more corrections than introduced errors, accounting for 204 corrections and 78 introduced errors. This shows that useful lessons can also introduce errors.

Validation-based filtering. We randomly sampled 1,000 held-out validation studies, with no patient overlap with construction, training, or testing. For each study, we evaluated its top-three retrieved lessons independently against the same initial draft, producing 3,000 lesson–case evaluations covering all 76 lessons. We removed a lesson when its introduced errors exceeded its corrections on validation cases. This removed 12 lessons, retaining 64 of 76 lessons. Both stores were then evaluated on the same test set using the base policy, common initial drafts, and the same inference procedure.

Table 3. Effect of validation-based lesson filtering.

OutcomeOriginal 76 lessonsFiltered 64 lessons
Incorrect labels corrected225248
Newly introduced errors123136
Net corrections102112
RadGraph ↑18.5618.58
CheXbert ↑53.7953.97
GritLM ↑65.1465.16
FBA ↑67.6967.82

With the retrieval budget fixed at (K=3), filtering changes which lessons are retrieved. Corrected findings increase from 225 to 248, alongside more introduced errors (123→136), while net corrections improve from 102 to 112.

These results demonstrate that validation-based filtering identifies lessons with negative validation outcomes and improves net corrections while maintaining overall report-generation performance, with FBA changing from 67.69 to 67.82.

W2+Q3

To isolate the effect of policy training, we compared the untrained (base policy) and trained revision policies using identical initial drafts, images, observations, and retrieved lessons in the same order, with the acceptance procedure unchanged.

Table 4. Effect of policy training with fixed drafts and retrieval.

Metric ↑Base policyTrained policyDifference
RadGraph18.5618.65+0.09
CheXbert53.7955.67+1.88
GritLM65.1465.29+0.15
FBA67.6969.49+1.80

With drafts and retrieval fixed, policy training improves FBA by 1.80 points and CheXbert by 1.88 points, with smaller gains in RadGraph and GritLM. This demonstrates that training improves lesson-conditioned revision independently of changes to initial drafting or retrieval.

For reference, the original trained policy, which also generates its own initial drafts, achieves 69.97 FBA (Table 3 in the paper).

W2+Q4

We conducted the ablation by removing region annotations from the same historical lesson images, while keeping the lesson text, initial drafts, retrieved lessons, revision policy, and acceptance procedure unchanged.

Table 5. Ablation on region annotations.

Metric ↑Original lessonsUnmarked historical imagesDifference
RadGraph18.5618.59+0.03
CheXbert53.7953.13−0.66
GritLM65.1465.07−0.07
FBA67.6967.06−0.63

Removing region annotations decreases FBA by 0.63 points and CheXbert by 0.66 points, while RadGraph and GritLM change only slightly. These results show that region annotations provide additional benefits beyond the historical images alone.

W2 - Multi-case consolidation and applicability conditions

Multi-case consolidation. To isolate the contribution of consolidation, we replaced each lesson's consolidated text with its eight writing-case records presented together, while keeping the historical images, initial drafts, retrieved lessons, revision policy, and acceptance procedure unchanged.

Table 6. Effect of multi-case consolidation.

Metric ↑Consolidated lessonsEight writing caseDifference
RadGraph18.5616.1+2.46
CheXbert53.7950.59+3.2
GritLM65.1465.11+0.03
FBA67.6966.5+1.19

Consolidation improves FBA by 1.19 points, CheXbert by 3.20, and RadGraph by 2.46 compared with presenting the underlying case records together, demonstrating the benefit of consolidating historical experience into lessons.

Applicability conditions. We further removed the condition and exception fields from the lessons while retaining the reporting decision, visual cue, and historical images.

Table 7. Ablation on condition and exception in lessons.

Metric ↑Full lessonWithout condition and exceptionDifference
RadGraph18.5616.03+2.53
CheXbert53.7949.9+3.89
GritLM65.1465.82-0.68
FBA67.6965.92+1.77

Removing these fields decreases FBA by 1.77 points, CheXbert by 3.89, and RadGraph by 2.53, while GritLM increases by 0.68. This is because GritLM measures overall semantic similarity, which does not necessarily reflect finding accuracy. These results demonstrate the contribution of explicitly specifying when a lesson applies and when it should not be used.

Reviewer_sU9qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates multimodal large language model (MLLM) agents for chest X-ray report generation. Rather than relying on raw textual information, the authors propose consolidating agent reporting errors into a compact library of 76 multimodal "lessons" derived from about 8000 MIMIC-CXR training episodes. Each lesson comprises four structured text fields (condition, decision, visual cue, exception), some historical reference radiographs with marked Chest ImaGenome anatomical bounding boxes, and links to source episodes for provenance. During inference, an agent generates an initial draft report, retrieves up to K=3 lessons via standard multimodal embedding similarity, and sequentially tests them one by one. The first proposal that modifies the draft's target finding in the suggested direction is accepted as the final report. To prevent over-triggering, the authors train the reporting policy using Group Relative Policy Optimization (GRPO) on a balanced set of repair cases (where the lesson corrects an error) and control cases (where following the lesson introduces an error), optimizing a finding-level reward. The framework is evaluated on the MIMIC-CXR and IU-Xray datasets across several metrics (FBA, CheXbert, RadGraph, GritLM, BLEU, ROUGE, METEOR), across four agent frameworks (ReAct, ReWOO, Plan-and-Solve, Swin-Qwen3), and across base model sizes from 4B to 122B parameters.

Concerning presentation, we can notice thatkey operational constraints and critical implementation dependencies are largely relegated to the appendices. Specifically, the fact that inference terminates after the first accepted lesson—enforcing a hard cap of at most one finding modification per report—is only clarified in passing in the limitations and appendix. Additionally, the reliance on external human-annotated bounding boxes (Chest ImaGenome) for lesson creation is not discussed in the main text's description of visual memory consolidation.

Tha paper is partially informative.The core conceptual mechanism essentially functions as an external, retrieval-based rule engine that patches draft reports rather than a principled representation-learning or agentic reasoning paradigm.

Critical strengths

The modular design of visual lesson (i.e., pairing textual conditions with concrete image exemplars and maintaining explicit pointers back to historical source episodes) seems to provide auditability. As shown in Appendix D, when a model output changes, clinicians or system designers can trace the causal lesson, inspect the source cases, and may try to verify via counterfactuals whether removing the lesson reverts the decision.

The empirical evaluation is rather good: specific dataset generalization (MIMIC-CXR to IU-Xray without retraining), cross-framework compatibility (evaluating ReAct, ReWOO, Plan-and-Solve, and Swin-Qwen3), and cross-model capacity scaling (from 4B to 122B parameters), with 5,000-draw bootstrap confidence intervals and paired difference testing.

Critical weaknesses
  1. Practical limitation During inference, lessons are tested sequentially, and the first revision that alters the target finding becomes the final report, immediately terminating further modifications. This appears to be a rather unjustified choice. In clinical practice, abnormal chest radiographs frequently exhibit multiple co-occurring pathologies (e.g., cardiomegaly accompanied by pulmonary edema and bilateral pleural effusions). Restricting the system to a single lesson modification per examination is a severe limitation to accept the proposal as an effective one, making the system unable of handling complex, multi-pathology cases.

  2. Computational overhead To draft a report and test up to 3 multimodal lessons, the system requires up to 4 sequential full-context MLLM passes (each ingesting high-resolution patient radiographs, historical lesson exemplars, and past trajectory observations). This produce a huge increase in inference latency and memory occupation to edit at most one clinical sentence. In my opinion this is computationally impractical for clinical deployment compared to dedicated end-to-end report generators.

  3. Report generation quality: The base generator operates with somethinga very low generation baseline; the paper reports BLEU-4 of 4.06 on MIMIC-CXR, whereas standard report generation literature usually reports much hjigher BLEU-4 scores (Chen et al., 2020, Chen et al., 2021,Nicolson et al., 2022). The lexical metrics actually degrade or stagnate when lessons are added (BLEU-4 drops from 4.06 to 3.83 on MIMIC-CXR, and drops from 8.07 to 7.85 on IU-Xray; ROUGE-L and METEOR remain flat). The framework effectively seems to act as an expensive heuristic post-processing patch on top of an under-trained base model.

Chen et al. 2020. https://doi.org/10.48550/arxiv.2010.16056 Chen et al. 2021. https://doi.org/10.18653/v1/2021.acl-long.459 Nicolson et al. 2022. https://doi.org/10.1016/j.artmed.2023.102633

  1. Unvalidated LLM evaluation The primary metric where gains are reported (FBA) is computed by querying another large language model (GLM-4.6V-Flash) to extract 27 disease labels from generated reports. The authors concede in the limitations that "no radiologist has reviewed the generated reports, so the clinical value of the corrections is not established". Relying exclusively on an LLM parser to evaluate reports edited by an LLM-driven rule engine introduces severe risk of metric circularity, superficial keyword matching and clinical inadequacy (no plausibility).

  2. Dependence on ground-truth bounding boxes The visual memory consolidation is not autonomous. The lesson writer does not learn visual grounding or predict coordinates; it merely selects from a predefined menu of ground-truth bounding boxes provided by the Chest ImaGenome dataset. This assumes a heavy reliance on labor-intensive expert annotations, making the memory consolidation pipeline quite strict to X-rays and non-transferable to other modalities (e.g., CT, MRI) or institutions lacking pre-computed anatomical segmentations.

  3. Static store and detrimental or unusual lessons Since the lesson store is static, actively detrimental lessons remain in the library indefinitely, potentially hallucinating non-existent findings on patients with clear radiographs. The lack of an automated validation-based filtering, merging, or pruning mechanism poses serious clinical risks (see for instance case 2 in Appendix D).

Questions for authors
  1. Given that inference terminates after the first accepted lesson, how can the framework be adapted to correct multiple independent diagnostic omissions or false positives on complex studies without combinatorial explosion or conflicting edits?

  2. Have any subset of the generated reports, particularly the cases where lessons overrode the initial draft, been blindly evaluated by certified board-certified radiologists to verify clinical accuracy beyond automated LLM extraction?

  3. If Chest ImaGenome bounding box annotations were absent, how would the visual lesson writer construct localized image evidence? What happens to performance if the agent must generate its own bounding boxes?

  4. Why is there no validation-split filtering mechanism that automatically purges or refines lessons that demonstrate net-negative utility (such as the effusion lesson in Case 2 that produced 8 false positives and zero true positives) before deployment?

  5. What is the latency and total token consumption of the full multi-call retrieval and verification pipeline per study compared to single-pass baseline generators such as MAIRA-2 or CURV?

7 comments

We sincerely thank the reviewer for the constructive feedback.

Summary of changes. To address the reviewer's concerns, we conducted additional experiments on (1) sequential application and combination of multiple lessons, (2) inference latency and token consumption, (3) lesson construction without Chest ImaGenome bounding boxes, and (4) validation-based filtering of harmful lessons, and (5) a blinded clinical evaluation by a board-certified radiologist. We also clarify the operational constraints and the role of visual memory in the main text.

Presentation clarification. The original main text already describes the use of Chest ImaGenome boxes for lesson construction (§3.2) and the first-accepted-lesson stopping rule (§3.3; Figure 2). We will make these operational details more prominent, clarify that the stopping rule limits accepted lesson applications rather than the number of corrected findings, and further explain the role of visual memory.

We address the reviewer's specific points below.

W1 + Q1

One accepted lesson does not limit a report to one corrected finding. Each revision generates a complete replacement report, and the reward accounts for changes to both target and non-target findings (§§3.3–3.4; Appendix B.4).

To address the reviewer's concern, we evaluated sequential continuation and two approaches to combining up to three lessons, using the same base policy, initial drafts, observations, and lesson store.

Combining multiple lessons. We present up to three retrieved lessons and their historical images together in a single prompt. Within the same generation, the model first examines each lesson's conditions, visual cues, and exceptions, deciding whether to apply or ignore it. It then combines the applicable corrections into one complete report, revising potentially conflicting statements while preserving unrelated findings.

We evaluated the following approaches:

  • Sequential continuation: Allow up to three accepted lessons, applying them successively while preventing reversals of previously accepted target decisions.

  • Simple multi-lesson prompt: Present the original top-three retrieved lessons together and directly request one revised report, without explicit per-lesson applicability assessment.

  • Multi-lesson assessment + report generation: Select up to three lessons targeting distinct findings, skipping lessons whose suggested corrections are already reflected in the initial draft. The model determines which lessons apply and combines their corrections into one report.

In both multi-lesson approaches, the model generates a single report proposal by combining the retrieved lessons. The assessment-based approach can incorporate any applicable subset of the retrieved lessons. Its intermediate assessment is excluded from the final report. The proposal is accepted if at least one lesson's target finding changes in its suggested direction; otherwise, the initial draft is retained.

We evaluated these approaches on 1,000 test studies, including 461 studies with at least two initial-draft errors.

Table 1. Sequential and combined application of multiple lessons.

MethodRadGraph ↑CheXbert ↑GritLM ↑FBA ↑FBA on multi-error studies ↑
Original: stop after first acceptance18.5653.7965.1467.6961.38
Sequential: up to three acceptances18.5553.7965.1767.7461.43
Simple multi-lesson prompt18.6652.9465.0166.9160.23
Multi-lesson assessment + report generation18.3854.2365.1868.6162.78

Simply presenting multiple lessons together does not improve FBA. In contrast, assessing and combining applicable lessons improves FBA from 67.69 to 68.61 overall and from 61.38 to 62.78 on studies with multiple initial-draft errors, compared with the original procedure. CheXbert also improves from 53.79 to 54.23.

These results demonstrate that the framework can jointly assess and combine multiple applicable lesson corrections, improving performance on complex studies without enumerating lesson combinations.

W2+Q5

We measured inference latency and token consumption on 100 studies using a single A100-80GB GPU. We report the complete pipeline and its individual stages.

Table 2. Inference latency and token consumption

MethodTime (seconds)Total tokens
Agent without memory4.986,374.36
Agent + lessons21.6017,605.83
MAIRA-23.161,562.26
CURV4.07832.29

Values are averages per study. Total tokens include image and text inputs, generated outputs, and retrieval inputs across all calls.

To explain the additional computation, we measured the contribution of each stage.

Table 3. Cost breakdown of the lesson-guided pipeline.

StageTime (seconds)Total tokens
Visual inspection and reasoning3.804,784.10
Initial report generation1.181,590.26
Lesson retrieval0.30898.27
Lesson-conditioned revision3.957,484.26
Report editing and other processing0.96471.25
Acceptance checks11.412,377.69
Total21.6017,605.83

Where the computation is spent. Visual inspection and reasoning take 3.80 seconds, lesson-conditioned revision takes 3.95 seconds, and acceptance checks take 11.41 seconds. These checks extract finding labels from the draft and proposed report to determine whether the target finding changes in the lesson’s suggested direction (§3.3).

Accuracy and traceability. Adding lessons improves FBA by 2.90 points, CheXbert by 5.20, and RadGraph by 1.80 over the no-memory agent. These results quantify the computational cost and finding-level accuracy tradeoff of lesson-guided revision. In addition to the final report, the system records image inspections and intermediate observations, and links lesson-based revisions to textual guidance and marked historical images. These records allow radiologists to inspect which image regions the agent examined and which lesson evidence accompanied a proposed correction, supporting report review and error analysis.

W3

The BLEU-4 scores reported in Chen et al. (2020, 2021) and Nicolson et al. (2023) are not directly comparable with ours due to differences in report formatting, evaluation, and training. Nicolson et al. (2023) follow Chen et al. (2020), using lowercased reports with special characters removed and a 60-word limit, while Chen et al. (2021) evaluate only the Findings section. In contrast, our evaluation joins the Findings and Impression sections and computes per-report NLTK sentence BLEU (§4.1, Appendices B.1 and B.7). Under our common evaluation protocol, the no-memory agent obtains BLEU-4 scores comparable to MAIRA-2 and CURV: 4.06 vs. 3.95 and 3.90 on MIMIC-CXR, and 8.07 vs. 6.47 and 7.47 on IU-Xray (Table 1 in the paper).

Furthermore, the cited methods are trained directly on paired images and reference reports from the target datasets, whereas our base agent is not trained to generate reports. Our MIMIC-CXR lesson store also transfers to IU-Xray unchanged, improving FBA by 2.39, CheXbert by 2.47, and RadGraph by 2.39 points, without modification (§4.3).

Our primary objective is to improve the correctness of clinical findings rather than reference wording. BLEU measures lexical overlap, which does not necessarily reflect diagnostic accuracy, as also discussed by Nicolson et al. (2023). Adding lessons changes BLEU-4 by only −0.23 points on MIMIC-CXR and −0.22 on IU-Xray, while improving all four clinical and semantic metrics on both datasets, with every 95% paired interval excluding zero (Tables 1 and 5 in the paper). With policy training, BLEU-4 further increases to 4.34, alongside 69.97 FBA and 56.84 CheXbert (Table 3 in the paper).

W4+Q2

Beyond FBA, our evaluation includes non-LLM CheXbert and RadGraph, which provide independent measures of clinical-content agreement. The original paper also reports 94.7% agreement and κ = 0.89 between GLM extraction and CAD-Chest on MIMIC-CXR reference reports.

To further assess clinical accuracy, we invited a board-certified radiologist to conduct a blinded evaluation of 20 randomly selected studies in which lessons modified the initial drafts. The radiologist examined the chest radiographs and complete reports, judged which report was clinically better, and assessed whether the preferred report contained important errors or omissions.

Table 4. Radiologist Evaluation Result.

Clinical preferenceCases
Lesson-revised report preferred12
Initial draft preferred3
Clinically equivalent5

The radiologist preferred the revised report in 12 of 20 cases overall (60%), and 12 of 15 cases (80%) among those with a clear preference. Among the 12 preferred revised reports, 8 had no important residual errors or omissions, while 2 had important errors and 2 were uncertain.

These results provide direct clinical assessment of lesson-based revisions beyond automated metrics, supporting the clinical value of our proposed method.

W5+Q3

We reconstructed the lesson store using three alternatives to supplied Chest ImaGenome boxes: RGRG-predicted boxes [1], no boxes, and coordinates predicted directly by the agent.

Table 5. Reconstruct lesson store using alternative bounding boxes.

Localization used to construct lessonsRetained lessonsRadGraphCheXbertGritLMFBA
Original supplied boxes7618.5653.7965.1467.69
RGRG-predicted boxes8318.4554.1764.9567.78
No boxes4818.4352.3064.9066.68
Agent-predicted boxes5618.4952.7464.9267.01

Using the released RGRG detector achieves 67.78 FBA, compared with 67.69 using supplied boxes. When the agent generates its own boxes, FBA reaches 67.01, while construction without boxes achieves 66.68. Both remain above the no-memory baseline of 64.79. These results show that useful lessons can be constructed without supplied bounding boxes.

Furthermore, bounding-box annotations are not required for new patients when applying the lesson store. The original MIMIC-CXR store already transfers to IU-Xray unchanged (§4.3), demonstrating reuse across chest X-ray datasets without target-patient box annotations.

[1] Tanida et al., Interactive and Explainable Region-Guided Radiology Report Generation, CVPR 2023.

W6+Q4

As discussed in §5, the original study keeps the lesson store fixed during evaluation and identifies merging or retiring harmful lessons as future work. Case 2 illustrates that a lesson may not be identified as harmful on its held-out check cases but can still introduce errors on the test set. This motivates evaluating lessons on additional held-out cases to identify harmful behavior before deployment.

We therefore evaluated automatic validation-based filtering on 1,000 randomly sampled held-out studies, with no patient overlap with construction, training, or testing. For each study, we evaluated its top-three retrieved lessons independently against the same initial draft, producing 3,000 lesson–case evaluations covering all 76 lessons. We removed lessons whose introduced errors exceeded their corrections on validation cases, retaining 64 of 76 lessons. Both stores were then compared on the same test set using the base policy and common initial drafts.

Table 6. Effect of validation-based lesson filtering.

OutcomeOriginal 76 lessonsFiltered 64 lessons
Incorrect labels corrected225248
Newly introduced errors123136
Net corrections102112
RadGraph ↑18.5618.58
CheXbert ↑53.7953.97
GritLM ↑65.1465.16
FBA ↑67.6967.82

The filter removed 12 lessons, including six add-effusion lessons. One removed effusion lesson produced no corrections and four false positives on validation cases. These results demonstrate that the procedure can identify the type of harmful behavior highlighted by the reviewer. With the retrieval budget fixed at K=3K=3, filtering changes which lessons are retrieved. Corrected findings increase from 225 to 248, alongside more introduced errors (123→136), while net corrections improve from 102 to 112.

These results demonstrate an automatic mechanism for identifying and removing lessons with negative validation outcomes while maintaining overall report-generation performance. Further improvements to lesson validation and refinement remain directions for future work.

Comments on the paper

1 comment

We sincerely thank the reviewers and the Area Chair for their time and constructive feedback, which has been invaluable in strengthening our paper.

The main concerns focused on lesson construction and reliability, multiple-finding correction, clinical evaluation, and computational cost. We addressed these through additional experiments and clarifications:

  1. Lesson Construction and Memory Design (34EK, ryGM, sU9q): We clarified the lesson construction procedure and analyzed lesson coverage, allocation, and retrieval budget. We also conducted new ablations to isolate the contributions of multi-case consolidation, applicability conditions, and visual region annotations. Furthermore, we reconstructed lessons using predicted bounding boxes or no boxes, demonstrating that useful lessons can be constructed without supplied anatomical bounding boxes.

  2. Lesson Reliability and Validation (34EK, sU9q): We evaluated lessons on opposite-label cases and introduced validation-based filtering to identify harmful lessons. We further isolated the contribution of policy training using fixed initial drafts and retrieved lessons, providing stronger evidence for the effectiveness of learning to use memory.

  3. Multiple-Finding Correction (sU9q): We evaluated sequential and combined application of multiple lessons. In response to the reviewer's follow-up, we conducted additional experiments distinguishing lesson combination from single-lesson selection, demonstrating that combining lessons improves finding-level accuracy and enables more corrections within the same report. We also analyzed omission and false-positive corrections separately, clarifying the specificity and RadGraph trade-offs.

  4. Clinical Evaluation and Computational Cost (sU9q): We conducted a blinded evaluation by a board-certified radiologist, providing direct clinical assessment beyond automated metrics. We also measured inference latency and token consumption, analyzed computational costs, and clarified the interpretation of lexical metrics and cross-dataset generalization.

We are encouraged by the reviewers' recognition of the value of consolidated visual memory, its auditability, and its generalization across datasets, agent frameworks, and model sizes. We also appreciate their continued engagement, particularly the follow-up questions that motivated more experiments.

We will incorporate these new results and clarifications into the revised manuscript. We believe the additional evidence further strengthens the paper's contributions to multimodal agent memory and its application to chest X-ray report generation.

We would be happy to clarify any remaining questions from the reviewers or Area Chair. Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.