- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the use of multimodal large language model (MLLM) episodic memory towards enhancing report generation quality for chest X-rays (CXR), after initial organization into a compact set of multimodal lessons that are retrieved by case context (keys). A compact set of 76 lessons was found to improve finding-level balanced accuracy compared to using the original set of 8192 episodes for retrieval, on MIMIC-CXR. The lessons were moreover found to generalize to other datasets, agents and models, which increases its practical significance.
Critical strengths
- Valuable insight into the potential of using consolidated lessons instead of raw cases as the retrieval target for MLLMs
Critical weaknesses
- Design choices for constructing the lesson store are not well-described, despite the lesson store being the major product
- Uncertain as to whether the increased performance is due to the lesson paradigm, or simply because more data is effectively retrieved with (consolidated) lessons than individual cases
Questions for authors
-
In Section 3.2, a definition of each (visual) lesson is provided in Equation 2. It might be considered to indicate the variables , and with reference to concrete examples, such as the lesson traces in Appendix D Figure 3.
-
In Section 3.2, is stated to store the lesson's writing cases and outcomes on its check cases. This might be further explained, especially regarding the outcomes - possibly again with concrete examples, and in terms of and other variables later referenced in Section 3.3.
-
In Section 3.3, it is stated that "Matching the current case to the cases behind a lesson, rather than to the lesson's text, retrieves lessons by the situation in which the error occurred". However, the cases within a lesson appear to be generated from groups of 16 cases (8 writing cases; from Appendix A), with no particular guarantee as to the conditions and situations covered by the writing cases. Given that MIMIC-CXR contains 14 standard classes, it is unclear as to whether 76 lessons of three cases/images each would sufficiently or appropriately represent each class. This might be carefully discussed, also with respect to the major issue on the lesson construction protocol.
-
In Section 4, five questions (RQ1 to RQ5) are declared and addressed through experiments. However, the optimal construction procedure of the lesson store from the original reports does not appear fully addressed in Section 3.2 (or Appendix A), which presents the procedure without either theoretical or empirical basis - and which does not appear to be reflected in the RQs either (Section 3.4 then describes the optimization of lesson usage, assuming a fixed set of lessons). In particular, it is not clear as to whether lessons are generated by uniform sampling over cases, by assigning a fixed number of lessons per context or other feature (in which case the number would be a relevant tunable parameter), etc.
-
In Section 4.1, episodic memory is stated to retrieve the most similar cases from the 8192 episodes with the same policy, prompt structure and acceptance rule as a lesson. However, each lesson appears to include far more independent episodes (3, constructed from a group of 16 cases from Appendix A), in which case it is unclear as to whether the performance gain is from lessons per se, or from the additional data (9 cases, compared to 3). This might be justified, along with whether a comparable amount of data is also retrieved for ExpeL and ViLoMem, since Appendix B.2 suggest that ViLoMem also retrieves 3 cases only.
-
Related to the above, for lesson/case retrieval is a pertinent parameter that might be ablated on, to quantify the value of additional cases for performance.
-
In Section 4.2 relating to RQ1, some of the result descriptions do not appear readily referencable in Table 1. For example, it is stated that "Retrieval over all 8,192 source episodes raises FBA by 1.07 and CheXbert by 2.20 points and lowers RadGraph by 0.70", but it is not clear as to whether this refers to episodic memory, and might be clarified.
3 comments
Q4
Lesson construction. We identify reporting errors by comparing the findings in the base agent’s initial drafts with their corresponding reference reports across 8,192 MIMIC-CXR episodes. Each error specifies a target finding and correction direction: add when the reference reports a finding that the draft omits or denies, and remove when the draft asserts or hedges a finding that the reference doesn't contain. We then group cases requiring the same correction, such as add edema or remove pleural effusion (§3.2). Within each error type, we randomly select candidate cases using a fixed seed and form groups of 16 distinct patients: eight writing cases and eight reserved check cases, with no patient reused across groups. The number of candidate groups is limited separately for each error type. The writer consolidates the writing cases into textual guidance and one to three historical images. After visual-support and lesson-usability checks, 76 lessons are retained.
Lessons per error type. To address the reviewer’s question about lesson allocation, we evaluated how performance changes when retaining different numbers of lessons per finding–direction pair. We compared stores retaining at most one, two, or all available lessons per finding–direction pair, resulting in 8, 16, and 76 lessons, respectively. All settings use the same initial drafts, base policy, and retrieval budget (K=3).
Table 2. Effect of lessons per error type.
| Lessons in store | RadGraph ↑ | CheXbert ↑ | GritLM ↑ | FBA ↑ |
|---|---|---|---|---|
| 8 | 17.26 | 49.41 | 65.01 | 65.69 |
| 16 | 17.12 | 50.79 | 65.87 | 66.06 |
| 76 — original | 18.56 | 53.79 | 65.14 | 67.69 |
Increasing the store from 8 to 16 and 76 lessons improves FBA from 65.69 to 66.06 and 67.69, respectively. The original store also achieves the highest CheXbert and RadGraph scores. These results support the benefit of retaining multiple lessons per error type.
Q5
We first clarify that each lesson is constructed from eight writing cases, while the other eight cases are used for checking. At inference, each revision call receives one lesson with one to three historical images. With , nine images is the maximum across three separate attempts, not the input to a single revision call.
To isolate the contribution of consolidation, we replaced each lesson's consolidated text with its eight writing-case grounding notes presented together, keeping the historical images and boxes, initial drafts, retrieval order, revision policy, and acceptance procedure unchanged.
Table 3. Effect of multi-case consolidation.
| Metric ↑ | Consolidated lessons | Eight writing case | Difference |
|---|---|---|---|
| RadGraph | 18.56 | 16.1 | +2.46 |
| CheXbert | 53.79 | 50.59 | +3.2 |
| GritLM | 65.14 | 65.11 | +0.03 |
| FBA | 67.69 | 66.5 | +1.19 |
Consolidation improves FBA by 1.19 points, CheXbert by 3.20, and RadGraph by 2.46 over presenting the same underlying case records without consolidation. This demonstrates the benefit of organizing historical experience into lessons beyond providing additional case information.
For the other memory baselines, episodic memory retrieves three individual cases from the 8,192 episodes and attempts them separately. ExpeL provides six global rules and two retrieved reference reports, while ViLoMem retrieves three logical and three visual guidelines. Both ExpeL and ViLoMem are constructed from the same 608 writing cases as our lessons and use one revision call (Appendix B.2).
Q6
We evaluated the effect of the maximum number of retrieved lessons , keeping the lesson store, initial drafts, and base policy unchanged. Inference stops at the first accepted revision or after examining all candidates.
Table 4. Effect of retrieval budget .
| RadGraph ↑ | CheXbert ↑ | GritLM ↑ | FBA ↑ | |
|---|---|---|---|---|
| 1 | 18.35 | 52.60 | 65.03 | 66.99 |
| 2 | 18.58 | 53.36 | 65.09 | 67.13 |
| 3 — default | 18.56 | 53.79 | 65.14 | 67.69 |
| 5 | 18.52 | 54.33 | 65.19 | 68.18 |
Increasing from 1 to 3 improves FBA by 0.70 points and CheXbert by 1.19 points. Increasing further to 5 yields additional gains of 0.49 FBA and 0.54 CheXbert, while RadGraph remains similar. These results show that retrieving additional candidate lessons can further improve finding-level performance.
Q7
The statement refers to the “+ episodic memory” row in Table 1 of the submitted paper, compared with the “Agent, no memory” row. Retrieving the top (K=3) cases from the 8,192 source episodes increases FBA from 64.79 to 65.86 (+1.07) and CheXbert from 48.59 to 50.79 (+2.20), while RadGraph decreases from 16.76 to 16.06 (−0.70). We will clarify this baseline in §4.2.
We thank the authors for the clarifications.
We thank the reviewer for their positive assessment and constructive suggestions.
Summary of changes. We conducted additional experiments to evaluate (1) lessons per error type, (2) multi-case consolidation versus separate case records, and (3) the number of retrieved lessons. We also analyzed lesson coverage and clarified the lesson components, construction procedure, retrieval mechanism, and baseline comparisons. We will incorporate these clarifications and experimental results into the revised paper.
We address the reviewer's specific points below.
Q1-Q2
We clarify the components of a visual lesson, γ=(Lγ,Vγ,Aγ), using the “add edema” example in Appendix D, Figure 3:
For each check case, we apply the lesson to its saved context ξi=(xi,τi,di) following Eq. 3. We compare the returned report with the initial draft di, using the reference yi∗ to count corrected findings and introduced errors (Eq. 4). For example, correctly adding a missing edema finding counts as a correction, while introducing an incorrect finding elsewhere counts as an introduced error.
During inference, the reporting policy receives Lγ and Vγ; Aγ is retained for inspection and traceability.
Q3
The lesson store is designed to address recurring reporting errors, rather than provide a fixed number of lessons for every diagnostic label. Errors are assessed across the finding vocabulary, and selected cases are grouped by target finding and correction direction. A finding being present in a case does not necessarily mean that the base agent reported it incorrectly.
Each lesson is written from eight writing cases, with eight additional cases reserved for checking. The writer consolidates the writing-case evidence into a condition, reporting decision, visual cue, and exception. The one to three retained historical images illustrate this guidance; they are not the only cases used to construct the lesson.
Table 1. Findings in writing cases and retained lesson targets.
The table distinguishes findings present in historical cases from the findings targeted by retained lessons. For example, cardiomegaly appears in 236 writing cases, although no retained lesson explicitly targets it. Lesson targets depend on the selected reporting errors and available visual support. Consequently, a zero lesson count does not establish that the base agent makes no errors on that finding.
Retrieval ranks lessons by similarity to their writing-case contexts, rather than matching lesson text. The reporting policy then considers the lesson’s condition, visual cue, and exception against the current patient’s images to decide whether to apply it (§3.3). The construction procedure and the effect of retaining multiple lessons per error type are addressed in Q4.