- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MedMME, a benchmark for evaluating long-term memory of multimodal medical agents, covering two settings: multimodal clinical dialogues generated from longitudinal patient records, and surgical or endoscopic procedure videos. The benchmark contains about 28K questions, and provides gold memory to evaluate memory extraction, updating, and retrieval. Experiments on long-context models and nine memory mechanisms show that long-context models perform best overall, while memory methods degrade more slowly as sessions accumulate, and memory updating remains a common weakness.
Critical strengths
- The benchmark covers a wide range of clinical domains and modalities, including 2D images, 3D volumes, and procedure videos, built from real longitudinal data at a large scale.
- The gold memory annotations allow memory extraction, updating, and retrieval to be evaluated separately rather than only through final QA accuracy, which helps analyse why memory systems fail.
Critical weaknesses
- The necessity of long-term memory in clinical practice is not demonstrated. A simple baseline that summarizes each session and uses the summary in subsequent sessions is not compared, so it is unclear whether long-term memory mechanisms are needed. In addition, each memory stream is dedicated to a single patient, which seems impractical in real scenarios, where a model should handle all patients rather than allocating a separate model to each.
- The reliability of the gold memory is not well established. The LLM used to generate it is not specified, and the expert review only checks consistency with the source data, not completeness or clinical relevance.
- The evaluation of multimodal understanding is limited. The model's ability to read findings from images or video frames is not evaluated.
- Only the QA judge is validated against physicians. The extraction, update, and retrieval judges lack human validation.
- The precision metric does not take redundancy into account. As long as the output covers any gold fact, it is labeled as supporting.
Questions for authors
- How would the models perform if multiple patients shared one memory?
- More details on the gold memory construction are needed. Why is an LLM needed to construct the gold memory rather than deriving it directly from the related evidence? Which LLM is used? How does the gold memory differ from the candidate facts obtained from the evidence? How is the gold memory reviewed by experts?
- Image memory is scored only on file retention. How would the extraction results change if the images were excluded but the memory contained the critical findings from the images?
11 comments
Rebuttal by Authors [2/3]
(W2 & Q2) Details of the Construction & Expert Review of the Clinical Gold Memory
Gold Memory Construction. We thank the reviewer for raising this important point. The raw clinical evidence contains information without clear clinical significance, such as acquisition device models and acquisition parameters. Therefore, drawing on the organization of patient summaries in HL7 IPS, we require each gold memory point to be supported by at least one source clinical evidence item. All gold memory points are constructed by GPT-5.6 Luna. Text memory points consolidate evidence items that convey the same clinical meaning, whereas image-grounded memory points retain only the visual evidence associated with examinations, diagnostic findings, and outcomes.
Expert Review. Physicians review the gold memory points on a web-based annotation platform. The platform displays each gold memory point together with its linked dialogue turns and clinical evidence. Physicians then determine whether its content contains unsupported claims or factual contradictions. The platform also displays the associated earlier memory points, so that physicians can assess whether an updated memory point is clinically meaningful. We further provide screenshots of the review interface in Appendix B.8, showing the linked clinical evidence and the update memory points. As described in Sec. 2.4, initial review is conducted by ten physicians, and the reviewed and revised dialogues and gold memory points are then checked by one of three senior physicians. During this process, 1.9% of the gold memory points and dialogues are revised. We categorize the revised gold memory points and find that 526 are related to completeness and 112 to clinical relevance.
(W3 & Q3) Evaluation of Visual Information in Memory
Thank you for the thoughtful question. To further assess the contribution of visual information, we compare three memory settings: text-only memory, text memory with metadata, and text memory with images. The metadata contains cues about the images, such as session attribution, timestamps, and lesion identifiers. As shown in Table 3, text memory with images outperforms text memory with metadata across all three methods. This might be attributed to the more intuitive and richer multimodal cues provided by image memory, which help the model locate and utilize the relevant clinical evidence.
Table 3: Performance comparison of different methods under three memory settings.
| Method | Acc (text) | Acc (text + meta) | Acc (text + image) |
|---|---|---|---|
| UniversalRAG | 28.8 | 32.7 | 40.0 |
| Augustus | 16.5 | 18.6 | 24.2 |
| Omni-SimpleMem | 21.7 | 25.4 | 33.1 |
Rebuttal by Authors [3/3]
(W4) Human Verification of Memory Operation Judgments
We thank the reviewer for pointing this out. We further recruit physicians to score the extraction, updating, and retrieval of the corresponding questions on 100 sessions using the same evaluation criteria. As shown in Table 4, agreement between the LLM judge and expert consensus scores is high. The Cohen’s κ of the four methods across the three memory operations all exceed 0.86, supporting the reliability of automated scoring on the reviewed sample. We have mentioned this in Line 261-262 of the manuscript and added Table 4 to Appendix B.3.
Table 4: Agreement between the LLM judge and physician consensus on memory operations, measured by Cohen’s κ.
| Method | Extraction | Update | Retrieval |
|---|---|---|---|
| WeMM-Embedding | 0.887 | 0.894 | 0.881 |
| RAG-Anything | 0.868 | 0.905 | 0.875 |
| MIRIX | 0.935 | 0.907 | 0.880 |
| Omni-SimpleMem | 0.914 | 0.896 | 0.902 |
(W5) Memory Entry Precision Analysis
Thank you for your insightful comments. We initially compute precision over memory entries, as they are the native unit that each method stores and retrieves. We concatenate semantically similar memory entries in groups of 2, 4, and 8 and re-evaluate precision. As shown in Table 5, concatenation raises entry-level precision to some extent, but the gain is limited as a supporting entry requires all claims within it to be evidence-supported. For instance, the precision of MIRIX and Omni-SimpleMem drops when 8 entries are concatenated, since lengthy entries are more likely to contain unsupported claims. We further decompose memory entries into factual claims and compute the fraction of claims that correspond to gold memories (last column of Table 5). The factual-claim-level precision is close to entry-level precision and yields a consistent ranking. We have added the computation of factual-claim precision in Lines 1004–1010 and reported both types of precision in Table 4 of the manuscript.
Table 5: Comparison of precision under different semantic concatenation setting, along with factual-claim precision.
| Method | Entry-level Precision | Concat 2 | Concat 4 | Concat 8 | Factual-claim precision |
|---|---|---|---|---|---|
| RAG-Anything | 69.1 | 70.9 | 72.3 | 73.2 | 67.8 |
| MIRIX | 67.1 | 70.6 | 71.8 | 69.0 | 65.9 |
| MMA | 30.6 | 38.9 | 45.2 | 49.9 | 26.7 |
| Omni-SimpleMem | 66.3 | 69.1 | 69.8 | 67.3 | 65.6 |
Rebuttal by Authors [1/3]
We sincerely thank the reviewer for recognizing our broad coverage of clinical domains and modalities, our large-scale benchmark with real longitudinal data, and our gold memory annotations enable memory failure analysis. We sincerely hope that our responses below address your concerns.
(W1 & Q1) Comparison of Session Summary Baseline & Evaluation of Multiple Patient Memory
Session Summary Baseline. Thank you for your valuable suggestions. We have added a session-summary baseline that uses the same GPT-5.4 Nano backbone. At the end of each session, the model generates a summary of that session, and these summaries are used in subsequent sessions. As shown in Table 1, the session-summary baseline underperforms Omni-SimpleMem on all question types, with a particularly pronounced gap on multimodal questions. A session summary is a simple compression of the content and tends to lose key clinical information in longitudinal care and fine-grained visual evidence. In contrast, memory methods organize multimodal memory in a structured manner and continuously maintain it through explicit storage, updating, and retrieval mechanisms.
Multiple Patient Memory. We agree that a model needs to handle multiple patients in real scenarios. We would like to clarify that, in memory-augmented agents, the same backbone model serves all patients while each patient has an independent memory, which is consistent with how electronic health records are organized per patient. Following your suggestion, we further evaluate the setting where multiple patients share one memory. As shown in Table 2, the performance of all methods drops substantially when the information of N patients is written into the same memory. These results suggest that existing memory mechanisms cannot yet effectively store information from multiple patients simultaneously.
Table 1: Performance comparison with the session-summary baseline across QA families. Text: text-only questions; MM: multimodal questions.
Table 2: Performance comparison of different memory methods under shared memory across N patients.