acceptodds
Under review as a conference paper at ICLR 2027

MedMME: Benchmarking Long-Horizon Medical Memory for Multimodal Agents

16 views

Abstract

Long-term memory is a critical capability for multimodal medical agents to integrate evolving clinical information throughout extended interactions. However, existing medical memory benchmarks cover a limited range of modalities and tasks, failing to fully capture the multimodal and dynamic nature of clinical information. To bridge this gap, we introduce MedMME, a benchmark for evaluating long-horizon medical memory in multimodal agents. MedMME integrates datasets across 10 clinical domains and 9 medical visual modalities, covering two medical settings: multimodal clinical dialogue and medical procedure videos. We divide clinical trajectories into sessions to evaluate agent memory under a streaming input protocol. The benchmark also introduces physician-validated clinical memory to evaluate memory extraction, updating, and retrieval. We systematically evaluate 19 memory-augmented multimodal agents built on nine memory mechanisms. We find that: (i) agent performance declines as multimodal clinical interactions progress; (ii) existing memory mechanisms perform unevenly across medical visual modalities; and (iii) agents with medically adapted backbones perform better, while backbone models scaling alone yields limited gains. By making these memory capability gaps measurable, MedMME supports the evaluation of future multimodal agents for long-horizon medical interactions.

Primary area:
datasets and benchmarks
Keywords:
Medical Memory, Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WyvDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MedMME, a benchmark for evaluating long-term memory of multimodal medical agents, covering two settings: multimodal clinical dialogues generated from longitudinal patient records, and surgical or endoscopic procedure videos. The benchmark contains about 28K questions, and provides gold memory to evaluate memory extraction, updating, and retrieval. Experiments on long-context models and nine memory mechanisms show that long-context models perform best overall, while memory methods degrade more slowly as sessions accumulate, and memory updating remains a common weakness.

Critical strengths
  1. The benchmark covers a wide range of clinical domains and modalities, including 2D images, 3D volumes, and procedure videos, built from real longitudinal data at a large scale.
  2. The gold memory annotations allow memory extraction, updating, and retrieval to be evaluated separately rather than only through final QA accuracy, which helps analyse why memory systems fail.
Critical weaknesses
  1. The necessity of long-term memory in clinical practice is not demonstrated. A simple baseline that summarizes each session and uses the summary in subsequent sessions is not compared, so it is unclear whether long-term memory mechanisms are needed. In addition, each memory stream is dedicated to a single patient, which seems impractical in real scenarios, where a model should handle all patients rather than allocating a separate model to each.
  2. The reliability of the gold memory is not well established. The LLM used to generate it is not specified, and the expert review only checks consistency with the source data, not completeness or clinical relevance.
  3. The evaluation of multimodal understanding is limited. The model's ability to read findings from images or video frames is not evaluated.
  4. Only the QA judge is validated against physicians. The extraction, update, and retrieval judges lack human validation.
  5. The precision metric does not take redundancy into account. As long as the output covers any gold fact, it is labeled as supporting.
Questions for authors
  1. How would the models perform if multiple patients shared one memory?
  2. More details on the gold memory construction are needed. Why is an LLM needed to construct the gold memory rather than deriving it directly from the related evidence? Which LLM is used? How does the gold memory differ from the candidate facts obtained from the evidence? How is the gold memory reviewed by experts?
  3. Image memory is scored only on file retention. How would the extraction results change if the images were excluded but the memory contained the critical findings from the images?
11 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

We sincerely thank the reviewer for recognizing our broad coverage of clinical domains and modalities, our large-scale benchmark with real longitudinal data, and our gold memory annotations enable memory failure analysis. We sincerely hope that our responses below address your concerns.

(W1 & Q1) Comparison of Session Summary Baseline & Evaluation of Multiple Patient Memory

Session Summary Baseline. Thank you for your valuable suggestions. We have added a session-summary baseline that uses the same GPT-5.4 Nano backbone. At the end of each session, the model generates a summary of that session, and these summaries are used in subsequent sessions. As shown in Table 1, the session-summary baseline underperforms Omni-SimpleMem on all question types, with a particularly pronounced gap on multimodal questions. A session summary is a simple compression of the content and tends to lose key clinical information in longitudinal care and fine-grained visual evidence. In contrast, memory methods organize multimodal memory in a structured manner and continuously maintain it through explicit storage, updating, and retrieval mechanisms.

Multiple Patient Memory. We agree that a model needs to handle multiple patients in real scenarios. We would like to clarify that, in memory-augmented agents, the same backbone model serves all patients while each patient has an independent memory, which is consistent with how electronic health records are organized per patient. Following your suggestion, we further evaluate the setting where multiple patients share one memory. As shown in Table 2, the performance of all methods drops substantially when the information of N patients is written into the same memory. These results suggest that existing memory mechanisms cannot yet effectively store information from multiple patients simultaneously.

Table 1: Performance comparison with the session-summary baseline across QA families. Text: text-only questions; MM: multimodal questions.

DialoguesVideos
TextMMTextMM
Methodsgl.upd.cmp.agg.loc.trc.col.sgl.upd.cmp.agg.loc.trc.col.
Long Context84.255.064.545.610.55.58.525.86.032.924.029.649.414.0
Session-Summary49.516.924.213.65.63.94.317.74.513.812.612.117.19.6
Omni-SimpleMem53.418.729.315.128.416.914.326.112.419.026.719.728.119.1

Table 2: Performance comparison of different memory methods under shared memory across N patients.

MethodN=1N=2N=4N=8
WeMM-Embedding40.527.819.513.8
UniversalRAG40.031.321.813.6
M2A24.713.310.47.1
MMA36.720.612.99.3
Augustus24.212.18.25.3
Omni-SimpleMem33.114.410.97.4
AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

(W2 & Q2) Details of the Construction & Expert Review of the Clinical Gold Memory

Gold Memory Construction. We thank the reviewer for raising this important point. The raw clinical evidence contains information without clear clinical significance, such as acquisition device models and acquisition parameters. Therefore, drawing on the organization of patient summaries in HL7 IPS, we require each gold memory point to be supported by at least one source clinical evidence item. All gold memory points are constructed by GPT-5.6 Luna. Text memory points consolidate evidence items that convey the same clinical meaning, whereas image-grounded memory points retain only the visual evidence associated with examinations, diagnostic findings, and outcomes.

Expert Review. Physicians review the gold memory points on a web-based annotation platform. The platform displays each gold memory point together with its linked dialogue turns and clinical evidence. Physicians then determine whether its content contains unsupported claims or factual contradictions. The platform also displays the associated earlier memory points, so that physicians can assess whether an updated memory point is clinically meaningful. We further provide screenshots of the review interface in Appendix B.8, showing the linked clinical evidence and the update memory points. As described in Sec. 2.4, initial review is conducted by ten physicians, and the reviewed and revised dialogues and gold memory points are then checked by one of three senior physicians. During this process, 1.9% of the gold memory points and dialogues are revised. We categorize the revised gold memory points and find that 526 are related to completeness and 112 to clinical relevance.

(W3 & Q3) Evaluation of Visual Information in Memory

Thank you for the thoughtful question. To further assess the contribution of visual information, we compare three memory settings: text-only memory, text memory with metadata, and text memory with images. The metadata contains cues about the images, such as session attribution, timestamps, and lesion identifiers. As shown in Table 3, text memory with images outperforms text memory with metadata across all three methods. This might be attributed to the more intuitive and richer multimodal cues provided by image memory, which help the model locate and utilize the relevant clinical evidence.

Table 3: Performance comparison of different methods under three memory settings.

MethodAcc (text)Acc (text + meta)Acc (text + image)
UniversalRAG28.832.740.0
Augustus16.518.624.2
Omni-SimpleMem21.725.433.1
AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

(W4) Human Verification of Memory Operation Judgments

We thank the reviewer for pointing this out. We further recruit physicians to score the extraction, updating, and retrieval of the corresponding questions on 100 sessions using the same evaluation criteria. As shown in Table 4, agreement between the LLM judge and expert consensus scores is high. The Cohen’s κ of the four methods across the three memory operations all exceed 0.86, supporting the reliability of automated scoring on the reviewed sample. We have mentioned this in Line 261-262 of the manuscript and added Table 4 to Appendix B.3.

Table 4: Agreement between the LLM judge and physician consensus on memory operations, measured by Cohen’s κ.

MethodExtractionUpdateRetrieval
WeMM-Embedding0.8870.8940.881
RAG-Anything0.8680.9050.875
MIRIX0.9350.9070.880
Omni-SimpleMem0.9140.8960.902

(W5) Memory Entry Precision Analysis

Thank you for your insightful comments. We initially compute precision over memory entries, as they are the native unit that each method stores and retrieves. We concatenate semantically similar memory entries in groups of 2, 4, and 8 and re-evaluate precision. As shown in Table 5, concatenation raises entry-level precision to some extent, but the gain is limited as a supporting entry requires all claims within it to be evidence-supported. For instance, the precision of MIRIX and Omni-SimpleMem drops when 8 entries are concatenated, since lengthy entries are more likely to contain unsupported claims. We further decompose memory entries into factual claims and compute the fraction of claims that correspond to gold memories (last column of Table 5). The factual-claim-level precision is close to entry-level precision and yields a consistent ranking. We have added the computation of factual-claim precision in Lines 1004–1010 and reported both types of precision in Table 4 of the manuscript.

Table 5: Comparison of precision under different semantic concatenation setting, along with factual-claim precision.

MethodEntry-level PrecisionConcat 2Concat 4Concat 8Factual-claim precision
RAG-Anything69.170.972.373.267.8
MIRIX67.170.671.869.065.9
MMA30.638.945.249.926.7
Omni-SimpleMem66.369.169.867.365.6
Reviewer_RwfEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedMME, a benchmark for evaluating long-horizon multimodal medical memory in agentic systems. The benchmark combines longitudinal multimodal clinical dialogues and medical procedure videos, covering 15 public datasets, 10 clinical domains, and 9 medical visual modalities. It organizes each clinical trajectory into temporally ordered sessions and evaluates agents under a streaming protocol, where previous raw inputs are unavailable after each session and agents must rely on maintained external memory.

Critical strengths
  1. The paper addresses a practically important and insufficiently studied problem: maintaining patient-specific multimodal evidence over extended clinical interactions. Unlike conventional medical VQA or long-context clinical reasoning benchmarks, MedMME explicitly evaluates memory behavior under a streaming setting in which prior raw observations cannot simply be replayed.

  2. The benchmark scope is broad and clinically diverse, including 10 clinical domains, 9 visual modalities, longitudinal patient records, and procedural videos.

  3. The introduction of physician-verified clinical gold memory is a meaningful design choice. Evaluating extraction, updating, and retrieval separately from QA accuracy can help distinguish whether a failure comes from missing evidence, outdated patient-state representation, irrelevant retrieval, or downstream reasoning.

Critical weaknesses
  1. Could the authors clarify the precise research gap that MedMME isolates beyond existing long-context medical QA benchmarks? In particular, how can the reported gains or degradations be attributed specifically to memory construction, updating, and retrieval rather than to differences in context-window truncation, prompt formatting, or the amount of visual information retained?

  2. Could the authors provide a fully formal definition of the streaming protocol, including the exact input available at session tt, the memory state MtM_t, the memory-update operator Mt=U(Mt−1,Xt)M_t=\mathcal{U}(M_{t-1},X_t), the retrieval operator Rt=R(Mt,qt)R_t=\mathcal{R}(M_t,q_t), and the answer-generation input? The current description states that raw prior inputs are inaccessible, but it remains unclear whether prior textual summaries, image embeddings, image thumbnails, metadata, timestamps, and source annotations remain available.

  3. How do the authors ensure that procedure-video tasks do not leak answers through textual surgical-phase labels, action annotations, frame filenames, timestamps, or session boundaries derived from source annotations? Figure 3(a) shows highly informative phase/action labels such as “Calot triangle dissection” and “Gallbladder extraction,” but the paper does not clearly specify whether these labels are hidden from all evaluated agents.

  4. Could the authors clarify whether the QA pairs are independently validated against source evidence rather than being evaluated mainly through the same gold-memory representation from which they are generated? Since questions are constructed from gold memories and answers are deterministically derived from linked gold memories, please discuss the risk that the benchmark primarily measures conformity to the authors’ memory decomposition rather than clinically meaningful longitudinal reasoning.

Questions for authors
  1. Why is the extraction metric based on memory-entry counts in P=Nsupport⁡/Npred⁡P=N_{\operatorname{support}}/N_{\operatorname{pred}}, rather than factual-claim counts or token-normalized precision? Under the current definition, a method may place many facts into one very long memory entry and obtain high precision as long as the entry supports at least one gold requirement; please analyze whether this creates an incentive for over-aggregated or excessively verbose memory entries.

  2. How is the comparison between long-context systems and memory-augmented systems made computationally fair? Long-context models receive the complete raw history until truncation, whereas memory methods use externally constructed summaries or indexed memories; please report matched context budgets, image/frame budgets, total input tokens, total API calls, latency, and monetary cost for every baseline rather than only for a subset of memory methods in Table 6.

  3. How are the medical-specialization conclusions in Figure 5(a) controlled for confounding factors? Models such as Hulu-Med, HealthGPT Pro, MedGemma, and their general-purpose counterparts may differ not only in medical fine-tuning but also in architecture, visual encoder, training data scale, instruction tuning, and context length; please provide paired comparisons using identical architectures and identical memory mechanisms.

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/4]

We thank the reviewer for the valuable feedback and for recognizing the importance of our evaluated problem, the broad and clinically diverse scope of our benchmark, and the meaningful design of our physician-verified clinical gold memory. We sincerely hope our responses below address your concerns.

(W1) MedMME Positioning Beyond Long-Context QA & Error Analysis.

MedMME Positioning. Thank you for your insightful comments. MedMME focuses on evaluating long-horizon medical memory for multimodal agents. To our knowledge, it is the first multimodal medical memory benchmark that adopts multi-session streaming inputs. It differs from existing long-context medical QA benchmarks in two main aspects. First, long-context benchmarks present the entire input as a single long session together with the question, and the model directly performs localization and reasoning over the complete input. In contrast, MedMME follows a multi-session streaming protocol. Agents must answer questions using maintained multimodal memory, without direct access to raw inputs from previous sessions. Second, long-context medical QA benchmarks only score the final answer. MedMME provides physician-verified clinical gold memory for each session, which specifies the key clinical information and the patient states to retain.

Error analysis. With the clinical gold memory, we can diagnose the overall performance of multimodal agents in memory extraction, updating, and retrieval (Sec. 3.3 in the manuscript). We further use the gold memory to determine whether an individual incorrect answer stems from memory extraction, state updating, retrieval, or subsequent reasoning. As shown in Table 1, we sample 250 failure cases from Augustus and identify the error mode of each case against the gold memory. We have added Table 1 to Appendix B.5.

Table 1: Error mode distribution of 250 failure cases from Augustus.

Error ModeCauseRatio (%)
Extraction ErrorMissing Evidence13.6
Update ErrorOutdated-State Retention6.4
Retrieval ErrorMemory Synthesis Failure38.0
Clinical Misgrounding14.8
Cross-modal Mismatch12.8
Reasoning ErrorFaulty Clinical Inference4.4
Others-10.0
AuthorsOpenReview ↗

Rebuttal by Authors [2/4]

(W2) Formal Definition of the Streaming Protocol

Thank you very much for your thoughtful suggestions. We have added the following definition in Appendix B.2.

Under the streaming protocol, each patient trajectory consists of sessions t=1,…,Tt=1,\dots,T in chronological order, and every memory method follows

M0=∅,Mt=U(Mt−1,Xt),Rt=R(Mt,qt),a^t=G(qt,Iqt,Rt).M_0=\emptyset,\qquad M_t=\mathcal{U}(M_{t-1},X_t),\qquad R_t=\mathcal{R}(M_t,q_t),\qquad \hat a_t=\mathcal{G}(q_t,I_{q_t},R_t).

Session input. XtX_t denotes the content of session tt, and it is the only input from which the memory is built. For clinical dialogues, XtX_t consists of all dialogue turns of session tt in their original order, and each turn contains the speaker, the text, and any images attached to that turn. For procedure videos, XtX_t consists of the frames of session tt in temporal order. XtX_t contains no image captions, source metadata, separate timestamp fields, or source annotations. Turn numbers, timestamps, and record identifiers are assigned during benchmark construction for annotation and physician review only, and they are removed before the dialogue is given to the agent. Clinical temporal information such as the visit day or treatment week is available only when it is stated in the dialogue text itself.

Memory state and update. MtM_t denotes the memory after session tt. The memory is reset at the start of each patient trajectory, so M0M_0 is empty and no information carries over across patients. U\mathcal{U} denotes the memory-update operator of each method. It receives the turns or frames of XtX_t one at a time followed by a session-end signal, and it produces MtM_t from Mt−1M_{t-1} and XtX_t. It can access earlier sessions only through Mt−1M_{t-1}. Anything a method derives from XtX_t, including textual summaries, image embeddings, thumbnails, original images, and identifiers or timestamps generated by the method itself, may be stored in MtM_t and remains available in later sessions only through MtM_t.

Questions and retrieval. qtq_t denotes a question asked after session tt has been processed, and several questions may be asked after the same session. IqtI_{q_t} denotes the query images attached to a multimodal question. R\mathcal{R} denotes the retrieval operator of each method. Its output RtR_t is a ranked list of at most K=10K=10 memory entries retrieved from MtM_t for qtq_t. All methods use the same KK.

Answer generation. G\mathcal{G} denotes the answer generator, and within each comparison, all methods use the same model and prompt. Its input consists only of qtq_t, IqtI_{q_t}, and the text and images in RtR_t. G\mathcal{G} has no direct access to MtM_t, and it never receives the gold memory or source annotations. G\mathcal{G} also never receives any X1,…,XtX_1,\dots,X_t directly, and it accesses earlier content only through the at most KK memory entries in RtR_t. a^t\hat a_t denotes the resulting answer.

Accessibility of prior inputs. By “raw prior inputs are inaccessible” we mean that, for memory methods, once session tt has been processed, none of X1,…,XtX_1,\dots,X_t is provided to the agent again. Prior content reaches later steps only through what U\mathcal{U} stored in MtM_t and what R\mathcal{R} retrieves. Textual summaries, image embeddings, thumbnails and original images are available in later sessions only if the method itself stored them. Metadata, timestamps and source annotations are never provided as input at any session, and the agent builds its memory from the raw dialogue or frames alone. Allowing memory methods to store original images is deliberate, because retaining visual evidence across sessions is one of the abilities MedMME evaluates.

AuthorsOpenReview ↗

Rebuttal by Authors [3/4]

(W3) Explanation of Procedure-Video Labels

We thank the reviewer for raising this important point. The source annotations shown in Figure 3(a), including surgical-phase labels and action labels, are used only during benchmark construction to build the gold memory and reference answers. They are not passed to the evaluated agents. All frames are re-exported from the source videos and released with anonymized filenames. The filenames contain no information about the source video, phase, action, or time. Additionally, agents have no access to timestamps, and the source timestamps are retained only in the metadata used during construction. Each video is divided along the time axis into consecutive sessions of fixed duration. The timestamps of the source annotations are used only to align the annotations to the corresponding sessions when constructing the gold memory. They are not used to determine session boundaries. Therefore, a session may span multiple surgical phases or actions, and a phase or action may extend across multiple sessions. The session boundaries themselves carry no phase or action information. We have updated Lines 144–145 and 243–244 of the revised manuscript with this description.

(W4) Explanation of QA Validation

Thank you for pointing this out. We would like to clarify that, although the QA pairs are constructed from the gold memory, they are validated independently against the source clinical evidence. As described in Sec. 2.4 and Appendix A.5, GPT-4o first assesses the factual consistency of the QA pairs against the source clinical evidence. Physicians then review these QA pairs through a dedicated platform by referring to the same clinical evidence. Physician verification results in the exclusion of 5.5% of the QA pairs due to clinical ambiguity.

The gold memory is supported by clinical evidence and has clear clinical significance. It is also reviewed by physicians. Agents can organize their memory in any form, and QA only scores their final answers against the reference answers. We observe that long-context agents, which do not construct any memory, achieve higher accuracy on clinical dialogues. MedMME therefore primarily evaluates the ability of agents to retain, update, and integrate longitudinal multimodal clinical information, which is the foundation of longitudinal clinical reasoning.

(Q1) Analysis of Extraction Precision at Entry and Factual Levels

Thank you for your valuable feedback. We initially compute precision over memory entries, as they are the native unit that each method stores and retrieves. In our current results, we observe that precision does not increase monotonically with the average entry length. As shown in Table 2, Omni-SimpleMem has the shortest entries (29.3 tokens) yet achieves higher precision than M2A (53.3 tokens) and MMA (167.4 tokens). Following your feedback, we concatenate semantically similar memory entries in groups of 2, 4, and 8 and re-evaluate precision. As shown in Table 3, concatenation raises entry-level precision to some extent, but the gain is limited as a supporting entry requires all claims within it to be evidence-supported. For instance, the precision of MIRIX and Omni-SimpleMem drops when 8 entries are concatenated, since lengthy entries are more likely to contain unsupported claims. We further follow your suggestion and report factual-claim precision (last column of Table 3), which is close to entry-level precision and yields a consistent ranking. We have added the computation of factual-claim precision in Lines 1004–1010 and reported factual-claim precision in Table 4 of the manuscript.

Table 2: Performance comparison and average memory entry length across different methods.

MethodLengthEntry-level PrecisionF1
Omni-SimpleMem29.366.357.9
M2A53.357.157.9
RAG-Anything59.869.180.1
MIRIX75.567.163.3
Augustus114.878.585.2
MMA167.430.642.1

Table 3: Comparison of precision under different semantic concatenation settings, along with factual precision.

MethodEntry-level PrecisionConcat 2Concat 4Concat 8Factual-claim precision
RAG-Anything69.170.972.373.267.8
MIRIX67.170.671.869.065.9
MMA30.638.945.249.926.7
Omni-SimpleMem66.369.169.867.365.6
Reviewer_Bv2LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MedMME, a benchmark for long-horizon memory in multimodal medical agents. It transforms longitudinal clinical data and annotated procedure videos from 15 source datasets into temporally ordered sessions, and evaluates seven types of questions in a streaming setting where earlier raw inputs are unavailable after processing. The authors additionally construct physician-reviewed gold memory points to score extraction, state updating, and retrieval. They evaluate long-context models and agents using nine retrieval or external-memory mechanisms, followed by analyses across horizons, visual categories, medical specialization, model scale, and construction cost.

Critical strengths
  1. Inspectable memory targets. Linking each question to supporting gold memory, and evaluating extraction, updating, and retrieval separately, is more informative than reporting answer accuracy alone. The original-source grounding, two independent physician reviewers per session, and senior review of revisions are meaningful safeguards for a large benchmark.
  2. A meaningful difference between settings. Table 3 shows that the strongest evaluated memory method on clinical dialogues scores 47.2, below the matched GPT-5.4 Nano long-context baseline at 53.4, whereas several memory methods modestly exceed that baseline on procedure videos (up to 27.6 versus 24.8). Reporting this contrast is valuable because it gives future work a more specific question—when and why external memory helps—than a single overall leaderboard would.
Critical weaknesses
  1. The contribution of visual information has not been quantified. The paper reasonably includes image-to-session association and historical visual evidence as aspects of multimodal memory; I am not asking every question to require a new clinical interpretation of pixels. However, Appendix A.3 counts an image gold point as extracted when the requisite original image asset is retained and accessible, while answers are derived from gold memory and, for multimodal questions, image-to-session bindings. The current results therefore do not tell us what fraction of questions require visual content rather than image identity, metadata, or the accompanying text. Nor do they distinguish these sources of error in the modality breakdown. Appendix C3 itself appropriately notes that one cross-modal failure cannot be attributed uniquely to visual matching, textual retrieval, or answer generation. A stratified audit or controlled pixel/text/metadata ablation would clarify what the benchmark measures and make the modality claims more persuasive. This concerns the interpretation of the benchmark.
  2. The long-horizon comparison needs matched cases. Figure 4 reports weaker accuracy–horizon correlations for memory methods, but it does not give the number or composition of cases at each session index or image-count range. Patients reaching later sessions can differ systematically in source dataset, modality, and question mix from those at earlier sessions. Moreover, Appendix B.1 discards the oldest sessions from a long-context baseline when the context limit is exceeded. That operational comparison is legitimate, but the extent of truncation must be reported to interpret the crossover. Given the opposing dialogue and video patterns in Table 3, aggregate trends alone cannot establish that memory mechanisms intrinsically degrade more slowly. A matched-question analysis by setting, with case counts, truncation rates, and patient-level uncertainty, would provide a firmer basis for this central claim.
Questions for authors
  1. Could you report, by QA family and modality, how often the answer is recoverable from textual clinical evidence and image-to-session metadata without using image pixels? For a representative audited subset, a text/metadata-only control and a pixel-dependent subset would help interpret the benchmark’s multimodal claims.
  2. For Figure 4, how many independent patients and QA pairs contribute to each horizon bin, and how do source datasets and task types vary across bins? How often is the long-context baseline truncated? Does the relative trend persist when the same eligible questions are compared within each setting, with confidence intervals clustered by patient?
  3. Appendix A.3 defines retrieval at a fixed number K of entries rather than a fixed amount of information, while Table 6 shows large differences in entry counts and lengths. What is K? Are the Table 4 retrieval rankings stable under a common retrieved-token or image-asset budget? Also, the reported physician–judge agreement on 300 QA responses validates answer grading; has comparable human validation been performed for the distinct extraction, update, and retrieval judgments?
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

We would like to express our heartfelt gratitude to the reviewer for recognizing that our memory evaluation is informative, our source grounding and physician review provide meaningful safeguards, and our comparison across settings is valuable. We sincerely appreciate the insightful questions raised and have addressed them in detail below.

(W1 & Q1) Quantification of Visual Information Contribution.

Thank you for your thoughtful feedback. We agree that this analysis helps to characterize the multimodal capabilities evaluated by our benchmark more clearly.

Answerability ratios. The QA pairs are constructed from the gold memory of each session, and each question is linked to its supporting gold memory. Based on the construction rules, we identify whether text memory and metadata alone are sufficient to determine the answer to each question. Following your suggestion, we compute the ratio of such questions, with the results summarized in Table 1. We find that text memory and metadata alone can answer fewer than 20% of the questions across all multimodal QA types in clinical dialogues and all QA types in procedure videos. Metadata such as session attribution, timestamps, and lesion identifiers can provide cues for image localization, temporal reasoning, and cross-session association. Nevertheless, most multimodal questions still require the visual information in the images themselves.

Controlled ablation. To further assess the contribution of visual information, we partition the questions into a text-metadata subset and a pixel-dependent subset. With the question input kept unchanged, we compare three memory settings: text-only memory, text memory with metadata, and text memory with images. As shown in Table 2, on the pixel-dependent subset, text memory with images substantially outperforms text memory with metadata. On the text-metadata subset, text memory with images still achieves 2% higher accuracy than text memory with metadata. This might be attributed to the more intuitive and richer cues provided by image memory, which help the model locate and utilize the relevant clinical evidence.

Table 1: Ratio (%) of questions answerable with text memory and metadata across QA families. Dial.: multimodal clinical dialogues; Vid.: medical procedure videos; Text: text-only questions; MM: multimodal questions.

Dial.Vid.
TextMMTextMM
Methodsgl.upd.cmp.agg.loc.trc.col.sgl.upd.cmp.agg.loc.trc.col.
Text memory only64.839.254.740.70000000000
Text memory + Metadata71.250.961.359.75.010.918.98.910.19.519.38.514.610.7

Table 2: Performance comparison under three memory settings on different QA subsets.

QA SubsetAcc (text)Acc (text + meta)Acc (text + image)
Overall10.814.328.9
Text + Metadata subset38.741.443.4
Pixel-dependent subset03.723.3
AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

(W2 & Q2) Analysis of Long-horizon Trends with Balanced Subset & Statistics of Truncation.

Balanced Subset Analysis. Thank you for your insightful suggestion. The long-horizon analysis in Figure 4 is based on all evaluation samples. To further validate our analysis, we select an evaluation subset whose session-index bins are relatively balanced in data source, question type, and number of patients. Detailed statistics are provided in Table 3. As shown in Table 4, while the accuracy of both the long-context baseline and memory methods decreases as the session index increases, memory methods degrade more gradually and eventually outperform the long-context baseline. This is consistent with the trend observed in Figure 4(a) of the manuscript.

Matched Comparisons by Setting. We further compare memory methods with the long-context baseline on the evaluation subset, separately for the dialogue and video settings. As shown in Table 5, truncation of the long-context baseline occurs earlier and at a higher rate in the video setting. This might be attributed to the faster consumption of the context budget by dense visual inputs. In both settings, we still observe that memory methods degrade more gradually and eventually outperform the long-context baseline. We report 95% patient-clustered paired bootstrap confidence intervals of the accuracy differences, which provide statistical support for the relative performance trends. The long-horizon trends in Tables 4 and 5 are visualized in Appendix Figure 11, in the same format as Figure 4(a).

Table 3 : Statistics of the balanced subset across session index bins.

SettingSession index# Source dataset# Patients# Question types# QA
dialogue1-54687298
6-104687298
11-154667292
16-204647288
> 204647288
video1-53667210
6-103667210
11-153667210
16-203637203
> 203607191

Table 4: Performance comparison between long-context and memory methods across session index bins.

Session1-56-1011-1516-20>20Spearman ρ
Long-context48.739.430.920.316.7-0.79
Memory40.136.632.822.620.4-0.46

Table 5: Performance comparison, truncation rates, and patient-clustered confidence intervals across session index bins.

SettingSession indexMemory ACCLong-context ACCTruncation Rate95% CI of Δ\Delta ACCp-value
dialogue1-537.648.30[-13.1, -8.1]< 0.001
6-1034.439.70[-7.8, -2.9]< 0.001
11-1532.531.93.9[-1.6, 2.5]0.627
16-2022.120.425.1[1.2, 4.6]0.002
> 2019.215.384.0[2.4, 5.7]< 0.001
video1-543.649.20[-8.5, -2.7]< 0.001
6-1039.738.97.1[-2.3, 3.3]0.665
11-1533.229.434.7[0.2, 4.6]0.032
16-2023.420.181.3[1.6, 4.9]< 0.001
> 2022.318.8100[2.4, 6.2]< 0.001
AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

(Q3) Common Retrieval Budgets Evaluation & Human Verification of Memory Operation Judgments

Common Retrieval Budgets. We thank the reviewer for pointing this out. Our retrieval experiments use K=10, and we have stated this explicitly in Appendix A.3. We further compute retrieval coverage under common retrieved token and image asset budgets. As shown in Table 6, the retrieval coverage of RAG methods remains generally higher than that of external memory modules. Omni-SimpleMem has a lower extraction recall than Augustus, whereas its retrieval coverage is higher under all budget settings. The observation in Sec. 3.3 of the manuscript that more complete retention does not necessarily guarantee better evidence retrieval still holds after controlling for the retrieval budget. We have added Table 6 to Appendix B.4.

Human Validation. We further recruit physicians to score the extraction, updating, and retrieval of the corresponding questions for 100 sessions using the same evaluation criteria. As shown in Table 7, agreement between the LLM judge and expert consensus scores is high. The Cohen’s κ of the four methods across the three memory operations all exceed 0.86, supporting the reliability of automated scoring on the reviewed sample. We have mentioned this in Line 261-262 of the manuscript and added Table 7 to Appendix B.3.

Table 6: Performance comparison under common retrieved token and image asset budgets.

Retrieved Tokens1k2k
Image Assets10 img20 img10 img20 img
WeMM-Embedding61.362.767.969.1
UniversalRAG58.859.564.265.0
MMA36.238.640.743.2
MIRIX46.347.555.657.4
Augustus30.531.034.835.1
Omni-SimpleMem55.856.061.061.2

Table 7: Agreement between the LLM judge and physician consensus on memory operations, measured by Cohen’s κ.

MethodExtractionUpdateRetrieval
WeMM-Embedding0.8870.8940.881
RAG-Anything0.8680.9050.875
MIRIX0.9350.9070.880
Omni-SimpleMem0.9140.8960.902

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear Reviewers, AC, SAC, and PC,

Thank you for your time and effort in ensuring a high-quality review process for ICLR. We respectfully provide a summary of the discussion for the AC's reference. For clarity and simplicity, we will refer to Reviewers Bv2L, RwfE, and WyvD as R1, R2, and R3, respectively, in the following response.

  1. Important and insufficiently studied problem of maintaining patient-specific multimodal evidence over long-horizon clinical interactions under a streaming protocol (R2).
  2. Broad and clinically diverse benchmark built from real longitudinal data, covering 10 clinical domains and 9 visual modalities across multimodal clinical dialogues and medical procedure videos (R2, R3).
  3. Physician-verified clinical gold memory, which allows memory extraction, updating, and retrieval to be evaluated beyond QA accuracy (R1, R2, R3).
  4. Informative empirical findings on long-context baselines and memory-augmented agents across clinical dialogues and procedure videos (R1).

We will ensure that all additional experiments and clarifications from the rebuttal are included in the final version.

  1. Contribution of visual information (R1, R3). We quantified the ratio of questions answerable without images and compared four visual memory settings (Appendix B.10), confirming that MedMME evaluates the use of visual evidence.
  2. Reliability of memory-operation evaluation (R1, R2, R3). We added factual-claim precision (Table 4 in Section 3.3), validated the memory-operation judges against physicians (Appendix B.3), and re-evaluated retrieval under common retrieval budgets (Appendix B.4).
  3. Evaluation protocol and additional analyses (R2, R3). We added a formal definition of the streaming protocol (Appendix B.2), and reported the computational cost of baselines (Appendix B.6), an error mode analysis (Appendix B.5), and controlled comparisons for medical specialization (Appendix B.7).
  4. Evaluation scope beyond a single patient (R3). MedMME follows the patient-specific setting of existing medical memory benchmarks. Following the reviewer's suggestions, we added shared-memory and cross-patient knowledge sharing experiments (Section 4 and Appendix B.9).

R1 stated that our responses resolved their concerns and raised the rating to weak acceptance. R3 stated that all other concerns have been fully addressed. For multi-patient memory, we have provided additional experiments and clarifications. We did not hear back from R2 during the discussion period. We believe our rebuttal addresses the concerns with the requested analyses.

Finally, we sincerely thank the reviewers for the thoughtful feedback and for carefully engaging with our rebuttal. To our knowledge, MedMME is the first multimodal medical memory benchmark that adopts multi-session inputs under a streaming protocol. Tracking patient states across evolving multimodal evidence is a pressing challenge for reliable medical agents. To systematically study this problem, we construct an evaluation framework from real longitudinal data, covering 10 clinical domains and 9 visual modalities across multimodal clinical dialogues and medical procedure videos. We further introduce physician-verified clinical gold memory to enable a fine-grained evaluation of memory operations, including extraction, updating, and retrieval.

All requested modifications have been incorporated faithfully into the updated paper (highlighted in blue). Given the context, we respectfully request that AC take our points into consideration while making the final decision. Thank you again for your valuable time and thoughtful consideration.

Best regards,

Authors of Submission 4749

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.