- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on evaluating the long-term memory capabilities of LLM agents in chronic care scenarios. The authors point out that existing general-domain or short-term medical memory benchmarks fail to reflect the complex challenges of real-world chronic disease management, which involve long time horizons, numerous departments, and massive follow-up records. To address this, the paper proposes a new benchmark, MEME-Bench, synthesized from real chronic-care records.
Critical strengths
- The scale of MEME-Bench (often thousands of visits and 90k+ tokens) and its 8-dimensional design are highly systematic.
- The paper abstracts memory systems into a four-stage pipeline, pinpointing "retrieval failure" as the core bottleneck by tracking the flow of each fact through the pipeline.
- The study covers 11 current mainstream memory architectures and 7 different backbones.
Critical weaknesses
-
The core experimental data is synthesized from real chronic-care records. However, synthetic data often carries generation artifacts, overly smooth logic, or structural biases, which may cause the failure modes of memory systems (such as certain retrieval failures) to deviate from the noise, typos, and unstructured doctor notes found in real-world electronic health records.
-
With the proliferation of ultra-long context models (supporting millions of tokens), the capacity of direct long-context attention is rising rapidly. Although the paper includes a long-context baseline, it lacks a fine-grained "context length vs. performance degradation curve" to reveal the exact threshold at which memory systems truly outperform long-context models.
-
Although the paper emphasizes that MEME-Bench contains dynamic time-point questions, these questions and evaluation suites remain essentially offline-preset and statically evaluated. In real clinical interactions, patient disease evolution is highly open-ended and adversarial.
Questions for authors
See above.
6 comments
W2 “...lacks a fine-grained "context length vs. performance degradation curve...”
We did not include an isolated context length vs performance degradation because it entangles the number of evidence visits required, the context length accumulated when it is asked, and its evidence depth, defined as the token distance to the supporting evidence.
Fig 9-19 in appendix already visualizes this effect of these three axes on memory system performance, and we break it down below into tables. From this analysis, we observe that:
Evidence depth is the only axis where Long Ctx baseline shows wins. It is the best method from 32K to 128K, with the evidence inside its window. Beyond the window, it drops to 0.40, while the memory systems stay roughly flat.
On record length, Long Ctx never leads. The < 128K bin, where the whole record fits, is mostly Temporal Validity Inference questions, which is where Long Ctx is weak.
On number of evidence visits required, the best memory system leads Long Ctx at every count. The gap is significant at 2 to 9 visits. Every method dips at 2 visits.
Performance of memory systems and long context under different evidence and record conditions.
Number of evidence visits required v.s. performance
| Method | 1 | 2 | 3–4 | 5–9 | ≥ 10 |
|---|---|---|---|---|---|
| Long Ctx | 0.64 | 0.32 | 0.44 | 0.49 | 0.61 |
| BM25 | 0.65 | 0.33 | 0.40 | 0.37 | 0.42 |
| Dense | 0.62 | 0.30 | 0.38 | 0.44 | 0.47 |
| Letta | 0.79 | 0.56 | 0.54 | 0.59 | 0.69 |
| MIRIX | 0.71 | 0.57 | 0.58 | 0.64 | 0.67 |
| ReMem | 0.73 | 0.42 | 0.58 | 0.61 | 0.57 |
| HippoRAG | 0.74 | 0.35 | 0.50 | 0.47 | 0.52 |
| A-Mem | 0.61 | 0.34 | 0.38 | 0.48 | 0.47 |
| Mem0 | 0.65 | 0.22 | 0.31 | 0.44 | 0.47 |
| LightMem | 0.58 | 0.31 | 0.38 | 0.44 | 0.49 |
Record length v.s. performance
| Method | < 128K | 128–256K | 256–384K | ≥ 384K |
|---|---|---|---|---|
| Long Ctx | 0.40 | 0.45 | 0.59 | 0.39 |
| BM25 | 0.50 | 0.37 | 0.52 | 0.36 |
| Dense | 0.46 | 0.36 | 0.52 | 0.41 |
| Letta | 0.85 | 0.62 | 0.67 | 0.49 |
| MIRIX | 0.70 | 0.65 | 0.64 | 0.60 |
| ReMem | 0.84 | 0.61 | 0.59 | 0.55 |
| HippoRAG | 0.71 | 0.49 | 0.57 | 0.46 |
| A-Mem | 0.40 | 0.39 | 0.53 | 0.42 |
| Mem0 | 0.40 | 0.34 | 0.50 | 0.41 |
| LightMem | 0.34 | 0.33 | 0.53 | 0.41 |
Evidence depth v.s. performance
| Method | < 32K | 32–64K | 64–128K | ≥ 128K |
|---|---|---|---|---|
| Long Ctx | 0.58 | 0.67 | 0.66 | 0.40 |
| BM25 | 0.54 | 0.51 | 0.34 | 0.43 |
| Dense | 0.46 | 0.47 | 0.49 | 0.46 |
| Letta | 0.75 | 0.62 | 0.58 | 0.60 |
| MIRIX | 0.73 | 0.62 | 0.60 | 0.59 |
| ReMem | 0.73 | 0.48 | 0.50 | 0.58 |
| HippoRAG | 0.64 | 0.51 | 0.45 | 0.52 |
| A-Mem | 0.47 | 0.51 | 0.49 | 0.47 |
| Mem0 | 0.48 | 0.43 | 0.45 | 0.43 |
| LightMem | 0.46 | 0.45 | 0.45 | 0.45 |
W3 “Although the paper emphasizes that MEME-Bench contains dynamic time-point questions, these questions and evaluation suites remain essentially offline-preset and statically evaluated. In real clinical interactions, patient disease evolution is highly open-ended and adversarial.”
MEME-Bench is evaluated as a stream. Each system ingests visits one at a time in chronological order, and questions are posed at intermediate checkpoints using only the history up to that point, so memory must be maintained while the record grows (Section 3). The questions mirror decisions made during chronic follow-up: whether a referral was completed and what it found, whether a temporary restriction still applies, which drug is safe given recorded constraints, and how a measurement has changed. Closed-loop evaluation, in which an agent's decisions change the patient's later course, needs a patient simulator whose dynamics respond to treatment. We agree it is the natural next step and will add it to the conclusion as future work. MEME-Bench opens the door for future works that involve such evolving clinical world models or realistic longitudinal patient digital twins.
We thank the reviewer for the detailed and constructive review. We answer W1–W3 in the order of the review: (All additional results added in blue color text in appendix)
W1 “...synthetic data …may cause the failure modes of memory systems (such as certain retrieval failures) to deviate from the noise, typos, and unstructured doctor notes found in real-world electronic health records.”
We add noises (e.g. random non-clinical sentences, random change of full phrases into abbreviations, copy-over prior conversations, change date formats, etc.) and typos to every visit, affecting approximately 33% of the turns, 1.6% of the total token count, and see if our conclusions still hold on Qwen 3.6 27B.
Our main conclusions remain consistent. MIRIX, MedCache, Letta, REMem remain the first tier of memory systems.
Original hardest axes, Trajectory and Cross-department remain the hardest challenges.
What changed is that stronger agentic systems dropped more with noise addition, as we can observe Letta/MIRIX/REMem average loses .09 while the others suffer less.
Performance comparison of 11 memory systems and long context across 8 axes with and without added noise.