- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether complex memory management is necessary for long term conversation tasks. The paper proposes Raw-RAG, which includes dense retrieval, cross-encoder reranking etc., retrieving relevant evidence directly from memory. Experiments across Locomo, Longmemeval, MEME and MemoryAgentBench show that Raw-RAG could perform competitively with existing agent memory systems, even better in some cases. In addition, the paper further provides analysis of memory mutation frequency, retrieval configurations, computational efficiency, etc, telling why raw-history retrieval remains effective and if we really need complex memory management.
Critical strengths
- I think the research question is well motivated and the analysis/experiment are extensive to some degree.
The paper revises an important assumption underlying recent agentic memory research: whether increasingly complex memory management consistently provides benefits beyond effective retrieval from raw histories. The evaluation covers multiple benchmarks, memory systems and reader configurations, providing evidence that a carefully but simple RAG baseline can also effective compare to complex memory systems.
- Diagnostic analyses are useful to some degree beyond accuracy comparisons.
The paper goes beyond performance by researching memory mutation behaviour, retrieval depth, evidence coverage etc. In addition, distinguishing evidence availability from downstream answer generation provides a useful perspective for diagnosing memory system failures.
Critical weaknesses
- The diagnostic experiments do not sufficiently isolate the factors underlying the reported performance differences.
The coverage/utilisation analysis is informative, but it needs more careful explanation. For example, section 5.1 table shows Raw-RAG has 938 sufficient evidence questions, while evermemos only has 520 questions. Their utilizatiion scores are therefore calculated over different subsets of questions, potentially with different difficulty distributions. Although the authors acknowledge this limitation, the results do not directly establish how much of the performance difference is attributable to evidence coverage rather than differences in the covered question sets.
In addition, Appendix G.9 shows that the MemoryAgentBench sufficiency judge evaluates the text constructed by locating golden answer values and extracting nearby text, with a maximum of 9,000 characters, rather than consistently evaluating the complete reader-visible context. This may overlook supporting evidence located elsewhere in the context and could affect systems with different memory representations differently.
What is more, a related limitation appears in the complete store analysis (Figure 5). It seems different systems apply different gold informed evidence selection and packing procedures. As said in the Appendix C.5, the observed gaps cannot be attributed solely to information loss during memory construction.
Overall, the analysis explanation seems give me the feeling that they have not yet fully isolate retrieval and evaluation effects.
- The ablations of meemory management and bechmark results do not clearly establish when additional memory complexity is beneficial.
I think the memory management ablations in Section 4.3 seem a little difficult to interpret as evidence against memory organisation itself. For example, the atomic-fact variant replaces each raw window with short extracted facts while retaining the same top item budget. This changes both the representation and the amount and structure of evidence available to the reader. The resulting drop in TTL accuracy from 47.68% to 15.37% may therefore reflect disrupted structure rather than a general limitation of fact-based memory. Similarly, summary augmentation changes the indexed text and embeddings, and graph expansion changes the retrieval candidate set. These experiments evaluate particular modifications to Raw-RAG rather than fully developed memory management approaches.
In addition, experiment release the limitations of Raw-RAG in different tasks. For example, Raw-RAG only achieves 3.66% on MEME cascade reasoning with DeepSeek-V4-Flash. And Appendix A.3 also shows it falls behind that EverMemOS and MemOS. These results suggest that the relative value of structured memory may depend on the task, even more agentic tasks such as shopping etc.(such as AMA-bench or Memoryarena bench), but this distinction is not sufficiently developed.
Finally, although the paper provides ingestion/retrieval latency(Table 6), it does not provide a complete long-term cost comparison that includes answer generation and repeated queries (token cost, etc). Thus, the experiments show Raw-RAG seem as a strong baseline, but provide less conclusive evidence about when its simplicity is preferable to more sophisticated memory management.
Questions for authors
- Evidence coverage and utilisation (Section 5.1, Tables 7–8, Appendix G.9).
The reported utilisation scores are calculated on subsets of questions judged to have sufficient evidence, as said in the weakness. For example, Raw-RAG covers 938 MemoryAgentBench questions, compared with 520 for EverMemOS. Could the authors provide a comparison of conditional answer accuracy on the subset of questions jointly covered by the compared systems, to better distinguish evidence utilisation from differences in question difficulty?
Additionally, Appendix G.9 evaluates sufficiency using gold answer localised excerpts rather than complete reader visible contexts. Would the coverage rankings remain consistent if the complete contexts were evaluated?
- Memory management ablations (Section 4.3, Table 5, Appendix C.3).
In the atomic-fact ablation, raw windows are replaced by short extracted facts while the final Top item budget remains unchanged. Given the substantial decrease in TTL performance (47.68% to 15.37%), how much of this degradation is attributable to lost demonstration structure or reduced evidence volume rather than atomic-fact extraction itself? Similar to other actions such as raw summary etc.
- Applicability to complex memory tasks (Section 4.2, Table 4, Appendix A.3).
Raw-RAG performs strongly overall, but achieves only 3.66% on MEME cascade reasoning with DeepSeek-V4-Flash and fail behind EverMemOS and MemOS on PersonaMem-v2. How do the authors interpret these task-dependent differences? Do the results suggest specific conditions, such as complex state dependencies or personalisation requirements, under which structured memory management may be more beneficial? How do they perform on other agentic tasks such as ama-bench, memoryarena etc? If it is not suitable to agentic tasks, maybe it should say that the scope is in conversation? such as the paper title or the "Most case" of Section 5 title (WHY IS RAW-RAG ENOUGH IN MOST CASES)
- Long-term efficiency and token consumption (Section 4.3).
The paper reports ingestion/retrieval latency, but the latency comparisons exclude answer generation. Could the authors provide end-to-end token consumption and inference cost comparisons, including reader input and output tokens? Since structured or compressed memories may have higher construction costs but lower per-query costs, how would the total cost comparison change as the number of queries over the same conversation history increases?
15 comments
Response to Reviewer ocSV (2/3)
2. W2, Q2: Memory-management ablations
Atomic facts with matched evidence. We let the atomic-fact variant pass facts to the reader up to each question's Raw-RAG evidence-token count, instead of a fixed top 15.
| Task | Raw-RAG | Atomic, top 15 | Atomic, matched tokens |
|---|---|---|---|
| Accurate Retrieval | 67.85% | 57.25% | 58.27% |
| Test-Time Learning | 47.68% | 15.37% | 16.04% |
| Long-Range Understanding | 31.74% | 21.17% | 21.83% |
| Conflict Resolution | 46.00% | 21.00% | 21.00% |
| Total | 48.32% | 28.70% | 29.29% |
Matching the evidence volume recovers 0.59 of the 19.62-point loss, so volume explains almost none of the drop. The loss comes from the extraction step itself, which in TTL breaks up the demonstrations the task depends on (App. F).
Summary and graph. Neither reduces evidence volume. Each summary item holds the full raw window plus its summary, so it is longer than a raw chunk. Graph expansion changes which raw windows are retrieved, and the reader still sees raw windows. Without CR, their losses are 1.41 and 1.21 points (see our response to Reviewer 9LTs).
Scope. We agree that these are modifications of Raw-RAG rather than complete memory systems; complete systems are compared in the main tables. We will rephrase RQ2 as "Do these operations improve the fixed Raw-RAG pipeline?" and limit its conclusion to the tested implementations.
3. W2, Q3: Complex memory tasks and scope
PersonaMem-v2. Raw-RAG (49.62%) trails EverMemOS (53.25%) and MemOS (50.72%). These tasks require applying preferences aggregated across sessions, which favors systems that maintain a consolidated user profile. A profile component alone is not enough: MemU and MemoryOS also keep profiles and score 38.70% and 40.05%.
MEME. With DeepSeek-V4-Flash, Raw-RAG scores 43.19% overall, against 63.17% for Full-context, and 3.66% on cascades. The reader is not the bottleneck: given the phase-visible source sessions, it reaches 82.32% on cascades (Table 2). Similarity retrieval returns the changed fact but tends to miss the dependency rule or the downstream state it invalidates. No evaluated memory system exceeds 9.15% on cascades, so cascades remain unsolved by current designs, retrieval-based or structured.
AMA-Bench [1]. On the complete real-world subset, under the official Qwen3-32B answering and judging protocol, Raw-RAG reaches 52.80%. That is 10.53 points above the benchmark's own dense-retrieval baseline (Qwen3-Emb-4B), above HippoRAG2 and all eleven memory-agent baselines, and above the Qwen3-32B long-context reader (51.81%, Table 4 of [1]). It is 4.42 points below AMA-Agent (57.22%), which builds an explicit causality graph with tool-augmented retrieval. All rows other than Raw-RAG are the published Table 5 numbers of [1], obtained with the same backbone and judge.
| Method | Type | Recall | Causal inference | State updating | State abstraction | Average |
|---|---|---|---|---|---|---|
| Raw-RAG(ours) | RAG | 57.93% | 60.07% | 47.14% | 40.82% | 52.80% |
| AMA-Agent | AMA-Bench authors | 62.38% | 61.45% | 53.05% | 47.19% | 57.22% |
| HippoRAG2 | RAG | 45.79% | 50.80% | 44.03% | 35.38% | 44.80% |
| Qwen3-Emb-4B | RAG | 48.43% | 49.74% | 35.20% | 30.11% | 42.27% |
| BM25 | RAG | 33.01% | 42.64% | 34.50% | 24.98% | 34.36% |
| GraphRAG | RAG | 30.77% | 39.05% | 31.40% | 28.79% | 32.58% |
| EMem | Memory agent | 46.31% | 49.25% | 45.12% | 34.21% | 46.10% |
| MemoRAG | Memory agent | 47.08% | 54.97% | 42.57% | 36.59% | 46.06% |
| MemoryBank | Memory agent | 32.31% | 41.00% | 30.06% | 33.32% | 33.97% |
| MemGPT | Memory agent | 32.89% | 44.04% | 28.09% | 25.26% | 33.04% |
| A-Mem | Memory agent | 30.84% | 36.53% | 30.88% | 28.73% | 31.86% |
| Mem-alpha | Memory agent | 28.76% | 41.72% | 30.64% | 21.71% | 31.17% |
| HiMem | Memory agent | 29.45% | 40.18% | 31.26% | 22.38% | 29.95% |
| MemAgent | Memory agent | 25.50% | 33.80% | 28.49% | 22.02% | 27.68% |
| Mem0 | Memory agent | 20.11% | 26.45% | 21.01% | 15.16% | 21.04% |
| SimpleMem | Memory agent | 20.12% | 18.84% | 17.64% | 13.73% | 18.11% |
| Mem1 | Memory agent | 11.80% | 14.27% | 12.05% | 10.80% | 12.29% |
AMA-Agent leads Raw-RAG in all four capabilities, most on state updating (5.91 points) and state abstraction (6.37 points). By domain, Raw-RAG is weakest on Embodied AI, where state-updating questions are most common (150 of 360):
| Domain | Raw-RAG |
|---|---|
| Game | 69.72% |
| Open-world QA | 61.94% |
| Web | 56.18% |
| Text-to-SQL | 54.41% |
| Software | 40.97% |
| Embodied AI | 34.72% |
MemoryArena [2]. The evaluation is still running because of the benchmark's size. Partial runs show Raw-RAG struggling on the math and physics reasoning tasks; we will post the full results here when they finish.
Response to Reviewer ocSV (3/3)
In these settings Raw-RAG stays within 1.5 to 4.4 points of the best memory system overall but does not lead. It is strongest when the answer depends on recovering specific details, and weaker when the task requires aggregating preferences, tracking dependent state changes, or abstracting over long trajectories.
On scope: our evidence is strongest for long-term conversation. We will state this scope in the abstract and introduction, retitle Section 5 "Why Is Raw-RAG Competitive on the Evaluated Conversational Benchmarks?", and report AMA-Bench as an out-of-domain test, including the cases where AMA-Agent is stronger.
4. W2, Q4: End-to-end cost and repeated queries
Table 6 excluded answer generation. Below we add it, report reader tokens, and measure repeated queries directly.
In the two tables below, times are in seconds. Total generative tokens cover construction and answering; n/a marks runs whose token logs were not kept. Latency break-even is the first number of queries at which a system becomes faster than Raw-RAG, under with the construction time and the mean retrieval-plus-answer time; ∞ means never. Reader output tokens are similar across systems (1.9k to 2.4k on MEME, 17.6k to 21.3k on BEAM) and are omitted.
MEME, 10 histories, 110 queries
| System | Construction | Retrieval | Answering | Total | Reader input tokens per query | Total generative tokens | Latency break-even |
|---|---|---|---|---|---|---|---|
| Raw-RAG | 50.32 | 255.59 | 230.57 | 536.48 | 8,404 | 926,781 | reference |
| EMem | 1,082.37 | 479.59 | 151.78 | 1,713.75 | 314 | 3,842,889 | ∞ |
| MemOS | 2,591.59 | 177.97 | 157.24 | 2,926.80 | 1,055 | n/a | 1852 |
| MemMachine | 3,104.35 | 411.39 | 180.97 | 3,696.71 | 5,068 | n/a | ∞ |
| Mem0 | 4,025.23 | 117.72 | 153.92 | 4,296.87 | 1,255 | n/a | 2039 |
| EverMemOS | 8,217.69 | 1,083.79 | 154.17 | 9,455.65 | 1,922 | n/a | ∞ |
BEAM, 1 history, 20 queries
| System | Construction | Retrieval | Answering | Total | Reader input tokens per query | Total generative tokens | Latency break-even |
|---|---|---|---|---|---|---|---|
| Raw-RAG | 1,174.26 | 64.63 | 205.53 | 1,444.41 | 20,473 | 428,619 | reference |
| MemOS | 11,698.04 | 24.72 | 192.06 | 11,914.82 | 2,020 | 4,750,772 | 3,944 |
| Mem0 | 14,206.48 | 50.27 | 189.27 | 14,446.01 | 2,077 | 16,881,354 | 8,512 |
| MemMachine | 30,706.46 | 90.63 | 197.92 | 30,995.01 | 16,043 | n/a | ∞ |
| EMem | 65,501.85 | 133.98 | 185.18 | 65,821.01 | 1,354 | 12,756,578 | ∞ |
| Memora | 97,037.04 | 243.40 | 206.93 | 97,487.37 | 32,474 | n/a | ∞ |
Repeated queries, measured. We built each memory once on one complete LoCoMo conversation and answered all 152 of its questions in sequence. Generative tokens per query include query-time LLM calls (MemMachine, EverMemOS) and the reader's input and output. Token break-even is the first query count at which a system uses fewer cumulative generative tokens than Raw-RAG, projected as with the construction tokens and the mean tokens per query.
LoCoMo, 1 conversation, 152 queries
| Method | Construction (s) | Search + answer per query (s) | Time after 152 queries (s) | Construction tokens | Generative tokens per query | Tokens after 152 queries | Token break-even |
|---|---|---|---|---|---|---|---|
| Raw-RAG | 6.98 | 13.49 | 2,057.78 | 0 | 6,460 | 981,908 | reference |
| Mem0 | 1,880.63 | 13.67 | 3,958.80 | 1,619,859 | 3,922 | 2,215,935 | 639 |
| MemMachine | 2,624.82 | 14.67 | 4,854.58 | 2,304,416 | 4,075 | 2,923,777 | 967 |
| EverMemOS | 1,084.03 | 21.63 | 4,372.21 | 883,553 | 8,524 | 2,179,273 | ∞ |
Raw-RAG has the lowest cumulative time at every prefix from 1 to 152 queries, and no comparator catches up, since none is faster per query. Tokens behave differently. Raw-RAG gives the reader more context (6.5k tokens per LoCoMo query against 3.9k to 4.1k for Mem0 and MemMachine; 8.4k per MEME query against 0.3k to 5.1k), which also makes its MEME answering step the slowest. Mem0 and MemMachine therefore use fewer generative tokens than Raw-RAG from the 639th and 967th query on the same history. EverMemOS never does, because of its query-time LLM calls. These counts exclude embedding and reranking and weight input and output tokens equally.
Raw-RAG's advantage is construction cost. In wall-clock time it holds at any query count on LoCoMo and for at least 186 queries per history on MEME and BEAM; in generative tokens it holds for the first 638 queries on a LoCoMo history. Past these points, systems with shorter reader contexts are cheaper.
References
[1] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. arXiv:2602.22769, 2026.
[2] He et al. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. arXiv:2602.16313, 2026.
Response to Reviewer ocSV (1/3)
Thank you for the review.
1. W1, Q1: Coverage with complete contexts, and utilization on matched questions
Complete-context coverage. The original MAB judge saw a 9,000-character excerpt around the reference-answer values (App. G.9). We re-judged all 2,071 MAB questions for all seven systems with the complete reader context, keeping the original answers.
Raw-RAG stays first, 5.12 points above Mem0 (paired 95% CI [3.19, 7.00], Holm p = 8.1e-6), and all six differences remain significant after Holm correction. The excerpt did affect systems unequally, as you suspected: MemMachine rises from 21.54% to 34.86% and EverMemOS from 25.11% to 28.97%, the two systems whose context was least visible to the original judge (9.24% and 17.91%). All analyses below use the complete-context labels.
Matched utilization. Each row compares the two systems on the same questions.
On the 112 questions covered by all seven systems:
A question counts as correct if its score is at least 0.5. Intervals use 50,000 within-source paired bootstrap replicates and are unadjusted; p-values come from two-sided exact McNemar tests, Holm-adjusted over all 21 pairwise comparisons among the seven systems.
On matched questions, Raw-RAG and each comparator differ by less than 7 points. EMem and MemU are higher than Raw-RAG before correction (their intervals exclude zero), and no Raw-RAG difference is significant after correction. On the seven-system intersection, Raw-RAG is last. This agrees with Finding 1: Raw-RAG's utilization is not the highest, and its lead comes from coverage. The intersection is small (5.41% of MAB) and mostly LongMemEval-S (98 of 112 questions).
How much of the gap comes from coverage. We split the 2,071 questions into four cells by the two systems' coverage labels. Cell c contributes (nc/2071)(aRaw,c−acomp,c) to the question-weighted accuracy gap, and the four contributions sum to that gap. The gap uses the binary correctness above, so it differs from the task-weighted MAB Total.
Contribution of each cell to the gap (pp):
Jointly covered questions contribute -1.11 to +0.72 points. Raw-RAG alone covers 106 to 499 more questions than each comparator, and the two exclusive cells contribute a net +2.66 to +14.39 points; against EverMemOS they account for 8.64 of the 11.20-point gap. The neither-covered cell is also positive against four comparators, because Raw-RAG answers some questions correctly from evidence the judge rated insufficient. We do not attribute that cell to coverage.
Figure 5. We agree that the complete-store gaps are not pure ingestion loss. Removing the 5,000-token packing cap (App. A.5.2) still leaves all seven stores 7.58 to 29.38 points below raw evidence, so the cap does not explain them. To separate storage, selection, and the reader, we audited all 2,715 complete-store failures (details in our response to Reviewer UJ8J). Reader errors are similar across systems (204 to 273, except EMem at 118), while storage absence plus selection/packing omission ranges from 54 to 223 and largely follows the gaps in Figure 5.