acceptodds
Under review as a conference paper at ICLR 2027

Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought

Abstract

LLM memory is widely used in long-term conversation understanding. Existing agentic memory systems often manage memories through mutation, structural organization, learned retrieval strategies, or other complex operations. Is this complexity always necessary? We study Raw-RAG, a naive RAG baseline using dense retrieval and reranking over raw history to supply evidence to a reader LLM. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among the compared memory systems in six of seven benchmark–model settings, outperforming the strongest compared agentic memory system by 0.13–5.53 points in those settings. We then investigate why raw-history retrieval remains competitive in these settings. Evidence analysis shows that its higher coverage of necessary evidence can offset lower utilization. Useful evidence includes both answer values and the relations and dependency paths needed to derive them. Even complete access to transformed memory stores leaves an accuracy gap relative to raw evidence, highlighting the importance of preserving raw information. These findings challenge the need for complex agentic memory systems in the evaluated conversation tasks and establish naive RAG as an essential baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RAG, Memory Agents, Long-term QA, Long-term conversation understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ocSVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether complex memory management is necessary for long term conversation tasks. The paper proposes Raw-RAG, which includes dense retrieval, cross-encoder reranking etc., retrieving relevant evidence directly from memory. Experiments across Locomo, Longmemeval, MEME and MemoryAgentBench show that Raw-RAG could perform competitively with existing agent memory systems, even better in some cases. In addition, the paper further provides analysis of memory mutation frequency, retrieval configurations, computational efficiency, etc, telling why raw-history retrieval remains effective and if we really need complex memory management.

Critical strengths
  1. I think the research question is well motivated and the analysis/experiment are extensive to some degree.

The paper revises an important assumption underlying recent agentic memory research: whether increasingly complex memory management consistently provides benefits beyond effective retrieval from raw histories. The evaluation covers multiple benchmarks, memory systems and reader configurations, providing evidence that a carefully but simple RAG baseline can also effective compare to complex memory systems.

  1. Diagnostic analyses are useful to some degree beyond accuracy comparisons.

The paper goes beyond performance by researching memory mutation behaviour, retrieval depth, evidence coverage etc. In addition, distinguishing evidence availability from downstream answer generation provides a useful perspective for diagnosing memory system failures.

Critical weaknesses
  1. The diagnostic experiments do not sufficiently isolate the factors underlying the reported performance differences.

The coverage/utilisation analysis is informative, but it needs more careful explanation. For example, section 5.1 table shows Raw-RAG has 938 sufficient evidence questions, while evermemos only has 520 questions. Their utilizatiion scores are therefore calculated over different subsets of questions, potentially with different difficulty distributions. Although the authors acknowledge this limitation, the results do not directly establish how much of the performance difference is attributable to evidence coverage rather than differences in the covered question sets.

In addition, Appendix G.9 shows that the MemoryAgentBench sufficiency judge evaluates the text constructed by locating golden answer values and extracting nearby text, with a maximum of 9,000 characters, rather than consistently evaluating the complete reader-visible context. This may overlook supporting evidence located elsewhere in the context and could affect systems with different memory representations differently.

What is more, a related limitation appears in the complete store analysis (Figure 5). It seems different systems apply different gold informed evidence selection and packing procedures. As said in the Appendix C.5, the observed gaps cannot be attributed solely to information loss during memory construction.

Overall, the analysis explanation seems give me the feeling that they have not yet fully isolate retrieval and evaluation effects.

  1. The ablations of meemory management and bechmark results do not clearly establish when additional memory complexity is beneficial.

I think the memory management ablations in Section 4.3 seem a little difficult to interpret as evidence against memory organisation itself. For example, the atomic-fact variant replaces each raw window with short extracted facts while retaining the same top item budget. This changes both the representation and the amount and structure of evidence available to the reader. The resulting drop in TTL accuracy from 47.68% to 15.37% may therefore reflect disrupted structure rather than a general limitation of fact-based memory. Similarly, summary augmentation changes the indexed text and embeddings, and graph expansion changes the retrieval candidate set. These experiments evaluate particular modifications to Raw-RAG rather than fully developed memory management approaches.

In addition, experiment release the limitations of Raw-RAG in different tasks. For example, Raw-RAG only achieves 3.66% on MEME cascade reasoning with DeepSeek-V4-Flash. And Appendix A.3 also shows it falls behind that EverMemOS and MemOS. These results suggest that the relative value of structured memory may depend on the task, even more agentic tasks such as shopping etc.(such as AMA-bench or Memoryarena bench), but this distinction is not sufficiently developed.

Finally, although the paper provides ingestion/retrieval latency(Table 6), it does not provide a complete long-term cost comparison that includes answer generation and repeated queries (token cost, etc). Thus, the experiments show Raw-RAG seem as a strong baseline, but provide less conclusive evidence about when its simplicity is preferable to more sophisticated memory management.

Questions for authors
  1. Evidence coverage and utilisation (Section 5.1, Tables 7–8, Appendix G.9).

The reported utilisation scores are calculated on subsets of questions judged to have sufficient evidence, as said in the weakness. For example, Raw-RAG covers 938 MemoryAgentBench questions, compared with 520 for EverMemOS. Could the authors provide a comparison of conditional answer accuracy on the subset of questions jointly covered by the compared systems, to better distinguish evidence utilisation from differences in question difficulty?

Additionally, Appendix G.9 evaluates sufficiency using gold answer localised excerpts rather than complete reader visible contexts. Would the coverage rankings remain consistent if the complete contexts were evaluated?

  1. Memory management ablations (Section 4.3, Table 5, Appendix C.3).

In the atomic-fact ablation, raw windows are replaced by short extracted facts while the final Top item budget remains unchanged. Given the substantial decrease in TTL performance (47.68% to 15.37%), how much of this degradation is attributable to lost demonstration structure or reduced evidence volume rather than atomic-fact extraction itself? Similar to other actions such as raw summary etc.

  1. Applicability to complex memory tasks (Section 4.2, Table 4, Appendix A.3).

Raw-RAG performs strongly overall, but achieves only 3.66% on MEME cascade reasoning with DeepSeek-V4-Flash and fail behind EverMemOS and MemOS on PersonaMem-v2. How do the authors interpret these task-dependent differences? Do the results suggest specific conditions, such as complex state dependencies or personalisation requirements, under which structured memory management may be more beneficial? How do they perform on other agentic tasks such as ama-bench, memoryarena etc? If it is not suitable to agentic tasks, maybe it should say that the scope is in conversation? such as the paper title or the "Most case" of Section 5 title (WHY IS RAW-RAG ENOUGH IN MOST CASES)

  1. Long-term efficiency and token consumption (Section 4.3).

The paper reports ingestion/retrieval latency, but the latency comparisons exclude answer generation. Could the authors provide end-to-end token consumption and inference cost comparisons, including reader input and output tokens? Since structured or compressed memories may have higher construction costs but lower per-query costs, how would the total cost comparison change as the number of queries over the same conversation history increases?

15 comments
AuthorsOpenReview ↗

Response to Reviewer ocSV (1/3)

Thank you for the review.

1. W1, Q1: Coverage with complete contexts, and utilization on matched questions

Complete-context coverage. The original MAB judge saw a 9,000-character excerpt around the reference-answer values (App. G.9). We re-judged all 2,071 MAB questions for all seven systems with the complete reader context, keeping the original answers.

SystemExcerptComplete context
Raw-RAG45.29%44.52%
Mem040.03%39.40%
EMem34.72%38.48%
MemMachine21.54%34.86%
MemOS34.48%33.90%
EverMemOS25.11%28.97%
MemU19.07%20.42%

Raw-RAG stays first, 5.12 points above Mem0 (paired 95% CI [3.19, 7.00], Holm p = 8.1e-6), and all six differences remain significant after Holm correction. The excerpt did affect systems unequally, as you suspected: MemMachine rises from 21.54% to 34.86% and EverMemOS from 25.11% to 28.97%, the two systems whose context was least visible to the original judge (9.24% and 17.91%). All analyses below use the complete-context labels.

Matched utilization. Each row compares the two systems on the same questions.

ComparatorShared questionsRaw-RAGComparatorDifference, pp (95% CI)Holm p
EverMemOS50285.26%82.27%+2.99 [-0.20, 6.18]0.912
Mem063184.47%84.47%0.00 [-2.85, 2.85]1.000
EMem60387.23%90.38%-3.15 [-5.80, -0.50]0.360
MemMachine58681.91%80.20%+1.71 [-1.19, 4.61]1.000
MemOS51985.93%87.67%-1.73 [-4.62, 1.16]1.000
MemU33580.90%87.76%-6.87 [-11.34, -2.39]0.077

On the 112 questions covered by all seven systems:

SystemCorrect / 112Accuracy
EMem10694.64%
Mem010190.18%
MemU10190.18%
MemOS10089.29%
MemMachine9786.61%
EverMemOS9383.04%
Raw-RAG9282.14%

A question counts as correct if its score is at least 0.5. Intervals use 50,000 within-source paired bootstrap replicates and are unadjusted; p-values come from two-sided exact McNemar tests, Holm-adjusted over all 21 pairwise comparisons among the seven systems.

On matched questions, Raw-RAG and each comparator differ by less than 7 points. EMem and MemU are higher than Raw-RAG before correction (their intervals exclude zero), and no Raw-RAG difference is significant after correction. On the seven-system intersection, Raw-RAG is last. This agrees with Finding 1: Raw-RAG's utilization is not the highest, and its lead comes from coverage. The intersection is small (5.41% of MAB) and mostly LongMemEval-S (98 of 112 questions).

How much of the gap comes from coverage. We split the 2,071 questions into four cells by the two systems' coverage labels. Cell cc contributes (nc/2071)(aRaw,c−acomp,c)(n_c/2071)(a_{\mathrm{Raw},c}-a_{\mathrm{comp},c}) to the question-weighted accuracy gap, and the four contributions sum to that gap. The gap uses the binary correctness above, so it differs from the task-weighted MAB Total.

ComparatorBoth coveredRaw-RAG onlyComparator onlyNeither
EverMemOS502420981,051
Mem0631291185964
EMem603319194955
MemMachine5863361361,013
MemOS519403183966
MemU335587881,061

Contribution of each cell to the gap (pp):

ComparatorBoth coveredRaw-RAG onlyComparator onlyNeitherTotal gap
EverMemOS+0.72+9.71-1.06+1.83+11.20
Mem00.00+4.78-2.12-1.79+0.87
EMem-0.92+7.77-1.88+4.01+8.98
MemMachine+0.48+9.90-1.35+5.70+14.73
MemOS-0.43+7.87-1.88-2.99+2.56
MemU-1.11+15.55-1.16+10.86+24.14

Jointly covered questions contribute -1.11 to +0.72 points. Raw-RAG alone covers 106 to 499 more questions than each comparator, and the two exclusive cells contribute a net +2.66 to +14.39 points; against EverMemOS they account for 8.64 of the 11.20-point gap. The neither-covered cell is also positive against four comparators, because Raw-RAG answers some questions correctly from evidence the judge rated insufficient. We do not attribute that cell to coverage.

Figure 5. We agree that the complete-store gaps are not pure ingestion loss. Removing the 5,000-token packing cap (App. A.5.2) still leaves all seven stores 7.58 to 29.38 points below raw evidence, so the cap does not explain them. To separate storage, selection, and the reader, we audited all 2,715 complete-store failures (details in our response to Reviewer UJ8J). Reader errors are similar across systems (204 to 273, except EMem at 118), while storage absence plus selection/packing omission ranges from 54 to 223 and largely follows the gaps in Figure 5.

AuthorsOpenReview ↗

Response to Reviewer ocSV (2/3)

2. W2, Q2: Memory-management ablations

Atomic facts with matched evidence. We let the atomic-fact variant pass facts to the reader up to each question's Raw-RAG evidence-token count, instead of a fixed top 15.

TaskRaw-RAGAtomic, top 15Atomic, matched tokens
Accurate Retrieval67.85%57.25%58.27%
Test-Time Learning47.68%15.37%16.04%
Long-Range Understanding31.74%21.17%21.83%
Conflict Resolution46.00%21.00%21.00%
Total48.32%28.70%29.29%

Matching the evidence volume recovers 0.59 of the 19.62-point loss, so volume explains almost none of the drop. The loss comes from the extraction step itself, which in TTL breaks up the demonstrations the task depends on (App. F).

Summary and graph. Neither reduces evidence volume. Each summary item holds the full raw window plus its summary, so it is longer than a raw chunk. Graph expansion changes which raw windows are retrieved, and the reader still sees raw windows. Without CR, their losses are 1.41 and 1.21 points (see our response to Reviewer 9LTs).

Scope. We agree that these are modifications of Raw-RAG rather than complete memory systems; complete systems are compared in the main tables. We will rephrase RQ2 as "Do these operations improve the fixed Raw-RAG pipeline?" and limit its conclusion to the tested implementations.

3. W2, Q3: Complex memory tasks and scope

PersonaMem-v2. Raw-RAG (49.62%) trails EverMemOS (53.25%) and MemOS (50.72%). These tasks require applying preferences aggregated across sessions, which favors systems that maintain a consolidated user profile. A profile component alone is not enough: MemU and MemoryOS also keep profiles and score 38.70% and 40.05%.

MEME. With DeepSeek-V4-Flash, Raw-RAG scores 43.19% overall, against 63.17% for Full-context, and 3.66% on cascades. The reader is not the bottleneck: given the phase-visible source sessions, it reaches 82.32% on cascades (Table 2). Similarity retrieval returns the changed fact but tends to miss the dependency rule or the downstream state it invalidates. No evaluated memory system exceeds 9.15% on cascades, so cascades remain unsolved by current designs, retrieval-based or structured.

AMA-Bench [1]. On the complete real-world subset, under the official Qwen3-32B answering and judging protocol, Raw-RAG reaches 52.80%. That is 10.53 points above the benchmark's own dense-retrieval baseline (Qwen3-Emb-4B), above HippoRAG2 and all eleven memory-agent baselines, and above the Qwen3-32B long-context reader (51.81%, Table 4 of [1]). It is 4.42 points below AMA-Agent (57.22%), which builds an explicit causality graph with tool-augmented retrieval. All rows other than Raw-RAG are the published Table 5 numbers of [1], obtained with the same backbone and judge.

MethodTypeRecallCausal inferenceState updatingState abstractionAverage
Raw-RAG(ours)RAG57.93%60.07%47.14%40.82%52.80%
AMA-AgentAMA-Bench authors62.38%61.45%53.05%47.19%57.22%
HippoRAG2RAG45.79%50.80%44.03%35.38%44.80%
Qwen3-Emb-4BRAG48.43%49.74%35.20%30.11%42.27%
BM25RAG33.01%42.64%34.50%24.98%34.36%
GraphRAGRAG30.77%39.05%31.40%28.79%32.58%
EMemMemory agent46.31%49.25%45.12%34.21%46.10%
MemoRAGMemory agent47.08%54.97%42.57%36.59%46.06%
MemoryBankMemory agent32.31%41.00%30.06%33.32%33.97%
MemGPTMemory agent32.89%44.04%28.09%25.26%33.04%
A-MemMemory agent30.84%36.53%30.88%28.73%31.86%
Mem-alphaMemory agent28.76%41.72%30.64%21.71%31.17%
HiMemMemory agent29.45%40.18%31.26%22.38%29.95%
MemAgentMemory agent25.50%33.80%28.49%22.02%27.68%
Mem0Memory agent20.11%26.45%21.01%15.16%21.04%
SimpleMemMemory agent20.12%18.84%17.64%13.73%18.11%
Mem1Memory agent11.80%14.27%12.05%10.80%12.29%

AMA-Agent leads Raw-RAG in all four capabilities, most on state updating (5.91 points) and state abstraction (6.37 points). By domain, Raw-RAG is weakest on Embodied AI, where state-updating questions are most common (150 of 360):

DomainRaw-RAG
Game69.72%
Open-world QA61.94%
Web56.18%
Text-to-SQL54.41%
Software40.97%
Embodied AI34.72%

MemoryArena [2]. The evaluation is still running because of the benchmark's size. Partial runs show Raw-RAG struggling on the math and physics reasoning tasks; we will post the full results here when they finish.

AuthorsOpenReview ↗

Response to Reviewer ocSV (3/3)

In these settings Raw-RAG stays within 1.5 to 4.4 points of the best memory system overall but does not lead. It is strongest when the answer depends on recovering specific details, and weaker when the task requires aggregating preferences, tracking dependent state changes, or abstracting over long trajectories.

On scope: our evidence is strongest for long-term conversation. We will state this scope in the abstract and introduction, retitle Section 5 "Why Is Raw-RAG Competitive on the Evaluated Conversational Benchmarks?", and report AMA-Bench as an out-of-domain test, including the cases where AMA-Agent is stronger.

4. W2, Q4: End-to-end cost and repeated queries

Table 6 excluded answer generation. Below we add it, report reader tokens, and measure repeated queries directly.

In the two tables below, times are in seconds. Total generative tokens cover construction and answering; n/a marks runs whose token logs were not kept. Latency break-even is the first number of queries at which a system becomes faster than Raw-RAG, under T(N)=A+NqT(N)=A+Nq with AA the construction time and qq the mean retrieval-plus-answer time; ∞ means never. Reader output tokens are similar across systems (1.9k to 2.4k on MEME, 17.6k to 21.3k on BEAM) and are omitted.

MEME, 10 histories, 110 queries

SystemConstructionRetrievalAnsweringTotalReader input tokens per queryTotal generative tokensLatency break-even
Raw-RAG50.32255.59230.57536.488,404926,781reference
EMem1,082.37479.59151.781,713.753143,842,889∞
MemOS2,591.59177.97157.242,926.801,055n/a1852
MemMachine3,104.35411.39180.973,696.715,068n/a∞
Mem04,025.23117.72153.924,296.871,255n/a2039
EverMemOS8,217.691,083.79154.179,455.651,922n/a∞

BEAM, 1 history, 20 queries

SystemConstructionRetrievalAnsweringTotalReader input tokens per queryTotal generative tokensLatency break-even
Raw-RAG1,174.2664.63205.531,444.4120,473428,619reference
MemOS11,698.0424.72192.0611,914.822,0204,750,7723,944
Mem014,206.4850.27189.2714,446.012,07716,881,3548,512
MemMachine30,706.4690.63197.9230,995.0116,043n/a∞
EMem65,501.85133.98185.1865,821.011,35412,756,578∞
Memora97,037.04243.40206.9397,487.3732,474n/a∞

Repeated queries, measured. We built each memory once on one complete LoCoMo conversation and answered all 152 of its questions in sequence. Generative tokens per query include query-time LLM calls (MemMachine, EverMemOS) and the reader's input and output. Token break-even is the first query count at which a system uses fewer cumulative generative tokens than Raw-RAG, projected as C(N)=B+NuC(N)=B+Nu with BB the construction tokens and uu the mean tokens per query.

LoCoMo, 1 conversation, 152 queries

MethodConstruction (s)Search + answer per query (s)Time after 152 queries (s)Construction tokensGenerative tokens per queryTokens after 152 queriesToken break-even
Raw-RAG6.9813.492,057.7806,460981,908reference
Mem01,880.6313.673,958.801,619,8593,9222,215,935639
MemMachine2,624.8214.674,854.582,304,4164,0752,923,777967
EverMemOS1,084.0321.634,372.21883,5538,5242,179,273∞

Raw-RAG has the lowest cumulative time at every prefix from 1 to 152 queries, and no comparator catches up, since none is faster per query. Tokens behave differently. Raw-RAG gives the reader more context (6.5k tokens per LoCoMo query against 3.9k to 4.1k for Mem0 and MemMachine; 8.4k per MEME query against 0.3k to 5.1k), which also makes its MEME answering step the slowest. Mem0 and MemMachine therefore use fewer generative tokens than Raw-RAG from the 639th and 967th query on the same history. EverMemOS never does, because of its query-time LLM calls. These counts exclude embedding and reranking and weight input and output tokens equally.

Raw-RAG's advantage is construction cost. In wall-clock time it holds at any query count on LoCoMo and for at least 186 queries per history on MEME and BEAM; in generative tokens it holds for the first 638 queries on a LoCoMo history. Past these points, systems with shorter reader contexts are cheaper.

References

[1] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. arXiv:2602.22769, 2026.

[2] He et al. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. arXiv:2602.16313, 2026.

Reviewer_UJ8JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a well-configured RAG pipeline over raw conversation history can compete with more complex memory systems. Raw-RAG combines dense retrieval and reranking, achieving the best reported score among compared memory systems in six of seven main settings while reducing ingestion cost. Further analyses suggest that preserving and retrieving supporting evidence is important to its performance.

Critical strengths
  1. The paper makes a useful practical point: improvements from complex memory systems should be measured against a strong retrieval baseline. The evaluation across several benchmarks and the retrieval ablations make this argument convincing.

  2. I appreciate the analysis of evidence coverage, especially the distinction between retrieving an answer value and retrieving the temporal or dependency relations needed to use it. This is relevant to designing memory components for agents.

Critical weaknesses
  1. The complete-store experiment does not fully isolate information loss during ingestion. Selection, ordering, and reader behavior still differ across representations, and Raw-RAG itself falls short of the raw-evidence control. The authors acknowledge this, but the explanation for the performance gaps remains less convincing than the main empirical result.

  2. The boundary of the finding needs more attention. On MEME with DeepSeek-V4-Flash, Raw-RAG scores 43.19 versus 63.17 for full context, with particularly poor cascade performance. It also trails the leading systems on PersonaMem-v2. These cases could provide more useful guidance about when additional memory organization helps.

Questions for authors
  1. In the complete-store failures, is the necessary evidence actually missing from storage, missed during selection, or present but used incorrectly by the reader? Have you audited examples to distinguish these cases?
  2. What explains the PersonaMem-v2 gap, and does it suggest a specific benefit from maintaining user profiles?
5 comments
AuthorsOpenReview ↗

Response to Reviewer UJ8J (1/2)

Thank you for the review.

1. W1, Q1: Where complete-store failures arise

We tried to label all 2,715 complete-store failures in Figure 5 with an LLM judge, but on manual review its labels were not reliable enough to report, mainly because it often missed errors made by the reader. (The sufficiency labels in Table 7 come from a judge that never sees the reader's answer, App. G.9; Table 8 uses deterministic checks.) Below we use what Figure 5 itself implies, together with failures we inspected by hand.

What the gap to raw evidence contains. The complete-store condition and the raw-evidence control share the gold signals, task policy, reader, and 5,000-token budget. They differ only in what they select from: the system's stored items, or the annotated source sessions. A system's gap to the control therefore combines evidence missing from its store with the cost of selecting and packing that store's items.

Raw-RAG stores the full history, so nothing can be missing from its store. Its 4.06-point gap is the selection and packing cost alone, and this is why Raw-RAG falls short of the control: it selects 256-token windows rather than whole annotated sessions. MemMachine, which also keeps the raw conversational content, falls 4.14 points short. The five systems that transform the conversation fall 10.01 to 30.54 points short. Selecting from transformed items may cost more than selecting raw windows, so this excess is not all storage loss, but storage loss is the one component the two raw stores cannot have. The control itself reaches only 61.60%, largely because readers still fail with the evidence in hand (cases 1 and 2).

Inspected failures. We checked each case against the source, the eligible store, the reader context, and the answer. The cases show how each mechanism arises, not how often.

  1. Reader error, invalidated value (Raw-RAG). Asked for the database connection string, the reader returned pyravar://data-primary:3306/aurora. Its context contained that string, the statement that it would change if the database changed, and the later update "We switched our database to Thalnex Store." The task rule required abstaining when the upstream entity changes without a new value. The evidence and the rule were both in context; the reader did not apply them.

  2. Reader error, incomplete aggregation (MemMachine). Asked "What do I do in my free time? List everything you know," the reader answered "Cycling, woodworking." Its context contained "My hobby is woodworking," "I play cycling," and "I'm part of a cycling club," and the task asked for hobby, sport, and club membership. The club was in the context, so better retrieval would not have helped.

  3. Storage absence, lost deletion request (EMem). The source said "I go to Quivira Springs Community College. Please forget that." EMem stored "The User attends Quivira Springs Community College as of March 22, 2023," and no removal instruction appears anywhere in its eligible store. Asked "Where am I studying?", the reader named the school. Gold-informed selection can retrieve the school fact, but not the instruction that it must not be disclosed.

  4. Storage absence, summary in place of content (EverMemOS). Asked to recite an error log verbatim, the reader answered "The User provided an exact error log regarding a PanicException related to an out-of-bounds issue." The source log contained the thread name, error values, call chain, and failover timestamp. The eligible store (EverMemOS's final persisted representation, filtered by source provenance) kept only summaries; tokio-runtime-worker and 2024-04-02T14:18:33Z, for example, appear nowhere in it.

In the revision, Finding 3 will state that the gap to raw evidence includes a selection and packing cost, about 4 points for the two raw stores, and Appendix A.5.2 will include these cases.

AuthorsOpenReview ↗

Response to Reviewer UJ8J (2/2)

2. W2, Q2: Where Raw-RAG falls short

PersonaMem-v2. Raw-RAG (49.62%) trails EverMemOS (53.25%) and MemOS (50.72%), both of which maintain user profiles, and the tasks require applying preferences aggregated across sessions. A profile component alone does not explain the gap: MemU and MemoryOS also maintain profiles and score 38.70% and 40.05%, below Raw-RAG. The results support a benefit from well-built profiles, not from having a profile component.

MEME. With DeepSeek-V4-Flash, Raw-RAG scores 43.19% overall, against 44.71% for MemMachine and 63.17% for Full-context, and 3.66% on cascades. The reader is not the bottleneck: given the phase-visible source sessions, it reaches 82.32% on cascades (Table 2). Similarity retrieval returns the changed fact but tends to miss the dependency rule or the downstream state it invalidates. No evaluated memory system exceeds 9.15% on cascades, so cascades remain unsolved by current designs, retrieval-based or structured. Table 4 shows two further cases where organized memory wins with DeepSeek-V4-Flash: aggregation (EMem 36.00% vs. 21.00%) and deletion (MemMachine 48.00% vs. 29.00%).

Agent trajectories. On AMA-Bench [1], whose questions concern trajectories of states, actions, observations, and tool outputs, Raw-RAG reaches 52.80% under the official Qwen3-32B protocol. That is above every RAG and memory-agent baseline in Table 5 of [1] (best: EMem, 46.10%), but 4.42 points below AMA-Agent (57.22%), which builds an explicit causality graph. AMA-Agent's lead is largest on state updating (5.91 points) and state abstraction (6.37 points). By domain, Raw-RAG is weakest on Embodied AI (34.72%), where state-updating questions are most common (150 of 360). Partial MemoryArena [2] runs show Raw-RAG struggling on the math and physics reasoning tasks; we will post the full results when the evaluation finishes.

In these settings Raw-RAG stays within 1.5 to 4.4 points of the best memory system overall but does not lead. It is strongest when the answer depends on recovering specific details, and weaker when the task requires aggregating preferences, tracking dependent state changes, or abstracting over long trajectories. We will state these boundaries in Section 6.

References

[1] Zhao et al. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. arXiv:2602.22769, 2026.

[2] He et al. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks. arXiv:2602.16313, 2026.

AuthorsOpenReview ↗

Follow-up Response to Reviewer UJ8J

Thank you again for your feedback. We provide two updates below. First, we have completed a human audit of the complete-store failures you asked about (W1, Q1). It supplies the breakdown our earlier automated attribution could not. Second, we have now completed the MemoryArena evaluation for the math and physics reasoning subsets, which bears on the boundary issue you raised (W2). Evaluations on the remaining MemoryArena subsets are still ongoing.

W1, Q1: Where complete-store failures arise

Setup. We audited 140 failed instances from the system-question phase. The sample is stratified by system and task, with 20 instances per system. One author inspected each instance (its source history, the evidence available in the store, the reader's context, the task instructions, and the system's answer) and manually assigned a failure type.

Results by system. Stated as percentages of audited instances (20 per system; 140 in total):

SystemStorage insufficiencySelection/assembly omissionReader errorScoring disputeUnresolved
Raw-RAG0.0%0.0%85.0%10.0%5.0%
MemMachine25.0%5.0%65.0%0.0%5.0%
EverMemOS35.0%5.0%50.0%10.0%0.0%
EMem35.0%0.0%60.0%0.0%5.0%
MemOS30.0%0.0%65.0%5.0%0.0%
Mem050.0%0.0%45.0%5.0%0.0%
MemU30.0%10.0%55.0%0.0%5.0%
Overall29.3%2.9%60.7%4.3%2.9%

How the labels are defined.

  • Storage insufficiency: the evidence the question requires is absent from the eligible exported store.
  • Selection/assembly omission: that evidence is present in the store but was never passed to the reader.
  • Reader error: the reader received sufficient evidence and instructions, yet the answer was wrong.
  • Scoring dispute: closer inspection accepts the system's answer.
  • Unresolved: the case could not be labelled confidently from the available artifacts.

Reading the aggregate. Reader error is the largest category overall, but that answers a different question: not why complete-store access falls short of raw evidence, but why it fails at all. To isolate the raw-evidence effect, we split the same audited failures by whether the raw-evidence control had answered the corresponding phase correctly.

Human-assigned categoryRaw correct, complete-store wrong (n=65)Both wrong (n=75)
Storage insufficiency49.2%12.0%
Selection/assembly omission3.1%2.7%
Reader error36.9%81.3%
Scoring dispute6.2%2.7%
Unresolved4.6%1.3%

Storage insufficiency accounts for 49.2% of the first group (32/65), compared with 12.0% of the second (9/75). Among the cases where raw evidence succeeded and complete-store access did not, it is the most frequent category, and its share is 37 percentage points higher than in the both-wrong group. When raw evidence succeeded, missing evidence becomes the leading cause. This concentration supports insufficient retained evidence as an important explanation for the raw-evidence advantage.

We will add this audit to our paper.

W2: Completed MemoryArena results for math and physics

The partial runs reported earlier have now been completed for both subsets (40 math groups/354 sessions; 20 physics groups/86 sessions). Raw-RAG, MemoryArena RAG, EverMemOS, and Mem0 were run locally under identical Qwen3.8-Flash-Next (BF16, thinking disabled) prompts, session orders, and final-verdict parsing. Metrics: final-session success (last session correct per group), mean group progress (mean within-group accuracy), and session accuracy (all sessions equally weighted).

SubsetMethodFinal-session successMean group progressSession accuracy
MathRaw-RAG30.00%35.31%33.05% (117/354)
MathMemoryArena RAG30.00%35.35%32.77% (116/354)
MathEverMemOS25.00%35.60%33.90% (120/354)
MathMem022.50%33.96%32.20% (114/354)
PhysicsRaw-RAG45.00%50.76%45.35% (39/86)
PhysicsMemoryArena RAG30.00%52.31%43.02% (37/86)
PhysicsEverMemOS60.00%58.81%50.00% (43/86)
PhysicsMem050.00%53.47%48.84% (42/86)

On math, Raw-RAG ties for the best final-session success (30.00%) and session accuracy is tightly clustered (32.20%–33.90%), so the partial-run impression of a math weakness does not hold. On physics, EverMemOS and Mem0 lead Raw-RAG on all three metrics (EverMemOS: 60.00% vs. 45.00% final-session success; 50.00% vs. 45.35% session accuracy). This supports the task-dependent boundary you raised. Structured memory can help on interdependent reasoning tasks, while Raw-RAG remains a strong baseline. We will include these results alongside AMA-Bench in the revision and share the remaining subsets when complete.

Reviewer_9LTsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAW-RAG is a memory pipeline that uses raw history, with timestamps and image captions is split into 256-token windows with 32-token overlap and embedded into a flat per-conversation index. Then it retrieves 128 candidates using cosine similarity, reranks them using a cross-encoder, and supplies the top 15 to the LLM.

Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, RAW-RAG obtains the highest score among the compared memory systems in 6 out of 7 benchmark–model settings. The paper also investigates why RAW-RAG performs well. The paper concludes that evidence access and preservation often matter more than elaborate memory management.

Critical strengths
  1. The paper’s finding that evidence availability varies far more across systems than conditional answer accuracy does gives an actionable insight for designing better memory systems.
  2. The investigation of temporal and dependency-path preservation.
Critical weaknesses
  1. Most baseline rows, including Full-context, are published numbers from other papers. Each used its own reader prompt, judge, and evidence budget, so no paired test is possible.
  2. For Conflict Resolution’s FactConsolidation sources, Raw-RAG indexes one numbered fact per chunk with no overlap (Appendix B.1). This is a task-specific choice rather than a default. The adaptation fixes CR in every chunk-size ablation (Table 5). CR also accounts for most of the RQ2 losses.
  3. Works like SmartSearch [1], Back-to-basics [2] overlap a lot with RAW_RAG motivation and diagnostic principles and the authors should explain precisely what additional knowledge the RAW-RAG establishes and include a comparison where feasible.
  4. LongMemEval [3] already showed that storing summaries or extracted facts in place of raw sessions lowers QA accuracy because information is lost. That is close to Finding 3 and the RQ2 ablations.

[1] SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval

[2] Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation

[3] LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Questions for authors
  1. Did you run EverMemOS, MemMachine, and the current Mem0 release on LoCoMo and LongMemEval-S, under exactly the same reader prompt, judge, and evidence budget as Raw-RAG?
  2. What does Full-context score on LoCoMo with GPT-4o-mini and GPT-4.1-mini, and on LongMemEval-S with GPT-4.1-mini?
  3. What median share of the reader's context did the judge see?
  4. How were MEME sufficiency labels actually produced?
  5. Does path completeness predict answering success beyond value coverage?
  6. Where does explicit state maintenance help?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 9LTs (1/2)

1. W1, Q1: Local Baseline Results

We evaluated the requested baselines locally under aligned conditions: the same LLM, embedding model, reranker, reader prompt, 5,000-token evidence budget, and GPT-4o-mini judge, averaging three judgments per answer. We preserved each system's native memory mechanisms. These are complete-system comparisons under shared model and evaluation settings.

For Mem0, we used the stable release 2.0.11, consistently with our MEME and MemoryAgentBench evaluations. This is not the newest 2.2.1 release, but belongs to the same 2.x major release series.

LoCoMo / GPT-4o-mini.

MethodSingle-hopMulti-hopTemporalOpen-domainOverall
Raw-RAG94.57%86.29%82.04%73.61%89.13%
EverMemOS83.99%81.56%71.13%69.44%79.96%
Mem093.18%84.16%83.49%72.22%88.20%
MemMachine89.34%80.61%72.38%53.13%81.95%

LoCoMo / GPT-4.1-mini.

MethodSingle-hopMulti-hopTemporalOpen-domainOverall
Raw-RAG96.79%92.32%90.86%80.21%93.70%
EverMemOS87.79%87.23%81.20%71.53%85.30%
Mem094.81%89.48%87.54%75.35%91.10%
MemMachine89.73%81.80%73.10%52.08%82.47%

LongMemEval-S / GPT-4.1-mini.

MethodSingle-session userSingle-session assistantSingle-session preferenceMulti-sessionKnowledge updateTemporal reasoningOverall
Raw-RAG98.57%98.21%96.67%74.94%95.30%89.47%89.20%
EverMemOS98.57%100.00%93.33%75.19%95.30%69.92%84.07%
Mem095.71%62.50%96.67%71.43%81.20%55.14%72.53%
MemMachine94.29%100.00%60.00%61.15%82.05%51.88%70.87%

As for MEME and MAB (MemoryAgentBench), the results in our paper are already aligned local run products.

2. W1, Q2: Full-Context Results

BenchmarkGPT-4o-miniGPT-4.1-mini
LoCoMo82.71%94.59%
LongMemEval-S--80.53%

We will replace the literature-derived Full-context entries with these local measurements.

3. W2: Conflict-Resolution Chunking

In FactConsolidation, each fact is already shorter than our chunk-size, so there is no need to split it. Technically, we could concatenate the facts and then apply the default sliding window like other subset, but neighboring facts in CR are not semantically related, so we therefore index each fact as a separate chunk. This task-specific choice remains implemented in the compared variants, so we didn't consider it as an unfair setting.

However, we agree that CR does not provide an equally favorable test for all variants. Its inputs are already brief, so summarization or atomic-fact extraction may offer little benefit. We therefore exclude CR and report the remaining results below.

Memory-operation ablation excluding CR.

ConfigurationAccurate RetrievalTest-Time LearningLong-Range UnderstandingMean excluding CR
Raw-RAG67.85%47.68%31.74%49.09%
+ Atomic facts57.25%15.37%21.17%31.26%
+ Raw summary67.53%45.89%29.62%47.68%
+ Graph-expanded retrieval65.42%46.49%31.74%47.88%

4. W3, W4: Differences from Related Work

SmartSearch [1] and Back to Basics / Nano-Memory [2]. We share their motivation for simplifying memory. Beyond a minimal dense-retrieval-and-reranking pipeline, we contribute:

  1. Broader tasks: MEME tests state updates, deletion, and cascades; MemoryAgentBench tests test-time learning (TTL), long-range understanding (LRU), and conflict resolution (CR). SmartSearch evaluates LoCoMo / LongMemEval-S; Nano-Memory evaluates LoCoMo / Long-MT-Bench+ / LongMemEval.

  2. Mem0 mutation audit: We distinguish proposed from effective mutations and test disabling UPDATE/DELETE, with conclusions limited to the evaluated configuration.

  3. Seven-system coverage/utilization decomposition: We separate evidence availability from conditional answer accuracy, supplemented by full-context judgments and jointly covered question comparisons on MAB.

  4. Path completeness: We distinguish answer-value coverage from temporal/dependency-path preservation, with an additional timestamp ablation testing the contribution beyond value availability.

  5. Complete-store oracle and failure attribution: Gold-informed access to seven complete stores tests recoverability; an additional audit of 2,715 failures separates storage absence, selection/packing omission, and reader error. The oracle gap alone does not isolate ingestion loss.

Direct comparisons with SmartSearch and Nano-Memory are ongoing.

AuthorsOpenReview ↗

Response to Reviewer 9LTs (2/2)

LongMemEval. We agree that its compression results anticipate Finding 3's general principle and our atomic-fact ablation. Our contribution is an extension. Finding 3 tests whether gold-informed access to seven systems' complete stored representations closes the gap to raw evidence. RQ2 additionally tests summary augmentation and graph-expanded retrieval that retain raw text, so their losses are not simply compression losses, and extends evaluation to demonstration-sensitive test-time learning.

5. Q3: Judge Context Visibility

This table shows the median visible share of the judge:

SystemMABMEME
Raw-RAG52.12%4.89%
EverMemOS17.91%81.29%
MemMachine9.24%6.17%
Mem095.83%38.31%
MemOS100.00%35.00%
EMem100.00%100.00%
MemU100.00%100.00%

We originally used reference-answer-localized excerpts because complete evidence could nearly fill the judge's context window, raising concerns about judgment reliability. We have now reassessed every MAB question using the complete reader evidence, giving 100% visibility while keeping the original answers fixed:

SystemFull-context coverage
Raw-RAG44.52%
Mem039.40%
EMem38.48%
MemMachine34.86%
MemOS33.90%
EverMemOS28.97%
MemU20.42%

Raw-RAG retains the highest coverage, exceeding every comparator after Holm correction.

6. Q4: MEME Labels and Path Completeness

The main MEME coverage table uses deterministic task-specific checks. The semantic audit uses an LLM judge, with complete contexts for the chain-sensitive reassessment. Requirements are: target values for recall; all requested values for aggregation; chronological order for tracking; explicit deletion events; and linked rules, triggers, and target states for cascade/invalidation. Benchmark annotations decide the requirements.

7. Q5: Path Completeness

Thank you for distinguishing relational completeness from its incremental predictive value. Our original table measures �0��0� and does not establish whether path completeness predicts answering success beyond value coverage.

To test whether temporal information contributes beyond answer-value availability, we conducted a paired timestamp-ablation experiment on all 95 Raw-RAG post-update tracking questions previously judged value- and path-complete. We compared the retrieved contexts with versions in which only the path were masked, leaving all other text unchanged. Accuracy fell from 71/95 (74.74%) to 29/95 (30.53%), a paired difference of 44.21 percentage points (95% episode-bootstrap CI: 34.74 to 53.68).

These results suggest that path completeness captures an additional contribution to answering success beyond answer-value coverage.

8. Q6: Where Explicit State Maintenance Helps

Thank you for emphasizing the importance of identifying the boundries of Raw-RAG and we will discuss where Raw-RAG does not have an advantage.

PersonaMem-v2. Raw-RAG scores 49.62%, versus 53.25% for EverMemOS and 50.72% for MemOS. A plausible explanation is that these tasks emphasize understanding user profiles and integrating preferences across sessions rather than recalling isolated details. Many agentic memory systems, including EverMemOS, MemOS, MemU, and MemoryOS, include dedicated profile components for storing and maintaining user profiles, which may be better aligned with these tasks. These results suggest a potential benefit from profile abstraction and maintenance.

MEME. With DeepSeek-V4-Flash, Raw-RAG scores 43.19% overall versus Full-context's 63.17%, MemMachine's 44.71%, and only 3.66% on cascades. One possible explanation is that embedding similarity retrieves semantically related evidence but does not explicitly trace logical dependencies. Retrieval may therefore omit links needed to reconstruct the current state after a change, while the reader must still apply the relevant update and invalidation rules correctly. Explicit dependency organization may help in such settings.

Our preliminary interpretation is that Raw-RAG tends to be less effective when preference aggregation or complex dependency reconstruction is required(but actually Raw-RAG can still remain competitive in these settings although not SOTA). We will make these boundaries more explicit in the paper.

References

[1] Derehag, Calva, and Ghiurau. SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval. arXiv:2603.15599, 2026. Sections 3-5 and Appendix D.

[2] Wu et al. Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation. arXiv:2604.11628, 2026. Sections 3-4 and Appendices A-C.

AuthorsOpenReview ↗

Follow-up Response to Reviewer 9LTs

Following your request to compare with SmartSearch and Back to Basics, we report additional LoCoMo results below.

Evaluation protocol. These runs use the same protocol as the LoCoMo experiment in our response to Q1. All systems answer the same 1,540 non-adversarial questions (category 5 excluded) with the same reader LLM, embedding model, reranker, reader prompt, 5,000-token evidence budget, and judge as Raw-RAG. Each system keeps its own memory construction and retrieval.

LoCoMo, GPT-4o-mini

MethodSingle-hopMulti-hopTemporalOpen-domainOverall
Raw-RAG94.57%86.29%82.04%73.61%89.13%
Nano-Memory (Back to Basics)82.76%64.66%61.58%63.54%73.83%
SmartSearch89.77%80.02%81.10%58.33%84.22%

LoCoMo, GPT-4.1-mini

MethodSingle-hopMulti-hopTemporalOpen-domainOverall
Raw-RAG96.79%92.32%90.86%80.21%93.70%
Nano-Memory (Back to Basics)83.00%63.83%61.47%62.50%73.72%
SmartSearch92.79%90.78%88.37%77.08%90.52%

Raw-RAG outperforms both baselines in both settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.