acceptodds
Under review as a conference paper at ICLR 2027

Less Context, More Accuracy: Structured Long-Term Memory Beyond Equal-Budget RAG

Abstract

Long-term memory for LLM agents is often justified against replaying the entire interaction history, which is expensive and can become less accurate as distractors accumulate. Yet outperforming full context does not establish that structured memory adds value beyond ordinary retrieval: a token-matched RAG baseline may recover the same gain. Existing evaluations also change the reader, judge, or context budget together with the memory system, making gains difficult to attribute and reproduce. We present Engram, an open-source, dual-process memory engine built on a bi-temporal data model. A fast write path appends lossless episodes without an LLM on the critical path; an asynchronous consolidation path extracts atomic (subject, predicate, object) facts, builds a bi-temporal knowledge graph, and resolves contradictions without an LLM call per fact—invalidating, never deleting, so every fact retains provenance and a supersession chain. A hybrid read path fuses dense, lexical, graph, and recency/salience signals, applies a point-in-time (“as-of”) temporal filter, and assembles a compact, provenance-tagged context. We evaluate frozen configurations on all 500 questions of LongMemEval-S with the official category-specific judge prompts. Streamlined Engram reaches 78.8%, significantly outperforming an equal-budget dense+BM25 RAG control at 72.4% (+6.4 points; McNemar p=0.001) and full context at 61.6% (+17.2 points), while using 6.7k rather than 103.5k mean context tokens (15.5x compression). The 86.0% annotated-evidence oracle is +7.2 points higher. The predefined 440-question held-out partition preserves the margins. Five single-variable interventions show that gains are category-specific and that an evidence planner and hierarchy blocks reduce aggregate accuracy; repeated generations and a 2x2 reader-judge cross test robustness. A separate streaming experiment on all 500 LongMemEval-M questions measures the shared episode index without conflating retrieval recall with QA accuracy. Every reported value is generated from fingerprinted, append-only context, answer, and judgment artifacts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-term memory, LLM agents, retrieval-augmented generation, bi-temporal knowledge graphs, reproducible evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CMMqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ENGRAM, a structured long-term memory system that asynchronously extracts and consolidates facts into a bi-temporal knowledge graph, and retrieves a combination of structured facts and raw conversation chunks. The authors evaluate ENGRAM on LongMemEval-S against full-context prompting and an equal-budget RAG baseline. Results show that ENGRAM achieves 78.8% accuracy, compared with 72.4% for RAG and 61.6% for full context, while using substantially fewer context tokens than full history.

Critical strengths
  1. Comparing structured memory against an equal-budget RAG baseline is important for understanding whether the benefits come from memory organization or simply better context selection.
  2. Strong reader models and retrievers are considered.
  3. The empirical analysis is thorough. The authors provide category-level results, component ablations, statistical significance tests, and evaluations across different readers and judges.
Critical weaknesses
  1. The evaluation is limited to LongMemEval. While the authors additionally report retrieval results on LongMemEval-M, end-to-end QA is only evaluated on LongMemEval-S. It would be helpful to include other memory benchmarks such as LoCoMo [1] and PersonaMem [2].
  2. The technical contribution is somewhat limited. The proposed system largely combines existing techniques, including fact extraction, temporal knowledge graphs, and hybrid retrieval.
  3. The comparison with existing memory systems is insufficient. While the equal-budget RAG baseline is useful, the paper does not directly compare ENGRAM with established memory systems discussed in the related work, such as Mem0, Zep/Graphiti, or HippoRAG.
  4. The efficiency analysis is incomplete. The paper reports context token usage and answer-generation latency, but does not fully account for the cost of memory construction and consolidation.

References

[1] Maharana et al. Evaluating Very Long-Term Conversational Memory of LLM Agents. ACL 2024.

[2] Jiang et al. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. COLM 2025.

Questions for authors

Please refer to the weaknesses

Reviewer_65SCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents Engram, a memory system combining raw conversation episodes, extracted facts, a bi-temporal knowledge graph, and hybrid retrieval. It separates fast episode ingestion from asynchronous consolidation and preserves superseded facts for historical queries. On LongMemEval-S, the planner-free configuration achieves 78.8% accuracy, compared with 72.4% for dense+BM25 RAG under the same token cap and 61.6% for full context. The paper also reports component interventions, reader–judge comparisons, and a separate retrieval-only experiment on LongMemEval-M.

Critical strengths
  • The comparison against budget-controlled hybrid RAG is useful. Holding the reader, judge, answer prompt, embedder, and context cap fixed provides a more informative comparison than evaluating only against full history.

  • The paper reports negative component results openly. Showing that planning and hierarchy can reduce accuracy is useful for understanding the system’s behavior. The paired results and category breakdowns also make the findings easier to assess.

Critical weaknesses

My first concern is that the experiments do not isolate the contribution of the proposed memory structure. Engram receives LLM-extracted facts and additional context-assembly mechanisms, whereas the RAG control retrieves raw material without those transformations. Consequently, the 6.4-point gain could reflect fact extraction and compression without requiring the proposed graph or temporal-state machinery. Table 3 reinforces this concern: removing graph retrieval increases accuracy from 75.4% to 77.0%, and removing temporal reconciliation increases it to 78.0%. These interventions use the complete configuration, so their effects on the headline planner-free system remain unknown. Moreover, bi-temporal facts, source provenance, and hybrid graph retrieval already feature in Zep/Graphiti; the paper provides no controlled comparison against this closest design. (The current evidence therefore leaves the distinct benefit of Engram’s memory design unresolved.

My second concern is that the selection and robustness checks do not consistently cover the headline configuration. Table 1 implies that the complete system answers 49/60 development questions correctly, while the planner-free system answers 48/60. The paper nevertheless headlines the latter because it performs best in the final matrix. Freezing all candidate configurations before evaluation does not establish that the headline candidate was selected independently of the 440 held-out questions. The selection criterion and timing need clarification. In addition, Tables 4 and 5 evaluate the 75.4% complete system rather than the 78.8% planner-free system. Table 4 repeats no RAG baseline, and Table 5 shows margins over RAG of only 1.2 and 1.0 points with Reader 2. These checks provide limited support for the robustness of the headline 6.4-point improvement.

The final point concerns the novelty of the method, which is also why I awarded a score of 1 for Informativeness. Zep/Graphiti already combines bi-temporal facts, non-destructive invalidation, provenance, and hybrid retrieval. Engram’s cheap-then-escalate conflict-resolution policy could provide a meaningful distinction, but the paper offers neither a targeted ablation nor a direct comparison demonstrating its accuracy–cost advantage. The methodological advance beyond combining established components therefore remains unclear.

Questions for authors
  • Was the planner-free system selected before evaluating the 440 held-out questions? What development-set criterion selected it?

  • Were graph retrieval and temporal reconciliation also ablated in the 78.8% planner-free system? If so, what were the scores?

Reviewer_N3LFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the debate over the utility of structured memory versus ordinary retrieval in LLM memory systems. It identifies two methodological gaps: the advantage of selective context over full context cannot be cleanly attributed to structured memory or to the retriever under the same token budget, and existing memory benchmarks lack strong reproducibility with respect to ingestion, answer prompts, and judges. The paper therefore asks: after controlling for generic context selection, does structured long-term memory still improve answer quality? The answer suggests that a filtered and precisely retrieved slice can be both more accurate and more efficient than a full-context alternative. This connects directly to the paper’s long-term vision of building a scalable and auditable long-term memory system. The paper introduces ENGRAM, a dual-process memory system whose design is motivated by the System-1/System-2 division: System 1 mainly preserves lossless episodes, analyses across conversations, and computes a light embedding for asynchronous embedding, while System 2 extracts atomic triplets to build an entity-relation graph that supports conflict handling and salience computation. The evaluation uses all 500 LongMemEval_S questions with the official category-specific judge prompts. It compares four systems (annotated-evidence oracle, full context, token-matched dense+BM25 RAG, and engram_final) plus five single-variable interventions. The authors make five main claims:

A selective context window not only saves tokens but can achieve higher accuracy than full context. Streamlined ENGRAM reaches 78.8% versus 61.6% for full context (+17.2 points), using 6.7k versus 103.5k mean context tokens. Structured memory adds gains beyond general retrieval. Against the equal-budget dense+BM25 RAG control, streamlined ENGRAM reaches 78.8% versus 72.4% (+6.4 points, McNemar p = 0.001). The additional gains from ENGRAM are selective across question types and mechanisms. Relative to RAG, the largest category shifts are +36.7 points on preference and +9.0 points on temporal reasoning, versus −3.6 points on single-session assistant questions. The human-labelled evidence oracle indicates remaining retrieval headroom. The oracle reaches 86.0%, 7.2 points above streamlined ENGRAM. The evaluation protocol is highly reproducible.

Critical strengths
  1. The equal-budget control principle is highly valuable. The RAG baseline shares ENGRAM’s BGE embedder and 8k-token cap, and the reader, judge, answer prompt, and questions are held fixed across systems. Under this control, RAG alone recovers +10.8 points over full context, which lets the paper separate gains from generic context selection from gains attributed to structured memory.

  2. The auditing materials are highly valuable. The numerical values in the paper show strong internal auditability after verification by re-running final_results.py, which is a creditable practice. Contexts, answers, and judgments are stored as separate append-only JSONL artifacts joined by SHA-256 (4,500 contexts, 8,000 answers, 16,000 judgments), and all paper values are generated from them by paper/final_results.py.

Critical weaknesses

The strongest result, the +6.4-point margin over RAG, seems to originate from a selected setting. The main text specifies engram_final as the complete configuration, while no_planner is one of the five full-set interventions. The +6.4-point margin is the paper’s evidence for its first contribution, that structured memory adds value beyond equal-budget RAG. For the pre-specified engram_final configuration, Table 1 reports 75.4% versus 72.4% for RAG on all 500 questions, and 74.5% versus 72.5% on the 440 held-out questions.

Questions for authors
  1. Was the 78.8% no_planner configuration specified as the headline system before evaluation on the 440 held-out questions? Section 4 states that all candidates were frozen on the 60-question development set before final grading. It describes engram_final_no_planner as the strongest frozen automatic configuration in the final matrix of full-set interventions.
  2. Does the current experiment measure persistent long-term memory, or structured compression driven by the test questions? The answer directly affects whether the advantage can be attributed to persistent structured memory rather than to query-time context compression. Section 4 states that the context builder extracts facts from at most 8 retrieved sessions, while Section 3.3 places fact extraction in asynchronous consolidation off the read path.
  3. Which structured components actually contribute to the headline advantage? The current single-variable interventions are anchored on the 75.4% result of the complete configuration, whereas the actual headline is the 78.8% configuration with the planner removed. All four removals in Table 3 score above the complete configuration (75.8%, 78.8%, 77.0%, and 78.0% versus 75.4%), and no intervention starts from engram_final_no_planner.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.