acceptodds
Under review as a conference paper at ICLR 2027

MemFit: Efficient Long-Term Agentic Memory

Abstract

Long-term memory systems for large language models (LLMs) have gained popularity for extending reasoning capabilities across applications. Current memory systems rely on LLM agents to organize and consolidate memory, resulting in costly, inefficient write operations. To address this limitation, we propose MemFit, a long-term memory system for conversational agents that reduces the cost and latency of memory operations. Unlike existing systems that rely on expensive LLM calls for memory construction or discard surface-level details through compression, MemFit stores each turn verbatim in an append-only store with near-instantaneous, LLM-free insertion, indexing turns with segment summaries rather than replacing them. Additionally, MemFit uses an LLM-free, multi-path retrieval strategy that combines lexical and semantic signals with cross-encoder reranking over caption-augmented episodes in both textual and multimodal settings. Empirical results on three widely used benchmarks, LoCoMo, MemGallery, and LongMemEval-S, show that MemFit achieves state-of-the-art performance while reducing memory construction time and cost several-fold, providing a scalable and efficient solution for persistent agentic memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Agentic AI, Agentic Memory Systems, LLM Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nsfiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

MemFit removes the LLM from the write path and from retrieval ranking, relying on verbatim turn storage, dual BM25 and dense vector indexes, segment summaries used as an index, and cross-encoder reranking to achieve low-cost memory construction and precise retrieval. It achieves competitive results on the LoCoMo, MemGallery, and LongMemEval-S benchmarks across multiple backbone models, while showing significantly efficiency than comparable baselines.

Critical strengths

MemFit achieves competitive performance against multiple baselines on the LoCoMo, MemGallery, and LongMemEval-S benchmarks, while requiring significantly fewer LLM token cost / inference time than those baselines.

Critical weaknesses
  1. The font in Fig. 1 is too small, especially in the “memory insertion append only” part, hard to follow.
  2. Although the authors claim “LLM free retrieval,” summary expansion clearly depends on LLM generated summaries. The quality of these summaries also appears to have a significant impact on the candidate pool. Moreover, authors emphasize “no consolidation,” the summary layer in Memfit is essentially a form of consolidation.
  3. The authors do not clearly explain why consolidation in iterative retrieval baselines such as Mem0, SeCom, and MemTree is unnecessary (in related work). The discussion in this part is unclear.
  4. Segmenting still requires an LLM call, so this is not “no LLM call per turn.” Also, why is no consolidation necessarily a reasonable design choice?
  5. The 'additive context window' design is naive. After retrieving an episode, the method adds neighboring turns from the same session to the pack with w=1w=1. However, ww is fixed and does not adapt to session length, turn type, or query. No sensitivity analysis about this parameter in the experiments.
  6. Overclaim. Neither authors claimed “LLM free write path” nor the “hybrid retrieval and cross encoder reranking” seems novel. The motivation for the former overlaps with that of many existing memory works, while the latter is a standard practice in information retrieval.
  7. The authors claim the “summary is an index, not compression,” but do not explain why using the summary as an index is better than using it as a memory unit. In my opinion, the 'index' in this paper is also a type of summary.
Questions for authors

Please see Weaknesses

1 comment

Thank you for the review.

(4) Segmented does not require an LLM call. Segmentation uses TextTiling over the stored embeddings (Sec. 3.2), instead of an LLM. The only write-time LLM cost is one summary per segment, which costs about 437 calls for LoCoMo's 5,882 turns.

Consolidation decides at write time which version of a fact survives, before any question is known, and it cannot recover the discarded evidence. Many questions need exactly what consolidation discards. MemFit keeps all interactions, with their dates, and lets the LLM process them correctly when the question is known. Four results support this:

  1. The reader resolves conflicts reliably. On LongMemEval-S knowledge-update questions, MemFit retrieved the outdated and the current value for 67 of 72 questions, and the reader chose the current one 79.1% of the time (Response to R1, W2).
  2. Consolidating baselines do worse on exactly these questions. MemFit's knowledge-update accuracy (78.2%) is above Mem0 (70.1), LangMem (66.7), A-MEM (64.1), and MemoryOS (48.7); only LightMem is higher.
  3. Consolidation did not help when we added it. We built an LLM reconciliation layer over stored memories and measured it; it gave no gain on LoCoMo or MemGallery, so MemFit does not include one.
  4. Consolidation is the main write-time cost. It is why Mem0, A-MEM and SimpleMem issue 10,554–15,736 LLM calls on LoCoMo against MemFit's 1,977 (Fig. 3).

(5) Regarding "No sensitivity analysis of w.” The paper does include a sensitivity analysis:

  • w=0 is the "w/o context window" ablation. Pooled over three readers, it costs 2.58 judge points (p<0.001), the largest single component.
  • w=2 is in the sensitivity study (Appendix Table 10): +1.82 (p=0.036), not significant after Holm correction over eight settings, at 1.44× the input tokens. On MemGallery, it changes the judge score by +0.06.

The fixed context window is intentionally simple because making it adaptable requires additional LLM calls or a learned or thresholded rule, which requires tuning per dataset and reduces practical applicability.

(2) Regarding "LLM-free retrieval" and summary quality, "LLM-free retrieval" means search and ranking do not use an LLM call. The single call per question is the answer. We write summaries once at storage time. On summary quality, the paper contains a direct test: replacing summaries written by Qwen3-8B with ones written by Gemma3-27B did not change accuracy (judge 64.4 vs. 64.6, p=0.77; Appendix C). The new experiment below shows that on LoCoMo summaries matter only as an index.

Regarding "the summary layer is consolidation”, we use "consolidation" in the same way the memory literature does. “Consolidation” means operations that rewrite, merge, or delete stored memories (Mem0's UPDATE/DELETE, A-MEM's note evolution, MemoryBank's forgetting). MemFit never does this. Summaries are added beside the turns. We agree that summarizing is a form of abstraction. The case against destructive consolidation is that it must choose what to keep before the question is known, and errors are unrecoverable (see Response to R1, W2).

(7) Regarding why we use summaries as an index rather than as memory units, we present new results using the same summaries, same retrieval, and same reader (LoCoMo, Gemma3-27B):

Summaries used asAvg F1JudgeTokens per question
Memory units: reader sees summaries only14.8819.431,081
Units: reader sees 2 retrieved segments verbatim28.5438.122,484
Index only: never shown to the reader44.9564.422,250
MemFit45.4264.552,181

Used as memory units, the summaries locate the right stretch of conversation for 75.6% of questions, yet judge accuracy falls by 45 points because the answer itself (a date, a name, a count) does not survive compression. Retrieving whole segments verbatim loses 26 points and uses more tokens than MemFit, because segment-level search is too coarse to place the evidence turn in context. Turn-level search with summaries as an index recovers both. This is the paper's distinction: the summary finds a stretch of conversation, and the raw turn answers the question.

(3) Regarding related work, we agree that the paragraph was unclear. Mem0, SeCom, and MemTree are not iterative-retrieval methods: they consolidate or restructure memory at write time (Mem0, MemTree), or compress segments into the retrieval unit (SeCom). We have rewritten the paragraph.

(6) Regarding novelty, we agree that BM25, dense retrieval, and cross-encoders are standard IR components. The contribution is a memory design and its evidence:

  • verbatim append-only storage, with summaries that index rather than replace turns;
  • additive candidate expansion;
  • per-kind score normalization so summaries and turns can compete;
  • one configuration across three benchmarks and five backbones.

This design matches or beats LLM-managed memory systems with 81–87% fewer LLM calls.

(1) Figure 1 is larger, with a larger font, in the revision.

Reviewer_SoR2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemFit, an append-only long-term memory system for conversational agents. Each dialogue turn is stored verbatim as an episode with timestamp, speaker, metadata, a BM25 representation, and a dense embedding. The system avoids per-turn LLM calls and does not merge, rewrite, or delete episodes. TextTiling partitions the episode stream into segments, and one LLM call per segment produces a short summary that acts as an index into its original member episodes rather than replacing them.

At query time, MemFit uses hybrid BM25/dense candidate generation, metadata boosts, additive expansion through summary provenance and pseudo-relevance feedback, and a frozen cross-encoder for reranking. The top episodes are augmented with neighboring turns and passed to a reader in one answer-extraction call. In multimodal mode, image captions are appended to episode text for retrieval and images can be passed to the multimodal reader. The system is evaluated on LoCoMo, MemGallery, and LongMemEval-S with several proprietary and open-weight readers. The paper reports strong accuracy, large reductions in write-side LLM calls and construction time relative to LLM-managed memory systems, component ablations, evidence-retrieval diagnostics, permutation tests, sensitivity studies, and detailed end-to-end cost measurements.

Critical strengths
  1. MemFit is a simple, reproducible, and practically compelling systems design. Keeping source turns verbatim preserves corrections and fine-grained evidence, while the combination of lexical retrieval, dense retrieval, pseudo-relevance feedback, summary provenance, cross-encoder reranking, and local context windows produces strong results without iterative LLM search. The design is easy to implement and avoids expensive memory rewrites.

  2. The evaluation is unusually transparent about both retrieval quality and cost. The paper evaluates multiple readers and three benchmarks, reports gold-evidence hit rates independently of the reader, uses paired permutation tests in the ablations, and breaks cost into insertion, segmentation, summarization, retrieval, and answer generation. It also openly reports the trade-off that MemFit reduces write-side cost while using longer answer prompts and having a total monetary cost similar to LightMem in one LoCoMo setting.

Critical weaknesses
  1. The multimodal comparison may use privileged image captions. The paper uses ground-truth captions when available but does not show that all baselines receive the same captions or that these annotations are allowed inputs. Results with ground-truth, generated, and no captions should be reported separately.

  2. The scalability and efficiency claims are not sufficiently validated. The append-only store and dense scan grow with history length, but no scaling experiment is provided. Gains over efficient baselines are also small or mixed on MemGallery and LongMemEval-S, so matched-budget and long-history experiments are needed.

  3. The contribution of individual components is not consistently demonstrated. Most LoCoMo component ablations are not statistically significant, and the MemGallery ablations are largely flat. Stronger matched retrieval baselines and uncertainty estimates are needed to show which parts of the pipeline drive the reported gains.

Questions for authors
  1. Are the ground-truth image captions allowed inputs, and do all MemGallery baselines receive the same captions? Please separate ground-truth, generated, and no-caption results.

  2. How do memory use, retrieval latency, and accuracy scale as the episode store grows?

  3. Which components provide statistically reliable gains over a matched hybrid-retrieval baseline?

2 comments

Thank you for the review.

W1/Q1: All MemGallery baselines receive the same captions. They are included in the benchmark as part of its text-only protocol, and they are also used in the published baseline numbers we report. As requested, we report all settings using Qwen2.5-VL-7B:

CaptionsF1Judge
Benchmark-provided (as in the paper)67.6766.86
Generated by our own reader, Qwen2.5-VL-7B65.0864.35
None (image identifier only)62.2161.84
Best published memory system (A-MEM, benchmark captions)62.28–

With captions that MemFit writes itself, MemFit still exceeds every published memory system, all of which used the benchmark's captions. Without any captions, it matches the best of them (62.21 vs. 62.28 F1). The captions therefore help, as they should, but they don't account for the margin.

W2/Q2:

(a) Long histories. We ran the full LongMemEval-M: the same 500 questions, but each history holds ~4,900 turns (~1.5M tokens, 13× LongMemEval-S). The reader (Qwen3-8B) and configuration are unchanged, and both runs are graded by the GPT-4o judge.

LongMemEval-SLongMemEval-M
Turns per history~4904,893
Accuracy (overall / task-averaged)57.0 / 60.051.6 / 55.0
Gold evidence in reader's pack (≥1 turn / all turns)94.5 / 81.387.0 / 68.7
Write-time LLM calls per history4060
Build time per history~43 s106 s (61 s LLM-free storage)
LLM calls / latency per question1 / 1.4 s1 / 1.1 s

With 13× more history, accuracy falls by only 5.4 points (paired over the same 500 questions). The drop concentrates on questions where every piece of evidence must be found among many more turns, such as multi-session questions that fall from 40.6% to 29.3% and preference questions from 26.7% to 16.7%. Performance on single-session questions remains about the same. Additionally, write cost stays bounded because summaries are capped at 60 per memory, so 13× more history costs only 1.5× the write-time calls. Retrieval still reaches the evidence for 87% of questions.

(b) Cost vs. store size, from 10310^3 to 10610^6 stored episodes (on LongMemEval-M turns):

EpisodesInsert, excl. embedding (ms per episode)Retrieval latency (s, mean / p95)BM25 scan (s)Dense scan (s)Embedding matrix (GB)
10³10.70.07 / 0.080.0010.0040.0015
10⁴13.30.12 / 0.140.0020.0060.015
10⁵16.20.19 / 0.210.0210.0170.15
10⁶19.11.81 / 1.920.390.181.5

Insertion stays near-constant: 10.7 to 19.1 ms per episode over a 1,000-times larger store. Retrieval grows sublinearly up to 10510^5 episodes (0.07 s to 0.19 s, about 20 LongMemEval-M histories) and stays under 2 s at 10610^6. The cross-encoder and the answer call do not grow with the store.

Regarding gains over efficient baselines: On MemGallery, MemFit is +5.4 F1 above the strongest published memory system. On LongMemEval-S, we read the result as equal to LightMem, but the matched-budget comparison is LoCoMo. On LoCoMo, LightMem has the same call budget (1,952 vs. 1,977 LLM calls), and MemFit uses 5.1× fewer write-time tokens and scores 50.26 vs. 24.63 average F1 with GPT-4.1-mini (Table 1, Figure 3).

W3/Q3: The paper's "plain retrieval" row is the matched hybrid baseline: same store, same BM25 and dense paths, same cross-encoder, same reader, one call per question. MemFit beats it by +4.6 to +7.9 judge points on each of five readers, all with p<0.001 (Appendix Table 6). Individual components overlap, so single-reader leave-one-out tests are underpowered. Pooled over the three readers of our ablation (4,615–4,618 paired questions, stratified permutation test):

Removed from MemFitΔ judge (pooled)95% CIp
Context window−2.58[−3.53, −1.60]<0.001
Summaries−0.97[−1.73, −0.17]0.017
Pseudo-relevance feedback−0.93[−1.62, −0.24]0.010
Wide first-stage scan−0.91[−1.58, −0.26]0.008
Summary expansion−0.41[−0.93, +0.11]0.150
All (plain retrieval)−5.93[−7.08, −4.83]<0.001

Every component except summary expansion is significant when pooled. We also add two retrieval ablations (Gemma3-27B):

  • Removing the cross-encoder costs 7.9 judge points (p<0.001), and the share of questions with gold evidence in the reader's pack falls from 82.5% to 60.5%.
  • Each single retrieval path costs 1.2–1.8 judge points (Response to R1).

MemGallery's ablation is flat for a measurable reason: plain retrieval already places gold evidence in the reader's pack for 95.5% of questions, so expansion has little to recover. On LoCoMo (79.2%), the same components add 5.5 judge points.

Thank you for your reply. I will keep the original score unchanged.

Reviewer_3XCnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MemFit, a database technique for storing and retrieving conversation histories. The main argument of the paper is to reduce the need for expensive LLM calls in the process (not per record), and replace that with lexical and semantically similarity-based retrieval and a reranking process. Overall, the technique is lightweight, efficient, and evaluation shows it beats in memory built time vs traditional system like mem0, but also beats sleep-consolidation-based approach like simplemem.

Critical strengths
  1. MemFit is a very practical approach to reduce memory overhead in conversational setting. It especially beats the common need of leveraging LLMs to process memory in order to obtain retrieval friendly summaries. I'm generally quite surprised that lexical and semantic retrieval is that effective overall.

  2. The evaluation is very clear in the sense of both offline and online cost.

Critical weaknesses
  1. the evaluation doesn't provide sufficient comparison of lexical and semantic aspect's benefits. In my experience working on the area, for memory items related logical relations like time etc are generally not easy to capture by these relation (thus they need some more intricated graph construction to relate these content). I'm mostly curious how the simple retrieval approach can capture such content effectively as more expensive processes. (or it's just the dataset isn't really that complicated in this sense?)
  2. The mechanism doesn't really have a forgetting mechanism, and I think it's primiarly relying on the agent to make calls at runtime. Previuos studies show that incorrect contexts (e.g., lost in conversation paper from ICLR 26). I'm wondering if the paper has an oversight on such evaluation settings.
Questions for authors
  1. explain the contribution of Lex and Semantic similarity retrieval.
  2. The technique actually seem quite useful for agentic tasks like web/coding as well. I'm wondering if it can also be useful for context management in these settings? (in fact, my intuition reading the paper is that it might work better in these settings over long conversation).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.