acceptodds
Withdrawn from ICLR 2027

SwiftMem: Towards Lightweight and Efficient Memory for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-based memory systems enable agents to autonomously manage long-term memories, but may introduce substantial token overhead through unnecessary retrieval and excessive memory returns. We propose SwiftMem, a lightweight and token-efficient memory system that uses plain text files and three simple memory tools. SwiftMem introduces memory probing to guide retrieval decisions, tag-based query enhancement to improve retrieval quality, and Adaptive Memory Subset Selection to balance retrieval utility and cost. Experiments on two benchmarks show that SwiftMem achieves superior memory performance while substantially reducing token consumption compared with existing tool-based memory systems (up to 1/2). We provide the source code of SwiftMem in https://anonymous.4open.science/r/SwiftMem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, agent memory, tool use, retrieval-augmented generation, token efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ofbKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SwiftMem is about memory systems for LLM agents, where the agent calls tools to read and write its own memory. The authors argue this setup wastes a lot of tokens. They found that the agent often searches memory when there’s nothing relevant to find, and the results include a long tail of irrelevant memories when it does search. Author proposed Every memory is a plain text file with a header and the agent gets three tools: UpdateMemory, SearchMemory, and ViewMemory. Search only returns short summaries, and the agent has to call View to see the full content. Search is done with BM25 or grep, so there’s no embedding model or graph database involved. Before the agent sees the question, the system checks whether any words in the question overlap with memory tags. If so, it injects the matching summaries into the prompt.

Critical strengths

The system is simple and with effective experimental results, seems like just text files and BM25. The combination of agent controlled retrieval and system level guidance interesting, as it avoids relying entirely on either the agent's own retrieval decisions or rigid retrieval rules. The progressive disclosure design is another sensible choice, allowing the agent to inspect memory summaries before committing to full memory access.

Critical weaknesses
  1. The AMSS method estimates memory utility and viewing probability using BM25 scores, but the paper does not validate these estimates. Also, the expected cost formula appears to miss part of the token cost during memory viewing. These issues raise questions about the accuracy of the optimization objective.

  2. The paper only reports token consumption during QA, while SwiftMem also uses background agents for memory updates and maintenance. These operations may introduce additional costs. It is unclear whether SwiftMem still provides significant savings when all memory operations are included.

  3. The paper does not compare AMSS with simpler methods such as dynamic top-k or similarity thresholding under the same token budget. Also, the authors replace the original retrieval methods of existing baselines with BM25. This makes it difficult to determine whether the improvements come from AMSS or simpler retrieval choices.

Questions for authors
  1. How do the authors justify using BM25 scores to estimate memory utility and viewing probability? Have these estimates been compared with actual agent behavior?

  2. Do the reported token costs include background memory inspection and memory updates? If not, could the authors provide the total token consumption across the full memory lifecycle?

  3. How does AMSS compare with simpler approaches, such as dynamic top-k or similarity thresholding, under the same token budget?

Reviewer_YqbDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SwiftMem, a lightweight, tool-based memory system for LLM agents that uses plain text files as the storage medium. The authors identify two sources of unnecessary overhead in existing tool-based memory systems: unnecessary memory retrieval and an inappropriately large retrieval scope that injects many irrelevant memories into the context. SwiftMem addresses these issues with three mechanisms: memory probing with prompt injection, tag-based query enhancement, and Adaptive Memory Subset Selection (AMSS). The system provides three tools—SearchMemory, ViewMemory, and UpdateMemory—and adopts a progressive disclosure workflow. Experiments are conducted on LongMemEval and BEAM, reporting QA accuracy/rubric scores and input token consumption, together with ablation studies and a marginal-returns analysis. Overall, the problem motivation is clear, the system design is pragmatic, and the token-efficiency results are promising; however, external validity, baseline fairness, and system-level cost reporting remain concerns.

Critical strengths
  1. The problem is well motivated and the system design has practical value. The paper does not simply aim to retrieve more memories; it explicitly targets unnecessary retrieval and excessive retrieval scope in tool-based memory systems. The combination of probing, prompt injection, progressive disclosure, and AMSS is useful for deploying lightweight agent memory systems in practice.

  2. The token-efficiency results are reasonably convincing. On LongMemEval, SwiftMem achieves 84.5% overall accuracy with about 6.4K tokens, compared with 77.7% and about 35K tokens for RAG with top-k=10. The ablation study further shows that removing PI, AMSS, or QE degrades either accuracy or token consumption. The marginal-returns analysis also supports the efficiency advantage.

Critical weaknesses
  1. External validity and baseline fairness are still insufficient. The experiments use only one backbone model and replace all vector-based similarity measures in the baselines with BM25. This may disadvantage RAG and TA-Mem baselines that were originally designed around embedding-based retrieval. In addition, several question types are excluded. On BEAM, SwiftMem outperforms AgeMem by only 0.1 overall points and is clearly lower than AgeMem on categories such as Contra and Event. Therefore, the claim of “comprehensive improvement” should be stated more cautiously. I recommend adding multiple backbones, reporting the original vector-based baseline versions, and providing category-level error analysis.

  2. The evidence for “lightweight and efficient” is mainly limited to QA input tokens, and system-level overhead is incompletely reported. The paper runs a background memory-inspection sub-agent with the full toolset once every K=5 turns, but does not state whether the token, latency, and invocation costs of this sub-agent are included in the reported total cost. Retrieval latency, storage overhead, and the actual runtime of AMSS are also not reported. If the background overhead were included, the token-efficiency advantage over the baselines might shrink. I recommend adding an end-to-end cost accounting and latency/storage analysis.

Questions for authors
  1. The background memory-inspection sub-agent runs every five turns. Are its token and latency costs included in the reported total cost? If included, do the token-efficiency conclusions relative to AgeMem and RAG still hold?

  2. Memory probing relies on lexical overlap between query tokens and memory tags. For semantically related queries without lexical overlap, the system injects a suppressive instruction. What is the fallback retrieval success rate in such cases? How sensitive is performance to tag noise or missing tags generated by the agent?

  3. Is it fair to replace all vector similarities in the baselines with BM25? Can you report results for RAG and TA-Mem with their original embedding-based retrieval? Also, what explains the lower performance on BEAM Contra and Event compared with AgeMem?

  4. Are the results in Tables 2 and 3 averaged over multiple runs? Are variance or significance tests reported? What are the sample size and confidence intervals for the preliminary observations that “67% of retrieval turns fail” and “81% of irrelevant memories are in the tail”?

  5. The Limitations section mentions the lack of queries that implicitly require memory and queries with no relevant memory. Can you add a small-scale stress test to verify that SwiftMem indeed reduces unnecessary retrieval in these scenarios?

Reviewer_oKn1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a light-weigted memory system for llm-based agents, authors propose SWIFTMEM, a memory system backed by plain-text .mdc files and mechanisms like suppressive hint, pre-query Injection w/ AMSS, and post-query sub-agent, to solve the unnecessary retrieval and excessive memory returns in recent memory designs.

They evaluated their design turn-by-turn with DeepSeek-V4-Flash-Preview on filtered subsets of LongMemEval and BEAM, SWIFTMEM reports higher QA accuracy (84.5% and 67.6%) and lower QA-turn input token usage (6,362 and 11,522 tokens) than AgeMem, TA-Mem, and top-K RAG baselines.

Critical strengths
  1. Clean progressive-disclosure abstraction, it has transparent systems design that avoids the operational bloat of external (graph/other) databases.

  2. Clear observation and a well-innovated study, decomposing memory overhead into when to retrieve (unnecessary tool invocations) versus how much to retrieve (tail truncation in similarity rankings), set up a clear, diagnostic lens for evaluating agentic memory.

  3. It shows attempts to improve the Agent's awareness of memory.

Critical weaknesses
  1. Lack of memory design innovation: In Section 1, the author criticizes traditional memory systems as fixed modules independent of the agent, where the timing of read/write operations is hardcoded by developers, and champions tool-based memory systems, which return control to the agent, allowing it to make autonomous decisions. However, in Section 3's design, isn't this turning a tool-based agent back into a traditional pipeline, consisting of a fixed, rule-based RAG step at the start of every round, plus a periodic background reflection process every five rounds?

  2. Mathematically, it should be a simple design to align with the name 'swift'; however, the paper shows an over-parameterized and mathematically questionable AMSS formulation. The integer fractional programming formulation in Equations (lines 250–277) dresses up a simple heuristic ranking in unnecessary complexity. Plus, the formula doesn't specify what value is set for the basic utility constant δ\delta, and the pipeline introduces at least seven hand-tuned hyperparameters without detailed analysis

  3. The evaluation setting is unfair: The experimental accounting method "counts only the gains while ignoring the costs," and furthermore, it discards the very strengths that a baseline model possesses in long-context scenarios.

  4. Regarding the omission of literature related to "adaptive retrieval gating" and "progressive reading": In addition to the literature mentioned in Section 2 that is limited to the "tool-memory" category, there has already been extensive research on the decision mechanism of "whether to perform a retrieval" and the two-stage reading mode of "first skimming and then detailed reading".

  5. Rather than addressing why irrelevant memories score poorly in the first place (e.g., coarse memory granularity and lexical mismatch in BM25), AMSS applies a post-hoc tail-truncation optimization driven by the very same noisy BM25 score (Sˉi\bar{S}_i) that created the ranking.

Questions for authors

This memory bank relies on independent plain text files (.mdc) coarsely categorized, lacking an integrated mechanism for ambiguity resolution and state merging during writes. How does SWIFTMEM prevent the accumulation of fragmented, contradictory memory states when agents update user preferences over extended periods or when conflicts arise?

Furthermore, without explicit cross-memory state merging, how does your progressive disclosure architecture handle entries that are simultaneously returned as duplicates or superseded entries in the candidate set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.