acceptodds
Under review as a conference paper at ICLR 2027

FlashMEM: A Lightweight Memory System for Language Agents

Abstract

Long-running language agents repeatedly draw on growing interaction histories. Memory construction organizes history into compact context, but summaries formed before future queries are known may omit needed details. More elaborate construction and maintenance also incur additional model calls, repeated encoding, and persistent state storage. We revisit the role of constructed memory: it should provide directly readable, organized context and help locate relevant sources, rather than serve as the sole carrier of historical evidence. We introduce , which co-designs agent-side memory representation and server-side key–value (KV) cache reuse. On the agent side, one-pass construction produces compact, source-linked memory. Answering combines retrieved memory with a small amount of original evidence from its linked sources. On the server side, reuses recurring context across construction calls and retains KV states for compact memory rather than the full history across queries. Across three datasets, \ours achieves relative F1 gains of 16.1%, 33.1%, and 41.5%, while reducing construction costs by 97.0% and inference costs by 92.2%. \ours's server-side optimizations reduce time to first token (TTFT) by 66.8%, selection overhead for KV recomputation by 99.3%, and long-term PIC KV storage costs by 84.9–93.8%. The framework also accelerates other agent memory methods. We will release our code after anonymous review.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent Memory, Large Language Model Agents, Retrieval-Augmented Generation, KV Cache Reuse, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2wSwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focus on the scenario that long running agent will compact the context as summary for several times. The problem is that

  1. Compacted summary is constructed on previous request instead future one which lead to the risk of omiting important context
  2. Poorly organized kv storage and retainment strategy will introduce storage and computation storage.

Flashmem introduce the following framework to solve the previous problems

  1. Flashmem mechanism to construct the memory construction with source-linked evidence retrieval
  2. PIC based KV-reuse pipeline for memory construction and inference
Critical strengths
  • The motivation of this paper is strong and realistic. The context compression is critical in current llm service workflow.
Critical weaknesses
  • The related work part is insufficient. KVCache compression algorithm such as KVZip is also query agnonist but neigther introduced in the related work nor the baseline.
  • The construction mechanism is lack of novelty. This mechanism is introduced by openhands one year ago and also claude code so the contribution of flashmem part will be reduced to the recall quality.
  • The KVCache reuse pipeline doesn't form a strong contribution. Cacheclip already introduce the similar mechanism that use a small model to identify the important token and recompute. Cacheslide use the PIC like position encoding for agentic workflow KVCache reuse. The contribution is considered as incremental.
  • There is no figure for system overview or algorithm mechanism.
Questions for authors
  • For realistic llm serving workload, like trace from mooncake, what's the ttft tpot throughput p50 p95 p99 and latency for requests? What's the hbm pressure at these workload comparing to baseline and what's the computation overhead during the prefilling, select and recompute.
  • How does the system accuracy performance comparing to KVZip and other related works?
  • Comparing to Cacheclip and Cacheslide, what's the key difference of flashmem?
  • Comparing to openhands etc. what's the key difference of memory construction, recall and source link.
  • For figure 10, did you introduce the kvcache eviction mechanism to baseline algorithms? If not it won't be a fair comparison.
  • For flashmem and kvcache reuse pipeline, could you provide a proper ablation study?

I would raise my score if all weakness and problems are solved.

8 comments

To Reviewer 2wSw (1/3)

We thank the reviewer for the detailed and constructive feedback. Before addressing each point, we clarify FlashMem's setting, since it affects several comparisons below.

FlashMem targets persistent agent memory: memory is built once from long interaction histories and retrieved across future sessions and queries. This differs from in-session context compaction, which frees the current window so that an ongoing task can continue. FlashMem makes two contributions:

  1. A source-linked memory organization that keeps persistent memory compact while preserving access to raw evidence.

  2. A KV-reuse pipeline (PIC) that makes serving this memory efficient.

W1&Q2: Comparison with KVZip

Thank you for pointing out KVZip.

  • KVZip prunes existing KV caches to reduce storage and decoding costs, whereas FlashMem's PIC reuses cached KV to reduce prefill costs.

  • FlashMem also reduces KV storage, but at the prompt level. Unlike KVZip, which compresses the KV representation of a given context, FlashMem reduces the context itself before KV generation. It constructs compact, source-linked memories that preserve essential facts and source pointers, retrieving original evidence on demand to reduces persistent KV storage while retaining access to raw evidence. KVZip(KV prunes) and FlashMem (KV reuse) are complementary topic.

  • KVZip compresses the KV cache of a context that has already been prefilled. Agent histories often exceed the model's context window. In our additional benchmarks and application case, histories range from 500K tokens (BEAM-500K, AMA-Bench) to ~25M tokens (LongMemEval-V2), so a single full-history KV cache cannot be built in the first place. KVZip is evaluated in contexts of up to ~170K tokens. While evaluating their integration is beyond the scope of this work, we will to explore the oppotunity of combining KV compression and KV reuse in future work.

W3&Q3 Comparison with Related Work.

<1>. Differences from CacheClip.

  • Selection efficiency. CacheClip performs a separate chunk-conditioned query prefill for each retrieved chunk, whereas FlashMem prefills the query once and scores all chunks via (QK⊤QK^\top) using the last 12 layers' cached keys. For Qwen2.5-0.5B with eight 1,024-token chunks and a 128-token query, this reduces estimated selection computation by 86% (0.80→0.115 TFLOPs). Retaining only these layers' keys further reduces auxiliary KV storage by 75% (384→96 MiB for 32K BF16 tokens).

  • Token selection. CacheClip independently normalizes attention scores within each chunk-conditioned context, concatenates them for global Top-kk selection, and applies sliding-window grouping. FlashMem instead applies a shared Softmax across all retrieved chunks, followed by direct token-level Top-kk selection avoiding the potential amplification of irrelevant chunks caused by local normalization.

<2>. Differences from CacheSlide.

  • Scope and position handling. CacheSlide targets relative-position-dependent reuse, where prompt-template chunks retain their relative order despite shifts in absolute positions. Its CCPE learns chunk position ranges through target-model LoRA training. In contrast, FlashMem handles position-independent reuse, where retrieved memories and raw-evidence spans vary in combination and order across queries. We adopt training-free RoPE correction without modifying the target model and do not claim novelty in position correction.

  • Recomputation selection. Like CacheBlend [1], CacheSlide relies on KV-deviation-based selection, requiring full-prompt computation at the target model's first layer before selecting tokens. FusionRAG [2] improved this criterion through query-guided attention scores. In contrast, FlashMem performs token selection with a lightweight draft model, avoiding full-prompt prefill at every target-model layer, including the first.

[1] Yao, Jiayi, et al. "Cacheblend: Fast large language model serving for rag with cached knowledge fusion." Proceedings of the twentieth European conference on computer systems. 2025.

[2] Wang, Jiahao, et al. "From prefix cache to fusion rag cache: Accelerating llm inference in retrieval-augmented generation." Proceedings of the ACM on Management of Data 4.1 (SIGMOD (2026): 1-28.

To Reviewer 2wSw (2/3)

Q1: Realistic llm serving workload experiment.

FlashMem uses HiCache to manage reusable KV across multiple storage tiers. All persistent memory chunk KV is stored on NVMe SSD and loaded into DRAM when the inference engine starts. Active KV blocks are then transferred to HBM as needed. During serving, KV blocks may reside in HBM, DRAM, or SSD, following the same hierarchical cache management mechanism as conventional inference systems.

Here we evaluate Kimi-K2.6 (TP=8) using the official SGLang v0.5.23 as the full-prefill baseline. FlashMem uses a fixed 30% recomputation budget. Public serving traces such as Mooncake release only request metadata (timestamps, token lengths, and block hash IDs), not request text. We therefore use a real serving workload.

  • Workload . A real serving workload from anonymized production API requests traces with 469 (93.8%) involve tool calls, and 392 (78.4%) contain at least two tool-call steps after the final user message. The replayed input lengths are 27,585/31,940/81,990 tokens at P50/95/P99.

Table 1: Serving performance on real workload requests.

Input (req/s)MethodThroughput (req/s)Output (tok/s)TTFT P50/P95/P99 (s)TPOT P50/P95/P99 (ms)E2E P50/P95/P99 (s)
0.15SGLang0.1464.656.89 / 16.74 / 21.0619.70 / 375.52 / 420.228.47 / 21.50 / 28.49
0.15Ours DRAM0.1464.663.25 / 7.97 / 12.3217.73 / 24.01 / 38.963.82 / 8.53 / 12.92
0.15Ours SSD0.1464.663.50 / 8.44 / 12.8320.08 / 28.78 / 37.694.14 / 9.06 / 13.50
0.30SGLang0.2467.88154.47 / 307.51 / 318.31164.86 / 435.57 / 526.03159.58 / 310.36 / 325.91
0.30Ours DRAM0.2919.314.32 / 12.88 / 18.0018.13 / 25.74 / 38.944.95 / 13.68 / 18.47
0.30Ours SSD0.2919.314.76 / 13.70 / 19.2718.65 / 29.77 / 42.055.35 / 14.55 / 20.05

1) HBM pressure. All configurations use the same 110K-token GPU KV pool. At 0.30 req/s, FlashMem uses 3.43 GiB more peak HBM per GPU than SGLang (94.02% vs. 90.44% utilization), while P95 KV pool occupancy remains comparable at 93.96%/94.82%/94.90% (SGLang/DRAM/SSD). All configurations peak near 99.8% KV pool occupancy.

Table 2. HBM usage and KV pool occupancy under serving workloads.

Input (req/s)MethodPeak HBM per GPU (MiB)HBM UtilizationKV Pool Utilization (P95 / Peak)
0.15SGLang88,50790.43%78.80% / 99.86%
0.15Ours DRAM92,02394.02%80.08% / 98.67%
0.15Ours SSD92,02394.02%79.43% / 98.59%
0.30SGLang88,51190.44%93.96% / 99.82%
0.30Ours DRAM92,02394.02%94.82% / 99.81%
0.30Ours SSD92,02394.02%94.90% / 99.75%

2) Computation overhead. To explain the serving latency, we further profile the per-stage TTFT costs in Table 3. Draft selection takes 191/183 ms (DRAM/SSD). Target-model prefill takes 2.05/2.05 s, recomputing/prefill selected tokens and question while reusing cached KV for the rest. Request dispatch, queueing, and KV preparation account for the remaining latency, yielding a mean TTFT of 3.47/3.55 s.

Table 3. FlashMem TTFT latency breakdown at 0.15 req/s (mean per request).

ComponentKV in DRAM (ms)KV in SSD (ms)
Selection model overhead191.51183.49
Selector queueing9.428.75
Request dispatch & prefill waiting952.091,006.34
Batch scheduling & KV preparation274.82302.82
Target prefill (selective recomputation)2,046.482,045.98
Mean TTFT3,474.313,547.38

3) KV loading. For this workload with 32K input tokens, isolated cool KV restoration takes 1.06/1.20 s (P50) from DRAM/SSD. During serving, asynchronous loading overlaps scheduling waits, leaving 112.09/167.94 ms (P50) of foreground waiting per restore call on rank 0, included in KV preparation. Host-to-GPU transfers average 70.90/74.41 ms per batch, with 63.78/60.97 ms overlapping target execution.

Q5: Clarifying Persistent KV Storage Accounting in Figure 10.

Thank you for the question. We clarify that Figure 10 measures the persistent KV storage required for each method's final memory store, rather than runtime cache occupancy or intermediate states generated during construction. In agent memory benchmarks, all final memory entries are retained for future RAG retrieval. To support PIC reuse across queries, their KV states must also be persistently stored; otherwise, retrieved entries require recomputation. This comparison essentially reflects the size of each method's final memory store, which could also be measured in text tokens or bytes.

KV eviction is a separate serving-level mechanism. HiCache manages KV residency across HBM, DRAM, and SSD without changing the persistent storage requirements measured in Figure 10. We will clarify this distinction in the revised manuscript.

To Reviewer 2wSw (3/3)

W2&Q4: Differences from OpenHands and Claude Code.

Thank you for highlighting these systems. We acknowledge that summarization and source references already exist in them. The difference lies in what the links are used for. The source-link in context compaction and persistent memory serve different purposes.

Context compaction. OpenHands' source link happen in its Context compaction. LLMSummarizingCondenser records summaries alongside forgotten_event_ids, and identifies the original events. These links provide traceability for solving current context window limitations, rather than indexing evidence for future retrieval.

Persistent memory. OpenHands SDK and Claude Code use file-based memory indexes (e.g., MEMORY.md) and read linked files on demand. Following a reference is agent-driven: the LLM must decide that more information is needed, choose a file, and issue tool calls. Each lookup therefore costs extra LLM inference and tool latency. FlashMem instead integrates source linking into the memory construction and rag retrieval pipeline. It constructs independently retrievable memory entries linked to original records. At recall, these links guide raw-evidence retrieval under a separate token budget, without additional agent-planned file access. The distinction lies in how source links support construction and retrieval, rather than source linking itself.

We will expand the Related Work section in the revised version to discuss these works in greater detail. Thank you again for highlighting them and for your questions.

W4: Paper Improve.

We apologize for the lack of visual clarity. We will add a pipeline diagram illustrating the overall architecture and key algorithmic mechanisms in the revised version.

Q6: More Ablation Study.

Thank you for the suggestion. We further examine two key components of FlashMem: memory organization and KV reuse.

Memory organization. Table 3 already compares summary-only, raw-only, independent retrieval, and source-linked retrieval. To further isolate source linking, we compare independent RAG and source-linked retrieval using only raw evidence for answering.

Table 4: Raw-evidence-only QA on LoCoMo (150-token budget; Accuracy %).

ModelRAGSource-linkedΔ [95% CI]
Mistral-7B-Instruct-v0.334.1643.44+9.29 [7.50, 11.09]
Llama-3.1-8B-Instruct29.1637.27+8.12 [5.52, 10.93]
Gemma-4-26B-A4B-it42.8654.29+11.43 [9.04, 13.79]
Qwen3-32B46.1057.14+11.04 [9.24, 12.92]
GLM-4.5-Air43.7053.57+9.87 [7.66, 12.52]
DeepSeek-V3.2 FP844.1656.62+12.47 [9.58, 15.31]
Kimi-K2.649.4860.78+11.30 [8.98, 13.54]

KV reuse. We conduct controlled ablations on LoCoMo/Qwen3-8B/Qwen-0.6b Embedding with 30% recomputation budget. We first vary layer aggregation, including the first half, last layer, and the 10 layers with the highest attention-score entropy. We then examine score normalization: chunk-wise Softmax normalizes QK scores independently within each chunk before global Top-k selection. For token selection, window grouping selects 8-token windows containing at least 5 candidate tokens. Finally, we disable RoPE correction to evaluate positional alignment. All other settings remain unchanged.

Table 5: KV reuse component ablation on LoCoMo (Accuracy %).

ScoringLayer AggregationToken SelectionRoPE FixAcc. (%)
Global QK (Ours)Later halfToken Top-k✓60.4
Global QKFirst halfToken Top-k✓49.1
Global QKLast layerToken Top-k✓57.5
Global QKEntropy Top-10Token Top-k✓57.0
Chunk-wise SoftmaxLater halfToken Top-k✓57.7
Global QKLater halfWindow (8, ≥5)✓58.9
Global QKLater halfToken Top-k×58.2

We hope these additional experiments and clarifications address your concern. Thank you for guiding us toward a clearer and more polished presentation!

Reviewer_qGLnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FlashMem has two components where the agent side writes summaries that keep source IDs, then retrieves raw windows only from the sources the selected summaries cite and the server side caches KV only for summaries, and a Qwen2.5-0.5B selector picks which positions the target model recomputes. With Qwen3-8B, Table 2 reports F1 of 50.22/44.41/38.75 on LoCoMo/LME-S/HaluMem against best-baseline 42.78/33.36/23.87. It also reports a 30.2/41.0/31.4% TTFT cut over FusionRAG and 84.6-93.7% less persistent KV.

Critical strengths
  1. The exp result, like tab 3, cleanly supports the source-linked evidence view.
  2. The cost accounting is useful and careful.
Critical weaknesses
  1. The headline pairs quality from one configuration with efficiency from another. The F1 gains come from FlashMem without KV reuse, while the latency and much of the cost savings come from FlashMem† and ‡. With reuse in both phases (‡), LoCoMo F1 on Qwen3-8B drops from 50.22 to 43.79, and on LoCoMo the ‡ variant falls below SimpleMem with GLM-4.5-Air and Kimi-K2.6. For a paper framed as a co-design, that is two separate trade-offs, not one joint gain.
  2. The quality gains cant be attributed to the memory design. FlashMem uses BGE-M3 plus a bge-reranker-v2-m3 cross-encoder, while the baselines keep their own retrievers. The Raw RAG retriever and the budgets are never given. Table 3's R column is also lower than Table 2's Raw RAG (49.14), so the setups differ in ways the paper doesn't explain. Without baselines on the same retrieval stack at a matched evidence budget, I can't tell how much of the +2 to +10 accuracy points on Qwen3-8B comes from source-linked memory and how much from a stronger retriever.
Questions for authors
  1. What retriever does Raw RAG use in Table 2?
  2. FlashMem‡ falls below SimpleMem on LoCoMo with GLM-4.5-Air and Kimi-K2.6. Which configuration do you recommend deploying, and how does it compare with the strongest baseline?
  3. Which table cells produce the abstract's 97.1% and 66.8%, and does 84.9-93.8% exclude the selector keys that Fig. 10 includes?
2 comments

To Reviewer qGLn (1/2)

W1&Q2 Joint Quality–Efficiency Gains and Deployment

Thank you for raising this concern. We clarify that FlashMem's memory organization and PIC acceleration are two orthogonal designs. The former improves both quality and efficiency without KV reuse, while the latter provides optional, tunable acceleration.

1. Joint gains without PIC. We conduct additional experiments on LoCoMo/Qwen3-8B with a shared Qwen3-Embedding-0.6B retriever. As shown in Table 1, FlashMem achieves the highest F1/Accuracy (47.46%/62.92%), outperforming the strongest quality baseline with 1.31× speedup over the fastest competing baseline (LightMem) and 1.99× over SimpleMem.

Table 1. LoCoMo/Qwen3-8B: Quality and TTFT without PIC.

MethodF1AccuracyAvg. TTFTP50 TTFTP95 TTFTP99 TTFTP95 TTFT Speedup
SimpleMem42.24%50.19%12.09s14.87s15.32s15.36s1.99x
LightMem39.42%43.96%8.30s9.65s10.10s10.15s1.31x
StructMem38.50%48.25%16.67s21.57s22.02s22.06s2.86x
MemoryOS39.25%44.16%24.51s31.62s34.00s34.04s4.42x
A-MEM44.47%53.44%23.97s31.46s31.90s33.22s4.15x
FlashMem47.46%62.92%6.41s7.25s7.69s7.74s1.00x

2. PIC as an optional, tunable acceleration. PIC is a general KV-reuse mechanism applicable to other agent memory frameworks that support chunk-level KV caching. Its recomputation rate (r) controls the quality–latency trade-off. As shown in Table 2, at (recompute rate=0.3), FlashMem avoids recomputing approximately 70% of eligible cached tokens, achieving 60.39% accuracy with a 3.18× P95 TTFT speedup. Increasing (r) to 0.8 recovers accuracy to 62.66%, close to full recomputation (62.92%), while retaining a `1.30× speedup. PIC offers an additional quality–efficiency trade-off rather than being required for FlashMem's joint gains.

Table 2. LoCoMo/Qwen3-8B: Quality–latency trade-off across recomputation rates.

Recompute rateF1 (%)Acc. (%)Avg./P50/P95/P99 TTFT (s)P95 Speedup
10%45.0959.420.62s / 0.70s / 0.96s / 1.01s7.98x
15%45.6060.320.89s / 0.97s / 1.42s / 1.47s5.40x
30%45.8860.391.66s / 1.84s / 2.42s / 2.47s3.18x
45%45.8861.492.76s / 3.49s / 3.94s / 3.98s1.95x
60%46.0462.533.31s / 3.04s / 4.73s / 4.78s1.62x
80%46.6662.664.85s / 5.36s / 5.94s / 5.98s1.30x
Full (100%)47.4662.926.41s / 7.25s / 7.69s / 7.74s1.00x

3. Cross-model validation and deployment. Here we further evaluate all methods with the same embedding and reranker models. As shown in Table 3, FlashMem‡ (recompute rate=0.3) achieves 63.12%/73.31% accuracy, compareable to SimpleMem's 62.79%/72.14%.

Table 3. LoCoMo: Accuracy (%) with a shared embedding model and reranker.

MethodGLM-4.5-AirKimi-K2.6
LightMem48.90%62.27%
StructMem53.57%67.60%
SimpleMem62.79%72.14%
FlashMem71.82%76.23%
FlashMemt†66.56%74.48%
FlashMem‡63.12%73.31%

Deployment recommendation. We recommend FlashMem without PIC for quality-sensitive workloads, as it already provides joint quality and latency gains. For efficiency-oriented workloads, PIC can be enabled with (r=0.3) or a higher rate according to quality requirements. We will revise the paper to clearly distinguish the quality and efficiency results of each configuration.

To Reviewer qGLn (2/2)

W2&Q1. Retrieval fairness and source-linked memory

Thank you for raising these concerns. We first clarify that Raw RAG in Table 2 uses BGE-M3 to retrieve the top-40 relevant history turns, the same embedding model used by FlashMem for memory retrieval. The lower R accuracy in Table 3 (45.2% vs. 49.14%) arises from its much smaller raw-evidence budget (150 tokens vs. top-40 turns).

To isolate the contribution of source linking under matched settings, we further compare independent RAG and source-linked retrieval on LoCoMo, excluding summaries from the answering context. Both use Qwen3-Embedding-0.6B, bge-reranker-v2-m3, and the same 150-token raw-evidence budget. GLM-5.3 serves as the accuracy judge.

Table 4: Raw-evidence-only QA on LoCoMo (150-token budget; Accuracy %).

ModelRAGSource-linkedΔ [95% CI]
Mistral-7B-Instruct-v0.334.1643.44+9.29 [7.50, 11.09]
Llama-3.1-8B-Instruct29.1637.27+8.12 [5.52, 10.93]
Gemma-4-26B-A4B-it42.8654.29+11.43 [9.04, 13.79]
Qwen3-32B46.1057.14+11.04 [9.24, 12.92]
GLM-4.5-Air43.7053.57+9.87 [7.66, 12.52]
DeepSeek-V3.2 FP844.1656.62+12.47 [9.58, 15.31]
Kimi-K2.649.4860.78+11.30 [8.98, 13.54]

Q1. Raw RAG retriever

  • Thank you for the question. Raw RAG in Table 2 uses BGE-M3 to retrieve the top-40 most relevant turns.

  • We further evaluate Raw RAG and FlashMem with five alternative embedding retrievers and four reader models on LoCoMo. Each cell reports FlashMem's F1/accuracy gains (pp) over Raw RAG. FlashMem consistently improves both metrics across all 20 configurations. GLM-5.3 serves as the accuracy judge.

Table 5: Retrieval backbone comparison (F1/Acc gain., %).

RetrieverQwen3-8BQwen3-32BDeepSeek-V3.2 FP8Gemma-4-26B
MiniLM-L6-v2+14.26 / +16.10+19.63 / +12.21+13.51 / +9.55+11.77 / +17.86
Qwen3-Embedding-0.6B+16.49 / +20.20+20.91 / +14.68+15.63 / +14.55+15.24 / +18.90
E5-large-v2+15.40 / +17.86+19.83 / +12.92+12.89 / +13.12+13.26 / +17.79
Jina-v3+15.46 / +17.92+20.58 / +13.57+12.64 / +11.82+12.60 / +17.53
ReasonIR-8B+13.42 / +14.81+18.38 / +10.13+10.26 / +7.27+8.99 / +11.36
Mean+15.01 / +17.38+19.87 / +12.70+12.99 / +11.26+12.37 / +16.69

Q3: Writing Improvements.

Thank you for your feedback. We will clarify the definitions and comparison baselines of these metrics. Inference cost is measured by total inference tokens against SimpleMem (Table 2: 16.2k + 736 vs. 1.1k + 10 tokens). TTFT reduction is measured for FlashMem relative to full prefill (Fig. 7: 2.71× speedup), rather than using A-MEM's answer-latency reduction. KV storage reduction includes both target-model KV and selector keys, consistent with Fig. 10. Based on these definitions, we will revise the abstract to:

"FlashMem achieves relative F1 gains of 17.4%, 33.1%, and 62.3%, while reducing inference costs by 93.4%. FlashMem's server-side optimizations reduce time to first token (TTFT) by 63.1%, selection overhead for KV recomputation by 99.3%, and long-term KV storage costs by 84.6–93.7%. The framework also accelerates other agent memory methods."

We will correct these in the revision in subsequent editions. Thank you again for guiding us toward a clearer and more polished presentation!

Reviewer_jXTDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlashMem constructs the agent memory by organizing the history and locate evidence while preserving access to the original records. FlashMem uses one construction-model call per history segment consolidates content into self-contained tagged entries. At inference time, it retrieves relevant entries and supplements them with short summaries from their linked sources. While serving It persistently caches memories and uses a small auxiliary model to select target-model tokens for KV recomputation.

FlashMem is tested on LoCoMo, LongMemEval-S, and HaluMem-Medium, with Qwen3-8B, GLM-4.5-Air, and Kimi-K2.6, against LightMem, A-MEM, MemoryOS, SimpleMem, StructMem, and raw RAG. FlashMem showed F1 and accuracy improvements using only 1–1.7k inference tokens per query.

Critical strengths
  1. The paper connects agent memory with position-independent caching. This integration sounds promising (reducing construction calls, repeated prefill, and persistent KV capacity) for many real world agentic applications
  2. LME-S accuracy is 61.4 vs. 51.0 (Qwen3-8B), 75.2 vs. 66.6 (GLM-4.5-Air), and 83.2 vs. 74.4 (Kimi-K2.6), using ~1.6–1.7k inference tokens vs. 13.3–17.3k for SimpleMem.
Critical weaknesses
  1. The paper is framed around long-horizon language agents, but every benchmark is conversational QA with histories of at most ~160K tokens; neither agentic tasks nor longer horizons are evaluated.
  2. It is unclear how much of the improvement comes from source-linked memory. In Table 3, retrieving raw evidence only from the cited sources adds just +1 to +4.1 accuracy points over retrieving it independently, and the paper neither states that budgets are matched nor reports confidence intervals.
  3. FlashMem uses BGE-M3 with a cross-encoder reranker, while the baselines keep their own retrievers. Since the embedding model strongly affects ranking, all methods should be compared with a shared retriever, otherwise FlashMem’s gains cannot be separated from retrieval quality.
  4. KV reuse changes memory content, not just latency. When KV is reused during construction, the length of the construction model’s output shifts by −36% to +64% (Table 2). Yet memory fidelity is never analyzed, the paper reports no entry counts, source-ID validity, or extraction recall.
Questions for authors
  1. Figure 1 establishes the quality–token-cost tradeoff, is it possible to extend this analysis to measured latency across recomputation budgets and isolate construction-only reuse.
  2. Can a higher recomputation budget recover most of the LoCoMo quality loss while retaining the speedup?
  3. What kind of error can occur while answering multi-hop, update, and conflict-resolution questions using FlashMem
  4. On LoCoMo/Qwen3-8B, 35.4k of 53.2k executed construction-prefill tokens are saved. How much of this is conventional prefix-cache hits, and what else recurs?
  5. Where does persistent KV lives? What is the cost of loading it, concurrency, throughput and reranking latency
  6. Please include a limitations section
  7. Can you tests the co-design hypothesis (fig 3 and 4) on flashmem aswell.
5 comments

To Reviewer jXTD (1/3)

We sincerely thank you for your detailed and constructive feedback. Your comments have helped us identify key areas for clarification and improvement. Below, we address your questions and the noted weaknesses.

W1. Agentic tasks and longer histories

Here we extended our evaluation to LongMemEval-V2 (web-agent trajectories), BEAM (500K–1M-token histories), and AMA-Bench subset (multi-session agentic tasks). As shown below, FlashMem consistently achieves the highest accuracy on all three completed long-history settings.

Table1: Additional benchmark results (model: Qwen3-8B; Accuracy %).

BenchmarkHistoryLightMemSimpleMemFlashMem
AMA-Bench>500K10.6723.3332.67%
LME-V2-S~25M20.4427.1130.22%
BEAM-500K~500K25.6140.9146.67%
BEAM-1M~1M37.9739.3842.97%

W2. Contribution of source-linked evidence

Table 3 measures the marginal gain from source linking when summaries are already available, which can limit the additional benefit of raw evidence. To isolate the contribution of source linking, we conduct a evidence-only evaluation on LoCoMo, excluding summaries from the answering context. Both approaches use Qwen3-Embedding-0.6B, bge-reranker-v2-m3, and the same 150-token budget.

Table2: Raw-evidence-only QA on LoCoMo (150-token budget; Accuracy %).

ModelRAGSource-linkedΔ [95% CI]
Mistral-7B-Instruct-v0.334.1643.44+9.29 [7.50, 11.09]
Llama-3.1-8B-Instruct29.1637.27+8.12 [5.52, 10.93]
Gemma-4-26B-A4B-it42.8654.29+11.43 [9.04, 13.79]
Qwen3-32B46.1057.14+11.04 [9.24, 12.92]
GLM-4.5-Air43.7053.57+9.87 [7.66, 12.52]
DeepSeek-V3.2 FP844.1656.62+12.47 [9.58, 15.31]
Kimi-K2.649.4860.78+11.30 [8.98, 13.54]

We further examine whether these QA gains reflect better evidence localization. Gold-turn Hit@K improves consistently across retrieval depths.

Table3: Gold-turn Hit@K on LoCoMo (%, paired 95% CI).

KRAGSource-linkedΔ [95% CI]
151.5753.29+1.72 [0.29, 3.09]
366.5771.92+5.35 [4.13, 6.48]
570.6176.93+6.33 [4.82, 7.78]
1074.1481.68+7.54 [5.84, 9.30]
2075.7785.36+9.59 [7.62, 11.23]
5076.1886.78+10.60 [8.96, 12.01]

Q3: Multi-category performance

We analyze FlashMem's performance across question categories. Three potential failure modes are: (1) Multi-hop: missing or failing to combine evidence across sessions; (2) Knowledge Update: overlooking later updates and relying on outdated information; and (3) Conflict Resolution: failing to distinguish conflicting facts and their temporal relationships. Table 6 reports category-wise accuracy on LoCoMo and LongMemEval-S, while Table 7 presents two cases illustrating how source-linked retrieval supports multi-hop reasoning and knowledge updates.

Table 6: Category-wise accuracy on datasets (Qwen3-8b/Qwen-0.6b-Embedding, accuracy%).

LoCoMo datasetMulti-hop RetrievalTemporal ReasoningOpen-domain KnowledgeSingle-hop RetrievalOverall
FlashMem40.460.133.374.962.9
SimpleMem31.229.027.167.350.2
LME-S datasetKnowledge UpdateMulti-sessionSingle-session AssistantSingle-session PreferenceSingle-session UserTemporal ReasoningOverall
FlashMem69.242.185.753.388.640.658.0
SimpleMem44.917.344.640.060.026.334.4

Table 7: Case studies of multi-hop retrieval and knowledge updates.

Case 1: Multi-hop RetrievalCase 2: Knowledge Update
QuestionWhat are Melanie's pets' names?How many engineers do I lead when I just started my new role as Senior Software Engineer? How many engineers do I lead now?
Evidence 12023-07-12 Melanie: “Luna and Oliver! They are so sweet and playful......”2023-05-11 User: “I apologize for the confusion! I lead a team of 4 engineers in my new role as .....”
Evidence 22023-08-23 (+42 days) Melanie: “Yeah, it's normal to .....- we got another cat named Bailey too. Here's a pic of Oliver. ......2023-10-24(+166 days) User: “Thats a great list! I think I will ....., especially the part where I now lead a team of five engineers - its been a great experience so far,......”
ReferenceOliver, Luna, BaileyInitially 4 engineers; now 5 engineers
FlashMemLuna, Oliver, Bailey ✓4 initially, 5 now ✓
SimpleMemLuna and Oliver ✗4 initially, 4 now ✗

To Reviewer jXTD (2/3)

W3. Shared retrieval models

To control differences in model backbones, we evaluate all methods on LongMemEval-S using shared embedding (BGE-M3), reranking (BGE-reranker-v2-m3), and inference (GLM-4.5-Air) models. FlashMem outperforms all baselines.

Table 4. LongMemEval-S accuracy with shared retrieval and inference models.

MethodFlashMemSimpleMemLightMemStructMemMemoryOSA-MEM
Accuracy (%)75.2066.8762.0565.8657.4356.22
FlashMem gain (pp)—+8.33+13.15+9.34+17.77+18.98

We further vary the embedding backbone to Qwen3-Embedding-0.6B/4B. FlashMem consistently leads the baselines under both settings, suggesting that its gains are robust to the choice of embedding model.

Table5: Additional embedding-model comparison on LoCoMo dataset (Accuracy, %).

MethodGLM-4.5&0.6B EmbeddingQwen3-8B &0.6B EmbeddingGLM-4.5&4B Embedding
LightMem48.90%43.96%49.16%
StructMem53.57%48.25%55.00%
MemoryOS48.51%44.16%48.05%
A-Mem51.43%53.44%55.52%
SimpleMem62.79%50.19%65.39%
FlashMem71.82%62.92%73.77%

Q4. Prefix-cache hits and recurring construction inputs

We respectfully clarify that all 35.4K saved tokens come from KV reuse beyond conventional prefix caching. Constr. Table 2 reports the actual prefill workload after excluding RadixCache hits. We apologize for this ambiguity and provide a detailed breakdown below.

Table 8: Construction tokens on LoCoMo per session/Qwen3-8B (K).

MethodSimpleMemLightMemStructMemMemoryOSFlashMem
Prompt Tokens42.691.9361.5335.274.5
Prompt Tokens w/o Prefix41.933.8217.9200.953.2
Available Reuse31.332.1213.0127.650.6

Recurring inputs arise from two sources: (1) During agent inference, conversation history is already prefilled to generate responses, producing KV caches that can be reused in later memory construction(Like FlashMem‡ 's 53.2k->17.8k under 30% recomputation rate); (2) Intermediate outputs from memory-construction steps.

Q1&Q2. Recomputation budget and quality recovery

We extend Fig. 1 with measured TTFT across recomputation budgets on LoCoMo/Qwen3-8B. Increasing the budget progressively recovers quality while retaining latency gains. At 80%, accuracy is close to full prefill (62.66% vs. 62.92%) with a 1.30× P95 TTFT speedup. The 30% setting offers a faster trade-off, achieving 60.39% accuracy with a 3.18× speedup.

Here P95 TTFT speedups are measured relative to FlashMem with full recomutation (no Kv reuse).

Table 9 : LoCoMo/Qwen3-8B/qwen3-0.6B embedding: Accuracy and P50/P95/P99 TTFT speedup.

Recomputation budgetF1AccuracyavgP50P95P99P95 TTFT speedup
10%45.09%59.42%0.62s0.70s0.96s1.01s7.98x
15%45.60%60.32%0.89s0.97s1.42s1.47s5.40x
30%45.88%60.39%1.66s1.84s2.42s2.47s3.18x
45%45.88%61.49%2.76s3.49s3.94s3.98s1.95x
60%46.04%62.53%3.31s3.04s4.73s4.78s1.62x
80%46.66%62.66%4.85s5.36s5.94s5.98s1.30x
FlashMem (rate=1)47.46%62.92%6.41s7.25s7.69s7.74s1.00x
simplemem42.24%50.19%12.09s14.87s15.32s15.36sN/A
lightmem39.42%43.96%8.30s9.65s10.10s10.15sN/A
structmem38.50%48.25%16.67s21.57s22.02s22.06sN/A
memoryOS39.25%44.16%24.51s31.62s34.00s34.04sN/A
A-mem44.47%53.44%23.97s31.46s31.90s33.22sN/A

Construction-only reuse. Here we independently enable KV reuse during construction and inference. With inference unchanged, construction-only reuse preserves retrieval F1 within approximately 1.2 points of normal construction across both evaluated configurations.

Table10: Stage-wise KV reuse ablation (LoCoMo, GLM-4.5-Air, 30% budget; retrieval F1, %).

ConstructionInferenceQwen3-4B EmbeddingQwen3-0.6B Embedding
NormalNormal52.1951.67
NormalKV reuse51.1850.15
KV reuseNormal51.0550.51
KV reuseKV reuse49.4648.78

W4 Impact of KV Reuse on Memory Fidelity.

We compare full prefill and 30% recomputation on the same LoCoMo dialogue packets across three construction models. Semantic similarity is the average packet-level cosine similarity between mean BGE-M3 embeddings of memory entries. Source-link validity measures the fraction of cited IDs that resolve to original passages in input packets. Gold-evidence coverage measures the fraction of annotated evidence IDs cited by memory entries in the same dialogue.

Table11: Impact of KV reuse on memory fidelity. (full attention->30% recompute)

Construction ModelSemantic SimilarityValid Source LinksGold-Evidence Coverage
Qwen3-8B0.929100% → 100%99.1% → 92.2%
GLM-4.5-Air0.915100% → 100%92.9% → 89.0%
Kimi-K2.60.928100% → 100%98.7% → 96.9%

To Reviewer jXTD (3/3)

Q5: KVcache Manage & Serving performance.

(1) Persistent KV storage. FlashMem uses HiCache to manage reusable KV across multiple storage tiers. All persistent memory chunk KV is stored on NVMe SSD and loaded into DRAM when the inference engine starts. Active KV blocks are then transferred to HBM as needed. During serving, KV blocks may reside in HBM, DRAM, or SSD, following the same hierarchical cache management mechanism as conventional inference systems.

(2) Concurrent serving performance. We evaluate Kimi-K2.6 (TP=8) using the official SGLang v0.5.23 as the full-prefill baseline. FlashMem uses a fixed 30% recomputation budget. We replay requests at input rates of 6 and 8 req/s. The throughput difference between DRAM and SSD remains below 0.3% at both input rates.

Table 12. Serving performance on LoCoMo requests.

Input (req/s)MethodThroughput (req/s)Output (tok/s)TTFT P50/P95/P99 (s)TPOT P50/P95/P99 (ms)E2E P50/P95/P99 (s)
6SGLang5.992191.743.37 / 9.35 / 10.26136.03 / 203.12 / 232.667.79 / 13.36 / 14.52
6Our DRAM6.010192.251.23 / 2.11 / 2.6016.17 / 20.91 / 24.021.75 / 2.69 / 3.19
6Our SSD6.008192.331.27 / 2.25 / 2.8116.66 / 22.00 / 24.031.81 / 2.91 / 3.46
8SGLang6.269200.6133.35 / 52.39 / 54.8160.06 / 138.70 / 143.9236.16 / 55.03 / 57.41
8Our DRAM7.929253.003.18 / 6.19 / 7.3326.89 / 44.46 / 49.714.07 / 7.34 / 8.41
8Our SSD7.906253.733.91 / 8.29 / 9.3632.46 / 39.56 / 47.474.84 / 9.50 / 10.54

(3) Loading cost. To explain the loading overhead, we further profile each stage of FlashMem serving at 6 req/s.

Table 13. FlashMem serving latency breakdown.

ComponentKV in DRAM (ms)KV in SSD (ms)
Section model inference59.2958.90
Selector queueing19.5238.47
Dispatch & batch waiting427.44444.25
Batch scheduling & KV preparation51.4060.22
GPU execution to first token672.49671.07
TTFT1,230.141,272.91

Note: Isolated KV restoration takes 98.36 ms (SSD) versus 6.63 ms (DRAM) at P50. During serving, SSD KV blocks are asynchronously prefetched into pinned DRAM while requests await GPU scheduling, with 97.2% of rank-0 blocks ready before use and only 0.081 ms scheduler-side restoration latency (P95). Both configurations incur approximately 22 ms per batch for host-to-GPU KV transfer, partially overlapping GPU execution.

(3) Reranking Cost: For document reranking, BGE-reranker-v2-m3 processes 40 summary candidates and 96 raw sources per LoCoMo query, taking 89.79 ms (P50), followed by raw-span reranking at 257.65 ms (P50). The measured total P50 is 341.09 ms.

Q6: Writing Improve

Thank you for the suggestion. we will add the Limitations section in the later version, addressing evaluation scope, memory-construction reliability, trade-offs.introduced by partial KV recomputation.

Q7: Hypothesis on FlashMem

  • Thank you for the question. Figure 3 analyzes token-surprisal concentration across datasets, independent of memory methods. We therefore extend Figure 4's evidence-retrieval analysis to FlashMem. In the table 14, we reranked raw turns, LightMem entries, and FlashMem entries on the LoCoMo using identical retriever settings. FlashMem achieved higher first-gold MRR than both raw turns and LightMem in all 18 groups.

Table 14: Each cell reports first-gold MRR in the order Raw turns / LightMem / FlashMem.

Evidence distanceQuestionsBGE-M3MiniLM DenseHybrid (RRF)
0–53070.476 / 0.495 / 0.5640.340 / 0.453 / 0.4850.437 / 0.504 / 0.535
6–102300.564 / 0.572 / 0.6210.394 / 0.528 / 0.5730.537 / 0.551 / 0.605
11–152830.448 / 0.491 / 0.5790.313 / 0.438 / 0.5550.414 / 0.470 / 0.588
16–202730.360 / 0.413 / 0.5170.286 / 0.410 / 0.4810.371 / 0.429 / 0.517
21–252240.437 / 0.435 / 0.5770.344 / 0.456 / 0.5130.449 / 0.452 / 0.577
26–312160.361 / 0.424 / 0.5360.260 / 0.448 / 0.5150.366 / 0.444 / 0.560
Overall1,5330.441 / 0.472 / 0.5650.323 / 0.454 / 0.5190.428 / 0.475 / 0.562

We hope these additional experiments and clarifications address your concern. Thank you for guiding us toward a clearer and more polished presentation!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and All Reviewers:

We thank all reviewers for their constructive feedback. Following their suggestions, we conducted additional experiments and analyses. We summarize the key analyses below.

  1. Contribution of source linking (jXTD, qGLn, 2wSw). On LoCoMo, we compared source-linked retrieval with independent RAG using the same embedding model, reranker, and budget. Readers received only the retrieved raw evidence. Source linking improved accuracy by 8.1–12.5 percentage points across seven reader models. Gold-turn Hit@10 increased from 74.14% to 81.68%. (jXTD, Tables 2–3.)
  2. Comparisons under shared retrieval settings (jXTD, qGLn). With a shared embedding model, reranker, and reader, FlashMem outperformed the strongest baseline on LongMemEval-S by 8.33 percentage points. On LoCoMo, it improved both F1 and accuracy over Raw RAG across all 20 retriever–reader configurations tested. (jXTD, Table 4 and qGLn, Table 5.)
  3. Long histories and agentic tasks (jXTD). We extended evaluation to an AMA-Bench subset, LongMemEval-V2-S, and two BEAM settings, covering histories up to approximately 25 million tokens. FlashMem achieved the highest accuracy in all four settings. (jXTD, Table 1.)
  4. Quality–latency trade-off (jXTD, qGLn, 2wSw). Without PIC, FlashMem achieved 62.92% accuracy and 7.69 s P95 time to first token (TTFT), compared with 10.10 s for the fastest competing baseline. With PIC, 30% recomputation achieved 60.39% accuracy and a 3.18× P95 TTFT speedup over full recomputation. At 80%, accuracy recovered to 62.66% with a 1.30× speedup. We also measured memory fidelity and ablated individual PIC components. (jXTD, Tables 9–11, qGLn, Tables 1–2, and 2wSw, Table 5.)
  5. Serving evaluation (jXTD, 2wSw). We evaluated two serving workloads. On production traces, FlashMem reduced P95 TTFT from 16.74 s to 7.97 s at 0.15 requests/s. At 0.30 requests/s, P95 TTFT dropped from 307.51 s to 12.88 s, while throughput improved from 0.246 to 0.291 requests/s. We also profiled GPU memory usage, KV loading, and prefill overhead. (jXTD, Tables 12–13; 2wSw, Part 2, Tables 1–3.)
  6. Integration with KVZip (2wSw). At 70% KV retention, the prototype reduced persistent KV and selector-key storage from 5.32 to 3.77 GiB. At 30% recomputation, accuracy was 59.9% with FlashMEM+KVZip and 60.4% with FlashMem alone. (KVZip integration appendix.)

For the reviewers' reference, we have created an anonymous repository at https://anonymous.4open.science/r/anon-review-7F32D9/.

We will incorporate these results, related-work comparisons, and limitations, along with the other clarifications and improvements suggested by the reviewers, into the revised manuscript. We thank the reviewers again for their valuable time.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.