- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focus on the scenario that long running agent will compact the context as summary for several times. The problem is that
- Compacted summary is constructed on previous request instead future one which lead to the risk of omiting important context
- Poorly organized kv storage and retainment strategy will introduce storage and computation storage.
Flashmem introduce the following framework to solve the previous problems
- Flashmem mechanism to construct the memory construction with source-linked evidence retrieval
- PIC based KV-reuse pipeline for memory construction and inference
Critical strengths
- The motivation of this paper is strong and realistic. The context compression is critical in current llm service workflow.
Critical weaknesses
- The related work part is insufficient. KVCache compression algorithm such as KVZip is also query agnonist but neigther introduced in the related work nor the baseline.
- The construction mechanism is lack of novelty. This mechanism is introduced by openhands one year ago and also claude code so the contribution of flashmem part will be reduced to the recall quality.
- The KVCache reuse pipeline doesn't form a strong contribution. Cacheclip already introduce the similar mechanism that use a small model to identify the important token and recompute. Cacheslide use the PIC like position encoding for agentic workflow KVCache reuse. The contribution is considered as incremental.
- There is no figure for system overview or algorithm mechanism.
Questions for authors
- For realistic llm serving workload, like trace from mooncake, what's the ttft tpot throughput p50 p95 p99 and latency for requests? What's the hbm pressure at these workload comparing to baseline and what's the computation overhead during the prefilling, select and recompute.
- How does the system accuracy performance comparing to KVZip and other related works?
- Comparing to Cacheclip and Cacheslide, what's the key difference of flashmem?
- Comparing to openhands etc. what's the key difference of memory construction, recall and source link.
- For figure 10, did you introduce the kvcache eviction mechanism to baseline algorithms? If not it won't be a fair comparison.
- For flashmem and kvcache reuse pipeline, could you provide a proper ablation study?
I would raise my score if all weakness and problems are solved.
8 comments
To Reviewer 2wSw (2/3)
Q1: Realistic llm serving workload experiment.
FlashMem uses HiCache to manage reusable KV across multiple storage tiers. All persistent memory chunk KV is stored on NVMe SSD and loaded into DRAM when the inference engine starts. Active KV blocks are then transferred to HBM as needed. During serving, KV blocks may reside in HBM, DRAM, or SSD, following the same hierarchical cache management mechanism as conventional inference systems.
Here we evaluate Kimi-K2.6 (TP=8) using the official SGLang v0.5.23 as the full-prefill baseline. FlashMem uses a fixed 30% recomputation budget. Public serving traces such as Mooncake release only request metadata (timestamps, token lengths, and block hash IDs), not request text. We therefore use a real serving workload.
- Workload . A real serving workload from anonymized production API requests traces with 469 (93.8%) involve tool calls, and 392 (78.4%) contain at least two tool-call steps after the final user message. The replayed input lengths are 27,585/31,940/81,990 tokens at P50/95/P99.
Table 1: Serving performance on real workload requests.
| Input (req/s) | Method | Throughput (req/s) | Output (tok/s) | TTFT P50/P95/P99 (s) | TPOT P50/P95/P99 (ms) | E2E P50/P95/P99 (s) |
|---|---|---|---|---|---|---|
| 0.15 | SGLang | 0.146 | 4.65 | 6.89 / 16.74 / 21.06 | 19.70 / 375.52 / 420.22 | 8.47 / 21.50 / 28.49 |
| 0.15 | Ours DRAM | 0.146 | 4.66 | 3.25 / 7.97 / 12.32 | 17.73 / 24.01 / 38.96 | 3.82 / 8.53 / 12.92 |
| 0.15 | Ours SSD | 0.146 | 4.66 | 3.50 / 8.44 / 12.83 | 20.08 / 28.78 / 37.69 | 4.14 / 9.06 / 13.50 |
| 0.30 | SGLang | 0.246 | 7.88 | 154.47 / 307.51 / 318.31 | 164.86 / 435.57 / 526.03 | 159.58 / 310.36 / 325.91 |
| 0.30 | Ours DRAM | 0.291 | 9.31 | 4.32 / 12.88 / 18.00 | 18.13 / 25.74 / 38.94 | 4.95 / 13.68 / 18.47 |
| 0.30 | Ours SSD | 0.291 | 9.31 | 4.76 / 13.70 / 19.27 | 18.65 / 29.77 / 42.05 | 5.35 / 14.55 / 20.05 |
1) HBM pressure. All configurations use the same 110K-token GPU KV pool. At 0.30 req/s, FlashMem uses 3.43 GiB more peak HBM per GPU than SGLang (94.02% vs. 90.44% utilization), while P95 KV pool occupancy remains comparable at 93.96%/94.82%/94.90% (SGLang/DRAM/SSD). All configurations peak near 99.8% KV pool occupancy.
Table 2. HBM usage and KV pool occupancy under serving workloads.
| Input (req/s) | Method | Peak HBM per GPU (MiB) | HBM Utilization | KV Pool Utilization (P95 / Peak) |
|---|---|---|---|---|
| 0.15 | SGLang | 88,507 | 90.43% | 78.80% / 99.86% |
| 0.15 | Ours DRAM | 92,023 | 94.02% | 80.08% / 98.67% |
| 0.15 | Ours SSD | 92,023 | 94.02% | 79.43% / 98.59% |
| 0.30 | SGLang | 88,511 | 90.44% | 93.96% / 99.82% |
| 0.30 | Ours DRAM | 92,023 | 94.02% | 94.82% / 99.81% |
| 0.30 | Ours SSD | 92,023 | 94.02% | 94.90% / 99.75% |
2) Computation overhead. To explain the serving latency, we further profile the per-stage TTFT costs in Table 3. Draft selection takes 191/183 ms (DRAM/SSD). Target-model prefill takes 2.05/2.05 s, recomputing/prefill selected tokens and question while reusing cached KV for the rest. Request dispatch, queueing, and KV preparation account for the remaining latency, yielding a mean TTFT of 3.47/3.55 s.
Table 3. FlashMem TTFT latency breakdown at 0.15 req/s (mean per request).
| Component | KV in DRAM (ms) | KV in SSD (ms) |
|---|---|---|
| Selection model overhead | 191.51 | 183.49 |
| Selector queueing | 9.42 | 8.75 |
| Request dispatch & prefill waiting | 952.09 | 1,006.34 |
| Batch scheduling & KV preparation | 274.82 | 302.82 |
| Target prefill (selective recomputation) | 2,046.48 | 2,045.98 |
| Mean TTFT | 3,474.31 | 3,547.38 |
3) KV loading. For this workload with 32K input tokens, isolated cool KV restoration takes 1.06/1.20 s (P50) from DRAM/SSD. During serving, asynchronous loading overlaps scheduling waits, leaving 112.09/167.94 ms (P50) of foreground waiting per restore call on rank 0, included in KV preparation. Host-to-GPU transfers average 70.90/74.41 ms per batch, with 63.78/60.97 ms overlapping target execution.
Q5: Clarifying Persistent KV Storage Accounting in Figure 10.
Thank you for the question. We clarify that Figure 10 measures the persistent KV storage required for each method's final memory store, rather than runtime cache occupancy or intermediate states generated during construction. In agent memory benchmarks, all final memory entries are retained for future RAG retrieval. To support PIC reuse across queries, their KV states must also be persistently stored; otherwise, retrieved entries require recomputation. This comparison essentially reflects the size of each method's final memory store, which could also be measured in text tokens or bytes.
KV eviction is a separate serving-level mechanism. HiCache manages KV residency across HBM, DRAM, and SSD without changing the persistent storage requirements measured in Figure 10. We will clarify this distinction in the revised manuscript.
To Reviewer 2wSw (3/3)
W2&Q4: Differences from OpenHands and Claude Code.
Thank you for highlighting these systems. We acknowledge that summarization and source references already exist in them. The difference lies in what the links are used for. The source-link in context compaction and persistent memory serve different purposes.
Context compaction. OpenHands' source link happen in its Context compaction. LLMSummarizingCondenser records summaries alongside forgotten_event_ids, and identifies the original events. These links provide traceability for solving current context window limitations, rather than indexing evidence for future retrieval.
Persistent memory. OpenHands SDK and Claude Code use file-based memory indexes (e.g., MEMORY.md) and read linked files on demand. Following a reference is agent-driven: the LLM must decide that more information is needed, choose a file, and issue tool calls. Each lookup therefore costs extra LLM inference and tool latency. FlashMem instead integrates source linking into the memory construction and rag retrieval pipeline. It constructs independently retrievable memory entries linked to original records. At recall, these links guide raw-evidence retrieval under a separate token budget, without additional agent-planned file access. The distinction lies in how source links support construction and retrieval, rather than source linking itself.
We will expand the Related Work section in the revised version to discuss these works in greater detail. Thank you again for highlighting them and for your questions.
W4: Paper Improve.
We apologize for the lack of visual clarity. We will add a pipeline diagram illustrating the overall architecture and key algorithmic mechanisms in the revised version.
Q6: More Ablation Study.
Thank you for the suggestion. We further examine two key components of FlashMem: memory organization and KV reuse.
Memory organization. Table 3 already compares summary-only, raw-only, independent retrieval, and source-linked retrieval. To further isolate source linking, we compare independent RAG and source-linked retrieval using only raw evidence for answering.
Table 4: Raw-evidence-only QA on LoCoMo (150-token budget; Accuracy %).
| Model | RAG | Source-linked | Δ [95% CI] |
|---|---|---|---|
| Mistral-7B-Instruct-v0.3 | 34.16 | 43.44 | +9.29 [7.50, 11.09] |
| Llama-3.1-8B-Instruct | 29.16 | 37.27 | +8.12 [5.52, 10.93] |
| Gemma-4-26B-A4B-it | 42.86 | 54.29 | +11.43 [9.04, 13.79] |
| Qwen3-32B | 46.10 | 57.14 | +11.04 [9.24, 12.92] |
| GLM-4.5-Air | 43.70 | 53.57 | +9.87 [7.66, 12.52] |
| DeepSeek-V3.2 FP8 | 44.16 | 56.62 | +12.47 [9.58, 15.31] |
| Kimi-K2.6 | 49.48 | 60.78 | +11.30 [8.98, 13.54] |
KV reuse. We conduct controlled ablations on LoCoMo/Qwen3-8B/Qwen-0.6b Embedding with 30% recomputation budget. We first vary layer aggregation, including the first half, last layer, and the 10 layers with the highest attention-score entropy. We then examine score normalization: chunk-wise Softmax normalizes QK scores independently within each chunk before global Top-k selection. For token selection, window grouping selects 8-token windows containing at least 5 candidate tokens. Finally, we disable RoPE correction to evaluate positional alignment. All other settings remain unchanged.
Table 5: KV reuse component ablation on LoCoMo (Accuracy %).
| Scoring | Layer Aggregation | Token Selection | RoPE Fix | Acc. (%) |
|---|---|---|---|---|
| Global QK (Ours) | Later half | Token Top-k | ✓ | 60.4 |
| Global QK | First half | Token Top-k | ✓ | 49.1 |
| Global QK | Last layer | Token Top-k | ✓ | 57.5 |
| Global QK | Entropy Top-10 | Token Top-k | ✓ | 57.0 |
| Chunk-wise Softmax | Later half | Token Top-k | ✓ | 57.7 |
| Global QK | Later half | Window (8, ≥5) | ✓ | 58.9 |
| Global QK | Later half | Token Top-k | × | 58.2 |
We hope these additional experiments and clarifications address your concern. Thank you for guiding us toward a clearer and more polished presentation!
To Reviewer 2wSw (1/3)
We thank the reviewer for the detailed and constructive feedback. Before addressing each point, we clarify FlashMem's setting, since it affects several comparisons below.
FlashMem targets
persistent agent memory: memory is built once from long interaction histories and retrieved across future sessions and queries. This differs from in-session context compaction, which frees the current window so that an ongoing task can continue. FlashMem makes two contributions:A source-linked memory organization that keeps persistent memory compact while preserving access to raw evidence.
A KV-reuse pipeline (PIC) that makes serving this memory efficient.
Thank you for pointing out KVZip.
KVZip prunes existing KV caches to reduce storage and decoding costs, whereas FlashMem's PIC reuses cached KV to reduce prefill costs.
FlashMem also reduces KV storage, but at the prompt level. Unlike KVZip, which compresses the KV representation of a given context, FlashMem reduces the context itself before KV generation. It constructs compact, source-linked memories that preserve essential facts and source pointers, retrieving original evidence on demand to reduces persistent KV storage while retaining access to raw evidence. KVZip(KV prunes) and FlashMem (KV reuse) are complementary topic.
KVZip compresses the KV cache of a context that has already been prefilled. Agent histories often exceed the model's context window. In our additional benchmarks and application case, histories range from 500K tokens (BEAM-500K, AMA-Bench) to ~25M tokens (LongMemEval-V2), so a single full-history KV cache cannot be built in the first place. KVZip is evaluated in contexts of up to ~170K tokens. While evaluating their integration is beyond the scope of this work, we will to explore the oppotunity of combining KV compression and KV reuse in future work.
<1>. Differences from CacheClip.
Selection efficiency. CacheClip performs a separate chunk-conditioned query prefill for each retrieved chunk, whereas FlashMem prefills the query once and scores all chunks via (QK⊤) using the last 12 layers' cached keys. For Qwen2.5-0.5B with eight 1,024-token chunks and a 128-token query, this reduces estimated selection computation by
86%(0.80→0.115 TFLOPs). Retaining only these layers' keys further reduces auxiliary KV storage by75%(384→96 MiB for 32K BF16 tokens).Token selection. CacheClip independently normalizes attention scores within each chunk-conditioned context, concatenates them for global Top-k selection, and applies sliding-window grouping. FlashMem instead applies a shared Softmax across all retrieved chunks, followed by direct token-level Top-k selection avoiding the potential amplification of irrelevant chunks caused by local normalization.
<2>. Differences from CacheSlide.
Scope and position handling. CacheSlide targets relative-position-dependent reuse, where prompt-template chunks retain their relative order despite shifts in absolute positions. Its CCPE learns chunk position ranges through target-model LoRA training. In contrast, FlashMem handles position-independent reuse, where retrieved memories and raw-evidence spans vary in combination and order across queries. We adopt training-free RoPE correction without modifying the target model and do not claim novelty in position correction.
Recomputation selection. Like CacheBlend [1], CacheSlide relies on KV-deviation-based selection, requiring full-prompt computation at the target model's first layer before selecting tokens. FusionRAG [2] improved this criterion through query-guided attention scores. In contrast, FlashMem performs token selection with a lightweight draft model, avoiding full-prompt prefill at every target-model layer, including the first.