- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper asks when an LLM agent's memory should be stored as text and when it should be kept as a KV cache. Text storage requires prefill computation whenever the memory is read. A KV cache avoids that work but consumes storage and may have to be loaded. The authors argue that the right choice depends on deployment-specific costs, model size, measured read costs, and how often the memory is reused.
They propose METER, a policy that calculates a break-even interval from these costs. An item enters the cache after two reads occur within that interval, and later reads extend its lease. The paper also presents a multi-tier formulation and proves a factor-two bound on per-gap cost for a version that does not use admission.
The evaluation replays LongMemEval and LoCoMo traces across 240 reader–residency combinations. Under the modeled assumptions, METER's cost is at most 1.51 times that of an offline optimum. Experiments on two accelerators examine single-tier execution and placement, while Mooncake trace replays illustrate cases where admission should be disabled.
Critical strengths
- The paper addresses a practical question: retaining KV state is not necessarily cheaper than recomputation, especially for large memories that are read infrequently.
- The cost model is understandable and adaptable. It distinguishes storage footprint, load cost, and recomputation cost, making the placement decision explicit.
- METER is relatively simple and needs little workload-specific tuning. Using the same break-even interval for admission and expiration gives the policy an intuitive economic basis.
- The evaluation includes useful comparisons and variations, including fixed TTL policies, calibration transfer, delayed bursts, different reader models, and hardware runs on two accelerators.
Critical weaknesses
- Most of the evaluation tests an independent-item cost model rather than the behavior of a complete serving system. Shared capacity limits, contention, migration costs, and end-to-end latency are not evaluated together.
- The reported cost ratios depend on the accounting model. The analytical replay uses a reference storage price; in the hardware experiment, prefill is priced using calibration, while resident cache hits are treated as having no incremental compute cost. These results do not by themselves demonstrate equivalent savings in a real deployment.
- Evidence about agent-memory reuse is limited. LoCoMo's read timing is synthetic, while LongMemEval contains little reuse, a setting in which never caching is close to optimal. More representative long-term traces would make the practical case stronger.
- The emphasis on matching the offline optimum's cached-item set may overstate what that result shows. METER observes only a short interval after its own run ends, while the offline policy can cache in advance of that interval. Agreement on selected items does not imply comparable hit rates or schedules.
Questions for authors
I appreciate the paper's explicit treatment of KV storage costs and the simple interpretation of its policy. It makes a useful point: the value of retaining an item depends not only on how often it is reused, but also on the device and the time between reads. Overall, I lean toward acceptance.
My main concern is how well the modeled savings carry over to an actual serving deployment. The formulation prices storage per byte over time and compute using calibrated accelerator occupancy. This is a useful starting point, but real systems have fixed accelerator allocations, limited cache capacity, batched prefill, and transfers that may overlap with other work. As a result, lower modeled cost may not translate directly into a lower bill or greater useful throughput.
Two kinds of experiments would help bridge this gap:
- Fix accelerator and cache capacity, then compare METER with renewable-TTL and capacity-aware baselines on throughput, latency, prefill work, and cache occupancy.
- For a fixed workload and latency target, measure the compute and storage each policy requires, including loading, cache management, and migration.
These comparisons would clarify when METER's cost model yields practical resource savings.
I also have several questions:
- Could the authors test more realistic memory-read traces? The paper appropriately notes LoCoMo's synthetic chronology, but admission and expiration depend directly on read intervals. Results across timing permutations, as well as bursty and changing reuse patterns, would help establish robustness.
- How should an operator choose online between METER with and without admission? Their relative performance reverses across readers in the serving experiments. A switching rule that uses only information available at runtime would make the recommendation more actionable.
- Could the paper emphasize cache hits, avoided prefills, and cost regret alongside item-level admission F1? In the executed experiment, METER records 302 hits versus 751 for the offline reference despite selecting the same items. This difference matters when interpreting the selection result.
3 comments
Post-Rebuttal Comment
Thank you to the authors for the rebuttal and clarifications. I have read the response and considered it alongside the paper. I am maintaining my score of Weak Accept.
Thank you
Dear Reviewer VvcT,
Thank you for reading our rebuttal and for your continued support of the paper. Your suggested experiments, the fixed-accelerator run and the runtime rule for switching admission, shaped the revision directly, and we are grateful for the time you gave to it.
Best regards, The authors
Response to Reviewer VvcT
Thank you for a constructive review with concrete experiments. In short: (1) we ran both experiments you proposed; (2) the switching rule exists and uses only runtime information; (3) you are right about hits against F1, and the revision reports hits and regret.
Experiment 1 (fixed accelerator and cache capacity; throughput, latency, prefill work). Executed in vLLM on one accelerator (Table 20 in the revision, also in the general response). Retaining the cache raises throughput 48% and cuts median TTFT from 391 s to 8 s on tool-agent. Between policies, the engine's LRU (METER without admission in a paid-for tier) and METER with admission are equal on conversation and at 30% lighter load; near saturation, admission loses 3 points of hits and queueing turns that into 125 s against 8 s. So the executed evidence supports the paper's recommendation for serving and sharpens it: admission off under dense reuse.
Experiment 2 (fixed workload and latency target; resources required). Full-hour traces, HBM fixed at the executed size, a rented DRAM pool behind it. Cheapest configuration of each family that meets the target (LRU's knob is pool size, METER's its lease multiple, TTL's its lifetime); cost covers prefill, loading and the DRAM actually occupied.
In the low-cost half of the latency range, shown here, the lease with admission is 0.3% to 12% cheaper than a sized LRU pool and 5% to 25% cheaper than fixed TTLs. At tighter targets the lease without admission and LRU trace the same curve, so neither is consistently cheaper there (Figure 11 in the revision shows the full curves). Two caveats: TTFT here is modelled from measured prefill and load times without queueing, and this tier is simulated. The simulator matches the executed engine's hit rates within 1.2 points on the HBM row when given executed timings.
Switching rule. During a warm-up the controller prices both configurations on the reads it has logged, with the paper's cost function, and runs the cheaper one. Charged exactly, with METER running during the warm-up:
It picks the cheaper configuration in 258 of 264 cells at warm-ups of 10% and 20%.
W4 (hits, avoided prefills, regret beside F1). Agreed. Over the 120 permuted-LoCoMo cells METER selects the optimum's items (F1 1.00) but serves 2,613 of the optimum's 6,993 cache hits, 37%, at a mean regret of 16.4%. The executed ratio you cite, 302 of 751, is 40%. The cause is structural: METER builds at the reuse, the optimum before it. The revision adds hits and regret beside F1 (Table 24) and corrects the abstract.
W3 (more realistic traces, changing reuse). We agree that timestamped agent-memory traces would strengthen this part. We know of no public one, and the revision says so. The paper has 30 permuted orders and 27 constructed bursts. For changing reuse we joined a sparse month of LongMemEval, the Mooncake tool-agent hour and a second sparse month. The one-shot rule decides in the sparse phase and stays at METER's cost (1.22 at Qwen2.5-7B, where no admission costs 1.03). So the rule is never worse than METER when reuse changes; re-deciding as reuse changes is the natural extension.
W2 (dependence on the accounting model). The executed results are in hits, prefill tokens, requests per second and seconds; no price enters them. Prices enter only the dollar comparisons, which the paper sweeps.
W1. Capacity, contention and latency are now evaluated together by execution in HBM. Migration is simulated; the host-to-device copy it relies on measured 1.15 GB/s through shared memory and 2.82 GB/s pinned.
Your proposed experiments were the right ones. They confirm the cost model's practical consequences where we could execute, and they fix where admission should not be used. We hope this strengthens your assessment.