acceptodds
Under review as a conference paper at ICLR 2027

Agent Memory That Pays Its Own Rent

Abstract

An agent can keep each memory item as text or as its key-value (KV) cache. Text is prefilled again at every read; the cache skips that prefill but pays for storage while it is kept. Serving engines such as SGLang and Mooncake keep the cache, and agent memory systems such as MemGPT and Mem0 keep text. The cheaper choice depends on the deployment. At the same hourly price, the reuse needed for caching to pay off differs 26.3 times between two accelerators. We present METER, a controller that derives a break-even period, over which storing a cache costs as much as recomputing it, from model configuration, storage prices, and measured prefill and load times. METER admits an item once two of its reads fall within one period; each read then renews the cache for another period, after which it lapses to text. On LongMemEval and LoCoMo, across 240 combinations of reader model and memory lifetime, METER costs at most 1.51 times an offline optimum that knows all future reads, and caches exactly the items that optimum caches. Never caching reaches 1.94 and METER without admission 1.99; rules that ignore storage prices reach 160 to 3,191. Twenty-one executed runs on two accelerators reproduce the item selection. On Mooncake serving traces, with over a thousand times more reuse, METER without admission costs 1.01 against 2.23 for never caching, and is provably within twice the cheapest choice on every gap between reads.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache, agent memory, LLM serving, prefix caching, cache placement, online algorithms, ski rental

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VvcTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks when an LLM agent's memory should be stored as text and when it should be kept as a KV cache. Text storage requires prefill computation whenever the memory is read. A KV cache avoids that work but consumes storage and may have to be loaded. The authors argue that the right choice depends on deployment-specific costs, model size, measured read costs, and how often the memory is reused.

They propose METER, a policy that calculates a break-even interval from these costs. An item enters the cache after two reads occur within that interval, and later reads extend its lease. The paper also presents a multi-tier formulation and proves a factor-two bound on per-gap cost for a version that does not use admission.

The evaluation replays LongMemEval and LoCoMo traces across 240 reader–residency combinations. Under the modeled assumptions, METER's cost is at most 1.51 times that of an offline optimum. Experiments on two accelerators examine single-tier execution and placement, while Mooncake trace replays illustrate cases where admission should be disabled.

Critical strengths
  • The paper addresses a practical question: retaining KV state is not necessarily cheaper than recomputation, especially for large memories that are read infrequently.
  • The cost model is understandable and adaptable. It distinguishes storage footprint, load cost, and recomputation cost, making the placement decision explicit.
  • METER is relatively simple and needs little workload-specific tuning. Using the same break-even interval for admission and expiration gives the policy an intuitive economic basis.
  • The evaluation includes useful comparisons and variations, including fixed TTL policies, calibration transfer, delayed bursts, different reader models, and hardware runs on two accelerators.
Critical weaknesses
  • Most of the evaluation tests an independent-item cost model rather than the behavior of a complete serving system. Shared capacity limits, contention, migration costs, and end-to-end latency are not evaluated together.
  • The reported cost ratios depend on the accounting model. The analytical replay uses a reference storage price; in the hardware experiment, prefill is priced using calibration, while resident cache hits are treated as having no incremental compute cost. These results do not by themselves demonstrate equivalent savings in a real deployment.
  • Evidence about agent-memory reuse is limited. LoCoMo's read timing is synthetic, while LongMemEval contains little reuse, a setting in which never caching is close to optimal. More representative long-term traces would make the practical case stronger.
  • The emphasis on matching the offline optimum's cached-item set may overstate what that result shows. METER observes only a short interval after its own run ends, while the offline policy can cache in advance of that interval. Agreement on selected items does not imply comparable hit rates or schedules.
Questions for authors

I appreciate the paper's explicit treatment of KV storage costs and the simple interpretation of its policy. It makes a useful point: the value of retaining an item depends not only on how often it is reused, but also on the device and the time between reads. Overall, I lean toward acceptance.

My main concern is how well the modeled savings carry over to an actual serving deployment. The formulation prices storage per byte over time and compute using calibrated accelerator occupancy. This is a useful starting point, but real systems have fixed accelerator allocations, limited cache capacity, batched prefill, and transfers that may overlap with other work. As a result, lower modeled cost may not translate directly into a lower bill or greater useful throughput.

Two kinds of experiments would help bridge this gap:

  1. Fix accelerator and cache capacity, then compare METER with renewable-TTL and capacity-aware baselines on throughput, latency, prefill work, and cache occupancy.
  2. For a fixed workload and latency target, measure the compute and storage each policy requires, including loading, cache management, and migration.

These comparisons would clarify when METER's cost model yields practical resource savings.

I also have several questions:

  • Could the authors test more realistic memory-read traces? The paper appropriately notes LoCoMo's synthetic chronology, but admission and expiration depend directly on read intervals. Results across timing permutations, as well as bursty and changing reuse patterns, would help establish robustness.
  • How should an operator choose online between METER with and without admission? Their relative performance reverses across readers in the serving experiments. A switching rule that uses only information available at runtime would make the recommendation more actionable.
  • Could the paper emphasize cache hits, avoided prefills, and cost regret alongside item-level admission F1? In the executed experiment, METER records 302 hits versus 751 for the offline reference despite selecting the same items. This difference matters when interpreting the selection result.
3 comments
AuthorsOpenReview ↗

Response to Reviewer VvcT

Thank you for a constructive review with concrete experiments. In short: (1) we ran both experiments you proposed; (2) the switching rule exists and uses only runtime information; (3) you are right about hits against F1, and the revision reports hits and regret.

Experiment 1 (fixed accelerator and cache capacity; throughput, latency, prefill work). Executed in vLLM on one accelerator (Table 20 in the revision, also in the general response). Retaining the cache raises throughput 48% and cuts median TTFT from 391 s to 8 s on tool-agent. Between policies, the engine's LRU (METER without admission in a paid-for tier) and METER with admission are equal on conversation and at 30% lighter load; near saturation, admission loses 3 points of hits and queueing turns that into 125 s against 8 s. So the executed evidence supports the paper's recommendation for serving and sharpens it: admission off under dense reuse.

Experiment 2 (fixed workload and latency target; resources required). Full-hour traces, HBM fixed at the executed size, a rented DRAM pool behind it. Cheapest configuration of each family that meets the target (LRU's knob is pool size, METER's its lease multiple, TTL's its lifetime); cost covers prefill, loading and the DRAM actually occupied.

tracemean TTFT targetLRUMETER familyfixed TTL
conversation0.53 s$4.01$3.81$4.00
0.50 s$4.30$4.28$5.07
tool-agent0.27 s$4.32$3.82$5.10
0.26 s$4.32$4.18$5.10

In the low-cost half of the latency range, shown here, the lease with admission is 0.3% to 12% cheaper than a sized LRU pool and 5% to 25% cheaper than fixed TTLs. At tighter targets the lease without admission and LRU trace the same curve, so neither is consistently cheaper there (Figure 11 in the revision shows the full curves). Two caveats: TTFT here is modelled from measured prefill and load times without queueing, and this tier is simulated. The simulator matches the executed engine's hit rates within 1.2 points on the HBM row when given executed timings.

Switching rule. During a warm-up the controller prices both configurations on the reads it has logged, with the paper's cost function, and runs the cheaper one. Charged exactly, with METER running during the warm-up:

METERno admissionwith the rule
240 benchmark cells, worst1.512.001.39
240 benchmark cells, mean1.0831.7801.066
24 serving cells, mean1.2471.1151.146

It picks the cheaper configuration in 258 of 264 cells at warm-ups of 10% and 20%.

W4 (hits, avoided prefills, regret beside F1). Agreed. Over the 120 permuted-LoCoMo cells METER selects the optimum's items (F1 1.00) but serves 2,613 of the optimum's 6,993 cache hits, 37%, at a mean regret of 16.4%. The executed ratio you cite, 302 of 751, is 40%. The cause is structural: METER builds at the reuse, the optimum before it. The revision adds hits and regret beside F1 (Table 24) and corrects the abstract.

W3 (more realistic traces, changing reuse). We agree that timestamped agent-memory traces would strengthen this part. We know of no public one, and the revision says so. The paper has 30 permuted orders and 27 constructed bursts. For changing reuse we joined a sparse month of LongMemEval, the Mooncake tool-agent hour and a second sparse month. The one-shot rule decides in the sparse phase and stays at METER's cost (1.22 at Qwen2.5-7B, where no admission costs 1.03). So the rule is never worse than METER when reuse changes; re-deciding as reuse changes is the natural extension.

W2 (dependence on the accounting model). The executed results are in hits, prefill tokens, requests per second and seconds; no price enters them. Prices enter only the dollar comparisons, which the paper sweeps.

W1. Capacity, contention and latency are now evaluated together by execution in HBM. Migration is simulated; the host-to-device copy it relies on measured 1.15 GB/s through shared memory and 2.82 GB/s pinned.

Your proposed experiments were the right ones. They confirm the cost model's practical consequences where we could execute, and they fix where admission should not be used. We hope this strengthens your assessment.

Reviewer_VvcTOpenReview ↗

Post-Rebuttal Comment

Thank you to the authors for the rebuttal and clarifications. I have read the response and considered it alongside the paper. I am maintaining my score of Weak Accept.

Dear Reviewer VvcT,

Thank you for reading our rebuttal and for your continued support of the paper. Your suggested experiments, the fixed-accelerator run and the runtime rule for switching admission, shaped the revision directly, and we are grateful for the time you gave to it.

Best regards, The authors

Reviewer_J4H9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper provides rough estimates of KV caching and prefill costs and identifies the conditions under which caching becomes beneficial. It also proposes a new caching policy and claims that the policy incurs only a modest cost overhead relative to an oracle policy.

Critical strengths

The attempt to analyze costs based on memory rental prices is very interesting. I also appreciate the effort to mathematically characterize the break-even point.

Critical weaknesses

In real serving environments, memory capacity constraints are unavoidable once the machine configuration is chosen, and most KV cache evictions are driven by memory pressure. Renting a machine also fixes the relative capacities of its memory tiers(HBM, DRAM, SSD..). An analysis that ignores these constraints is far removed from real deployment conditions.

Furthermore, time spent on prefill is time that cannot be spent on decoding, which effectively increases serving costs. The rental charges continue to accrue while the hardware performs prefill instead of decoding.

Prefill and decoding costs vary dynamically with the number of concurrent users and the workload. At a minimum, the analysis should have accounted for the number of concurrent users.

Questions for authors
  1. It appears that experiments were also conducted with MoE models. Why were the MoE results not included?

  2. Did the analysis account for the number of concurrent users?

  3. After a request arrives, the KV cache may remain in use on the device while text is generated during decoding. Was this additional period accounted for in the admission and reuse policies?

4.Once the hardware configuration is fixed, the memory capacities and their ratios across memory-tiers are constrained by that configuration. Why were these constraints not incorporated into the model’s assumptions?

Reviewer_9DBYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the economic tradeoff between retaining agent memory as text and storing its corresponding KV-cache representation. The authors propose METER, a cost-aware memory placement controller that determines whether caching is beneficial based on model architecture, storage pricing, prefill computation cost, and cache loading overhead. Specifically, an item is admitted when two reads occur within the break even period, and subsequent reads extend its retention lifetime. The authors evaluate METER using LongMemEval, LoCoMo, and Mooncake serving traces, comparing against an offline clairvoyant optimum and alternative caching policies. Across 240 evaluation configurations, METER achieves a worst case observed cost ratio of 1.51 relative to the offline optimum.

Critical strengths

The paper is well-written and addresses a practically relevant and underexplored aspect of agent memory management: the economic cost of maintaining reusable intermediate representations over time. Unlike approaches that primarily optimize retrieval accuracy or cache hit rates, METER explicitly accounts for the trade-off between storage expenditure and recomputation, providing a useful cost-based perspective on memory management. The derivation of a hardware-dependent break even period is intuitive and interpretable, allowing the placement decision to adapt to different model architectures and deployment costs without fitting workload specific parameters. The experimental study is relatively comprehensive in terms of model configurations and memory lifetimes, and the authors appropriately distinguish the theoretical guarantee for the no-admission variant from the empirical behavior of the full controller.

Critical weaknesses

The technical novelty of the proposed mechanism is limited as the underlying algorithmic mechanism is closely related to classical ski-rental and time-to-live caching strategies. While the hardware-aware cost calibration is useful, the extent of the fundamental algorithmic contribution remains somewhat limited. More importantly, the theoretical guarantee applies only to the no-admission variant rather than the full METER policy recommended for agent memory. The system model further assumes largely independent memory items and simplified storage accounting, without fully addressing finite cache capacity, concurrent workloads, shared prefixes, contention, and the complexity of composing multiple cached memory items into a single agent context. The hardware evaluation covers only single-tier placement on two accelerators, while the multi-tier extension remains analytical. The evaluation emphasizes monetary or normalized cost, and the improvements in end-to-end agent performance, latency, or operational efficiency are also important to present.

Questions for authors
  1. Can the authors derive a competitive bound for the complete METER algorithm, including both admission and renewable retention, or construct a worst-case counterexample that characterizes its limitations?

  2. How does METER handle memory reuse involving multiple independently stored passages, particularly when KV-cache composition requires positional adjustments or introduces additional computation?

  3. Can the authors demonstrate improvements in end-to-end agent execution cost and latency using an integrated multi-tier serving system rather than relying primarily on analytical accounting and trace replay?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.