- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines where to keep the KV caches of inactive agent sessions: in GPU memory, host memory, storage, or nowhere (by recomputing them later). The authors model these choices as a multi-state dynamic power-management problem. Each tier has a cost for retaining a cache and a cost for bringing it back into use, allowing the paper to draw on learning-augmented ski-rental algorithms with stated consistency and robustness guarantees.
Using real coding-agent traces, the authors train a predictor based on information available when a request ends. The type of request—tool call or human interaction—provides a useful signal, but the model has trouble estimating the idle duration within each category. With the paper’s reference costs, a policy tuned on validation data cuts modeled GPU-seconds by roughly 11% compared with the prediction-free lower-envelope policy. That improvement mostly vanishes when the authors use measured hardware costs instead. The paper also derives a crossover context length for loading versus recomputation and measures the relevant costs across several GPU and storage setups.
Critical strengths
- This is a worthwhile systems problem. Keeping KV caches for inactive agent sessions consumes expensive memory, while evicting them too soon can mean substantial recomputation and added delay when a session resumes.
- The model is clear and easy to interpret. Treating each cache tier in terms of holding and wake-up costs connects the problem to established online algorithms and makes the assumptions behind the guarantees visible.
- The trace evaluation takes care to avoid leakage and separates data by user. In particular, excluding tool latency observed during an idle interval is important for making the predictability results credible.
Critical weaknesses
- The case for deploying the learned policy is still narrow. Its roughly 11% gain is tied to a chosen reference configuration; across the measured configurations, the improvement is only 0–0.9%. This leaves limited evidence that prediction is preferable to a robust policy that does not use predictions.
- The per-session model does not account for shared capacity limits or competition among sessions. Host-memory pressure is captured by a fixed rent parameter, even though that choice has a strong influence on whether prediction and storage help.
- The evaluation pairs trace simulation with component-level measurements, rather than running the policy inside a serving system. As a result, modeled GPU-seconds alone do not demonstrate improvements in throughput, resource use, or latency under concurrent workloads.
Questions for authors
Thank you for a well-prepared submission. I found the paper’s formulation clean and its analysis of when prediction helps—and when it does not—careful and useful. The combination of real traces, explicit robustness guarantees, and measured hardware costs gives readers a helpful view of the problem. I am generally positive about the paper and lean toward acceptance.
My main reservation is about the operating conditions in which the learned policy is actually useful.
The measurements change the picture suggested by the reference configuration. When host-memory transfers are inexpensive relative to recomputation, the prediction-free policy already performs well, and validation mostly opts not to trust the predictor. At present, the strongest practical takeaway may therefore be a robust prediction-free placement policy, along with guidance on when learning is worth adding.
It would help to make that deployment choice more concrete. How can a system relate the host-memory rent parameter to observable resource pressure, and decide whether it is in a regime where prediction is likely to save much? The roughly 10% gain at higher host-memory rent is intriguing, but its real-world relevance depends on whether those rent values occur in practical fleet settings.
I also have several questions:
- How should a deployed system set the holding-cost parameters? The interpretation of HBM occupancy is intuitive, but the DRAM and storage rents still seem to be modeling choices. Could they be tied to queueing effects, memory pressure, or useful work displaced by retaining caches?
- What happens when host memory has a hard capacity limit? Independent per-session decisions could send many caches to DRAM at once, creating contention or forcing evictions that are not covered by the current guarantees.
- Could the authors compare the duration regressor with a direct breakpoint classifier or a cost-sensitive tier predictor? Because duration accuracy does not necessarily translate into good placement decisions, this comparison could show whether the modest gains come from limited predictive signal or from optimizing the wrong target.
- How much do the learned policy and trust parameters vary across user splits? The trace contains many gaps but only 43 developers, and the appendix includes a case where the validation-selected trust level does not transfer. Repeated user-level splits with confidence intervals would help establish how robust the results are.