- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RR-Evict, an eviction algorithm that evicts the tail of each trajectory in a round-robin manner to achieve lower tail TTFT. In comparison with LRU that may evict the whole trajectory, the experiments show that RR-Evict can reduce P99 TTFT by up to 75.4% and P99 uncached prompt tokens to 65.7%.
Critical strengths
- The proposed policy is simple to implement and does not require predictions of future request arrivals or tool execution latency.
- Addresses a practical problem in agentic LLM serving. Conventional LRU eviction may lead to high tail TTFT.
- Good implementation brings good evaluation results.
Critical weaknesses
- Limited novelty and technical contribution. The core idea of RR-Evict is straightforward, and is to reduce the workload of the long-tail tasks.
- The round-robin policy ignores workload heterogeneity. The policy treats idle agent trajectories uniformly, without considering their request arrival frequencies or expected time until the next invocation. Neither does the author consider the non-linear different prefill cost when the length is different. The analytical model is also weak to this.
- Insufficient evaluation. The evaluation does not reveal: (1) whether it has a impact on the average TTFT statistically, which may happen since the concurrency of prefill may drop; (2) omit other implementations such as T-LRU [1], ECE [2], Continnum [3], or Random Attention [4] (which may be considered as concurrent work); (3) the performance of recency tiering; (4) different model types, especially the models that involve linear attentions or GDN, such as Qwen 3.8, Kimi K3 and GLM-5.3 Flash; (5) the performance of RR-Evict in cases of context compactation.
[1] W. Zhang, Y. Li, et al. Tail-Optimized Caching for LLM Inference. NeurIPS 25.
[2] Y. Ren, L. Chen, et al. Transcending Cost-Quality Tradeoff in Agent Serving via Session-Awareness. NeurIPS 25.
[3] H. Li, R. He, et al. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. Arxiv 25.
[4] H. Wang, J. Qiu, et al. Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning. Arxiv 26.
Questions for authors
- How's the RR-Evict's performance compared to other algorithms listed in the Weaknesses ([1-4]) under your current setting?
- Can you please evaluate how recency-tiering works with the cases where no explicit stop notification exist? And can you please study how RR-Evict performs in its current form when context compactation happens?
- How sensitive is RR-Evict to heterogeneous request arrival frequencies? If some agents resume frequently while others remain idle for long periods, does uniform eviction still improve tail TTFT?
- What is the overhead of RR-Evict? How robust are the results to eviction chunk size and memory pressure? The evaluation uses 1024-token eviction chunks. How sensitive are the results to this choice? If possible, I suggest the authors also evaluate other models as detailed in the Weakness 3.