- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper shows how re-tokenization and chat templates break prefix-cache reuse across turns. It measures the extra prefill and KV-memory costs and tests three fixes in vLLM. DriftEvict evicts unused KV branches first without changing model inputs or sampling.
Critical strengths
- The paper studies a real serving problem: unchanged conversation text does not guarantee a reusable token prefix. The measurements clearly show the extra prefill and KV-memory costs across models and languages.
- The issue also matters for agent harnesses that omit earlier reasoning when rebuilding history. This case is not evaluated here, but it reinforces the practical relevance of the problem.
- The evaluation reports quality regressions rather than treating every fix as harmless. DriftEvict leaves model inputs and sampling unchanged and gives useful latency gains under load.
Critical weaknesses
- DriftEvict is tested with ordinary user replies and fixed rendering settings. A discussion of harness-side history changes and tool continuations would help clarify its scope.
- The serving tests use one idle-time distribution and report mean TTFT. Tail latency and different idle times would give a fuller picture of the benefit.
Questions for authors
- Can DriftEvict account for harness-side changes, such as dropping earlier reasoning, when predicting which KV blocks will no longer be reused?
- Can you report tail TTFT from the existing runs? How sensitive are the gains to idle time?
3 comments
Thanks for the additional experiments. The tail-latency and idle-time results address my concerns. I also appreciate the clarification about reasoning removal in the GLM template. I’ll keep my weak accept recommendation.
Thank you for the follow-up and for your thoughtful review. We are glad that the tail-latency and idle-time results address your concerns. The revised PDF includes both (Appendix A.9, and Section 4.2 with Appendix A.11).
Q2 follow-up (idle-time sensitivity)
The idle-time runs are complete. We scaled every sampled idle time by 0.5× and 2×, which gives medians of 36.8 s and 147 s against 73.5 s in the main setting. We kept the sessions, arrival rates, and measurement window of Figure 3. Mean TTFT (s), baseline → DriftEvict, at the four arrival rates of Figure 3 (lowest to highest):
Idle time sets how long each conversation's KV states wait for its next request. At a given arrival rate, longer idle times therefore leave more conversations waiting to reuse their KV states.
Doubling idle time brings the gain to lower arrival rates. At 2×, DriftEvict improves both mean TTFT and decoding throughput at every tested arrival rate on all three models. At the lowest rate, mean TTFT falls from 1.04 to 0.78 s on Qwen3.8-27B, from 0.32 to 0.25 s on Qwen3.5-4B, and from 1.46 to 0.85 s on GLM-4.7-Flash. At the highest rate, throughput rises from 316 to 332, from 596 to 651, and from 349 to 357 tokens/s.
Halving idle time keeps the gain at the highest rate, where mean TTFT falls from 2.93 to 1.92 s, from 0.46 to 0.40 s, and from 1.03 to 0.78 s. The revised PDF reports these results in Section 4.2 and shows the full comparison in Appendix A.11 (Figure 10).