- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a harness-agnostic KV-cache prefetch engine that reduces LLM agent latency by warming cached prefixes during tool execution. It handles cache eviction, newly returned tool outputs, and subagent cold starts using background inference requests. A Thompson Sampling policy decides when prefetching is beneficial to avoid unnecessary overhead. Experiments across Claude Code, Hermes, and Codex show 19.6–68.9% reductions in aggregate time to first token, with only 0.5% additional prompt-token overhead at eight concurrent sessions. The main limitation is that prefetching consumes inference capacity and may interfere with foreground requests under high load.
Critical strengths
- Good direction on agentic inference.
Critical weaknesses
- Terrible use of AI technology in paper writing. The graphs are AI generated and screenshot into the paper.
- No comparison with previous methods over the past 2 years...
Questions for authors
- How is this novel and not done by previous people? YOu have no comparison with prior work!! This is outrageous