- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes HorizonKV, a two stage sparse recovery framework for KV cache reuse across different agent contexts. In KV-Cache reuse scenario, HorizonKV computes the relevance score of each token using mean of dynamic Qs in early layers and select the tokens to re-prefill, while leaving the other tokens' KV cache unchanged. HorizonKV also adopts K-sparse attention with the same relevance scores. HorizonKV presents additional ideas built upon CacheBlend (Yao et al., 2024).
Critical strengths
None.
Critical weaknesses
-
Weak logical, experimental justification of using mean-Q. Broadly speaking, the idea of predicting future attention demand using prompt queries is a fairly well-established idea. However, the justification for using query mean as a proxy is not enough, comparing this only with the last query and a random query. we can compare this with recent queries' attention patterns (SnapKV, PyramidKV, FastKV, ...) or CacheBlend method (From Fig.3, we can see that cacheblend can recall upto ~50% of top-k for the future Q). More fundamentally, since HorizonKV also predicts later layers from early layers using the current tokens, it belongs to the category of query/context relevance, and the term "future relevance" is a stretch.
-
Baselines are weak. Since CacheBlend, several other efficient KV reuse methods have been appeared, so it is natural to compare HorizonKV with them. Comparable works include RelayCaching, KVCOMM, ProphetKV. Like these works, HorizonKV ultimately aims to reduce latency (TTFT, TPOT) with minimal accuracy degradation. Contrary to the statement in Appendix E, these methods are not complementary to HorizonKV: combining them with HorizonKV is either infeasible or, even if feasible, would further aggravate accuracy degradation. Therefore, a head-to-head comparison is necessary.
-
The paper lacks the explanations needed to verify that the comparison with the baselines is fair:
- HorizonKV performs full attention up to layer 4. Does CacheBlend perform full attention up to layer 2?
- The recomputation ratio described in the Appendix appears to exclude sink tokens and early layers from the count. How was this applied to the baselines?
- From Figure 6 and Table 2, Random-Q appears to achieve higher accuracy than CacheBlend. This implies that some factor in HorizonKV other than T1 and T2 already gives it an advantage over the CacheBlend baseline. This needs to be specified and justified.
- Table 2 shows a notable accuracy gain from T2 (Table 7 also shows.). This is a phenomenon frequently observed in KV eviction studies, arising from the removal of attention noise from unimportant tokens. If so, can HorizonKV be compared with CacheBlend combined with a KV eviction method such as SnapKV?
In other words, some results suggest that CacheBlend and HorizonKV are not being compared fairly, and the Evaluation section does not clearly resolve this issue.
Questions for authors
-
I would like to see detailed configurations, a sound interpretation of the results, and additional experiments that address the issues raised in Critical Weaknesses 2 and 3.
-
Please specify which segments are treated as cached(historical) vs dynamic for each agent trace (multi, single), which segments are the cached KV, where they are from, and the positional relationship/distribution of dynamic and historical context. For example, we can describe an agent trace like this: Agent1 prefills A+B and decodes C. Agent2 prefills A+D+C and decodes E. Lastly Agent3 prefills A+F+C+E and decodes G. For Agent3, A,F,C,E are historical, C,E are cached KV, and the last few tokens of F are the dynamic tokens.