- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SideKV, a KV cache compression method that builds on the intuition that the compressed context does not need to contain the information that remaining (initial and final) parts already have. The paper proposes a new training objective to train the encoder with this capability. A thorough evaluation is provided, spanning over a large number of models and compression approaches.
Critical strengths
This paper provides a well-motivated and novel approach, that allows compressed soft inputs to focus on modeling the parts that remaining parts do not already have. The experiments are extremely broad and thorough, spanning a wide range of models and compression approaches. The resulting approach enables significant speedup compared to full kv cache setup.
Critical weaknesses
The core weakness is practical value. Most evaluations rely on continuation negative log-likelihood (NLL) measured on pretraining corpora, which may not necessarily translate to practical downstream performance. The paper provides evaluation on LongBench in Table 4, but the performance drop compared to the full baseline is significant (dropping from 30.9 to 18.8 on average), making practical applicability of the approach questionable.
I acknowledge that the performance is better than the baselines compared in Table 4, and much faster than other high-performant baselines in Table 14, but the overall performance drop would be unacceptable in practical terms. The large drop in some of other baselines’ performance may also suggest that the specific experiment setup causes a large universal drop in performance (e.g. too aggressive compression ratio), making the comparison itself less valuable. It would be useful to know if the performance gain would also translate to more practical evaluation regimes (e.g., with modest compression ratio).
Questions for authors
Please see critical weakness.