- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper suffers from very limited novelty, highly selective experimental evaluation, and a complete lack of latency measurements. I strongly encourage the authors to thoroughly reassess the quality and completeness of their experimental evaluation before submitting to a top-tier conference, rather than leaving reviewers to identify such fundamental shortcomings.
Critical strengths
Practical motivation. Compressing the KV cache before queries arrive and reusing the same compressed cache for multiple queries is a meaningful and practically relevant problem.
Critical weaknesses
- Very limited novelty. The proposed method is highly similar to VATP and CriticalKV, essentially combining KeyDiff with existing value-aware eviction strategies. Table 2 further shows that single-stage CriticalKV achieves nearly identical performance, making the novelty of VG-Evict highly questionable.
- Highly selective and insufficient evaluation. The main experimental results (Table 1 and Figures 1 and 3) focus almost exclusively on Passage Retrieval, just one of the 16 English datasets in LongBench. I think it unacceptable for a KV cache compression paper to build its main claims around a single favorable dataset rather than providing comprehensive benchmark evaluations.
- No latency evaluation. The paper provides no compression latency, inference latency, or throughput measurements, despite proposing a method intended to improve inference efficiency. Without these results, its practical efficiency remains unsubstantiated.
- Narrowly selected RULER evaluation. Table 4 evaluates only a specific subset of 100 RULER documents with four questions each, rather than providing a comprehensive evaluation. This again raises concerns about selective experimental reporting, especially given the very small improvement over Compactor.
- Poor performance on broader benchmarks. Table 11 clearly shows that VG-Evict underperforms Compactor and substantially trails KVzip on the broader 13-task RULER evaluation at both 16K and 32K context lengths. These unfavorable results are relegated to the appendix, while the main paper emphasizes results from narrowly selected tasks.
Questions for authors
see weakness