- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TridentKV, a training-free KV cache eviction framework for efficient long-context LLM inference. The method combines uniqueness-based layer budget allocation, behavior-guided head budget allocation, and spatial-temporal token score aggregation. It aims to preserve diverse attention behaviors while improving token retention under constrained cache budgets. Experiments on LongBench, RULER, and Needle-in-a-Haystack across Llama, Mistral, and Qwen models demonstrate competitive performance compared with existing KV cache eviction methods. The paper also reports decoding acceleration and memory savings under long-context settings.
Critical strengths
- The paper addresses an important practical problem in long-context LLM inference and proposes a training-free solution.
- The method combines layer-level and head-level budget allocation with spatial-temporal token importance aggregation.
- The experiments cover multiple LLMs and benchmarks, with competitive results and ablation studies supporting the proposed components.
Critical weaknesses
- The proposed budget allocation relies on attention-pattern diversity, but its relationship with the functional importance of individual attention heads is not sufficiently established.
- Although prefill and decoding latency are reported separately, the end-to-end inference benefits over existing KV eviction methods remain unclear.
- The reported LongBench results differ between Table 1 and Table 10 despite using the same stated cache budget and model configurations.
Questions for authors
- Could the authors provide a small-scale analysis of whether attention-pattern similarity reflects the functional importance of attention heads, for example, through head-level output perturbation measurements?
- Could the authors clarify whether the additional prefill overhead affects the overall inference efficiency compared with existing KV eviction methods?
- Could the authors explain the differences in the LongBench results between Table 1 and Table 10 under the same stated experimental settings?