- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Random Attention, a KV cache eviction method that preserves the prompt and randomly retains reasoning tokens independently across attention heads. Across four reasoning models and six tasks, it achieves accuracy comparable to strong training-free eviction baselines and delivers 32–43% higher throughput than TriAttention in the evaluated vLLM setup. Controlled experiments suggest that prompt protection accounts for much of the performance gap between existing methods, while redundancy in reasoning traces helps tolerate random eviction. However, the method struggles with facts stated only once and needed much later, and its efficiency gains depend on workload and implementation.
Critical strengths
- The paper offers an interesting perspective: preserving the prompt can help maintain long-context capabilities in extended reasoning tasks, reducing the need for sophisticated KV cache eviction metrics.
- The authors propose a method that combines prompt preservation with random eviction, improving inference efficiency while maintaining competitive performance, and integrate it with existing inference kernels.
- The authors conduct extensive comparisons with training-free KV cache compression methods across multiple models and benchmarks, demonstrating the advantages of their approach.
- Further analysis shows that incorporating prompt preservation into existing methods can also improve their performance.
Critical weaknesses
- Is this method applicable to agentic and multi-turn settings? When the prompt spans multiple conversational turns, does the method remain effective?
- Could existing KV cache compression methods, particularly training-based approaches, implicitly learn to preserve the prompt? In other words, could prompt preservation account for much of their performance advantage over some training-free methods?
- The effectiveness of the proposed method should be validated across a broader range of model families.
- Does the method fail when the prompt is long or the KV cache budget is small? I suggest that the authors include a comparative analysis of its limitations under these conditions against other methods.
Questions for authors
Seeing Weakneesses.