- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper present DASH-KV, using approximate retrieval in a learned local sensitive hash (LSH) to reduce KV Cache IO during inference. It use different hash encoders for query and key cache, hamming-distance-based relevance estimation with calibration cross heads and layers. It split attention scheme into 3 level, exact computing for highly relevant tokens, hash-based scoring with learned correction to moderately relevant tokens, and ignoring the rest tokens to reduce KV Cache IO. The hash and residual correction modules are trained via attention mass distribution distillation while keeping the LLM backbone frozen. Evaluation on LongBench tasks and 3 LLM backbones shows accuracy close to original full attention while reduce single-layer attention latency for long contexts.
Critical strengths
- DASH-KV reduce KV Cache IO with only slightly accuracy loss across different LLM backbone and tasks.
- S1The implementation with packed XOR and POPCNT matching, fused routing and sparse attention kernels is solid in engineering.
- The paper provides extensive ablations and layer-wise analyses, including asymmetric and symmetric Q/K encoder for hashing, hash-code length, encoder depth and correction...
Critical weaknesses
- Limited novelty and lack literature survey and comparison to closely related retrieval based methods.
Beyond KV Cache eviction methods like H2O and StreamingLLM, there are already a lot of retrieval based methods for long context inference using ANNS and LSH. For example, RetrievalAttention [1] build ANN indices over KV vectors, MagicPIG [2] use LSH-based sampling, HashAttention [3] separate Q/K mappings into hamming space and HATA [4] applies similar learned binary hashing. All these works are published 1-2 years ago, and already find the Out of Distribution (OOD) problem for attention QK retrieval with more details and use similar "asymmetric" methods to solve the problem. But this paper neither discusses its differences to these related works nor include these more relevant retrieval methods as baselines. This is particularly concerning because Section 2.2 claim that hashing and transformer attention "has not been extensively investigated", "first systematic integration of hashing techniques for attention optimization". Given the substantial prior work, these novelty claims appear overstated, and the actual methodological contribution of DASH-KV remains unclear.
- Insufficient study and ablation for the proposed design choices.
Several key design choice are insufficiently motivated or isolated.
- First, like W1 talks, prior works already identified the gap between classical ANNS to Q/K retrieval, or the OOD issue. Since Q and V cache are transformed by different linear projection, they have different distribution. So previous works [1-4] deploys different methods including "asymmetric" methods to solve the problem. But this work only shows that "asymmetric > symmetric" without further study about whether the gain coms from Q/K OOD, larger nolinear query encoder or simply more learned capacity.
- Second, this work always keeps the first 4 token and most recent 64 token in full precision. Similar recent-window and first "sink-token" strategies have already appeared in StreamingLLM[6], KiVi[5] and Keyformer[7]. And some recent works and models try to make "sink-free" like GatedAttention[11] in Qwen3.5 family. Yet this work lacks abaltions on the recent window and sink-token accuracy contribution to the accuracy and the difference on "sink-free" GatedAttention models.
- Finally, the necessity for training a residual corrector is unclear. Methods like CaliDrop[8] and DapQ[9] uses historical queries and position-aware pseudo to achieve training-free correction. This work does not discuss why the learned correction is needed or can it outperform training-free methods.
- Insufficient serving-level evaluation of the claimed acceleration.
In efficiency evaluation part, latency is mainly compared with full attention, while even those baselines used in accuracy comparison are omit in the efficiency comparison, not to mention the more relevant hash and retrieval methods in W1 [1-4]. What's worse, the efficiency comparison only act on single attention layer in decoding. The retrieval overheads in the real generation like the hash initialization overhead after prefill, the new generated token hash insert, the recent-window maintenance may be omitted in such evaluation. And the evaluation is also primarily at batch size = 1. The end-to-end benefit like TPOT and TTFT improvement under different context length, batch size also absence.
- The overall deployment benefit remains unclear beyond local attention acceleration.
DASH-KV retains the full-precision KV cache and additionally stores hash codes and auxiliary parameters. Table 2A reports 1,722.66 MiB for FullKV versus 1,740.30 MiB for DASH-KV, so this work do not reduce memory cost. In contrast, retrieval methods such as MagicPIG [2] can offload KV-related retrieval and attention computation to CPU memory, reducing GPU memory pressure. Together with the missing end-to-end results in W3, the overall speed--memory--quality benefit of DASH-KV is therefore unclear.
The generalization of the learned hash/residual modules is also insufficiently validated. They are trained on data from NarrativeQA, HotpotQA, Qasper, MultiNews, GovReport, and TriviaQA, which are the sources of LongBench evaluation's 6 tasks. Although the appendix clarifies that training uses the original datasets' training splits rather than LongBench test split, this mainly demonstrates same-task generalization rather than cross-task transfer. A stronger evaluation should freeze trained DASH-KV module and test it without adaptation on unseen task families, e.g., math reasoning, code generation, or long-context agent workloads, or explicitly characterize whether workload-specific retraining is required.
[1] Di Liu, Meng Chen, Baotong Lu, et al.
"RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval."
arXiv:2409.10516, 2024.
[2] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, et al.
"MagicPIG: LSH Sampling for Efficient LLM Generation."
International Conference on Learning Representations (ICLR), 2025.
[3] Aditya Desai, Shuo Yang, Alejandro Cuadron, Matei Zaharia,
Joseph E. Gonzalez, and Ion Stoica.
"HashAttention: Semantic Sparsity for Faster Inference."
International Conference on Machine Learning (ICML), 2025.
[4] Ping Gong, Jiawei Yi, Shengnan Wang, et al.
"HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention
for Scalable Large Model Inference."
Findings of the Association for Computational Linguistics: ACL, 2025.
[5] Zirui Liu, Jiayi Yuan, Hongye Jin, et al.
"KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache."
International Conference on Machine Learning (ICML), 2024.
[6] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
"Efficient Streaming Language Models with Attention Sinks."
International Conference on Learning Representations (ICLR), 2024.
[7] Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, et al.
"Keyformer: KV Cache Reduction through Key Tokens Selection
for Efficient Generative Inference."
Proceedings of Machine Learning and Systems (MLSys), 2024.
[8] Yi Su, Quantong Qiu, Yuechi Zhou, et al.
"CaliDrop: KV Cache Compression with Calibration."
arXiv:2507.19906, 2025.
[9] Zhenxu Tian, Yi Su, Juntao Li, and Min Zhang.
"Where Matters More Than What: Decoding-aligned KV Cache Compression
via Position-aware Pseudo Queries."
arXiv:2603.11564, 2026.
[10] Yushi Bai, Xin Lv, Jiajie Zhang, et al.
"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding."
Proceedings of the 62nd Annual Meeting of the Association for Computational
Linguistics (ACL), 2024.
[11] Qiu Z, Wang Z, Zheng B, et al.
Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free[J].
Advances in Neural Information Processing Systems, 2026, 38: 100092-100118.
Questions for authors
In Table 1, the results on MultiNews (MN) and GovReport (GR) are somewhat counter-intuitive. Across all 3 backbones, eviction-based baselines such as StreamingLLM, H2O, and SnapKV consistently achieve substantially higher scores than Full Attention on these two tasks, but DASH-KV remains close to Full Attention. Could the authors explain this behavior? Is this improvement caused by the evaluation metric or differences in generated output length, or does aggressive KV eviction/sparsification genuinely improve performance on these summarization tasks? Additional analysis of generation length, decoding behavior, and the corresponding evaluation metrics would help clarify whether this is a real modeling effect or an evaluation artifact.