acceptodds
Under review as a conference paper at ICLR 2027

DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing

Abstract

The increasing memory footprint of the KV cache and the high computational cost of query-key similarity computation constitute two fundamental bottlenecks for large language models in long-context inference. While existing KV cache compression methods primarily alleviate the former, they often sacrifice generation quality and leave the latter largely unaddressed. This paper introduces DASH-KV, an innovative acceleration framework that reformulates attention as approximate nearest neighbor search via asymmetric deep hashing. Under this paradigm, we design an asymmetric encoding architecture that differentially maps queries and keys to account for their distinctions in precision and reuse characteristics. To balance efficiency and accuracy, we further introduce a dynamic mixed-precision mechanism that adaptively retains full-precision computation for critical tokens. Experiments on six LongBench tasks across three backbones show that DASH-KV achieves performance comparable to full attention. At a 128K context length, DASH-KV achieves a 2.732x speedup over pre-expanded SDPA for the complete single-layer decoding attention path on Qwen2-7B-Instruct.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient Inference, KV Cache Compression, Large Language Models, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gnTiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper present DASH-KV, using approximate retrieval in a learned local sensitive hash (LSH) to reduce KV Cache IO during inference. It use different hash encoders for query and key cache, hamming-distance-based relevance estimation with calibration cross heads and layers. It split attention scheme into 3 level, exact computing for highly relevant tokens, hash-based scoring with learned correction to moderately relevant tokens, and ignoring the rest tokens to reduce KV Cache IO. The hash and residual correction modules are trained via attention mass distribution distillation while keeping the LLM backbone frozen. Evaluation on LongBench tasks and 3 LLM backbones shows accuracy close to original full attention while reduce single-layer attention latency for long contexts.

Critical strengths
  1. DASH-KV reduce KV Cache IO with only slightly accuracy loss across different LLM backbone and tasks.
  2. S1The implementation with packed XOR and POPCNT matching, fused routing and sparse attention kernels is solid in engineering.
  3. The paper provides extensive ablations and layer-wise analyses, including asymmetric and symmetric Q/K encoder for hashing, hash-code length, encoder depth and correction...
Critical weaknesses
  1. Limited novelty and lack literature survey and comparison to closely related retrieval based methods.

Beyond KV Cache eviction methods like H2O and StreamingLLM, there are already a lot of retrieval based methods for long context inference using ANNS and LSH. For example, RetrievalAttention [1] build ANN indices over KV vectors, MagicPIG [2] use LSH-based sampling, HashAttention [3] separate Q/K mappings into hamming space and HATA [4] applies similar learned binary hashing. All these works are published 1-2 years ago, and already find the Out of Distribution (OOD) problem for attention QK retrieval with more details and use similar "asymmetric" methods to solve the problem. But this paper neither discusses its differences to these related works nor include these more relevant retrieval methods as baselines. This is particularly concerning because Section 2.2 claim that hashing and transformer attention "has not been extensively investigated", "first systematic integration of hashing techniques for attention optimization". Given the substantial prior work, these novelty claims appear overstated, and the actual methodological contribution of DASH-KV remains unclear.

  1. Insufficient study and ablation for the proposed design choices.

Several key design choice are insufficiently motivated or isolated.

  • First, like W1 talks, prior works already identified the gap between classical ANNS to Q/K retrieval, or the OOD issue. Since Q and V cache are transformed by different linear projection, they have different distribution. So previous works [1-4] deploys different methods including "asymmetric" methods to solve the problem. But this work only shows that "asymmetric > symmetric" without further study about whether the gain coms from Q/K OOD, larger nolinear query encoder or simply more learned capacity.
  • Second, this work always keeps the first 4 token and most recent 64 token in full precision. Similar recent-window and first "sink-token" strategies have already appeared in StreamingLLM[6], KiVi[5] and Keyformer[7]. And some recent works and models try to make "sink-free" like GatedAttention[11] in Qwen3.5 family. Yet this work lacks abaltions on the recent window and sink-token accuracy contribution to the accuracy and the difference on "sink-free" GatedAttention models.
  • Finally, the necessity for training a residual corrector is unclear. Methods like CaliDrop[8] and DapQ[9] uses historical queries and position-aware pseudo to achieve training-free correction. This work does not discuss why the learned correction is needed or can it outperform training-free methods.
  1. Insufficient serving-level evaluation of the claimed acceleration.

In efficiency evaluation part, latency is mainly compared with full attention, while even those baselines used in accuracy comparison are omit in the efficiency comparison, not to mention the more relevant hash and retrieval methods in W1 [1-4]. What's worse, the efficiency comparison only act on single attention layer in decoding. The retrieval overheads in the real generation like the hash initialization overhead after prefill, the new generated token hash insert, the recent-window maintenance may be omitted in such evaluation. And the evaluation is also primarily at batch size = 1. The end-to-end benefit like TPOT and TTFT improvement under different context length, batch size also absence.

  1. The overall deployment benefit remains unclear beyond local attention acceleration.

DASH-KV retains the full-precision KV cache and additionally stores hash codes and auxiliary parameters. Table 2A reports 1,722.66 MiB for FullKV versus 1,740.30 MiB for DASH-KV, so this work do not reduce memory cost. In contrast, retrieval methods such as MagicPIG [2] can offload KV-related retrieval and attention computation to CPU memory, reducing GPU memory pressure. Together with the missing end-to-end results in W3, the overall speed--memory--quality benefit of DASH-KV is therefore unclear.

The generalization of the learned hash/residual modules is also insufficiently validated. They are trained on data from NarrativeQA, HotpotQA, Qasper, MultiNews, GovReport, and TriviaQA, which are the sources of LongBench evaluation's 6 tasks. Although the appendix clarifies that training uses the original datasets' training splits rather than LongBench test split, this mainly demonstrates same-task generalization rather than cross-task transfer. A stronger evaluation should freeze trained DASH-KV module and test it without adaptation on unseen task families, e.g., math reasoning, code generation, or long-context agent workloads, or explicitly characterize whether workload-specific retraining is required.


[1] Di Liu, Meng Chen, Baotong Lu, et al.
"RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval."
arXiv:2409.10516, 2024.

[2] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, et al.
"MagicPIG: LSH Sampling for Efficient LLM Generation."
International Conference on Learning Representations (ICLR), 2025.

[3] Aditya Desai, Shuo Yang, Alejandro Cuadron, Matei Zaharia,
Joseph E. Gonzalez, and Ion Stoica.
"HashAttention: Semantic Sparsity for Faster Inference."
International Conference on Machine Learning (ICML), 2025.

[4] Ping Gong, Jiawei Yi, Shengnan Wang, et al.
"HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention
for Scalable Large Model Inference."
Findings of the Association for Computational Linguistics: ACL, 2025.

[5] Zirui Liu, Jiayi Yuan, Hongye Jin, et al.
"KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache."
International Conference on Machine Learning (ICML), 2024.

[6] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
"Efficient Streaming Language Models with Attention Sinks."
International Conference on Learning Representations (ICLR), 2024.

[7] Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, et al.
"Keyformer: KV Cache Reduction through Key Tokens Selection
for Efficient Generative Inference."
Proceedings of Machine Learning and Systems (MLSys), 2024.

[8] Yi Su, Quantong Qiu, Yuechi Zhou, et al.
"CaliDrop: KV Cache Compression with Calibration."
arXiv:2507.19906, 2025.

[9] Zhenxu Tian, Yi Su, Juntao Li, and Min Zhang.
"Where Matters More Than What: Decoding-aligned KV Cache Compression
via Position-aware Pseudo Queries."
arXiv:2603.11564, 2026.

[10] Yushi Bai, Xin Lv, Jiajie Zhang, et al.
"LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding."
Proceedings of the 62nd Annual Meeting of the Association for Computational
Linguistics (ACL), 2024.

[11] Qiu Z, Wang Z, Zheng B, et al.
Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free[J].
Advances in Neural Information Processing Systems, 2026, 38: 100092-100118.

Questions for authors

In Table 1, the results on MultiNews (MN) and GovReport (GR) are somewhat counter-intuitive. Across all 3 backbones, eviction-based baselines such as StreamingLLM, H2O, and SnapKV consistently achieve substantially higher scores than Full Attention on these two tasks, but DASH-KV remains close to Full Attention. Could the authors explain this behavior? Is this improvement caused by the evaluation metric or differences in generated output length, or does aggressive KV eviction/sparsification genuinely improve performance on these summarization tasks? Additional analysis of generation length, decoding behavior, and the corresponding evaluation metrics would help clarify whether this is a real modeling effect or an evaluation artifact.

Reviewer_qd3oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes an attention acceleration method based on asymmetric learned hashing. It encodes queries with an MLP and keys with a linear projection, uses hash-based relevance estimates to select computation, and combines exact attention scores with hash-based approximations and residual correction. The backbone remains frozen, and full-precision keys and values remain cached. The paper evaluates quality on long-context benchmarks and reports single-layer decoding attention speedups.

Critical strengths

The paper implements packed binary-code operations and fused kernels, and measures the complete single-layer attention path, including encoding and routing overhead. This provides more concrete evidence of local computational savings than theoretical operation counts alone. However, this evidence does not establish a useful end-to-end quality–latency trade-off.

Critical weaknesses
  1. The near-lossless quality results are qualified by limited and selectively chosen layer replacement. In particular, the main Qwen2 result replaces only one layer. A systematic quality–latency evaluation across increasing deployment depth is needed to establish the method’s practical operating range.

  2. The practical advantage over query-aware sparse attention methods such as Quest remains insufficiently established. Both approaches retain full K/V and reduce query-time access through candidate selection. DASH-KV introduces learned hashing, residual correction, and cross-head/cross-layer calibration, but the reported single-layer speedup does not establish an end-to-end benefit under the same deployment configuration used for quality evaluation. Its compatibility with tensor-parallel attention is also unclear: cross-head consensus may require communication across head shards, while restricting consensus to local heads would change the algorithm. An end-to-end comparison with Quest at comparable quality, including tensor-parallel execution and realistic batching with paged KV caches, would help determine whether the additional complexity yields practical benefits.

  3. The small improvement over Quest (38.99 vs. 38.83) is difficult to interpret without uncertainty estimates or repeated runs and does not establish a clear quality advantage.

  4. The residual-correction mechanism is not specified consistently. The main formulation conditions the residual on binary codes, whereas Appendix F.1 attributes magnitude recovery to continuous representations. Clarifying the actual inputs is important for assessing both the mechanism’s expressiveness and its execution cost.

Questions for authors

Please see the weakness part.

Reviewer_6MpLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DASH-KV, a training-based, plug-in sparse attention framework that reformulates Q–K similarity matching as approximate nearest-neighbor retrieval in a binary Hamming space. Queries are encoded by a lightweight three-layer MLP, while Keys are encoded by a linear projection; both are binarized into 128-bit codes, and relevance is scored by Hamming distance with cross-head consensus and cross-layer momentum calibration. A dynamic mixed-precision routing scheme then retains highly relevant Keys in full precision, applies hash-plus-residual approximation to moderately relevant Keys, and masks weakly relevant Keys for the current query, while the backbone remains frozen and each replaced layer is trained separately via KL distillation. Experiments on LongBench with Qwen2-7B, Llama-3.1-8B, and Qwen2.5-14B report performance close to Full Attention, though the main Qwen2 result replaces only Layer 18 and the more realistic eight-layer configuration is reported separately. Speed measurements are limited to a single decoding layer at 128K on an RTX 4090, reporting 2.732× over pre-expanded SDPA and 8.021× over stock GQA-expanded SDPA.

Critical strengths
  • The paper makes a conceptually novel attempt to reformulate Q–K similarity matching as binary hash retrieval and to integrate asymmetric deep hashing into the attention mechanism.
Critical weaknesses
  • Hamming distance and dot-product attention have fundamentally different dynamics. Hamming distance discards magnitude information and forces an isotropic binary space, whereas softmax attention relies on continuous dot-product logits. The paper introduces cross-head voting, cross-layer momentum, residual correction, asymmetric temperature scaling, and layer-wise training to compensate for this mismatch, but provides no theoretical justification for why these patches should generalize. The reported Recall@100 of only 86.06% further confirms that the coarse retrieval is lossy.

  • The main experimental comparisons are not conducted under equal intervention scopes. In Table 1, DASH-KV replaces only Layer 18 on Qwen2-7B, while StreamingLLM, H2O, and SnapKV are applied across the full model. Table 5 is also selectively reported: it presents only three single-layer configurations (Layers 1, 18, and 27) plus one eight-layer set, rather than the complete per-layer replacement curve, and Layer 18 happens to be one of the most favorable points.

  • The paper omits the most directly comparable sparse-decoding baselines. It does not compare against SeerAttention-R, a training-based, decode-focused sparse attention method that also freezes the backbone and trains only a lightweight gate. It also omits DSA and MagicPig. Quest appears only once in Table 2A.

  • The speed evaluation compares only against SDPA variants and internal ablations, with comparison against other sparse attention methods. No end-to-end throughput, batch-size scaling, or production-serving latency is reported.

  • The evaluation is limited to 7B–14B dense models under a single-GPU, batch-size-1 setting. All experiments use Qwen2-7B, Llama-3.1-8B, and Qwen2.5-14B on an RTX 4090 with FP16 and batch size 1. There is no validation on larger models, reasoning-heavy models, multi-GPU serving, or production workloads.

Questions for authors
  • How does DASH-KV behave on models with hybrid Linear attention / SWA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.