acceptodds
Under review as a conference paper at ICLR 2027

Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference

10 views

Abstract

The memory usage and decoding latency of LLM inference grow rapidly with context length. To reduce these costs, key-value (KV) cache compression methods selectively retain cached states based on token importance or differences in attention patterns across heads. However, we discover that retrieval capability varies substantially with relative distance, even within the same attention head. To exploit this structure, we introduce Distance-KV, which learns a static KV retention pattern over the joint space of layers, attention heads, and relative distances. The pattern is learned offline with the language model frozen and reused across inputs to prune and compact the KV cache without online importance scoring. Across three backbone models and four long-context benchmarks, Distance-KV consistently achieves the best overall performance among competing KV cache compression methods, exceeding the strongest compression baseline by up to 9.3 points on RULER at 128K. On Llama-3.1-8B-Instruct at 128K, Distance-KV reduces KV cache memory by 65.4% and achieves a decoding speedup relative to Dense. Together, these results identify relative distance as an important structural dimension for understanding how LLMs retrieve information over long contexts and for designing more efficient inference methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, long-context inference, KV cache compression, efficient inference, attention mechanisms

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mBFDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Distance-KV, a query-independent KV cache compression method motivated by the observation that an attention head’s retrieval behavior varies with relative distance. It learns a static binary retention pattern over layers, query heads, and distance bins using synthetic retrieval examples while freezing the language model. Optimization combines Hard Concrete gates with straight-through global Top-K refinement. After dense context prefill, the learned pattern is used to prune and compact the cache; sink and recent tokens are preserved, and newly appended tokens are retained in full. For GQA models, logical query-head decisions are combined by union within each shared KV head.

The paper evaluates three backbones using NIAH, LongBench, RULER, and SCBench. It reports the highest aggregate scores among the evaluated compression baselines and, for Llama-3.1-8B at 128K, a 65.4% reduction in context KV memory and a 1.66× decoding speedup over Dense. Structural ablations support the usefulness of jointly modeling heads and distance.

Critical strengths

Critical Strengths

  1. The paper identifies useful structure within attention heads and provides supporting ablations. Figure 1 illustrates distance-dependent retrieval behavior. At matched physical KV retention, shuffling distance assignments reduces the RULER task average from 63.7 to 54.7, while head-only retention achieves 44.5 (Table 3). These results support the importance of the learned correspondence between heads and distance intervals.

  2. A reusable static policy achieves promising quality and cache savings. One pattern per backbone is reused across benchmarks, including shared-context multi-turn evaluation. On Llama-3.1, LongBench performance remains close to Dense (47.30 versus 47.53), alongside substantial context-cache savings at 128K. Avoiding query-dependent rescoring makes the approach practically attractive for reusable context caches.

Critical weaknesses

Critical Weaknesses

  1. The deployed GQA attention policy differs from the policy optimized during training, and the effect is not evaluated. Appendix A.2 and the released training implementation apply independent distance masks to query heads. At inference, the implementation unions these masks within each shared KV head and passes the resulting cache to FlashAttention without reapplying the individual query-head masks. A query head therefore attends to positions excluded by its own training-time mask, changing both the softmax normalization and attention output. This is consistent with the stated union operation, but it means the deployed policy is not the exact policy optimized by the retrieval objective. Since both modern backbones use GQA, this discrepancy is central to interpreting the method. In addition, the budget constrains logical query-head entries rather than physical KV storage.

  2. The released artifacts do not fully support verification of the reported results. Applying the Figure 6(b) definition—heads retaining some but not all learnable distance bins—to the released 20%-budget patterns gives 60.84% for Llama-2, 48.44% for Llama-3.1, and 39.16% for Qwen2.5, compared with the reported 62.5%, 70.2%, and 77.9%. The release therefore appears to contain different patterns or use a different statistic, without sufficient provenance to resolve the discrepancy. The package also imports a missing internal module (training.audits.gradient_audit), and the downloaded release lacks benchmark, baseline, and efficiency-measurement scripts. This prevents checking important experimental controls, including the unusually large improvement over Qwen Dense at 128K. Main-method comparisons also lack performance–physical-memory curves across budgets, while efficiency reporting omits pruning/packing latency and end-to-end cost despite requiring dense prefill.

Questions for authors

Questions For Authors

The following questions concern the current method and reported results.

  1. GQA training–deployment consistency: The released inference implementation uses the GQA-group union without enforcing individual query-head masks. How much does this change retrieval loss and benchmark performance relative to the training-time binary policy? Please provide a controlled comparison and explain why optimizing the independent masks is appropriate for the deployed union policy.

  2. Released-pattern provenance: What explains the discrepancy between Figure 6(b) and the partially retained query-head fractions calculated from the released budget20.pt files? Please provide the exact statistic, figure-generation script, and checkpoint or pattern identifiers used for both the figure and the main benchmark results.

  3. Physical-resource comparison: Can you provide comparisons with the strongest baselines at matched physical KV memory, preferably across several budgets? Please also explain how physical retention was matched in the structural ablations and how the restricted variants were optimized.

  4. Evaluation and efficiency controls: Please provide the benchmark and timing scripts, including exact checkpoint, RoPE/YaRN, attention-kernel, and position-index settings for every method. What explains Qwen’s RULER group average of 23.47 for Dense versus 68.38 for Distance-KV at 128K? Please report pruning/packing latency and end-to-end inference time, and clarify whether peak-memory measurements include dense prefill.

  5. Reproducibility and checkpoint selection: Please restore the missing internal module and document the complete training-to-export procedure. Which training checkpoints produced the released patterns and reported results? In particular, what explains the difference between the Distance-KV task averages in Table 3 (63.7) and Table 8 (71.64), and how were the ablation checkpoints selected?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mBFD (Part 1/2)

We thank the reviewer for the careful evaluation and constructive comments. We provide a new anonymous repository containing the Distance-KV training, inference, and structural ablation code, benchmark and efficiency evaluation scripts for Distance-KV and all evaluated baselines, and the required patterns and configurations: https://anonymous.4open.science/r/Distance-KV-Pattern-New-5AF1/. We also provide additional end-to-end efficiency measurements in the responses below.

W1 / Q1: GQA training–inference consistency

We thank the reviewer for pointing this out. After checking the training and inference code, we confirm that the original description in Appendix A.2—query heads independently modulate attention and are combined only at inference—was incorrect. During training, we already combine the retention decisions of query heads sharing the same KV head before computing attention; inference uses the same within-group union rule. Thus, the training–inference inconsistency implied by our description does not occur in the actual implementation. We have corrected this description in Appendix A.2 of the revised manuscript. The related code and training records are available in the new anonymous repository.

W2: Completeness of released artifacts and verification of results

We have included the missing training.audits.gradient_audit module, along with benchmark and efficiency evaluation code for all evaluated methods, in the new anonymous repository. The remaining concerns are addressed individually in Q2–Q5 below.

Q2: Released-pattern provenance and the discrepancy with Figure 6(b)

The budget20.pt files in the original anonymous repository were exported at step44, whereas Figure 6 uses patterns exported from each model’s final training checkpoint, resulting in different statistics. For each model, the main benchmark evaluations, efficiency measurements, and Figure 6 all use the same final pattern: step60 for Llama-2, step132 for Llama-3.1, and step80 for Qwen2.5. Step44 is used only as the reference for the structural ablations. The new anonymous repository includes these final patterns. Figure 6(b) reports the fraction of query heads that retain some but not all learnable distance bins, computed by tools/figures/plot_pattern_analysis_v10.py. The corresponding percentages are 62.5%, 70.2%, and 77.9%, consistent with the paper.

Q3: Physical-resource comparison

The paper reports physical KV cache sizes measured during execution. For Distance-KV, these measurements reflect the actual cache after the within-group GQA union and physical pruning/packing, including the additional KV positions retained through the union. We do not treat the 20% logical query-head retention ratio as the physical KV retention ratio. The following table summarizes accuracy and physical KV usage for the main baselines:

ModelMethodKV @128K (GiB)LongBench Avg.RULER Avg.SCBench Avg.
Llama3.1DuoAttention7.8346.9262.9740.0
Llama3.1MoA27.3243.8648.5227.4
Llama3.1Distance-KV5.4147.3068.6641.0
Qwen2.5DuoAttention3.4343.2859.0828.6
Qwen2.5MoA24.0842.0453.6125.7
Qwen2.5Distance-KV3.2844.0068.3834.2

On Llama-3.1, Distance-KV achieves the highest scores on all three benchmarks while using 30.9% less physical KV memory than DuoAttention, the smallest baseline in the table. On Qwen2.5, it also achieves the highest scores while using approximately 4% less physical KV memory than DuoAttention.

The structural ablations use closely matched physical KV budgets after the GQA-group union. Using the same 128k-token decode-start setting as the efficiency evaluation, the calculated physical KV sizes are:

MethodCalculated physical KV (GiB)Retention relative to Dense
Distance-KV (ablation reference)5.442934.83%
Head-only5.414534.65%
Distance-only5.447434.86%
Distance-shuffled5.442634.83%

Head-only uses the closest achievable budget under its whole-head retention constraint; the other controls closely match the reference budget.

Head-only and Distance-only independently train their gate parameters from retrieval-based initialization under their respective structural constraints, using the same training data and optimization procedure as Distance-KV. Distance-shuffled permutes distance assignments within each KV group and requires no retraining.

AuthorsOpenReview ↗

Response to Reviewer mBFD (Part 2/2)

Q4: Evaluation settings and end-to-end efficiency

We follow the official RULER evaluation protocol and each method’s original implementation with the necessary model adaptations. Dense and Distance-KV use the same Qwen2.5-7B-Instruct backbone and YaRN settings (factor 4, original context length 32768). The evaluation and timing scripts, together with the checkpoint, RoPE/YaRN, attention-kernel, and position-index configurations for each method, are provided in the new anonymous repository.

Qwen2.5’s performance degradation at 128K is not unique to our experiments. Table 16 of the official Qwen technical report shows that without DCA and YaRN, Qwen2.5-7B-Instruct’s RULER score decreases from 89.4 at 32K to 31.4 at 128K [1]. RULER identifies failure to ignore distractors as an important failure mode [2], while Differential Transformer shows that suppressing attention noise can improve long-context retrieval [3]. Distance-KV’s learned distance selection may similarly reduce interference from irrelevant positions.

We further measured pruning/packing latency, end-to-end inference time, and full-pipeline peak GPU memory on an A800, following the original efficiency experiment’s 128K setting and reporting medians over three measurements. After full-context prefill, Distance-KV uses the precomputed static pattern to identify retained KV positions and gathers the selected KV entries into contiguous buffers, without online importance scoring.

ModelDense prefill (s)Distance-KV prefill (s)Distance-KV pruning/packing (s)
Llama3.133.59833.5440.046
Qwen2.527.50927.4610.032

Prefill latency is close to Dense, and pruning/packing costs less than 0.15% of prefill time on both models.

The following table reports end-to-end time and full-pipeline peak allocated GPU memory for the 512-step setting. End-to-end time includes prefill, each method’s cache preparation/compression, query processing, 16 warmup steps, and 512 measured decoding steps.

MethodLlama3.1 end-to-end (s)Llama3.1 full-pipeline peak (GiB)Qwen2.5 end-to-end (s)Qwen2.5 full-pipeline peak (GiB)
Dense59.25144.78116.59938.08
StreamingLLM50.44844.7841.86338.08
DuoAttention56.57836.9442.85638.09
MoA97.99959.4089.84557.61
KVzip-CI57.25932.3442.68523.14
Distance-KV50.92844.7840.88438.08

Regarding peak memory, the original paper reports peak memory during decoding; its measurement interval excludes dense prefill. The full-pipeline peaks above include dense prefill and account for model weights, KV caches, and temporary tensors. Distance-KV’s full-pipeline peak is close to Dense because it completes dense prefill before pruning; its memory savings apply to the retained KV cache and the decoding stage. The revised manuscript reports end-to-end latency and full-pipeline peak memory in Table 10 (Appendix D.1).

Q5: Reproducibility and checkpoint selection

As explained in Q2, the main experiments use each model’s final pattern.

To control computational cost and compare different structures at the same training step, we consistently use step44 for the structural ablations. The shuffled controls are also generated from the reference pattern at this step. Thus, Table 3 reports the step44 result (63.7), whereas Table 8 reports the final step132 result (71.64). The revised manuscript lists the final checkpoints in Appendix A.3 and documents the common ablation step in Appendix E.

The complete training-to-pattern-export procedure, related code, configurations, and corresponding patterns are provided in the new anonymous repository.

[1] Qwen Team. Qwen2.5 Technical Report. arXiv:2412.15115, 2024.

[2] Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? COLM, 2024.

[3] Ye et al. Differential Transformer. ICLR, 2025.

Thank you for the detailed response. My main remaining question is about the Qwen2.5 Dense result in Table 9. Could you provide the exact command and output for the 128K run, and, if available, a no-pruning run through the Distance-KV execution path with the same YaRN and position settings? The cited 31.4 result disables both DCA and YaRN, so it does not directly explain the 24.68 observed in your YaRN-enabled setup.

Reviewer_3PT8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Distance-KV, a KV cache compression method that exploits the relationship between attention heads and relative token distances.

The authors observe that the retrieval capability of an attention head can vary significantly across relative distances. Based on this observation, Distance-KV learns a fine-grained retention pattern over layers, attention heads, and relative distances. The pattern is optimized offline using synthetic retrieval tasks while keeping the language model frozen, and can be reused across different inputs without online importance scoring.

Experiments across three backbone models and four long-context benchmarks show that Distance-KV achieves competitive accuracy while substantially reducing KV cache memory usage and decoding latency.

Critical strengths
  1. Meaningful motivation. The paper identifies an interesting limitation of conventional head-level KV cache compression: the retrieval capability of the same attention head may vary significantly across relative distances. Extending KV cache allocation to the joint space of layers, heads, and distances is a reasonable and well-motivated approach.

  2. Efficient inference with reusable static patterns. Although learning the retention pattern requires offline optimization, the resulting pattern can be directly reused during inference without query-dependent importance scoring or additional optimization. This makes the online inference procedure relatively simple and potentially practical for deployment.

  3. Promising experimental results. Distance-KV achieves strong performance across multiple models and long-context benchmarks, while reducing KV cache memory usage and decoding latency. The structural ablation studies also provide convincing evidence that the correspondence between attention heads and relative distances plays an important role in preserving retrieval performance.

Critical weaknesses
  1. Uncertain generalization of static retention patterns. Distance-KV relies on a model-specific static pattern learned from synthetic retrieval tasks. Although the authors evaluate the learned patterns across several benchmarks, it remains unclear whether the same patterns would generalize equally well to more diverse data distributions, complex reasoning tasks, or substantially different application scenarios. Since the patterns are fixed during inference, their ability to adapt to changes in information requirements is inherently limited.

  2. Potentially unfair compression-budget comparisons. The main experiments use a 20% logical query-head retention ratio for Distance-KV. However, due to shared KV heads in GQA, the actual physical KV retention ratio can be considerably higher. Meanwhile, the competing baselines use different compression configurations. Consequently, the reported accuracy improvements may not correspond to comparisons under equivalent physical KV cache budgets. A more controlled accuracy–compression trade-off evaluation would strengthen the empirical conclusions.

  3. Limited end-to-end efficiency analysis. The paper reports meaningful reductions in KV cache memory and decoding latency. However, Distance-KV requires a full-context prefill followed by physical KV pruning and packing. The additional preprocessing cost is not fully characterized in an end-to-end inference evaluation. It would be helpful to understand how these costs affect overall latency under different generation lengths and workload characteristics.

Questions for authors
  1. How robust are the learned static retention patterns when applied to substantially different data distributions or tasks beyond the synthetic retrieval setting used for optimization? Have the authors investigated whether different calibration datasets produce similar patterns?

  2. Could the authors provide accuracy–compression trade-off curves under matched physical KV cache budgets? In particular, it would be valuable to compare Distance-KV with existing methods at equivalent memory usage, especially for GQA models.

  3. Could the authors report end-to-end inference latency that includes full-context prefill, KV cache pruning and packing, and decoding? This would help clarify the practical benefits of Distance-KV for different input and output lengths.

  4. On Qwen2.5-7B at 128K, Distance-KV substantially outperforms even the dense attention baseline on RULER. Could the authors provide further analysis of this result? Understanding why compression improves retrieval accuracy so significantly would strengthen the paper's conclusions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3PT8 (Part 1/2)

We thank the reviewer for recognizing the motivation behind distance-aware KV retention and the practical benefits of reusable static patterns. Below, we clarify the generalization rationale, physical KV comparisons, and Qwen2.5’s performance at 128K, and provide additional pruning/packing and end-to-end latency measurements.

W1 / Q1: Generalization of static retention patterns

Our design is based on the premise that retrieval is a fundamental capability for long-context processing, shared across data distributions and tasks. Although question answering, summarization, and code tasks have different objectives, they all require accessing relevant information within long contexts. Distance-KV uses synthetic retrieval tasks to learn attention heads’ retrieval capabilities at different relative distances and preserves the pathways supporting this shared capability. This provides the basis for reusing a static pattern across tasks. The static pattern fixes the KV retention policy, while attention weights over retained positions still vary dynamically with the input and query, allowing the model to select and use information according to current needs.

The cross-benchmark evaluations in the paper provide direct evidence supporting this rationale: each model learns a single pattern, which is applied directly to NIAH, LongBench, RULER, and SCBench without benchmark-specific retraining or adjustment. These evaluations cover synthetic retrieval and real-data tasks involving question answering, summarization, and code, demonstrating transfer beyond the calibration tasks.

Regarding different calibration datasets, we have not conducted a controlled comparison. Based on the rationale above, we believe retrieval datasets that cover different relative distances and provide effective retrieval training signals can serve as calibration data. We also expect them to capture similar head–distance capability structures.

W2 / Q2: Physical KV budgets and comparison fairness

The paper reports physical KV cache sizes measured during execution. For Distance-KV, these measurements reflect the actual cache after the within-group GQA union and physical pruning/packing, including the additional KV positions retained through the union. We do not treat the 20% logical query-head retention ratio as the physical KV retention ratio. The following table presents comparisons on Llama3.1 and Qwen2.5:

ModelMethodKV @128K (GiB)LongBench Avg.RULER Avg.SCBench Avg.
Llama3.1DuoAttention7.8346.9262.9740.0
Llama3.1MoA27.3243.8648.5227.4
Llama3.1Distance-KV5.4147.3068.6641.0
Qwen2.5DuoAttention3.4343.2859.0828.6
Qwen2.5MoA24.0842.0453.6125.7
Qwen2.5Distance-KV3.2844.0068.3834.2

On Llama3.1, compared with DuoAttention—the strongest-performing baseline and the one with the smallest KV footprint in the table—Distance-KV uses 30.9% less physical KV memory while achieving higher average scores on all three benchmarks.

On Qwen2.5, the two methods have comparable physical KV footprints, with Distance-KV still using approximately 4% less memory. Under this comparable budget, Distance-KV improves the average LongBench, RULER, and SCBench scores by 0.72, 9.30, and 5.6 points, respectively, showing larger performance gains.

These comparisons therefore show that Distance-KV’s performance advantage does not arise from a larger physical KV budget; it is achieved with lower or comparable actual cache usage.

AuthorsOpenReview ↗

Response to Reviewer 3PT8 (Part 2/2)

W3 / Q3: End-to-end inference efficiency

We further measured pruning/packing latency and end-to-end inference time on an A800, following the original efficiency workload and reporting medians over three measurements. After full-context prefill, Distance-KV uses the precomputed static pattern to identify retained KV positions and gathers the selected KV entries into contiguous buffers, without online importance scoring.

At the 128K setting, the measured prefill and pruning/packing times are:

ModelDense prefill (s)Distance-KV prefill (s)Distance-KV pruning/packing (s)
Llama3.133.59833.5440.046
Qwen2.527.50927.4610.032

Prefill latency is close to Dense, and pruning/packing costs less than 0.15% of prefill time on both models at this setting.

To examine different input and output lengths, the following table reports end-to-end times at the original efficiency workload’s 32K and 128K settings, with 128 and 512 measured decoding steps. All times are in seconds and include full-context prefill, each method’s cache preparation/compression, query processing, 16 warmup decoding steps, and measured decoding.

ModelMethod32K / 128 steps32K / 512 steps128K / 128 steps128K / 512 steps
Llama3.1Dense7.85718.27640.69859.251
Llama3.1StreamingLLM8.49420.72138.20150.448
Llama3.1DuoAttention8.59421.08639.96056.578
Llama3.1MoA16.46833.82157.75897.999
Llama3.1KVzip-CI7.95718.56139.69457.259
Llama3.1Distance-KV7.26515.98938.43150.928
Qwen2.5Dense10.73430.17751.937116.599
Qwen2.5StreamingLLM7.40717.96531.35841.863
Qwen2.5DuoAttention7.03616.78731.43142.856
Qwen2.5MoA14.87130.71752.08789.845
Qwen2.5KVzip-CI6.74915.44431.06042.685
Qwen2.5Distance-KV6.48714.46231.24440.884

Distance-KV achieves lower end-to-end latency than Dense across both input lengths and both decoding lengths. Its end-to-end speedup over Dense increases as decoding extends from 128 to 512 steps, showing that the pruning/packing overhead does not offset the gains from faster decoding.

Appendix D.1 of the revised manuscript reports pruning/packing latency and the single-request end-to-end results (Table 10). The scripts and configurations for these supplemental efficiency experiments are available in our new anonymous repository: https://anonymous.4open.science/r/Distance-KV-Pattern-New-5AF1/

Q4: Why Distance-KV outperforms Dense on Qwen2.5 at 128K

Retaining all KV entries does not necessarily lead to better retrieval performance. In long contexts, numerous irrelevant positions also participate in attention computation and may divert attention from relevant information. Distance-KV learns attention heads’ retrieval capabilities at different relative distances and selects retained positions accordingly. A plausible explanation is that this selection preserves effective retrieval pathways while reducing attention interference from irrelevant positions, allowing cache compression to improve retrieval performance.

This explanation is consistent with prior research: RULER identifies failure to ignore distractors as an important failure mode [2], while Differential Transformer shows that suppressing attention noise from irrelevant context can improve long-context retrieval [3]. The official Qwen technical report also documents long-context degradation: without DCA and YaRN, Qwen2.5-7B-Instruct’s RULER score decreases from 89.4 at 32K to 31.4 at 128K [1].

[1] Qwen Team. Qwen2.5 Technical Report. arXiv:2412.15115, 2024.

[2] Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models?. COLM, 2024.

[3] Ye et al. Differential Transformer. ICLR, 2025.

AuthorsOpenReview ↗

Response to Reviewer 3PT8 (Follow-up on Q4)

We would like to provide a correction concerning the Qwen2.5 Dense results at 128K discussed in Q4. Following Reviewer mBFD’s further questions about the YaRN settings, we inspected the configuration actually active after model loading. We found that the original Dense loading path did not forward the modified configuration to the model, so the intended YaRN settings were not activated. We have fixed this issue and reevaluated all six context lengths and all 13 tasks, totaling 39,000 samples, using the same frozen inputs, generation budgets, and scoring protocol.

Averaged over the eight task groups used in Figure 4, the Dense score at 128K is corrected from 23.47 to 66.67. The previously observed large performance gap was substantially affected by this configuration issue, and the comparison should therefore be based on the corrected results.

We have updated Figure 4(b) and Table 9 of the manuscript. The corrected code, launch commands, configuration-check evidence, and complete scoring results are available in the anonymous repository. Other benchmarks, other methods, and efficiency measurements are unaffected by this configuration-forwarding issue.

Reviewer_PxvnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Distance-KV, a static KV cache compression method that exploits the distance-dependent retrieval behavior of attention heads. The authors observe that the retrieval capability of an individual head can vary substantially across relative distances, suggesting that existing head-level retention policies may be insufficiently fine-grained. Motivated by this observation, Distance-KV learns a binary retention pattern over layers, query heads, and relative-distance bins. The pattern is optimized offline using synthetic retrieval examples with the backbone model frozen, and is reused across inputs without online importance scoring. Experiments on three backbone models and four long-context benchmarks demonstrate promising results. In particular, Distance-KV reduces KV cache memory by 65.4% and achieves a 1.66× decoding speedup over Dense on Llama-3.1-8B-Instruct at 128K context length.

Critical strengths
  1. The paper provides a systematic analysis showing that retrieval capability can vary significantly across relative distances, even within the same attention head. This observation offers a meaningful refinement of existing head-level characterizations and motivates distance-conditioned KV retention. The resulting design is intuitive and allows more flexible retention patterns than conventional head-level or sliding-window policies.
  2. Distance-KV achieves strong performance across several long-context benchmarks while substantially reducing physical KV cache usage and decoding latency. The structural ablations, particularly the head-only and distance-shuffled variants, provide useful evidence that the learned correspondence between attention heads and distance ranges contributes to the observed improvements.
Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the proposed design's contribution.

The proposed method appears to be an incremental extension of existing head-wise KV cache compression and sparse attention methods. In particular, DuoAttention has already introduced an optimization-based framework for learning static head-level retention policies using synthetic retrieval data, while MoA has explored heterogeneous attention spans across layers and heads. The main distinction of Distance-KV appears to be its finer-grained, potentially non-contiguous distance-bin retention pattern. The optimization techniques, including Hard Concrete gates, sparsity regularization, and straight-through Top-K selection, are also well established.

Although the current ablations demonstrate the benefits of the learned pattern over head-only or shuffled alternatives, they do not sufficiently establish whether such a highly flexible distance-bin representation is necessary. For example, simpler distance-aware alternatives, such as learned per-head sliding windows, two-window retention patterns, or coarser distance partitions, might capture much of the observed benefit. Comparisons with these alternatives under matched physical KV budgets would help clarify the contribution beyond existing head-wise sparse attention approaches.

  1. Insufficient baseline coverage and end-to-end evaluation to substantiate the claimed efficiency advantages.

The experimental evaluation does not sufficiently establish the competitiveness of Distance-KV against existing KV cache compression approaches, particularly in terms of the trade-off between accuracy and end-to-end efficiency.

First, the paper compares only against KVzip-CI rather than the original KVzip. This omission is difficult to justify given the stated evaluation protocol. The original KVzip is context-dependent but query-agnostic: it performs compression without knowing future queries and can reuse the resulting cache across subsequent queries. Therefore, query independence does not require restricting the comparison to context-independent methods. A direct comparison with the original KVzip, including its compression overhead, would be important. Furthermore, more recent approaches to adaptive KV budget allocation and learned token selection, such as Ada-KV (NeurIPS'25) and LKV (EMNLP'26), deserve consideration. Comparisons at matched physical KV retention budgets would also strengthen the evaluation.

Second, the paper excludes query-aware compression methods and primarily reports per-token decoding latency rather than end-to-end inference performance. While avoiding query-dependent scoring is a potential advantage of Distance-KV, the current experiments do not demonstrate whether this advantage translates into better overall latency or throughput while maintaining competitive accuracy.

This issue is particularly important for the shared-context multi-query setting emphasized by the paper. Although SCBench evaluates accuracy under cache reuse, the corresponding cumulative inference costs are not reported. A comparison covering prefill, compression or selection overhead, decoding, and repeated queries would be necessary to establish the practical advantages and limitations of the proposed static strategy.

Questions for authors

Q1. Necessity of fine-grained distance patterns.

Could the authors compare Distance-KV with simpler distance-aware alternatives, such as optimized per-head sliding windows, two-window patterns, or coarser distance partitions, under matched physical KV budgets? In particular, how much of the improvement can be attributed specifically to allowing arbitrary non-contiguous distance intervals?

Q2. Missing baselines and end-to-end efficiency.

Why is the original query-agnostic KVzip excluded from the evaluation? Could the authors provide comparisons with the original KVzip and representative query-aware compression methods, reporting accuracy, physical KV cache usage, and end-to-end latency?

For the shared-context setting, it would be particularly useful to evaluate how the total inference cost changes as the number of queries sharing the same context increases.

Q3. KVzip-CI configuration.

Appendix B states that KVzip-CI uses a "head-level pruning ratio of 0.60", whereas the official KVzip implementation defines ratio as the retained fraction and recommends level head ratio 0.6 for its context-independent configuration.

Does the reported ratio correspond to 60% retention or 60% eviction? Could the authors clarify the exact configuration and resulting physical KV retention, especially given the very low performance of KVzip-CI on several tasks?

Q4. GQA training semantics and query-head-level parameterization.

The released implementation appears to combine query-head gates within each GQA group during training, whereas Appendix A.2 states that these gates independently modulate their corresponding attention distributions and are combined only when mapped to physical KV heads at inference.

Could the authors clarify this discrepancy? Furthermore, since the resulting physical retention pattern operates at the KV-head level, would directly learning KV-head-level distance patterns achieve comparable performance under the same physical cache budget?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Pxvn (Part 1/3)

We thank the reviewer for the detailed feedback on our methodological contribution and evaluation. Below, we clarify the contribution of head–distance modeling and the implementation details, and provide additional comparisons with the original KVzip, together with end-to-end measurements for both single-query and shared-context multi-query workloads.

W1 / Q1: Methodological contribution and fine-grained distance patterns

Our contribution is to identify and exploit distance-dependent retrieval capability within individual attention heads. Section 3.1 and Figure 1 show substantial within-head variation, including intermediate-distance peaks and nonmonotonic profiles, which a single head-level label or attention span cannot fully characterize. Section 4.5 and Figure 6 further show that 62.5%, 70.2%, and 77.9% of query heads in the three models retain only a subset of distance bins. Distance-KV learns retention decisions over this head–distance structure; Hard Concrete, regularization, and Top-K are established tools used to optimize it.

The original baselines already include representative simpler distance-aware strategies. MoA optimizes sliding-window lengths across layers and heads, while DuoAttention combines full-context retrieval heads with fixed sink-plus-recent retention for streaming heads. Across the evaluated NIAH, LongBench, RULER, and SCBench benchmarks, Distance-KV achieves better overall performance than both methods, while using less physical KV memory on Llama3.1 and Qwen2.5. These comparisons support the practical value of finer-grained allocation at lower cache usage.

Table 3 further examines the structural contribution at closely matched physical KV budgets and the same training step:

MethodRULER Avg.
Head-only44.5
Distance-only12.7
Distance-shuffled54.7
Distance-KV63.7

Head-only and Distance-only are trained independently under their respective constraints, removing distance variation and head-specific variation, respectively. Distance-shuffled preserves per-head retained-bin counts and closely matches the physical KV cost but permutes distance assignments, causing a 9.0-point drop. Together, these results support the importance of the learned head–distance correspondence, beyond increased representational flexibility alone.

W2 / Q2: Original KVzip comparison and end-to-end efficiency

The original manuscript uses KVzip-CI to compare retention policies that do not require recomputing importance scores for each context. Although the original KVzip is context-dependent, it does not depend on subsequent query content and supports shared-context cache reuse. We therefore supplement the evaluation with the original KVzip, denoted KVzip-CD below, reporting accuracy, actual physical KV usage, and full end-to-end costs. Our comparisons focus on query-agnostic methods that do not access subsequent query content.

RULER results across context lengths. The following results cover two models, six context lengths, and all 13 tasks. Scores at each length follow the same task-group aggregation as Figure 4 in the paper; “Avg.” is the mean across the six context lengths.

ModelMethod4K8K16K32K64K128KAvg.
Llama3.1Dense94.6594.4689.0780.5579.1769.3084.53
Llama3.1DuoAttention93.7892.6385.0478.8076.1062.9781.55
Llama3.1KVzip-CD94.2093.5290.4381.4576.8566.5483.83
Llama3.1Distance-KV94.6094.1190.4381.2577.4368.6684.41
Qwen2.5Dense93.2792.2592.2587.0569.4123.4776.28
Qwen2.5DuoAttention88.8783.7981.0475.5472.6459.0876.83
Qwen2.5KVzip-CD90.0587.5387.0182.0378.6866.1081.90
Qwen2.5Distance-KV90.9188.8386.7680.8578.1868.3882.32

Distance-KV achieves higher overall RULER scores than the original KVzip on both models while avoiding per-context reconstruction-based scoring.

AuthorsOpenReview ↗

Response to Reviewer Pxvn (Part 2/3)

W2 / Q2 (continued): End-to-end efficiency

Single-request physical KV usage and end-to-end costs. The following efficiency measurements are collected on the same A800 GPU, using the original efficiency experiment’s 128K setting and 512 measured decoding steps, with medians over three measurements. End-to-end time includes full-context prefill, importance scoring where applicable, pruning/packing, query processing, 16 warmup decoding steps, and measured decoding. Appendix D.1 (Table 10) of the revised manuscript reports the single-request results for the methods evaluated in the original submission.

ModelMethodActual physical KV (GiB)End-to-end time (s)
Llama3.1Dense15.6359.251
Llama3.1StreamingLLM0.1450.448
Llama3.1DuoAttention7.8356.578
Llama3.1MoA27.3297.999
Llama3.1KVzip-CI9.3557.259
Llama3.1KVzip-CD4.70125.619
Llama3.1Distance-KV5.4150.928
Qwen2.5Dense6.84116.599
Qwen2.5StreamingLLM0.0641.863
Qwen2.5DuoAttention3.4342.856
Qwen2.5MoA24.0889.845
Qwen2.5KVzip-CI4.0342.685
Qwen2.5KVzip-CD2.0599.452
Qwen2.5Distance-KV3.2840.884

KVzip-CD uses a smaller physical cache, while Distance-KV achieves higher average scores and lower end-to-end latency. KVzip-CD’s context-reconstruction scoring takes 74.604 s and 58.989 s on the two models, respectively. Distance-KV requires no online importance scoring, and its pruning/packing takes only 0.046 s and 0.032 s. Per-context reconstruction-based scoring therefore introduces substantial additional latency for KVzip-CD, which Distance-KV avoids.

Multi-query costs with a shared context. We further evaluate a real SCBench many-shot sample with a context length of approximately 26K tokens, following the original five-query generation procedure, output limits, and EOS stopping rules. All methods use the same sample, with medians over three repetitions. Cumulative timing covers context preparation, query processing, and generation. KVzip-CD performs scoring and compression only once; subsequent queries reuse its compressed cache.

ModelMethodThrough Q1 (s)Through Q2 (s)Through Q3 (s)Through Q4 (s)Through Q5 (s)Total output tokensCumulative end-to-end time/token (s)
Llama3.1Dense3.1993.3573.5393.6373.736190.1966
Llama3.1StreamingLLM3.2173.6033.9634.3194.676500.0935
Llama3.1DuoAttention2.9843.1733.4013.5623.787150.2525
Llama3.1MoA11.75620.69229.78838.65348.122242.0051
Llama3.1KVzip-CI3.2163.3963.5963.7673.966240.1652
Llama3.1KVzip-CD9.92110.08210.26110.41410.592210.5044
Llama3.1Distance-KV3.0003.1483.3143.4533.731220.1696
Qwen2.5Dense2.9893.4663.9444.0654.187320.1308
Qwen2.5StreamingLLM2.8113.1573.4663.7764.084500.0817
Qwen2.5DuoAttention2.7633.0473.3283.6073.888500.0778
Qwen2.5MoA10.67619.09627.63536.02344.925500.8985
Qwen2.5KVzip-CI2.7742.8642.9543.0883.359250.1344
Qwen2.5KVzip-CD7.9428.1988.4528.7088.960500.1792
Qwen2.5Distance-KV2.7813.0323.2793.5273.774500.0755

The final column divides cumulative end-to-end time through five queries by the actual number of generated tokens. It includes one-time context preparation and is not isolated decode latency. Output counts are reported to make the actual generation workload explicit.

KVzip-CD’s main additional cost occurs during initial context preparation, after which its cache is reused. Distance-KV avoids this one-time reconstruction-based scoring cost. In this five-query workload, Distance-KV takes 3.731 s and 3.774 s on the two models, compared with 10.592 s and 8.960 s for KVzip-CD, demonstrating the practical cost advantage of static retention in this shared-context setting.

All relevant method implementations, evaluation scripts, and required patterns and configurations are available in our new anonymous repository: https://anonymous.4open.science/r/Distance-KV-Pattern-New-5AF1/

AuthorsOpenReview ↗

Response to Reviewer Pxvn (Part 3/3)

Q3: KVzip-CI configuration

We use ratio=0.6 and level="head", where 0.60 specifies a target retention ratio of 60%, rather than 60% eviction. We have replaced “head-level pruning ratio” with “head-level retention ratio” in Appendix B of the revised manuscript.

At the 128K efficiency setting, the actual physical KV sizes are 9.35 GiB for Llama3.1 and 4.03 GiB for Qwen2.5, corresponding to physical retention ratios of approximately 59.8% and 59.0%, respectively. The complete implementation and experimental configurations are provided in the new anonymous repository listed in W2 / Q2.

Q4: GQA training semantics and query-head-level parameterization

After checking the training and inference code, we confirm that the original description in Appendix A.2—query heads independently modulate attention and are combined only at inference—was incorrect. During training, we already combine the retention decisions of query heads sharing the same KV head before computing attention; inference uses the same within-group union rule. Thus, the training–inference inconsistency implied by our description does not occur in the actual implementation. We have corrected this description in Appendix A.2 of the revised manuscript.

Regarding parameterization, our motivation is that an individual query head’s retrieval capability varies with relative distance. We therefore naturally use query-head–distance pairs as the units of analysis and learning, aligning the parameterization with the capability structure we investigate. For GQA, these decisions are mapped to the shared physical KV cache through the within-group union.

Since the resulting retention policy operates at the KV-head level, we expect that directly learning KV-head-level distance patterns could also achieve comparable performance. This is an alternative parameterization worth further comparison under matched physical KV budgets.

The related training and inference code, together with training records, is available in the new anonymous repository listed in W2 / Q2.

Comments on the paper

2 comments

Dear Reviewer,

I hope this message finds you well. I am writing to kindly remind you that the authors’ rebuttal has been posted and is now available for your review in the system.

We would greatly appreciate it if you could review the rebuttal and provide your assessment at your earliest convenience, as your input is important for us to move forward with a timely decision.

Thank you very much for your time and continued support.

ACs

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.