- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Distance-KV, a query-independent KV cache compression method motivated by the observation that an attention head’s retrieval behavior varies with relative distance. It learns a static binary retention pattern over layers, query heads, and distance bins using synthetic retrieval examples while freezing the language model. Optimization combines Hard Concrete gates with straight-through global Top-K refinement. After dense context prefill, the learned pattern is used to prune and compact the cache; sink and recent tokens are preserved, and newly appended tokens are retained in full. For GQA models, logical query-head decisions are combined by union within each shared KV head.
The paper evaluates three backbones using NIAH, LongBench, RULER, and SCBench. It reports the highest aggregate scores among the evaluated compression baselines and, for Llama-3.1-8B at 128K, a 65.4% reduction in context KV memory and a 1.66× decoding speedup over Dense. Structural ablations support the usefulness of jointly modeling heads and distance.
Critical strengths
Critical Strengths
-
The paper identifies useful structure within attention heads and provides supporting ablations. Figure 1 illustrates distance-dependent retrieval behavior. At matched physical KV retention, shuffling distance assignments reduces the RULER task average from 63.7 to 54.7, while head-only retention achieves 44.5 (Table 3). These results support the importance of the learned correspondence between heads and distance intervals.
-
A reusable static policy achieves promising quality and cache savings. One pattern per backbone is reused across benchmarks, including shared-context multi-turn evaluation. On Llama-3.1, LongBench performance remains close to Dense (47.30 versus 47.53), alongside substantial context-cache savings at 128K. Avoiding query-dependent rescoring makes the approach practically attractive for reusable context caches.
Critical weaknesses
Critical Weaknesses
-
The deployed GQA attention policy differs from the policy optimized during training, and the effect is not evaluated. Appendix A.2 and the released training implementation apply independent distance masks to query heads. At inference, the implementation unions these masks within each shared KV head and passes the resulting cache to FlashAttention without reapplying the individual query-head masks. A query head therefore attends to positions excluded by its own training-time mask, changing both the softmax normalization and attention output. This is consistent with the stated union operation, but it means the deployed policy is not the exact policy optimized by the retrieval objective. Since both modern backbones use GQA, this discrepancy is central to interpreting the method. In addition, the budget constrains logical query-head entries rather than physical KV storage.
-
The released artifacts do not fully support verification of the reported results. Applying the Figure 6(b) definition—heads retaining some but not all learnable distance bins—to the released 20%-budget patterns gives 60.84% for Llama-2, 48.44% for Llama-3.1, and 39.16% for Qwen2.5, compared with the reported 62.5%, 70.2%, and 77.9%. The release therefore appears to contain different patterns or use a different statistic, without sufficient provenance to resolve the discrepancy. The package also imports a missing internal module (
training.audits.gradient_audit), and the downloaded release lacks benchmark, baseline, and efficiency-measurement scripts. This prevents checking important experimental controls, including the unusually large improvement over Qwen Dense at 128K. Main-method comparisons also lack performance–physical-memory curves across budgets, while efficiency reporting omits pruning/packing latency and end-to-end cost despite requiring dense prefill.
Questions for authors
Questions For Authors
The following questions concern the current method and reported results.
-
GQA training–deployment consistency: The released inference implementation uses the GQA-group union without enforcing individual query-head masks. How much does this change retrieval loss and benchmark performance relative to the training-time binary policy? Please provide a controlled comparison and explain why optimizing the independent masks is appropriate for the deployed union policy.
-
Released-pattern provenance: What explains the discrepancy between Figure 6(b) and the partially retained query-head fractions calculated from the released
budget20.ptfiles? Please provide the exact statistic, figure-generation script, and checkpoint or pattern identifiers used for both the figure and the main benchmark results. -
Physical-resource comparison: Can you provide comparisons with the strongest baselines at matched physical KV memory, preferably across several budgets? Please also explain how physical retention was matched in the structural ablations and how the restricted variants were optimized.
-
Evaluation and efficiency controls: Please provide the benchmark and timing scripts, including exact checkpoint, RoPE/YaRN, attention-kernel, and position-index settings for every method. What explains Qwen’s RULER group average of 23.47 for Dense versus 68.38 for Distance-KV at 128K? Please report pruning/packing latency and end-to-end inference time, and clarify whether peak-memory measurements include dense prefill.
-
Reproducibility and checkpoint selection: Please restore the missing internal module and document the complete training-to-export procedure. Which training checkpoints produced the released patterns and reported results? In particular, what explains the difference between the Distance-KV task averages in Table 3 (63.7) and Table 8 (71.64), and how were the ablation checkpoints selected?
6 comments
Response to Reviewer mBFD (Part 2/2)
Q4: Evaluation settings and end-to-end efficiency
We follow the official RULER evaluation protocol and each method’s original implementation with the necessary model adaptations. Dense and Distance-KV use the same Qwen2.5-7B-Instruct backbone and YaRN settings (factor 4, original context length 32768). The evaluation and timing scripts, together with the checkpoint, RoPE/YaRN, attention-kernel, and position-index configurations for each method, are provided in the new anonymous repository.
Qwen2.5’s performance degradation at 128K is not unique to our experiments. Table 16 of the official Qwen technical report shows that without DCA and YaRN, Qwen2.5-7B-Instruct’s RULER score decreases from 89.4 at 32K to 31.4 at 128K [1]. RULER identifies failure to ignore distractors as an important failure mode [2], while Differential Transformer shows that suppressing attention noise can improve long-context retrieval [3]. Distance-KV’s learned distance selection may similarly reduce interference from irrelevant positions.
We further measured pruning/packing latency, end-to-end inference time, and full-pipeline peak GPU memory on an A800, following the original efficiency experiment’s 128K setting and reporting medians over three measurements. After full-context prefill, Distance-KV uses the precomputed static pattern to identify retained KV positions and gathers the selected KV entries into contiguous buffers, without online importance scoring.
| Model | Dense prefill (s) | Distance-KV prefill (s) | Distance-KV pruning/packing (s) |
|---|---|---|---|
| Llama3.1 | 33.598 | 33.544 | 0.046 |
| Qwen2.5 | 27.509 | 27.461 | 0.032 |
Prefill latency is close to Dense, and pruning/packing costs less than 0.15% of prefill time on both models.
The following table reports end-to-end time and full-pipeline peak allocated GPU memory for the 512-step setting. End-to-end time includes prefill, each method’s cache preparation/compression, query processing, 16 warmup steps, and 512 measured decoding steps.
| Method | Llama3.1 end-to-end (s) | Llama3.1 full-pipeline peak (GiB) | Qwen2.5 end-to-end (s) | Qwen2.5 full-pipeline peak (GiB) |
|---|---|---|---|---|
| Dense | 59.251 | 44.78 | 116.599 | 38.08 |
| StreamingLLM | 50.448 | 44.78 | 41.863 | 38.08 |
| DuoAttention | 56.578 | 36.94 | 42.856 | 38.09 |
| MoA | 97.999 | 59.40 | 89.845 | 57.61 |
| KVzip-CI | 57.259 | 32.34 | 42.685 | 23.14 |
| Distance-KV | 50.928 | 44.78 | 40.884 | 38.08 |
Regarding peak memory, the original paper reports peak memory during decoding; its measurement interval excludes dense prefill. The full-pipeline peaks above include dense prefill and account for model weights, KV caches, and temporary tensors. Distance-KV’s full-pipeline peak is close to Dense because it completes dense prefill before pruning; its memory savings apply to the retained KV cache and the decoding stage. The revised manuscript reports end-to-end latency and full-pipeline peak memory in Table 10 (Appendix D.1).
Q5: Reproducibility and checkpoint selection
As explained in Q2, the main experiments use each model’s final pattern.
To control computational cost and compare different structures at the same training step, we consistently use step44 for the structural ablations. The shuffled controls are also generated from the reference pattern at this step. Thus, Table 3 reports the step44 result (63.7), whereas Table 8 reports the final step132 result (71.64). The revised manuscript lists the final checkpoints in Appendix A.3 and documents the common ablation step in Appendix E.
The complete training-to-pattern-export procedure, related code, configurations, and corresponding patterns are provided in the new anonymous repository.
[1] Qwen Team. Qwen2.5 Technical Report. arXiv:2412.15115, 2024.
[2] Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? COLM, 2024.
[3] Ye et al. Differential Transformer. ICLR, 2025.
Thank you for the detailed response. My main remaining question is about the Qwen2.5 Dense result in Table 9. Could you provide the exact command and output for the 128K run, and, if available, a no-pruning run through the Distance-KV execution path with the same YaRN and position settings? The cited 31.4 result disables both DCA and YaRN, so it does not directly explain the 24.68 observed in your YaRN-enabled setup.
Response to Reviewer mBFD (Part 1/2)
We thank the reviewer for the careful evaluation and constructive comments. We provide a new anonymous repository containing the Distance-KV training, inference, and structural ablation code, benchmark and efficiency evaluation scripts for Distance-KV and all evaluated baselines, and the required patterns and configurations: https://anonymous.4open.science/r/Distance-KV-Pattern-New-5AF1/. We also provide additional end-to-end efficiency measurements in the responses below.
W1 / Q1: GQA training–inference consistency
We thank the reviewer for pointing this out. After checking the training and inference code, we confirm that the original description in Appendix A.2—query heads independently modulate attention and are combined only at inference—was incorrect. During training, we already combine the retention decisions of query heads sharing the same KV head before computing attention; inference uses the same within-group union rule. Thus, the training–inference inconsistency implied by our description does not occur in the actual implementation. We have corrected this description in Appendix A.2 of the revised manuscript. The related code and training records are available in the new anonymous repository.
W2: Completeness of released artifacts and verification of results
We have included the missing
training.audits.gradient_auditmodule, along with benchmark and efficiency evaluation code for all evaluated methods, in the new anonymous repository. The remaining concerns are addressed individually in Q2–Q5 below.Q2: Released-pattern provenance and the discrepancy with Figure 6(b)
The
budget20.ptfiles in the original anonymous repository were exported at step44, whereas Figure 6 uses patterns exported from each model’s final training checkpoint, resulting in different statistics. For each model, the main benchmark evaluations, efficiency measurements, and Figure 6 all use the same final pattern: step60 for Llama-2, step132 for Llama-3.1, and step80 for Qwen2.5. Step44 is used only as the reference for the structural ablations. The new anonymous repository includes these final patterns. Figure 6(b) reports the fraction of query heads that retain some but not all learnable distance bins, computed bytools/figures/plot_pattern_analysis_v10.py. The corresponding percentages are 62.5%, 70.2%, and 77.9%, consistent with the paper.Q3: Physical-resource comparison
The paper reports physical KV cache sizes measured during execution. For Distance-KV, these measurements reflect the actual cache after the within-group GQA union and physical pruning/packing, including the additional KV positions retained through the union. We do not treat the 20% logical query-head retention ratio as the physical KV retention ratio. The following table summarizes accuracy and physical KV usage for the main baselines:
On Llama-3.1, Distance-KV achieves the highest scores on all three benchmarks while using 30.9% less physical KV memory than DuoAttention, the smallest baseline in the table. On Qwen2.5, it also achieves the highest scores while using approximately 4% less physical KV memory than DuoAttention.
The structural ablations use closely matched physical KV budgets after the GQA-group union. Using the same 128k-token decode-start setting as the efficiency evaluation, the calculated physical KV sizes are:
Head-only uses the closest achievable budget under its whole-head retention constraint; the other controls closely match the reference budget.
Head-only and Distance-only independently train their gate parameters from retrieval-based initialization under their respective structural constraints, using the same training data and optimization procedure as Distance-KV. Distance-shuffled permutes distance assignments within each KV group and requires no retraining.