acceptodds
Under review as a conference paper at ICLR 2027

WaveKV: Where Attention Meets Signal Processing for KV-Cache Compression

19 views

Abstract

Key-value (KV) caching enables efficient text generation with Large Language Models (LLMs), but its near-linear growth with generated tokens creates increasing memory demands and a memory wall. Existing compression methods rely on query-dependent token eviction or model fine-tuning, which either discard useful KV states or require model adaptation, limiting simultaneous reductions in KV storage and attention computation. Motivated by structured attention patterns and signal-energy redundancy of KV states in the frequency and discrete-wavelet domains, we propose , a training-free KV-cache compression framework that combines discrete cosine transforms (DCT) and discrete wavelet transforms (DWT) to jointly reduce KV-cache storage and attention computation. first performs a one-time, query-independent attention calibration to capture positional attention structure and retrieval-sensitive layers. The resulting positional policy generalizes across datasets and models to guide adaptive DCT-based compression. It then uses adaptive, attention-guided DWT-based hierarchical routing to selectively access the compressed representation during attention computation without directly evicting tokens. Across LLaMA-3.1-8B, Phi3-medium-128k, and Qwen3-8B, achieves \sim\textbf{2\times} KV-cache compression, improving average LongBench and RULER scores over similarly compressed baselines by % and %, respectively, while maintaining competitive language-modeling and long-context capability. At the system level, reduces attention scoring compared to dense full-KV by up to \textbf{11.7\times} and peak memory by % at 64K context, while supporting 128K contexts where full KV runs out of memory. Our code is available at https://anonymous.4open.science/r/WaveKV-95DF

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Long-Context LLMs, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_36sMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a KV cache compression method that combines Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT). It employs a one-time attention calibration process to determine the compression strategy and utilizes hierarchical routing to reduce the computational overhead of attention mechanisms. The authors evaluate the approach across various models in terms of language modeling performance, long-context task capabilities, and system efficiency, which aims to balance cache capacity and computational costs without the need for fine-tuning.

Critical strengths
  1. The method accounts for both the size of cached representations and the cost of attention-based access. It is designed to balance these two types of overhead, with a clear system-level motivation.
  2. The experiments encompass multiple models and diverse task types, and include ablation studies on compression and routing, which helps clarify the method's scope of applicability.
Critical weaknesses
  1. In Appendix E.7, the query-dependent importance scores for certain baselines are replaced with scores derived solely from key vectors. This modification could significantly alter the performance of the original methods. However, the main tables continue to use the original method names, making it difficult to assess the actual advantages over the original, unmodified versions.
  2. There is a clear inconsistency in the "SnapKV" row of Table 2: the arithmetic mean of the 14 listed task scores is approximately 6.40, yet the reported average is 37.40, a value higher than any of the individual task scores in that row. Furthermore, improvements in average performance relative to baselines can be skewed by the inclusion of weaker baselines. It is more meaningful to prioritize reporting gains relative to the strongest comparable method.
Questions for authors
  1. Regarding the task scores and their averages for SnapKV in Table 2, which specific values require correction? Does this correction affect the primary comparisons or the improvement percentages cited in the abstract?
  2. Could you clearly distinguish between the "vanilla" (original) baselines and the query-agnostic modified versions? Furthermore, which of these comparisons do your current results actually support?
  3. Does the reduction in attention score computations in WaveKV lead to tangible end-to-end inference acceleration? It is recommended to provide data on decoding throughput and peak VRAM usage across various context lengths under consistent cache budget and hardware conditions, while accounting for the additional overheads introduced by the wavelet transform and cache maintenance.
2 comments
AuthorsOpenReview ↗

Response to reviewer questions regarding numerical consistency, baselines for comparison, system performance.

We thank the reviewer for the constructive questions. We have corrected the reported averages, clarified the baseline protocols, and distinguished attention-scoring reduction from measured inference performance.

Q1. SnapKV correction and abstract comparisons

The 14 task-level scores for query-independent SnapKV in Table 2 are correct. Only its LLaMA-3.1-8B LongBench average changes from 37.40 to 6.40. Each LongBench average is weighted by the number of evaluation samples in each task. We have also reconciled the Full KV scores across Tables 2 and 12 (Table 10 in initial submission). Neither correction changes WaveKV's measured scores. We recalculated the abstract's LongBench comparison against all 12 baselines in the query-independent evaluation, rather than selecting the next-best method. These comprise GemFilter, SnapKV, PyramidKV, LaCache, FastKV, CaM, D2O, ThinK, StarKV, WaveFilter, FreqKV, and PyramidInfer. Query-dependent selection is adapted to key-only importance where applicable, while inherently query-independent components remain unchanged (Appendix E.7). We first compute each method's sample-weighted average over 14 LongBench tasks, then average equally across the 12 baselines and three models. The resulting baseline mean is 27.36, compared with 41.62 for WaveKV, giving 52.1% relative improvement, revised from the original 40% abstract claim. The remaining abstract comparisons are unaffected by this LongBench correction. We have clarified the aggregation and corrected Tables 2 and 12.

Q2. Vanilla, unmodified, and query-independent baselines

Full KV (vanilla) denotes inference without compression. Unmodified baselines use each published method's native importance scoring, token selection, and compression. The query-independent variants in Appendix E.7 replace query-derived importance scores with key-only statistics, preserving their cache budgets and other compression settings. The latter variants isolate the effect of query-independent cache construction. Several existing approaches use attention from recent queries to select and irreversibly evict states, potentially discarding information needed by later queries at arbitrary context positions. WaveKV instead compresses independently of the decoding query and uses query-dependent hierarchical routing over its retained representation. The query-independent comparisons therefore serve as a controlled evaluation, not a replacement for the published implementations. We additionally evaluated the unmodified methods under comparable cache budgets. On LLaMA-3.1-8B, WaveKV scores 44.56, versus 43.08 for PyramidInfer. On Qwen3-8B, it scores 43.19, versus 41.55 for PyramidInfer and 41.36 for FastKV. On Phi-3-medium-128k, WaveKV scores 37.12, while SnapKV (40.38) and PyramidKV (40.36) perform better. Thus, the relative advantage varies by model. Importantly, LongBench averages alone do not characterize retrieval. At 16K, WaveKV achieves 86.57% RULER accuracy on LLaMA-3.1-8B, versus 45.66% for PyramidInfer and 89.08% for Full KV. Figure 10 also demonstrates stronger Needle-in-a-Haystack retrieval across needle positions and context lengths, consistent with retaining compressed context rather than irreversible eviction. We distinguish both protocols in Section 4.1.2, Tables 2 and 12, and Appendices E.7 and G.2 of the revised submission.

Q3. Attention-scoring reduction and system performance

The requested system measurements were included in Section 4.3 (Section 4.2 in the initial submission) and Appendices E and F of the initial submission. Figure 11 reports 11.7× fewer attention-scoring operations at 32K, while Tables 5–8 report measured decoding throughput, peak GPU memory, and prefill latency. Attention-workload reduction is not equivalent to an end-to-end speedup because wavelet routing, cache maintenance, and other inference operations also contribute to runtime. These costs are reflected in the measured throughput. At 16K/32K/64K, WaveKV achieves 22.4/23.1/22.4 tokens/s, versus 25.3/25.4/24.9 for optimized Full KV, a 9–12% throughput difference. At 16K, WaveKV is approximately 70% faster than FreqKV (22.4 versus 13.2 tokens/s), with comparable throughput at 32K–64K. At 16K, WaveKV uses 46.8 GB versus 42.3 GB for Full KV. At 32K, peak GPU memory reduces by 15.3% (32.1 versus 37.9 GB), reaching 40.1% at 64K (33.2 versus 55.4 GB). At 64K, WaveKV also reduces TTFT from 10.70 to 9.42 s relative to Full KV. At 128K, WaveKV uses 35.4 GB of peak memory and sustains 21.5 tokens/s, whereas Full KV encounters OOM and achieves lower TTFT than FreqKV (19.30 versus 22.02 s). These results demonstrate joint savings in KV storage and attention computation with competitive long-context decoding and improved memory scalability.

The rebuttal questions also address the critical weaknesses.

Reviewer_36sMOpenReview ↗

My concerns have been adequately addressed

Thanks for the detailed response and additional experiments. My concerns have been adequately addressed, and I have therefore raised my score from 2 to 3.

Reviewer_RVCsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WaveKV, a training-free KV-cache compression framework that combines DCT-based frequency compression with Haar-DWT hierarchical routing. A one-time attention calibration captures a three-part Head/Info/Tail positional structure and an attention-growth function, which guide adaptive block-wise DCT energy allocation; a DWT tree then performs coarse-to-fine, query-dependent routing to reduce attention scoring. A needle-in-a-haystack calibration further assigns layers to PROTECT/SOFT/BLIND tiers to modulate compression budgets. Experiments cover LLaMA-3.1-8B, Qwen3-8B, and Phi3-medium-128k on PG-19, LongBench, RULER, NIAH, GSM8K, and MMLU, with system-level measurements on H100.

Critical strengths
  1. The system-level evaluation is solid and honest. Besides quality metrics, the authors report attention-scoring reduction, realized throughput, peak memory, recompression counts, and TTFT across batch sizes and context lengths (Tables 5–8, Figures 11, 18). They also explicitly state that the reduction in attention FLOPs does not translate proportionally into wall-clock speedup (Section 4.2.2, Appendix A), without overclaiming. For a compression paper whose main selling point is system benefit, this is more credible than much of the related work, and it lets readers judge the practical deployment value (mainly the 128K OOM-avoidance regime).

  2. The RULER 16K results are convincing empirical evidence. On LLaMA-3.1-8B, WaveKV reaches 86.57% vs. 89.08% for full KV, while PyramidInfer (45.66%), WaveFilter (54.60%), and FreqKV (31.80%) largely collapse; the same trend holds on Qwen3 and Phi3. Retrieval is exactly where eviction and naive frequency compression tend to fail, and the gaps of tens of points are hard to explain by tuning noise. This supports the core design of coupling compression, routing, and retrieval-aware layer tiers.

Critical weaknesses
  1. The "40%/75% over similarly compressed baselines" claim in the abstract does not hold up and is inflated by weak baselines. On LongBench LLaMA-3.1-8B (Table 2), WaveKV scores 44.56 vs. 43.08 for PyramidInfer, a relative gain of only about 3%, not 40%. The 40% figure only appears when collapsed baselines such as SnapKV (8.59), FastKV (4.22), ThinK (8.96), and StarKV (7.90) are averaged in, and several of these are query-free variants constructed by the authors themselves (Appendix E.7). The "~70–75%" RULER claim is similarly pulled up by FreqKV at 31.80%. This matters because the abstract, introduction, and conclusion all foreground these percentages as the main evidence of effectiveness; if the honest comparison is roughly on par with PyramidInfer and only modestly above FreqKV, the contribution is materially smaller than advertised.

  2. Quality results are inconsistent across benchmarks, and on several the method is worse than full KV or than simpler baselines, which undercuts the "competitive language modeling" claim. On PG-19 (Table 1), WaveKV is worse than full KV at every context length (e.g., 8.97 vs. 8.42 at 2K; 9.22 vs. 8.25 at 32K) and worse than fine-tuned FreqKV at 1K/2K/8K/16K/32K. On GSM8K (Table 4), it drops to 74.50 vs. 82.50 for full KV, while WaveFilter, dismissed as providing "no KV-cache storage compression," reaches 84.00. On LongBench Phi3 (Table 10), WaveKV (37.12) is below D2O (38.63) and CaM (38.15). This matters because the paper positions itself as training-free compression that preserves quality; a consistent 5–10% PPL degradation and an 8-point GSM8K drop suggest the compression is not as lossless as claimed, and the abstract's focus on the best-case benchmark (RULER) obscures this. Readers cannot tell from the main text when WaveKV is actually preferable to full KV.

Questions for authors
  1. Attention-scoring reduction vs. throughput. Given that realized throughput is lower than full KV at 16K/32K/64K (Table 5: 22.4/23.1/22.4 vs. 25.3/25.4/24.9), what is the concrete deployment regime where WaveKV is a net win? Is it purely the 128K OOM-avoidance case, and if so, how large is that regime relative to the claimed 11.7× scoring reduction?

  2. Calibration generalization. Calibration is done on Pile-GitHub/Wikipedia and the PROTECT/SOFT/BLIND tiers are fixed model-specific layer sets (Appendix D.2). How sensitive are the results to the calibration corpus? If calibration were done on a different domain (e.g., code vs. narrative), would the Head/Info/Tail boundaries and tier assignments transfer, or does this reintroduce a hidden domain dependence that undercuts the "generalizes across datasets" claim?

2 comments
AuthorsOpenReview ↗

Response to reviewer questions regarding WaveKV deployment benefits, and sensitivity of calibration corpus.

We thank the reviewer for the constructive questions. We have clarified WaveKV's system-level benefits and added cross-domain calibration experiments in the revised manuscript.

Q1. Attention-scoring reduction and deployment benefits

WaveKV is designed to jointly address KV-cache storage, attention computation, and long-context inference quality through training-free, query-independent cache compression. A one-time offline calibration determines model-specific compression and routing configurations, stored as reusable metadata without model fine-tuning. Query-independent DCT compression maintains a compact context representation, while query-dependent DWT hierarchical routing selectively accesses relevant KV states during decoding. This enables adaptive attention computation without query-specific cache construction or irreversible token eviction.

The benefits of this design are reflected in long-context evaluations. At 16K, WaveKV achieves 86.57% RULER accuracy, close to Full KV (89.08%) and substantially higher than PyramidInfer (45.66%) and FreqKV (31.80%). At 32K, WaveKV achieves an 11.7× reduction in attention-scoring operations, demonstrating the effectiveness of hierarchical routing while preserving long-context retrieval capabilities.

At the system level, WaveKV maintains competitive decoding throughput despite the additional operations associated with routing and cache maintenance. Across 16K–64K, WaveKV incurs only a 9–12% throughput reduction relative to optimized Full KV, while achieving approximately 70% higher throughput than FreqKV at 16K and comparable throughput at 32K–64K. Importantly, these benefits extend to contexts where Full KV remains operational. At 32K, WaveKV reduces peak GPU memory by 15.3%, increasing to 40.1% at 64K, while sustaining 22.4 tokens/s versus 24.9 tokens/s for Full KV at 64K. At 128K, WaveKV sustains 21.5 tokens/s where Full KV encounters OOM, while achieving lower TTFT than FreqKV (19.30 s versus 22.02 s).

Thus, WaveKV's practical utility extends beyond avoiding memory exhaustion at extreme context lengths. It combines training-free deployment, query-independent compression, improved memory efficiency, and reduced attention computation with competitive long-context accuracy and decoding throughput. The 11.7× scoring reduction characterizes the reduction in attention workload rather than an equivalent end-to-end speedup, while the measured throughput and memory results demonstrate benefits across both memory-feasible and memory-constrained operating regimes.

Q2. Transferability of calibration across domains

Appendix D.1 (Figure 13) shows similar positional attention structures across different context lengths and domains, including Pile-GitHub and Pile-Wikipedia. The Head/Info/Tail decomposition remains consistent, supporting fixed Head (64 tokens) and Tail (256 tokens) boundaries.

To quantify calibration sensitivity, we independently calibrated LLaMA-3.1-8B on Wikipedia, GitHub, and arXiv. We measure the overlap of layer assignments using Jaccard similarity, Jt(c,c′)=∣Lt(c)∩Lt(c′)∣∣Lt(c)∪Lt(c′)∣J_t(c,c')=\frac{|\mathcal{L}_t^{(c)}\cap\mathcal{L}_t^{(c')}|}{|\mathcal{L}_t^{(c)}\cup\mathcal{L}_t^{(c')}|}, where Lt(c)\mathcal{L}_t^{(c)} denotes the layer set assigned to tier tt using corpus cc. Overall agreement measures the fraction of layers assigned the same tier under both calibrations.

Using Wikipedia as the reference, we obtain

CorpusPROTECT JJSOFT JJBLIND JJAll-tier agreement
GitHub0.905 (19/21)0.200 (1/5)0.636 (7/11)84.4% (27/32)
arXiv1.000 (21/21)0.500 (2/4)0.778 (7/9)93.8% (30/32)

The PROTECT assignments exhibit at least 90.5% Jaccard similarity, while overall tier agreement remains above 84%. This indicates that retrieval-sensitive layers remain largely consistent across domains, with most variations affecting SOFT and BLIND assignments.

To examine whether these variations influence downstream retrieval, we evaluate RULER at 8K using each calibration configuration.

Calibration corpusRULER 8K average
Wikipedia93.62
GitHub93.22
arXiv93.66

RULER accuracy varies by at most 0.44% across the three calibration corpora, indicating minimal sensitivity to these layer-assignment differences in the evaluated setting. Together with the consistent positional attention structure, these results support the transferability of WaveKV's calibration across the evaluated domains without requiring workload-specific recalibration. As described in Appendix D.2, calibration uses samples independent of evaluation datasets and is performed once per model, with configurations fixed during inference. The additional results are provided in Section 4.2, Tables 5 and 6.

Critical Weaknesses not addressed by the questions have been discussed in Appendices G.2 and G.4 of the revised submission.

Area_Chair_K9j1OpenReview ↗

Reminder: Rebuttal Discussion Deadline

Dear Reviewer,

The discussion deadline is only a few hours away. Could you please respond to the authors' rebuttal before the deadline?

Best regards, Area Chair

Reviewer_5CDnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WaveKV combines adaptive DCT-based KV compression with Haar-wavelet hierarchical routing without model fine-tuning. Offline attention calibration determines compression budgets and routing policies across positions or layers. Experiments on three model families report language-modeling and long-context results, reduced attention-scoring work, and lower peak memory at long contexts.

Critical strengths

The shared calibration policy offers a concrete way to coordinate compression and routing, rather than applying them independently. Figure 9 supports the benefit of adaptive routing over a uniform beam, making this a potentially useful training-free design.

The implementation demonstrates practical memory savings.

Critical weaknesses

Section 3.3 ranks subtrees using mean keys, which can conceal isolated keys with high attention scores. Coarse-scale energy concentration does not establish reliable query-specific routing. Although adaptive beams and protected layers mitigate this risk, evidence on the attention mass missed by routing is needed to assess its impact.

Calibration proxies may not capture compression sensitivity. Section 3.2 allocates compression budgets using spectral energy and needle-attention peaks. Low-energy components can still carry essential details, while layers without sharp retrieval peaks may contribute through distributed attention. The paper should establish whether these proxies predict the actual output changes caused by compression.

Reported results are internally inconsistent. In Table 2, SnapKV’s 14 displayed task scores average approximately 6.40, but its reported average is 37.40. Llama Full KV also differs between Tables 2 and 10, with averages of 47.86 and 37.19. These discrepancies affect the interpretation of relative performance. Consistent task-level results and aggregation rules are necessary before relying on the headline improvements.

Questions for authors

How do the unmodified baselines perform under matched, measured memory budgets?

Which LADDER/TIER/FF configurations produced each quality and efficiency result, and what are their actual KV and routing-index sizes? Does PROTECT preserve original or previously compressed states?

Can you reconcile the task scores, averages, and Full KV results across Tables 2 and 10?

3 comments
AuthorsOpenReview ↗

Response to reviewer questions on baselines, WaveKV configurations with their KV/routing sizes, and numerical consistency.

We thank the reviewer for the constructive questions. We have clarified baseline evaluation protocols, configuration choices, and LongBench reporting in the revised manuscript.

Q1. Unmodified baselines under matched budgets

We expanded LongBench to include the unmodified implementations of existing KV-cache compression methods, preserving their published importance scoring, token selection, and compression mechanisms under comparable cache budgets and evaluation settings. On LLaMA-3.1-8B, WaveKV scores 44.56, exceeding PyramidInfer (43.08), the strongest competing compressed baseline. On Qwen3-8B, WaveKV scores 43.19, versus PyramidInfer (41.55) and FastKV (41.36). On Phi-3-medium-128k, WaveKV scores 37.12, close to PyramidInfer (37.20), but below SnapKV (40.38) and PyramidKV (40.36). However, LongBench averages do not fully characterize long-context retrieval. Although some methods match or exceed WaveKV on LongBench, WaveKV attains 86.57% on 16K RULER for LLaMA-3.1-8B, compared with 45.66% for PyramidInfer and 89.08% for Full KV. Figure 10 likewise shows stronger Needle-in-a-Haystack retrieval across needle positions and context lengths.

Importantly, unmodified baselines do not necessarily share WaveKV's training-free, query-independent cache-construction constraints. Several methods use recent-query attention scores to select or irreversibly evict KV states, potentially discarding information needed by future queries at arbitrary positions. Some competing approaches additionally require model-specific training or fine-tuning. WaveKV instead constructs its compressed representation independently of the current query and uses query-dependent hierarchical routing during decoding. Therefore, we evaluate query-independent adaptations using key-only importance statistics while retaining other compression settings and budgets. Several baselines degrade further under this constraint. This controlled evaluation isolates query-independent cache construction and complements, rather than replaces, the unmodified comparisons. The new results appear in Section 4.1.2, Table 2 and Appendix G.2, Table 12.

Q2. Configurations and KV/routing sizes

Section 3.3 and Appendix D.2 of the main paper describe LADDER and TIER, while Appendix F describes Fast-Forward (FF).

  • LADDER uses position-adaptive DCT blocks of 256/128/64 tokens and corresponding routing beam widths of 32/64/128. The Head (64 tokens) and Tail (256 tokens) remain uncompressed.
  • TIER classifies layers as PROTECT, SOFT, or BLIND with energy-retention targets of 1.0/0.5/0.2. PROTECT retains the original full-resolution KV cache, with no compression or routing, and uses dense attention. SOFT and BLIND apply LADDER compression with beam widths 128 and 64, respectively.
  • FF uses an 8192-state rolling prefill cache and 0.4 DCT retention for older regions before standard compression and decoding.

The zero-indexed layer sets below are ordered PROTECT / SOFT / BLIND, with counts in parentheses.

  • LLaMA-3.1-8B (17/9/6). {5,6,8–11,13–17,19–21,24–26} / {2,7,18,22,23,27–30} / {0,1,3,4,12,31}.
  • Qwen3-8B (22/10/4). {1–15,17–23} / {16,24,26,28–34} / {0,25,27,35}.
  • Phi-3-medium-128k (23/10/7). {8–28,30–31} / {2–7,29,32,33,35} / {0,1,34,36–39}.

PG-19, LongBench, GSM8K, and MMLU use LADDER. RULER and Needle-in-a-Haystack additionally use TIER, retaining LADDER compression in non-PROTECT layers. FF is used only in hardware evaluations to accelerate prefill. Stored KV sizes depend on full-resolution PROTECT layers and retained DCT coefficients in compressed layers. Routing beam width kk limits leaf-level candidates to at most 2k2k per tree, excluding densely attended regions. With TT retained states, the padded binary tree has P=2⌈log⁡2T⌉P=2^{\lceil\log_2 T\rceil} leaves and P−1P-1 internal subtree-mean keys for routing. These configurations and index dimensions are described in Section 3.3 and Appendix D.2.

Q3. Consistency of Tables 2 and 10

The query-independent SnapKV task-level scores are correct. Its LLaMA-3.1-8B LongBench average is corrected from 37.40 to 6.40. We also reconciled Full KV results across Tables 2 and 12 (Table 10 in the initial submission). All LongBench averages are sample-weighted across 14 tasks using each task's evaluation sample count, not an unweighted mean. The corrected tables now state this procedure explicitly.

Critical Weaknesses not addressed by the questions have been discussed in Appendix D.2 of the revised submission.

Thank you for the corrections and clarifications. The reliability of the compression and routing criteria, and the quality–memory trade-off under matched configurations, remain insufficiently established. I will maintain my rating of 2.

AuthorsOpenReview ↗

Response to reviewer question on reliability of compression and quality-memory trade-off.

We appreciate the reviewer's continued assessment. However, the intended scope of the concerns regarding the reliability of compression and routing criteria and the quality–memory trade-off under matched configurations remains somewhat unclear to us. To the best of our understanding, we address the theoretical foundations of WaveKV's mechanisms and the evaluation of quality under matched compression budgets.

1. Reliability of compression and routing criteria

DCT-based compression. As established by FreqKV and characterized in Figure 2, KV representations exhibit substantial low-frequency energy concentration. For an orthonormal DCT, Parseval's theorem establishes that the squared reconstruction error from coefficient truncation equals the energy of discarded coefficients:

∣x−x^∣22=∑k∈D∣Xk∣2.\lvert x-\hat{x} \rvert_2^2=\sum_{k\in\mathcal D}|X_k|^2.

Thus, DCT energy retention provides a mathematically grounded measure of representation fidelity rather than an arbitrary proxy for semantic importance. WaveKV further uses attention-based calibration to allocate this fidelity. The Head and Tail remain uncompressed, retention increases toward recent Info positions, and retrieval-sensitive PROTECT layers preserve their full KV representations. Appendix D.2 (Figures 16–17) demonstrates limited perplexity variation under moderate perturbations of these calibration parameters.

DWT-based routing. Haar-DWT exploits the multiscale structure of KV representations to organize keys into progressively finer regions. Rather than treating coarse-scale energy as a definitive indicator of query relevance, WaveKV evaluates subtree-mean keys against the current query, retains the top-kk branches at each level, and performs exact query-key scoring over up to 2k2k leaf candidates. Adaptive beam widths (k=32,64,128k=32,64,128 in LADDER) provide broader candidate coverage in finer-grained, higher-attention regions. In retrieval-oriented TIER, PROTECT layers use dense attention, while SOFT and BLIND use k=128k=128 and k=64k=64, respectively.

Although subtree means cannot guarantee that isolated high-attention keys survive pruning, adaptive beams mitigate this limitation. At a matched ∼6×\sim6\times attention-scoring reduction, WaveKV achieves ∼5×\sim5\times lower perplexity error than uniform routing (Figure 9). Appendix D.2 (Figure 17) further demonstrates limited sensitivity to moderate beam-width variations. RULER and Needle-in-a-Haystack provide complementary evidence of retrieval effectiveness, although they do not directly measure missed attention mass.

2. Quality–memory trade-off under matched configurations

Our evaluation targets approximately 50% of Full-KV storage across compressed baselines, with budgets configured according to each method's native compression mechanism. Eviction-based methods control retained KV states, merging-based methods control stored representations, and channel/frequency-based methods control retained channels or coefficients. Method-specific protected regions and compression settings are preserved. The query-independent adaptations in Appendix E.7 retain their corresponding baseline cache budgets, ensuring controlled comparisons without altering the intended compression ratio. WaveFilter retains the full KV cache and is therefore distinguished from storage-compressing methods. We also distinguish nominal KV-cache compression from measured peak GPU memory, which includes model parameters, routing structures, and temporary allocations. Table 1 reports PG-19 perplexity across context lengths, while the hardware evaluations quantify actual inference memory requirements. At 64K, WaveKV reduces peak GPU memory from 55.4 GB to 33.2 GB (40.1% reduction), while supporting 128K inference where Full KV encounters OOM.

FreqKV provides a relevant quality–memory comparison. Although fine-tuned FreqKV achieves lower PG-19 perplexity at several context lengths (Table 1), its advantage does not consistently extend to retrieval-intensive workloads. Figure 10 demonstrates substantial NIAH degradation beyond its evaluated 8K fine-tuning regime, while WaveKV maintains strong retrieval performance through 16K without model adaptation. On 16K RULER, FreqKV achieves 31.80% average accuracy and 23.51% on NIAH, compared with 86.57% and 91.89% for WaveKV. These results demonstrate that favorable perplexity does not necessarily translate into effective long-context retrieval.

Furthermore, WaveKV requires no model fine-tuning, whereas the fine-tuned FreqKV configuration incurs additional offline model-adaptation costs, which we distinguish from inference-time memory. Thus, WaveKV's advantage is not uniformly superior perplexity or minimum memory consumption, but a training-free quality–memory trade-off combining substantial inference-memory savings with strong long-context retrieval performance, without query-dependent cache construction or model fine-tuning.

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.