acceptodds
Desk rejected at ICLR 2027

Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are increasingly applied to very long inputs, such as whole documents, codebases, and conversations that run for many turns. The key-value (KV) cache stores the attention keys and values of all past tokens. Because the cache grows with the context length and is re-read in full at every generated token, it dominates the inference memory at long context. Most existing methods reduce this cost by quantizing the cache: storing every value at the same low precision. These methods reach about two bits per value, but they rarely go below two bits, because accuracy drops sharply at that point. The reason is that two bits allow only four levels per value, which are too few for a cache with outlier channels, which is known as the 2-bit cliff. In our study, we find that once we project the cache into a latent space that removes the correlations between channels, the contributions of the different channels of the KV cache to the attention output are highly uneven. And we find that spending the bit budget on those few important channels is far more accurate than spreading the bit budget evenly. Guided by these observations, we develop SPECTRA, a training-free compression method for the KV cache that requires no change to the model. SPECTRA transforms the KV cache into a latent space in which the channels are ordered by their contribution to the attention output, and then allocates the bit budget across these ordered channels. On long-context benchmarks and three instruction-tuned models, every uniform quantization methods that we compare against has fallen below the 2-bit cliff at 8 compression rate, and none of these methods has a configuration beyond 9. SPECTRA matches the uncompressed model at 8 on Llama-3.1-8B, stays within 1 point of that accuracy up to 11, and recovers every planted needle in needle-in-a-haystack at 12.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM, KVCache, Quantization, Pruning, Efficiency, Compression

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jfKyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPECTRA, a training-free method for compressing the KV cache by combining an activation-weighted latent transformation with non-uniform bit allocation. The paper argues that raw KV-cache channels are strongly correlated, making uniform quantization or direct channel pruning inefficient. SPECTRA instead constructs a model- and activation-dependent transform using a G-weighted factorization of the key and value projection matrices. The resulting latent coordinates are approximately orthogonal and ordered by the amount of activation-weighted energy they carry.

After transforming the cache into this latent space, SPECTRA applies reverse-water-filling-style bit allocation. High-energy latent groups receive more bits, low-energy groups receive fewer bits, and groups assigned zero bits are dropped completely. Thus, quantization and rank reduction are treated as two endpoints of a single allocation procedure. The method also uses group-wise quantization with per-token scales and zero-points, and applies a low-rank output-projection correction for the value path.

The paper evaluates SPECTRA on LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Qwen2.5-7B-Instruct using eight English LongBench tasks and needle-in-a-haystack retrieval. The calibration uses 16 windows of 2,048 tokens, half from WikiText-2 and half from source code. On LLaMA-3.1-8B-Instruct, the dense model obtains a LongBench average of 53.29. SPECTRA obtains 53.29 at 7.65x effective compression, 52.99 at 8.82x, and 52.44 at 11.09x. At the same time, the evaluated uniform quantization baselines generally lose substantial accuracy around the 8x compression regime. The paper also reports 1.9x to 3.4x throughput improvements at equal GPU memory on an H100 NVL, depending on context length and compression ratio.

Critical strengths
  1. The paper presents a coherent explanation for why a transformed latent space can make aggressive KV-cache compression easier. The distinction between a weight-space transform and an activation-weighted transform is technically meaningful, and the G-weighted factorization gives a principled reason for ordering latent coordinates by their produced energy.

  2. The experimental results are strong in the high-compression regime. On LLaMA-3.1-8B-Instruct, SPECTRA remains at 52.99 at 8.82x effective compression and 52.44 at 11.09x, compared with a dense LongBench average of 53.29. The paper also counts quantization metadata in the reported effective ratios and provides equal-memory throughput measurements, which makes the systems evaluation substantially more credible.

  3. The paper connects the representation choice to an actual serving path. By performing attention directly in the latent dimension and folding the up-projection into the query and output computation, SPECTRA can obtain memory and throughput benefits rather than merely reporting lower reconstruction error.

Critical weaknesses
  1. The robustness of the calibration-dependent transform is not yet fully established. The most aggressive operating point is highly sensitive to the calibration corpus: on LLaMA-3.1-8B, the 11.09x point improves from 47.90 with WikiText-only calibration to 52.44 after adding source-code calibration. This is a 4.54-point difference, much larger than the differences at moderate compression. The result suggests that the transform and retained latent directions may be strongly distribution-dependent, which weakens the generality of the “drop-in” claim for unseen workloads.

  2. The evidence for the main long-context claim is narrower than the motivation suggests. Quality experiments use eight English LongBench tasks with inputs capped at approximately 31,500 tokens, and needle-in-a-haystack experiments go up to 32K. The 128K results are primarily throughput measurements rather than task-accuracy measurements. In addition, at high compression the comparison becomes incomplete because several baselines have no operating point beyond approximately 8x or do not support the Qwen model. Thus, the claim that SPECTRA is uniquely effective beyond the 2-bit cliff should be qualified as applying to the evaluated implementations and operating ranges.

Questions for authors
  1. The calibration ablation shows a 4.54-point improvement at 11.09x on LLaMA-3.1-8B when source-code windows are added to WikiText-2. How should a user choose calibration data for a new workload? Is there a principled way to estimate whether a calibration corpus is sufficiently representative before deployment?

  2. How does SPECTRA perform when the evaluation distribution is substantially different from the calibration distribution, for example on documents, mathematical text, multilingual inputs, tool traces, or repository code not represented in the calibration set?

  3. Please report task-level accuracy at longer contexts, especially 64K, 128K, and beyond, rather than only throughput at those lengths. The paper’s central motivation concerns very long contexts, while the main quality evaluation is capped near 32K.

  4. At compression ratios above 8x, several baselines have no reported operating points. Can the authors compare against additional low-rank, eviction, mixed-precision, or latent-cache baselines at matched effective compression ratios, even if this requires implementing the missing operating points?

  5. On Qwen2.5-7B, the released code supports fewer baselines than on LLaMA and Mistral. How much of the observed advantage is robust when the same baseline set is used across all three models?

  6. The transform uses activation Gram matrices and inverse square-root factors. How is numerical conditioning handled when the Gram matrix is rank-deficient or ill-conditioned? Please report the regularization, precision, and stability behavior for different layers and model families.

  7. What is the exact runtime cost of constructing the transformed projections and executing latent-space attention compared with an optimized dense attention kernel? The equal-memory throughput results are promising, but latency at equal batch size and end-to-end time-to-first-token would clarify the deployment trade-off.

Reviewer_g6Y6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats KV-cache compression as a rate-distortion problem. Instead of fixing the number of retained tokens, the rank, or the bit-width in advance, it asks where each bit should go. The method whitens the key and value projections with the second moment G of the layer input, takes an SVD, and obtains a latent that is second-moment orthogonal and ordered by energy. It then allocates bits across latent coordinates by reverse water-filling on their variances, and drives low-variance coordinates to zero bits, so rank reduction becomes one endpoint of the same control. The method is training-free. It needs one small calibration pass and replaces the existing key and value projections at deployment. A closed-form reduced-rank correction is folded into the output projection to absorb the error introduced on the value path.

Critical strengths

A clean formulation with a low engineering cost. Collapsing the retained rank and the per-channel precision into a single control, where a zero-bit channel is simply deleted, puts two previously separate families on one rate-distortion curve. The transform has a closed-form solution, its coefficients come from calibration statistics, and inference only replaces the key and value projections. No fine-tuning and no architectural change is required. This is the most substantial contribution of the paper. A solid evaluation. Three models, eight tasks, full test splits, one harness, byte-accounted compression ratios, throughput measured at a fixed memory budget, and cell-level NIAH grids. The appendix reports the effect of the calibration corpus, the lower ceiling on Qwen, and several tricks that were screened out. This level of transparency is uncommon in this literature.

Critical weaknesses

First, the case for going beyond the 2-bit cliff rests on baselines having no configuration at those ratios, not on baselines failing at the same bit-width. In Table 1, KIVI and OTT reach 52.17 and 53.30 at 7.11x, which is about 2.25 bits, and sit less than 1.1 points below dense. The methods that actually collapse to 46 to 47 points are TurboQuant and RotateKV at their 8x configurations. The spread between two 2-bit methods is nearly 7 points, which says the differences come mostly from implementation quality. At matched quality, SPECTRA needs 7.65x where OTT needs 7.11x, roughly a seven percent advantage in ratio. The abstract states that SPECTRA matches dense at 8x, but the point that matches dense is measured at 7.65x; the next point, 8.82x, scores 52.99 and is 0.30 below dense. The only strictly matched-ratio comparison in the paper is Qwen at 8.00x, where SPECTRA leads TurboQuant by 1.4 points, and TurboQuant is the weakest of the 2-bit methods. The decisive control experiment is cheap: combine Palu's low-rank truncation with KIVI or OTT at 2 bits and see whether it also passes 9x. The paper does not run it. The binning in Table 1 is also inconsistent, since SPECTRA at 4.57x is placed in the about 5x bin while Palu at 4.41x is placed in the about 4x bin. That placement lets SPECTRA avoid KIVI's 53.92 in the low-compression region.

Second, neither core component is validated on end-task accuracy, and the high-compression points depend on the calibration corpus. The difference between the G-weighted transform and a plain SVD, and the difference between importance-weighted and uniform allocation, are compared only on reconstruction error and attention-output error in Figure 1. The main text says the first difference shows up in end-task accuracy, but no LongBench result is given. Table 5 shows that on Llama at 11.09x, calibration on WikiText alone gives 47.90, while adding code gives 52.44, a gain of 4.54 points driven by LCC and RepoBench-P. In other words, a substantial part of the best-looking operating point comes from a calibration distribution that covers the evaluation distribution. The trick screen in Table 6 was also run on four LongBench test tasks; full test splits were used, but the reported benchmark is still what selects the configuration. The 11x and 12.2x results should therefore be read as in-domain results, not general ones.

Questions for authors
  1. Is the transform fitted separately for each KV head, or jointly over the concatenated 1024 dimensions? The text writes the key projection as d by d_kv with rank at most d_kv, yet the rank ceiling is swept up to 2048, which implies a joint fit over the full width. If the fit is joint, the query-folding step in Appendix F holds only when the up-projection is block-diagonal per head. Please state the actual implementation and the effective rank retained per head.
  2. At which step is RoPE applied? If post-RoPE keys are compressed, the energy ordering shifts with position and a single basis is no longer optimal. If pre-RoPE keys are compressed, RoPE must be applied after reconstruction, and the query-folding argument in Appendix F that avoids materializing full K no longer applies. Please give the exact procedure and its cost.
  3. Please add a matched-bit-rate, same-harness control: Palu low-rank combined with KIVI or OTT at 2 bits. If that combination also passes 9x and stays near dense, the gain here comes mainly from the transform rather than from unified allocation.
  4. Please add an end-task ablation: uniform bit allocation on the same latent basis, and a plain SVD in place of the G-weighted transform. Report the LongBench average and per-task scores.
  5. How much of the high-compression result depends on the calibration corpus? Please calibrate without any code, report on a long-context benchmark that took no part in the screen, and report variation across several calibration seeds. Do the 0.1 to 0.7 point differences at low compression fall inside that variation?
  6. The optimality of reverse water-filling assumes a high-rate Gaussian source, while the operating average rate is below 2 bits, bit-widths are shared by groups of 64 coordinates, and scales are per-token min and max. How much advantage does the allocation retain over a simple heuristic at this rate?
Reviewer_8pxfOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors proposed SPECTRA - training-free method for KV cache cache compression, based on rotation of KV cache into its spectral basis, and selecting a subset of most important channels. The SPECTRA projection is different from plain low-rank factorization, and done based on activation recovery for some calibration traces. Then KV storage budget is distributed only between these channels: more bits to high-importance channels, less bits or 0 to weak channels. Authors demonstrated that SPECTRA outperform compression methods based on uniform quantization.

Critical strengths

The paper proposed a good training-free method for KV cache compression. The number of well planned experiments demonstrate that it outperforms tradional qauntization based methods. The paper is well written

Critical weaknesses

None

Questions for authors

I would recommend to move overview of related work from Introduction to separate section "Related work"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.