acceptodds
Under review as a conference paper at ICLR 2027

Architecture-Conditioned KV Compressibility

Abstract

KV-cache compression is usually evaluated as a deployment method, but a prior question is often left implicit: which native cache representations contain structure worth compressing? We study KV compressibility as a conditional property of representation, architecture, context length, domain, and tensor folding. The main audited screen contains nine completed instruction-tuned models from Qwen2.5, Llama, and Mistral, spanning 0.5–22.2B parameters, four domains, four contexts from 512 to 32K, and three seeds: 432 valid cells and 1,296 representation rows. A separate exact-length matrix contains 420 cells from three models, four domains, seven contexts from 512 to 32K, and five seeds. Three results emerge. First, representation ordering is universal in both protocols: pre-RoPE keys beat post-RoPE keys, which beat values, in 432/432 and 420/420 paired cells. Second, quantized tensor-train (QTT) compatibility is architecture-conditioned and non-monotone: Qwen2.5-7B reaches a 7.32× median pre-RoPE-key QTT ratio in the main screen, while neighboring Qwen2.5-3B and 14B are near 1×; the protocol-separated Qwen2.5-1.5B exact-context matrix reaches 26.82×. Third, long context amplifies representation mismatch: from 512 to 32K, rank-16 error rises by 0.0246 for pre-RoPE K, 0.1448 for post-RoPE K, and 0.0819 for V. The paper provides a structural screening prior, not a claim of resident-memory or latency improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, Efficient transformer inference, Long-context language models, Tensor-train decomposition, Architecture-conditioned compression, Attention geometry, Tensor trains

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NTaKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is a measurement study rather than a new compression method. It uses low-rank and QTT diagnostics to compare how compressible pre-RoPE keys, post-RoPE keys, and values are across nine open models at contexts up to 32K, with a separate exact-length study on three models. The main findings are that pre-RoPE keys always have the lowest fixed-rank error and values the highest, that QTT compressibility varies sharply across models in a way that model size alone does not explain, and that longer contexts raise the fixed-rank error most for post-RoPE keys. The paper presents these results as a screening prior, includes a systems pilot and a factor-native decoder pilot, and does not claim quality, resident-memory, or latency gains.

Critical strengths

The study is large (852 cells) and statistically careful, and the ordering of rank-16 error across representations holds in every cell. The paper is also careful about the scope of its claims and reports negative systems results. This makes the measurements credible as descriptive statistics.

Critical weaknesses

The novelty of the main findings is limited. Finding 1, that pre-RoPE K has lower fixed-rank error than post-RoPE K and V, has already been reported and used in method design by ShadowKV (Sun et al., ICML 2025). ShadowKV states that pre-RoPE keys are far more low-rank than post-RoPE keys and values, and that values are not low-rank, and it therefore stores low-rank pre-RoPE keys. Loki (Singhania et al.) also reports lower rank for pre-rotary than post-rotary keys. Neither work is cited. Finding 2 is the central result referred to in the title, but it only establishes that QTT ratios differ across models, and the authors themselves describe it as an observational result without a mechanistic explanation. Moreover, the QTT ratio is computed on a fixed 128-token probe and stays constant from 512 to 32K tokens in Figure 6, so it does not reflect the long-context caches the paper studies.

There is no validation linking the structural metrics to actual compression outcomes. The stated goal is a screening prior for choosing compression methods, but Sec. 5 explicitly says the selector has not been validated, and the quality grid in the supplementary material has only 2,449 of its 5,175 cells completed. The systems results do not show a usable gain either. The Sec. 6 pilot still materializes the full K/V, so the resident cache does not shrink. The rank-16 factor-native decoder does reduce persistent bytes, but it has a next-token KL of 3.89 and 5.60 at 512 and 2048 tokens and top-1 agreement of only 0.270 and 0.120, and it supports only the post-RoPE path. The paper therefore neither shows that these measurements help in choosing a compression method nor demonstrates a compression gain at acceptable quality.

Questions for authors
  1. On which matrix are the rank-16 error and rank95 computed? Are they computed per head or over all KV heads of a layer concatenated, and how are layers aggregated? Since head dimensions and numbers of KV heads differ across models, can e16 be compared directly across models?
  2. Which tokens does the 128-token probe use? Since the QTT ratio in Figure 6 does not change from 512 to 32K, how do the QTT results represent longer caches?
  3. The paper never says which four domains or which datasets are used (only retrieval is mentioned), and M0–M6 are never defined. What are M0–M6, and which three models and context lengths does that pilot use? Supplementary Figure 3 only calls it a 3-model pilot.
  4. Could you validate the screening prior with an actual compression method? For example, you could apply the same compression to pre-RoPE K on models with high and low QTT ratios and compare generation quality at the same memory budget.
4 comments
AuthorsOpenReview ↗

Corrected QTT protocol and downstream validation

Thank you for the detailed review. We agree with three points and have changed the paper accordingly.

For completeness, we also clarified the protocol details you asked about. For each layer, rank-16 error is computed after reshaping K to a token-by-feature matrix T × (H_KV d_head), with KV heads concatenated; rank95 is computed per layer and aggregated over layers and cells as specified in the revised protocol. Because head dimensions and KV-head counts differ across models, we do not interpret rank-16 error as a directly comparable cross-model compression budget; cross-model claims use measured factor ratios and explicit size controls.

The original QTT probe uses the first 128 tokens of each stored prefix. That is why its ratio was repeated across longer context labels: it was a short structural diagnostic, not a measurement of the full longer cache. We now state this limitation explicitly and replace it for new claims with assembled full-length reconstructions at 64–8192 tokens under a measured global K-error ceiling.

The four structural domains are fiction, code, scientific text, and retrieval. The protocol-separated three-model pilot used Qwen2.5-1.5B, Llama-3.1-8B, and Mistral-7B at 512, 2048, and 8192 tokens with five seeds; we keep it separate from the newer main matrices. The later M0–M6 systems study defines M0 as the full-cache baseline, M1–M3 as incremental SVD variants, M4 as an offline paired-layer basis, and M5 and M6 as per-layer and cross-layer TT-ICE variants. All M0–M6 paths materialize full K/V, so this is a quality and latency boundary study rather than a resident-memory result.

First, the original 128-token QTT statistic did not certify assembled TT compression at 5% total error. In a published Qwen2.5-7B code cell, the stored ranks gave an estimated 7.16x factor ratio but 8.10% measured reconstruction error; a conservative assembled TT-SVD gave 1.94x at 4.40% error. We now state this correction explicitly and replace the old headline interpretation with full-length measured-error reconstructions. Under the corrected protocol, the 23-book matrix gives 1.62x for Qwen2.5-3B and 10.74x for Qwen2.5-7B at 8192 tokens, with measured global K error at most 5%. The phenomenon survives, but the quantitative claim is now narrower and properly certified.

Second, we no longer present the pre-RoPE versus post-RoPE ordering as a new observation by itself. We cite and discuss SALS, ShadowKV, and Loki, and treat the ordering as a controlled replication that establishes the reliability of our measurement protocol. The new contribution is the model- and context-conditioned non-monotonicity, the folding and matched-shape controls, and the explicit separation between structural, quality, and serving evidence.

Third, we ran the requested downstream validation with an actual compression method. The refined 20-setting TT grid covers 10 models, 8 context lengths, and 8 source-backed retrieval documents per model and length, for 12,800 responses and 1,600 aggregates. At 8192 tokens and nominal tolerance 0.05, Qwen2.5-7B retains 8/8 conditional retrieval accuracy at a median K ratio of 10.05x, while Qwen2.5-1.5B retains 3/8 at 30.45x. At 2048 tokens the corresponding values are 8/8 at 5.60x for 7B and 1/8 at 18.63x for 1.5B. The incomplete earlier F3 grid is not used as the main quality claim; the refined grid is the completed downstream validation.

We also ran the requested matched-budget quality comparison. On four held-out Qwen models, TT and a per-layer SVD control were given approximately 20% of dense K factor bytes, with exact V and 64 teacher-forced continuation tokens (32 cells in total). TT has lower mean next-token KL than SVD for all four models; for Qwen2.5-7B the means are 0.193 and 0.244, respectively. This is a matched logical K-budget quality comparison with a materialized cache, not a resident-memory comparison. Together, the equal-budget audit and the refined frontier show that structural measurements can guide which budgets and representations to test, but they do not by themselves guarantee quality or identify a universally best compressor.

These results show that structural compressibility is useful as a screening variable, but not as a universal quality predictor or compressor selector. The quality grid is conditional on a short-code retrieval task, exact V, and materialized K. It does not establish general-generation quality, resident-memory savings, latency, or throughput. The paper now defines these boundaries explicitly and reports the factor-native experiment separately because its persistent-byte reduction did not yet meet the quality gate.

Thanks for the response. Replacing the 128-token probe with full-length reconstructions and filling in the missing setup details and citations fixes the technical issues I raised, and I appreciate the honesty there. Even so, I am keeping my score at 1. The new downstream results actually make me more skeptical of the main takeaway: Qwen2.5-1.5B has the best structural ratio by far (30.45x), yet its retrieval accuracy drops to 3/8. If a high ratio can still fail that badly, the structural screen doesn't really avoid running the downstream evals anyway. With Finding 1 now framed as a replication, the paper leans heavily on the 3B-vs-7B gap, where I still don't know why it happens. One more thing—you mention in the reply that rank-16 error isn't directly comparable across models; doesn't that undercut the scaling trend in Figure 5 and Supplementary Table 5?

Thank you for the follow-up. We agree that the Qwen2.5-1.5B result shows that a large structural factor ratio is not, by itself, a quality predictor and cannot replace downstream evaluation. Our intended use of the structural map is as a first-stage screen: it identifies representation–model–length regimes with structural headroom, while quality and systems performance remain separate gates. The matched-budget TT/SVD audit is closer to a method-selection test; at approximately 20% of dense K factor bytes, TT has lower mean next-token KL than SVD on all four held-out Qwen models.

On rank-16 error, we should have distinguished a fixed algebraic-rank comparison from an equal-memory comparison. Relative e16e_{16} is comparable across models as the error obtained under the same rank-1616 constraint, so Figure 5 and Supplementary Table 5 remain valid as descriptive fixed-rank trends. However, because feature widths and KV-head layouts differ, rank 16 corresponds to different factor-byte budgets across architectures. We will therefore revise the wording so these results are not interpreted as equal-memory compression scaling; cross-model compression-efficiency claims will rely on measured factor ratios and matched-byte controls.

Thank you for the review and for the follow-up.

We also received an OpenReview email notification stating that, after our response, you intended to raise the score to 2 because the main technical issues had been addressed. Since the visible comment currently says that the score remains 1, we just wanted to check whether this was an intentional revision or whether the score update may not have been recorded.

Reviewer_SKurOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how KV-cache compressibility changes across models and context lengths. The authors find that pre-RoPE keys are easier to compress than post-RoPE keys and values. They also show that QTT compression varies across models, and longer contexts make compression harder, especially for post-RoPE keys.

Critical strengths

The paper provides a fairly broad evaluation across different models and context lengths. The consistent results for pre-RoPE K, post-RoPE K, and V make the findings more convincing.

Critical weaknesses
  1. The QTT results are interesting, but it is still unclear why the compression ratios vary so much across models. Could this be due to tensor shapes, folding choices, or QTT settings rather than the architectures themselves? The authors need to look into these factors.

  2. The pre-RoPE vs. post-RoPE finding is not really new. SALS (NeurIPS 2025) already discussed how RoPE affects the rank and compressibility of keys. Also, while the factor-native implementation is interesting, the paper does not show clear benefits in generation quality, memory usage, or latency.

Questions for authors
  1. Why does QTT compression vary so much across Qwen models? Have the authors checked whether tensor shapes, folding choices, or QTT settings could explain these differences?

  2. Can the authors show whether the structural compression results actually predict generation quality or help select a better compression method? Right now, it is unclear how useful these measurements are in practice.

3 comments
AuthorsOpenReview ↗

Shape and folding controls, plus the quality frontier

Thank you for asking whether the Qwen differences could be explained by tensor shape, folding, or QTT settings. We tested these possibilities directly. In a prespecified five-folding audit at 2048 tokens (five models, 23 books, 575 model, book, and folding cells), the fixed-folding median ratios are 1.35x for Qwen2.5-3B and 6.59x for Qwen2.5-7B. The best median among the five tested foldings is 1.48x for 3B and 6.84x for 7B. A matched-shape control keeps the 7B ratio above the 3B ratio in all 24 paired subset cells. Folding changes the magnitude, but neither the tested folding choices nor the shape control removes the contrast. We therefore make no causal architecture-only claim and do not present one folding as universally optimal.

We also credit SALS for the prior observation that RoPE changes the rank and compressibility of keys. Our additional contribution is the broader controlled map across representation, model, and context length, together with reconstruction, quality, and systems checks. We tested a mean/residual explanation for the 3B–7B contrast. At 2048 tokens, the median token-mean energy fractions are 0.694 for 3B and 0.676 for 7B, so mean energy alone does not explain the result. Residual spectra, head composition, TT cuts, and model shape remain observationally entangled.

We then tested whether the structural measurements are useful for downstream quality. The refined grid evaluates 20 TT settings on 10 models, 8 context lengths, and 8 source-backed retrieval documents per model and length: 12,800 task-level responses and 1,600 aggregates. At 8192 tokens and nominal TT tolerance 0.05, Qwen2.5-0.5B, 3B, and 7B retain 8/8 conditional retrieval accuracy at median K ratios of 9.56x, 1.60x, and 10.05x, while Qwen2.5-1.5B retains only 3/8 at 30.45x. At 2048 tokens, 1.5B retains 1/8 at 18.63x, whereas 7B retains 8/8 at 5.60x. The 1.5B result is especially informative: the largest structural ratio does not imply the best quality.

Thus the measurements provide a model- and length-conditioned compression–quality frontier for budget calibration and risk screening. They are not a scalar quality predictor and do not select a compressor universally. The grid measures conditional accuracy on a short-code retrieval task with exact V and a materialized K attention path. It is therefore quality evidence, not a general-generation, resident-memory, or latency result.

Thanks for the response. The folding and matched-shape controls resolve my Q1. However, Finding 1 is an acknowledged replication of the pre/post-RoPE ordering, and Finding 2 cannot be attributed to a mechanism. I don’t think the contribution is sufficient, so I maintain my rating.

Thank you for clarifying. We agree that the pre/post-RoPE ordering is a replication and that our controls do not establish a causal architectural mechanism for the 3B–7B contrast. The revised paper does not present either of these as its sole contribution. It reports an audited full-length measurement of model- and representation-conditioned KV approximability, with measured-error reconstruction, folding and matched-shape controls, and downstream quality validation.

The magnitude of the effect is itself significant for a measurement study: under the same refined protocol at 8192 tokens, the median K ratio is 1.60× for Qwen2.5-3B and 30.45× for Qwen2.5-1.5B, while conditional retrieval accuracy is 8/8 and 3/8, respectively. This is a large, reproducible and quality-relevant model difference, not a small measurement fluctuation.

Establishing its causal mechanism is an important follow-up, but it is not required for the empirical contribution we claim here: documenting and quantifying this variation under controlled protocols. The degree of compressibility and the stability of the results have been studied, and they clearly indicate the potential for future, more theoretical research, but they do not guarantee the presence of direct interpretability.

We hope the revised scope makes that contribution clear, even if it does not change your rating.

Reviewer_2u5sOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the compressibility of transformer key and value activations varies with architecture, layer, token position, domain, and context length. It evaluates nine instruction-tuned models from three model families using paired low-rank diagnostics, and reports a separate exact-length protocol as a robustness check. Across both protocols, pre-RoPE keys are more compressible than post-RoPE keys, which in turn are more compressible than values. The paper also finds that tensor-train compressibility varies non-monotonically across model sizes and that long contexts particularly affect post-RoPE keys.

Critical strengths

1.I find this paper valuable. Although it does not propose a new method, it provides an in-depth validation and synthesis of the current literature and offers useful guidance for future research and application in this area. In an era where many works claim SOTA after marginal improvements on public benchmarks, such a study is particularly meaningful.

2.The paper is well structured and clearly written.

Critical weaknesses

1.The striking QTT ratios are measured on a fixed 128-token sequence probe. The paper does not establish whether the reported non-monotonic size effects persist across probe lengths or across substantially different token sequences, which limits the generality of the QTT-based conclusions.

2.The paper uses the low-rank approximation error and factor-size ratio under a fixed rank-16 / fixed 5% QTT target as a proxy for “compressibility.” However, Frobenius error is not attention-sensitive, the QTT ratio is not a resident-memory ratio, and the ranking may depend on the chosen rank or threshold budget. Moreover, the paper does not verify whether this structural ranking translates into downstream quality, memory, or latency benefits. Consequently, the current evidence supports representation ranking primarily at the structural screening level. I recommend explicitly using “structural compressibility” or “low-rank approximability” and annotating the 128-token probe nature when first reporting QTT ratios, so that readers do not misinterpret “pre-RoPE K is easier to compress” as a deployment benefit.

Questions for authors

na

1 comment
AuthorsOpenReview ↗

Corrected full-length measurements and deployment scope

Thank you for asking us to separate structural approximability from deployment benefits. We agree that our original 128-token QTT probe was too easy to overinterpret: its ratios were estimated factor sizes from independently selected unfolding ranks, not assembled full-cache compression certified at 5% total error. We now describe this as a short structural diagnostic and use measured-error reconstructions for new claims.

We replaced the single-probe analysis with an assembled TT-SVD audit on full-length pre-RoPE K tensors at 64, 128, 256, 512, 1024, 2048, 4096, and 8192 tokens. The protocol covers 23 books and five models and keeps the measured global K reconstruction error at or below 5%. For Qwen2.5-7B, the median K factor-size ratio is 3.14x at 128 tokens and 10.74x at 8192; for Qwen2.5-3B it is 1.02x and 1.62x. Part of the increase is expected because dense storage grows linearly with context while the selected TT factor size grows more slowly. We therefore interpret this as persistence of the structural effect beyond the original probe, not as evidence that the representation becomes intrinsically easier to approximate. The fixed-rank error analysis still shows increasing error with context length.

We also clarified the terminology throughout the paper. We use structural compressibility or low-rank approximability for the offline tensor measurements, and reserve resident-memory or serving claims for the separate systems audit. The ordinary TT attention path materializes full K/V, so these measurements do not establish resident-memory, latency, or throughput gains. They identify model, representation, and context combinations that are promising enough to justify a downstream compression experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.