acceptodds
Under review as a conference paper at ICLR 2027

CoFeatKV: Coalesced-Feature Low-Rank Compression for Post-RoPE KV Caches

Abstract

During long-context inference, the KV cache grows linearly with the sequence length and becomes a major source of memory usage and data movement. Existing low-rank key compression faces a trade-off: compressing pre-RoPE keys typically preserves a stronger low-rank structure, but decoding requires key reconstruction followed by RoPE; compressing post-RoPE keys avoids repeated positional rotation, but their weaker low-rank structure often leads to larger compression error. We propose CoFeatKV, a training-free method for post-RoPE key compression. Within each token page, CoFeatKV first groups key features with similar structure, builds a shared representation for each group, and then applies low-rank compression to the grouped representation. The compressed cache stores only the shared representatives, feature coefficients, and group assignments. During decoding, the query is mapped directly into the compressed group space to compute attention scores, avoiding both full reconstruction of historical keys and repeated RoPE on cached keys. CoFeatKV uses page-local updates to support autoregressive decoding while retaining all historical tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Feature Grouping, Low-Rank Approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eYqDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoFeatKV is a training-free method that compresses post-RoPE keys to shrink the KV cache in long-context inference. It slices keys into fixed-size pages along the sequence, compressing each page once it is full while keeping the active page uncompressed. Inside a sealed page it groups feature dimensions by attention importance, represents each group with one shared direction, and stores only a coefficient plus a group index per feature — much lighter than a dense projection matrix. At decode time it computes attention directly in this compressed space, so it never rebuilds keys or re-applies RoPE. On Qwen3-8B / WikiText-2 it saves up to ~45% of key storage and lifts decode throughput ~23–26%, with perplexity close to full KV cache.

Critical strengths
  1. The storage saving is structural rather than tuned, because each feature stores only one coefficient and a group index instead of a dense projection matrix, which cuts per-page projection cost by roughly an order of magnitude.
  2. Decoding gets faster instead of slower, since it needs no key reconstruction and no second RoPE pass, so the method does not trade speed away for the storage saving.
  3. It is training-free and streaming, computing grouping locally per sealed page with no retraining or global optimization, which makes it easy to drop into an existing inference pipeline.
Critical weaknesses
  1. RULER is already saturated, with both the method and full KV cache scoring near the ceiling. The evaluation would be more convincing with hard long-reasoning tasks such as AIME-style problems, which stress the cache far more than RULER or WikiText perplexity.
  2. There is no code release or reproducibility statement, the paper gives no repository link or supplementary material, so the results cannot be correctly verified or reproduced.
  3. Grouping is greedy and page-local with no global-optimization guarantee, and the calibration is frozen at sealing time, so it may mismatch queries that arrive later.
Questions for authors

The author stated that their benchmarks include MMLU and use Llama-2-7B. But I see no MMLU and Llama-2-7B experiments.

In your experiments, a page holds 32 tokens, so each group direction is a 32-dimensional vector. 32 dimensions can contain at most 32 independent directions. The paper asks for 64 to 96 group directions per page. That means most of those directions are just linear combinations of the first 32. Is this design reasonable?

I think this paper needs more efforts. A lot of things haven't been fully thought through.

1 comment

We sincerely thank the reviewer for the careful review and constructive comments, and we apologize for the late response due to the time needed to run the additional experiments.

Q1: RULER is close to saturation. Should long-reasoning tasks such as AIME be included?

Response: We have added a long-reasoning evaluation on AIME 2024/2025. We use Qwen3-8B with thinking enabled on 60 problems in total, with a maximum generation length of 32,768 tokens and an average output length of about 16K tokens.

MethodAIME 2024AIME 2025OverallK cache savingsKV cache savings
FullKV74.17%67.50%70.83%0%0%
CoFeatKV m=6475.83%60.83%68.33%45.31%22.66%
CoFeatKV m=8075.00%67.50%71.25%32.81%16.41%

The accuracy of m=80 is close to that of FullKV; m=64 decreases the overall accuracy by 2.50 percentage points, mainly on AIME 2025, reflecting the quality cost of stronger compression. These results show that CoFeatKV performs well in long-context settings beyond RULER: with outputs averaging about 16K tokens, m=80 matches FullKV (71.25% vs. 70.83%).

Q2: The paper does not provide code, a repository link, or a reproducibility statement. How can the results be verified?

Response: We thank the reviewer for pointing this out. Our code is publicly available via an anonymous link: https://anonymous.4open.science/r/cofeatsvd-5155/.

Q3: The grouping is greedy and page-local, and it is frozen after sealing. Could it mismatch later queries?

Response: Paging, greedy assignment, and freezing after sealing are design choices made to control the cost of streaming inference; we do not claim global optimality. Once the grouping is determined, the shared direction of each group is the optimal solution to the weighted rank-one approximation problem for that group.

Regarding query mismatch, CoFeatKV can operate completely independently of queries. Denoting the query weights in the paper as wjw_j, setting wj←1w_j \leftarrow 1 for all jj yields the Key-only variant: grouping and compression use only the keys themselves without any historical query information, so there is no mismatch with later queries. On Qwen3-8B at 32K with m=64, the Key-only variant achieves 88.00 on FWE, comparable to 87.67 for the Q-weighted version; on multi-value NIAH, both achieve 99.17. This shows that the effectiveness of the method does not rely on query calibration.

Q4: The paper mentions MMLU and Llama-2-7B. Why are there no corresponding results?

Response: We thank the reviewer for pointing this out. After checking, the MMLU experiment was actually conducted with Llama-3.1-8B-Instruct; the description of Llama-2-7B in the paper has no corresponding experimental results, which is our mistake. The MMLU experiment covers 57 subjects and 14,042 questions in the 0-shot setting. The results are as follows:

MethodMMLU accuracy (%)
FullKV68.44
PaLu55.79
OjaKV64.98
ReCalKV58.81
CoFeatKV (m=64)67.24
CoFeatKV (m=80)67.94

CoFeatKV m=80 is only 0.50 percentage points below FullKV, and m=64 is 1.20 percentage points below, both higher than the other compression baselines. In the revised version, we will add these results, remove the descriptions related to Llama-2-7B, and unify the model and task lists in the abstract, experimental setup, and result tables.

Q5: With a page length of only 32, why are 64–96 grouping directions needed? Does the linear dependence among directions indicate an unreasonable design?

Response: The observation about the rank is correct: a 32-dimensional space has at most 32 linearly independent directions. However, the number of groups mm denotes the number of shared representative directions, not the rank of a matrix. In CoFeatKV, each feature can select only one representative direction and multiply it by one coefficient. Under this constraint, a direction may still be necessary even if it is a linear combination of other directions. For example, in a two-dimensional space, "northeast" can be obtained by combining "east" and "north," but it cannot be represented by scaling "east" or "north" alone.

Therefore, using more than 32 representative directions reduces the error of this "single direction + single coefficient" representation and does not require these directions to be linearly independent. If 32 basis vectors were used instead, each feature would typically require multiple combination coefficients, losing the current compact storage structure. Taking P=32P=32, d=128d=128, and m=80m=80 as an example, the representative vectors, coefficients, and group indices take 5,504 bytes in total, 32.8% less than the 8,192 bytes of the original BF16 K. Therefore, the error and efficiency should be compared under the same byte budget, rather than judging the design solely by m>Pm>P.

Reviewer_QGDfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoFeatKV, a training-free low-rank compression method for post-RoPE keys. Within each completed post-RoPE key page, CoFeatKV selects the top-m feature dimensions by attention importance as group representatives, assigns the remaining features to the most similar groups, and stores only one scalar coefficient and one group index per feature instead of a dense d×m projection matrix. During decoding, queries use the same sparse feature-to-group mapping to compute attention directly in the compressed space, without reconstructing historical keys or reapplying RoPE.

Critical strengths
  1. Clear problem formulation and targeted design. The paper correctly identifies the weakened low-rank structure of post-RoPE keys as a key difficulty and proposes attention-aware feature grouping to mitigate it. The motivation and method are well aligned.

  2. Compact representation and efficient decoding. Replacing dense projection matrices with one coefficient and one group index per feature reduces per-page projection storage. Decoding via (qA)C^T avoids key reconstruction and repeated RoPE, which is elegant and practical.

  3. Broad experimental coverage with promising results. The evaluation covers 7B/8B/32B models, 16K/32K contexts, and multiple benchmarks. CoFeatKV is close to or occasionally above FullKV on RULER, preserves 100% NIAH accuracy on 32B, and reports storage and throughput benefits

Critical weaknesses
  1. Page-local greedy grouping is not globally optimal. After selecting top-m representatives, remaining features are assigned by pairwise cost; this does not solve the combinatorial clustering problem, and the pairwise merge cost is not the final residual of a multi-feature group. Grouping quality may vary across layers, heads, and page sizes.

  2. Feature importance depends on query energy distribution. The importance score relies on estimated query weights. If future queries shift from the estimation distribution, anchor selection may become suboptimal, and the error bound does not guarantee task accuracy.

  3. Sensitivity to hyperparameters and page design. The method depends on group budget m, page size, and weight estimation. Small m saves more but increases error, while large m improves quality but reduces compression; task-level variation already appears on CWE, FWE, and QA-1.

  4. Actual memory and throughput benefits are insufficiently measured. The K saving in Table 5 is for the compact storage format, not measured allocation savings in the archived timing runs. Experiments use a single RTX 3090 with batch size 1 and lack multi-batch, long-prefill, serving-scenario, and real-kernel comparisons.

  5. Compression scope and experimental comparison are limited. The method mainly compresses keys while the value cache remains dense, so overall KV-cache benefits require joint consideration. LongBench covers only four QA tasks, 32B is compared only with FullKV, and matched-byte comparisons with same-scale low-rank baselines are missing.

Questions for authors

Under the conditions of "strict matched-byte (including all metadata), real GPU memory allocation, and end-to-end decoding latency", will CoFeatKV's advantages in quality, storage, and throughput still hold?

2 comments

We sincerely thank the reviewer for the careful review and constructive comments, and we apologize for the late response due to the time needed to run the additional experiments.

Q1: The page-local greedy grouping has no global optimality guarantee, and the pairwise cost is not equal to the final group residual.

Response: We agree with this point. The pairwise cost is used only to guide the assignment and is not an exact surrogate for the final residual of a multi-feature group. Once the grouping is determined, we re-solve the weighted rank-one approximation for the entire group, obtaining the optimal shared direction under that grouping. Page-local processing and greedy assignment are used to control the cost of online compression, and we do not claim that they achieve globally optimal clustering. The ablations on the number of groups and the page length in Q3 show the impact of this approximation on downstream tasks.

Q2: The importance weights rely on historical queries. Are they still reliable when future queries change? Can the error bound guarantee task accuracy?

Response: The error bound constrains the approximation error of the attention scores; it cannot directly guarantee task accuracy, nor can it guarantee small errors under arbitrary shifts in the query distribution. Freezing the representation after sealing avoids repeated compression, but it does carry a risk of mismatch with future queries, and we will list this as a limitation in the revised version.

To examine how much the method relies on query calibration, we added a Key-only ablation that does not use query energy estimation (Qwen3-8B, 32K, m=64): on the same 100 samples, FWE is 88.00, compared with 87.67 for the Q-weighted version; on 90 paired multi-value NIAH samples, both achieve 99.17. This indicates that our method has limited reliance on query calibration.

Q3: Is the method sensitive to the number of groups, the page length, and the weight estimation?

Response: There is a clear quality–storage trade-off. The ablation results on the number of groups and the page length with Qwen3-8B, 32K, and Q-weighting are as follows:

Page length PNumber of groups mK savingsCWEFWEQA-1
326445.31%76.3087.6774.00
328032.81%79.2089.6775.00
329620.31%81.2090.3375.00
166440.62%77.7089.0076.00

With the page length fixed at 32, increasing the number of groups from 64 to 96 improves all three tasks, while the K cache savings decrease from 45.31% to 20.31%. With m=64 fixed, reducing the page length from 32 to 16 improves CWE/FWE/QA-1 by 1.40/1.33/2.00 points, while the K cache savings decrease from 45.31% to 40.62%. Therefore, both increasing the number of groups and reducing the page length improve quality. The impact of weight estimation is shown by the Key-only ablation in Q2.

Q4: Can the logical byte savings translate into actual GPU memory and throughput gains? Are the experiments limited to the RTX 3090 with batch size 1?

Response: We additionally measured throughput, per-step decoding latency, and KV cache usage under different concurrency levels on the A6000 (Qwen3-8B, 16K input, m=64).

ConcurrencyMethodThroughput (tok/s)P95 latency (ms)P99 latency (ms)Logical KV usage (GiB)
1FullKV26.3938.7538.912.320
1CoFeatKV30.0637.3472.221.795
2FullKV40.1251.0351.294.641
2CoFeatKV53.4543.7456.723.589
4FullKV54.3574.9375.109.281
4CoFeatKV83.4752.5983.407.178

CoFeatKV reduces KV usage by about 22.7%, and the gains increase with concurrency: at concurrency 1/2/4, throughput improves by 13.91%/33.22%/53.58%, and P95 latency decreases by 3.64%/14.29%/29.81%. The P99 latency is higher than that of FullKV because page-sealing compression is performed once every 32 decoding steps (about 3% of the steps); this overhead is reflected only in P99 and does not affect P95.

Q5: V is not compressed; LongBench includes only four QA tasks; and the 32B model lacks low-rank baselines.

Response: CoFeatKV compresses only K, and we will clearly distinguish K savings from full KV savings in the revised version. With P=32P=32, d=128d=128, BF16, and uncompressed V, the K savings of m=64/80 are 45.31%/32.81%, corresponding to full KV savings of 22.66%/16.41%.

We added classification, code, and summarization tasks on Qwen3-8B/32B, using the same first 50 samples for each task, and added the low-rank baseline PaLu on 32B. The results on Qwen3-32B are as follows:

TaskFullKVCoFeatKV m=80PaLu
TREC (classification)74.0074.0074.00
RepoBench-P (code)16.2217.5617.00
QMSum (summarization)23.2224.1124.00
GovReport (summarization)33.3933.7133.41

As shown in the table, CoFeatKV is no lower than FullKV and PaLu on all four tasks.

Q6: Does the advantage still hold when strict byte matching, actual GPU memory, and end-to-end latency are all satisfied simultaneously?

Response: Our existing experiments verify each of these three conditions, although under different settings:

  • Byte matching: On Qwen3-32B at 32K, the KV cache and metadata of CoFeatKV with m=80 take 7,180,648,448 bytes in total, compared with 7,179,534,336 bytes for PaLu, a difference of about 0.016%. On the same first 50 RULER samples, CoFeatKV is no lower than PaLu on all four tasks, with CWE/FWE of 54.80/90.00 compared with 45.00/62.00 for PaLu.
  • Actual GPU memory: On the A6000 with Qwen3-8B, m=64, and 32K input, the peak allocated memory, including the model, cache, and temporary tensors, decreases from 21.532 GiB to 20.952 GiB, and the peak reserved memory decreases from 22.820 GiB to 21.721 GiB; at 16K, the reserved memory increases slightly (18.617→18.758 GiB).
  • End-to-end latency: Under the same configuration with batch size 1 and 512 decoding steps, the total time including prefill, compression, and decoding decreases from 34.44 s to 28.05 s (18.57%) at 32K on the A6000; across the four 16K/32K settings on the RTX 3090 and A6000, the total time decreases by 8.01%–18.57%.
Reviewer_uAPSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoFeatKV, a training-free approach for post-RoPE Key-cache compression in LLM inference. The authors address the trade-off between compression quality and decoding efficiency: pre-RoPE compression preserves stronger low-rank structures but introduces reconstruction overhead, while post-RoPE compression enables efficient decoding at the cost of larger approximation errors.

Critical strengths
  1. The paper identifies feature-level redundancy in post-RoPE Keys and introduces an attention-aware grouping strategy with structured low-rank representations. The mathematical formulation is well-developed, including the derivation of the optimal pairwise merging cost, the optimal shared direction within each group, and attention error bounds. The sparse feature-to-group mapping reduces projection storage overhead while enabling reconstruction-free attention, offering a practical approach to balancing compression quality and decoding efficiency.
Critical weaknesses
  1. CoFeatKV introduces additional computation during page compression, including feature importance estimation, grouping, and eigenvector computation for each group. These operations may incur substantial overhead across multiple layers, KV heads, and pages. However, the paper does not provide a detailed breakdown of compression latency, prefill overhead, or end-to-end inference latency. It remains unclear whether the decoding speedups can offset these additional costs, particularly under realistic long-context and continuous-batching workloads.

  2. Although the paper compares against several low-rank compression methods, the evaluation does not consistently compare methods under matched actual memory budgets, including projection and metadata overhead. This makes it difficult to fully assess whether the improvements result from the proposed representation or differences in compression configurations.

Questions for authors
  1. Could the authors quantify the overhead of feature grouping, eigenvector computation, and page compression? It remains unclear whether the reported decoding speedups offset these additional costs in end-to-end inference, particularly under long-context and high-concurrency workloads.

  2. Could the authors clarify whether the baseline methods are evaluated under comparable actual memory budgets, including projection matrices and metadata overhead? A comparison at matched storage costs, particularly against page-wise SVD methods such as PuzzleKV, would help establish whether the improvements primarily stem from the proposed structured representation.

1 comment

We sincerely thank the reviewer for the careful review and constructive comments, and we apologize for the late response due to the time needed to run the additional experiments.

Q1: Can the overhead of feature grouping, direction computation, and page compression be offset by the decoding gains? How does the method perform with long contexts and high concurrency?

Response: Under the single-request long-context workloads we measured, the compression overhead is offset by the decoding gains, yielding end-to-end speedups. The test configuration is Qwen3-8B in BF16, Key-only, m=64, page length 32, batch size 1, and 512 decoding steps, with one warm-up run followed by three measured runs. The timing includes cache allocation, prefill, compression, and online page sealing during decoding, and excludes model loading, tokenization, one-time weight packing, and warm-up. FullKV uses the native SDPA and dynamic cache in Transformers.

GPUInput lengthFullKV total time (s)CoFeatKV total time (s)Total time reduction
RTX 309016K21.9320.178.01%
RTX 309032K35.1830.7612.56%
A600016K22.78419.78813.15%
A600032K34.44028.04618.57%

On the RTX 3090, the preparation stage takes 0.66/1.03 s longer than FullKV at 16K/32K, while the decoding stage saves 2.41/5.45 s; on the A6000, the preparation stage takes only about 0.25/0.40 s longer. All of these overheads are included in the total time.

A profiled run on the A6000 gives the time of each stage:

Input lengthGrouping (ms)Shared direction computation (ms)Overall page compression (ms)
16K9586991,994
32K2,1911,3045,684

"Overall page compression" already includes grouping and shared direction computation. These CUDA event spans include scheduling waits, and compression overlaps with model computation, so they do not equal the additional end-to-end time.

In summary, the compression overhead is offset by the decoding gains in all measured settings, and the gains grow with context length: from 16K to 32K, the total time reduction increases from 8.01% to 12.56% on the RTX 3090 and from 13.15% to 18.57% on the A6000. This is because the decoding savings grow faster than the compression overhead: on the RTX 3090, the former increases from 2.41 s to 5.45 s, while the latter increases only from 0.66 s to 1.03 s.

Q2: Does the storage accounting include projections and metadata? Is the comparison with methods such as PuzzleKV aligned on the actual storage budget?

Response: We account for storage based on the complete persistent representation and have added baseline comparisons under nearly identical budgets. With BF16, page length PP, and feature dimension dd, the K storage of CoFeatKV per page per head is 2Pm+2d+d2Pm+2d+d bytes, corresponding to the representative vectors, coefficients, and uint8 group indices, respectively; the two BF16 factors of per-page SVD take 2r(P+d)2r(P+d) bytes. When P=32P=32 and d=128d=128, CoFeatKV with m=64 and per-page SVD with r=14 both take 4,480 bytes; therefore, the two methods should not be compared at the same value of mm and rr.

For each of the four RULER tasks, we evaluate Qwen3-8B on the same first 50 samples:

TaskFullKVCoFeatKVCoFeatKV + V INT8PaLuPuzzleKV
KV savings0%22.65%46.08%30%40%
VT100.00100.00100.0099.2099.76
CWE43.2054.8078.2045.0062.80
FWE90.6790.0087.3362.0095.93
QA-176.0078.0076.0076.0054.18

CoFeatKV can be combined with INT8 quantization of V. With this combination, the KV savings reach 46.08%, higher than those of PaLu (30%) and PuzzleKV (40%). Even at this higher compression ratio, the scores on VT, CWE, and QA-1 are no lower than those of either method; in particular, CWE reaches 78.20, compared with 45.00 for PaLu and 62.80 for PuzzleKV. Since the source code of PuzzleKV is not publicly available, its results are taken from the original paper; its evaluation setting is not fully identical to ours, so these results are for reference only.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.