acceptodds
Under review as a conference paper at ICLR 2027

VFold: Symmetry-Aware Cross-Layer Value Cache Compression

Abstract

While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However, existing techniques necessitate architectural changes to the transformer and incur substantial overhead. In this work, we propose a symmetry-aware value cache merging strategy that reduces cache memory while avoiding both harmful performance degradation and architectural overhead during decoding. Furthermore, we show that this redundancy can be exploited alongside existing cache compression techniques, seamlessly composing with high-ratio quantization and key cache pruning methods. Ultimately, our findings reveal a major source of underutilized capacity in the value cache, opening a simple yet highly effective avenue for scaling context windows under strict memory constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, folding, merging, transformer, compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QxHfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VFold, a cross-layer value-cache compression method for pre-trained LLMs. The motivation is that adjacent transformer layers have value vectors with high CKA similarity (but near-zero cosine similarity), suggesting they live in the same rotated basis and can be aligned and merged. It computes a CCA-based linear alignment map between adjacent layers' value caches, and fold it into the attention weights, then average the two layers' value caches into a shared inter-layer cache. They use Llama-3.1-8B-Instruct, Mistral-Small-3.1-24B-Instruct, and Qwen3-8B for evaluation on RULER and LongBench.

Critical strengths
  1. The idea of value cache compression is well motivated by the asymmetry of KV, where RoPE is applied on key states but not value.
  2. The weight-folding reparameterization is clean and practical.
  3. Experimental results show the effectiveness and efficiency of the proposed method.
Critical weaknesses
  1. The paper overstates the advantage of VFold in the main body only with a compression ratio of 25%. The performance drops as the compression ratio increases to 37.5%.
  2. The evaluation is conducted on RULER and LongBench. More challenging benchmarks like InfiniteBench and LongReason are recommended.
  3. The intuition of using CCA is not clear enough. Since no RoPE is applied on value states, could CCA be replaced by other mapping functions?
Questions for authors
  1. What do you mean in line 152: "For CKA similarities, we simply swap out the metric given the same setup for measuring cosine similarity"?
  2. How does VFold perform under larger compression ratios?
  3. Why CCA vs. SVD? CommonKV uses joint SVD across layers to find a shared subspace. VFold uses CCA + Hungarian matching. Have you directly compared these two alignment strategies on the same models and benchmarks?
4 comments

We thank the reviewer for their time, and for acknowledging the motivation, practicality, and effectiveness of the proposed method. We address reported weaknesses and questions as follows:

W1/Q2 overstated advantage, size of compression ratios: We respectfully note that all main body claims are scoped to the 25% setting (and carefully rechecked, §1, §6.1). The main body also contains higher compression ratios like 50% (with ThinK), and 4.27x compression (with KIVI). It is true that performance drops as we increase the compression ratio of the method; we note that the maximal possible compression ratio attainable is 50% using VFold alone (analogous to 0% in methods that compress keys and values), which is why we report 50% reduction of the value cache (25% overall), and 75% reduction of the value cache (37.5% of overall). We would like the claims made to be well supported with evidence in our work; would you be able to note more specifically where you believe the claims about VFold’s advantage is overstated?

We also agree that the achievable compression ratio matters. However, it is the case that we target values only by design, which limits overall compression ratios to an unattainable 50% maximum (from deleting all value vectors). Given that the work chooses to investigate values deeply, our view is that larger ratios should come from quantization rather than continuing to grow VFold group sizes; we demonstrate it is orthogonal to quantization and a key-side method. KIVI-4bit + VFold reaches 4.27× compression within 0.3 points of VFold alone. The Qwen3-8B MK3 drop occurs at group size of 4, which was included to demonstrate VFold’s limits at a larger compression ratio. At the recommended setting, VFold shows strong performance. Our hypothesis re MK3 is that it requires exact retrieval among several needles with similar formats, and averaging four layers’ value vectors blurs these distinctions.

W2 Additional evaluation settings: Thank you for the suggestion. Our original benchmark selection of RULER 16K was because performance differences became evident at this length given the models we were using, which is not the case with shorter lengths. We now include additional results on RULER 32K and InfiniteBench suggested by the reviewer. We report average scores here for visibility, and will report results on all models on Infinibench (too long to run for rebuttal period) for the camera ready. We will incorporate these present results into our revision as well.

RULER 32K @ 25% compression:

ModelFull cacheVFoldVFold-NaiveCommonKVMiniCacheMiniCache-V
Llama-3.1-8B-Instruct86.1584.9278.9475.2438.7858.99
Mistral-Small-3.1-24B-Instruct94.4792.0488.9792.9023.2976.04
Qwen3-8B88.5584.1469.9979.936.2459.52

InfiniteBench @ 25% compression, Llama-3.1-8B-Instruct:

TaskFullKVVFoldCommonKVMiniCache-VMiniCache
Avg52.4248.1139.6933.6631.71

We see that VFold demonstrates strong performance on these additional benchmarks, strongly outperforming baselines on Llama-3.1-8B-Instruct and Qwen3-8B, and performing comparably to CommonKV for Mistral.

W3/Q3: Intuition of CCA, compare to SVD: Thanks for your great question which allows us to improve our motivation of using CCA in our work. In the stated weakness, you are right that CCA could be replaced by other maps, we characterize the permissible set of maps for head-level value states to be the general linear group (GL) of size d_h (line 193). However, we choose CCA because it finds directions in which the two layers are most correlated on the same tokens, and it is invariant to invertible maps, which matches the symmetry group.

Other potential maps fall short of its benefit. Rotational maps do not address scale differences between vectors and can lose accuracy due to large scale changes. Least-squares maps end up with poor condition numbers due to essentially overfitting. Another large benefit of CCA is that maps are well conditioned (Table 6). Joint SVD assumes both layers receive the same input since it applies to the weights, whereas CCA is fit on each layer’s actual values. The effect of SVD vs CCA can be found in CommonKV, who similarly use reprojection plus layer merging to achieve compression; however, VFold reduces many issues related to decoding latency, which is another benefit of avoiding low-rank approaches. We will improve our motivation regarding CCA maps in our revision; thanks for pointing this gap out.

Q1: CKA vs cosine settings: This was just meant to communicate that the adjacent layer similarity computation was the same up to metric. We agree that this description was not the most clear and will update the text in the revision.

We thank the reviewer for their helpful feedback and questions, which have helped strengthen our work. Please let us know if any points need further clarification.

Thanks for your response. My concerns on the limited performance remain. Besides, I would like to suggest more discussion and analysis on design choice, which could better motivate the method. Therefore, my ratings toward the paper will be maintained.

Do you mind clarifying what specifically in the author response regarding 1) performance and 2) the design choice fall short of sufficient clarification? We would appreciate this information as it would help us potentially address your remaining concerns and improve our work.

Reviewer_frGQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VFOLD proposes a cross-layer value cache compression method that aligns adjacent layers using CCA-based transformations and head-level permutations. By folding these transformations into attention weights, VFOLD enables direct value cache averaging without additional alignment or reconstruction overhead during decoding, achieving 25% KV cache reduction with minimal accuracy loss.

Critical strengths
  • this paper empirically demonstrates that keys and values exhibit substantially different robustness to merging, despite having high inter-layer CKA similarity.
  • Beyond identifying high CKA similarity, VFOLD effectively improves inter-layer cosine similarity through CCA-based alignment and head-level permutation, without requiring additional runtime alignment operations or fine-tuning.
  • VFOLD generally outperforms existing cache-merging methods in both accuracy and hardware efficiency across the evaluated models and task. The paper also provides clear ablation studies that demonstrate the contributions of averaging and alignment.
Critical weaknesses
  • Calibration cost and domain generalization: How expensive is the offline CCA calibration across all head pairs? Also, evaluations on domain-specific generation tasks and calibration sensitivity analyses would help assess the generalization of the alignment maps.
  • Limited compression benefits: Unlike eviction or quantization, VFOLD does not directly reduce attention computation or per-layer memory traffic, while leaving the key cache uncompressed. This makes its achievable compression ratio particularly important. However, the severe accuracy drop on certain tasks at 37.5% KV reduction (e.g., 98.0% → 41.2% on Qwen3-8B MK3) raises concerns about its accuracy–compression trade-off at higher compression ratios.
  • Baseline comparison fairness: Although the paper highlights the different compression sensitivities of keys and values, most baselines are matched only by total KV cache reduction. Additional value-only comparisons would help better isolate the benefits of the proposed alignment method.
Questions for authors

Please address the concerns raised in Weaknesses.

4 comments

We thank the reviewer for their time, and for acknowledging the empirical key/value asymmetry investigation, similarity investigations before and after VFold alignment, and the task & efficiency performance of our method compared to baselines. We address reported weaknesses and questions as follows:

W1: Calibration cost, domain generalization, calibration sensitivity analysis: Thanks for raising these important points. We address calibration cost, domain generalization, and calibration sensitivity with new experiments in our response.

First, we report the offline CCA calibration cost on Llama-3.1-8B-Instruct. This is on the paper setting, using 128 chunks of 2048 tokens from Wikitext for computing maps, and then computing all CCA maps with Hungarian matching to pick final maps. We report timing and expense of the forward pass & map building on a single NVIDIA L40S 48GB. Total time for Llama-3.1-8B-Instruct is about 5 min, or 8 min if we include model loading/writing.

Whole stageCCA fittingScoring + Hungarian
335.4 s270.5 s64.9 s

On domain-specific generalization, we include an experiment on Llama-3.1-8B-Instruct where we vary the calibration data to be either Wikitext data or GSM8K data (one in-domain dataset, and one-OOD dataset) but test GSM8K only (500 samples only). As seen in the table, the method performs well but is insensitive to the domain shift between in-domain and general data.

MethodScore
FullKV83.2
VFold-Naive79.0
VFold, WikiText maps82.4
VFold, GSM8K maps82.2

Finally, we include an analysis of the sensitivity of the method to the number of samples used to create calibration maps. The paper uses 128 samples as standard. In this experiment, we use {8, 32, 128, 512} samples, and limit our eval for the time being to 6 tasks within LongBench. We will include a full eval in our revision. As seen in the table, VFold is not very sensitive to the amount of data used, and overall demonstrates robustness to many decisions regarding calibration data.

Calibration samplesqasperhotpotqagov_reporttreclccrepobench-pAvg
FullKV (no compression)46.7958.1034.4572.5065.0658.3055.87
846.5357.8031.0272.5064.8557.4755.03
3245.7057.9030.8472.5064.6057.6854.87
12845.1457.9031.5272.5065.2257.3754.94
51244.6757.8931.3772.5065.2057.0154.77

W2: Limited compression benefits

The reviewer is correct in noting that VFold, among other similar methods, does not decrease attention computation or per-layer cache sizes, but instead focuses on decreasing cache memory which in turn helps allow larger batches or longer contexts as seen in Table 4.

We also agree that the achievable compression ratio matters. However, it is the case that we target values only by design, which limits overall compression ratios to an unattainable 50% maximum (from deleting all value vectors). Given that the work chooses to investigate values deeply, our view is that larger ratios should come from quantization rather than continuing to grow VFold group sizes; we demonstrate it is orthogonal to quantization and a key-side method. KIVI-4bit + VFold reaches 4.27× compression within 0.3 points of VFold alone. The Qwen3-8B MK3 drop occurs at group size of 4, which was included to demonstrate VFold’s limits at a larger compression ratio. At the recommended setting, VFold shows strong performance. Our hypothesis re MK3 is that it requires exact retrieval among several needles with similar formats, and averaging four layers’ value vectors blurs these distinctions.

W3: Additional value-only baselines

Thank you for your suggestion. We agree that value-only baselines are a great way to isolate the benefits of VFold, which is why we implement MiniCache-V and VFold-Naive, which are value-only methods not present in prior work, where we adapt MiniCache to values only and report VFold without an alignment step. What is left is CommonKV; however, this is unable to be modified into a value-only version. The method creates a joint SVD breakdown of the concatenated key and value projection matrices to create a shared latent space; adapting this method to a values-only variety would require design choices beyond what is straightforward for incorporating a new baseline. Otherwise, it is the case that there are no other value-only cache compression methods to the best of our knowledge.

We thank the reviewer for their helpful feedback and questions, which have helped strengthen our work. Please let us know if any points need further clarification.

Reviewer_frGQOpenReview ↗

Response to the Authors' Rebuttal

Thank you for the detailed response and additional clarifications.

Regarding the calibration cost, I think the reported overhead is reasonable, which addresses my concern on this point. However, I still have some remaining concerns. For domain generalization, the response mainly refers to GSM8K, which was already included in the original paper, rather than providing results on new benchmarks. For calibration sensitivity, I appreciate the additional results, although I think further analysis is needed to explain why performance decreases as the number of calibration samples increases. I also feel that concerns regarding the reliance on other methods for higher compression ratios and the limited comparison with value-only baselines remain insufficiently addressed.

Overall, while I appreciate the authors' efforts and acknowledge that some of my concerns have been addressed, several important questions remain. Taking these points into account, I will maintain my original score.

Thank you for your quick response. We would like to respond to your remaining concerns with additional experiments and clarification:

  1. Additional domain-specific tasks & new benchmarks

We run 2 additional benchmarks as part of our updated evaluation. These are InfiniteBench and RULER 32K for far longer generations. We would also like to just clarify the range of domains covered in RULER, LongBench, and InfiniteBench: LongBench covers narrative fiction (NarrativeQA), scientific papers (Qasper), wikipedia (HotpotQA which is multihop, 2WIkiMQA, MuSiQue), Government data (govreports), meetings (QMSum), news (multinews), dialogue (SAMSum), and code (LCC, Repobench). RULER covers a number of synthetic stress tests designed to understand long-context performance. Finally, InfiniteBench covers novels, dialogue, code, and math. It would be helpful to us to understand where the current evaluation lacks in domain coverage, in order to strengthen the claims in the current work.

We note that calibration data is always Wikitext2; we misunderstood your original point to concern domain matching between calibration and evaluation, which is what we included in that new experiment. We hope these additional results on new benchmarks helps address your clarified concern about additional results on new benchmarks. We report average scores here for visibility.

RULER 32K @ 25% compression:

ModelFull cacheVFoldVFold-NaiveCommonKVMiniCacheMiniCache-V
Llama-3.1-8B-Instruct86.1584.9278.9475.2438.7858.99
Mistral-Small-3.1-24B-Instruct94.4792.0488.9792.9023.2976.04
Qwen3-8B88.5584.1469.9979.936.2459.52

InfiniteBench @ 25% compression, Llama-3.1-8B-Instruct:

TaskFullKVVFoldCommonKVMiniCache-VMiniCache
Avg52.4248.1139.6933.6631.71

We see that VFold demonstrates strong performance on these additional benchmarks, strongly outperforming baselines on Llama-3.1-8B-Instruct and Qwen3-8B, and performing comparably to CommonKV for Mistral.

  1. Performance across calibration samples

We have since rerun this experiment examining VFold's sensitivity to the amount of calibration data across 3 seeds (data sampling randomness) and report results as follows:

Calibration samplesqasperhotpotqagov_reporttreclccrepobench-pAvg
FullKV46.7958.1034.4572.5065.0658.3055.87
846.24 ± 0.8857.89 ± 0.1030.99 ± 0.1472.50 ± 0.0064.81 ± 0.3357.74 ± 0.3655.03 ± 0.25
3245.95 ± 0.2457.93 ± 0.0631.17 ± 0.3072.50 ± 0.0064.26 ± 0.3557.62 ± 0.2254.91 ± 0.03
128 (paper)45.57 ± 0.4557.93 ± 0.0631.31 ± 0.2272.50 ± 0.0064.96 ± 0.2657.48 ± 0.1354.96 ± 0.09
51244.83 ± 0.3958.00 ± 0.1831.11 ± 0.2372.50 ± 0.0064.90 ± 0.2857.48 ± 0.4154.80 ± 0.09

Given these results, several things are evident. 1, it is not the case that average performance decreases with more samples (these results are not monotone with sample size). The span of scores is only 0.23. Finally, the 8 sample setting has the highest standard deviation which suggests the performance at this smaller calibration level is less stable. We take this finding to be positive, implying that the maps are learnable with only a few samples, and do not depend on increasing data to improve performance, suggesting that the map fitting is data efficient and robust. If this interpretation does not seem well justified, it would be helpful to understand where this additional calibration sensitivity falls short of addressing the stated concern.

  1. Value-only baselines We provided our response to the limited values only baseline concern which included the fact that CommonKV can not be adapted to a value only version, and that others do not exist (to the best of our knowledge). Would you be able to point to or describe additional value-only baselines you had in mind? These types of baselines would be beneficial in improving our work.

  2. Limited compression benefits: Finally, we acknowledge the maximum 50% KV cache compression ratio as a known and inherent limitation of our work as we decided to take a closer look at values-only compression in a way that exploits key-value asymmetries. We would like to note though that the contributions of this work are not intended to lie in a SOTA method KV cache method, which would be very difficult intentionally focusing on values only, but instead to shed light on key-value asymmetries, focus heavily on the cross-layer potential of value caches, and demonstrate a low-overhead method that exploits this phenomenon and performs competitively with other cross-layer methods; despite only applying to a single cache.

Reviewer_edSuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VFold, a method for inference in LLMs with long contexts, which combines "symmetry-aware alignment" with "cross-layer value-cache merging." The approach employs a mapping based on Canonical Correlation Analysis (CCA) to align value representations before averaging them. Experiments demonstrate that this method reduces the memory footprint of the KV cache while maintaining accuracy on long-context tasks; the evaluation covers three distinct models.

Critical strengths
  1. The paper is well-written, easy to follow, and features clear figures.
  2. Given the high inference costs of LLMs, KV cache compression is an effective strategy for reducing GPU memory usage and enabling longer context windows. The authors propose an efficient compression method that mitigates the performance degradation typically associated with merging value caches.
  3. The method is evaluated across a variety of LLMs and datasets, outperforming the baselines in five out of the six primary experimental settings.
Critical weaknesses
  1. Limited methodological innovation. VFold primarily integrates existing techniques: cross-layer cache merging was explored in MiniCache [1]; CCA-based representation alignment was applied in CCA Merge [2]; and weight transformations that preserve attention mechanisms were studied by Zhang et al. [3]. Overall, the approach appears to be an incremental combination of existing concepts regarding cache sharing and alignment.
  2. Lack of comparison against highly relevant baseline methods. Although KVSharer [4] and EchoKV [5] are mentioned in the text, no experimental comparisons with these cross-layer compression methods are provided. Without a comparative analysis under equivalent budgets, it is impossible to determine whether this framework offers a significant advantage over existing approaches.
  3. The experiments cover Llama-3.1-8B-Instruct, Mistral-Small-3.1-24B-Instruct, and Qwen3-8B, but do not include larger-scale models or those based on the MoE architecture. Since the effectiveness of "per-head alignment" and "cross-layer averaging" may depend on the underlying architecture, further experiments—for instance, using models like Gemma-4-31B and Qwen3.6-35B-A3B—are necessary to better validate the framework's generalizability.
  4. The evaluation is primarily based on LongBench and 16K RULER, omitting other benchmarks such as LongBench v2 [6] and InfiniteBench [7]. Consequently, the current evidence is insufficient to demonstrate that this compression strategy remains effective for more challenging reasoning tasks or longer contexts. Additionally, why did the authors not conduct evaluations on longer RULER datasets, such as those with 32K and 64K?
  5. Performance improvements are not always significant. For instance, with a 37.5% reduction in the total KV cache size, VFold underperforms compared to CommonKV on the RULER benchmark using the Qwen3-8B model (85.18 vs. 87.46). Furthermore, Table 4 indicates that at a batch size of 1, the decoding speed is slower than when using the full cache (27.8 ms/token vs. 21.0 ms/token).

[1] Liu A, Liu J, Pan Z, et al. Minicache: Kv cache compression in depth dimension for large language models [C]. Advances in Neural Information Processing Systems, 2024, 37: 139997-140031.

[2] Horoi S, Camacho A M O, Belilovsky E, et al. Harmony in diversity: Merging neural networks with canonical correlation analysis [C]. Forty-first International Conference on Machine Learning, 2024.

[3] Zhang B, Zheng Z, Chen Z, et al. Beyond the permutation symmetry of transformers: The role of rotation for model fusion [C]. Forty-second International Conference on Machine Learning, 2025.

[4] Yang Y, Cao Z, Chen Q, et al. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing [J]. arXiv preprint arXiv:2410.18517, 2024.

[5] Ji S, Wang Y, Liu Y, et al. EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction [J]. arXiv preprint arXiv:2603.22910, 2026.

[6] Bai Y, Tu S, Zhang J, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks [C]. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 3639-3664.

[7] Zhang X, Chen Y, Hu S, et al. ∞ Bench: Extending long context evaluation beyond 100K tokens [C]. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 15262-15277.

Questions for authors
  1. How does VFold differ from approaches that combine existing techniques—such as cross-layer cache merging, CCA-based alignment, and transformations that preserve the attention mechanism?
  2. Could the authors provide further comparisons with related methods, covering larger-scale model architectures and MoE models, along with experimental results on LongBench v2 and InfiniteBench?
  3. Could the authors explain why the improvements in accuracy are inconsistent, and why there is additional decoding overhead?
4 comments

We thank the reviewer for their time, for noting the clarity of the presentation of the work, and for noting the effectiveness of the strategy proposed. We address reported weaknesses and questions as follows, including many additional experiments:

W1/Q1: Methodological innovation follows 3 prior works: We agree with the reviewer that these 3 citations (MiniCache, CCAMerge, Zhang et al. Beyond permutation symmetries) are critical prior work that help motivate the current work. However, we would like to posit that the contribution of the work, despite its sources of inspiration, is indeed not limited in methodological innovation and instead promotes a novel and exciting direction for KV cache compression. Firstly, our work spends considerable effort discussing results that motivate the particular method, including results on cache similarity (admittedly, previously explored) and novel results on key vs value cache averaging sensitivity, which was critically underexamined in MiniCache (in fact, had they done this, they could have proposed MiniCache-V which we implement as a baseline stronger than MiniCache). Secondly, we expand upon the intra-attention head rotation symmetry class proposed in Zhang et al. to include head-wise permutation matching, which is missing in their symmetry description. In both Zhang et al. and CCAMerge, folded maps are present to align weight matrices to combine disjoint models, whereas in our work, we use these maps to align adjacent caches for downstream merging. It was not clear until experimentation that a folding-based approach would work well applied to very different objects (averaging caches versus averaging weights). Finally, a subtle contribution that we would like to emphasize when talking about novelty is that VFold, by design, allows new tokens to be generated in an uncompressed manner, which was not the case in the MiniCache paper (no official implementation to check), cannot be the case in CommonKV and related papers (due to low-rank by design), and is frequently not the case otherwise (activation quantization, etc.). Our work emphasizes this as an important design feature for compression techniques that operate independently of the sequence length [see lines 202-203]. Many of these design choices are in place in order to produce a low-overhead, more performant (much more than MiniCache) answer to prior work on cross-layer cache compression, which is both understudied as well as previously prohibitively expensive to attempt to deploy in a realistic setting.

W2/Q2: Specific addl baseline methods: Thanks for your suggestion. We agree a direct comparison would be informative; to this end, we will expand our discussion of both listed methods in the paper. KVSharer directly shares caches between dissimilar layers without any averaging; we note that the original paper only reports results on perplexity and short tasks, and sometimes performs poorly on even these tasks. We include a small comparison that shows this discrepancy compared to VFold-Naive (worse than VFold), on Llama-3.1-8B-Instruct with GSM8K, with results reported below:

Llama-3.1-8B-Instruct @ 25% compression, evaluated on GSM8K (accuracy):

Full cacheVFold-NaiveKVSharer
80.7%79.5%1.82%
Given these results, we do not include it as a main baseline as it is not very competitive.

EchoKV is a learned method that trains a linear predictor to reconstruct a full KV cache from a truncation and another layer cache; it runs this reconstruction at every decode step before attention is computed. VFold is quite different in spirit from EchoKV; it is training-free, parameter-free, and lightweight by design. While we do not have the computational capacity to benchmark EchoKV at the moment as we would need to train the predictor for each setting and model; we intend to include it for the camera ready version with more time and compute power since it is related to inter-layer cache relationships. We will include more discussion of these two methods in the updated PDF.

W3/Q2: More models to strengthen claim across architectures Thank you for the suggestion. VFold applies only to Transformer attention sublayers & their caches, interacting specifically with WVW_V and WOW_O weight matrices. This means it applies straightforwardly to models with either dense or MoE FFN layers; our current evaluation includes 2 GQA variants (with and without QKNorm) where our method is shown to work well on either, and 24B parameter Mistral is already in parameter range of the suggested models, given common release sizes for open-weight models. Additionally, the proposed models both include interleaved attention variations including partial vs full attention. This means that at initialization, these models already reduce cache pressure by design. We intend to focus on targeting models with heavier cache pressure where cache memory becomes a decoding bottleneck more quickly.

W4/Q2: Additional benchmarks Thank you for the suggestion. Our original benchmark selection of RULER 16K was because performance differences became evident at this length given the models we were using, which is not the case with shorter lengths. We now include additional results on RULER 32K and InfiniteBench suggested by the reviewer. We run the 9 English tasks in InfiniteBench. We report average scores here for visibility, and will report results on all models on InfiniteBench (too long to run for rebuttal period) for the camera ready. We will add these present results to our revision as well.

RULER 32K @ 25% compression:

ModelFull cacheVFoldVFold-NaiveCommonKVMiniCacheMiniCache-V
Llama-3.1-8B-Instruct86.1584.9278.9475.2438.7858.99
Mistral-Small-3.1-24B-Instruct94.4792.0488.9792.9023.2976.04
Qwen3-8B88.5584.1469.9979.936.2459.52

InfiniteBench @ 25% compression, Llama-3.1-8B-Instruct:

TaskFullKVVFoldCommonKVMiniCache-VMiniCache
Avg52.4248.1139.6933.6631.71

We see that VFold demonstrates strong performance on these additional benchmarks, strongly outperforming baselines on Llama-3.1-8B-Instruct and Qwen3-8B, and performing comparably to CommonKV for Mistral.

W5/Q3: Performance comparisons & decode speed This observation about decoding speed is correct, that the decoding speed is slightly slower than the full cache run (at bsz 1) and is expected behavior. Our method reduces total cache memory needed during decoding; it does not reduce the attention load, nor does it claim to. Our method, however, speeds up decoding significantly over the CommonKV baseline (84.7 ms/token) which was our goal and we take to represent great promise for cross-layer cache compression. Additionally, since the submission, we removed some per-step CPU-GPU syncs and a slower tensor copy to further streamline our method; remeasured TPOT on the same hardware (3 run avg):

SettingFull cacheVFold
batch 120.5 ms24.2 ms
batch 32162.4 ms146.9 ms
The main benefit of VFold is cache memory reduction, which means the total throughput attainable at maximum batch is improved for VFold:* 233.8 tok/s* (batch 38) vs 204.0 tok/s (batch 33). At batch 1, the overhead comes from merging tokens evicted from the recent buffer and copying new tokens to the buffer.

Re performance comparisons, we do not claim VFold outperforms every baseline in every setting, but it is the case that VFold outperforms baselines within a majority of settings, or performs comparably when not outperforming. The benefit of VFold also lies in its much reduced overhead compared to CommonKV, where TTFT and TPOT are far larger (13% and 250% respectively). When considering both performance and efficiency, we believe VFold to be superior to the reported baselines. Regarding why we think it the case that VFold underperformed on the cited setting (Qwen on RULER @ 37.5% reduction), this could be two fold. One, CommonKV leaves 2 of 9 layer groups uncompressed and compresses the rest more (Table 9). Two, CommonKV is a prefill-only compression method whereas VFold also compresses generated tokens.

We thank the reviewer for their helpful feedback and questions, which have helped strengthen our work. Please let us know if any points need further clarification.

Thank you for the additional results and clarifications. My concern about novelty remains. I appreciate the clarification that VFold applies CCA to adjacent value caches and adds head-wise permutation matching, but the rebuttal does not establish why these changes constitute a substantial methodological advance over prior work on cross-layer cache merging, CCA-based alignment, and attention-preserving transformations.

The RULER 32K results extend the evaluation, but several experiments from my review remain missing. The rebuttal provides no comparison with EchoKV, no evaluation on larger or MoE models, and no results on LongBench v2 or RULER 64K. These gaps leave me unconvinced that VFold generalizes across architectures and remains competitive with related methods on more challenging long-context tasks. I will therefore maintain my original rating.

Thanks for your updated assessment. We respond to your remaining concerns and ask for more clarification on certain concerns to help improve our work:

Novelty It remains unclear why the unique combination of the three listed ideas (cross layer cache merging, CCA-based alignment, and exploiting existing weight symmetries in attention) into a functional KV cache compression method does not constitute sufficient novelty. If there is a specific analysis that would make the contribution clearer, we would welcome the suggestion.

Additional benchmarks Additionally, to clarify your concern about context length, our provided results on InfiniteBench (>100K) far surpass the context length of RULER 64K as suggested by the reviewer. To us, it seems running RULER 64K has diminishing returns on adding to the empirical story of the paper, provided that RULER 16K, 32K, and the even longer InfiniteBench show consistent results. If there is a specific insight the reviewer has in mind about the remaining benchmarks suggested, we would welcome the discussion.

More baselines We implemented and tested KVSharer, suggested by the reviewer, and found it to perform very poorly compared to other methods. We were not able to train the expensive EchoKV compressor adaptors within the short discussion period and acknowledge this as a limitation. EchoKV is different in spirit from many of the existing baselines in that it adds paramterization to models, occuping a different design space compared to the existing training free, parameter-free methods.

More models If there is an aspect of MoE models that is expected to interact with value-cache merging, we would be glad to discuss it. Our current analysis includes what we believe to be more important architectural elements that could affect the method directly, like GQA (addressed with Kronecker product design (App. A), and QK-normalization (present in Qwen3-8B and not the other models).

A note on empiricism and sufficient evidence As an empirical paper, there is no finite set of experiments we can provide that would conclusively demonstrate that VFold works in every setting. We just aim to show sufficient evidence supporting our claims, within the scope we state. Within this scope, results are consistent for 3 models x 4 benchmarks (including new ones from the rebuttal period), and compositions with additional methods. Because we would like to improve our work and find out where the community might find our evidence to fall short of what is sufficient to support our claims, it would be very helpful to understand among the reviewer's requests what specifically is lacking from the existing evidence where the reviewer might expect VFold to fail in a different environment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback. Reviewers noted that the paper is well written and easy to follow (edSu), that the paper provides clear ablation studies that demonstrate the contributions of averaging and alignment (frGQ), that the key-value asymmetry analysis is a key strength (frGQ, QxHf), and that the weight folding reparameterization is clean and practical (QxHf). We summarize new results and clarifications from the discussion period below, and changes appear in the updated PDF in blue:

New Results

  1. RULER 32K @ 25% KV reduction (50% value cache reduction)
ModelFull cacheVFoldVFold-NaiveCommonKVMiniCacheMiniCache-V
Llama-3.1-8B-Instruct86.1584.9278.9475.2438.7858.99
Mistral-Small-3.1-24B-Instruct94.4792.0488.9792.9023.2976.04
Qwen3-8B88.5584.1469.9979.936.2459.52

-- Demonstrates consistent performance to RULER 16K in main text

  1. InfiniteBench (9 English tasks, 100K+ tokens) @ 25%, Llama-3.1-8B-Instruct
FullKVVFoldCommonKVMiniCache-VMiniCache
52.4248.1139.6933.6631.71

-- Demonstrates consistent performance at much longer contexts

  1. KVSharer (GSM8K, Llama-3.1-8B-Instruct, 25%):
FullKVVFold-NaiveKVSharer
80.7%79.5%1.82%

-- Justifies our exclusion of KVSharer as a main baseline because it fails “easier” testing experiments

  1. Decode efficiency & throughput numbers after improving data movement
SettingFull cacheVFold
batch 120.5 ms24.2 ms
batch 32162.4 ms146.9 ms

--Answers question about decoding speed in context of total throughput instead. CommonKV’s TPOT is 84.7ms for batch 1. Full utilization throughput is 233.8 tok/s for VFold (batch 38) vs. 204.0 tok/s for the full cache (batch 33).

  1. Calibration:

Fitting maps for Llama-8B takes ~6 min on a L40S. Additional results show insensitivity to the calibration domain (GSM8K math vs Wikitext), and calibration data size (8-512 samples, steady performance) suggesting symmetry maps can be fitted quickly, efficiently, and robustly.

Design choices and supporting analyses

  1. Targeting values only: This decision is supported by Table 1 which demonstrates that values are much more amenable to simple averaging (preserves 98.5% performance vs keys’ 3.4% on GSM8K). This naturally limits our maximum KV cache compression ratio to 50% (the same as key-only methods, like ThinK [1]). This is a deliberate decision given this evident asymmetry.
  2. CCA maps to align caches before averaging: This decision separates VFold and VFold-Naive; the former outperforms the latter by 4-13 RULER points across models @ 50% value cache reduction. CCA allows for well-conditioned (stably invertible) maps (Table 6).
  3. Folding maps into weights: This crucial decision is function-preserving, does not add overhead, and runs attention unmodified
  4. Composition: Due to decision 1 and the maximum compression ratio limited by the deliberate focused study, we demonstrate 2 experiments that compose VFold with KIVI quantization and ThinK which are realistic compositions and demonstrate successful composition with preserved performance.

[1] Xu et al., 2024. ThinK: Thinner Key Cache by Query-Driven Pruning

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.