- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VFold, a cross-layer value-cache compression method for pre-trained LLMs. The motivation is that adjacent transformer layers have value vectors with high CKA similarity (but near-zero cosine similarity), suggesting they live in the same rotated basis and can be aligned and merged. It computes a CCA-based linear alignment map between adjacent layers' value caches, and fold it into the attention weights, then average the two layers' value caches into a shared inter-layer cache. They use Llama-3.1-8B-Instruct, Mistral-Small-3.1-24B-Instruct, and Qwen3-8B for evaluation on RULER and LongBench.
Critical strengths
- The idea of value cache compression is well motivated by the asymmetry of KV, where RoPE is applied on key states but not value.
- The weight-folding reparameterization is clean and practical.
- Experimental results show the effectiveness and efficiency of the proposed method.
Critical weaknesses
- The paper overstates the advantage of VFold in the main body only with a compression ratio of 25%. The performance drops as the compression ratio increases to 37.5%.
- The evaluation is conducted on RULER and LongBench. More challenging benchmarks like InfiniteBench and LongReason are recommended.
- The intuition of using CCA is not clear enough. Since no RoPE is applied on value states, could CCA be replaced by other mapping functions?
Questions for authors
- What do you mean in line 152: "For CKA similarities, we simply swap out the metric given the same setup for measuring cosine similarity"?
- How does VFold perform under larger compression ratios?
- Why CCA vs. SVD? CommonKV uses joint SVD across layers to find a shared subspace. VFold uses CCA + Hungarian matching. Have you directly compared these two alignment strategies on the same models and benchmarks?
4 comments
Q1: CKA vs cosine settings: This was just meant to communicate that the adjacent layer similarity computation was the same up to metric. We agree that this description was not the most clear and will update the text in the revision.
We thank the reviewer for their helpful feedback and questions, which have helped strengthen our work. Please let us know if any points need further clarification.
Thanks for your response. My concerns on the limited performance remain. Besides, I would like to suggest more discussion and analysis on design choice, which could better motivate the method. Therefore, my ratings toward the paper will be maintained.
Do you mind clarifying what specifically in the author response regarding 1) performance and 2) the design choice fall short of sufficient clarification? We would appreciate this information as it would help us potentially address your remaining concerns and improve our work.
We thank the reviewer for their time, and for acknowledging the motivation, practicality, and effectiveness of the proposed method. We address reported weaknesses and questions as follows:
W1/Q2 overstated advantage, size of compression ratios: We respectfully note that all main body claims are scoped to the 25% setting (and carefully rechecked, §1, §6.1). The main body also contains higher compression ratios like 50% (with ThinK), and 4.27x compression (with KIVI). It is true that performance drops as we increase the compression ratio of the method; we note that the maximal possible compression ratio attainable is 50% using VFold alone (analogous to 0% in methods that compress keys and values), which is why we report 50% reduction of the value cache (25% overall), and 75% reduction of the value cache (37.5% of overall). We would like the claims made to be well supported with evidence in our work; would you be able to note more specifically where you believe the claims about VFold’s advantage is overstated?
We also agree that the achievable compression ratio matters. However, it is the case that we target values only by design, which limits overall compression ratios to an unattainable 50% maximum (from deleting all value vectors). Given that the work chooses to investigate values deeply, our view is that larger ratios should come from quantization rather than continuing to grow VFold group sizes; we demonstrate it is orthogonal to quantization and a key-side method. KIVI-4bit + VFold reaches 4.27× compression within 0.3 points of VFold alone. The Qwen3-8B MK3 drop occurs at group size of 4, which was included to demonstrate VFold’s limits at a larger compression ratio. At the recommended setting, VFold shows strong performance. Our hypothesis re MK3 is that it requires exact retrieval among several needles with similar formats, and averaging four layers’ value vectors blurs these distinctions.
W2 Additional evaluation settings: Thank you for the suggestion. Our original benchmark selection of RULER 16K was because performance differences became evident at this length given the models we were using, which is not the case with shorter lengths. We now include additional results on RULER 32K and InfiniteBench suggested by the reviewer. We report average scores here for visibility, and will report results on all models on Infinibench (too long to run for rebuttal period) for the camera ready. We will incorporate these present results into our revision as well.
RULER 32K @ 25% compression:
InfiniteBench @ 25% compression, Llama-3.1-8B-Instruct:
We see that VFold demonstrates strong performance on these additional benchmarks, strongly outperforming baselines on Llama-3.1-8B-Instruct and Qwen3-8B, and performing comparably to CommonKV for Mistral.
W3/Q3: Intuition of CCA, compare to SVD: Thanks for your great question which allows us to improve our motivation of using CCA in our work. In the stated weakness, you are right that CCA could be replaced by other maps, we characterize the permissible set of maps for head-level value states to be the general linear group (GL) of size d_h (line 193). However, we choose CCA because it finds directions in which the two layers are most correlated on the same tokens, and it is invariant to invertible maps, which matches the symmetry group.
Other potential maps fall short of its benefit. Rotational maps do not address scale differences between vectors and can lose accuracy due to large scale changes. Least-squares maps end up with poor condition numbers due to essentially overfitting. Another large benefit of CCA is that maps are well conditioned (Table 6). Joint SVD assumes both layers receive the same input since it applies to the weights, whereas CCA is fit on each layer’s actual values. The effect of SVD vs CCA can be found in CommonKV, who similarly use reprojection plus layer merging to achieve compression; however, VFold reduces many issues related to decoding latency, which is another benefit of avoiding low-rank approaches. We will improve our motivation regarding CCA maps in our revision; thanks for pointing this gap out.