- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose JoLT, a training-free KV-cache compression method combining partial Tucker decomposition along the token and feature dimensions with quantization of the rotated residual. A joint optimization allocates decomposition ranks and quantization bit-widths under a shared budget. They also introduce FlashJoLT, a faster approximation based on randomized SVD. Experiments across different model families and attention architectures show good performance in terms of perplexity.
Critical strengths
-
An interesting and, to my knowledge, novel approach to jointly optimizing low-rank factorization and residual quantization under a shared memory budget.
-
Extensive evaluation across multiple model families and attention architectures.
-
Strong compression results in the 2x to 3× regime, with informative ablations supporting the proposed design choices.
Critical weaknesses
-
Although the authors evaluate JoLT on several downstream benchmarks, comparisons against competing compression methods are largely limited to GSM8K, where JoLT does not outperform quantization baselines. Comparisons on RULER and LongBench in terms of accuracy, not only perplexity, would help establish the advantages of the method.
-
The experiments do not include strong eviction-based approaches such as KVzip [1]. Comparing their memory-quality trade-offs under equivalent budgets would help contextualize the results.
-
Despite the introduction of FlashJoLT, the reported decoding overhead remains substantial. The authors suggest that memory savings could improve throughput in batched serving scenarios; this is not experimentally demonstrated.
[1] KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction, Kim et al.
Questions for authors
-
Could the authors evaluate throughput under realistic batched serving or memory-constrained scenarios, where the reduced KV-cache footprint might offset the additional computation?
-
Could the authors provide downstream comparisons with competing compression methods on long-context benchmarks such as RULER and LongBench, including one or two strong KV-cache eviction baselines?
We thank the reviewer for the detailed review.
W1 (RULER and LongBench). We added these comparisons (Appendix H). On RULER at 64K (LLaMA-3.1-8B), JoLT stays within 0.6 points of the baseline up to 4x (88.7 against 89.3). Near 4x, int4 scores 84.6 and TurboQuant 78.7, and KIVI scores 68.0 at 5.3x. On LongBench at 16K (Mistral-7B), all methods up to 4x stay within 0.52 points of the baseline.
W2 (KVzip). Due to limited time we ran KVzip on RULER only, at JoLT's stored bytes (Appendix H). KVzip scores 93.0 and 94.5 at 2x and 3x, above both its own baseline (88.8) and JoLT (89.8), because eviction helps on the multi-key tasks with distractor needles. Eviction decides which tokens to keep and JoLT how to store them, so the two are complementary.
W3 / Q1 (batched serving). We added batched serving (Appendix G). At 8K context, the compressed cache fits 45 and 62 sequences at 2x and 3x against 23, but total throughput is about a quarter of the dense cache's (50 and 43 against 192 tokens/s). The smaller footprint raises capacity without offsetting the slower decode, so we scope JoLT to stored prompt caches, which decode at dense speed (Section 4.5).
We hope these changes address your concerns and would be glad to clarify anything further.