acceptodds
Under review as a conference paper at ICLR 2027

JoLT: Near-Lossless KV Cache Compression via Joint Rank-Bit Allocation

Abstract

The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference. Existing compression methods apply low-rank factorization or quantization independently, without jointly allocating rank and precision under a shared storage budget. We introduce JoLT, a training-free compressor that treats grouped prefill caches as fourth-order tensors and applies partial Tucker decomposition along the token and feature modes, the two axes that carry low-rank structure, while leaving the head and layer modes intact. A rotated low-bit quantizer captures the truncation residual, and a single Lagrangian dual allocates per-group Tucker ranks and residual bit-widths under a global byte constraint. FlashJoLT replaces the exact token-mode SVD with a randomized approximation that matches JoLT within the free zone at a fraction of the compression cost, and a fused Triton decode kernel evaluates attention directly over the stored factors without materializing dense KV tensors. Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2 - 3x compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3x and declines by only 0.90 and 2.40pp at 4x and 5x, respectively. JoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache compression, large language models, Tucker decomposition, low-rank approximation, quantization, Lagrangian relaxation, rate-distortion, long-context inference, randomized SVD, fused attention kernel

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PU3BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose JoLT, a training-free KV-cache compression method combining partial Tucker decomposition along the token and feature dimensions with quantization of the rotated residual. A joint optimization allocates decomposition ranks and quantization bit-widths under a shared budget. They also introduce FlashJoLT, a faster approximation based on randomized SVD. Experiments across different model families and attention architectures show good performance in terms of perplexity.

Critical strengths
  • An interesting and, to my knowledge, novel approach to jointly optimizing low-rank factorization and residual quantization under a shared memory budget.

  • Extensive evaluation across multiple model families and attention architectures.

  • Strong compression results in the 2x to 3× regime, with informative ablations supporting the proposed design choices.

Critical weaknesses
  • Although the authors evaluate JoLT on several downstream benchmarks, comparisons against competing compression methods are largely limited to GSM8K, where JoLT does not outperform quantization baselines. Comparisons on RULER and LongBench in terms of accuracy, not only perplexity, would help establish the advantages of the method.

  • The experiments do not include strong eviction-based approaches such as KVzip [1]. Comparing their memory-quality trade-offs under equivalent budgets would help contextualize the results.

  • Despite the introduction of FlashJoLT, the reported decoding overhead remains substantial. The authors suggest that memory savings could improve throughput in batched serving scenarios; this is not experimentally demonstrated.

[1] KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction, Kim et al.

Questions for authors
  • Could the authors evaluate throughput under realistic batched serving or memory-constrained scenarios, where the reduced KV-cache footprint might offset the additional computation?

  • Could the authors provide downstream comparisons with competing compression methods on long-context benchmarks such as RULER and LongBench, including one or two strong KV-cache eviction baselines?

1 comment

We thank the reviewer for the detailed review.

W1 (RULER and LongBench). We added these comparisons (Appendix H). On RULER at 64K (LLaMA-3.1-8B), JoLT stays within 0.6 points of the baseline up to 4x (88.7 against 89.3). Near 4x, int4 scores 84.6 and TurboQuant 78.7, and KIVI scores 68.0 at 5.3x. On LongBench at 16K (Mistral-7B), all methods up to 4x stay within 0.52 points of the baseline.

W2 (KVzip). Due to limited time we ran KVzip on RULER only, at JoLT's stored bytes (Appendix H). KVzip scores 93.0 and 94.5 at 2x and 3x, above both its own baseline (88.8) and JoLT (89.8), because eviction helps on the multi-key tasks with distractor needles. Eviction decides which tokens to keep and JoLT how to store them, so the two are complementary.

W3 / Q1 (batched serving). We added batched serving (Appendix G). At 8K context, the compressed cache fits 45 and 62 sequences at 2x and 3x against 23, but total throughput is about a quarter of the dense cache's (50 and 43 against 192 tokens/s). The smaller footprint raises capacity without offsetting the slower decode, so we scope JoLT to stored prompt caches, which decode at dense speed (Section 4.5).

We hope these changes address your concerns and would be glad to clarify anything further.

Reviewer_5B4FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed JoLT in this paper, a training-free KV-cache compression method that combines partial Tucker decomposition, which truncates only the token and feature modes, with a rotated low-bit residual. A Lagrangian allocator distributes Tucker ranks and residual bit-widths across layer groups, keys, and values under a shared byte budget. The authors also introduce FlashJoLT, which replaces the exact token-mode SVD with a randomized SVD to reduce compression cost, and a fused Triton kernel that computes attention directly over the compressed representation.

Across five models, JoLT achieves 2 – 3X compression with less than 0.2% perplexity degradation. Downstream evaluations on RULER, GSM8K, and LongBench further support its compression quality, although decoding from the compressed representation introduces substantial latency overhead.

Critical strengths
  1. The 2 – 3 X near-lossless results are impressive, especially across different model architectures, e..g, MHA, GQA, and MoE. The 64K RULER evaluation is also encouraging. These results make a convincing case that the compression approach preserves useful information.

  2. Combining low-rank factors with a quantized residual is sensible, and the rank-only ablation shows clearly that the residual stage is necessary. I also appreciate that the authors implemented a fused decode kernel rather than stopping at offline reconstruction metrics, and that they report its latency candidly.

Critical weaknesses
  1. Section 4.5 reports that the fused kernel is 7.5 – 65.9 X slower than dense attention in a single-layer, batch-1 microbenchmark. On RULER NIAH with Mistral-7B at 8K context, end-to-end latency (including compression and per-step reconstruction) increases from 1.6s to 18.2 – 20.7s. Meanwhile, the measured peak KV decode memory drops by only 1.43 X and 2.40 X at the 2 X and 3 X targets. That’s a large latency cost for a modest memory reduction. Moreover, at 2 X, int8 per-channel and fp8 reach the same quality (Table 2) with standard, low-overhead kernels. The authors acknowledge this limitation, which I appreciate. Still, without throughput or batched-serving results, it’s hard to tell whether the memory savings actually improve serving capacity. This is my biggest concern.

  2. Joint rank-bit allocation is the paper’s main methodological contribution, but the ablations don’t test it directly. The joint vs. K/V-decoupled comparison in Tables 11 and 12 changes only whether keys and values share one budget. Both variants still allocate ranks and bits jointly. Even that comparison is mixed: the two are identical at 2 X, the decoupled version is better at 4 X, and the joint version wins at 8 X for n=512n = 512 and n=1024n = 1024 but not for n=2048n = 2048. There is also no 3 X point. The comparison that would isolate the contribution is missing: a fixed residual bit-width (e.g., b=4b = 4 for every cell) with Lagrangian rank allocation. Section 3 motivates the joint allocator by arguing that such a fixed policy cannot adapt, so this experiment seems necessary.

Questions for authors

Please refer to the Weakness section

1 comment

We thank the reviewer for the detailed review.

W1 (memory, throughput, serving). We agree that int8 and fp8 match JoLT at 2x with cheaper kernels. The resident KV memory during fused decode now matches the targets (2.00x to 2.04x and 2.97x to 3.02x at 32K, Appendix F). In batched serving (Mistral-7B, 8K context), the compressed cache fits 45 and 62 sequences at 2x and 3x against 23, but aggregate throughput is 21% to 28% of the dense cache's (Appendix G). Capacity improves and throughput does not, so we scope JoLT to stored prompt caches, which decode at dense speed (Section 4.5).

W2 (isolating the joint allocator). We ran this ablation (Appendix I). No fixed width works across ratios: fixed 2-bit loses at 3x (6.685 against 6.539), and fixed 4-bit cannot reach 4x. At 4x and 5x, fixed 2-bit beats the submitted joint allocator (6.719 against 6.940 at 4x), because the allocator gives zero residual bits to many key cells. Removing 0 from its bit menu gives the best perplexity at 4x and 5x and matches the joint allocator at 3x.

We hope these changes address your concerns and would be glad to clarify anything further.

Reviewer_RjYBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes JoLT, a training-free KV-cache compression framework combining partial Tucker decomposition, quantized residuals, and joint rank-bit allocation. The proposed method exploits low-rank structures along the token and feature dimensions while preserving the layer and attention-head dimensions. A Lagrangian optimization framework allocates Tucker ranks and residual quantization bit-widths under a shared memory budget. The authors also introduce FlashJoLT, which accelerates decomposition using randomized SVD, and a fused Triton kernel for decoding from compressed representations. Experiments across five LLMs demonstrate near-lossless perplexity at 2-3x KV-cache compression without retraining. However, the proposed decoding implementation incurs substantial latency overhead.

Critical strengths
  • Compression quality is meaningful. JoLT preserves perplexity well at 2-3x compression across five models, with less than 0.2% degradation.

  • Evaluation of compression quality is comprehensive. Experiments cover MHA, GQA, and MoE architectures, together with perplexity, reasoning, and long-context retrieval benchmarks.

Critical weaknesses
  • Latency overhead is severe - this is my primary concern. Section 4.5 reports that the fused attention kernel is 7.5-65.9x slower than uncompressed attention. More importantly, the end-to-end RULER latency increases from 1.6 seconds to 20.7 and 18.2 seconds at 2x and 3x compression, respectively. These represent approximately 13x and 11.4x slowdowns. Although JoLT reduces KV-cache memory, such substantial performance degradation makes its practical value questionable.

  • Comparison with existing methods is not fair. Appendix E reveals substantial modifications to existing methods. In particular, the Palu baseline replaces its original offline weight factorization with per-prompt KV-cache SVD; optional latent quantization is also disabled. ReCalKV undergoes similar modifications. Consequently, the large reported performance gaps cannot be interpreted as reliable improvements over their (Palu's and ReCalKV's) published implementations. This is a significant weakness of experimentation.

  • Competitiveness against modern quantization methods is limited. The authors' own results show that TurboQuant achieves approximately 3.82x compression with only 0.28% perplexity degradation on Mistral, whereas JoLT incurs 5.65% degradation at 4x. Furthermore, standard 8-bit quantization already achieves near-lossless performance at 2x without JoLT's elaborate decomposition procedure. The advantage of JoLT is thus restricted to a relatively narrow compression range; beyond the "sweet" range, JoLT is not particularly superior.

  • JoLT performs global tensor decomposition on an existing prefill cache, requiring expensive SVD operations and rank allocation. Its token-mode factors depend on the entire cached sequence, making JoLT poorly suited to continuous autoregressive inference. FlashJoLT reduces the SVD computation but does not resolve this architectural limitation. Even for reusable prompt caches, the paper does not quantify the number of cache reuses required to amortize compression and decoding overhead.

Questions for authors

See "Critical Weaknesses" for my concerns and questions.

1 comment

We thank the reviewer for the detailed review.

W1 (latency). We agree that decoding from the compressed cache is slower than dense decoding. The submitted Section 4.5 mixed three measurements, which the revision separates. The 7.5x to 65.9x figure is one attention layer in isolation. The 20.7 s and 18.2 s RULER figures include compression and per-step dense reconstruction, so they do not measure the fused kernel. Over a whole-model decode step (Mistral-7B, 32K context, batch 1), the fused path takes 1.8x and 2.9x as long as dense decoding at the 2x and 3x targets. The remaining cost comes from RoPE, which forces full key reconstruction per tile. For stored prompt caches, decoding runs at dense speed: on Mistral-7B at 32K, the stored file is 2x and 3x smaller and loads and rebuilds in 1.74 s and 1.22 s, against 2.20 s for the uncompressed cache.

W2 (baseline fairness). We agree. Table 2 now runs the authors' released Palu pipeline with latent quantization (Appendix E). JoLT ties Palu with 8-bit latents at 2x, is better than every Palu configuration at 3x (+0.04% against +2.44%), and is worse at 4x than Palu with 4-bit latents, whose compression there comes from quantization. We removed the claim of a large gap at 4x. The earlier Palu and ReCalKV arms are labelled as per-prompt adaptations, and we make no claim about the published ReCalKV.

W3 (quantizers). We agree that int8 and fp8 match JoLT at 2x and that quantizers have lower perplexity near 4x. Our claim is that joint rank-precision allocation makes low-rank compression competitive with quantization in the near-lossless regime, not that JoLT is best at every ratio. Near 3x, JoLT scores 89.8 on RULER at 64K against 87.3 for TurboQuant 5-bit (Appendix H).

W4 (global decomposition, reuse count). Factorizing the finished prefill cache is shared by per-prompt low-rank methods such as xKV, ShadowKV and SVDq. We scope JoLT to reusable prompt caches and now report the reuse count (Section 4.5, Appendix F). The one-time compression pays for itself after about 15 reuses at 2x and 12 at 3x against recomputing the prefill, and after 62 and 30 against storing the uncompressed cache.

We hope these changes address your concerns and would be glad to clarify anything further.

Comments on the paper

1 comment

We thank the reviewers for their careful reading and concrete suggestions. Three concerns were shared across reviews: the decode latency of the fused kernel, the memory it saves in practice, and the fairness and breadth of the baselines. We have uploaded a revised paper with all changes marked in blue:

  1. Section 4.5: separate kernel, whole-model and end-to-end timings, and stored-prompt-cache results (load time, stored size, reuse count).
  2. Appendix F: measured KV memory at the 2x and 3x targets, with a per-component breakdown.
  3. Appendix G: batched-serving capacity and throughput.
  4. Table 2 and Appendix E: Palu from the authors' released pipeline, adapted arms labelled as per-prompt, and the 4x claim over Palu removed.
  5. Appendix H: RULER and LongBench against KIVI, TurboQuant and int4, and KVzip at matched memory.
  6. Appendix I: fixed residual bit-width ablation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.