- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes KVRecode, a lossless compression format and runtime for BF16 KV caches that stay resident on the GPU during generation. Each 256-word page of a K or V tensor is checked for how many distinct exponent values it contains. This gives a best-case object ratio of about 0.797. No calibration and no per-value exception handling are needed.
Critical strengths
None
Critical weaknesses
- While I appreciate the authors' detailed writing in the method section, I think it's a bit too deep in system-level programming, which might be hard for the unfamiliar readers to comprehend. I would recommend a more reader-friendly version of the method in the main text, while the existing one should be used as detailed system-info level in the appendix.
- Novelty is limited, and the components are not evaluated individually. Palette (dictionary) coding with fixed-width indices and a raw fallback is a standard data-compression technique. Splitting off the exponent bits follows SplitZip. Reusing a shared layer-sized decode buffer is already in ECF8. Keeping recent tokens raw resembles KIVI's full-precision residual. The contribution is mainly a careful engineering combination of these, and the ML insight relevant to ICLR is thin.
- Experimental scope is limited. Everything runs on one consumer GPU (RTX 4090D), at batch size 1, in HuggingFace Transformers with Flash SDPA, without integration into a paged-attention serving engine (vLLM, SGLang). The study also only uses two models.
- The runtime adds reconstruction traffic on every decoding step. Per Section 3.3 and Eq. 9–10, each layer's sealed slabs are decoded into flat scratch, copied into a shared full-layer BF16 workspace, and only then consumed by an unmodified SDPA kernel. Because a single workspace is shared across layers (and, per the text, across decoding steps), this reconstruction appears to be repeated for every layer at every step. Relative to Raw, which reads the cache once, this adds at least one full BF16 write, a scratch-to-workspace copy, and a second read of the reconstructed data. I believe a fused approach that decodes inside the attention kernel (analogous to ZipServ's fused decompression for weights, which the paper cites) would eliminate the workspace traffic and the workspace memory.
Questions for authors
- Can authors show a setting where the memory saving pays off: higher throughput at larger batch sizes, more concurrent requests under a fixed memory limit, or longer contexts that Raw cannot fit?
- What are the rates of raw fallback broken down by K vs V, layer, and head, and how do they depend on the page layout?
- Following up on 4, can we try to fuse decoding inside the attention kernel to avoid extra read/write?
- Can we compare with more lossless compression baselines like DF11 or ECF8? The authors mention some of them in the paper, yet never have a head-to-head comparison.
Nitpicking:
- I think using the term "word" in the definition is a bit unconventional for writing within this direction "We partition U into G= ⌈N/P⌉pages of P = 256 words".
6 comments
[2/2] Response to Reviewer oy77
Question 2What are the rates of raw fallback broken down by K vs V, layer, and head, and how do they depend on the page layout?
Author's response. We measured the percentage of pages stored raw across all layers and heads, nine inputs per model, two tasks, and 4K/8K/16K contexts.
| Model | Tensor | Head–token–dimension | Token–head–dimension |
|---|---|---|---|
| Llama | K | 0.00783% | 0.00339% |
| Llama | V | 0.00192% | 0.00028% |
| Qwen | K | 0.30906% | 0.41349% |
| Qwen | V | 0.00123% | 0.00021% |
The 16-entry palette covers 99.69% of Qwen K pages in the default layout and over 99.99% in the other groups. Qwen K varies most across layers (0.00698–8.38497% fallback). By head, fallback spans 0.05219–0.61718% (element-weighted). By task, it is 0.32322% on HotpotQA and 0.28074% on QMSum.
Weakness 4The runtime adds reconstruction traffic on every decoding step. Per Section 3.3 and Eq. 9–10, each layer's sealed slabs are decoded into flat scratch, copied into a shared full-layer BF16 workspace, and only then consumed by an unmodified SDPA kernel. Because a single workspace is shared across layers (and, per the text, across decoding steps), this reconstruction appears to be repeated for every layer at every step. Relative to Raw, which reads the cache once, this adds at least one full BF16 write, a scratch-to-workspace copy, and a second read of the reconstructed data. I believe a fused approach that decodes inside the attention kernel (analogous to ZipServ's fused decompression for weights, which the paper cites) would eliminate the workspace traffic and the workspace memory.
Question 3Following up on 4, can we try to fuse decoding inside the attention kernel to avoid extra read/write?
Author's response. Thank you for identifying the extra write and copy on each decoding step. Each step reuses the allocation and reconstructs every layer. Your suggestion eliminates a full BF16 writeback. We tested tile decoding inside Attention.
| Single-layer measurement | Condition | Fused prototype |
|---|---|---|
| Peak allocation vs Raw SDPA | 8K / 16K | −20.03% / −18.60% |
| Host latency / Raw SDPA | hot / rotating inputs | 3.80× / 3.63× |
| Host latency / separate decode + SDPA | hot / rotating inputs | 1.42× / 1.85× |
Fusion reduced peak memory at both lengths. We chose separate decode + Flash SDPA for generation because it was faster in both conditions. We will test whether coalesced loads and overlapping tile decode with Attention retain these savings at lower latency, checking bit-exact KV recovery.
Question 4Can we compare with more lossless compression baselines like DF11 or ECF8? The authors mention some of them in the paper, yet never have a head-to-head comparison.
Author's response. We tested official codecs via KV adapters in five paired processes. Single-layer tests used identical real 16K K/V, verified bitwise recovery and Flash SDPA. Cells give KVRecode / external codec in ms or MiB of encoded K/V.
| External codec | Input / GPU | Encode | Decode | Decode + Attn | Size (MiB) |
|---|---|---|---|---|---|
| DietGPU | Llama / A800 | 1.121 / 0.351 | .214 / .251 | .279 / .314 | 51.00 / 43.80 |
| DietGPU | Qwen / A800 | 1.114 / 0.352 | .214 / .254 | .279 / .306 | 51.00 / 43.97 |
| DietGPU | Mistral / 4090D | .688 / .384 | .181 / .293 | .244 / .349 | 51.00 / 43.77 |
| DFloat11 | Llama / 4090D | .688 / 24,748 | .181 / .194 | .243 / .261 | 51.00 / 43.91 |
| DFloat11 | Qwen / 4090D | .689 / 25,288 | .181 / .195 | .242 / .263 | 51.00 / 44.02 |
| DFloat11 | Mistral / 4090D | .688 / 24,637 | .181 / .194 | .244 / .262 | 51.00 / 43.84 |
- Decode + Attention. Paired latency fell by 10.89%, 8.54%, and 30.19% versus DietGPU on Llama, Qwen, and Mistral, and by 7.02%, 7.90%, and 7.10% versus DFloat11. DietGPU encoded faster and produced smaller objects. A800 used a shared host and 80 MiB workspace. With 64 MiB and overflow allocations, both A800 reductions were 4.26%. Charging encoded K/V, workspace and BF16 decode output gives 115.00 / 187.80 MiB on Llama and 115.00 / 187.97 MiB on Qwen for KVRecode / DietGPU (one A800 layer). DFloat11 encode includes CPU tables and transfers.
- Full generation. On Mistral-7B, both adapters used the same nine 16K inputs and 512-token outputs. Each matched the reference on 55/55 pairs including warmups.
| Method | Request (s) | TTFT (s) | TPOT (ms) | Cache (MiB) | Peak allocated / reserved (MiB) |
|---|---|---|---|---|---|
| DietGPU | 19.532 | 2.392 | 33.542 | 1631.68 | 17,274 / 18,573 |
| KVRecode | 16.705 | 2.401 | 27.994 | 1788.51 | 17,428 / 18,300 |
KVRecode reduced request latency by 14.47% (95% CI 14.44–14.49%) and TPOT by 16.54% with 9.61% more cache storage.
ECF8's released codec accepts FP8 E4M3 weights. Once ECF8 releases a BF16 KV interface, we will compare it with KVRecode on the same tensors and benchmarks, checking bitwise recovery before timing.
Thank the authors for the additional experiments. I'm still not very convinced by the novelty argument, but I will raise my score.
[1/2] Response to Reviewer oy77
Author's response. Thank you for identifying where the method loses the reader. We will explain the three operations in the order a request encounters them. New KV is appended, a full slab is sealed, and one layer at a time is reconstructed for attention. We will move the buffer and indexing details to the appendix and replace “words” with “BF16 values”.
Author's response. Thank you for identifying these precedents. Our question is how to keep a growing KV cache compressed on the GPU when every decoding step needs the exact BF16 values.
We changed one runtime choice at a time against unmodified KVRecode with 16K input tokens, 512 generated tokens and five paired processes.
The shared workspace more than halves the resident cache footprint compared with per-layer workspaces. Sealing once avoids the request latency added by repeated encoding.
Author's response. Five more sessions fit on one 80 GB A800. Using Qwen3-8B with 16K input and 512 output tokens, we kept every session’s KV resident and advanced the sessions in round-robin order.
This raises measured resident capacity by 18.52%. At an equal load of 26 sessions, whole-model peak allocation fell from 77.23 to 66.13 GiB. In a separate pause/resume test on Qwen, the first token took 42.85 ms with resident KVRecode and 128.67 ms after a pinned-CPU restore.
We also integrated KVRecode into the vLLM V1 scheduler and generation loop. KVRecode and its Raw-matched control both used eager execution and per-request Flash SDPA. With one or four requests in flight, KVRecode reduced peak active-cache bytes by 17.88–19.48%, and all 540 paired outputs matched (180 measured, 360 warmup).
We additionally tested Mistral-7B-Instruct-v0.3. Cache storage fell 15.32% relative to Raw, with identical generated tokens in all 55 paired requests (including warmups).