acceptodds
Under review as a conference paper at ICLR 2027

KVRecode: Lossless KV Cache Compression with Local Coding and Buffer Reuse

Abstract

Long-context generation places increasing memory demands on the key and value (KV) cache. However, compressing this cache losslessly requires repeated reconstruction before attention, making inference cost depend on both the stored representation and its runtime. To address this challenge, we introduce KVRecode, a lossless BF16 cache representation and runtime that combines complete local exponent palettes with reusable reconstruction buffers. KVRecode encodes exponents with fixed-width indices into page-local tables while preserving sign and mantissa bits, retaining pages in raw form when their exponent sets exceed the palette capacity. This construction enables exact recovery without calibration or per-value exception handling. As the cache grows, the runtime seals completed token segments and reconstructs successive layers into shared buffers, limiting reconstruction memory while preserving the existing attention backend. Across two 8B models and nine natural inputs at 4K, 8K, and 16K contexts, KVRecode reduces request latency for 512-token outputs by 10.45% relative to SplitZip under matched resident-cache execution. It also uses 15.19% less resident cache memory than Raw, including tails and reconstruction workspaces. KVRecode preserves cache contents and generated outputs exactly. These results demonstrate the benefits of coupling local coding with buffer reuse for efficient lossless cache access during generation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, lossless compression, large language model inference, long-context generation, memory-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oy77OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes KVRecode, a lossless compression format and runtime for BF16 KV caches that stay resident on the GPU during generation. Each 256-word page of a K or V tensor is checked for how many distinct exponent values it contains. This gives a best-case object ratio of about 0.797. No calibration and no per-value exception handling are needed.

Critical strengths

None

Critical weaknesses
  1. While I appreciate the authors' detailed writing in the method section, I think it's a bit too deep in system-level programming, which might be hard for the unfamiliar readers to comprehend. I would recommend a more reader-friendly version of the method in the main text, while the existing one should be used as detailed system-info level in the appendix.
  2. Novelty is limited, and the components are not evaluated individually. Palette (dictionary) coding with fixed-width indices and a raw fallback is a standard data-compression technique. Splitting off the exponent bits follows SplitZip. Reusing a shared layer-sized decode buffer is already in ECF8. Keeping recent tokens raw resembles KIVI's full-precision residual. The contribution is mainly a careful engineering combination of these, and the ML insight relevant to ICLR is thin.
  3. Experimental scope is limited. Everything runs on one consumer GPU (RTX 4090D), at batch size 1, in HuggingFace Transformers with Flash SDPA, without integration into a paged-attention serving engine (vLLM, SGLang). The study also only uses two models.
  4. The runtime adds reconstruction traffic on every decoding step. Per Section 3.3 and Eq. 9–10, each layer's sealed slabs are decoded into flat scratch, copied into a shared full-layer BF16 workspace, and only then consumed by an unmodified SDPA kernel. Because a single workspace is shared across layers (and, per the text, across decoding steps), this reconstruction appears to be repeated for every layer at every step. Relative to Raw, which reads the cache once, this adds at least one full BF16 write, a scratch-to-workspace copy, and a second read of the reconstructed data. I believe a fused approach that decodes inside the attention kernel (analogous to ZipServ's fused decompression for weights, which the paper cites) would eliminate the workspace traffic and the workspace memory.
Questions for authors
  1. Can authors show a setting where the memory saving pays off: higher throughput at larger batch sizes, more concurrent requests under a fixed memory limit, or longer contexts that Raw cannot fit?
  2. What are the rates of raw fallback broken down by K vs V, layer, and head, and how do they depend on the page layout?
  3. Following up on 4, can we try to fuse decoding inside the attention kernel to avoid extra read/write?
  4. Can we compare with more lossless compression baselines like DF11 or ECF8? The authors mention some of them in the paper, yet never have a head-to-head comparison.

Nitpicking:

  1. I think using the term "word" in the definition is a bit unconventional for writing within this direction "We partition U into G= ⌈N/P⌉pages of P = 256 words".
6 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer oy77

Weakness 1 While I appreciate the authors' detailed writing in the method section, I think it's a bit too deep in system-level programming, which might be hard for the unfamiliar readers to comprehend. I would recommend a more reader-friendly version of the method in the main text, while the existing one should be used as detailed system-info level in the appendix.

Nitpicking 1 I think using the term "word" in the definition is a bit unconventional for writing within this direction "We partition U into G= ⌈N/P⌉pages of P = 256 words".

Author's response. Thank you for identifying where the method loses the reader. We will explain the three operations in the order a request encounters them. New KV is appended, a full slab is sealed, and one layer at a time is reconstructed for attention. We will move the buffer and indexing details to the appendix and replace “words” with “BF16 values”.


Weakness 2 Novelty is limited, and the components are not evaluated individually. Palette (dictionary) coding with fixed-width indices and a raw fallback is a standard data-compression technique. Splitting off the exponent bits follows SplitZip. Reusing a shared layer-sized decode buffer is already in ECF8. Keeping recent tokens raw resembles KIVI's full-precision residual. The contribution is mainly a careful engineering combination of these, and the ML insight relevant to ICLR is thin.

Author's response. Thank you for identifying these precedents. Our question is how to keep a growing KV cache compressed on the GPU when every decoding step needs the exact BF16 values.

  • What determines which pages compress? SplitZip uses calibrated exponent codebooks and sparse escapes. KVRecode uses the exponents actually present in each 256-value page. Up to 16 distinct exponents fit a complete local palette. A page exceeding 16 stays raw. Both paths preserve every BF16 bit.
  • When should new KV be encoded? KIVI keeps a full-precision residual alongside a quantized cache. KVRecode appends to a raw tail and seals it once at 256 tokens. Older and newer KV therefore retain the same precision while encoding work is paid once per slab.
  • How long is a reconstructed layer needed? ECF8 shares a decode buffer for FP8 weights. KVRecode reuses one layer-sized BF16 workspace across layers and generation steps while the sealed KV remains compressed. As a result, only the current layer needs a full reconstruction for attention.

We changed one runtime choice at a time against unmodified KVRecode with 16K input tokens, 512 generated tokens and five paired processes.

Change from KVRecodeMetric relative to KVRecodeLlamaQwen
Keep a reconstruction workspace per layerResident cache bytes2.144×2.155×
Re-encode the active slab on every appendRequest latency+28.59%+38.50%

The shared workspace more than halves the resident cache footprint compared with per-layer workspaces. Sealing once avoids the request latency added by repeated encoding.


Weakness 3 Experimental scope is limited. Everything runs on one consumer GPU (RTX 4090D), at batch size 1, in HuggingFace Transformers with Flash SDPA, without integration into a paged-attention serving engine (vLLM, SGLang). The study also only uses two models.

Question 1 Can authors show a setting where the memory saving pays off: higher throughput at larger batch sizes, more concurrent requests under a fixed memory limit, or longer contexts that Raw cannot fit?

Author's response. Five more sessions fit on one 80 GB A800. Using Qwen3-8B with 16K input and 512 output tokens, we kept every session’s KV resident and advanced the sessions in round-robin order.

MethodSessions completing the workloadNext observed memory limit
Raw BF1627 (1/1 run)Allocating the 28th cache during restoration
KVRecode32 (5/5 runs)Continuing generation with 33 sessions

This raises measured resident capacity by 18.52%. At an equal load of 26 sessions, whole-model peak allocation fell from 77.23 to 66.13 GiB. In a separate pause/resume test on Qwen, the first token took 42.85 ms with resident KVRecode and 128.67 ms after a pinned-CPU restore.

We also integrated KVRecode into the vLLM V1 scheduler and generation loop. KVRecode and its Raw-matched control both used eager execution and per-request Flash SDPA. With one or four requests in flight, KVRecode reduced peak active-cache bytes by 17.88–19.48%, and all 540 paired outputs matched (180 measured, 360 warmup).

vLLM output tokens/sRaw-matched, C=1KVRecode, C=1Raw-matched, C=4KVRecode, C=4
Llama46.3136.5871.9650.54
Qwen41.3533.2765.2945.63

We additionally tested Mistral-7B-Instruct-v0.3. Cache storage fell 15.32% relative to Raw, with identical generated tokens in all 55 paired requests (including warmups).

AuthorsOpenReview ↗

[2/2] Response to Reviewer oy77

Question 2 What are the rates of raw fallback broken down by K vs V, layer, and head, and how do they depend on the page layout?

Author's response. We measured the percentage of pages stored raw across all layers and heads, nine inputs per model, two tasks, and 4K/8K/16K contexts.

ModelTensorHead–token–dimensionToken–head–dimension
LlamaK0.00783%0.00339%
LlamaV0.00192%0.00028%
QwenK0.30906%0.41349%
QwenV0.00123%0.00021%

The 16-entry palette covers 99.69% of Qwen K pages in the default layout and over 99.99% in the other groups. Qwen K varies most across layers (0.00698–8.38497% fallback). By head, fallback spans 0.05219–0.61718% (element-weighted). By task, it is 0.32322% on HotpotQA and 0.28074% on QMSum.


Weakness 4 The runtime adds reconstruction traffic on every decoding step. Per Section 3.3 and Eq. 9–10, each layer's sealed slabs are decoded into flat scratch, copied into a shared full-layer BF16 workspace, and only then consumed by an unmodified SDPA kernel. Because a single workspace is shared across layers (and, per the text, across decoding steps), this reconstruction appears to be repeated for every layer at every step. Relative to Raw, which reads the cache once, this adds at least one full BF16 write, a scratch-to-workspace copy, and a second read of the reconstructed data. I believe a fused approach that decodes inside the attention kernel (analogous to ZipServ's fused decompression for weights, which the paper cites) would eliminate the workspace traffic and the workspace memory.

Question 3 Following up on 4, can we try to fuse decoding inside the attention kernel to avoid extra read/write?

Author's response. Thank you for identifying the extra write and copy on each decoding step. Each step reuses the allocation and reconstructs every layer. Your suggestion eliminates a full BF16 writeback. We tested tile decoding inside Attention.

Single-layer measurementConditionFused prototype
Peak allocation vs Raw SDPA8K / 16K−20.03% / −18.60%
Host latency / Raw SDPAhot / rotating inputs3.80× / 3.63×
Host latency / separate decode + SDPAhot / rotating inputs1.42× / 1.85×

Fusion reduced peak memory at both lengths. We chose separate decode + Flash SDPA for generation because it was faster in both conditions. We will test whether coalesced loads and overlapping tile decode with Attention retain these savings at lower latency, checking bit-exact KV recovery.


Question 4 Can we compare with more lossless compression baselines like DF11 or ECF8? The authors mention some of them in the paper, yet never have a head-to-head comparison.

Author's response. We tested official codecs via KV adapters in five paired processes. Single-layer tests used identical real 16K K/V, verified bitwise recovery and Flash SDPA. Cells give KVRecode / external codec in ms or MiB of encoded K/V.

External codecInput / GPUEncodeDecodeDecode + AttnSize (MiB)
DietGPULlama / A8001.121 / 0.351.214 / .251.279 / .31451.00 / 43.80
DietGPUQwen / A8001.114 / 0.352.214 / .254.279 / .30651.00 / 43.97
DietGPUMistral / 4090D.688 / .384.181 / .293.244 / .34951.00 / 43.77
DFloat11Llama / 4090D.688 / 24,748.181 / .194.243 / .26151.00 / 43.91
DFloat11Qwen / 4090D.689 / 25,288.181 / .195.242 / .26351.00 / 44.02
DFloat11Mistral / 4090D.688 / 24,637.181 / .194.244 / .26251.00 / 43.84
  • Decode + Attention. Paired latency fell by 10.89%, 8.54%, and 30.19% versus DietGPU on Llama, Qwen, and Mistral, and by 7.02%, 7.90%, and 7.10% versus DFloat11. DietGPU encoded faster and produced smaller objects. A800 used a shared host and 80 MiB workspace. With 64 MiB and overflow allocations, both A800 reductions were 4.26%. Charging encoded K/V, workspace and BF16 decode output gives 115.00 / 187.80 MiB on Llama and 115.00 / 187.97 MiB on Qwen for KVRecode / DietGPU (one A800 layer). DFloat11 encode includes CPU tables and transfers.
  • Full generation. On Mistral-7B, both adapters used the same nine 16K inputs and 512-token outputs. Each matched the reference on 55/55 pairs including warmups.
MethodRequest (s)TTFT (s)TPOT (ms)Cache (MiB)Peak allocated / reserved (MiB)
DietGPU19.5322.39233.5421631.6817,274 / 18,573
KVRecode16.7052.40127.9941788.5117,428 / 18,300

KVRecode reduced request latency by 14.47% (95% CI 14.44–14.49%) and TPOT by 16.54% with 9.61% more cache storage.

ECF8's released codec accepts FP8 E4M3 weights. Once ECF8 releases a BF16 KV interface, we will compare it with KVRecode on the same tensors and benchmarks, checking bitwise recovery before timing.

Thank the authors for the additional experiments. I'm still not very convinced by the novelty argument, but I will raise my score.

Reviewer_nHX1OpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVRecode, a lossless BF16 KV-cache compression method using local exponent palettes and reusable reconstruction buffers. The compressed cache remains on the GPU and is reconstructed before attention, while newly appended tokens are temporarily stored uncompressed. Experiments evaluate memory usage, generation latency, and output equivalence against uncompressed caching and other lossless codecs.

Critical strengths
  1. KVRecode preserves BF16 cache values exactly, with matching generated outputs in the reported evaluations.

  2. The evaluation accounts for reconstruction workspaces and append tails, providing a useful view of the memory and latency costs beyond compressed payload size alone.

Critical weaknesses
  1. The practical benefit of the memory–latency trade-off is insufficiently demonstrated. For 512-token outputs, Table 1 reports approximately 15% lower resident cache memory but 25% higher request latency than uncompressed BF16 caching. Whole-model savings are smaller: Table 17 shows only about a 1.7% reduction in peak allocated memory for Llama at 16K context. The experiments do not establish when these savings justify the additional latency.

  2. The evaluation does not establish an advantage over KV-cache offloading under GPU memory constraints. Comparisons with resident-cache codecs do not answer whether keeping compressed KV on the GPU is preferable to transferring exact KV values from host memory. A comparison under the same GPU memory budget is needed to establish its deployment value.

  3. The batch-size-one evaluation leaves serving scalability unclear. Repeated cache reconstruction introduces additional computation and memory traffic, whose impact may change with concurrent requests. The paper does not demonstrate whether compression enables larger batches or higher aggregate throughput that compensates for this overhead.

  4. The mechanism diagram in Figure 2 is visually cluttered. Dense labels and decorative elements make the encoding and reconstruction flow difficult to follow.

Questions for authors
  1. Under what memory-constrained settings do the cache savings outweigh the reconstruction overhead, particularly compared with exact KV-cache offloading?

  2. Can KVRecode support larger batches or longer contexts than uncompressed caching, and what performance does it achieve in those settings?

10 comments
AuthorsOpenReview ↗

Response to Reviewer nHX1

Thank you for asking when a smaller cache changes what a server can do. We measured how many sessions fit and how quickly a paused session can return its first token.

Weakness 1 The practical benefit of the memory–latency trade-off is insufficiently demonstrated. For 512-token outputs, Table 1 reports approximately 15% lower resident cache memory but 25% higher request latency than uncompressed BF16 caching. Whole-model savings are smaller: Table 17 shows only about a 1.7% reduction in peak allocated memory for Llama at 16K context. The experiments do not establish when these savings justify the additional latency.

Question 1 Under what memory-constrained settings do the cache savings outweigh the reconstruction overhead, particularly compared with exact KV-cache offloading?

Author's response. On one 80 GB A800, we kept all Qwen3-8B session caches on the GPU and resumed the sessions in rotation. Each had a 16K-token input and generated 512 tokens.

MethodSessions completing 512 tokensSuccessful runsWhole-model peak allocated
Raw BF16271/177.913 GiB
KVRecode325/577.788 GiB

Five more sessions (+18.52%) could resume from GPU memory on the same card. Raw ran out of memory allocating the 28th cache while restoring a pool of 32. KVRecode reached its limit during generation with 33 sessions.

Weakness 2 The evaluation does not establish an advantage over KV-cache offloading under GPU memory constraints. Comparisons with resident-cache codecs do not answer whether keeping compressed KV on the GPU is preferable to transferring exact KV values from host memory. A comparison under the same GPU memory budget is needed to establish its deployment value.

Author's response. We compared two uses of exact CPU offloading.

  • Offloading during generation. On a 4090D, all methods used the same 20 GiB PyTorch allocator budget, including the model and runtime. We timed 16K/512 requests across five paired processes. Offload includes the adapter's explicit synchronization.
Request latency (s)RawOffloadKVRecode
Llama-3.1-8B13.58115.3117.15
Qwen3-8B16.22129.4519.99

At 20 GiB, KVRecode cut request latency by 85.1% on Llama and 84.6% on Qwen relative to this adapter. At 18 and 19 GiB, only Offload completed.

  • Restoring a paused session once. We compared keeping KV resident with restoring the whole live prefix from pinned CPU memory on A800/Qwen3-8B. Each of three 16K inputs was paused at two positions across five paired processes. Times start at resume. Initial setup and offload were timed separately.
Time from resume (ms)Raw residentCPU restoreKVRecode
First token28.73128.6742.85
First 8 tokens228.91329.07294.88
All 128 tokens3661.843763.964630.27

The first token arrived 66.70% sooner with KVRecode than with CPU restore (latency ratio 0.333, 95% CI [0.329, 0.337]). That advantage persisted through the eighth token of the same continuation. CPU restore finished the full 128 tokens sooner.

Weakness 3 The batch-size-one evaluation leaves serving scalability unclear. Repeated cache reconstruction introduces additional computation and memory traffic, whose impact may change with concurrent requests. The paper does not demonstrate whether compression enables larger batches or higher aggregate throughput that compensates for this overhead.

Question 2 Can KVRecode support larger batches or longer contexts than uncompressed caching, and what performance does it achieve in those settings?

Author's response. We integrated KVRecode into vLLM 0.11 V1. Both the Raw control and KVRecode used eager execution and per-request Flash SDPA. We tested one and four concurrent requests at 16K input and 512 output tokens with five paired processes.

Four concurrent requestsPeak active KV (GiB), Raw / KVRecodeOutput tokens/s, Raw / KVRecode
Llama-3.1-8B8.31 / 6.7071.96 / 50.54
Qwen3-8B9.35 / 7.5365.29 / 45.63

The vLLM scheduler admitted, advanced and released requests with 17.88%–19.48% less active-cache memory across both settings, including tails and reconstruction buffers. All 540 paired outputs matched the Raw control (180 measured, 360 warmup). In the longer-context check, both methods completed all three natural-length 32K inputs.

Weakness 4 The mechanism diagram in Figure 2 is visually cluttered. Dense labels and decorative elements make the encoding and reconstruction flow difficult to follow.

Author's response. Thank you for pointing out the clutter. We will simplify Figure 2 to make the encoding and reconstruction paths easier to follow.

Thank you for the additional experiments. The increased resident-cache capacity is useful, but it remains unclear whether it justifies the substantial performance penalty compared with optimized offloading systems such as Strata [1]. Existing evaluations such as Strata model requests arriving over time and users returning after pauses between conversation turns. The fixed-pool, round-robin experiment does not capture these basic serving conditions, leaving its practical relevance unclear. A more realistic workload and a stronger offloading baseline are needed to establish the benefit. I therefore maintain my current score.

[1] Strata: Hierarchical Context Caching for Long Context Language Model Serving. OSDI 2026. https://www.usenix.org/conference/osdi26/presentation/xie-zhiqiang

AuthorsOpenReview ↗

Response to Reviewer nHX1

Follow-up: Thank you for the additional experiments. The increased resident-cache capacity is useful, but it remains unclear whether it justifies the substantial performance penalty compared with optimized offloading systems such as Strata [1]. Existing evaluations such as Strata model requests arriving over time and users returning after pauses between conversation turns. The fixed-pool, round-robin experiment does not capture these basic serving conditions, leaving its practical relevance unclear. A more realistic workload and a stronger offloading baseline are needed to establish the benefit. I therefore maintain my current score.

[1] Strata: Hierarchical Context Caching for Long Context Language Model Serving, OSDI 2026

Author's response. Thank you for asking whether the extra cache capacity pays off when users return. To address this concern, we compared KVRecode directly with Strata as new sessions arrived and users returned after a pause. Compared with Strata configured with a 32 GB pinned host KV pool, KVRecode used 32 GB less dedicated host memory. In our long-document repeat, KVRecode reduced median paired completion time by 11.6% in group A and 14.4% in group B. On short conversations, the median extra wait for the first token was 5 ms in each group.

Serving conditions. Each system had a 24 GiB GPU KV budget and ran BF16 Llama-3.1-8B-Instruct on the same physical A800 as its comparator. New sessions arrived according to a Poisson process at 0.25 sessions/s, with up to four requests in flight. Each user returned 60 s after their own previous answer finished. The next question included that engine's generated answer, and timing included queueing from the scheduled arrival. We used the Strata first author's public SGLang branch with its hierarchical scheduler and native CUDA restore path. With --disable-overlap-schedule, restoring a 14,972-token prefix reproduced all 32 checked tokens from cold generation. Strata implementation

1. Long-document follow-ups. Users asked three published questions about each of 48 LooGLE documents, with initial inputs of 13.5K–32.2K tokens. Groups A and B each contributed 48 returns. We repeated the same arrival schedules and documents in fresh service processes. The system columns give each system's median. For the savings columns, we computed the difference and percentage reduction for each matched return, then took the median.

LooGLE runStrata time to full answerKVRecode time to full answerTime saved by KVRecode vs. StrataKVRecode latency reduction vs. Strata
Initial A7.279 s7.159 s0.447 s5.7%
Initial B11.065 s9.784 s1.630 s13.7%
Repeat A7.184 s6.589 s0.605 s11.6%
Repeat B10.012 s7.700 s0.965 s14.4%
LooGLE repeatStrataKVRecodeKVRecode benefit vs. Strata
Median time to first token, A / B3.285 / 4.424 s2.814 / 3.500 s0.541 / 0.830 s earlier on paired returns
Returns where the system delivered the first token sooner28/9668/96KVRecode led on 40 more returns
Returns where the system finished first34/9662/96KVRecode led on 28 more returns
GPU-board energy per completed request, A / B995.6 / 1066.3 J928.8 / 1004.6 J6.7% / 5.8% less energy

Energy covers the entire replay, including user pauses, and is divided by all completed requests.

2. Short conversations. We replayed the first three consecutive human questions from each of 48 CoQA conversations under the same arrival and return schedule.

The GPU peaks below include each engine's allocation policy. In a separate same-engine comparison, KVRecode used 10.5% and 11.1% fewer final session-cache bytes than raw BF16 in groups A and B. Both served GPU-resident prefixes on all 96 returns.

CoQA measurement, A / BStrataKVRecodeKVRecode vs. Strata
Pinned host memory reserved for historical KV32 GB0 GB32 GB less reserved
Sampled peak GPU memory40.62 / 40.65 GiB18.42 / 18.46 GiBabout 22.2 GiB lower in each group
Median time to first token21 / 22 ms27 / 26 ms5 / 5 ms later on paired returns
Median time to full answer78 / 81 ms111 / 146 ms41 / 57 ms later on paired returns

KVRecode left 32 GB of pinned host memory available for other work compared with this Strata configuration. On 90 of 96 short-chat returns, the extra wait for a complete answer was at most 250 ms. On the long-document replay, KVRecode delivered the complete answer before Strata on 62 of 96 returns.

If these results address your concern, we would be grateful if you would consider raising your score.

Reviewer_A9qtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes KVRecode, a lossless BF16 KV-cache representation plus runtime. Each 256-word page stores a complete local exponent palette (≤16 distinct exponents) with packed 4-bit indices, keeping sign/mantissa bits intact; pages exceeding capacity fall back to raw. Reconstruction is exact integer bit-assembly, no calibration, no exceptions. The runtime adds sealed 256-token slabs, a raw append tail, and a shared layer-sized BF16 workspace reused across layers and decoding steps, feeding the existing Flash SDPA backend. Experiments on Llama-3.1-8B and Qwen3-8B at 4K/8K/16K report ~15% resident-memory saving vs Raw and 10.45% lower request latency vs a resident-cache SplitZip adapter, with bit-exact outputs.

Critical strengths
  1. Clean and well-specified lossless format with real runtime co-design. The page-local palette is fully specified and provably invertible, including NaN payloads, signed zeros, and subnormals; whole-page raw fallback avoids exception handling and calibration. More importantly, the paper does not stop at a codec — the sealed-slab/tail/shared-workspace design tackles the actual obstacle of lossless KV compression, namely that the cache must be reconstructed at every decoding step, so decode cost matters as much as ratio. This coupling makes the latency results meaningful rather than a storage-only claim.

  2. Careful matched-condition evaluation with exactness checks. Memory accounting charges tails, workspaces, descriptors, and alignment; timing includes online encode/decode and allocation sync. Exactness is verified by exhaustive 65,536-pattern codec gates, native K/V hash comparison, token-level agreement over all 2,160 requests, and unchanged LongBench scores. The paired bootstrap and the honest per-cell reporting (including cells where KVRecode is slower than SplitZip) increase confidence in the headline numbers.

Critical weaknesses
  1. The baseline choice is weak for the stated goal, and the framing overstates the practical case. The main comparison is SplitZip, a transfer-oriented codec adapted by the authors into a resident runtime it was not designed for. Against the actually relevant baseline — Raw BF16 — KVRecode is 12.7–25.2% slower in request latency and 22–28% slower in TPOT (Table 1b), while saving only ~15% memory. The abstract highlights the 10.45% win over SplitZip but pushes the Raw-relative slowdown into a table. Also, Table 17 shows whole-model reserved peak is higher with KVRecode than Raw at 16K (20,216 vs 19,676 MiB on Llama), because workspace and scratch offset the cache saving. This weakens the central motivation.

  2. Narrow scale and workload coverage. All experiments are batch-one greedy decoding on one RTX 4090D, two 8B models, nine inputs from only two LongBench tasks. The method depends on ≤16 distinct exponents per 256-word page, which is a data-distribution property; fallback rates are reported only for two frozen 16K inputs (Figure 4) and never broken down by layer, head, K vs V, or task. Batch >1, tensor parallelism (which shards K/V heads and changes page composition), and other architectures are untested. Since the whole value is a ~15% memory saving, the absence of any memory-bound regime (e.g., a batch size that OOMs with Raw but fits with KVRecode) leaves practical significance unestablished.

Questions for authors

Raw-relative trade-off. Table 1b shows KVRecode is 12.7–25.2% slower than Raw. Under what deployment condition is a 15% memory saving worth a 25% latency regression? Does the trade flip at larger batch sizes or under cache offloading? A batch-size sweep identifying any crossover would materially affect the assessment.

2 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer A9qt

Weakness 1 The baseline choice is weak for the stated goal, and the framing overstates the practical case. The main comparison is SplitZip, a transfer-oriented codec adapted by the authors into a resident runtime it was not designed for. Against the actually relevant baseline — Raw BF16 — KVRecode is 12.7–25.2% slower in request latency and 22–28% slower in TPOT (Table 1b), while saving only ~15% memory. The abstract highlights the 10.45% win over SplitZip but pushes the Raw-relative slowdown into a table. Also, Table 17 shows whole-model reserved peak is higher with KVRecode than Raw at 16K (20,216 vs 19,676 MiB on Llama), because workspace and scratch offset the cache saving. This weakens the central motivation.

Question 1 Raw-relative trade-off. Table 1b shows KVRecode is 12.7–25.2% slower than Raw. Under what deployment condition is a 15% memory saving worth a 25% latency regression? Does the trade flip at larger batch sizes or under cache offloading? A batch-size sweep identifying any crossover would materially affect the assessment.

Author's response. Thank you for recognizing our exactness checks. Your question led us to measure how many complete sessions one GPU can keep resident. In the submitted single-session test, KVRecode saved about 15% of resident cache while increasing request latency by 12.7–25.2% relative to Raw. On Llama at 16K, peak reserved memory rose from 19,676 to 20,216 MiB. We therefore measured a workload where resident caches occupy most of the card.

  • Resident capacity. All Qwen3-8B BF16 caches stayed on one 80 GB A800 during round-robin continuation (16K input, 512 output).
MethodResident sessionsSuccessful runsPeak allocated, GiB
Raw271/177.913
KVRecode325/577.788

KVRecode completed five more sessions (+18.52%). The 32-session workload contained 11 HotpotQA, 3 QMSum and 18 GovReport documents. Raw hit OOM allocating cache 28 in a target pool of 32. KVRecode hit OOM resuming 33 sessions.

  • Performance at equal load. We compared 26 sessions using an expandable allocator (five paired processes).
MetricRawKVRecodeChange
Peak allocated, GiB77.2366.13−14.37%
Peak reserved, GiB77.3775.77−2.06%
Cache, GiB60.3350.80−15.80%
Output tokens/s32.2725.75−20.21%

Whole-model peak allocation fell by 11.10 GiB. Raw retained higher throughput. For a server keeping paused sessions ready to resume, the gain was five more complete caches on the same card.

  • Exact CPU offload. All methods used 16K/512, five paired processes and a 20 GiB PyTorch allocator cap on 4090D. Offload includes transfer sync.
ModelRaw, sCPU offload, sKVRecode, s
Llama13.58115.3117.15
Qwen16.22129.4519.99

KVRecode finished each request sooner than this synchronized offload adapter on both models. At 18 and 19 GiB, only the offload adapter completed.

  • External codec. We tested official DietGPU in our common resident adapter on Mistral-7B-v0.3 (16K/512, five paired processes).
MethodRequest, sTPOT, msCache, MiB
Raw13.14921.0912112.0
DietGPU19.53233.5421631.7
KVRecode16.70527.9941788.5

Versus DietGPU, KVRecode used 9.61% more cache with 14.47% lower latency (95% CI 14.44–14.49% reduction) and 16.54% lower TPOT. We will state the Raw trade-off in the abstract and main tables and label the SplitZip adapter.

AuthorsOpenReview ↗

[2/2] Response to Reviewer A9qt

Weakness 2 Narrow scale and workload coverage. All experiments are batch-one greedy decoding on one RTX 4090D, two 8B models, nine inputs from only two LongBench tasks. The method depends on ≤16 distinct exponents per 256-word page, which is a data-distribution property; fallback rates are reported only for two frozen 16K inputs (Figure 4) and never broken down by layer, head, K vs V, or task. Batch >1, tensor parallelism (which shards K/V heads and changes page composition), and other architectures are untested. Since the whole value is a ~15% memory saving, the absence of any memory-bound regime (e.g., a batch size that OOMs with Raw but fits with KVRecode) leaves practical significance unestablished.

Author's response. Thank you for asking how the results extend across hardware, models and workloads.

1. Batch and resident capacity on one A800 (80 GB). Static batches used 16K inputs, 512 output tokens and a 40 GiB PyTorch allocator budget. Entries are Raw / KVRecode.

ModelLargest tested batch completedBatch-4 cache (GiB)Batch-4 output tokens/s
Llama-3.1-8B4 / 48.25 / 6.89122.14 / 51.94
Qwen3-8B6 / 49.28 / 7.7297.93 / 46.21

At batch four, KVRecode used less cache and Raw had higher throughput. With restored Qwen sessions (Part 1), resident capacity rose from 27 to 32.

2. More documents and longer contexts. After GPU prefill, each session generated 512 tokens in rotation. The 16K pool grew from 9 to 39 documents (25 GovReport, 11 HotpotQA, 3 QMSum).

Input poolModelSessions completed by bothCache GiB, Raw / KVRecode
16K, 39 documentsLlama2959.81 / 50.65
16K, 39 documentsQwen2660.33 / 50.80
32K, 3 GovReport documentsLlama312.19 / 10.30
32K, 3 GovReport documentsQwen313.71 / 11.53

3. Model, engine and task coverage. All three model families use grouped-query attention.

EvaluationScopeMatched result
Mistral-7B-Instruct-v0.3, RTX 4090D6 HotpotQA + 3 QMSum, 16K/512, five paired processesCache 2,112.00 → 1,788.51 MiB (−15.32%). Request time 13.15 → 16.71 s. All 55 output pairs matched, including warmups.
vLLM V1, A800Llama and Qwen, concurrency 1 and 4, eager per-request Flash SDPA17.88–19.48% lower peak active cache than matched Raw. All 540 output pairs matched (180 measured, 360 warmup).
Original LongBench quality panel25 inputs per task per model, 8 tasks¹All 400 output pairs matched Raw. Both methods averaged 52.70 on Llama and 54.65 on Qwen across tasks.

¹ HotpotQA, Qasper, TriviaQA, QMSum, MultiNews, SAMSum, LCC and RepoBench-P.

4. Which pages fall back to Raw? Page-weighted fallback across 54 captures, all layers and all heads (9 inputs per model at 4K/8K/16K).

ModelHead–token–dimension, K / VToken–head–dimension, K / V
Llama0.00783% / 0.00192%0.00339% / 0.00028%
Qwen0.30906% / 0.00123%0.41349% / 0.00021%
Qwen keys in default head–token–dimension layoutRaw fallback
Layers 6 / 0 (minimum / maximum)0.00698% / 8.38497%
Head range, weighted by contributed elements0.05219–0.61718%
HotpotQA / QMSum, weighted by pages0.32322% / 0.28074%

Over 99.69% of pages fit the default 16-entry palette in each model and tensor group. Whole-head splits aligned to 256 values preserved page and fallback counts in all 3,672 projections. We will verify this on distributed KV-head shards and measure per-rank memory and decode latency.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear Area Chair and Reviewers,

Thank you for the time and care you have given KVRecode. The discussion helped us answer two questions central to the paper: why a small local table can encode KV values exactly, and when the resulting memory savings matter to users.

  • Reviewer A9qt (3, Weak Acceptance) highlighted the exactness checks and matched evaluation, and asked when compression pays off in practice. On one 80 GB A800, KVRecode kept 32 Qwen3-8B sessions resident versus 27 with Raw BF16, each with a 16K-token input and 512-token continuation. Reviewer nHX1 also acknowledged this capacity gain.
  • Reviewer oy77 raised their score from 1 to 2 while retaining a concern about novelty. Our follow-up isolated the contribution of choosing exponents locally. On the same Qwen key tensor, a fixed 16-entry table missed 6,731 pages; a page-local table of the same width missed only six. This explains why four-bit indices can preserve even globally rare exponents exactly.
  • Reviewer nHX1 stated that they would raise their score from 1 to 2, although the displayed rating remains 1. They credited the load sweep and asked about Strata’s overlap scheduler and the benefit over Raw. In our final follow-up, overlap-enabled Strata exceeded the backlog limit on both targeted replays, while KVRecode and Raw met the service target. Both sustained the same tested load. On the 164 return turns where only KVRecode retained the prefix, users received the first token 1.19 seconds earlier and the full answer 1.07 seconds earlier in paired medians.

We respectfully ask the Area Chair to consider the reviewers’ stated assessments together with the follow-up evidence, particularly the experiments completed in response to the final questions. We would welcome the opportunity to share KVRecode and these findings with the ICLR 2027 community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.