acceptodds
Under review as a conference paper at ICLR 2027

ByteKV: Reusable KV Cache Compression via Bit Allocation and Model Adaptation

Abstract

The key–value (KV) cache accelerates autoregressive decoding in large language models by avoiding redundant context computation, but storing long contexts incurs substantial memory overhead. Reusing a compressed cache across questions requires preserving context information within a tight byte budget before downstream queries are revealed. Here we present ByteKV, a framework combining existing query-blind selection rules with bit allocation and model adaptation. Lowering KV precision enables ByteKV to retain more context positions per byte, while a shared low-rank adapter (LoRA) adapts the model to the sparse, quantized cache through self-distillation from a frozen teacher using a full 16-bit KV cache (all context positions). Across four selection rules evaluated on Llama and Qwen models, 4-bit ByteKV outperforms matched frozen 16-bit baselines on RULER-13, delivering absolute accuracy gains of 17.7 to 58.1 percentage points at a nominal 3% budget relative to full 16-bit KV storage. Integrated with custom fused packed-KV attention and vLLM memory management, ByteKV delivers 1.91× the peak output throughput of full-cache serving with the same adapted weights under concurrent serving. Our code is available at https://anonymous.4open.science/r/ByteKV.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, KV cache compression, efficient inference, quantization, low-rank adaptation, KV cache reuse, query-agnostic eviction, query-blind eviction, low-bit quantization, LoRA, self-distillation, inference memory budget, RULER

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_AHu8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates reusable KV cache compression in scenarios where subsequent query conditions are unknown. The proposed method employs low-bit storage to retain a greater number of context positions within a fixed byte budget, and utilizes self-distillation to train low-rank adapters, enabling the model to adapt to a sparse, quantized cache. The authors conduct experiments using various selection criteria and implement fused attention kernels, along with integration into serving systems.

Critical strengths
  1. The authors distinguish between two sources of gain, which increases reserved slots and improving model adaptation, and further provide controlled experiments comparing actual byte-level matching against fixed reserved slots, making the results easier to analyze.
  2. The approach includes both the execution of compressed caching and a practical implementation within a serving system, demonstrating that storage savings can be translated into tangible gains in concurrent service capacity, rather than remaining merely a theoretical estimate.
Critical weaknesses
  1. Most of the performance gains shown in Table 1 stem from retaining more positions after reducing precision, while the contribution from the model adaptation mechanism is relatively minor. Given that existing research already explores the combination of quantization and token eviction, the specific contribution of the adaptation mechanism introduced in this paper requires clearer delineation.
  2. The adaptation mechanism suffers from significant issues regarding task transferability. As shown in Section 5.6, the LongBench score for Llama (at a 3% cache budget) drops from 41.74, achieved using a frozen 4-bit scheme, to 31.16. Furthermore, no consistent gains are observed on SCBench. This undermines the method’s credibility as a general-purpose, reusable caching solution.
  3. The evaluation of benefits for practical deployment remains insufficient. An improvement in task scores under a fixed cache budget does not automatically translate to enhanced end-to-end inference efficiency. It is recommended to provide a more comprehensive analysis by comparing metrics such as time-to-first-token (TTFT), decoding throughput, and adaptation costs, accounting for actual VRAM usage (including quantization metadata and adaptation overhead). This would clarify whether the observed performance gains can be effectively realized as tangible system-level benefits.
Questions for authors
  1. In the context of actual byte-level matching, what are the primary gains and performance regressions of the adaptation module compared to the frozen 4-bit baseline when evaluated on real-world question-answering tasks?
  2. Can the results from the existing tasks be used to explain the causes of these performance regressions? Furthermore, could you clarify for which specific tasks or data conditions the use of this adaptation module is recommended?
3 comments

Thank you for connecting the transfer questions with practical deployment benefits and costs.

W1. What adaptation adds

The frozen 4-bit control already receives the coverage benefit of lower precision. Adaptation then improves accuracy while the retained positions and storage stay fixed (Table 8, KVzip). This shows that adaptation adds value beyond storing more positions. ByteKV trains a shared LoRA adapter to produce and use the sparse, quantized cache, with a frozen full-cache teacher supplying the targets.

W2, Q1, Q2. Transfer, equal storage, and use cases

The new adapters improve real QA scores at unchanged context-KV storage. ByteKV-KVzip uses RestoreKV's released LongAlpaca, PG-19, and FLAN mixture, with one seed per model. Both training data and selector differ from the original KeyDiff experiments, and some training text overlaps evaluation documents. Frozen and adapted 4-bit caches share retained positions, quantizer, and metadata. At a nominal 3% of full 16-bit KV payload, adaptation gains are:

ModelLongBench, 12 tasksSCBench book QA
Llama+4.00+9.40
Qwen3+8.07+5.41

At 3%, the gains extend beyond classification: Qasper, HotpotQA, and MuSiQue improve, and removing TREC leaves mean gains of 1.81 on Llama and 6.26 on Qwen3. Code completion (RepoBench-P) declines on both models. Both benchmark means also improve at 5%, but Llama's LongBench gain there depends on TREC: excluding it gives 41.24 versus frozen 4-bit's 41.35. At 10%, Llama's LongBench mean is 0.89 points below frozen 4-bit.

Document and book QA at tight budgets are the supported use cases. The original corpus trains identifier lookup, while the broader mixture includes semantic matching and answer synthesis. That closer task match offers a plausible explanation for the QA gains. The general response provides the full baseline comparison.

W3. Deployment benefits and costs

Section 5.8 evaluates KeyDiff serving with identical LoRA-merged weights for compressed and full-cache inference. Merging removes separate adapter operations at runtime. Timing fixes context length at 8K and output length at 1024 tokens; accuracy is measured separately. At concurrency 64:

MeasurementByteKV-KeyDiffAdapted full cache
Paged KV per request165.75 MiB1168 MiB
Output throughput723.0 tokens/s397.5 tokens/s
Median time to first token9.8 s135.4 s
95th-percentile inter-token latency45.4 ms25.1 ms

The serving benefit is higher concurrent capacity. Smaller caches let more requests stay resident, reducing queueing before the first token. Inter-token latency rises, while single-request decoding is essentially unchanged. Peak throughput reaches 1.91 times the full-cache peak. KV storage includes metadata, page rounding, and uncompressed question and generated-token KV; weights and workspaces consume additional VRAM.

The new KVzip adapters cost about 12.3 GPU-hours on Llama and 45.2 on Qwen3 to train, excluding score precomputation. Each is trained once and reused across all tested tasks and budgets; no new training is required for each context or question.

Reviewer_AHu8OpenReview ↗

Raised my score from 2 to 3

Thanks to the authors for their detailed response. My concerns have been adequately addressed, and I have thus raised my score from 2 to 3 accordingly.

AuthorsOpenReview ↗

Thank you for the discussion

Thank you for your updated assessment. Your feedback helped us clarify ByteKV’s contribution and strengthen the evaluation. We appreciate your time and engagement.

Reviewer_DFvDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ByteKV, a query-agnostic KV cache compression method that combines existing eviction methods with KV quantization, where lowering KV cache precision allows more positions to be stored within the same memory byte budget. Then, ByteKV trains a LoRA adapter through the self-distillation from a full KV cache teacher to improve the compression accuracy under the same budget. The main result is shown in Table 2, on RULER benchmark dataset across selectors and budgets, 4-bit quantization substantially improves on frozen 16-bit eviction, and the LoRA adapter adds further gains in nearly all cases. However, Sec. 5.6. shows that LoRA fails to transfer to LongBench or SCBench, exposing a major weakness of the proposed approach.

Critical strengths
  • The authors honestly report experimental results in the Appendix sections, including those that could weaken their claims.
  • The authors reported the actual memory and throughput gains under the serving engine, vLLM.
Critical weaknesses
  • One major concern is the novelty of the proposed methods. Combining eviction with quantization has already been explored. For instance, (Zhang et al., 2025) report that “storing more tokens with lower precision significantly outperforms standalone KV pruning or KV quantization methods under various KV cache budgets, highlighting the importance of balancing token count and precision in KV cache compression.”. KVzip also presented experimental results under 4-bit quantization. Therefore, the memory budget reallocation gain should be considered as a baseline rather than a contribution.
  • Adapting a model or learned module is also not new (e.g., KVLinC, KV-Distill, etc), and recent works (Abdellatif, 2026) and (Baek et al., 2026) apply a LoRA adapter with output reconstruction error. The novelty concern is not simply about the existence of the related work, but about the straightforwardness of combining these methods.
  • The proposed LoRA adapter transfers poorly beyond the training task. Across all budgets and models, the default LoRA model consistently fails to generalize on LongBench and SCBench benchmarks (Tables 18, 19, and 20), limiting their application to practical utilization.
  • The original KVzip baseline, without any quantization or LoRA adapter, exhibits 90.50% accuracy on Llama 3.1-8B under 10% budget, which outperforms all the results in Table 2. This result should be included in both Table 2 and Table 8 for a fair comparison. However, it is only reported in Table 9, making it difficult for readers to compare the proposed method against this strong baseline.
Questions for authors
  • From a practical viewpoint, what accuracy drop is acceptable for the memory and throughput gains?
  • The abstract reports accuracy gains as "..., delivering absolute accuracy gains of 17.7 to 58.1 percentage points at a nominal 3% budget relative to full 16-bit KV storage." Do you think this covers the overall gain of ByteKV, or its most favorable setting?

Minor

  • Why are there arrows in Figure 2 (A)? The figure looks like selection, bit allocation, and LoRA are applied sequentially.
8 comments

Thank you for recognizing the experimental reporting and serving measurements.

W1, W2. What the combined design contributes

Zhang et al. and KVzip establish the precision–coverage trade-off. ByteKV couples allocation with learning to produce and use the compressed cache. Lower precision retains more context, but selection and quantization change attention's inputs. A shared LoRA adapter learns under both changes from a frozen full-cache teacher's predictions. It participates in prefill, question processing, and decoding; the selection rule and bit-width stay fixed.

The submitted KVzip control shows accuracy recovery without changing the positions selected by the frozen model. Thus, adaptation can help even when selection stays unchanged. With 2-bit KeyDiff, adaptation also improves 12 of 13 tasks on each model. These results use one seed and a nominal 3% of full 16-bit KV storage (Tables 8 and 10):

ModelConfigurationFrozenByteKV
Llama-3.1-8BKVzip, 4-bit37.6663.99
Qwen3-8BKVzip, 4-bit29.9740.72
Llama-3.1-8BKeyDiff, 2-bit12.6349.55
Qwen3-8BKeyDiff, 2-bit40.5061.95

The related methods differ in which stage and representation they train:

MethodLearned operation
KV-DistillLearns selection and context encoding; encoder LoRA is off for later questions and decoding.
RestoreKV, Baek et al. (2026)Generates complementary KV entries; LoRA is off for later questions and decoding.
KVLinCUpdates an extra matrix and vector to correct quantized attention.
Abdellatif (2026)Trains attention LoRA for quantization with all positions retained.
ByteKVShared LoRA produces context KV and later uses its selected, quantized form.

We could not run KV-Distill: no implementation or checkpoint was available when checked on 10 November. KVLinC's all-position 2-bit codes require 12.5% of full 16-bit storage. Our budgets require adding eviction and accounting for its correction states. We directly compare with RestoreKV, a learned query-blind method, at all three budgets.

W3. Transfer

Transfer is model-dependent in the submitted results. The Qwen3 adapter improves LongBench at all three budgets: at 3%, it scores 34.38 versus frozen 4-bit's 32.46. Llama loses accuracy on LongBench, and neither default adapter improves book QA (Tables 19 and 20).

With RestoreKV's released training mixture, new ByteKV-KVzip adapters improve both models on LongBench and book QA at 3% and 5%. Each model uses one seed. The new recipe changes both data and selector, and some training text overlaps evaluation documents. At 3%, gains over frozen 4-bit are:

BenchmarkLlama gainQwen3 gain
LongBench, 12 tasks+4.00+8.07
SCBench book QA+9.40+5.41

These gains establish useful adaptation beyond identifier lookup. ByteKV-KVzip also exceeds the 16-bit KVzip and RestoreKV baselines at 3% and 5% on both models, and at 10% on Qwen3. At 10% on Llama, RestoreKV leads both benchmarks, and frozen 4-bit also leads on LongBench. We will add the complete transfer and baseline comparisons to the appendix.

W4. The original KVzip baseline

KVzip's 90.50 at 10% exceeds every Llama result in Table 2. Here are the submitted results together (Tables 8 and 9):

ModelBudgetOriginal frozen 16-bitFrozen 4-bitByteKV-KVzip, 4-bit
Llama10%90.5079.8686.97
Llama5%63.4754.7778.35
Llama3%27.0537.6663.99
Qwen310%87.2274.5691.74
Qwen35%38.0342.8266.15
Qwen33%19.4029.9740.72

Original KVzip scores higher for Llama at 10%; ByteKV-KVzip exceeds it in the other five settings. Adaptation improves the matched 4-bit configuration in all six. Thus, adaptation helps within this selector, while the advantage over original KVzip depends on the budget and model. The 4-bit pair shares positions, protection rules, and evaluator. Table 2 reports matched controls; Table 9 retains published configurations.

Q1. Acceptable accuracy loss

The application's quality requirement determines the usable budget. ByteKV-KVzip scores 48.69 on Qwen3 LongBench at a nominal 5%, versus 50.81 with full cache: about two score points lower. The general response shows the trade-off at the other budgets

Q2. The abstract's range

The 17.7 to 58.1 percentage-point range covers all eight model–selector pairs at 3% in Table 2, against matched frozen 16-bit controls. Tables 1 and 2 also report 5% and 10%.

M1. Figure 2(A)

The arrows compare configurations at one budget. LoRA participates in cache construction during prefill, before selection and quantization, and in later question processing and decoding. We will revise Figure 2(A)'s arrows and labels to clarify these roles.

AuthorsOpenReview ↗

Coupling Allocation with Adaptation (W1, W2)

Thank you for highlighting the need to explain why these components belong together. We have uploaded a revised manuscript, with new and revised text marked in blue.

W1, W2. Coupling allocation with adaptation

Lower precision makes room for more context, but selection and quantization change the KV representation used by attention. ByteKV trains a shared adapter for cache construction and question answering through that representation, using a frozen full-cache teacher. The selection rule and bit-width remain fixed.

What the fixed-position control establishes

In the submitted KVzip control, retained positions stay fixed while adaptation changes the KV vectors at those positions and how later questions use them. At a nominal 3% of full 16-bit KV storage, RULER accuracy rises from 37.66 to 63.99 on Llama and from 29.97 to 40.72 on Qwen3 (Table 8, one training seed). These gains show that learning to represent and use retained context improves accuracy without selecting more positions or increasing KV storage.

W3. Transfer under both training recipes

Section 5.6 now distinguishes the two training settings. The original KeyDiff adapter improves tasks absent from its identifier-lookup training: at 3%, Llama's single-hop QA rises from 22.40 to 38.55 and Qwen3's variable tracking from 72.60 to 87.34 (four-seed means). This transfer is task-dependent; the original Llama adapter loses accuracy on LongBench.

With RestoreKV's released mixture and the same distillation objective, each new ByteKV-KVzip checkpoint is evaluated on LongBench and SCBench book QA at all three budgets. Adaptation improves 11 of 12 means over frozen 4-bit KV; Llama's LongBench result at 10% is the exception. At 3% and 5%, it also exceeds 16-bit KVzip and RestoreKV on both models' LongBench and book QA means (Table 27). These results give a practical benefit for the same combined design beyond the original lookup task.

Manuscript changes: we have made this training design explicit in the Introduction and brought the KV-Distill comparison into Section 2.

We would appreciate your assessment of whether these controls and direct comparisons change your view of the combined design.

Thank you for the rebuttal. I will read the rebuttal and respond to the points. However, do the authors plan to include the original KVzip result in Tables 2 and 8, as I commented in W4? If not, could you explain why it remains only in Table 9 of the revised manuscript?

Reviewer_KFT2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a KV-cache compression framework that combines cache quantization, selection algorithm and model adaptation to improve accuracy with the compressed kv cache. The main contribution of the paper is reducing the performance gap between full-cache and compressed-cache inference by using self-distilation. The method is validated on Llama, Qwen3 and Mistral, for 4 different cache selection method. The authors also show that the method can reduce kv cache storage size in practice by implementing custom kernel for vLLM.

Critical strengths

The method is widely applicable to query-blind KV-cache selection method and evaluated with open-sourced models, including Qwen3, Llama, and Mistral.

Critical weaknesses

The paper is difficult to read because it heavily depends on the appendices for its experimental results. For example, section 5.4 to 5.6 rely on results presented only in appendices. Some important results, such as performance on general benchmark after adaptation is included only in appendix and difficult to find. The paper includes a lot of experimental results with various setting but insights from these experiments are not clear. For example, there is no explanation in the main text about why full-cache consistency term is used in Qwen3 adaptation, not Llama.

Questions for authors
  • In table 24, What is the performance of Qwen3 with anchor=0 on these benchmarks?
  • In section 5.4 and Table 11, Do you have any insights about why V-only adaptation works much better for Llama than Qwen3? Which difference in model makes the V-only adaptation more effective for Llama?
2 comments

Thank you for highlighting the anchor and projection questions, and the need for clearer experimental interpretation.

W1, Q1. Qwen3 without the full-cache anchor

The anchor is an optional regularizer that penalizes disagreement with the teacher when compression is disabled. The original KeyDiff recipe adds it for Qwen3 with λ=1; Llama uses compressed-cache distillation alone, with λ=0. The new Qwen3 control tests whether that extra constraint is needed, holding data, learning rate, and training seed fixed. Evaluation retains all positions at 16 bits:

BenchmarkFrozenWith anchor, λ=1Without anchor, λ=0
WikiText perplexity, lower is better10.9110.4710.42
MMLU49.0349.0749.29
ARC-Easy79.4679.9280.05
HellaSwag68.6368.9468.95
GSM8K14.1812.2814.03

Without the anchor, Qwen3 retains comparable full-cache performance and improves compressed RULER-13 accuracy from 57.86 to 58.84 at 3% (Appendix A.4). This single-seed control supports using compressed-cache distillation alone for Qwen3 in this setting, as Llama already does.

Q2. Why V-only differs across models

Llama's mean gain in Table 11 is concentrated in the multi-key retrieval task niah_multikey_2. The task's 4-bit KeyDiff scores at 3%, using one training seed, are:

ModelFrozen 4-bitValue projection onlyAll attention projections
Llama-3.1-8B3.2061.0061.80
Qwen3-8B1.209.2020.60

V-only nearly matches full adaptation on this Llama task, explaining much of its reported mean gain. Qwen3 benefits more from updating the other projections. V updates also alter later-layer inputs, which can change later keys and selected positions (Appendix C.1). This explains how V-only adaptation can affect retrieval beyond the current layer. The ablation varies trainable projections within each pretrained model. It identifies which updates recover accuracy on this task. Attributing the difference to a specific architectural component would require varying that component independently.

We will place the full-cache control and projection results beside their main-text discussions.

Thank you for the response.

However, I find it difficult to learn much insight from these results. If Qwen3 without the anchor retains comparable full-cache performance, the motivation or observation remains unclear to me why the full-cache anchor only on Qwen3 and not to Llama. Response to Q2 also doesn't address the details in projection placement. Besides these two results, several results throughout the paper would benefit from more explanation of the motivation behind them, or how they support the main contribution.

I am maintaining my score of 2.

Reviewer_DhmVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ByteKV has two parts1. store the kept tokens at 4 bits instead of 16, so ~4x more tokens fit. No training is needed for this. 2. train a LoRA adapter so the model works better with the compressed cache. The lora adapter is used both during prefill and decoding and this can provide gains on standard bencharmks. The authors also have a 4-bit vLLM backend offering with 1.91x peak throughput.

Critical strengths

Honest ablations eg frozen 4-bit, exact byte matching. But this kinda shows that their new parts on top of 4bit doesn't add much.

Critical weaknesses

Most of the gains are because of 4-bit storage... which is known. In your own table 1 going from 16 to 4 does most of the job and the additional lora adds small acc gains. Also trading precision for more tokens is present in works like "More Tokens, Lower Precision" (Zhang et al. 2025) and MiKV both cited.

The adapter is trained basically on what ruler tests and that hurts outside ruler. Training is synthetic key lookup and ruler is actually mostly synthetic retrieval so the the (very) in domain gains are not that surprising. But according to table 18 on LongBench the adapter makes Llama worse than plain 4-bit (about 10 points lower). That seems problematic. On SCBench it also doesn’t help that much (or hurts) checking table 20. It also hurts llama when compression is off, eg perplexity goes up. why?

Questions for authors
  1. Just 4 bit without the adapter beats bytekv on LongBench for Llama (table 18). The adapter also makes the uncompressed llama worse (table 24). Is there a setting where adding the adapter clearly helps on real & non-synthetic tasks without hurting the uncompressed model? If not why would one use the adapter instead of just storing more tokens at 4 bit?

  2. At the same memory how does bytekv compare with existing 4-bit methods that need no training like KIVI or MIKV?

1 comment

Thank you for focusing on whether the adapter is worth using beyond synthetic retrieval.

W1. Why combine allocation with adaptation?

Zhang et al. and MiKV establish the precision–coverage benefit. ByteKV couples allocation with learning to produce and use the sparse, quantized cache it creates. More retained positions preserve context, while distillation trains a shared LoRA adapter under the resulting selection and quantization. Adaptation can recover accuracy even without changing which positions are retained. With KVzip, RULER accuracy rises from 37.66 to 63.99 on Llama and from 29.97 to 40.72 on Qwen3 (Table 8, one seed). Both 4-bit configurations use identical positions at a nominal 3% budget relative to full 16-bit storage.

W2, Q1. Real-task gains and uncompressed capabilities

We've add new ByteKV-KVzip adapters and they are trained on RestoreKV's released mixture improve LongBench and SCBench book QA at 3% and 5% on both models. Each model uses one seed. The new recipe changes both selector and training data, and some training text overlaps evaluation documents. Frozen and adapted 4-bit caches have identical positions and storage. At a nominal 3% of full 16-bit KV payload:

Benchmark, modelFrozen 4-bitByteKV-KVzip, 4-bit
LongBench, Llama35.1539.16
LongBench, Qwen336.3244.39
SCBench book QA, Llama10.8320.23
SCBench book QA, Qwen314.5319.94

These gains support using the adapter for compressed QA at tight budgets. At 10%, Llama's LongBench mean is 0.89 points below frozen 4-bit.

The same adapters do not preserve every tested full-cache capability. On the five benchmarks from Table 24, Qwen3 improves four metrics but loses 2.88 points on GSM8K. Llama improves GSM8K and regresses on the other four; WikiText perplexity rises from 7.97 to 9.94. Training fits behavior under sparse, quantized KV. Restoring all positions at full precision changes attention's inputs, which may contribute to these regressions.

Q2. KIVI and MiKV at the same memory budget

We have no direct same-budget results for KIVI or MiKV. Their native designs need extensions for our query-blind 3%, 5%, and 10% budgets:

MethodRequired extension
KIVIIt retains all positions without selection. Even its 2-bit codes require 12.5% of full 16-bit storage before residuals and metadata. Our budgets require adding a selector.
MiKVIt assigns mixed precision using importance scores. Its evaluated scorer uses query-dependent attention; reusable caches need a query-blind policy and a corresponding byte allocation.

The frozen 4-bit control holds selection and quantization fixed to test the adapter's benefit. The general response adds complete-method comparisons with 16-bit KVzip and RestoreKV at all three budgets.

Reviewer_26omOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a query-blind KV-cache compression framework that trades precision for coverage and adapts the model to the resulting sparse, quantized cache. ByteKV selects positions with an existing rule, stores retained KV entries at lower precision, and trains a shared Q/K/V/O LoRA adapter by distillation from a full-cache teacher. On Llama-3.1-8B and Qwen3-8B, the main KeyDiff experiments show that at a nominal 3% payload, 4-bit coverage improves over a frozen 16-bit selection baseline and adaptation adds a further 4.8-5.6 points in the main RULER-13 table. The paper also reports results for three other selectors, longer contexts, LongBench and SCBench transfer, and a fused Triton/vLLM implementation. The serving experiment reports 7.05x lower resident prefix KV storage and 1.91x peak output throughput at high concurrency. However, the adapter is trained mainly on synthetic identifier-retrieval examples, and the transfer results are mixed: on Llama LongBench at 3%, ByteKV scores 31.16 versus 41.74 for frozen 4-bit KV, while Qwen3 improves only modestly. The reported gains therefore establish a useful within-task recovery effect more clearly than a general-purpose reusable-cache solution.

Critical strengths
  1. The paper separates the effects of lower precision and adaptation using frozen 16-bit, frozen 4-bit, and adapted 4-bit controls under the same selector. At the 3% nominal budget, reallocation produces the largest improvement, while adaptation still gives a positive additional gain across the main model-budget cells. This is a useful and clearly testable decomposition of the cache-compression problem.

  2. The work goes beyond offline accuracy. It provides exact-byte accounting, a packed 4-bit attention kernel, page allocation, and a concurrency sweep. The reported reduction in resident prefix storage and the throughput gain at high concurrency make the systems implications more concrete than a purely simulated compression study.

Critical weaknesses
  1. The main adapter is trained on 701 synthetic 4K retrieval examples with random identifiers, and transfer is not reliable. At 3% on Llama, LongBench drops from 41.74 for frozen 4-bit KV to 31.16 for ByteKV; multi-format synthetic training raises it only to 35.90. SCBench also often matches or falls below frozen 4-bit KV. This is consequential because the paper motivates reusable caches for downstream questions that are not known at construction time, while the evidence shows that the learned adapter can specialize to the distillation question distribution.

  2. The headline comparisons still leave a large gap to the full cache, and some comparisons use a nominal rather than exactly realized byte budget. At 3%, ByteKV reaches 68.06 on Llama and 57.75 on Qwen3, compared with full-cache scores of 95.75 and 95.36. The 4-bit representation also uses 106.6-111.3% of the matched frozen 16-bit control before exact-byte correction. Thus, the results support recovery relative to a sparse frozen baseline, but not near-lossless reusable-cache quality at the stated budget.

  3. Validation across selectors is uneven. The main KeyDiff results average four seeds, but most adapted cells for Expected Attention, LagKV, and k-norm use one seed. The paper itself reports that the Llama 10% KeyDiff gain is smaller than its seed standard deviation, and KVzip runs span 7.21 points. This makes it difficult to know how consistently the adaptation effect survives selector and seed changes.

  4. The serving comparison does not fully close the quality accounting. Figure 3 compares ByteKV with an adapted full-cache reference, but the merged full-cache vLLM accuracy is explicitly unmeasured; the accuracy references come from a separate 4K four-needle workload, whereas timing uses an 8K context and 1024 generated tokens. This does not invalidate the memory measurement, but it weakens the claim that the throughput point represents an equivalent-quality serving comparison.

Questions for authors
  1. How does ByteKV perform on a broader training distribution that includes semantic retrieval, multi-hop QA, summarization, and code tasks, rather than synthetic random-identifier retrieval? In particular, can you report a matched comparison against frozen 4-bit KV on LongBench and SCBench with several training seeds?

  2. Please report main-table results with exactly matched realized bytes, including scales, zero points, retained-index metadata, and rounding. How much do the 3%, 5%, and 10% conclusions change after replacing nominal budgets with exact per-example byte matching?

  3. For the non-KeyDiff selectors, can you provide multiple-seed results and confidence intervals for the adapted cells? The current one-seed results make it difficult to assess whether the positive adaptation effect is robust across selection rules.

  4. Can you measure the merged adapted full-cache model with the same served vLLM configuration and the same 8K/1024-token workload used for Figure 3? Please report its accuracy and clarify whether the throughput comparison changes when all configurations are evaluated at matched quality and software conditions?

1 comment

Thank you for the detailed questions about transfer, byte accounting, seeds, and serving quality.

W1, Q1. Broader training and transfer

We've add new ByteKV-KVzip adapters and they improve LongBench and book QA at 3% and 5% on both models. Training uses RestoreKV's released LongAlpaca, PG-19, and FLAN mixture: 4,738 examples for Llama and 5,206 for Qwen3. This changes both data and selector from the original KeyDiff experiments; some training text overlaps evaluation documents. Each model uses one training seed and one adapter across all 12 LongBench tasks and SCBench book QA. In book QA, each compressed cache is reused across questions. Below are gains over frozen 4-bit at identical positions and KV storage. Budgets are nominal fractions of full 16-bit KV payload.

Benchmark, model3%5%10%
LongBench, Llama+4.00+2.62−0.89
LongBench, Qwen3+8.07+7.77+3.39
SCBench book QA, Llama+9.40+12.54+3.13
SCBench book QA, Qwen3+5.41+5.70+3.70

These results extend adaptation's benefit beyond synthetic identifier retrieval. The general response also compares 16-bit KVzip and RestoreKV.

W2, Q2. Exact bytes and the full-cache gap

Both allocation and adaptation gains survive exact-byte accounting. Appendix A.2 fits each KeyDiff cache within the realized byte allowance of its 16-bit control, including codes, scales, zero points, indices, and rounding. Fewer positions are retained to accommodate metadata. Allocation still improves RULER accuracy by 18.4 to 34.5 points over 16-bit selection, and adaptation improves every paired cell:

ModelReference budgetFrozen 4-bitByteKV-KeyDiff, 4-bit
Llama10%79.4281.52
Llama5%67.4872.50
Llama3%61.2966.81
Qwen310%74.2977.17
Qwen35%61.4764.80
Qwen33%49.0856.05

Llama uses four training seeds and Qwen3 one. Byte correction lowers adapted scores by 1.07 to 1.86 points relative to nominal runs in the same evaluator. Full-cache reference scores are 95.75 for Llama and 95.36 for Qwen3.

W3, Q3. Training-seed variation

The KVzip adaptation gain in paper repeats across all three tested Llama seeds and both Qwen3 seeds, at every budget. These RULER runs use the original training corpus, separately from the new transfer experiments. The 7.21-point Llama spread at 10% changes the gain's magnitude; every tested run still improves over frozen 4-bit. Llama KeyDiff at 10% is seed-sensitive: two of four seeds fall below frozen 4-bit. Expected Attention, LagKV, k-norm, and the new QA adapters each have one training seed. We report their paired scores without training-seed confidence intervals. The repeated KVzip gains provide evidence that adaptation remains useful across independently trained checkpoints for that selector.

W4, Q4. Served quality and software conditions

Figure 3 controls request length and prefix sharing with synthetic 8K token sequences and 1024 output tokens. These sequences have no reference answers, so accuracy is measured separately. At the same compression setting, served merged ByteKV scores 63.45 on 4K retrieval, versus 63.65 with the runtime adapter. The 0.20-point difference shows close agreement between these two compressed serving paths.

Both timed configurations use identical merged weights and the same vLLM version. Prefix caching reuses no blocks in this workload. Full-cache serving enables chunked prefill. ByteKV's backend requires the whole context in one prefill pass, so chunking is disabled.

The requested served full-cache accuracy on labeled 8K tasks has not been measured. The 1.91-fold result establishes the capacity gain at the tested setting. A matched-quality comparison would first score both served paths on labeled 8K tasks and select ByteKV budgets that meet a common accuracy target. Timing those configurations with matched chunking settings would then answer the equal-quality throughput question.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

Thank you for the feedback on novelty and transfer. We have added broader training and direct comparisons with KVzip and RestoreKV.

ByteKV addresses two effects of compression: fewer context positions and altered KV values. It allocates bytes to more low-bit positions and trains a shared LoRA adapter to produce and use the compressed cache. Training matches predictions from a frozen teacher with all positions retained at 16 bits.

The new ByteKV-KVzip adapters use RestoreKV's released LongAlpaca, PG-19, and FLAN mixture. Each model uses one adapter across all budgets and both benchmarks, with one training seed. LongBench covers our 12 tasks; SCBench book QA reuses each compressed 32K context across questions.

Test, modelBudgetFrozen full cacheKVzip, 16-bitRestoreKV, 16-bitFrozen 4-bitByteKV-KVzip, 4-bit
LongBench, Llama-3.1-8B3%49.6525.3530.0435.1539.16
LongBench, Llama-3.1-8B5%49.6531.8238.4938.5541.18
LongBench, Llama-3.1-8B10%49.6545.9148.2245.6644.77
LongBench, Qwen3-8B3%50.8119.5623.4236.3244.39
LongBench, Qwen3-8B5%50.8125.9530.5140.9248.69
LongBench, Qwen3-8B10%50.8145.6545.1648.4651.85
SCBench book QA, Llama-3.1-8B3%27.925.4110.8310.8320.23
SCBench book QA, Llama-3.1-8B5%27.9214.2517.9512.5425.07
SCBench book QA, Llama-3.1-8B10%27.9221.6525.3620.2323.36
SCBench book QA, Qwen3-8B3%27.354.849.6914.5319.94
SCBench book QA, Qwen3-8B5%27.357.6915.3818.2323.93
SCBench book QA, Qwen3-8B10%27.3519.3725.0722.2225.93

Budgets are nominal percentages of full 16-bit KV payload, excluding quantization metadata. For Llama, the full-cache, frozen 4-bit, and ByteKV-KVzip runs use fp16. The KVzip and RestoreKV baselines use bf16. All Qwen3 runs in this table use bf16.

At 3% and 5%, ByteKV-KVzip is better than all three compressed baselines on both benchmark means for both models. Qwen3 also leads at 10%. On Llama at 10%, RestoreKV scores higher on both benchmarks; frozen 4-bit and KVzip also score higher on LongBench.

Besides, it is interesting that under test case - LongBench + Qwen3-8B + 10% budget, ByteKV-KVzip (4-bit) is even better than the frozen full cache. The compressed KV cache may reduce the context noise in full KV cache which helps improve the model performance of extreme compressed KV cache.

Separately, the submitted KeyDiff serving implementation delivers 1.91x the peak throughput of full-cache serving with the same adapted weights (Section 5.8).

AuthorsOpenReview ↗

Revised Manuscript and Main Updates

Thank you for considering the additional experiments. We have uploaded a revised manuscript, with new and revised text marked in blue.

1. Original training: transfer beyond identifier lookup

The original ByteKV-KeyDiff adapter learns to retrieve one identifier from an explicit key. This supervision covers only part of RULER. It does not train natural-language QA, variable tracking, or multiple-answer retrieval. At a nominal 3% of full 16-bit KV storage, we compare three KeyDiff configurations with the same protection rules. ByteKV scores average four training seeds:

ModelRULER taskFrozen 16-bitFrozen 4-bitByteKV-KeyDiff
Llama-3.1-8BSingle-hop QA19.2022.4038.55
Llama-3.1-8BMulti-hop QA17.6018.6027.35
Qwen3-8BVariable tracking2.8872.6087.34
Qwen3-8BMultiple-value retrieval20.2080.8090.15

The adapted results appear in Table 13 of the original submission (Appendix C.2), rounded to one decimal. The frozen columns come from the per-task results underlying Table 1's baselines.

Every adapted seed exceeds frozen 4-bit in each listed task. Adaptation therefore transfers to task types absent from its training supervision. The benefit is task-dependent: word-frequency extraction declines. On LongBench, Qwen3 improves at all three budgets; Llama improves on Qasper but loses overall accuracy. Neither default adapter improves SCBench book QA.

2. Broader training: gains on document and book QA

The new ByteKV-KVzip adapters use RestoreKV's released LongAlpaca, PG-19, and FLAN mixture [1] with the same distillation objective. Each model uses one training seed and the same checkpoint across LongBench, SCBench book QA, and all three budgets.

Adaptation improves 11 of 12 benchmark–model–budget means over frozen 4-bit KV (Table 27). Each pair has identical retained positions and KV storage. The only mean decline is Llama's LongBench result at 10%.

For example, at 5%:

  • Qwen3, LongBench: the score rises from 40.92 with frozen 4-bit KV to 48.69 with ByteKV-KVzip, compared with 50.81 for full cache.
  • Llama, SCBench book QA: the score rises from 12.54 with frozen 4-bit KV to 25.07 with ByteKV-KVzip, compared with 27.92 for full cache.

These comparisons support using adaptation for document and book QA beyond synthetic identifier retrieval. Table 27 also includes 16-bit KVzip [2] and RestoreKV. ByteKV-KVzip exceeds both at 3% and 5% on both models and benchmarks. At 10%, Qwen3 retains this advantage, while RestoreKV leads on Llama.

3. Changes in the revised manuscript

  • Method and motivation: the Introduction connects the training design to these controls and QA results. Section 2 clarifies the relation to KV-Distill [3], and Figure 2(A) clarifies the adapter's roles in cache construction and use.
  • Transfer evidence: Section 5.6 now separates the original and broader training results. Appendix C.2 adds the original task-level controls; Appendix H gives the complete broader-training comparisons.
  • Other controls: Table 24 includes Qwen3 without the full-cache consistency loss; Section 5.4 explains projection results; Table 28 reports the new adapters' full-cache results.

We would appreciate your assessment of whether these comparisons address the concern that adaptation adds little practical value beyond low-bit storage.

References

[1] Baek et al. RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction. arXiv:2608.01247, 2026.

[2] Kim et al. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. NeurIPS 2025.

[3] Chari et al. KV-Distill: Nearly Lossless Learnable Context Compression for LLMs. arXiv:2503.10337, 2025.

AuthorsOpenReview ↗

Updated Manuscript and Completed KVzip Comparisons

Thank you for the continued discussion. The six matched 16-bit KVzip controls are now complete. We have uploaded an updated manuscript with KVzip comparisons added to Figure 1 and Tables 2 and 8. New entries and revised text are marked in blue.

Both allocation and adaptation improve accuracy in all six matched model–budget settings. The original KVzip results remain in Table 9 and are discussed in Section 5.3. Our latest reply to Reviewer DFvD (W4) gives the complete comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.