acceptodds
Under review as a conference paper at ICLR 2027

Don't Drop Tokens, Dim Them: Bit Allocation for Multi-Turn KV-Cache Compression

Abstract

Agentic language models spend substantial wall-clock time paused while waiting for tool responses or user input. During these pauses, their KV caches either re- main resident in GPU memory or are managed by a memory store and transferred between storage and the GPU. How to compress the paused cache, and which compression best preserves answer quality over the turns that follow, has no set- tled answer: the two training-free families, token selection and quantization, are both built and evaluated for a single prompt answered immediately. We show that both fail in the paused regime, for opposite reasons. Token selection commits a keep-set before the queries arrive, so methods perfectly recover a planted fact on a single-turn needle collapse on SCBench key–value retrieval. Quantization makes no such commitment but spends its budget uniformly, and on multi-turn retrieval it collapses at 2 bits, which is also the most it can compress. We trace both failures to one cause: precision is uniform and token selection is binary. We therefore propose SOFT-BIT KV. Rather than choosing which tokens survive, it gives every token its own bit-width, fixed once at the pause by rate–distortion allocation under a measured-byte budget. A small MLP scores each cached to- ken from the resuming query’s own per-head attention, and reverse water-filling spends a byte budget over those scores. Average across four models, SOFT-BIT KV retains 92% of full-cache accuracy on multi-turn SCBench at 5.2× compres- sion, compared with 85% for TurboQuant and 72% for the best token-dropping method. At ≈ 11× compression, SOFT-BIT KV retains 99% on single-turn needle retrieval and 65% on long-context RULER, compared with 61% and 56% for the best token-dropping method, and 38% and 23% for TurboQuant. At the system level, it reduces the paused cache of a 98k-token Llama-3.1-8B trajectory by 81% and accelerates host-to-device transfer upon resumption by 5.2×.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM KV Cache, Agentic LLM, multi-turn conversation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kqw2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SOFT-BIT KV, which assigns 1-, 2-, or 8-bit precision to cached tokens under a memory budget. A learned MLP estimates token importance from question-window attention features, and a constrained allocation rule determines precision. The default mode fixes the allocation once; the reload variant stores multiple precision versions and selects among them for each new question.

Critical strengths
  1. Coherent design that preserves all tokens while allocating more precision to important information.
  2. The learned importance detector is a meaningful distinction from heuristic mixed-precision methods.
  3. Promising results against token-dropping and quantization baselines.
  4. The reload variant provides a practical mechanism for adapting precision to changing questions.
Critical weaknesses
  1. Unclear generalization. The detector is trained on synthetic factual retrieval, which closely resembles important components of the evaluation benchmarks. Transfer to mathematical reasoning, coding agents, and other task families remains insufficiently evaluated.
  2. Incomplete compression-rate comparison. SCBench is evaluated primarily at 5.2×, while RULER is evaluated at 11.8×. This prevents separating benchmark sensitivity from compression severity.
  3. Limited evidence isolating the contribution. Importance-aware mixed precision has substantial prior work. More controlled ablations are needed to establish how much improvement comes from the learned detector versus the allocation rule.
  4. Limited evaluation of conversational memory. Additional benchmarks such as LoCoMo and LongMemEval would strengthen the practical generalization claim.
Questions for authors
  1. Can the authors evaluate both SCBench and RULER at both 5.2× and 11.8× compression, ideally with intermediate budgets?
  2. How well does the retrieval-trained detector transfer to mathematical reasoning and coding-agent benchmarks?
  3. How sensitive are the results to the detector’s training distribution? Would training on another task family preserve retrieval performance?
  4. Under a common quantizer and byte budget, how much improvement comes from the MLP versus heuristic scoring, and from constrained allocation versus simpler precision assignment?
  5. Does the method maintain consistent benefits on LoCoMo and LongMemEval at both compression ratios?
2 comments

Dear Reviewer kqw2,

Thanks again for the reviews! We deeply appreciate the time and effort you have invested in evaluating our work :)

We've carefully considered your comments and have prepared a detailed rebuttal addressing each point raised. We are hoping reviewers would be able to review our responses and to engage in a discussion before the end of the discussion period. Please let us know if there are any specific areas you'd like us to clarify further or discuss in more detail. We look forward to your thoughts and are eager to engage in a productive dialogue.

Thank you once again for your commitment to the review process.

Best regards,

Authors

Dear Reviewer kqw2,

We thank the reviewer again for the review. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further : )

Best regards,

Authors

Reviewer_DPdROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compresses the KV cache of a paused multi-turn session once, before the follow-up questions are known. SOFT-BIT KV keeps every token: a small MLP scores tokens from first-question attention, the scores set each token's width (1, 2 or 8 bits), TurboQuant codes the result, and a tier-bucketed kernel attends over it directly.

Critical strengths
  1. The learned scorer is backed by a clean comparison in Appendix H: storage is fixed and only the token ranking changes. On these 2k-token probes, the MLP keeps full-cache retrieval accuracy up to a higher compression ratio than head-averaging or a retrieval-head committee in 10 of 12 settings. On Qwen3-4B, the full method lifts Retr.KV from 2.6 under TurboQuant to 19.0, matching the full cache.

  2. The tier-bucketed kernel makes per-token precision practical: attention reads the packed codes directly, and decode-attention time at 98k tokens falls from 4.59 ms (fp16) to 1.41 ms at 5.2× (Table 5).

Critical weaknesses
  1. It is not clear how much of the gain survives past the first question. The scorer reads the attention of q1 and is trained to find the answer to the visible question, so its benefit on later turns needs direct evidence. Table 2 reports only turn averages, and Appendix D gives per-turn numbers for FullKV but not for Soft-bit or TurboQuant.

  2. The eviction comparison may not be budget-matched. On SCBench, Appendices B and D fix the eviction baselines at 4,096 tokens (about 24× at Llama's 98k-token mean prefill), while Table 2 labels them 5.2×. If so, the headline 94% vs. 72% comparison (Section 5.1) is not like-for-like.

On novelty, MiKV (cited in Section 2) already keeps tokens at low precision rather than evicting them, arguing that future needs cannot be predicted, especially in multi-turn conversations. KVzip (NeurIPS 2025, not cited) compresses once without task queries and, on Qwen2.5-7B-1M, reports preserving accuracy on SCBench Retr.KV and RepoQA at a 30% cache ratio, which bears on Section 3's claim that a better ranking cannot fix eviction. The new parts are mainly the learned score and the kernel. I lean toward weak rejection; budget accounting and per-turn results could change my view.

Questions for authors
  1. How many tokens and bytes did each method actually keep on SCBench?

  2. Can every method use q1 when it selects or quantizes the cache? Please report turn 1 separately from later turns on the retrieval tasks.

  3. Minor: How is the 0.8-bit average in Table 10 obtained with a 1-bit minimum? Gemma Soft-bit's 12 task scores in Table 2 average 31.45, not 32.3.

2 comments

Dear Reviewer DPdR,

Thanks again for the reviews! We deeply appreciate the time and effort you have invested in evaluating our work :)

We've carefully considered your comments and have prepared a detailed rebuttal addressing each point raised. We are hoping reviewers would be able to review our responses and to engage in a discussion before the end of the discussion period. Please let us know if there are any specific areas you'd like us to clarify further or discuss in more detail. We look forward to your thoughts and are eager to engage in a productive dialogue.

Thank you once again for your commitment to the review process.

Best regards,

Authors

Dear Reviewer DPdR,

We thank the reviewer again for the review. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further : )

Best regards,

Authors

Reviewer_X7pGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Soft-bit KV for multi-turn KV-cache compression. It retains all tokens (though the proposed formulation also accommodates token dropping) and allocates different precision levels using learned importance scores under a storage budget. Experiments cover multi-turn retrieval, long-context benchmarks, and cache-resume efficiency.

Critical strengths
  1. The paper addresses a meaningful problem: information that appears unimportant at compression time may be needed by later queries. Preserving low-precision representations is a sensible approach.

  2. The evaluation covers several models and tasks, and the comparisons between learned and attention-based scoring provide useful evidence for the proposed importance predictor.

Critical weaknesses
  1. The compression-budget descriptions are inconsistent, raising concerns about comparison fairness. Tables 1–2 report token-dropping baselines at 5.2× compression, whereas Appendices B and D specify a fixed 4,096-token budget on SCBench. For Llama’s reported mean prefill length of 98k, this would imply approximately 24× compression. It is therefore unclear whether the main comparisons actually use matched storage budgets.

  2. Comparisons with closely related mixed-precision methods, particularly MiKV and ZipCache, are missing, leaving the incremental contribution insufficiently established.

Questions for authors
  1. Could the authors reconcile the 4,096-token budget for the token-dropping methods (Appendix B, Per-benchmark budgets; Appendix D, Benchmark and protocol) with the reported 5.2× compression? With Llama's 98k mean prefill, a 4,096-token budget implies roughly 24× compression. If the budgets were not matched, how do the results change under matched budgets?

  2. The paper uses the MSE-only variant of TurboQuant. Beyond the implementation constraints discussed in Appendix G, could the authors explain how omitting the QJL correction might affect the comparison? Would the conclusions be expected to hold with the QJL corrected variant? A qualitative discussion would be sufficient.

  3. MiKV and ZipCache, both cited in the paper, are closely related to the proposed method. Could the authors clarify how SOFT-BIT KV differs from them and, if feasible, compare against them under the same pause-time protocol and storage budget?

3 comments

Dear Reviewer X7pG,

Thanks again for the reviews! We deeply appreciate the time and effort you have invested in evaluating our work :)

We've carefully considered your comments and have prepared a detailed rebuttal addressing each point raised. We are hoping reviewers would be able to review our responses and to engage in a discussion before the end of the discussion period. Please let us know if there are any specific areas you'd like us to clarify further or discuss in more detail. We look forward to your thoughts and are eager to engage in a productive dialogue.

Thank you once again for your commitment to the review process.

Best regards,

Authors

Thank you for the clarifications and additional experiments. The explanation of the compression budgets, the comparisons with MiKV and ZipCache, and the discussion of QJL address my main concerns. I have raised my rating to 3.

Dear Reviewer X7pG,

We sincerely thank the reviewer for the reply and for raising the score :) Glad to know that the main concerns are addressed!

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

To All Reviewers

We'd like to thank all the reviewers for their valuable feedback! During rebuttal, we add the following content to the main paper and appendix to further improve the quality of the paper (all revised text is marked in orange in the PDF):

  1. Byte accounting and matched budgets (App. B, Tab. 6-7\color{blue}{\text{App. B, Tab. 6-7}}): We added one definition of the compression ratio used in every table and corrected the stale 4,096-token description.
  2. Comparison with MiKV and ZipCache (Sec. 2, App. M, Tab. 21\color{blue}{\text{Sec. 2, App. M, Tab. 21}}): We rewritten the related-work to add previous works reviewers suggest, and re-implemented MiKV-style and ZipCache-style baselines under the same coder and byte budget on the needle probe and RULER.
  3. Scorer × allocator ablation (App. M, Tab. 22\color{blue}{\text{App. M, Tab. 22}}): We added six scorers (uniform, H2O, ZipCache, SnapKV-style mean, retrieval-head committee, our MLP) crossed with three allocators to separate the contribution of the learned score from that of the allocation rule.
  4. Compression-ratio sweep (App. G, Tab. 14\color{blue}{\text{App. G, Tab. 14}}): Added TurboQuant vs. Soft-bit KV at 3.9 / 5.1 / 7.5 / 11.8× on the needle probe and RULER across two models.
  5. Detector generalization (Sec. 4.1, App. J, Tab. 17-18\color{blue}{\text{Sec. 4.1, App. J, Tab. 17-18}}): we trained detectors trained on code (MBPP), math (GSM8K) and mixed data to test whether retrieval can be generalized across training data. We further tested whether the retrieval-trained detector can be transferred to math reasoning tasks.
  6. Conversational memory (App. N, Tab. 23\color{blue}{\text{App. N, Tab. 23}}): Added LoCoMo and LongMemEval-S benchmarks on Llama-3.1-8B and Qwen2.5-7B at 8.0× and 11.8× to test long-memory.
  7. QJL-corrected TurboQuant (App. H, Tab. 15\color{blue}{\text{App. H, Tab. 15}}): added storage analysis and a matched-byte retrieval comparison against the MSE-only coder used in the paper.
  8. Fair access to the first question (App. B, Inference protocol\color{blue}{\text{App. B, Inference protocol}}): further clarified which methods see q1 at compression time, and cache is frozen for every method afterwards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.