acceptodds
Under review as a conference paper at ICLR 2027

Read What Matters: Query-Adaptive Quantization for KV Caches

Abstract

KV-cache entries are stored before their future queries are known, but each de- coding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. READKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, com- putes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders. Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more accurate than storing and fully reading four bits at the same payload-read budget. Long-context question answering and retrieval on two instruction-tuned models provide additional quality evidence. On the tested 8K-token, batch-one, single-layer workload on an NVIDIA A10G, a restricted eight-bit READKV reader with a two- bit mean payload-read budget has 39% lower latency than the tested TurboQuant codec.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Query-Adaptivity, Asymmetric read and write

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fMSzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes READKV, which separates the precision for the KV cache from the precision needed to fetch for each query. Keys and values are stored using a progressive scalar code, so different prefixes of the same entry can be read without rewriting the cache. The current query determines the key-channel precision, and attention computed from the reconstructed keys determines the value-token precision. The authors provide separation results on constructed input families and show that a greedy allocator optimizes the stated stagewise distortion objectives. Experiments show small perplexity losses at an average four-bit read budget from an eight-bit cache, while a restricted GPU implementation reduces single-layer latency relative to the baseline.

Critical strengths
  1. The distinction between stored precision and fetched precision is useful for KV caches, where entries are written once and read repeatedly. The proposed two-stage reader follows this setting naturally: it uses the query to allocate key precision and the reconstructed attention to allocate value precision. The fixed-split key-stage ablation and intermediate-depth controls provide evidence that query information and multilevel reconstruction both contribute to the results.

  2. The quality results are fairly consistent. Across six base models, the C4 perplexity increase is small, and READKV is more accurate than fully reading the four-bit version of the same code at a matched payload-read budget.

Critical weaknesses

My main concern is the additional dependency introduced by query-adaptive reads. Key-prefix selection must wait for the current query, and value-prefix selection depends on the attention computed from the reconstructed keys. Unlike fixed-precision attention, the value addresses are therefore not known in advance, which may limit prefetching and overlap between memory transfers and computation. A breakdown of the execution pipeline would help establish when the bandwidth savings outweigh these additional dependencies.

Questions for authors

How do the two-stage read dependencies affect KV prefetching, overlap with computation, and decoding latency?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fMSz

We thank the reviewer for the constructive feedback and positive comments on our work.

Complete-model gains include both allocation stages

ReadKV delivers faster complete-model decoding and higher throughput, including both allocation stages, reconstruction, and scheduling. For Qwen2.5-7B on L40S at 8K, decode-step latency falls by 17.5% at batch 32 and 23.2% at batch 64 (Table 8).

On A10G, the CUDA-core reference reader fits 29 concurrent 8K requests versus Dense’s 15, delivering 1.42× total throughput (Table 8). Dense cannot fit the 29-request batch, so that point has no equal-batch latency comparison. The performance table in point 1 of the General response reports L40S throughput gains of 1.67× at 8K and 1.65× at 32K, using larger supported batches.

ReadKV’s compressed cache allows more requests to share model-weight reads, while selective reading reduces the cache traffic each request generates.

Attention-dependent value reads and prefetching

ReadKV reduces fetched data while preserving stored detail for future queries. It chooses key precision from the query, then value precision from reconstructed attention weights.

Fixed-precision readers know value addresses in advance; ReadKV’s value-refinement addresses depend on attention. Metadata and predetermined base regions can be prefetched, and key-refinement addresses are known after key allocation. Figure 4 profiles this dependency chain in the earlier A10G single-layer fixture at 8K/batch one. Overlap was not measured separately; the complete-model speedups in the General response’s performance table (point 1) include selection and scheduling costs, showing that the net benefit survives these dependencies.

GPU implementation and layout optimization

Tensor-core value aggregation packs selected tokens for computation and reduces instruction and memory-traffic overhead. A separate warm-cache L40S kernel profile at 8K/batch 32 measures 20.0% fewer instructions per scalar and 27.0% less total DRAM traffic, with unchanged payload and allocation decisions (Appendix G.3).

The key layout also supports three-bit reads without fetching a fourth bit plane: one plane across 256 tokens occupies 32 bytes, allowing three planes to be fetched separately. Value packing is handled separately (Appendix L). Lower per-query bandwidth demand also motivates exploring larger, slower memory tiers (Appendix A.8).

AuthorsOpenReview ↗

Follow Up and Gentle Reminder

Dear Reviewer fMSz,

Thank you again for your constructive feedback, particularly your questions about allocation costs and dependencies. With the reviewer update deadline tomorrow, we wanted to follow up on our general response and our reply to your comments, posted a few days ago.

We added complete-model measurements demonstrating faster decoding with the costs of both allocation stages, reconstruction, and scheduling included. We also clarified the dependencies involved in selective reading and provided kernel profiling results to explain the implementation improvements.

We hope these additions and general comments address your concerns. If so, we would be grateful if you would consider revising your score upward. Please let us know if you have any further questions.

Reviewer_KSE2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces READKV, a progressive KV-cache codec with a two-stage query-adaptive reader. It separates retained cache precision from per-query payload reads: the current query selects key-channel prefix depths, reconstructed keys compute attention weights, and the resulting attention selects value-token prefix depths. The paper establishes finite-dimensional separations between query-dependent and query-independent access, proves exact stagewise greedy optimality for calibrated separable proxy objectives, and evaluates the method on language modeling, long-context QA, passkey retrieval, and single-layer GPU benchmarks.

Critical strengths
  1. Addresses a significant problem in the field of systems and modeling. This paper identifies a practical limitation in traditional KV-cache quantization: while cache entries are written only once, subsequent queries may require varying levels of precision across different key channels and value tokens. For long-context decoding tasks, decoupling "storage precision" from "read precision" is a well-conceived and highly relevant approach. This distinction between "storage capacity" and "repeated read traffic" is conceptually clear and innovative, with potential applications that extend well beyond the specific encoding-decoding frameworks explored in this paper.
  2. Presents a technically interesting query-adaptive design. The proposed two-stage reader is a sensible design: query-dependent key precision is selected before reading keys, while value precision is selected after reconstructing attention. The progressive code makes it possible to retain information for later queries without reading all retained bits at every decoding step. The use of a shared progressive representation, rather than writing a new cache representation for each query, is an appealing aspect of the method.
  3. Includes nontrivial theoretical analysis. The paper goes beyond an empirical heuristic. It gives explicit finite-dimensional families on which query-dependent access outperforms query-independent branching readers, proves exact greedy allocation for the stated separable objective under diminishing gains, and relates the surrogate objectives to attention-output error through calibration and residual-correlation factors. The paper is generally careful to state that these are restricted-family separations rather than universal dominance results.
  4. Provides broad empirical evidence and useful controls. This evaluation encompasses six foundation language models evaluated on the C4 benchmark, two instruction-tuned models, and three datasets from the LongBench benchmark. The assessment also covers "passkey" retrieval tasks, various configurations for memory read/write operations, and several mechanistic control interventions. By comparing these against full-read models—which utilize the same progressive code—the study yields valuable insights. Furthermore, the appendix provides useful discussions on topics such as current-query allocation, intermediate precision levels, and physical DRAM traffic.
Critical weaknesses
  1. The main quality results and the main hardware results concern different READKV readers. My primary concern lies in the discrepancy between the configuration used for accuracy and the one profiled on hardware. The main language-modeling and QA evaluations rely on a flexible reader that pools budgets across heads, permits depth zero, uses dev-selected K/V splits, and operates at individual token granularity. In contrast, the GPU latency benchmarks use a significantly restricted reader (per-head budgets, fixed 1:1 K/V rates, a minimum depth of one, and paired 2-token packets). As Appendix F.8 confirms, these restrictions cause substantial quality degradation in low-read regimes. Specifically, the restricted reader exhibits noticeable NLL degradation at an average read budget of R = 2, and passkey retrieval at 8K collapses from 200/200 down to ~100/200 keys. Consequently, the paper currently demonstrates the quality of "READKV-Flex" and the runtime speedup of "READKV-Restricted", but fails to show an apples-to-apples quality–latency Pareto frontier for a single deployable artifact.
  2. The baseline comparison does not fully isolate quality, retained capacity, read cost, workspace, and implementation quality. While the authors are transparent regarding the distinct trade-off targets among KIVI, SparQ, TurboQuant, and READKV, the lack of iso-resource controls obscures whether READKV truly offers Pareto superiority. In particular, READKV (8, 4) preserves substantially more persistent cache capacity than TurboQuant K4/V4, leaving open whether its lower perplexity is merely a consequence of higher retained state. Similarly, the nominal read rate R captures only payload depth rather than total traffic, the hardware benchmark evaluates disparate isolated implementations, and the internal FULL READER baseline relies on READKV's own transforms rather than an independent 4-bit standard. To rigorously establish its advantages, the paper needs strictly matched-capacity and matched-traffic comparisons, supported by a holistic evaluation spanning generation quality, persistent memory, temporary workspace overhead, and actual runtime latency. The practical serving claim is not yet supported by end-to-end evidence. The A10G experiment is a useful microbenchmark, but it measures a single attention layer under CUDA Graph replay, beginning from an existing cache. It excludes cache construction and does not measure full-model decoding, batching under memory pressure, concurrent requests, or end-to-end tokens per second.
  3. The paper itself notes that DENSE remains faster in the tested single-layer workload and that the measurements do not establish an end-to-end serving advantage. This limitation should be reflected more prominently in the abstract and conclusion. In particular, the claim of 39% lower latency should always be scoped to the specific 8K-token, batch-one, single-layer A10G benchmark against the tested TurboQuant implementation. A convincing systems evaluation would include complete-model decode latency, memory-limited throughput, larger batch sizes, continuous batching, cache insertion cost, and total memory including the substantially larger READKV workspace. Statistical evidence is limited for some of the small reported differences.
  4. The C4 evaluation relies on 32 held-out 2,048-token documents per model across up to three transform seeds, while the LongBench benchmark evaluates 100 questions per condition. Although these provide reasonable initial evidence, several reported margins over competing methods are small enough that statistical uncertainty becomes a critical factor. The main tables would benefit considerably from explicit confidence intervals or paired significance tests, particularly for marginal perplexity or F1 score differences. This validation is especially needed because low-read behavior varies substantially across architectures: the R = 2 configuration remains relatively competitive on Qwen, but incurs severe degradation on Yi, DeepSeek, and Mistral. The paper should better characterize this architectural sensitivity and report whether the development-selected K/V budget splits and calibration artifacts remain stable across different calibration sets, transform seeds, and held-out evaluation documents.
Questions for authors
  1. Question on Notation and Editorial Artifact in Appendix E.2.3. Could the authors clarify the notation in Appendix E.2.3 on page 30? In the sentence introducing the dependent rounding step, there appears to be an unresolved editorial mark "(?)" immediately following the marginal probability statement for T_i. Could you confirm whether this question mark is simply a residual drafting artifact to be removed? In addition, would you consider formally defining the rounded depth variable T_i before it is referenced in this introductory paragraph, as its explicit construction only appears later in the formal proof?
  2. Flexible versus restricted reader. Could the authors report the full C4 and LongBench quality tables for the exact restricted GPU policy timed in Appendix G? Alternatively, can they report GPU latency and physical traffic for the flexible policy used in the main quality results? What is the end-to-end quality-latency tradeoff of one fixed deployable configuration?
  3. Matched baseline conditions. Can the authors provide controlled comparisons at matched persistent KV capacity, matched logical-read budget, and matched workspace? How much of the apparent advantage over TurboQuant or KIVI remains after fixing the same dense-layer exemptions and the same implementation/timing boundary?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KSE2

We appreciate the reviewer’s thoughtful comments and recognition of our contributions.

W1 / Q2. Quality and latency of a fixed GPU policy

For Qwen2.5-7B at 8K, tensor-core value aggregation reduces L40S decode-step latency by 17.5% at batch 32 and 23.2% at batch 64 (General response, point 1).

Both GPU readers retain eight bits and read two bits per scalar on average, using the same (3,1) key/value split. The same tensor-core implementation’s single-seed C4 check on A10G gives PPL 10.60 versus Dense’s 10.13 on the same 32 documents (Table 8). The CUDA-core implementation has separate C4 and question-answering results in Table 7 and recovers 200/200 passkeys at 8K on Qwen2.5-7B-Instruct (Table 7). These are direct quality and complete-model runtime measurements of the reported implementations.

W3. Memory-limited concurrency and throughput

ReadKV supports more concurrent requests and higher total decoding throughput within the same GPU memory. On A10G at 8K, CUDA-core ReadKV fits 29 requests versus Dense’s 15, delivering 1.42× throughput (Table 8). Dense cannot fit the 29-request batch, so it has no latency measurement there. On L40S, the tensor-core reader fits 125 versus 64 requests at 8K and 30 versus 15 at 32K, yielding 1.67× and 1.65× throughput, respectively (Table 8).

These measurements run each method at its largest fitting batch on the same GPU, including workspace. ReadKV’s smaller cache allows more requests to share model-weight reads. W1 separately shows lower latency for the same 32 or 64 requests on L40S. Section 5 and Appendix G.2 give the complete-model measurements and protocols.

W2 / Q3. Controls for storage, read volume, and allocation

Retained capacity and repeated read traffic impose different costs. ReadKV retains precision for future queries while each query reads only the refinements it needs. Reading two of eight stored bits cuts ideal payload demand fourfold and reduces the bandwidth a larger, slower memory tier must supply (Appendix A.8). We therefore evaluate quality, storage, and reads jointly, including matched-storage comparisons.

At comparable retained storage, ReadKV W6/R3 cuts accounted reads by about 49%, from KIVI-4’s 37.6% to 19.3% of Dense. Across six models, PPL increases over Dense are 0.71–1.59% for ReadKV versus 0.06–0.10% for KIVI-4 (Table 12); the Qwen3B byte accounting includes side information and dense layers.

The same-code controls show allocation’s contribution: with the eight-bit cache fixed, adaptive reading gives lower perplexity on all six models at both two- and four-bit average reads (Table 12). A broader whole-model paired analysis favors ReadKV in 94 of 96 equal-budget comparisons (Table 16). A separate coding control compares Full Reader with nonprogressive four-bit Lloyd-Max under the same transforms; their perplexity increases differ by only 0.01–0.15 percentage points across four models (Table 12).

Among the timed implementations in Table 10, CUDA-core ReadKV alone combines about 13% of Dense’s logical reads, a 4.54% perplexity increase, and 12% lower step latency. Separate tests of the all-layer FP8/NVFP4 configurations give perplexity increases of 780% and 10,765%. Table 10 also reports layer-protected variants’ quality at 38–57% of Dense’s read bytes; these variants remain untimed.

W4. Statistical support and architecture sensitivity

The additional paired C4 analysis supports ReadKV’s quality advantage: W8/R4 achieves lower perplexity in all 18 model-by-seed comparisons against Full Reader and all 18 against TurboQuant, with identical protected layers in the TurboQuant comparison. Every 95% interval excludes zero (Table 13). The expanded logical-reader QA study uses 200 questions per condition, with paired intervals for all six 7,500-token conditions. The pooled W8/R2–KIVI-2 point estimate favors ReadKV by 0.65 F1, with a 95% interval of [−0.18, +1.50]; the interval includes zero (Table 14).

At R4, transform-seed PPL spread is at most 0.10%, and no headline comparison changes sign. Repeating split selection on two further seeds preserves 9 of 12 choices, including Yi and DeepSeek at both rates (Table 15). Using disjoint calibration documents changes PPL by at most 0.067% across all six models at R4; at R2, changes are at most 0.33% for five models and 2.32% for Mistral (Table 15).

The Mistral layout control demonstrates a concrete quality improvement. In the single-seed logical test at the fixed (3,1) split, switching from paired to per-token values reduces perplexity from 25.51 to 8.26; hybrid packing gives 8.02. The deployed Mistral-7B base-model reader selects (2,2) at R2 (Appendix L; Table 7). This identifies a controllable value-packing choice behind the degradation in this comparison.

Q1. Notation in Appendix E.2.3

The sentence should have cited Gandhi et al. (2006). We have corrected the typo and defined Ti=ℓi+ZiT_i=\ell_i+Z_i, Zi∈{0,1}Z_i\in\{0,1\}, and P(Zi=1)=fiP(Z_i=1)=f_i before Proposition 9 (Appendix E.2.3).

AuthorsOpenReview ↗

Follow Up and Gentle Reminder

Dear Reviewer KSE2,

Thank you again for your detailed review and helpful suggestions. With the reviewer update deadline tomorrow, we wanted to follow up on our general response and our reply to your comments, posted a few days ago.

Your feedback prompted additional quality and runtime measurements of the implemented GPU readers, comparisons controlling for storage and read budgets, and expanded statistical and calibration analyses. We hope these additions address your questions about practical performance, fair comparisons, and stability, and further strengthen the paper.

If these results and general comments address your concerns, we would be grateful if you would consider revising your score upward. Please let us know if you have any further questions.

Reviewer_XWmMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes READKV, which separates KV-cache storage precision from query-time read precision. It uses progressive codes and a two-stage reader that allocates key precision from the query and value precision from reconstructed attention. Theoretical results and experiments demonstrate promising quality–read tradeoffs, while practical serving benefits remain less established.

Critical strengths

S1: KV cache compression is important for serving efficiency.

S2: Separating retained capacity from read volume captures the repeated-access nature of KV caches. Progressive codes enable query-specific precision without rewriting stored entries. The two-stage allocation naturally reflects the different roles of keys and values.

S3: The paper establishes advantages over query-independent readers on explicit input families.

Critical weaknesses

W1: The quality and efficiency evidence concerns materially different settings. The main quality experiments use flexible allocation, whereas the GPU implementation introduces several restrictions. These restrictions significantly affect low-rate retrieval: READKV (8,2) under the restricted settings recovers only 100–101 of 200 passkeys, compared with 200 of 200 under the flexible settings. Therefore, the paper has not yet demonstrated its strongest quality and latency benefits under the same settings.

W2: The systems evaluation does not establish an end-to-end serving advantage. The measurements isolate a single attention layer with an existing cache, and DENSE is faster than READKV throughout the reported timing grid. At 8K tokens and batch size one, READKV (8,2) also remains slower than FULLREADER (4,4), despite fetching less payload. Complete-model measurements under memory-limited concurrency are needed to determine whether reduced traffic offsets allocation overhead, additional storage, and workspace requirements.

W3: The paper identifies Xie et al. (2025) as already separating retained precision from fetched precision, but does not report a direct experimental comparison with that approach.

W4: The allocation objective may not reliably reflect actual attention-output error. It ignores interactions between reconstruction errors across tokens, which Appendix F.7 shows can be significant. Further analysis is needed to establish whether lower objective values consistently correspond to lower actual output error.

Questions for authors

As stated in weak points.

2 comments
AuthorsOpenReview ↗

Response to Reviewer XWmM

We thank the reviewer for the careful review and for highlighting the strengths of our work.

W1. Complete-model quality and latency

For Qwen2.5-7B at 8K, tensor-core value aggregation reduces complete-model L40S decode-step latency by 17.5% at batch 32 and 23.2% at batch 64 (performance table in point 1 of the General response). In a single-seed C4 check on A10G, the same implementation gives PPL 10.60 versus Dense’s 10.13 on the same 32 documents (Table 8).

The CUDA-core reader has three-seed C4 quality measurements and 11.9–13.0% lower L40S latency at the same batches (Table 4). It recovers all 200 passkeys at 8K on Qwen2.5-7B-Instruct (Table 7). Both implementations store eight bits and read two bits per scalar on average, with the same (3,1) key/value budget; the results include the policy and test protocols.

W2. Complete-model throughput under memory constraints

On A10G at 8K, the CUDA-core reader fits 29 simultaneous requests versus Dense's 15 and delivers 1.42× the decode tokens/s, including all 28 layers and each method's workspace (Table 8). Dense cannot fit the 29-request batch, so it has no latency measurement there.

On L40S, tensor-core value aggregation supports 125 versus 64 requests at 8K and 30 versus 15 at 32K, yielding 1.67× and 1.65× the decoding throughput (Table 8). Larger batches share model-weight reads across more requests; ReadKV reduces the added KV-cache storage.

W3. What ReadKV adds to Xie-style selective reading

We thank the reviewer for emphasizing the comparison with Xie et al. (2025). ReadKV combines compressed codes with decodable prefixes and query-dependent key precision, followed by attention-dependent value precision, under explicit read budgets (Appendix B.3).

Our code/allocation control on Qwen3B uses about 13% of Dense's reads. Xie-style tier selection with float truncation raises perplexity by 42.90%; substituting ReadKV's code reduces this to 8.73%, and using ReadKV's allocation too reduces it to 3.85%. Code substitution improves every tested model at both rates. Table 11 reports the results; Appendix H specifies the truncation-and-tier-selection control and metadata accounting.

W4. Allocation objectives and attention-output error

Theorem 13 proves exact optimization of the stagewise proxy objectives under diminishing gains (Appendix E.4). Fixed-input tests across four models measure total attention-output error, including cross-token interactions. Against sampled random and permuted equal-budget allocations, ReadKV’s chosen key allocation has lower error in 89.9–99.9% of comparisons; its value allocation has lower error in every comparison (Table 17). Separate whole-model paired comparisons favor ReadKV in 94 of 96 equal-budget cases (Table 16).

AuthorsOpenReview ↗

Follow Up and Gentle Reminder

Dear Reviewer XWmM,

Thank you again for your thoughtful review. With the reviewer update deadline tomorrow, we wanted to follow up on our general response and our reply to your comments, posted a few days ago.

In response to your feedback, we added complete-model quality and performance measurements, a controlled comparison with Xie-style selective reading, and experiments measuring actual attention-output error. We believe these results clarify ReadKV’s contribution and strengthen the case for exploiting read–write asymmetry in KV-cache quantization.

We hope the general comments and specific responses address your concerns. If so, we would be grateful if you would consider revising your score upward. Please let us know if you have any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment Explaining the central claims and new results

We thank the reviewers for recognizing the value of retaining information for future queries while reading selectively, and for their questions about prior work, allocation cost, and complete-model performance.

During decoding, each step writes only the new token’s KV entry but reads from the accumulated cache. Repeated reads therefore dominate KV-cache bandwidth demand. For Dense Qwen2.5-7B on L40S at 8K/batch 32, writing the initial cache once has an estimated minimum time of 0.5–0.7 ms per request, based on memory bandwidth. Across the subsequent 127 decode steps, repeated cache reads and attention computation together account for about 85 ms per request when averaged across the batch—47% of decoding time. For a scalar read 128 times, eight-bit storage with two-bit reads moves 264 payload bits: 8 written and 256 read. Fully reading a four-bit or uncompressed 16-bit cache instead moves 516 or 2,064 bits, respectively. Compared with the four-bit cache, ReadKV writes four extra bits once but saves 256 bits across repeated reads (Appendix A).

Retained capacity and repeated read traffic impose different costs. ReadKV’s progressive code retains precision for future queries, allowing different queries to access different refinements while each stays within an explicit read budget. Our theory identifies settings where query-dependent reading outperforms query-independent reading. Complete-model tests show faster decoding and higher throughput, including the cost of choosing what to read.

1. Faster decoding and more concurrent requests on the same GPU

For Qwen2.5-7B, ReadKV provides faster decoding at the same batch size and higher total throughput within the same GPU memory:

GPU/contextComparisonReadKV vs. DensePaper
L40S, 8KSame batches: 32 and 6417.5% and 23.2% lower decode-step latencyTable 8
L40S, 8KLargest fitting batch125 vs. 64 concurrent requests; 1.67× total decode throughputTable 8
L40S, 32KLargest fitting batch30 vs. 15 concurrent requests; 1.65× total decode throughputTable 8
A10G, 8KLargest fitting batch29 vs. 15 concurrent requests; 1.42× total decode throughputTable 8

Equal-batch results show faster decoding; memory-limited results show more concurrent requests and higher throughput on the same GPU. Dense cannot fit the 29-request A10G batch, so that point has no equal-batch latency comparison.

The L40S results use tensor-core value aggregation; the A10G capacity results use the CUDA-core reference reader. Each L40S capacity result comes from one successful search per configuration. Measurements include all 28 layers and workspace, with fixed batches during decoding. Prefill and cache construction: Appendix A. Tables 4 and 8 give each implementation’s timing, C4 quality and protocols.

2. Contributions of progressive coding and read allocation

ReadKV’s progressive code reduces prediction-quality loss, and query-dependent allocation reduces it further. We isolate these contributions by replacing the code and then the allocation in a control emulating Xie-style float truncation and ranked-tier selection (Xie et al., 2025) under explicit read budgets. On Qwen3B at about 13% of Dense’s accounted reads:

ReaderPerplexity increase over uncompressed Dense (lower is better)
Xie-style tiers + float truncation42.90%
Same tiers + ReadKV code8.73%
ReadKV code + allocation3.85%

Changing only the code improves the tier reader on all six models at both tested read budgets. The Qwen3B comparison shows allocation’s additional benefit: perplexity increase falls from 8.73% to 3.85%. Appendix H and Table 11 give the control and accounting.

3. Quality benefit and runtime cost of read allocation

At fixed stored precision and equal read budgets, paired comparisons favor ReadKV in 94 of 96 whole-model allocation tests (Table 16). In the logical Qwen7B control, uniform prefixes raise perplexity by 96.45%, versus 2.99% for adaptive ReadKV using the same eight-bit cache and total read budget (Table 12). A separate L40S timing comparison at 8K/batch 32 measures about 1.5 ms more per step for adaptive reading with CUDA-core aggregation than uniform two-bit reads of the same eight-bit cache. Appendix G.3 gives the protocols. The complete-model latency gains above include allocation.

4. Quality comparisons across models and read budgets

Across six models and three seeds, all 36 paired C4 comparisons with eight-bit storage and four-bit average reads (W8/R4) favor ReadKV over Full Reader and TurboQuant; the TurboQuant comparison uses identical protected layers, and every 95% interval excludes zero (Table 13). At storage comparable to KIVI-4, six-bit storage with three-bit average reads (W6/R3) roughly halves accounted reads. Table 12 reports quality, storage, and reads together. Lower read demand also motivates retaining precision in larger, slower memory tiers (Appendix A.8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.