- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes READKV, which separates the precision for the KV cache from the precision needed to fetch for each query. Keys and values are stored using a progressive scalar code, so different prefixes of the same entry can be read without rewriting the cache. The current query determines the key-channel precision, and attention computed from the reconstructed keys determines the value-token precision. The authors provide separation results on constructed input families and show that a greedy allocator optimizes the stated stagewise distortion objectives. Experiments show small perplexity losses at an average four-bit read budget from an eight-bit cache, while a restricted GPU implementation reduces single-layer latency relative to the baseline.
Critical strengths
-
The distinction between stored precision and fetched precision is useful for KV caches, where entries are written once and read repeatedly. The proposed two-stage reader follows this setting naturally: it uses the query to allocate key precision and the reconstructed attention to allocate value precision. The fixed-split key-stage ablation and intermediate-depth controls provide evidence that query information and multilevel reconstruction both contribute to the results.
-
The quality results are fairly consistent. Across six base models, the C4 perplexity increase is small, and READKV is more accurate than fully reading the four-bit version of the same code at a matched payload-read budget.
Critical weaknesses
My main concern is the additional dependency introduced by query-adaptive reads. Key-prefix selection must wait for the current query, and value-prefix selection depends on the attention computed from the reconstructed keys. Unlike fixed-precision attention, the value addresses are therefore not known in advance, which may limit prefetching and overlap between memory transfers and computation. A breakdown of the execution pipeline would help establish when the bandwidth savings outweigh these additional dependencies.
Questions for authors
How do the two-stage read dependencies affect KV prefetching, overlap with computation, and decoding latency?
2 comments
Follow Up and Gentle Reminder
Dear Reviewer fMSz,
Thank you again for your constructive feedback, particularly your questions about allocation costs and dependencies. With the reviewer update deadline tomorrow, we wanted to follow up on our general response and our reply to your comments, posted a few days ago.
We added complete-model measurements demonstrating faster decoding with the costs of both allocation stages, reconstruction, and scheduling included. We also clarified the dependencies involved in selective reading and provided kernel profiling results to explain the implementation improvements.
We hope these additions and general comments address your concerns. If so, we would be grateful if you would consider revising your score upward. Please let us know if you have any further questions.
Response to Reviewer fMSz
We thank the reviewer for the constructive feedback and positive comments on our work.
Complete-model gains include both allocation stages
ReadKV delivers faster complete-model decoding and higher throughput, including both allocation stages, reconstruction, and scheduling. For Qwen2.5-7B on L40S at 8K, decode-step latency falls by 17.5% at batch 32 and 23.2% at batch 64 (Table 8).
On A10G, the CUDA-core reference reader fits 29 concurrent 8K requests versus Dense’s 15, delivering 1.42× total throughput (Table 8). Dense cannot fit the 29-request batch, so that point has no equal-batch latency comparison. The performance table in point 1 of the General response reports L40S throughput gains of 1.67× at 8K and 1.65× at 32K, using larger supported batches.
ReadKV’s compressed cache allows more requests to share model-weight reads, while selective reading reduces the cache traffic each request generates.
Attention-dependent value reads and prefetching
ReadKV reduces fetched data while preserving stored detail for future queries. It chooses key precision from the query, then value precision from reconstructed attention weights.
Fixed-precision readers know value addresses in advance; ReadKV’s value-refinement addresses depend on attention. Metadata and predetermined base regions can be prefetched, and key-refinement addresses are known after key allocation. Figure 4 profiles this dependency chain in the earlier A10G single-layer fixture at 8K/batch one. Overlap was not measured separately; the complete-model speedups in the General response’s performance table (point 1) include selection and scheduling costs, showing that the net benefit survives these dependencies.
GPU implementation and layout optimization
Tensor-core value aggregation packs selected tokens for computation and reduces instruction and memory-traffic overhead. A separate warm-cache L40S kernel profile at 8K/batch 32 measures 20.0% fewer instructions per scalar and 27.0% less total DRAM traffic, with unchanged payload and allocation decisions (Appendix G.3).
The key layout also supports three-bit reads without fetching a fourth bit plane: one plane across 256 tokens occupies 32 bytes, allowing three planes to be fetched separately. Value packing is handled separately (Appendix L). Lower per-query bandwidth demand also motivates exploring larger, slower memory tiers (Appendix A.8).