- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PiCoKV, a training-free KV-cache quantization framework that performs attention directly on packed binary representations. It factorizes keys and values into binary codes and shared decoding matrices, enabling both query-key scoring and value aggregation through bitwise operations without reconstructing full-precision global KV states. The method combines Grouped Progressive Coding (GPC), RoPE-aware High-Frequency Masking (HFM), and Error-Aware Selection (EAS). Experiments on Llama-3.1-8B-Instruct and Qwen3-8B demonstrate competitive accuracy on RULER, LongBench, InfiniteBench, and NIAH. The authors report up to 3.45× decoding throughput over FlashAttention-2 and 6.3× KV-cache compression at a 256K context.
Critical strengths
-
Efficient code-domain attention design. The main contribution is extending binary representations to both query-key scoring and weighted-value aggregation. Unlike conventional KV quantization, PiCoKV performs these operations directly on packed codes without reconstructing full-precision global KV states. Although its key-side formulation builds on BinaryPC, extending this approach to compressed full attention, particularly value aggregation, represents a meaningful technical contribution.
-
Promising accuracy and efficiency results. PiCoKV achieves accuracy close to full attention across multiple long-context benchmarks while substantially improving decoding throughput. The reported 3.45× speedup at 256K is encouraging. Ablations on GPC, HFM, EAS, and code length provide useful evidence supporting the design choices.
Critical weaknesses
-
Insufficient comparisons with closely related methods. The evaluation primarily compares against KIVI, OScaR, and PolarQuant, but lacks comparisons with direct quantized-attention implementations such as HACK, TurboAttention, and BitDecoding. Moreover, the relationship to BinaryPC could be better clarified. Experiments are also limited to two 8B models, leaving scalability to other architectures insufficiently explored.
-
Incomplete end-to-end overhead analysis. The paper does not sufficiently evaluate the complete cost of online compression. The default online variant requires input-specific projection fitting, residual updates, EAS selection, and bit packing, whereas Appendix B.1 reports prefill latency only for PiCoKV-Fixed. The 3.45× throughput result is measured during decoding with a synthetic 256K KV cache, rather than complete requests. Additionally, the reported 6.3× compression lacks a detailed accounting of decoding matrices, dense tokens, EAS exceptions, and auxiliary buffers.
Questions for authors
Q1. Novelty and comparison with BinaryPC. BinaryPC already introduces binary key factorization, bitwise query-key scoring, and optimized CUDA kernels for hashing and retrieval. Could the authors clarify the differences in PiCoKV's coding algorithm, bitwise kernels, and prefill pipeline? A comparison of GPC against BinaryPC's coding algorithm at matched code lengths would help isolate the algorithmic contribution. Although BinaryPC uses sparse attention rather than KV quantization, comparing accuracy, prefill latency, decoding throughput, and memory footprint under consistent settings would help demonstrate PiCoKV's practical advantages.
Q2. Online encoding overhead. Could the authors report complete online prefill latency, including projection fitting, GPC, EAS, and packing, and evaluate end-to-end request latency across different input/output lengths?
Q3. Memory accounting and cache maintenance. Does the reported 6.3× compression include decoding matrices, dense tokens, EAS exceptions, metadata, and temporary buffers? How are newly generated KV states managed during long decoding sequences?
Q4. Stronger efficiency baselines. Could the authors compare against HACK, TurboAttention, and BitDecoding under matched hardware, accuracy, and physical memory budgets, and provide a breakdown of the decoding overheads?
4 comments
W2/Q3: Memory Accounting
The following table reports the complete KV cache footprint on Llama-3.1-8B-Instruct with batch size 1. It includes packed codes, decoder matrices, retained dense K/V, and all necessary persistent auxiliary data. Online uses EAS = 0.01, and its footprint includes EAS-retained dense K/V; Fixed does not use EAS.
| Method | KV Cache at 128K (GiB) | KV Cache at 256K (GiB) |
|---|---|---|
| FlashAttention-2 | 16.01 | 32.01 |
| PiCoKV Online | 2.87 | 5.64 |
| PiCoKV-Fixed | 2.60 | 5.10 |
The reported 6.3× compression corresponds to Fixed at 256K: 32.01 / 5.10 ≈ 6.3.
Temporary buffers are not included in the persistent KV cache footprint. To account for runtime memory requirements, we additionally report peak GPU memory on a single H800 GPU at 128K with batch size 1.
| Method | Prefill Peak (GiB) ↓ | Decode Peak (GiB) ↓ |
|---|---|---|
| FlashAttention-2 | 45.56 | 31.25 |
| KIVI | 32.50 | 19.54 |
| OScaR | 32.58 | 18.03 |
| PolarQuant | 32.80 | 18.56 |
| BitDecoding INT2 | 32.56 | 18.01 |
| PiCoKV Online | 25.36 | 17.91 |
| PiCoKV-Fixed | 25.11 | 17.64 |
These peak measurements include model weights, activations, temporary buffers, and the KV cache. Both PiCoKV variants have lower prefill and decoding peak memory than all evaluated baselines.
Q3: KV Cache Maintenance During Decoding
In the current anonymous repository, newly generated K/V states are appended to the dense cache in their original precision, while the compressed global cache remains unchanged.
For longer outputs, we have implemented a periodic maintenance mechanism that encodes and packs the oldest local tokens using existing projections, moves them into the compressed global cache, and retains sink and recent local tokens in dense form. However, this mechanism is not included in the version currently available in our anonymous repository.
Q4: Breakdown of Decoding Time
The following table reports PiCoKV-Fixed’s accumulated GPU kernel time on Llama-3.1-8B-Instruct using a single H800 GPU, 320/320-bit codes, and batch size 1. Times are in ms/token.
Accumulated GPU kernel time sums individual kernel durations, counting overlapping execution separately while excluding host-side overhead and gaps between kernels; it therefore differs from wall-clock decoding latency, and its reciprocal does not directly correspond to the reported throughput.
| Component | Operation | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|
| Attention | Packed attention | 1.49 | 2.69 | 5.00 | 9.55 |
| Attention | Fused value decoding & merge | 2.21 | 2.22 | 2.23 | 2.21 |
| Attention | Query quantization & packing | 0.10 | 0.10 | 0.10 | 0.10 |
| Attention | Dense sink/local attention | 0.25 | 0.25 | 0.26 | 0.26 |
| Other | Model weight GEMMs | 6.21 | 6.20 | 6.26 | 6.24 |
| Other | Cache append | 0.33 | 0.34 | 0.34 | 0.34 |
| Other | Normalization / elementwise | 1.63 | 1.64 | 1.66 | 1.65 |
| Total | GPU kernel time | 12.31 | 13.52 | 15.92 | 20.42 |
The listed components account for over 99% of the reported total GPU time.
Query quantization and packing converts the projected query to INT8 and packs it into 64-bit words. Fused value decoding and merge applies the Value decoder to aggregated codes, combines the dense contribution, and normalizes the output. Model weight GEMMs include MLP, Q/K/V projections, and the LM head.
Packed attention accounts for the growing decoding cost as context length increases, while the other listed operations remain relatively stable.
We thank the reviewer again for the valuable feedback. If you have any further questions or concerns, we would be happy to address them before the discussion period ends. If our responses have adequately addressed your concerns, we would appreciate it if you could reconsider your assessment.
Thank you for your detailed response. I will maintain my score of acceptance.
We thank the reviewer for the valuable feedback. We address the concerns below.
W1/Q1: Novelty and Comparison with BinaryPC
BinaryPC uses binary Key codes for token retrieval and then computes sparse attention using the original K/V. PiCoKV encodes both K and V and computes attention directly on packed codes.
Matched-code-length comparison. We replace PiCoKV’s coding scheme with BinaryPC coding, using 320-bit codes for both K and V in online mode. We compare each complete coding scheme: PiCoKV uses GPC with HFM, while BinaryPC uses BPC without HFM.
PiCoKV’s complete coding scheme achieves higher quality at the same coding budget. Figure 4 additionally demonstrates the encoding-latency benefit of grouped fitting.
Complete-pipeline comparison. The following results use Llama-3.1-8B-Instruct at 128K with batch size 1. BinaryPC uses 64-bit Key retrieval codes; PiCoKV uses 320-bit codes for both K and V.
At 128K, PiCoKV achieves higher quality and decoding throughput, lower prefill latency, and substantially lower KV cache storage.
W1/Q4: Comparison with Direct Quantized-Attention Methods
We additionally evaluate BitDecoding INT2 on Llama-3.1-8B-Instruct at 128K, using a single H800 GPU with batch size 1. Both PiCoKV variants use 320/320-bit codes. Online uses EAS = 0.01; Fixed does not use EAS.
Under the same hardware, both PiCoKV variants achieve higher quality and decoding throughput with lower storage than the evaluated BitDecoding configuration. Fixed also has lower prefill latency.
For HACK, we inspected the public code but could not establish a complete runnable model integration for the evaluated workload. For TurboAttention, we did not locate a public implementation during our search.
W1: Extension to MLA
We extend PiCoKV to GLM-4.7-Flash (MLA), encoding its shared 512-dimensional KV latent into 1280-bit codes for both QK and WV computation. Preliminary RULER results remain close to BF16:
W2/Q2: Online Prefill and Request Latency
Figure 8 reports PiCoKV-Fixed. We additionally report complete Online prefill latency, including projection fitting, GPC, EAS, and packing. Experiments use Llama-3.1-8B-Instruct, one H800 GPU, batch size 1, and 320/320-bit PiCoKV codes.
Request latency comprises prefill and decoding and can be estimated as prefill latency plus the number of decode steps multiplied by average per-token decoding latency. We report these components separately while varying input and output lengths.
Varying input length. We vary input length from 32K to 128K while fixing output length to 64 tokens:
Varying output length. We vary output length from 512 to 8K tokens while fixing input length to 128K. PiCoKV-Fixed enables periodic KV cache maintenance (Q3), similar to the other quantized baselines. The table reports average decode latency (ms/token).