- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces TaSQ, a vector-quantization method for accurate KV-cache compression in the challenging 1-bit regime. TaSQ tailors the quantization space through query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping while preserving the conventional lookup structure and adding little serving overhead. Across general, reasoning, and long-context retrieval tasks, it outperforms existing low-bit VQ baselines and enables up to 14x larger batches and 1.87x higher peak throughput than BF16 on the evaluated GPU.
Critical strengths
- TaSQ achieves strong performance in the challenging 1-bit regime, and the resulting KV-cache compression enables substantially larger batch sizes and higher peak throughput.
- The paper clearly motivates the limitations of existing VQ methods and introduces a targeted solution for each one. Moreover, the proposed components are sufficiently efficient, making the overall method practical for deployment.
- The evaluation across multiple models, together with empirical evidence for each component, provides strong support for the proposed methodology.
Critical weaknesses
- The latency evaluation is limited and would benefit from broader and more fine-grained validation:
- All efficiency experiments are conducted on a single RTX 6000 Ada GPU. Evaluating the method on other hardware would help establish whether the reported gains generalize across platforms.
- At the same batch size, the performance improvement over BF16 appears relatively modest.
- A more detailed latency breakdown would help clarify the sources of overhead, particularly those introduced by pre-RoPE quantization and on-the-fly RoPE computation.
- It would be better to have an ablation on actual generation tasks (Table 4). In particular, the PPL gains from the two components other than covariance-aware grouping seem minor (8.52 -> 8.45), making it difficult to assess how meaningful these improvements are, i.e., how much they translate into actual gains on generation tasks.
Questions for authors
No specific questions other that what i listed in the weakness section.
2 comments
Response to Reviewer bDgx (2/2)
[W2] Ablation on NIAH and AIME
We thank the reviewer for suggesting downstream ablations. We independently remove each TaSQ component and evaluate AIME24 on DeepSeek-R1-Distill-Llama-8B and RULER-NIAH on Llama-3.1-8B-Instruct (the model used in Table 3), averaging over three seeds.
| Method | AIME24 | NIAH 4k | NIAH 8k | NIAH 16k | NIAH 32k | NIAH 64k |
|---|---|---|---|---|---|---|
| BF16 | 53.33 ± 3.33 | 98.50 ± 1.73 | 98.83 ± 0.76 | 98.67 ± 0.29 | 98.67 ± 0.76 | 98.17 ± 1.04 |
| Full TaSQ | 48.89 ± 6.94 | 93.33 ± 2.08 | 93.17 ± 0.76 | 92.00 ± 1.32 | 93.67 ± 1.04 | 93.17 ± 0.76 |
| w/o query-guided weighting | 44.44 ± 5.09 | 92.00 ± 1.00 | 88.67 ± 1.26 | 88.33 ± 2.02 | 90.83 ± 0.76 | 91.33 ± 1.89 |
| w/o cross-head normalization | 41.11 ± 6.94 | 95.17 ± 2.57 | 93.00 ± 1.00 | 92.67 ± 1.26 | 92.33 ± 2.31 | 91.17 ± 0.76 |
| w/o covariance-aware grouping | 24.44 ± 5.09 | 16.17 ± 2.08 | 11.50 ± 3.04 | 7.50 ± 1.32 | 5.67 ± 0.58 | 3.33 ± 0.76 |
The results show that all three components contribute to downstream performance, with covariance-aware grouping having the largest impact. Removing it reduces AIME24 accuracy by 24.4 points and causes NIAH performance to collapse across all context lengths. Query-guided weighting and cross-head normalization also improve AIME24 accuracy by 4.4 and 7.8 points, respectively. For NIAH, query-guided weighting and cross-head normalization have relatively modest effects. These results suggest that even small perplexity differences in Table 4 can translate into meaningful differences in downstream accuracy.
We've included these results and the corresponding Qwen3-4B-Thinking-2507 ablations in Section 6.2, alongside the original perplexity ablation.
Response to Reviewer bDgx (1/2)
[W1] Efficiency analysis on different GPU types
To evaluate TaSQ's serving efficiency across different hardware, we repeated the throughput experiment of Figure 4(a) on an NVIDIA A100 40GB GPU, using 2,048-token prompts and 32,768-token outputs. We report the peak throughput speedup over BF16, with the optimal batch size selected for each method.
TaSQ achieves higher peak throughput on both models, demonstrating that its serving efficiency benefits extend beyond the RTX 6000 Ada GPU. The full throughput curves are provided in Appendix G (Figure 9).
[W1] Speedup at the Same Batch Size
We report throughput at matched batch sizes using the measurements from Figure 4(a), in the small-batch regime where BF16 remains feasible.
As the reviewer noted, TaSQ provides modest throughput gains at small batch sizes, reaching 1.13× at batch size 4 in the 32K-generation setting of Figure 4(a). However, we would also like to highlight two additional benefits of low-bit KV cache compression for serving efficiency.
Higher concurrency. A larger KV cache pool enables more concurrent long-context requests. In Figure 4(a), BF16 supports at most 6 concurrent requests, whereas TaSQ accommodates much larger batches, achieving 1.87× higher peak throughput. This increased capacity also allows more requests to remain in GPU memory, reducing the need for CPU offloading.
Reduced data movement. When offloading or prefill–decode disaggregation is required, a smaller KV footprint directly reduces the amount of data transferred. This can alleviate data movement bottlenecks in long-context serving.
These benefits are particularly relevant to long-context and agentic serving workloads, where KV cache capacity and data movement are increasingly important.
[W1] Latency Breakdown
To identify the sources of TaSQ's runtime overhead, we profile decoding and prefill on Qwen3-4B-Thinking-2507 using an RTX 6000 Ada GPU. We use CUPTI to measure individual kernels within CUDA graphs.
The latency breakdown consists of the following components:
Decode latency. We profile a single decoding step following a 32K-token input prompt at batch sizes 1 and 64. BF16 cannot accommodate batch size 64 due to its KV cache capacity limit. All latencies are in milliseconds per decoding step. RoPE overhead is reported separately and is also included in attention latency.
At batch size 1, KV encoding accounts for only 0.7% of TaSQ's decoding latency, while on-the-fly RoPE accounts for 14% of attention latency. Despite these overheads, TaSQ achieves 9% lower decoding latency than BF16, as reduced KV cache memory traffic outweighs the additional computation. At batch size 64, KV encoding accounts for only 1.0% of decoding latency, while RoPE accounts for 22% of attention latency. BF16 cannot support this batch size due to its KV cache capacity limit.
Prefill latency. We also profile prefill with a 32K-token input prompt at batch size 1, averaging three runs. All latencies are in milliseconds per prompt.
TaSQ's additional prefill cost primarily comes from KV encoding, which requires codebook matching for each 8-dimensional group and accounts for 9.6% of TTFT. FFN, projection, and attention costs remain comparable to BF16, resulting in approximately 10% higher TTFT. TaSQ requires no separate key transformation during prefill, as unrotated keys are already available from the key projection.