acceptodds
Under review as a conference paper at ICLR 2027

Tailoring the Quantization Space for 1-Bit KV Cache Compression

Abstract

The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwidth. To mitigate this bottleneck, vector quantization (VQ) has emerged as a promising approach for aggressive KV cache compression. However, existing VQ methods degrade substantially in the 1-bit regime. At such extreme compression, each codebook must represent a larger group of channels with a limited set of centroids, making effective use of its capacity increasingly challenging. To address this, we introduce , which tailors the VQ target space by combining query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping to better reflect the error sensitivity and statistical structure of cached activations. Since these transforms are RoPE-compatible and can be easily merged into projection weights and codebooks, TaSQ preserves the conventional VQ lookup structure and adds negligible serving overhead. Across general, long-chain-of-thought reasoning, and long-context retrieval benchmarks, TaSQ consistently outperforms existing low-bit KV cache VQ baselines while preserving reasoning stability. On a single RTX 6000 Ada GPU, its SGLang implementation supports up to larger batch sizes and achieves higher peak throughput compared to the BF16 baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, vector quantization, low-bit quantization, large language model inference, efficient serving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bDgxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces TaSQ, a vector-quantization method for accurate KV-cache compression in the challenging 1-bit regime. TaSQ tailors the quantization space through query-guided channel weighting, cross-head normalization, and covariance-aware channel grouping while preserving the conventional lookup structure and adding little serving overhead. Across general, reasoning, and long-context retrieval tasks, it outperforms existing low-bit VQ baselines and enables up to 14x larger batches and 1.87x higher peak throughput than BF16 on the evaluated GPU.

Critical strengths
  1. TaSQ achieves strong performance in the challenging 1-bit regime, and the resulting KV-cache compression enables substantially larger batch sizes and higher peak throughput.
  2. The paper clearly motivates the limitations of existing VQ methods and introduces a targeted solution for each one. Moreover, the proposed components are sufficiently efficient, making the overall method practical for deployment.
  3. The evaluation across multiple models, together with empirical evidence for each component, provides strong support for the proposed methodology.
Critical weaknesses
  1. The latency evaluation is limited and would benefit from broader and more fine-grained validation:
  • All efficiency experiments are conducted on a single RTX 6000 Ada GPU. Evaluating the method on other hardware would help establish whether the reported gains generalize across platforms.
  • At the same batch size, the performance improvement over BF16 appears relatively modest.
  • A more detailed latency breakdown would help clarify the sources of overhead, particularly those introduced by pre-RoPE quantization and on-the-fly RoPE computation.
  1. It would be better to have an ablation on actual generation tasks (Table 4). In particular, the PPL gains from the two components other than covariance-aware grouping seem minor (8.52 -> 8.45), making it difficult to assess how meaningful these improvements are, i.e., how much they translate into actual gains on generation tasks.
Questions for authors

No specific questions other that what i listed in the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer bDgx (1/2)

[W1] Efficiency analysis on different GPU types

To evaluate TaSQ's serving efficiency across different hardware, we repeated the throughput experiment of Figure 4(a) on an NVIDIA A100 40GB GPU, using 2,048-token prompts and 32,768-token outputs. We report the peak throughput speedup over BF16, with the optimal batch size selected for each method.

ModelTaSQ / BF16
Qwen3-4B-Thinking-25071.58×
DeepSeek-R1-Distill-Llama-8B2.27×

TaSQ achieves higher peak throughput on both models, demonstrating that its serving efficiency benefits extend beyond the RTX 6000 Ada GPU. The full throughput curves are provided in Appendix G (Figure 9).

[W1] Speedup at the Same Batch Size

We report throughput at matched batch sizes using the measurements from Figure 4(a), in the small-batch regime where BF16 remains feasible.

Batch size1248
TaSQ / BF161.01×1.09×1.13×BF16 OOM

As the reviewer noted, TaSQ provides modest throughput gains at small batch sizes, reaching 1.13× at batch size 4 in the 32K-generation setting of Figure 4(a). However, we would also like to highlight two additional benefits of low-bit KV cache compression for serving efficiency.

  1. Higher concurrency. A larger KV cache pool enables more concurrent long-context requests. In Figure 4(a), BF16 supports at most 6 concurrent requests, whereas TaSQ accommodates much larger batches, achieving 1.87× higher peak throughput. This increased capacity also allows more requests to remain in GPU memory, reducing the need for CPU offloading.

  2. Reduced data movement. When offloading or prefill–decode disaggregation is required, a smaller KV footprint directly reduces the amount of data transferred. This can alleviate data movement bottlenecks in long-context serving.

These benefits are particularly relevant to long-context and agentic serving workloads, where KV cache capacity and data movement are increasingly important.

[W1] Latency Breakdown

To identify the sources of TaSQ's runtime overhead, we profile decoding and prefill on Qwen3-4B-Thinking-2507 using an RTX 6000 Ada GPU. We use CUPTI to measure individual kernels within CUDA graphs.

The latency breakdown consists of the following components:

  • FFN: feed-forward network computation.
  • QKV/O projections: query, key, value, and output projections.
  • Attention: attention computation, including on-the-fly RoPE for TaSQ.
  • RoPE: estimated overhead of on-the-fly rotation, measured by disabling RoPE within the same fused attention kernel and taking the difference in execution time.
  • KV encoding: codebook matching to compress KV entries into quantized indices.
  • KV cache management: cache writes and index bookkeeping.
  • Other: normalization, activation, LM head, and sampling.
  • Wall time / TTFT: measured separately without profiling and may differ slightly from the sum of individual kernel times.

Decode latency. We profile a single decoding step following a 32K-token input prompt at batch sizes 1 and 64. BF16 cannot accommodate batch size 64 due to its KV cache capacity limit. All latencies are in milliseconds per decoding step. RoPE overhead is reported separately and is also included in attention latency.

Batch sizeMethodFFNQKV/O proj.AttentionRoPEKV encodingKV cache managementOtherWall time
1BF166.232.307.24––0.071.2416.93
1TaSQ6.192.375.020.710.110.431.2715.48
64TaSQ7.062.71290.963.83.081.341.95303.6

At batch size 1, KV encoding accounts for only 0.7% of TaSQ's decoding latency, while on-the-fly RoPE accounts for 14% of attention latency. Despite these overheads, TaSQ achieves 9% lower decoding latency than BF16, as reduced KV cache memory traffic outweighs the additional computation. At batch size 64, KV encoding accounts for only 1.0% of decoding latency, while RoPE accounts for 22% of attention latency. BF16 cannot support this batch size due to its KV cache capacity limit.

Prefill latency. We also profile prefill with a 32K-token input prompt at batch size 1, averaging three runs. All latencies are in milliseconds per prompt.

MethodFFNQKV/O proj.AttentionKV encodingKV cache managementOtherTTFT
BF161,2414722,380–122054,309
TaSQ1,1484732,3234521042074,730

TaSQ's additional prefill cost primarily comes from KV encoding, which requires codebook matching for each 8-dimensional group and accounts for 9.6% of TTFT. FFN, projection, and attention costs remain comparable to BF16, resulting in approximately 10% higher TTFT. TaSQ requires no separate key transformation during prefill, as unrotated keys are already available from the key projection.

AuthorsOpenReview ↗

Response to Reviewer bDgx (2/2)

[W2] Ablation on NIAH and AIME

We thank the reviewer for suggesting downstream ablations. We independently remove each TaSQ component and evaluate AIME24 on DeepSeek-R1-Distill-Llama-8B and RULER-NIAH on Llama-3.1-8B-Instruct (the model used in Table 3), averaging over three seeds.

MethodAIME24NIAH 4kNIAH 8kNIAH 16kNIAH 32kNIAH 64k
BF1653.33 ± 3.3398.50 ± 1.7398.83 ± 0.7698.67 ± 0.2998.67 ± 0.7698.17 ± 1.04
Full TaSQ48.89 ± 6.9493.33 ± 2.0893.17 ± 0.7692.00 ± 1.3293.67 ± 1.0493.17 ± 0.76
w/o query-guided weighting44.44 ± 5.0992.00 ± 1.0088.67 ± 1.2688.33 ± 2.0290.83 ± 0.7691.33 ± 1.89
w/o cross-head normalization41.11 ± 6.9495.17 ± 2.5793.00 ± 1.0092.67 ± 1.2692.33 ± 2.3191.17 ± 0.76
w/o covariance-aware grouping24.44 ± 5.0916.17 ± 2.0811.50 ± 3.047.50 ± 1.325.67 ± 0.583.33 ± 0.76

The results show that all three components contribute to downstream performance, with covariance-aware grouping having the largest impact. Removing it reduces AIME24 accuracy by 24.4 points and causes NIAH performance to collapse across all context lengths. Query-guided weighting and cross-head normalization also improve AIME24 accuracy by 4.4 and 7.8 points, respectively. For NIAH, query-guided weighting and cross-head normalization have relatively modest effects. These results suggest that even small perplexity differences in Table 4 can translate into meaningful differences in downstream accuracy.

We've included these results and the corresponding Qwen3-4B-Thinking-2507 ablations in Section 6.2, alongside the original perplexity ablation.

Reviewer_UuXROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies vector quantization of the KV cache at about one bit per channel. It changes the space in which keys are quantized. Keys are reweighted by query sensitivity, normalized across KV heads, and grouped by covariance while keeping RoPE pairs together. The transforms are folded into projection weights and codebooks, so serving keeps the standard VQ lookup. Experiments cover five models, general tasks, long chain-of-thought reasoning, and long-context retrieval, with an SGLang implementation on an RTX 6000 Ada.

Critical strengths
  1. Strong 1-bit retrieval: At 64K, TaSQ reaches 93.17 on Llama-3.1-8B and 81.50 on Qwen3-4B-Thinking, while NSNQuant gets 82.17 and 73.00 (Table 3). CQ and NovaKV collapse at this length.

  2. Practical serving path: The transforms are absorbed into weights and codebooks, so there is no runtime channel shuffle. The SGLang implementation raises the KV pool by 12.48x and peak throughput by 1.87x over BF16 (Figure 4).

Critical weaknesses
  1. Thin component ablation: Table 4 reports only WikiText-2 perplexity on one base model, with no variance. Removing grouping raises PPL from 8.4559 to 10.1731, but removing weighting or normalization changes it by only 0.05 to 0.06.

  2. Calibration and overhead claims: The channel weights, covariance, grouping, and Fisher weights all come from the same 64 GPQA/codeparrot windows, and no calibration sensitivity is reported. The abstract also calls the serving overhead negligible, while Section 6.1 reports a 10-14% TTFT increase.

Questions for authors
  1. How much do the results change with calibration size and domain?

  2. Can you report downstream accuracy for the component ablation, not just WikiText-2 PPL?

  3. What is the latency-throughput curve, and how does TaSQ behave for short outputs where the 10-14% TTFT overhead is not amortized?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UuXR (1/3)

[W1] Ablation on NIAH and AIME

We thank the reviewer for suggesting downstream ablations. We independently remove each TaSQ component and evaluate AIME24 on DeepSeek-R1-Distill-Llama-8B and RULER-NIAH on Llama-3.1-8B-Instruct (the model used in Table 3), averaging over three seeds.

MethodAIME24NIAH 4kNIAH 8kNIAH 16kNIAH 32kNIAH 64k
BF1653.33 ± 3.3398.50 ± 1.7398.83 ± 0.7698.67 ± 0.2998.67 ± 0.7698.17 ± 1.04
Full TaSQ48.89 ± 6.9493.33 ± 2.0893.17 ± 0.7692.00 ± 1.3293.67 ± 1.0493.17 ± 0.76
w/o query-guided weighting44.44 ± 5.0992.00 ± 1.0088.67 ± 1.2688.33 ± 2.0290.83 ± 0.7691.33 ± 1.89
w/o cross-head normalization41.11 ± 6.9495.17 ± 2.5793.00 ± 1.0092.67 ± 1.2692.33 ± 2.3191.17 ± 0.76
w/o covariance-aware grouping24.44 ± 5.0916.17 ± 2.0811.50 ± 3.047.50 ± 1.325.67 ± 0.583.33 ± 0.76

The results show that all three components contribute to downstream performance, with covariance-aware grouping having the largest impact. Removing it reduces AIME24 accuracy by 24.4 points and causes NIAH performance to collapse across all context lengths. Query-guided weighting and cross-head normalization also improve AIME24 accuracy by 4.4 and 7.8 points, respectively. For NIAH, query-guided weighting and cross-head normalization have relatively modest effects. These results suggest that even small perplexity differences in Table 4 can translate into meaningful differences in downstream accuracy.

We've included these results and the corresponding Qwen3-4B-Thinking-2507 ablations in Section 6.2, alongside the original perplexity ablation.

AuthorsOpenReview ↗

Response to Reviewer UuXR (2/3)

[W2, Q1] Robustness to Different Calibration Settings

To evaluate calibration robustness, we tested TaSQ, CQ, and NovaKV on Qwen3-4B under varying calibration sizes (16–128, maintaining 25% code data) and domains (GPQA-only, mixed, code-only, and general text from WikiText-2), while keeping all other settings unchanged.

GSM8K (BF16 score: 86.28)

Method163264 (paper)128GPQA onlyCode onlyWikiText-2Range
TaSQ85.3785.5285.4482.0385.4482.5684.843.5
CQ71.0464.2270.3662.8562.4055.2739.6531.4
NovaKV83.1783.0984.5383.6283.1781.2082.183.3

HumanEval (BF16 score: 81.71)

Method163264 (paper)128GPQA onlyCode onlyWikiText-2Range
TaSQ74.3978.0577.4479.2776.2279.8878.055.5
CQ60.9865.2465.8562.2060.3762.2058.547.3
NovaKV76.2276.8376.2273.1773.1771.9571.954.9

Across these calibration settings, TaSQ achieves the highest accuracy in 12 of 14 cases and maintains consistently high performance despite variations in calibration size and domain. Its accuracy varies by only 3.5 points on GSM8K and 5.5 points on HumanEval. In contrast, CQ exhibits greater sensitivity to calibration data, particularly on GSM8K, where its accuracy drops to 39.65 with WikiText-2 calibration. Overall, these results demonstrate that TaSQ is robust to changes in calibration settings.

We have included both tables in the appendix.

[W2] Serving Overhead

We thank the reviewer for pointing out the ambiguity. By "negligible serving overhead," we intended to refer to the additional overhead of TaSQ's components beyond standard VQ, rather than the overhead of VQ relative to BF16. Since the proposed transformations are folded into the projection weights and codebooks, TaSQ requires no additional runtime transformations. This is why we primarily compare its serving efficiency with CQ, which uses a standard VQ lookup.

AuthorsOpenReview ↗

Response to Reviewer UuXR (3/3)

[Q3] Throughput–Latency Analysis

We further analyze the throughput–latency trade-off using the setting of Figure 4(a), using different numbers of output tokens. We compare the end-to-end latency of both methods at similar throughput levels, using BF16's peak throughput as the reference. Here, latency includes both prefilling and decoding time.

Output tokensMethodBatch sizeThroughput (tok/s)Latency (s)Latency reduction
1,024BF16764,00958.2—
TaSQ644,16547.219%
2,048BF16572,49293.7—
TaSQ482,60875.420%
4,096BF16381,457160.3—
TaSQ321,574124.922%
8,192BF1622807279.2—
TaSQ16809202.627%
16,384BF1612432512.0—
TaSQ8429343.833%

At comparable throughput levels, TaSQ achieves 19–33% lower end-to-end latency by serving requests with smaller batch sizes. The latency reduction becomes more pronounced at longer output lengths, as TaSQ achieves comparable throughput with smaller batch sizes and shorter completion times. We include these results in the appendix.

[Q3] Efficiency with Shorter Outputs

We extended the experiment in Figure 4(a) to shorter generations, varying the output length from 1,024 to 32,768 tokens. For each output length, we report the peak throughput speedup of TaSQ over BF16, with the batch size optimized for each method.

Output tokensTaSQ / BF16
1,0241.10×
2,0481.23×
4,0961.29×
8,1921.47×
16,3841.64×
32,768 (Fig. 4a)1.87×

TaSQ exceeds BF16 in peak throughput at every output length, already 1.10× at 1K output tokens, and the speedup increases to 1.87× at 32K as KV cache capacity becomes increasingly important. Such long generations are common in agentic or reasoning workloads. For example, the median BF16 generation length on LiveCodeBench-v6 is 8.2K tokens.

We've included these results in the appendix.

Thank you for your response. All of my concerns have been addressed, and I will maintain my positive score.

Reviewer_Pi9DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TaSQ is a KV-cache vector quantizer at about 1.25 bits per value. Keys are quantized before RoPE after a query-guided channel weighting, a per-token RMS normalization and a RoPE-pair-preserving channel grouping, all folded into W_K and the codebooks. It beats CQ, NSNQuant and NovaKV on general, long-CoT and NIAH benchmarks, and in SGLang it gives a 12.48x larger KV pool and 1.87x peak throughput over BF16.

Critical strengths

The design is serving aware (the transforms fold offline and decoding stays a lookup), and there's a real SGLang and Triton implementation. The gains are largest where it matters: a long-CoT average of 60.55 vs 46.67 for the best baseline on Qwen3-4B-Thinking, and 93.17 vs 82.17 on NIAH at 64k with Llama-3.1-8B. NIAH and reasoning results use three seeds.

Critical weaknesses
  1. The only component ablation is WikiText-2 perplexity, where removing the query guided weighting or the normalization changes perplexity by less than 1%. The big wins are on NIAH and long-CoT, which aren't ablated, so it's unclear what drives them.
  2. Most components are inherited (pre-RoPE keys from KVQuant/CQ, Fisher-weighted k-means from CQ, normalization from NSNQuant), and the grouping cost is the Gaussian distortion bound from Optimized Product Quantization (Ge et al., 2013), which isn't cited. CommVQ which does 1-2 bit KV-cache VQ with a RoPE-commutative codebook, is neither cited nor compared.
  3. All baselines are reimplemented. NovaKV runs outside its published setting and scores 0.00 on NIAH from 16k up on Qwen3-4B-Thinking, which may reflect the configuration more than the method. There are no non-VQ baselines at matched memory, such as 2-bit KIVI or SnapKV.
  4. Efficiency is shown for one 4B model against BF16 and CQ only. Throughput at batch 16 is already about 1.56x the BF16 peak (Fig. 4a), so milder compression such as FP8 or 4-bit KV might get much of the gain at a smaller accuracy cost; AIME'25 on Qwen3-4B-Thinking drops from 74.44 to 56.67.
Questions for authors
  1. Can you ablate the components on NIAH and AIME?
  2. How does TaSQ compare with CommVQ at matched bits?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Pi9D (1/4)

[Q1, W1] Ablation on NIAH and AIME

We thank the reviewer for suggesting downstream ablations. We independently remove each TaSQ component and evaluate AIME24 on DeepSeek-R1-Distill-Llama-8B and RULER-NIAH on Llama-3.1-8B-Instruct (the model used in Table 3), averaging over three seeds.

MethodAIME24NIAH 4kNIAH 8kNIAH 16kNIAH 32kNIAH 64k
BF1653.33 ± 3.3398.50 ± 1.7398.83 ± 0.7698.67 ± 0.2998.67 ± 0.7698.17 ± 1.04
Full TaSQ48.89 ± 6.9493.33 ± 2.0893.17 ± 0.7692.00 ± 1.3293.67 ± 1.0493.17 ± 0.76
w/o query-guided weighting44.44 ± 5.0992.00 ± 1.0088.67 ± 1.2688.33 ± 2.0290.83 ± 0.7691.33 ± 1.89
w/o cross-head normalization41.11 ± 6.9495.17 ± 2.5793.00 ± 1.0092.67 ± 1.2692.33 ± 2.3191.17 ± 0.76
w/o covariance-aware grouping24.44 ± 5.0916.17 ± 2.0811.50 ± 3.047.50 ± 1.325.67 ± 0.583.33 ± 0.76

The results show that all three components contribute to downstream performance, with covariance-aware grouping having the largest impact. Removing it reduces AIME24 accuracy by 24.4 points and causes NIAH performance to collapse across all context lengths. Query-guided weighting and cross-head normalization also improve AIME24 accuracy by 4.4 and 7.8 points, respectively. For NIAH, query-guided weighting and cross-head normalization have relatively modest effects. These results suggest that even small perplexity differences in Table 4 can translate into meaningful differences in downstream accuracy.

We've included these results and the corresponding Qwen3-4B-Thinking-2507 ablations in Section 6.2, alongside the original perplexity ablation.

AuthorsOpenReview ↗

Response to Reviewer Pi9D (2/4)

[W2] Components Inherited from Previous Studies

We acknowledge that several components of TaSQ are inspired by prior work. However, we believe our main contribution lies not in simply combining existing techniques, but in identifying the challenges of near-1-bit KV cache quantization, applying targeted algorithms to address them, and empirically validating their effectiveness. Specifically:

  • Query-guided weighting: We observe substantial variation in query activation distributions across channels (Figure 1(a)), implying different sensitivities to key quantization errors. Motivated by this, we derive channel-wise weights from expected attention-logit errors and incorporate them into the quantization space, empirically reducing attention-logit error and attention KL divergence (Figure 3(a)).

  • Covariance-aware grouping: We observe highly non-uniform correlations between key channels (Figure 1(b)), motivating channel grouping that exploits these dependencies. We empirically validate the Gaussian distortion criterion even in the near-1-bit regime (Figure 3(b–c)) and propose hierarchical matching to efficiently find RoPE-pair-preserving channel groups, reducing reconstruction error compared with contiguous grouping (Figure 3(d)).

  • Pre-RoPE quantization: We observe that RoPE substantially disperses key distributions (Figure 1(c)), making them harder to represent with a finite codebook. We quantify this effect and show that pre-RoPE quantization reduces total reconstruction error by 35% across 32 layers (Figure 1(d)).

Together, these techniques address key challenges in KV cache quantization and enable accurate VQ in the near-1-bit regime, where existing methods suffer substantial accuracy degradation.

[W2] Relation to Optimized Product Quantization (OPQ)

We thank the reviewer for pointing out this relevant work. While we already cite Gersho & Gray (2012) for the classical rate-distortion theory underlying our grouping criterion, we agree that OPQ (Ge et al., 2013) is closely related and have added the citation. Unlike OPQ's general orthogonal transformations, our method optimizes RoPE-pair preserving channel permutations through hierarchical matching.

[W2, Q2] Comparison with CommVQ

We thank the reviewer for pointing out CommVQ. We additionally evaluate CommVQ using its released codebook for Llama-3.1-8B-Instruct. We follow the evaluation protocol of Table 1, using the same prompts, metrics, and 64-token full-precision sliding window.

MethodBitsGSM8KMATH500MBPPHumanEvalAvg.
BF161683.6242.2059.6062.2061.91
CommVQ-1bit1.0374.6814.6051.6045.1246.50
CommVQ-2bit2.0082.7934.4058.8062.2059.55
TaSQ1.27/1.2581.7334.0058.4058.5458.17

TaSQ achieves an 11.7-point higher average accuracy than CommVQ-1bit, with the largest improvement on MATH500 (34.0 vs. 14.6), while using a moderately higher bitwidth. Compared with CommVQ-2bit, TaSQ achieves a similar average accuracy (58.17 vs. 59.55) while using approximately 37% fewer bits. Moreover, TaSQ retains a single-stage VQ structure, whereas CommVQ requires multiple codebook lookups and aggregation due to its multi-stage residual quantization.

We have cited CommVQ and discussed these methodological differences in the related work.

AuthorsOpenReview ↗

Response to Reviewer Pi9D (3/4)

[W3] Baseline Implementation and Configuration

Baseline implementation. We implemented all baselines in a single SGLang serving stack so that every method is evaluated under identical conditions, with the same model weights, KV cache management, and evaluation pipeline. This setup minimizes framework-related differences, allowing a more direct comparison of the quantization methods in terms of accuracy and serving efficiency. NSNQuant and CQ do not provide SGLang implementations, so we integrated them into this stack, while for NovaKV we directly ported its official SGLang implementation rather than reimplementing it.

NovaKV configuration. We mostly keep NovaKV's original design, and only adjust its published configuration to target the 1-bit regime. The key codebook is set to match CQ and TaSQ for fair comparison, and the value quantization is reduced from 2 bits to 1 bit. To compare the key VQ methods independently of differences in value quantization, we also provide a key-only comparison with full-precision values in Figure 5.

Long-context degradation of NovaKV. Our position-wise analysis in Appendix D (Figure 7) provides further insight into NovaKV's NIAH degradation. Within the 2,048-token calibration range, NovaKV achieves lower attention-logit error than CQ and NSNQuant. However, its error increases by more than an order of magnitude at later positions, whereas TaSQ maintains the lowest error throughout. This behavior may arise because NovaKV calibrates on post-RoPE keys, whose position-dependent rotations extend beyond those observed during calibration.

[W3] Comparison with KIVI and SnapKV

We thank the reviewer for suggesting non-VQ baselines. Our original evaluation focused on VQ methods because they generally offer better accuracy in the extreme low-bit regime targeted by TaSQ. To provide a broader comparison, we additionally evaluate KIVI using its official implementation ported to our SGLang framework. SnapKV is a token eviction method and is largely orthogonal to TaSQ, as token eviction and quantization can be combined.

We evaluate both 1-bit and 2-bit KIVI. We use a group size of 128 to match the effective bitwidth of 1-bit KIVI (1.25 bits) to TaSQ's memory budget (1.258 bits). The 2-bit configuration uses approximately 1.8× more memory. All results are obtained on Qwen3-4B-Thinking-2507 over three seeds.

MethodBitwidthAIME24NIAH 4k8k16k32k64k
KIVI 1-bit (G=128)1.250.00 ± 0.000.00 ± 0.000.00 ± 0.000.000.000.00
KIVI 2-bit (G=128)2.2561.11 ± 5.0996.50 ± 1.5094.50 ± 0.8791.67 ± 1.7684.17 ± 2.2568.50 ± 2.18
TaSQ1.25868.89 ± 1.9299.83 ± 0.2998.67 ± 0.7698.83 ± 0.5898.50 ± 0.0081.50 ± 1.50

At a comparable memory budget, 1-bit KIVI fails on all evaluated tasks. Even with 1.8× more memory, 2-bit KIVI trails TaSQ by 7.8 percentage points on AIME24 and 14.3 and 13.0 points on NIAH at 32k and 64k, demonstrating TaSQ's advantage in both compression efficiency and long-context accuracy.

AuthorsOpenReview ↗

Response to Reviewer Pi9D (4/4)

[W4] Efficiency Analysis with an Additional Model

To evaluate the generality of TaSQ's serving efficiency, we additionally benchmarked DeepSeek-R1-Distill-Llama-8B on a single RTX 6000 Ada GPU, using the same serving setup as Figure 4(a). We report the peak throughput speedup over BF16 at each generation length, with the batch size optimized for each method.

Output lengthDeepSeek-R1-Distill-Llama-8BQwen3-4B-Thinking-2507
2,0481.28×1.23×
8,1921.82×1.47×
16,3842.22×1.64×
32,7682.45×1.87×

TaSQ consistently improves peak throughput on both models, with larger gains at longer generation lengths. On DeepSeek-R1-Distill-Llama-8B, the speedup increases from 1.28× at 2K to 2.45× at 32K, confirming that TaSQ's serving efficiency benefits extend beyond Qwen3-4B.

[W4] Benefits of 1-Bit Quantization over Milder Compression

We agree that milder KV cache quantization such as FP8 or 4-bit can offer a comparable throughput gain in some serving scenarios, particularly when a 2–4× reduction is enough to keep the workload compute-bound. However, peak throughput alone does not capture the benefits of aggressive compression, which lie in how much KV cache a GPU can hold and move.

  • Capacity. In the setting of Figure 4(a) (RTX 6000 Ada, 2,048-token prompts, 32,768 output tokens), the KV pool holds 6 concurrent requests in BF16. A 2–4× reduction raises this to roughly 12–24, while TaSQ's 12.48× pool holds 84. This larger KV capacity allows TaSQ to serve substantially more concurrent long-context requests without offloading, an advantage that becomes increasingly important as sequence lengths grow.
  • Data movement. KV cache transfers can become a bottleneck in long-context serving, particularly during CPU offloading or prefill–decode disaggregation. TaSQ reduces transfer volume by 12.48× compared with BF16, whereas FP8 and 4-bit quantization provide 2–4× reductions.

In summary, aggressive KV cache compression offers system-level benefits beyond peak throughput, particularly in memory-constrained long-context serving. TaSQ makes these benefits accessible in the 1-bit regime while maintaining practical model accuracy. In contrast, conventional scalar quantization struggles at such low bitwidths. KIVI-1bit scores 0 across all evaluated tasks, while KIVI-2bit uses 1.8× more memory than TaSQ yet achieves 7.8 points lower accuracy on AIME24 (see our response on KIVI).

Comments on the paper

1 comment
Area_Chair_SDyvOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.