- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SealKV, a low-bit KV cache compression framework that integrates online trellis coding, block-mean subtraction (BMS), and fused attention kernels. Experimental results demonstrate that SealKV achieves nearly lossless compression at 3 bits and above, while delivering state-of-the-art performance at 2 bits. On Qwen3-14B, SealKV reduces KV cache memory usage by 6.6x with only a 0.37-point accuracy drop, outperforming all evaluated baselines.
Critical strengths
-
BMS appears to be the main innovation of this work, and the design is effective. The results in Table 3 demonstrate that BMS significantly improves quantization accuracy across various quantizers, showing good generalizability.
-
The experimental evaluation is comprehensive. The paper integrates SealKV into vLLM and provides extensive results on accuracy, throughput, and ablation studies.
Critical weaknesses
-
The novelty of SealKV needs further clarification. The concurrently submitted CPTC paper already explores efficient trellis coding and short-window online KV cache quantization, which appears to overlap with SealKV's first contribution. Moreover, Table 3 shows that BMS-enhanced baselines can match or even outperform SealKV, raising questions about the additional benefits of its more complex trellis coding design.
-
Limited generalizability to real-world deployment scenarios. Throughput evaluation is restricted to Qwen3-8B on a single H100 GPU with homogeneous requests. Despite significant peak throughput gains, performance degrades under small batch sizes.
Questions for authors
-
Compared with BMS-enhanced quantizers (CPTC: A Concurrently Submitted Paper Referenced by SealKV), what additional accuracy-efficiency benefits does SealKV provide?
-
Under dynamic batching and heterogeneous requests, can the throughput gains translate into end-to-end latency improvements in real-world serving workloads?
4 comments
Rebuttal Reply to Reviewer 9QPU [2/3]
W2. Generality of the throughput results, and small batch sizes.
(1) To check other models and GPUs, we repeated the serving measurement of Fig. 8(e) on Qwen3-4B, 8B and 14B on one H200, where the paper used H100. Each request generates 512 tokens, every method gets the same memory cap, and each method is compared with the strongest BF16 in its own engine. Peak decode throughput relative to BF16:
Peak decode throughput relative to BF16 (one H200, new):
| Model | Context | SealKV K4V4 | SealKV K4V2 | SealKV K2V2 | SAW-INT4 |
|---|---|---|---|---|---|
| Qwen3-4B | 16K | 1.68× | 1.80× | 1.94× | 1.12× |
| 32K | 1.79× | 1.94× | 2.05× | 1.14× | |
| 64K | 1.88× | 2.04× | 2.17× | 1.17× | |
| Qwen3-8B | 16K | 1.64× | 1.75× | 1.90× | 1.20× |
| 32K | 1.84× | 2.02× | 2.19× | 1.23× | |
| 64K | 1.99× | 2.21× | 2.38× | 1.24× | |
| Qwen3-14B | 16K | 1.81× | 1.95× | 2.12× | 1.26× |
| 32K | 1.98× | 2.18× | 2.37× | 1.30× | |
| 64K | 2.50× | 2.75× | 3.01× | 1.35× |
SealKV reaches 1.64 to 3.01× BF16 on every model, against 1.12 to 1.35× for SAW-INT4, and as in the paper its gain grows with context. The gain is not only from fitting a larger batch. At BF16's own largest batch, SealKV is already 1.39 to 1.86× faster on Qwen3-4B and 14B.
(2) At batch 1 every compressed cache is slower than BF16 on H100 (Fig. 8c, d and the same-batch columns of Table 14), with SealKV at 0.77 to 0.93×, OSCAR 0.71 to 0.83×, SAW-INT4 0.40 to 0.68× and TurboQuant 0.23 to 0.50×. At batch 1 each decode step reads 16 GB of weights but only 2.4 to 9.7 GB of BF16 cache from 16K to 64K, so a smaller cache saves little, and SealKV's lookup-based attention kernel is slower than FlashAttention-3 there (0.68× at 32K, Table 15). On H200 SealKV is at parity at batch 1, 0.91 to 1.06× across the three models, against 0.28 to 0.70× for SAW-INT4 and 0.19 to 0.45× for TurboQuant.
The advantage appears quickly as the batch grows. In Table 14 on H100, SealKV is 1.22 to 1.29× BF16 at batch 8 at 16K, 1.48 to 1.53× at batch 8 at 32K and 1.45 to 1.51× at batch 4 at 64K, where no baseline exceeds 1.24×. The batch-1 gap also shrinks with context, as SealKV's attention kernel reaches 1.32× FlashAttention-3 at 128K and 2.31× at 1M (Table 15), and with 100K-token prompts a single request is served 3 to 7% faster than with BF16. Heterogeneous requests under dynamic batching are in our answer to Q2.
Q1. Benefits beyond BMS-enhanced quantizers and CPTC.
W1 compares SealKV with BMS-enhanced baselines and with a different trellis codebook. Here we remove one SealKV component at a time on Qwen3-8B and keep everything else unchanged.
| Component removed | Bits | Metric | Without it | SealKV |
|---|---|---|---|---|
| BMS (CPTC + RHT only) | K2V2 | AIME-26, 8 passes | 40.83 | 67.92 |
| BMS (CPTC + RHT only) | K2V2 | WikiText-2 KL ×10⁻³ | 1724.8 | 19.2 |
| RHT (CPTC + BMS only) | K4V4 | NIAH-MK 32K, 100 prompts | 58.0 | 96.0 |
Without BMS, SealKV loses 27 points on AIME-26 at 2 bits, and without RHT it loses 38 points of retrieval at 4 bits. The KL row keeps block means in BF16, at 3.125 bits against 2.125 without them. With the served 4-bit mean records, SealKV K2V2 is 2.41 bits at nearly the same KL.
The design also matters for speed. At L = 16, the trellis window used in the CPTC paper, the same Viterbi encoder is 41× slower at 4 bits and 93× slower at 2 bits than at SealKV's L = 10 on Hopper. For decoding, the L = 16 codebook takes 128 KiB in BF16, 64× SealKV's 2 KiB table, so it cannot be replicated per lane in shared memory as our kernel does, and read from global memory its lookups are 5.4× slower. L = 10 costs only 0.14 effective bits (Table 1). Block sealing, the fused encoder, which is up to 2.3× faster than CPTC's kernel (Sec. 3.2), and the fused attention kernel are what turn the smaller cache into the 2.81 to 3.44× serving gain at 64K.
Rebuttal Reply to Reviewer 9QPU [3/3]
Q2. dynamic batching and heterogeneous requests
We thank the reviewer for this system point. We added new measurements in two settings, both with vLLM's continuous batching.
(1) Real reasoning traces on Qwen3-8B, where requests replay the prompts and output lengths of our Qwen3-8B reasoning runs (GPQA-D, AIME-25/26, MATH-500, HumanEval), max_gen_len=32K. The inference engine keeps C requests in flight.
| Concurrent requests C | 1 | 4 | 16 | 32 | 64 |
|---|---|---|---|---|---|
| Request throughput vs. BF16 | 0.93–0.96× | 1.00–1.02× | 1.22–1.27× | 1.38–1.44× | 1.29–1.66× |
| Median end-to-end latency vs. BF16 | 1.08–1.10× | 1.05–1.07× | 0.79–0.83× | 0.75–0.92× | 0.50–0.76× |
Ranges span K4V4, K4V2 and K2V2; a latency ratio below 1 means faster. From 16 concurrent requests on, SealKV both serves more requests per second and finishes the median request sooner. At C = 64, K2V2 serves 1.66× the requests with half the median latency.
(2) For heterogeneous requests, we served the workload from vLLM's benchmark blog post with vLLM on H200, with 150 requests with 20K input and 2K output tokens (±15%), arriving either as a Poisson stream at 0.5 req/s or all at once.
| Arrivals | Metric | BF16 | SealKV | Speedup (↑) |
|---|---|---|---|---|
| Poisson, 0.5 req/s | Time to finish all 150 requests (s) | 426 | 341 | 1.25× |
| Poisson, 0.5 req/s | Median time to first token (s) | 35.7 | 1.75 | 20.4× |
| All 150 at once | Time to finish all 150 requests (s) | 405 | 297 | 1.36× |
| All 150 at once | Median time to first token (s) | 133 | 73 | 1.82× |
Under Poisson arrivals BF16 can't keep up and requests pile up in the queue. Its median time to first token is 35.7 s, while SealKV's is 1.75 s, 20× lower. Because SealKV's KV cache is smaller, more requests run at the same time, which matters most with long contexts and long outputs.
Dear Reviewer 9QPU,
Thank you again for your review. We posted our responses to every point and a revised PDF. We would sincerely appreciate that, when you have a moment, let us know whether they address your concerns. We will be happy to clarify anything further.
Rebuttal Reply to Reviewer 9QPU [1/3]
We sincerely thank the reviewer for the review and address each point below.
W1. Novelty relative to CPTC, and BMS-enhanced baselines in Table 3 (now Table 5).
We thank the reviewer for raising both points.
First, we need to clarify that SealKV is quantizer-agnostic. Its accuracy comes from SealKV's own design, whichever trellis codebook sits inside it. To show this, we replaced CPTC with QTIP's 1MAD codebook at the same L = 10 and kept everything else unchanged. Results on Qwen3-8B:
Either codebook on its own breaks the model, with KL 12.6 to 22.7× higher. Inside SealKV both match BF16 within noise on AIME-26 and GPQA-D, and CPTC keeps only a small KL edge.
From CPTC, SealKV takes the trellis code itself, meaning the codebook construction, the L = 10 operating point and the base Viterbi search (Sec. 2.1, Appendix F). CPTC is a quantization format. Its paper mostly evaluates weight quantization on LLaMA-3.1 and does not apply CPTC to KV caches at all. It notes that L = 10 would make its encoder fast enough for a KV cache (CPTC Sec. 4.3), but it reports no KV-cache accuracy and has no attention kernel or serving system.
What is new is everything that makes a trellis code work on a growing cache. Block sealing (Sec. 3.1) codes each complete 16-token block once, in a batched launch that never waits on the newest token. Block-mean subtraction (Sec. 2.2 and 3.2) removes per-channel offsets shared by neighbouring tokens, which rotation cannot remove. Without it, 2-bit KL is 90× higher and AIME-26 drops from 67.92 to 40.83. Together with an RHT and a per-token scale, it needs no calibration. The fused encoder is up to 2.3× faster than CPTC's and writes bit-identical codes, and the fused attention kernel decodes in registers and adds each block mean as one term. That gives 2.5 to 2.9× faster attention than FlashAttention-3 (Fig. 8a) and 2.81 to 3.44× BF16's peak decode throughput at 64K in vLLM (Table 14).
Second, Table 5 shows that BMS, our own technique, transfers to other quantizers, and it lifts all three by up to 32.2 points. The trellis code still matters on top of it. With BMS added, the baselines trail SealKV on AIME-25 and KL on both GPT-OSS models while using equal or more bits, and on GPQA-D and retrieval none is more than 0.8 points above it, within noise. Scores are from Table 13 and bits from Table 5, where BMS adds 0.25.
On Qwen3-8B we also ran a fully controlled comparison, giving every baseline both BMS and SealKV's 128-token sink and 256-token window, and measured KL from BF16 on WikiText-2 (×10⁻³).
Across the three models SealKV's KL is 1.2 to 3.4× lower than every BMS-enhanced baseline at a similar or lower bit rate, and it has the best AIME-25 on both GPT-OSS models by 3.4 to 8.3 points. The trellis code also pays off in speed, since SealKV reaches 2.81 to 3.44× BF16's peak decode throughput at 64K, against at most 1.38× for SAW-INT4 and 2.00× for OSCAR (Table 14).