acceptodds
Under review as a conference paper at ICLR 2027

SealKV: Accurate Low-Bit KV Compression for Faster LLM Inference

16 views

Abstract

Long-context reasoning and agentic workloads have made the key–value (KV) cache the dominant cost of LLM serving: it is re-read at every decoding step and grows with every generated token, bounding both decoding speed and concurrency. Compressing it pays off only if it combines high quality, low memory use, and fast serving, yet existing quantization methods sacrifice at least one: they require offline calibration, lose accuracy at low bit widths, or save memory without a measured speedup. We introduce SealKV, the first KV-cache compression method that achieves all three, by bringing a trellis-coding quantizer to a cache that grows continuously. Our key insight is that quantization can take place in sealed blocks, each coded only once all its tokens have arrived; sealing decouples trellis encoding from the newest token and exposes a subtractable per-channel offset, shared by neighboring tokens, that rotation alone cannot remove; subtracting it also lifts three existing quantizers by up to 32 points on GPT-OSS-120B retrieval. We also observe that a shorter trellis is also affordable online at little loss: the code's quality depends far less on its window size than its encoding cost does. Co-designed fused dequantization–attention kernels read the codes directly and reconstruct values in registers without writing them back to memory. Without calibration, SealKV is effectively lossless at three bits and above and state-of-the-art at two, where it shrinks KV cache memory by while losing as little as 0.37 points on Qwen3-14B and outscoring all evaluated baselines. Integrated end to end into vLLM, SealKV raises peak decode throughput at 64K context to that of BF16 at 4 bits and at 2 bits, ahead of all evaluated baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
calibration-free quantization, KV cache compression, efficient reasoning, efficient agentic serving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9QPUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SealKV, a low-bit KV cache compression framework that integrates online trellis coding, block-mean subtraction (BMS), and fused attention kernels. Experimental results demonstrate that SealKV achieves nearly lossless compression at 3 bits and above, while delivering state-of-the-art performance at 2 bits. On Qwen3-14B, SealKV reduces KV cache memory usage by 6.6x with only a 0.37-point accuracy drop, outperforming all evaluated baselines.

Critical strengths
  • BMS appears to be the main innovation of this work, and the design is effective. The results in Table 3 demonstrate that BMS significantly improves quantization accuracy across various quantizers, showing good generalizability.

  • The experimental evaluation is comprehensive. The paper integrates SealKV into vLLM and provides extensive results on accuracy, throughput, and ablation studies.

Critical weaknesses
  • The novelty of SealKV needs further clarification. The concurrently submitted CPTC paper already explores efficient trellis coding and short-window online KV cache quantization, which appears to overlap with SealKV's first contribution. Moreover, Table 3 shows that BMS-enhanced baselines can match or even outperform SealKV, raising questions about the additional benefits of its more complex trellis coding design.

  • Limited generalizability to real-world deployment scenarios. Throughput evaluation is restricted to Qwen3-8B on a single H100 GPU with homogeneous requests. Despite significant peak throughput gains, performance degrades under small batch sizes.

Questions for authors
  1. Compared with BMS-enhanced quantizers (CPTC: A Concurrently Submitted Paper Referenced by SealKV), what additional accuracy-efficiency benefits does SealKV provide?

  2. Under dynamic batching and heterogeneous requests, can the throughput gains translate into end-to-end latency improvements in real-world serving workloads?

4 comments
AuthorsOpenReview ↗

Rebuttal Reply to Reviewer 9QPU [1/3]

We sincerely thank the reviewer for the review and address each point below.

W1. Novelty relative to CPTC, and BMS-enhanced baselines in Table 3 (now Table 5).

We thank the reviewer for raising both points.

First, we need to clarify that SealKV is quantizer-agnostic. Its accuracy comes from SealKV's own design, whichever trellis codebook sits inside it. To show this, we replaced CPTC with QTIP's 1MAD codebook at the same L = 10 and kept everything else unchanged. Results on Qwen3-8B:

KV formatMethodBits/valueKLD ×10⁻³ ↓AIME-26 ↑GPQA-D ↑
BF1616065.0060.35
K4V4SealKV (CPTC)4.411.62765.0060.23
SealKV + QTIP-1MAD4.411.75365.8362.37
QTIP-1MAD alone4.12539.87--
CPTC alone4.12533.71--
K4V2SealKV (CPTC)3.414.54967.5060.73
SealKV + QTIP-1MAD3.415.20065.8359.34
QTIP-1MAD alone3.12566.95--
CPTC alone3.12557.53--

Either codebook on its own breaks the model, with KL 12.6 to 22.7× higher. Inside SealKV both match BF16 within noise on AIME-26 and GPQA-D, and CPTC keeps only a small KL edge.

From CPTC, SealKV takes the trellis code itself, meaning the codebook construction, the L = 10 operating point and the base Viterbi search (Sec. 2.1, Appendix F). CPTC is a quantization format. Its paper mostly evaluates weight quantization on LLaMA-3.1 and does not apply CPTC to KV caches at all. It notes that L = 10 would make its encoder fast enough for a KV cache (CPTC Sec. 4.3), but it reports no KV-cache accuracy and has no attention kernel or serving system.

What is new is everything that makes a trellis code work on a growing cache. Block sealing (Sec. 3.1) codes each complete 16-token block once, in a batched launch that never waits on the newest token. Block-mean subtraction (Sec. 2.2 and 3.2) removes per-channel offsets shared by neighbouring tokens, which rotation cannot remove. Without it, 2-bit KL is 90× higher and AIME-26 drops from 67.92 to 40.83. Together with an RHT and a per-token scale, it needs no calibration. The fused encoder is up to 2.3× faster than CPTC's and writes bit-identical codes, and the fused attention kernel decodes in registers and adds each block mean as one term. That gives 2.5 to 2.9× faster attention than FlashAttention-3 (Fig. 8a) and 2.81 to 3.44× BF16's peak decode throughput at 64K in vLLM (Table 14).

Second, Table 5 shows that BMS, our own technique, transfers to other quantizers, and it lifts all three by up to 32.2 points. The trellis code still matters on top of it. With BMS added, the baselines trail SealKV on AIME-25 and KL on both GPT-OSS models while using equal or more bits, and on GPQA-D and retrieval none is more than 0.8 points above it, within noise. Scores are from Table 13 and bits from Table 5, where BMS adds 0.25.

GPT-OSS-120BBitsAIME-25GPQA-DKLD ↓NIAH 16KNIAH 32K
SAW-INT4 + BMS4.7590.880.00.13594.881.3
TQ-4bit + BMS4.5090.080.00.14293.780.4
NVFP4 + BMS4.7588.380.80.13193.880.4
SealKV4.5294.280.40.08898.083.0
GPT-OSS-20BBitsAIME-25GPQA-DKLD ↓NIAH 16KNIAH 32K
SAW-INT4 + BMS4.7585.872.50.14695.285.9
TQ-4bit + BMS4.5082.572.50.15995.885.6
NVFP4 + BMS4.7584.272.70.13794.686.1
SealKV4.5290.873.00.04795.086.0

On Qwen3-8B we also ran a fully controlled comparison, giving every baseline both BMS and SealKV's 128-token sink and 256-token window, and measured KL from BF16 on WikiText-2 (×10⁻³).

Qwen3-8BBitsKL ↓
SAW-INT4 + BMS4.532.21
TQ-4bit + BMS4.471.88
NVFP4 + BMS4.782.65
SealKV K4V44.381.52
TQ-3bit + BMS3.475.44
SealKV K4V23.414.64

Across the three models SealKV's KL is 1.2 to 3.4× lower than every BMS-enhanced baseline at a similar or lower bit rate, and it has the best AIME-25 on both GPT-OSS models by 3.4 to 8.3 points. The trellis code also pays off in speed, since SealKV reaches 2.81 to 3.44× BF16's peak decode throughput at 64K, against at most 1.38× for SAW-INT4 and 2.00× for OSCAR (Table 14).

AuthorsOpenReview ↗

Rebuttal Reply to Reviewer 9QPU [2/3]

W2. Generality of the throughput results, and small batch sizes.

(1) To check other models and GPUs, we repeated the serving measurement of Fig. 8(e) on Qwen3-4B, 8B and 14B on one H200, where the paper used H100. Each request generates 512 tokens, every method gets the same memory cap, and each method is compared with the strongest BF16 in its own engine. Peak decode throughput relative to BF16:

Peak decode throughput relative to BF16 (one H200, new):

ModelContextSealKV K4V4SealKV K4V2SealKV K2V2SAW-INT4
Qwen3-4B16K1.68×1.80×1.94×1.12×
32K1.79×1.94×2.05×1.14×
64K1.88×2.04×2.17×1.17×
Qwen3-8B16K1.64×1.75×1.90×1.20×
32K1.84×2.02×2.19×1.23×
64K1.99×2.21×2.38×1.24×
Qwen3-14B16K1.81×1.95×2.12×1.26×
32K1.98×2.18×2.37×1.30×
64K2.50×2.75×3.01×1.35×

SealKV reaches 1.64 to 3.01× BF16 on every model, against 1.12 to 1.35× for SAW-INT4, and as in the paper its gain grows with context. The gain is not only from fitting a larger batch. At BF16's own largest batch, SealKV is already 1.39 to 1.86× faster on Qwen3-4B and 14B.

(2) At batch 1 every compressed cache is slower than BF16 on H100 (Fig. 8c, d and the same-batch columns of Table 14), with SealKV at 0.77 to 0.93×, OSCAR 0.71 to 0.83×, SAW-INT4 0.40 to 0.68× and TurboQuant 0.23 to 0.50×. At batch 1 each decode step reads 16 GB of weights but only 2.4 to 9.7 GB of BF16 cache from 16K to 64K, so a smaller cache saves little, and SealKV's lookup-based attention kernel is slower than FlashAttention-3 there (0.68× at 32K, Table 15). On H200 SealKV is at parity at batch 1, 0.91 to 1.06× across the three models, against 0.28 to 0.70× for SAW-INT4 and 0.19 to 0.45× for TurboQuant.

The advantage appears quickly as the batch grows. In Table 14 on H100, SealKV is 1.22 to 1.29× BF16 at batch 8 at 16K, 1.48 to 1.53× at batch 8 at 32K and 1.45 to 1.51× at batch 4 at 64K, where no baseline exceeds 1.24×. The batch-1 gap also shrinks with context, as SealKV's attention kernel reaches 1.32× FlashAttention-3 at 128K and 2.31× at 1M (Table 15), and with 100K-token prompts a single request is served 3 to 7% faster than with BF16. Heterogeneous requests under dynamic batching are in our answer to Q2.

Q1. Benefits beyond BMS-enhanced quantizers and CPTC.

W1 compares SealKV with BMS-enhanced baselines and with a different trellis codebook. Here we remove one SealKV component at a time on Qwen3-8B and keep everything else unchanged.

Component removedBitsMetricWithout itSealKV
BMS (CPTC + RHT only)K2V2AIME-26, 8 passes40.8367.92
BMS (CPTC + RHT only)K2V2WikiText-2 KL ×10⁻³1724.819.2
RHT (CPTC + BMS only)K4V4NIAH-MK 32K, 100 prompts58.096.0

Without BMS, SealKV loses 27 points on AIME-26 at 2 bits, and without RHT it loses 38 points of retrieval at 4 bits. The KL row keeps block means in BF16, at 3.125 bits against 2.125 without them. With the served 4-bit mean records, SealKV K2V2 is 2.41 bits at nearly the same KL.

The design also matters for speed. At L = 16, the trellis window used in the CPTC paper, the same Viterbi encoder is 41× slower at 4 bits and 93× slower at 2 bits than at SealKV's L = 10 on Hopper. For decoding, the L = 16 codebook takes 128 KiB in BF16, 64× SealKV's 2 KiB table, so it cannot be replicated per lane in shared memory as our kernel does, and read from global memory its lookups are 5.4× slower. L = 10 costs only 0.14 effective bits (Table 1). Block sealing, the fused encoder, which is up to 2.3× faster than CPTC's kernel (Sec. 3.2), and the fused attention kernel are what turn the smaller cache into the 2.81 to 3.44× serving gain at 64K.

AuthorsOpenReview ↗

Rebuttal Reply to Reviewer 9QPU [3/3]

Q2. dynamic batching and heterogeneous requests

We thank the reviewer for this system point. We added new measurements in two settings, both with vLLM's continuous batching.

(1) Real reasoning traces on Qwen3-8B, where requests replay the prompts and output lengths of our Qwen3-8B reasoning runs (GPQA-D, AIME-25/26, MATH-500, HumanEval), max_gen_len=32K. The inference engine keeps C requests in flight.

Concurrent requests C14163264
Request throughput vs. BF160.93–0.96×1.00–1.02×1.22–1.27×1.38–1.44×1.29–1.66×
Median end-to-end latency vs. BF161.08–1.10×1.05–1.07×0.79–0.83×0.75–0.92×0.50–0.76×

Ranges span K4V4, K4V2 and K2V2; a latency ratio below 1 means faster. From 16 concurrent requests on, SealKV both serves more requests per second and finishes the median request sooner. At C = 64, K2V2 serves 1.66× the requests with half the median latency.

(2) For heterogeneous requests, we served the workload from vLLM's benchmark blog post with vLLM on H200, with 150 requests with 20K input and 2K output tokens (±15%), arriving either as a Poisson stream at 0.5 req/s or all at once.

ArrivalsMetricBF16SealKVSpeedup (↑)
Poisson, 0.5 req/sTime to finish all 150 requests (s)4263411.25×
Poisson, 0.5 req/sMedian time to first token (s)35.71.7520.4×
All 150 at onceTime to finish all 150 requests (s)4052971.36×
All 150 at onceMedian time to first token (s)133731.82×

Under Poisson arrivals BF16 can't keep up and requests pile up in the queue. Its median time to first token is 35.7 s, while SealKV's is 1.75 s, 20× lower. Because SealKV's KV cache is smaller, more requests run at the same time, which matters most with long contexts and long outputs.

Dear Reviewer 9QPU,

Thank you again for your review. We posted our responses to every point and a revised PDF. We would sincerely appreciate that, when you have a moment, let us know whether they address your concerns. We will be happy to clarify anything further.

Reviewer_GjeyOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SealKV compresses the KV cache with trellis coding. Its main idea is block sealing. A completed block of 16 tokens is quantized once, after all its tokens have arrived, so a short trellis code can run online without waiting for the newest token. Before coding, SealKV applies a fixed Hadamard rotation, subtracts a per-channel block mean shared by the 16 tokens, and scales each token to the codebook variance. A fused attention kernel decodes the codes in registers and never writes dequantized K or V back to HBM.

The paper evaluates Qwen3 4B, 8B, and 14B on reasoning, coding, retrieval, and LongBench, GPT-OSS 20B and 120B, agentic tasks, KL divergence, kernel throughput, and vLLM serving on one H100. The authors report calibration-free accuracy within 0.7 points of BF16 at 3 bits and above, the best evaluated 2-bit accuracy at 2.41 bits per value with 6.6 times less KV memory, and 2.81 to 3.44 times BF16 decode throughput at 64K context.

Critical strengths

The system turns a lower bit rate into measured serving gains. The fused kernel reads codes directly, avoids dequantized writeback, and reaches 2.81 times BF16 throughput at 4 bits and 3.44 times at 2 bits at 64K. The equal-batch gains of 1.6 to 1.7 times near capacity at 16K and 32K show that the speedup is not only a capacity effect. This is the main reason to accept.

BMS is a simple and general finding. Adding it to SAW-INT4, TurboQuant, and NVFP4 improves all 30 model, quantizer, and metric combinations on GPT-OSS, with up to 32.2 points on 32K retrieval. This shows that the insight is not tied to CPTC and gives the community a cheap improvement for existing quantizers.

Critical weaknesses

The serving evaluation is decode-only, and quality at the headline 64K setting is not measured. The difference method cancels prefill, so the paper does not report prefill encoding cost or time to first token. At 64K the model uses YaRN with factor 4, but accuracy is reported only up to 32K NIAH. A 3.44 times decode speedup is less convincing if prefill becomes slower or if quality at 64K is unknown. This directly affects the main practical claim.

The baseline comparison conflates several advantages. SealKV protects 128 sink tokens and a 256-token window in BF16, while the paper does not report giving SAW-INT4, NVFP4, or TurboQuant the same protection. On GPT-OSS the gap is large, and the BMS ablation recovers most of it, but sink protection is not ablated on those baselines. The TurboQuant baseline is also unusually weak, with 0.00 NIAH on Qwen3-4B and 56.30 mean on Qwen3-8B. The state-of-the-art claim would be stronger with matched protected regions and a validated baseline implementation.

Questions for authors
  1. What are the prefill latency and time to first token for SealKV at 16K, 32K, and 64K, including the encoding cost? Can you also report end-to-end throughput for a workload with long prompts and short generations?
  2. What is the accuracy at 64K with YaRN factor 4, especially on NIAH and LongBench? The current accuracy results stop at 32K.
  3. How do SAW-INT4, NVFP4, and TurboQuant perform when they receive the same 128-token BF16 sink and 256-token BF16 window as SealKV? How much of the GPT-OSS gap remains after this control?
  4. Did the TurboQuant baseline reproduce the accuracy reported for that method? The 0.00 NIAH score on Qwen3-4B and the 56.30 mean on Qwen3-8B are much lower than the other 3-bit methods.
  5. What is the relationship between SealKV and the concurrently submitted CPTC paper? Are the authors the same, and which parts of the codec are new here? The paper cites CPTC as Anonymous 2026 and includes it as supplementary material.
  6. What is the actual KV memory footprint at 4K and 16K, including the BF16 sink and window? The 6.6 times figure is the asymptotic coded rate and does not include these regions.
7 comments
AuthorsOpenReview ↗

Rebuttal Reply to Reviewer Gjey [1/5]

We sincerely thank the reviewer for the review and address each point below.

W1. end-to-end serving evaluation including prefill profiling, and quality at 64K.

(1) We run RULER at 64K tokens with YaRN ×4, Qwen3-8B, five tasks.

MethodBitsFive-task meanvs. BF16vs. SealKV K4V4
BF161686.5
SealKV K4V44.4185.9−0.6
SAW-INT44.2584.3−2.2−1.6
TurboQuant K3V33.2510.9−75.6−75.0
OSCAR3.0073.1−13.3−12.7

Under the long-context benchmarks where we tested on RULER (NIAH multi-key/query/value, variable tracking, multi-doc QA), SealKV consistently achieves the best performance of all the baselines and stays very closely to the BF16 accuracy.

(2) We further profiled and recorded the e2e throughput win with prefill included on H200. The peak throughput vs. BF16:

MethodContext1K2K4K8K16K
SealKV K4V416K1.03×1.25×1.52×1.58×1.98×
64K1.11×1.28×1.58×1.91×2.22×
SealKV K4V216K1.03×1.24×1.60×1.72×2.15×
64K1.09×1.29×1.58×1.96×2.38×
SealKV K2V216K1.01×1.25×1.64×1.82×2.30×
64K1.08×1.30×1.63×2.05×2.53×
SAW-INT416K0.89×0.84×0.85×0.88×1.01×
64K1.02×1.00×0.99×1.00×1.15×

At every length, SAW-INT4 stays at 0.84 to 1.15×, so from 2K generated tokens on SealKV has the highest end-to-end throughput of the methods measured, up to 2.53x

For prefilling cost, we evaluate on Qwen3-8B with H200 and SealKV costs consistently 27% while TurboQuant incurs 43% overhead at 64K context length. Our fused prefill encoder writes the same codes as CPTC's kernel and is 2.26× faster at 4 bits with up to 2.3× on H100, shown in Sec. 3.2.

64K prompts, one H2001 prompt (s)8 prompts (s)32 prompts (s)Time vs. BF16
BF163.5628.4114.21.00×
SealKV4.5336.1144.21.27×

Under dynamic batching with about 20K-token prompts and 2K-token outputs, SealKV serves 1.25 to 1.36× BF16's throughput, answered in Q2.

SealKV pays a prefill premium and is ahead once generation passes about 1K tokens; our Qwen3-8B reasoning traces average > 10K output tokens. We have added these prefill and end-to-end results to the revision (Appendix C.1).

AuthorsOpenReview ↗

Rebuttal Reply to Reviewer Gjey [2/5]

W2. Matched sink+sliding window protection, and the TurboQuant baseline.

We thank the reviewer for pointing out the specs of TurboQuant and matched protection.

(1) We have evaluated new experiments with SAW-INT4, NVFP4 and TurboQuant the same protection as SealKV (a 128-token BF16 sink and a 256-token BF16 window) on Qwen3-4B/8B/14B and GPT-OSS. Full tables are in Q3.

We want to restate that SealKV keeps these regions based upon our observation. For the sink, in Fig.4 from the paper, the first 16 tokens are 1.5% of the cache but 38% of the attention-output error under 2-bit key quantization. Prior work on attention sinks studies where attention mass goes but not quantization errors; our Fig. 4 shows the same positions also dominate quantization error. The choice of sliding window holds each block until the block is complete, so each block is encoded once, in one batched launch, and BMS can use its mean. It also keeps the newest tokens exact.

We provide the results with the protection below on GPT-OSS-120B across different benchmarks.

ModelMethodBitsAIME-25GPQA-DNIAH 16KNIAH 32K
120BBF161687.576.89884
SAW-INT4 + protection4.5064.275.39263
NVFP4 + protection4.5083.374.28649
TQ-4bit + protection4.25/4.5071.775.08253
SealKV K4V44.5287.577.09682

Even with a controlled protection area, we can tell that SealKV still significantly outperforms other baselines on GPT-OSS-120B by large margin, with at least 4.2pt on AIME and 19pt on NIAH32K.

(2) The implementation of TurboQuant in paper, with detailed answers in Q4, has the codec matching vLLM's TurboQuant; its weak rows come from the number of quantized layers. We quantized every layer, the same as all other methods, whereas TurboQuant on vLLM by default keeps the first two and last two layers in BF16.

Qwen3MethodBitsLongBenchFive reasoning tasksNIAH-MK 32K
4BBF161652.3375.0495
TurboQuant, all layers3.2513.2620.270
TurboQuant, as shipped in vLLM4.6148.9069.6994
SealKV K4V23.4151.7774.4294
8BBF161653.5676.2297
TurboQuant, all layers3.2530.2162.5629
TurboQuant, as shipped in vLLM4.6151.0072.1589
SealKV K4V23.4153.4275.9995
14BBF161656.1378.4297
TurboQuant, all layers3.2553.7873.1993
TurboQuant, as shipped in vLLM4.4754.4675.4799
SealKV K4V23.4156.3978.7598

The five reasoning tasks are GPQA-D, HumanEval, AIME-25, AIME-26, MATH-500.

At 1.06 to 1.20 more bits per value, TurboQuant trails SealKV K4V2 by 1.9 to 2.9 points on LongBench and 3.3 to 4.7 on reasoning.

Q1. Prefill latency, TTFT, and long-prompt / short-generation throughput.

TTFT for one prompt, which measure the prefill latency in secs on Qwen3-8B and H200:

PromptBF16SealKVTime vs. BF16
32K1.261.641.30×
64K3.564.531.27×
128K10.3812.921.24×
64K prompts, one H2001 prompt (s)8 prompts (s)32 prompts (s)Time vs. BF16
BF163.5628.4114.21.00×
SealKV4.5336.1144.21.27×

The prefilling cost 27% overhead while TurboQuant incurs 43% overhead at 64K context length. Our fused prefill encoder writes the same codes as CPTC's kernel and is 2.26× faster at 4 bits with up to 2.3× on H100, shown in Sec. 3.2.

We further profiled and recorded the e2e throughput win with prefill included on H200. The peak throughput vs. BF16 and also evaluated SAW-INT4 baseline:

MethodContext1K2K4K8K16K
SealKV K4V416K1.03×1.25×1.52×1.58×1.98×
64K1.11×1.28×1.58×1.91×2.22×
SealKV K4V216K1.03×1.24×1.60×1.72×2.15×
64K1.09×1.29×1.58×1.96×2.38×
SealKV K2V216K1.01×1.25×1.64×1.82×2.30×
64K1.08×1.30×1.63×2.05×2.53×
SAW-INT416K0.89×0.84×0.85×0.88×1.01×
64K1.02×1.00×0.99×1.00×1.15×

At every length, SAW-INT4 stays at 0.84 to 1.15×, so from 2K generated tokens on SealKV has the highest end-to-end throughput of the methods measured, up to 2.53x

For long prompts with relatively short answers, SealKV can still bring higher throughput even if sealing adds about ~20 µs per prompt token. In 1K generated tokens: at 1K SealKV is already 1.02–1.11× faster, and its advantage grows to 1.58–2.51× from 8K to 16K generated tokens.

AuthorsOpenReview ↗

Rebuttal Reply to Reviewer Gjey [3/5]

Q2. 64K-Long-Context Performance.

We further evaluated RULER at 64K tokens on Qwen3-8B and YaRN factor 4 with original length 32K, we have evaluated on the following tasks. RULER tasks are significantly longer than LongBench and we report the new results below.

RULER task (64K)BF16SealKV K4V4SealKV K4V2SealKV K2V2
NIAH multi-key77.078.078.078.0
NIAH multi-query98.898.598.598.2
NIAH multi-value92.592.893.093.0
Variable tracking87.885.085.483.8
SQuAD QA76.375.072.475.1
Mean (five tasks)86.585.985.585.6
MethodBitsFive-task meanvs. BF16vs. SealKV K4V4
BF161686.5
SealKV K4V44.4185.9−0.6
SAW-INT44.2584.3−2.2−1.6
TurboQuant K3V33.2510.9−75.6−75.0
OSCAR3.0073.1−13.3−12.7

Under the long-context benchmarks where we tested on RULER (NIAH multi-key/query/value, variable tracking, multi-doc QA), SealKV consistently achieves the best performance of all the baselines and stays very close to the BF16 accuracy, only 0.6pt difference.

Q3 (1/2). Baselines with SealKV's protection: Qwen3.

We run new experiments by keeping SAW-INT4 and TurboQuant a 128-token BF16 sink and a 256-token BF16 window, the same as SealKV. OSCAR keeps its own sink and window by default, so it runs as published.

LongBench six-task mean (Table 8; * marks results from the paper):

MethodBits4B8B14B
BF16*1652.3353.5656.13
SAW-INT4 + protection4.2551.4453.2855.77
TurboQuant K3V3 + protection3.2516.4732.3755.24
OSCAR*3.0043.51
SealKV K4V2*3.4151.7753.4256.39
SealKV K4V4*4.4152.3753.8556.02

Reasoning, mean of AIME-26, GPQA-D and MATH-500:

MethodBits4B8B14B
BF16*1670.6273.2776.02
SAW-INT4 + protection4.2569.7673.8575.61
TurboQuant K3V3 + protection3.2523.1368.9275.39
OSCAR*3.0071.38
SealKV K4V2*3.4170.4974.2175.60
SealKV K4V4*4.4170.4672.9476.39

KL on WikiText-2 for Qwen3-8B (×10⁻³):

MethodBitsKL ↓
SAW-INT4 + protection4.253.29
NVFP4 + protection4.505.77
TurboQuant 4-bit + protection4.1911.98
OSCAR3.0038.47
SealKV K4V44.381.52

With the same protection, SealKV stays within 0.6 points of BF16 on LongBench and within 1 point on reasoning at every model size, and K4V2 matches or beats protected SAW-INT4 with 0.84 fewer bits per value. Its KL is 2.2 to 7.9× lower than every protected baseline and 25× lower than OSCAR's, while TurboQuant collapses at 4B and 8B and OSCAR trails K4V2 by 9.9 points on LongBench.

Q3 (2/2). Baselines with SealKV's protection: GPT-OSS.

Same as Q3(½), here we add 128 sink tokens and 256 sliding window size to each baseline and evaluated on GPT-OSS-120B:

We provide the results with the protection below on GPT-OSS-120B across different benchmarks.

ModelMethodBitsAIME-25GPQA-DNIAH 16KNIAH 32K
120BBF161687.576.89884
SAW-INT4 + protection4.5064.275.39263
NVFP4 + protection4.5083.374.28649
TQ-4bit + protection4.25/4.5071.775.08253
SealKV K4V44.5287.577.09682

Even with a controlled protection area, we can tell that SealKV still significantly outperforms other baselines on GPT-OSS-120B by large margin, with at least 4.2pt on AIME and 19pt on NIAH32K.

Reviewer_ktKcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents SealKV, a low-bit KV cache compression system that combines trellis-coded quantization, block-mean subtraction (BMS), and fused attention kernels. It achieves competitive accuracy at 2–4 bits and reports up to 3.44× higher peak decoding throughput than BF16 on an H100.

Critical strengths
  1. Strong empirical performance. SealKV achieves competitive accuracy at low bit widths, with up to 6.6× KV cache compression and substantial end-to-end throughput improvements.

  2. Comprehensive evaluation. The paper includes multiple model families, reasoning and retrieval benchmarks, agentic workloads, kernel profiling, and implementation details.

Critical weaknesses
  1. Limited algorithmic novelty. The main components closely resemble existing techniques. Delayed block-wise KV quantization has been explored by KIVI and related methods. The trellis quantizer is inherited from the concurrently submitted CPTC work, with the main adaptation being a shorter trellis and optimized implementation. Moreover, BMS closely resembles the mean subtraction technique in SageAttention. Although SealKV extends it to block-wise persistent KV caching, the paper does not sufficiently establish the novelty of this extension.
Questions for authors
  1. How does BMS fundamentally differ from SageAttention's key smoothing beyond applying mean subtraction to completed KV blocks? Can the authors provide a direct comparison?

  2. What are the independent algorithmic contributions of SealKV beyond adapting CPTC and combining existing KV compression techniques?

  3. If the authors can clearly articulate and substantiate the novel contributions beyond these existing techniques, I would be willing to raise my rating.

3 comments
AuthorsOpenReview ↗

Rebuttal Reply to Reviewer ktKc [1/2]

We sincerely thank the reviewer for the review and we address every point below.

W1. Novelty relative to KIVI, CPTC and SageAttention.

We thank the reviewer for asking about novelty.

First, we need to clarify that SealKV is quantizer-agnostic, and CPTC is the quantization format we plugged in, and SealKV is what makes a trellis codebook usable on a growing KV cache. The CPTC paper mostly evaluates weight quantization and does not apply CPTC to KV caches at all, so it has no KV-cache accuracy, attention kernel or serving path. SealKV adds block sealing, block-mean subtraction (BMS), an RHT with per-token scaling, and fused encoding and attention kernels, and together these give near-BF16 accuracy at 3.41 bits and up to 3.44× BF16's peak serving throughput. The short trellis is a systems choice as well. At L=10 the codebook fits in shared memory, while at L=16, the length used in the CPTC paper, encoding is 41 to 93× slower. To check that the gains do not come from CPTC itself, we replaced it with QTIP's 1MAD codebook at the same L=10 and kept the rest of SealKV unchanged. We further run results on Qwen3-8B:

KV formatMethodBits/valueKLD ×10⁻³ ↓AIME-26 ↑GPQA-D ↑
BF1616065.0060.35
K4V4SealKV (CPTC)4.411.62765.0060.23
SealKV + QTIP-1MAD4.411.75365.8362.37
QTIP-1MAD alone4.12539.87--
CPTC alone4.12533.71--
K4V2SealKV (CPTC)3.414.54967.5060.73
SealKV + QTIP-1MAD3.415.20065.8359.34
QTIP-1MAD alone3.12566.95--
CPTC alone3.12557.53--

Either codebook on its own breaks the model, with KL 12.6 to 22.7× higher. Inside SealKV both match BF16 within noise on AIME-26 and GPQA-D, and CPTC keeps only a small KL edge, so the accuracy comes from SealKV whichever trellis codebook it uses.

Second, compared to KIVI, both methods wait for a group of tokens before quantizing, but for different reasons, and they do different things with the group. KIVI quantizes keys per channel on a scalar grid and folds the channel offset into each group's zero point. SealKV seals a block so that a trellis code can be encoded once per block in a batched launch, and so that the block's shared offset can be measured once, stored as a vector and carried through attention in closed form (Sec. 3.3). SealKV is clearly more accurate at fewer bits. Change from BF16 in points, from Tables 7 and 8:

BenchmarkQwen3KIVI-2, 3.00 bitsSealKV K2V2, 2.41 bitsSealKV K4V2, 3.41 bits
Six-task mean4B−10.77−3.81−0.02
8B−3.62−2.35−0.19
14B−1.78−0.37+0.44
LongBench4B−5.42−2.27−0.56
8B−4.29−1.78−0.14
14B−2.93−1.11+0.26

With 0.59 fewer bits, K2V2 loses 1.3 to 7.0 fewer points than KIVI-2 on the six tasks and 1.8 to 3.2 fewer on LongBench, and K4V2 stays within 0.6 points of BF16 everywhere.

Third, SageAttention is not a KV-cache method. It subtracts one key mean over the whole sequence inside a single attention call, which needs the full sequence and so is not available while a cache grows. BMS subtracts per-block means from both K and V, stores them, and adds one term per block inside attention. The block version matters because K and V offsets are local. On Qwen3-8B, 16-token block means remove 71% of the key energy and a sequence-wide mean only 48%. In the direct comparison in Q1, with the same codec, BMS has 23% lower KL than an oracle SageAttention-style mean at 4 bits and 43% lower at 2 bits. A causal running mean, which could be used online, does about as well as the oracle, so the gain comes from block locality.

AuthorsOpenReview ↗

Rebuttal Reply to Reviewer ktKc [2/2]

Q1. BMS versus SageAttention's key smoothing, compared directly.

BMS differs from SageAttention's key smoothing in three ways. (1) SageAttention subtracts one key mean over the whole sequence, but K and V offsets are local. On Qwen3-8B, 16-token block means remove 71% of the key energy and 26% of the value energy, while one sequence-wide mean removes only 48% and 9.5%. (2) A sequence-wide mean needs every token, so it can't be computed while the cache grows, whereas each block mean is fixed once its block is sealed and stored with the codes. (3) BMS works on values as well as keys, inside attention. The key mean enters as one product per block and the value mean as one rank-1 update per block (Eqs. 6 and 7), so no per-token work is added.

The table compares them by KL from BF16 on WikiText-2 (×10⁻³). Every row uses the same CPTC codec, RHT, sink and window, and keeps means in BF16 as in our ablations. The first and last rows are from the paper and the middle three are new.

Mean subtractionUsable onlineBits, 4-bitKL, 4-bitBits, 2-bitKL, 2-bit
Noneyes4.1253.6302.1251724.8
One sequence-wide K mean (SageAttention-style, oracle)no4.1271.9782.12733.64
Sequence-wide K and V means (oracle)no4.1291.9462.12933.46
Causal running K and V means, stored per blockyes5.1251.8803.12532.79
BMS, per-block K and V means (SealKV)yes5.1251.5323.12519.17

BMS has 23% lower KL than the SageAttention-style mean at 4 bits and 43% lower at 2 bits, even though that mean is an oracle that sees future tokens. The causal running mean spends the same bits as BMS and still lands next to the oracle, so the gain comes from block locality rather than from storing a mean. In a paired test over the 32 text chunks, BMS beats every alternative at 4 bits, with t from 3.0 to 6.6 and lower KL on 28 to 31 chunks, and at 2 bits even BMS on keys alone wins all 32. With the served 4-bit mean records it costs 0.28 bits/value at nearly the same KL (Appendix E.2 and E.3). BMS is not tied to our codec either. Added to SAW-INT4, TurboQuant and NVFP4, it improves all 30 GPT-OSS model, quantizer and metric combinations, by up to 32.2 points (Table 5).

Q2. Independent contributions beyond adapting CPTC and combining existing techniques.

SealKV is an algorithm and system co-design and is quantizer-agnostic, and W1 shows its accuracy does not come from CPTC itself. With QTIP's 1MAD codebook at the same L, SealKV still matches BF16 within noise, while either codebook used alone breaks the model. Three contributions hold regardless of the codebook.

First, block sealing makes trellis coding usable on a growing cache. To our knowledge no prior KV-cache method uses trellis coding, and the CPTC paper does not apply CPTC to KV caches at all. Sealing complete blocks lets encoding run once per block, in batched launches that never wait on the newest token. It also sets the trellis length. At L = 10 the 2 KiB codebook fits per lane in shared memory, while at L = 16, the length used in the CPTC paper, encoding is 41 to 93× slower and codebook lookups come from global memory, 5.4× slower on H200. L = 10 costs only 0.14 effective bits (Table 1).

Second, BMS comes from a new observation. K and V carry per-channel offsets shared by neighbouring tokens, which rotation cannot remove (Fig. 2, Sec. 2.2). Removing them per sealed block cuts 2-bit KL 90× (Appendix E.2) and keeps 2-bit SealKV from collapsing, with AIME-26 at 40.83 without BMS against 67.92 with it. BMS also beats SageAttention-style mean subtraction and lifts three other quantizers on GPT-OSS (Q1).

Third, the kernels are built around this format. The fused prefill encoder is up to 2.3× faster than CPTC's and writes bit-identical codes. The fused attention kernel decodes in registers from a conflict-free per-lane codebook and folds BMS into one term per block, without writing back a dequantized cache. Attention is 2.5 to 2.9× faster than BF16 FlashAttention-3 from 16K to 1M context (Fig. 8a), and peak decode throughput in vLLM reaches 2.81 to 3.44× BF16 at 64K, against at most 2.00× for any baseline (Table 14).

No evaluated method matches this tradeoff among quality, speed and memory. SealKV stays within 0.7 points of BF16 at 3.41 and 4.41 bits at every Qwen3 size (Table 7), is the most accurate method at 2.41 bits, ahead of KIVI-2 and OSCAR at 3.00 (Tables 7 and 8), and has the highest serving throughput (Table 14).

Q3. We hope this makes the contributions concrete. In this rebuttal and revised paper we have clarified and added:

  • the codebook swap with QTIP-1MAD (W1) and the direct comparison with SageAttention-style mean subtraction (Q1);
  • ablations showing accuracy collapses without BMS or RHT (reply to 9QPU Q1);
  • baselines with SealKV's protection on Qwen3 and GPT-OSS, and accuracy at 64K.

We will be glad to clarify anything further.

Dear Reviewer ktKc,

Thank you again for your review. We posted our responses to every point and a revised PDF. We would sincerely appreciate that, when you have a moment, let us know whether they address your concerns. We will be happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all three reviewers for their reviews and have addressed every raised point. The table lists the main points reviewers raised and what and where we added in the revised PDF to address each of them (new text in blue).

Point raisedReviewersWhat we addedRevised PDF
Novelty vs. CPTC; does accuracy come from CPTC?9QPU, Gjey, ktKcWe want to show that SealKV is quantizer-agnostic so swapped CPTC for QTIP's 1MAD codebook inside SealKV and changed nothing else. Both match BF16 within noise on AIME-26 and GPQA-DSec. 4.6, Table 6
Controlled experiments with longer context and the same sink and window for baselinesGjeyOn RULER at 64K with YaRN ×4, SealKV K4V4 is 0.6 points below BF16. With SealKV's sink and window given to SAW-INT4, NVFP4 and TurboQuant on Qwen3 and GPT-OSS-120B, SealKV keeps the lowest KL and matches or beats them on accuracy. TurboQuant as shipped in vLLM keeps four layers in BF16 at 4.47 to 4.61 bits and still trails SealKV K4V2 at 3.41 bits on LongBench and reasoningSec. 4.2, Tables 3 and 4; Appendix A.2, Tables 9 and 10; Appendix A.3, Table 11
Efficiency, including end-to-end cost with prefill, dynamic batching and other configurationsGjey, 9QPUWith prefill counted, SealKV is 1.01 to 1.11× BF16's throughput at 1K generated tokens and up to 2.53× at 16K, and prefill takes 1.27× BF16's time at 64K. Counting the BF16 sink and window, K2V2 still saves 4.35× memory at 4K and 6.44× at 64K. On H200, SealKV reaches 1.64 to 3.01× BF16 peak decode on Qwen3-4B, 8B and 14B, and serves up to 1.66× the requests under continuous batchingAppendix C.1, Tables 17 and 18; Appendix H.3, Table 23; replies to 9QPU W2 and Q2
BMS effects, with BMS-enhanced baselines and SageAttention's key smoothing9QPU, ktKcWith BMS and the sink and window added to every baseline, SealKV still has the lowest KL, and removing BMS from SealKV costs 27 AIME-26 points at 2 bits. BMS also has 23% lower KL at 4 bits and 43% at 2 bits than SageAttention-style key smoothing, even with an oracle sequence-wide meanAppendix E.4, Table 21; replies to 9QPU W1 and Q1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.