- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that KV cache compression only becomes a serving gain when the runtime treats compressed KV as a first class representation. Sparse attention reduces how much KV each query reads, but the full cache must stay resident, since future queries may select any block. Sparsity lowers the read cost, not the memory that limits how many requests run at once. Existing compression work targets compact storage or faster kernels, and does not say when to compress, how attention runs over compressed states, or how compressed KV is managed next to native KV.
Duet is a serving native, low rank compression framework built on vLLM, using Quest as its sparse selector. It has three components: 1) a joint head latent compresses keys after RoPE so the projection folds into one query side term, and block summaries are still computed from the original keys, so the selector behaves as before. 2) a mixed attention operator reads native and compressed blocks where they are and merges the two partial results exactly. 3) a dual representation runtime makes the compressed slot the canonical block identity, batches compression at a watermark, and uses separate execution paths for decode and for prefill. Across Qwen3-4B, Llama-3.1-8B, and Mistral-Nemo-12B, Duet cuts KV memory 4x, raises serving throughput up to 1.8x, and cuts time to first token up to 16.8x, with about one point of average loss on LongBench.
Critical strengths
-
The contribution is systems integration rather than a new compression algorithm. Duet makes compressed KV a first-class representation in a real serving runtime: the compressed slot is the canonical block identity, leaving paged allocation, prefix caching, and CUDA graph replay untouched, while native KV acts as a cache over those blocks. Watermark-triggered batched compression and phase-specialized execution complete the lifecycle. The paper also evaluates on the metrics this design implies, reporting serving throughput, time to first token, queue length, and prefix-cache hit rate across offline, single-turn, and multi-turn settings, rather than only a compression-ratio-versus-accuracy curve. This end-to-end treatment is what converts a 4x footprint reduction into measured serving gains, and it distinguishes the work from representation-level KV compression.
-
Decoupling sparse selection from KV compression is a clean, well-motivated design. Because block summaries are always computed from the original keys, the selector behaves exactly as in the uncompressed system, and compression only approximates the contents of already-selected blocks. This directly explains the accuracy profile, since retrieval tasks barely move and single-needle retrieval on RULER is essentially lossless. The paper makes the design decision concrete by measuring the problem it avoids: summaries built from reconstructed keys would change up to 60 percent of selected blocks. The evaluation is also candid about costs, reporting that compressed attention is slightly slower than sparse attention alone at the kernel level and that inter-token latency rises about 7 percent under the larger batches the extra capacity enables.
Critical weaknesses
Weakness 1: The case for low-rank compression is thin, and the paper's own measurements raise doubts
-
The paper motivates low-rank latent compression by absorbing the projection into the query side, which makes attention over the latent cheaper. That saving only matters when attention touches many tokens, and sparse attention here already limits each query to about 2048. The two sources of speedup therefore work against each other. This is what Fig. 2b shows: sparse attention with compression is slightly slower than raw sparse attention. Given how much machinery the design adds, I would expect a clearer speed benefit from low rank itself.
-
The gains that do appear come mainly from capacity rather than from low rank. Compression fits more blocks in memory, and the prefix-cache hit rate rises from 6.9% to 63.8%, which lets larger batches run. Quantization would free the same capacity, and I could not find a comparison showing that low rank is needed here. Once capacity pressure is removed in the offline setting, Duet is only 1.15–1.3× faster than Quest.
Weakness 2: The evaluation compares only against uncompressed baselines, and some headline claims are hard to verify
-
The baselines are dense attention and Quest, so the experiments mainly show that compression helps relative to no compression. Quantization is the most direct alternative for capacity, and it is not compared. Palu and ShadowKV are discussed as related work but not measured.
-
The abstract reports less than 1% average accuracy loss, but the appendix shows gaps of 1.03 on Llama LongBench and 2.02 on Mistral RULER. I also could not trace the 16.8× TTFT reduction to any table, and throughput is quoted as 1.8× in the abstract and 1.9× in the body. The evaluation setting is relatively modest, with 4–12B models, context capped at 32K, and all runs on a single GPU without tensor parallelism.
Questions for authors
Question 1: Fig. 2b shows that sparse attention with compression is slightly slower than raw sparse attention, where DSA already limits each query to about 2048 tokens. What speed benefit does low rank itself provide once the capacity effect is set aside?
Question 2: The gains look capacity-driven: the prefix-cache hit rate rises from 6.9% to 63.8%, and quantization would free the same capacity. Since the only baselines are dense attention and Quest, how can we tell whether low rank, rather than extra capacity alone, drives the gain?
Question 3: The offline gain over Quest is only 1.15–1.3×, well below the online gain. How much of the online gain comes from the compression itself, rather than from serving effects such as prefix caching and larger batches?
Question 4: The abstract claims less than 1% accuracy loss, but the appendix reports gaps of 1.03 points on Llama LongBench and 2.02 on Mistral RULER.
Question 5: The 16.8× TTFT reduction appears in no table, and throughput is 1.8× in the abstract but 1.9× in the body. Where are these numbers measured?
Question 6: All runs cap context at 32K on a single GPU. Do the gains hold at longer contexts and under multi-GPU parallelism like TP and CP?
5 comments
W1 (2nd bullet) & W2 (1st bullet): Compared with quantization and compression baselines
We clarify that quantization and low-rank compression are complementary rather than mutually exclusive: they can be combined to achieve better compression–accuracy trade-offs under the same memory budget, as demonstrated in Table 10. To further characterize their trade-offs, we measure the Frobenius reconstruction error across different compression ratios. The results show two regimes: quantization performs better at moderate ratios (up to approximately 5×), while low-rank compression remains more viable at higher ratios. To ensure a strong quantization baseline, we evaluate both per-channel and per-token grouping and report the lower-error configuration for each tensor (following the KIVI-style setup).
| Compression | Scheme | Key error | Value error |
|---|---|---|---|
| 1.9x | Quant. only, INT8 | 0.0050 | 0.0069 |
| 3.8x | Quant. only, INT4 | 0.0844 | 0.1175 |
| 4.0x | Duet, rank 256 | 0.2691 | 0.3727 |
| 4.9x | Quant. only, INT3 | 0.1809 | 0.2521 |
| 7.1x | Quant. only, INT2 | 0.4271 | 0.5913 |
| 7.8x | Duet, rank 256 + INT8 | 0.2697 | 0.3729 |
| 10.2x | Duet, rank 256 + K4V8 | 0.3889 | 0.3729 |
| 15.5x | Duet, rank 128 + INT8 | 0.3334 | 0.5006 |
The results confirm that INT4 is more accurate than rank-256 Duet at similar ratios, but quantization degrades sharply below 3 bits: INT2 reaches only 7.1×, whereas rank-256 Duet + INT8 achieves 7.8× with approximately 1.6× lower error on both tensors. Moreover, our 10.2× Duet + K4V8 configuration would require approximately 1.3 bits per element from quantization alone. Combining the two techniques also incurs slight task-performance degradation: adding K4V8 to rank-256 Duet (D/4) changes the three LongBench averages by only -0.54, +0.01, and -0.33 points (Table 10). Thus, our claim is composability and extended compression reach, not an already optimal joint scheme; the reported combinations are untuned configurations with uniform global bit widths, and an interesting future direction is to jointly optimize low-rank decomposition and quantization. For example, by adaptively allocating quantization bits to different low-rank components based on their singular values and relative importance, where our systematic solution is still required.
Neither Palu nor ShadowKV is a directly comparable in-GPU capacity baseline for selector-preserving online sparse serving:
- Palu: Focuses on low-rank KV compression, but does not provide an integrated sparse-serving runtime with external block selection, mixed native/compressed execution, and online cache lifecycle management.
- ShadowKV: Uses hierarchical offloading, keeping compressed keys and landmarks on GPU while offloading values to CPU DRAM. It relies on its own landmark-based selection pipeline rather than preserving an external sparse selector, and reduces GPU residency through offloading rather than in-GPU compression of the full KV cache.
W2 (2nd bullet), Q4 & Q5: Headline accuracy, TTFT, and throughput
On Q4, the "1%" in the abstract refers to the average absolute score-point gap across three models, not relative percentage loss. At 1/4 compression ratio, the average gaps to Quest are 0.92 points on LongBench and 0.91 on RULER (Tables 2 and 9). We acknowledge that individual gaps reach 1.03 and 2.02 points, respectively, and will revise the abstract to clarify this.
On Q5, all numbers are measured on Qwen3-4B in the multi-turn setting (Figure 7). Token throughput counts both prompt and generated tokens, including prefix-cache hits, divided by the full run duration. At 0.4 req/s, throughput is 6,350 vs. 11,710 tokens/s for Quest and Duet (1.84×). The 1.8× vs. 1.9× discrepancy is due to inconsistent rounding of the same metric, not different counting conventions. The 16.8× TTFT reduction is measured using mean TTFT across all turns, from 47.94 s to 2.85 s. We will clarify the metric definition and revise this throughout the paper.
Q6: Longer contexts and multi-GPU scaling
Regarding scalability beyond our current evaluation scope, we discuss the following considerations:
- Longer contexts: The 4× KV payload reduction yields increasing absolute memory savings as context length grows. Meanwhile, the fixed sink and local-window regions account for a decreasing fraction of the cache, while the sparse attention budget remains fixed at 2,048 tokens.
- Tensor parallelism and data-parallel attention: Since the joint-head latent is shared across all KV heads in a layer, it cannot be naively sharded by KV head across TP ranks. Similar to MLA's shared-latent design, data-parallel attention provides a natural alternative. Duet could follow the same strategy.
- Context parallelism: The associative log-sum-exp merging in Equation 4 naturally supports ring- or tree-based reductions across token shards. Coordinating compression watermarks and native KV pools across shards remains future work.
I thank the authors for the additional experiments and the detailed discussion. They are helpful for better understanding the contribution. However, the discussion further deepened my concerns about the method's effectiveness.
The response confirms that low rank itself brings no attention speedup, and the serving gains come from the HBM capacity. My concern is that low rank does not appear to be the effective tool for this purpose. The authors' own comparison shows that at the paper's default ratio of about 4x, INT4 quantization reaches a lower reconstruction error than Duet at rank 256. And combined with INT8 is still much worse than INT4 alone, which means the accuracy loss mainly comes from low-rank.
Quantization seems to be a better solution, with lower reconstruction error. Additionally, quantization also speeds up the attention, since it reduces the bytes read per attention step during memory-bandwidth bound decoding. Quantization is also a more mature and widely deployed technique, and recent research such as DeepSeek-V4.1-Flash, NVFP4 KV cache, and BitDecoding-FP4 (Hunyuan) has already shown that aggressive KV quantization works well in serving.
Given this, I am not convinced that low rank is the right mechanism for the setting the paper targets, and I am concerned about where exactly the proposed scheme applies and how much it adds over the simpler alternative.
On the method's effectiveness I therefore have to change my rating to 2. I appreciate the authors' candor throughout the discussion, and recognize the contribution on the systems work.
We sincerely thank the reviewer for recognizing our end-to-end systems design and selector-decoupling mechanism, and provide detailed responses to your comments below.
W1 (1st bullet), Q1, Q2 & Q3: What low rank provides
We agree with your reading that Duet's serving gains come from KV capacity rather than faster attention kernels. That is our stated position in Section 2, where we frame compression under a dynamic sparse selector as a capacity mechanism rather than a kernel speedup, and Figure 2(b) deliberately reports the slight kernel-level cost of the compressed path.
Low rank and extra capacity are therefore not rival explanations: extra capacity is what low rank produces by storing fewer bytes per token. Once a sparse selector has removed most of the attention compute, HBM capacity is the remaining lever on throughput, and a mechanism that frees memory is what supplies it.
Accordingly, once capacity is set aside, low rank itself provides no kernel speedup, and we do not claim one. What our operator co-design provides is that compression does not cost speed; this is not automatic, as the obvious implementation would erase the capacity gain on the attention path:
Prefix retention and larger active batches are likewise the mechanisms by which smaller blocks become throughput, not confounders to be subtracted out. Offline runs isolate concurrency alone, with all requests submitted at once, no arrival queue, and no shared multi-turn prefixes, which is why the gain there is 1.15-1.3x over Quest (Figure 5). Under continuous multi-turn arrivals, the same 4x smaller KV payload additionally prevents prefix eviction and cuts admission queueing, raising the prefix-cache hit rate from 6.9% to 63.8% and dropping average waiting requests from 33.3 to 3.5 (Table 5).