acceptodds
Under review as a conference paper at ICLR 2027

From Memory Savings to Throughput Gains: Serving-Native KV Compression for Sparse-Attention LLMs

Abstract

Long-context LLM serving is increasingly bottlenecked by the growing memory and bandwidth demands of the KV cache. Sparse attention alleviates the KV access cost, but still leaves the full KV cache resident, limiting serving concurrency. Existing KV compression methods primarily target compact storage or optimized kernels, rather than the execution and lifecycle of compressed KV in online serving, where compression timing, efficient attention over compressed states, and cache management directly affect performance. We present Duet, a serving-native KV compression framework that turns KV memory savings into practical serving gains while preserving the underlying sparse-attention selection semantics. Duet first develops an inference-friendly low-rank representation that exploits cross-head redundancy and removes per-token reconstruction from sparse attention. Building on this, a mixed native-latent attention operator directly consumes selected KV blocks in either form without materializing compressed history. Finally, Duet integrates compression into the serving runtime through dual-representation KV management, watermark-triggered batched compression, and phase-specialized execution. Together, these designs reduce KV memory usage by while preserving efficient attention execution, delivering up to higher throughput and a reduction in TTFT, with less than 1% average accuracy loss.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
sparse attention, LLM serving, kv cache compression

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ZeD5OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that KV cache compression only becomes a serving gain when the runtime treats compressed KV as a first class representation. Sparse attention reduces how much KV each query reads, but the full cache must stay resident, since future queries may select any block. Sparsity lowers the read cost, not the memory that limits how many requests run at once. Existing compression work targets compact storage or faster kernels, and does not say when to compress, how attention runs over compressed states, or how compressed KV is managed next to native KV.

Duet is a serving native, low rank compression framework built on vLLM, using Quest as its sparse selector. It has three components: 1) a joint head latent compresses keys after RoPE so the projection folds into one query side term, and block summaries are still computed from the original keys, so the selector behaves as before. 2) a mixed attention operator reads native and compressed blocks where they are and merges the two partial results exactly. 3) a dual representation runtime makes the compressed slot the canonical block identity, batches compression at a watermark, and uses separate execution paths for decode and for prefill. Across Qwen3-4B, Llama-3.1-8B, and Mistral-Nemo-12B, Duet cuts KV memory 4x, raises serving throughput up to 1.8x, and cuts time to first token up to 16.8x, with about one point of average loss on LongBench.

Critical strengths
  • The contribution is systems integration rather than a new compression algorithm. Duet makes compressed KV a first-class representation in a real serving runtime: the compressed slot is the canonical block identity, leaving paged allocation, prefix caching, and CUDA graph replay untouched, while native KV acts as a cache over those blocks. Watermark-triggered batched compression and phase-specialized execution complete the lifecycle. The paper also evaluates on the metrics this design implies, reporting serving throughput, time to first token, queue length, and prefix-cache hit rate across offline, single-turn, and multi-turn settings, rather than only a compression-ratio-versus-accuracy curve. This end-to-end treatment is what converts a 4x footprint reduction into measured serving gains, and it distinguishes the work from representation-level KV compression.

  • Decoupling sparse selection from KV compression is a clean, well-motivated design. Because block summaries are always computed from the original keys, the selector behaves exactly as in the uncompressed system, and compression only approximates the contents of already-selected blocks. This directly explains the accuracy profile, since retrieval tasks barely move and single-needle retrieval on RULER is essentially lossless. The paper makes the design decision concrete by measuring the problem it avoids: summaries built from reconstructed keys would change up to 60 percent of selected blocks. The evaluation is also candid about costs, reporting that compressed attention is slightly slower than sparse attention alone at the kernel level and that inter-token latency rises about 7 percent under the larger batches the extra capacity enables.

Critical weaknesses

Weakness 1: The case for low-rank compression is thin, and the paper's own measurements raise doubts

  • The paper motivates low-rank latent compression by absorbing the projection into the query side, which makes attention over the latent cheaper. That saving only matters when attention touches many tokens, and sparse attention here already limits each query to about 2048. The two sources of speedup therefore work against each other. This is what Fig. 2b shows: sparse attention with compression is slightly slower than raw sparse attention. Given how much machinery the design adds, I would expect a clearer speed benefit from low rank itself.

  • The gains that do appear come mainly from capacity rather than from low rank. Compression fits more blocks in memory, and the prefix-cache hit rate rises from 6.9% to 63.8%, which lets larger batches run. Quantization would free the same capacity, and I could not find a comparison showing that low rank is needed here. Once capacity pressure is removed in the offline setting, Duet is only 1.15–1.3× faster than Quest.

Weakness 2: The evaluation compares only against uncompressed baselines, and some headline claims are hard to verify

  • The baselines are dense attention and Quest, so the experiments mainly show that compression helps relative to no compression. Quantization is the most direct alternative for capacity, and it is not compared. Palu and ShadowKV are discussed as related work but not measured.

  • The abstract reports less than 1% average accuracy loss, but the appendix shows gaps of 1.03 on Llama LongBench and 2.02 on Mistral RULER. I also could not trace the 16.8× TTFT reduction to any table, and throughput is quoted as 1.8× in the abstract and 1.9× in the body. The evaluation setting is relatively modest, with 4–12B models, context capped at 32K, and all runs on a single GPU without tensor parallelism.

Questions for authors

Question 1: Fig. 2b shows that sparse attention with compression is slightly slower than raw sparse attention, where DSA already limits each query to about 2048 tokens. What speed benefit does low rank itself provide once the capacity effect is set aside?

Question 2: The gains look capacity-driven: the prefix-cache hit rate rises from 6.9% to 63.8%, and quantization would free the same capacity. Since the only baselines are dense attention and Quest, how can we tell whether low rank, rather than extra capacity alone, drives the gain?

Question 3: The offline gain over Quest is only 1.15–1.3×, well below the online gain. How much of the online gain comes from the compression itself, rather than from serving effects such as prefix caching and larger batches?

Question 4: The abstract claims less than 1% accuracy loss, but the appendix reports gaps of 1.03 points on Llama LongBench and 2.02 on Mistral RULER.

Question 5: The 16.8× TTFT reduction appears in no table, and throughput is 1.8× in the abstract but 1.9× in the body. Where are these numbers measured?

Question 6: All runs cap context at 32K on a single GPU. Do the gains hold at longer contexts and under multi-GPU parallelism like TP and CP?

5 comments

We sincerely thank the reviewer for recognizing our end-to-end systems design and selector-decoupling mechanism, and provide detailed responses to your comments below.

W1 (1st bullet), Q1, Q2 & Q3: What low rank provides

We agree with your reading that Duet's serving gains come from KV capacity rather than faster attention kernels. That is our stated position in Section 2, where we frame compression under a dynamic sparse selector as a capacity mechanism rather than a kernel speedup, and Figure 2(b) deliberately reports the slight kernel-level cost of the compressed path.

Low rank and extra capacity are therefore not rival explanations: extra capacity is what low rank produces by storing fewer bytes per token. Once a sparse selector has removed most of the attention compute, HBM capacity is the remaining lever on throughput, and a mechanism that frees memory is what supplies it.

Accordingly, once capacity is set aside, low rank itself provides no kernel speedup, and we do not claim one. What our operator co-design provides is that compression does not cost speed; this is not automatic, as the obvious implementation would erase the capacity gain on the attention path:

Kernel-Level Comparison (Qwen3-4B)BaselineDuetSpeedup
Mixed operator vs. materialize-then-attend0.393 ms0.239 ms1.65x
Post-RoPE absorption vs. pre-RoPE reconstruction0.0708 ms0.0239 ms3x

Prefix retention and larger active batches are likewise the mechanisms by which smaller blocks become throughput, not confounders to be subtracted out. Offline runs isolate concurrency alone, with all requests submitted at once, no arrival queue, and no shared multi-turn prefixes, which is why the gain there is 1.15-1.3x over Quest (Figure 5). Under continuous multi-turn arrivals, the same 4x smaller KV payload additionally prevents prefix eviction and cuts admission queueing, raising the prefix-cache hit rate from 6.9% to 63.8% and dropping average waiting requests from 33.3 to 3.5 (Table 5).

W1 (2nd bullet) & W2 (1st bullet): Compared with quantization and compression baselines

We clarify that quantization and low-rank compression are complementary rather than mutually exclusive: they can be combined to achieve better compression–accuracy trade-offs under the same memory budget, as demonstrated in Table 10. To further characterize their trade-offs, we measure the Frobenius reconstruction error across different compression ratios. The results show two regimes: quantization performs better at moderate ratios (up to approximately 5×), while low-rank compression remains more viable at higher ratios. To ensure a strong quantization baseline, we evaluate both per-channel and per-token grouping and report the lower-error configuration for each tensor (following the KIVI-style setup).

CompressionSchemeKey errorValue error
1.9xQuant. only, INT80.00500.0069
3.8xQuant. only, INT40.08440.1175
4.0xDuet, rank 2560.26910.3727
4.9xQuant. only, INT30.18090.2521
7.1xQuant. only, INT20.42710.5913
7.8xDuet, rank 256 + INT80.26970.3729
10.2xDuet, rank 256 + K4V80.38890.3729
15.5xDuet, rank 128 + INT80.33340.5006

The results confirm that INT4 is more accurate than rank-256 Duet at similar ratios, but quantization degrades sharply below 3 bits: INT2 reaches only 7.1×, whereas rank-256 Duet + INT8 achieves 7.8× with approximately 1.6× lower error on both tensors. Moreover, our 10.2× Duet + K4V8 configuration would require approximately 1.3 bits per element from quantization alone. Combining the two techniques also incurs slight task-performance degradation: adding K4V8 to rank-256 Duet (D/4) changes the three LongBench averages by only -0.54, +0.01, and -0.33 points (Table 10). Thus, our claim is composability and extended compression reach, not an already optimal joint scheme; the reported combinations are untuned configurations with uniform global bit widths, and an interesting future direction is to jointly optimize low-rank decomposition and quantization. For example, by adaptively allocating quantization bits to different low-rank components based on their singular values and relative importance, where our systematic solution is still required.

Neither Palu nor ShadowKV is a directly comparable in-GPU capacity baseline for selector-preserving online sparse serving:

  • Palu: Focuses on low-rank KV compression, but does not provide an integrated sparse-serving runtime with external block selection, mixed native/compressed execution, and online cache lifecycle management.
  • ShadowKV: Uses hierarchical offloading, keeping compressed keys and landmarks on GPU while offloading values to CPU DRAM. It relies on its own landmark-based selection pipeline rather than preserving an external sparse selector, and reduces GPU residency through offloading rather than in-GPU compression of the full KV cache.

W2 (2nd bullet), Q4 & Q5: Headline accuracy, TTFT, and throughput

On Q4, the "1%" in the abstract refers to the average absolute score-point gap across three models, not relative percentage loss. At 1/4 compression ratio, the average gaps to Quest are 0.92 points on LongBench and 0.91 on RULER (Tables 2 and 9). We acknowledge that individual gaps reach 1.03 and 2.02 points, respectively, and will revise the abstract to clarify this.

On Q5, all numbers are measured on Qwen3-4B in the multi-turn setting (Figure 7). Token throughput counts both prompt and generated tokens, including prefix-cache hits, divided by the full run duration. At 0.4 req/s, throughput is 6,350 vs. 11,710 tokens/s for Quest and Duet (1.84×). The 1.8× vs. 1.9× discrepancy is due to inconsistent rounding of the same metric, not different counting conventions. The 16.8× TTFT reduction is measured using mean TTFT across all turns, from 47.94 s to 2.85 s. We will clarify the metric definition and revise this throughout the paper.

Q6: Longer contexts and multi-GPU scaling

Regarding scalability beyond our current evaluation scope, we discuss the following considerations:

  • Longer contexts: The 4× KV payload reduction yields increasing absolute memory savings as context length grows. Meanwhile, the fixed sink and local-window regions account for a decreasing fraction of the cache, while the sparse attention budget remains fixed at 2,048 tokens.
  • Tensor parallelism and data-parallel attention: Since the joint-head latent is shared across all KV heads in a layer, it cannot be naively sharded by KV head across TP ranks. Similar to MLA's shared-latent design, data-parallel attention provides a natural alternative. Duet could follow the same strategy.
  • Context parallelism: The associative log-sum-exp merging in Equation 4 naturally supports ring- or tree-based reductions across token shards. Coordinating compression watermarks and native KV pools across shards remains future work.

I thank the authors for the additional experiments and the detailed discussion. They are helpful for better understanding the contribution. However, the discussion further deepened my concerns about the method's effectiveness.

The response confirms that low rank itself brings no attention speedup, and the serving gains come from the HBM capacity. My concern is that low rank does not appear to be the effective tool for this purpose. The authors' own comparison shows that at the paper's default ratio of about 4x, INT4 quantization reaches a lower reconstruction error than Duet at rank 256. And combined with INT8 is still much worse than INT4 alone, which means the accuracy loss mainly comes from low-rank.

Quantization seems to be a better solution, with lower reconstruction error. Additionally, quantization also speeds up the attention, since it reduces the bytes read per attention step during memory-bandwidth bound decoding. Quantization is also a more mature and widely deployed technique, and recent research such as DeepSeek-V4.1-Flash, NVFP4 KV cache, and BitDecoding-FP4 (Hunyuan) has already shown that aggressive KV quantization works well in serving.

Given this, I am not convinced that low rank is the right mechanism for the setting the paper targets, and I am concerned about where exactly the proposed scheme applies and how much it adds over the simpler alternative.

On the method's effectiveness I therefore have to change my rating to 2. I appreciate the authors' candor throughout the discussion, and recognize the contribution on the systems work.

Reviewer_i7yROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Duet, a KV compression framework for sparse-attention serving. It combines post-RoPE joint-head low-rank compression, a mixed native–latent attention operator, and dual-representation cache management. Evaluations on three models report improved throughput and TTFT, primarily through increased concurrency and prefix-cache reuse, with some accuracy degradation.

Critical strengths
  1. The paper clearly distinguishes reduced KV access from reduced KV residency. Its observation that compression benefits sparse-attention serving primarily through capacity rather than kernel acceleration is useful.

  2. The design addresses practical serving requirements, including mixed representations, batched compression, and phase-specific execution. The multi-turn results demonstrate meaningful benefits from improved prefix reuse.

Critical weaknesses
  1. The incremental contribution needs stronger validation.

Low-rank KV compression and projection absorption have been explored in prior work. Duet’s distinctive contribution appears to be their integration with selector-preserving mixed execution and online cache management.

  1. The memory and accuracy trade-offs need clearer reporting.

Please distinguish the 4× KV payload reduction from the total runtime memory footprint, and clarify the native-versus-compressed residency used in accuracy evaluation. Although average accuracy degradation is modest, some tasks show noticeably larger drops.

  1. Limited evaluation across model architectures.

The evaluation does not cover hybrid architectures such as Qwen3.5, which combines linear and full attention, or Gemma models with interleaved local and global attention. These architectures already reduce full-history KV storage in part of the network, changing the opportunity for compression. It remains unclear whether Duet’s memory savings translate into comparable end-to-end serving gains in these settings.

Questions for authors

Beyond addressing the weaknesses above, could the authors discuss the complementary benefits of combining Duet with HiSparse?

Much of Duet’s reported improvement appears to come from increased concurrency and prefix-cache reuse. When hierarchical KV management reduces GPU residency, where would compression provide additional gains, and how would these compare with its execution overhead? A cost analysis or preliminary measurement would help clarify this trade-off.

2 comments

We thank the reviewer for thoughtful comments. Below, we provide detailed responses to your comments.

W1: Core contribution under sparse attention

We acknowledge that low-rank KV compression and projection absorption are not new, and do not claim them as our contributions. Duet makes low-rank compression practical for dynamic sparse attention and online serving. Naive integration introduces two problems: reconstructed-key summaries change up to 60% of selected blocks (App. A), while materialize-then-attend is 1.65–1.69× slower (Tab. 3).

1. Insights under sparse selection:

  • Capacity, not kernel speed: Since sparse attention already eliminates most KV traffic, compression primarily benefits serving by reducing KV residency and enabling higher concurrency, rather than accelerating attention kernels.
  • Decoupling selection from compression: Joint-head compression entangles KV heads, while the selector scores each head independently. Duet therefore computes summaries from original keys and maintains them independently of compressed payloads, preserving selector semantics.

2. Serving-system design. Duet addresses these challenges with three complementary designs: mixed native–latent attention without materialization, watermark-triggered batched compression, and dual-representation KV management with compressed slots as canonical block identities, preserving paged allocation, prefix caching, and CUDA Graph replay.

W2: Runtime footprint, residency, and accuracy

Duet's 4x KV payload reduction corresponds to a 3.40x total per-block footprint reduction, and Tables 2 and 9 were evaluated at 128-way concurrency with the unprotected history predominantly compressed.

KV payload versus total footprint. On Qwen3-4B with 4x compression ratio, resident bytes per logical block across all layers are:

ComponentQuestDuet
KV payload2,304 KiB576 KiB
Selector summaries144 KiB144 KiB
Block metadata—44 B
Total resident per block2,448 KiB720 KiB

Selector summaries are a sparse-attention cost paid identically by Quest and Duet. Separately, the native pool is sized by the maximum sequence length, the concurrency limit, and the compression watermark, so it is a fixed configuration constant whose share of the KV budget falls as the budget grows. Even accounting for this fixed reservation, Duet accommodates approximately 3× as many logical KV tokens in our experiments.

Accuracy residency and task drops. At 128-way concurrency, only 2 sink and 2 local-window blocks per request are protected, while the shared native pool provides approximately 448 additional tokens per request against 15K–40K prompts, leaving most history compressed. Task-level drops are concentrated in tasks requiring fine-grained information aggregation across selected blocks: e.g., GovReport on Llama-3.1-8B (-6.53 points), consistent with our analysis in Appendix G. On RULER, the larger gap on Mistral-Nemo-12B mainly comes from Variable Tracking (-8.2 points) and Multi-Query Retrieval (-6.0 points), while Duet matches Quest on QA tasks.

W3: Applicability to hybrid architectures

Duet's calibration, compression, and mixed attention are performed independently per layer. It can therefore target full/sparse-attention layers while leaving linear-attention and sliding-window layers unchanged.

Hybrid architectures (e.g., Qwen3.5 and Gemma) have a smaller KV footprint at a fixed context length because some layers maintain only bounded state. However, as context length and concurrency grow, the KV footprint of full-attention layers continues to scale linearly and may dominate the remaining memory cost. These layers therefore remain natural targets for Duet, with the overall benefit depending on their fraction of the model's total KV footprint.

Q: Combining Duet with HiSparse

We agree that combining Duet with HiSparse is a promising direction, as they address complementary bottlenecks: HiSparse reduces GPU residency through host offloading, while Duet reduces the representation size itself. Their combination naturally forms a three-tier hierarchy: native GPU blocks, compressed GPU blocks, and offloaded host latents.

  1. Smaller transfers and host footprint: Each offloaded block requires less storage, reducing PCIe traffic and host memory usage.
  2. Higher GPU cache capacity: Storing historical KV blocks as latents allows HBM to accommodate roughly 3× as many KV blocks, reducing GPU cache misses and the resulting PCIe transfers.

Cost versus gain. The additional cost comes from latent attention overhead and online compression, whose amortized per-block latency is reduced by up to 36× through batching. In return, each offloaded block requires smaller transfer volume, while the larger GPU cache can further reduce data transfers. Thus, combining the two could substantially improve memory and transfer efficiency.

Thanks for responses. keep the score.

Reviewer_P44jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Addressing the issue that dynamic sparse attention cannot reduce the memory occupation of KV cache, this paper proposes a service-oriented KV compression framework based on real service bottlenecks. While retaining the original selection semantics of the sparse attention mechanism, it converts the savings in KV memory into actual service performance improvements.

Critical strengths
  • Propose a framework for sparse attention methods that takes into account actual inference efficiency, and achieve inference acceleration compared to vLLM.
  • The method design is relatively reasonable, starting from real bottlenecks, and the structure of the paper is well-designed.
Critical weaknesses
  • Losing around 1 point on Longbench is actually not insignificant, and it doesn't seem worthwhile compared to the maximum 1.9x throughput improvement. It is estimated that the performance loss may mainly come from post-rope compression.
  • The inspiration for the architecture design primarily stems from MLA, yet this also hinders the generalization of this method to MLA architectures. With the increasing adoption of MLA in mainstream model architectures, this may become a drawback of the method.
  • Since some other related work has also been adapted Quest into the inference engine, such as hisparse[1] and vortex[2], it might be better to compare the throughput with related baselines; in addition, another contribution of this paper is that there is redundancy in the cross-head attention head kv. So existing related work, such as ResidualKV[3], should be discussed.
  • Due to the numerous method steps, there is a significant overhead, and it is estimated that acceleration cannot be achieved in the case of bs1.

[1] HiSparse: Scaling Sparse-Attention Decoding with Hierarchical KV Cache Management

[2] Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents

[3] ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference

Questions for authors
  • Can you compare some baselines such as hisparse or vortex?
  • Compared to compressing the kv cache, can offloading and then dynamically loading based on the index still significantly improve the hit rate of the prefix cache?
  • Could you please show the throughput comparison under the scenario of bs1 or smaller batch size?
2 comments

We sincerely thank the reviewer for recognizing our problem motivation and system design, and provide detailed responses to your comments below.

W1: Serving gains and post-RoPE trade-off

We consider this a favorable serving trade-off: Duet achieves up to 1.9× higher throughput and substantially reduces TTFT while supporting higher concurrency.

We acknowledge that post-RoPE compression trades reconstruction fidelity for inference efficiency. In a separate compression ablation, pre-RoPE compression achieves a LongBench macro-average of 48.08, compared with 46.68 for post-RoPE, providing an upper-bound estimate of the accuracy gap due to RoPE placement at our deployed compression setting. However, pre-RoPE prevents query-side absorption because RoPE lies between the query and compression basis, requiring per-token key reconstruction and re-rotation during decode. Consequently, its compressed attention kernel is approximately 3× slower. We choose post-RoPE to avoid this overhead.

W2: Generalization to MLA architectures

We agree that Duet's low-rank representation targets GQA models rather than native MLA models, which already train a shared KV latent. At the same time, a substantial number of recent models still use GQA (such as MiniMax-M2.5 and MiMo-V2.6), so the setting remains practically relevant.

W3 & Q1: HiSparse and ResidualKV

HiSparse and ResidualKV target a different layer of the serving stack and an orthogonal redundancy axis, respectively, so they are better seen as complementary to Duet than as alternative in-GPU compressors:

  • HiSparse: HiSparse is hierarchical KV offloading that keeps the full history in host DRAM behind a small fixed-size GPU cache, so it does not shrink a block but moves the bottleneck from HBM capacity to host-device bandwidth, which is also how that work frames its own cost. The two therefore compose: storing compressed blocks shrinks the host footprint and every per-miss transfer and fits more blocks in the GPU cache, while fetched latents can be attended directly.
  • ResidualKV: ResidualKV exploits KV redundancy along a different axis and at a different adoption cost: it performs inter-token reference-residual factorization rather than Duet's intra-token cross-head factorization, making the two composable rather than competing, and it trains a compressor and decompressor whereas Duet is training-free and needs only an offline calibration.

We thank the reviewer for these references and will add the discussion in the revision.

Q2: Prefix offloading versus GPU compression

Dynamic offloading with index-based loading raises the prefix-cache hit rate effectively in multi-turn interactive serving, but is much weaker in large-batch offline generation:

  • Multi-turn interactive serving: Live conversations greatly outnumber the requests active in any single batch and most of a conversation's KV is idle between turns, so dynamic offload plus index-based onload can substantially raise the logical hit rate.
  • Large-batch offline generation: The working set is the union of all running requests and is active simultaneously, so index-based loading degenerates into sustained host-device traffic and becomes bandwidth-bound rather than capacity-bound.

W4 & Q3: Throughput at batch size 1

On Llama-3.1-8B in the offline setting (16k input and 1k output tokens), Duet reaches about 96% of Quest's throughput at batch size 1 and moves ahead once the batch becomes capacity-bound:

Throughput (tok/s)BS = 1BS = 64
Quest1,0364,859
Duet990 (0.96x)5,942 (1.22x)

Compression buys cache capacity, and at batch size 1 a single request's history already fits in HBM, so that capacity has nothing to hold and none of it converts into speed; what remains is only the small kernel-level cost of the compressed path. Duet targets capacity-bound serving at high concurrency: we do not claim a batch-1 speedup, and the few-percent gap is the price of keeping the compressed path uniform rather than special-casing small batches.

Dear Reviewers,

Thank you again for your time. The authors have posted their rebuttal, please take a moment to read their response, share your follow-up thoughts, and update your rating if appropriate.

Best, Area Chair

Reviewer_eHW1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Duet, a KV compression system for serving LLMs with dynamic sparse attention. The starting observation is that selecting fewer KV blocks reduces attention traffic but does not remove the need to store the full history. Duet compresses keys and values jointly across KV heads using offline low-rank bases. Keys are compressed after RoPE, allowing the key projection to be absorbed into the query; value aggregation is performed in latent space before projecting the output back. A mixed native/latent attention operator avoids reconstructing every selected token. The runtime combines stable logical block identities, deferred reclamation, and watermark-triggered compression, while keeping the sparse selector’s original-key summaries separate.

The experiments use three models and cover accuracy, offline inference, single-turn serving, and multi-turn prefix reuse. The reported throughput gains come largely from greater serving capacity and reduced prefix eviction, rather than uniformly faster attention. Rank, quantization, and additional serving results are included in the appendix.

Critical strengths
  1. The representation and serving runtime are designed together. The paper goes beyond storing a smaller KV tensor. Query-side absorption, latent value aggregation, mixed-representation attention, and stable block management address the work that would otherwise consume the memory benefit. Table 3 supports this with a roughly 1.65–1.69× improvement over materialize-then-attend. This is the most substantial technical contribution.

  2. The serving results explain the source of the improvement. Table 5 is particularly helpful: on the multi-turn workload, prefix hits rise from 6.9% for Quest to 63.8% for Duet, while average waiting requests fall from 33.3 to 3.5. The paper also reports the approximately 7% single-turn P95 ITL increase instead of suggesting that every latency metric improves. These measurements make the system argument convincing.

Critical weaknesses
  1. The accuracy characterization is broader than the task-level evidence. The LongBench macro-average losses are small, but some tasks are affected appreciably: Llama GovReport falls from 34.31 to 27.78, and Qwen MultiFieldQA from 54.91 to 46.64. Appendix F also reports a two-point RULER average loss for Mistral. These results are disclosed, so the issue is not missing evaluation. It is whether “less than 1% average accuracy loss” adequately describes the operating point. More importantly, dual-resident blocks use native KV when available. The paper should state whether its accuracy evaluation reflects the compressed-only history encountered under memory pressure, or a mixture with substantial native residency.

  2. The end-to-end comparison does not isolate the advantage over a simpler compressed sparse-serving system. Dense attention and uncompressed Quest are useful controls, but they leave open how much of the serving gain requires Duet’s particular low-rank and mixed-representation design. Figure 9 and Appendix G show quantization added to Duet; they do not compare against quantization-only Quest with the same selector. A comparison at similar task accuracy and memory budget would help establish why this design is preferable, rather than only why more KV capacity is beneficial.

Questions for authors
  1. Under what native/latent residency conditions were Tables 2 and 9 evaluated? How does task accuracy change when the unprotected history is predominantly compressed? Please also clarify whether the abstract’s “1%” means relative loss or absolute score points.

  2. How does Duet compare with quantization-only Quest under the same selector and a comparable accuracy or resident-memory budget? The existing Duet-plus-quantization experiment answers a different question.

  3. What exactly is counted in token throughput, and what request count and measurement interval are used? Appendix D.1 specifies the 3,200-block native reservation; please give total resident KV bytes including that reservation and selector summaries, and identify the workload, baseline, and TTFT statistic behind the 16.8× headline.

2 comments

We sincerely thank the reviewer for the constructive evaluation and insightful questions. Below, we provide detailed responses to your comments.

W1 & Q1: Accuracy reporting and residency

Tables 2 and 9 already report accuracy under Duet's compressed-serving configuration.

Residency conditions. Both benchmarks ran at 128-way concurrency. Only 2 sink and 2 local-window blocks per request remain protected (64 of 2,048 attended tokens, or 3.1%). The shared native pool provides approximately 448 additional tokens per request. Thus, against a 15K–40K prompt, only approximately 1.3–3.4% of the history remains native, meaning over 96% is compressed. The native pool is shared across requests, and native copies are lazily reclaimed after compression under memory pressure.

Clarifying "1%". The abstract's "1%" refers to average absolute score-point gaps across three models, not relative percentage loss: 0.92 points on LongBench and 0.91 on RULER. Larger task-level drops occur when tasks require preserving and aggregating multiple fine-grained facts across selected blocks, e.g., GovReport (-6.53 points), and Multi-Query Retrieval (-6.0 points). Since selection remains unchanged, these drops reflect approximation of selected KV contents rather than selector errors. We will clarify these variations and revise the abstract.

W2 & Q2: Quantization-only Quest baseline

We agree that Fig. 9 and App. G demonstrate compatibility with quantization, but do not establish its advantage over quantization alone. We therefore conduct a matched-budget reconstruction-error comparison on Qwen3-4B, revealing two regimes: quantization is more accurate at moderate compression ratios (up to approximately 5×), while combining low-rank compression with quantization remains effective at more aggressive ratios.

Baseline fairness. We measure relative Frobenius error using asymmetric min-max quantization with both per-channel and per-token grouping. For each tensor, we report the lower-error configuration (KIVI-style: per-channel keys and per-token values), including scales and zero-points in the storage budget.

CompressionSchemeKey errorValue error
1.9xQuant. only, INT80.00500.0069
3.8xQuant. only, INT40.08440.1175
4.0xDuet latent, rank 2560.26910.3727
4.9xQuant. only, INT30.18090.2521
7.1xQuant. only, INT20.42710.5913
7.8xDuet, rank 256 + INT80.26970.3729
10.2xDuet, rank 256 + K4V80.38890.3729
15.5xDuet, rank 128 + INT80.33340.5006

Trade-off. Quantization is indeed more accurate at moderate ratios (INT4 vs. rank-256 Duet). However, its error increases sharply below 3 bits: at 7.8×, rank-256 Duet + INT8 achieves approximately 1.6× lower error on both tensors than INT2 at 7.1×. Our 10.2× Duet + K4V8 configuration further demonstrates the potential of combining the two techniques. These combined configurations are untuned, using uniform global bit widths, rather than jointly optimized schemes. Thus, our results establish the complementary benefits of low-rank compression and quantization, rather than claiming that low rank universally outperforms quantization. A promising future direction is to jointly optimize low-rank decomposition and quantization, for example, by allocating more bits to latent components associated with larger singular values.

Q3: Throughput, resident bytes, and TTFT

Token throughput. We report total-token throughput (prompt + generated tokens, including prefix hit), divided by the full run duration. Each multi-turn run contains 64 conversations up to three turns, with Gamma arrivals (burstiness 1).

16.8× TTFT headline. This is the reduction in mean TTFT across all turns, comparing Duet against uncompressed Quest with the same selector, on multi-turn Qwen3-4B at 0.4 req/s (Figure 7): 47.9 s vs. 2.85 s.

Resident KV bytes. For Qwen3-4B with 1/4 low-rank compression, resident bytes per logical block across all layers are:

ComponentQuestDuet
KV payload2,304 KiB576 KiB
Selector summaries (per-page min/max, page-valid flag)144 KiB144 KiB
Block metadata—44 B
Total resident per block2,448 KiB720 KiB

KV payload shrinks 4x and total per-block footprint shrinks 3.40x. The 144 KiB selector summaries are a sparse-attention cost paid identically by Quest and Duet and require original keys to leave selection unchanged (Section 3.1). Separately, Appendix D.1's 3,200-block native reservation is sized by the maximum sequence length, the concurrency limit, and the compression watermark W rather than by cache capacity, so it is a fixed configuration constant whose share of the KV budget falls as the budget grows, and the total footprint therefore amortizes toward the 3.40x per-block ratio.

Dear Reviewers,

Thank you again for your time. The authors have posted their rebuttal, please take a moment to read their response, share your follow-up thoughts, and update your rating if appropriate.

Best, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.