- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The authors propose DeSKV which keeps exact KV in host memory, routes with INT4 rank-32 key-weighted sketch. It reuses the previous step's selection to overlap routing with the forward pass. DeSKV obtains 48.35 LongBench vs 48.51 full cache and reduces decode latency by 33% at 97K context on Qwen3-8B .
Critical strengths
- Simple and useful finding that best query-key mixing weight shifts toward keys during decoding
- Extensive latency measurement with CUDA graphs and accounting of GPU footprint
- Step deferral is easy to implement and gives a clear speedup over immediate selection
Critical weaknesses
- Mostly a combination of known pieces with concepts from Loki routing, PQCache, InfLLM offload
- Mistral loses 3 points vs full cache but main text says near-full-cache. The latency gain is only at 97K and batch size , at 8K the method is slower than full cache
Questions for authors
- Authors should report TPOT at batch size 4, 8, 16 at 97K context
- Why does Mistral lose 3 points when Qwen and Llama do not?
5 comments
Gentle reminder: author–reviewer discussion ending in two days
Dear Reviewer 337L,
Thank you again for your review and helpful questions. With the author–reviewer discussion period ending in two days, we would like to briefly follow up on our responses and the revised manuscript.
Regarding batch scaling, Appendix E.3, Table 20 now reports results at batch sizes 1, 2, 4, 8, and 16 with distinct prompts. Deferral reduces latency relative to immediate DeSKV at every tested batch size; at batch 16, latency decreases from 46.03 to 36.19 ms/token per request.
Regarding Mistral’s larger quality loss, Appendix D.1 includes a diagnostic showing that a substantial gap is already present with exact-score immediate selection. Increasing the prompt budget from 256 to 1,024 improves the deployed six-task mean from 48.74 to 54.39, compared with 55.37 for full cache. Section 6.3 also explains why routing overhead outweighs attention savings at 8K context.
We would appreciate any follow-up feedback on whether these additions address your questions. If anything remains unclear, we would be happy to clarify it during the remaining discussion period.
Thank you again for your time and consideration.
Best regards,
The authors
Follow-up before the discussion deadline
Dear Reviewer 337L,
With one day remaining in the discussion period, we would appreciate any brief feedback on whether our responses and revised manuscript address your main concerns. If any questions remain, we would be happy to clarify them before the discussion closes.
Thank you again for your time and consideration.
Best regards, The authors
W1. Relationship to Loki, PQCache, and InfLLM
DeSKV addresses both routing accuracy and the cost of preparing a small working set for decoding. Loki constructs a low-rank basis from key statistics, PQCache uses product-quantized keys, and InfLLM retrieves blocks through representative keys. Our analysis examines how a shared query-key projection preserves routing scores as decode queries change. Sections 4.1–4.2 show that the mixture preferred during prefill differs from the one preferred during decoding. This finding motivates fitting the basis jointly, with more weight on the fixed prompt keys, and using the resulting compact sketch for token-level selection.
For execution, Figure 5(a,b) identifies two patterns: deferral incurs larger losses during the initial decode steps, and missed high-ranked targets tend to lie near selected positions. Priming uses current-step selection during the initial interval, while local expansion shares scores between neighboring positions within the same token budget. Appendix D.6 evaluates these components separately, showing that pooling helps prepare selections for step deferral.
The contribution lies in designing the router and selection policy for deferred selection. Our analysis shows when deferral loses the most attention mass and where missed next-step targets occur. DeSKV uses these findings to prepare a compact, token-level working set that supports deferred execution at 128–256 tokens per head. We will make this connection clearer in the revised paper.
Q1. TPOT at batch sizes 4, 8, and 16
We measured Qwen3-8B at 97K context on one 96 GB RTX PRO 6000, using a 256-token prompt budget per KV head. Each request has a different prompt and independent KV state. Table R11 reports TPOT with the complete decode step captured in a CUDA Graph.
Table R11. Batch-size scaling on Qwen3-8B at 97K context. latencies are medians of five 128-step runs after 16 warm-up steps, with the complete decode step captured in a CUDA Graph. OOM denotes insufficient GPU memory.
Deferral reduces TPOT from 25.43 to 20.11 ms at batch 4 and from 46.03 to 36.19 ms at batch 16. Routing and KV-transfer work increase with the number of requests, but overlapping this preparation continues to reduce decoding latency across the tested batch sizes.
At batch 16, the deferral generates approximately 442 tokens/s in total, with about 228 GB of exact KV stored in host memory. Full-cache execution exceeds GPU memory capacity at batches 8 and 16. We will include these measurements in the revised paper.
W2-1. Performance at shorter contexts
DeSKV targets long-context generation, where KV-cache storage and access become major memory-capacity/bandwidth bottlenecks, and reducing the GPU-resident working set and per-step KV reads provides greater benefits. Therefore at 8K context and batch size one, the time saved by reducing KV access is smaller than the remaining routing and fetching overhead, making DeSKV slightly slower than full-cache execution.
W2, Q2. Mistral's larger quality loss
Mistral is more sensitive to tight prompt-KV budgets than Qwen3 and Llama. To identify where the loss occurs, we separately examined working-set restriction, score approximation, and deferred selection. Table R12 reports results on the first 64 examples of six LongBench tasks: HotpotQA, 2WikiMQA, PassageRetrieval-en, Qasper, GovReport, and TREC.
Table R12. Mistral-7B-Instruct-v0.3, mean over six tasks. Budgets include 32-token prompt tail.
At a 256-token budget, Mistral's mean score drops from 55.37 with full cache to 48.99 with exact-score immediate selection. INT4 immediate selection scores 49.20, and the deployed policy scores 48.74. Thus, a substantial gap is already shown before score approximation or deferred execution is introduced.
We also compared exact-score selection across models on the three shared tasks: HotpotQA, 2WikiMQA, and TREC. At budget 256, the full-cache and exact-score selection means are 53.7 and 48.6 for Mistral, 57.1 and 56.8 for Llama, and 53.9 and 53.7 for Qwen3. The same working-set restriction therefore has a larger effect on Mistral.
Increasing DeSKV's prompt budget from 256 to 1,024 raises Mistral's six-task mean from 48.74 to 54.39, compared with 55.37 for full cache. These results support strong budget sensitivity on Mistral. We will make this model dependence explicit and scope the near-full-cache quality statement to the reported Qwen3 and Llama results.