- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents SparseEngine, a ground-up sparse-first inference engine built around a shared lifecycle contract. It supports 15 exisiting sparse methods in four categories (Dynamic sparse attention, KV eviction, KV compression, KV quantization). Lifecycle hooks and method-customized Cachemanager in their design support the easy implementation of new sparse methods. Chain Cache and Sparsity-Based Prefix-Cache Pruning enable the sparse methods to improve the actual memory footprint and efficiency.
Critical strengths
-
SparseEngine provides a well-defined shared interface for existing or future sparse attention methods to integrate with the capabilities of inference engines. This allows other researchers to integrate and test their sparse methods easily, which encourages new research in the sparse attention domain.
-
The results of serving performance show that SparseEngine can translate the sparse methods into practical performance gains. This can encourage local LLM users and LLM serving providers to adopt these methods, turning the existing research to real-world benefit.
Critical weaknesses
- The proposed interface is well-suited for the sparse methods presented in the paper. However, the authors do not discuss its limitations, which makes the claim of broad compatibility with the sparse methods less convincing. Some examples of potential limitations are listed in Questions section.
Questions for authors
-
Is it possible to apply two or more sparse methods from different families concurrently? For example, is it possible to serve SnapKV (KV Eviction) and TurboQuant (KV Quantization) combined in a single serving instance? Furthermore, what about Dynamic Sparsity family and KV Eviction family? If not, what additional work would be required (e.g. new implementation code for each combination)?
-
Could you provide the throughput comparisons of KV Quantization or KV Compression (if possible) against other comparable engines?
-
Are there examples of existing sparse methods or families (like KVComm, CacheBlend, or FastKV) which cannot be integrated with SparseEngine, or those which can be integrated but fail to translate their theoretical performance gains into practical gains?
4 comments
Response (2/2) to Reviewer eqsq
Q2: Throughput comparisons of KV Quantization or KV Compression against other comparable engines.
Conclusion: Under matched settings on Llama-3.1-8B (Table R13), SparseEngine matches vLLM on FP8 KV quantization (within 1.5%) and outperforms vLLM by 7.3% on TurboQuant. For KV Compression (e.g., Palu, LoRC, DeltaKV), no comparable serving engine currently supports payload-replacement contracts for direct throughput comparison.
Setup: All runs use 32K input tokens, batch size 8 and CUDA Graphs. Decoding throughput is measured over 256 consecutive steps, with 3 repetitions.
Table R13: Decoding throughput with KV cache quantization (Llama-3.1-8B).
| GPU | KV Format | SparseEngine (tokens/s) | vLLM (tokens/s) | SparseEngine / vLLM |
|---|---|---|---|---|
| PRO5000 | BF16 | 188.1 | 189.1 | 1.0× |
| PRO5000 | FP8 | 274.2 | 278.5 | 1.0× |
| H100 | BF16 | 452.6 | 447.1 | 1.0× |
| H100 | TurboQuant | 160.1 | 149.3 | 1.1× |
Q3: Are there examples of existing sparse methods or families (like KVComm, CacheBlend, or FastKV) which cannot be integrated with SparseEngine?
Conclusion: SparseEngine's abstraction boundary is defined by a per-request lifecycle contract anchored to native Transformer execution order (Sec. 3.1) and a prefix-aligned cross-request state contract (, Sec. 3.2 & App. B.2). Under these contracts, FastKV is directly supported (achieving 2.0× decode speedup), whereas CacheBlend faces bounded practical speedups and KVComm falls out of scope (Tables R14–R15):
- FastKV (In-scope & fast): Fits natively via
on_layer_endpropagation andCacheManagercompaction (172 LoC), achieving 2.0× decode speedup over Dense and +0.4 LongBench score over the original paper. - CacheBlend (Expressible, bounded speedup): Expressible within a single forward pass (113 LoC), but two boundaries limit latency gains: (1) it targets RAG non-prefix chunk splicing, whereas our cross-request reuse (
ChainCache/ Radix) assumes prefix-aligned turns (Sec. 3.2); and (2) mid-sequence recomputation introduces dynamic shapes that break static CUDA Graph replay (App. B). - KVComm (Out of scope): Violates our exact-prefix invariant (Sec. 3.2 & App. B.2) and single-request
CacheManagerownership (Sec. 3.1) by transferring and adapting KV across divergent prefixes or distinct agents/models, requiring an inter-agent communication layer above the engine.
Setup: Decoding throughput is measured with batch size 8 and 32K input tokens. FastKV retains 10% of KV tokens and 20% of TSP tokens, with token selection propagated at layer 15.
Table R14: Quality on LongBench and decoding throughput of FastKV (Llama-3.1-8B, H100).
| Method | LongBench Score | Δ vs. Dense / Origin Paper | Decode (tokens/s) | Speedup vs. Dense |
|---|---|---|---|---|
| Dense | 50.7 | — | 452.7 | — |
| FastKV | 48.8 | −1.9 / +0.4 | 918.7 | 2.0× |
Table R15: Method-specific lines of code for integrating new and composite sparse methods.
| Method | Runtime / Algorithm | Cache Manager | Activation Hooks | Method-specific Kernels | Total Effective LoC |
|---|---|---|---|---|---|
| FastKV | 49 | 110 | 13 | 0 | 172 |
| CacheBlend | 31 | 82 | 0 | 0 | 113 |
| Quest + FastKV | 14 | 97 | 0 | 0 | 111 |
| OmniKV + SnapKV | 81 | 20 | 0 | 0 | 101 |
| Quest + SnapKV | 11 | 125 | 0 | 54 | 190 |
Thank you for your efforts to address my concerns. I have reviewed the response and the revised PDF, and I believe the practical limitations and the potential of SparseEngine are now more clearly separated.
As I understand it, SparseEngine makes it possible to integrate per-request sparse methods into serving systems. This helps efficient LLM research achieve real speedups and be put into real-world use.
I will therefore keep my positive rating and raise my soundness score to 3.
Thank you very much for your helpful comments and suggestions. We sincerely appreciate your time and careful review, which have helped us improve the manuscript.
Response (1/2) to Reviewer eqsq
Thank you for your thoughtful review and for highlighting SparseEngine's unified interface for researchers and its ability to translate sparse attention methods into practical serving speedups.
Below, we address your limitation discussion request and three questions, and we have incorporated these additions into the updated PDF.
Thank you for this valuable suggestion. We agree that explicitly discussing SparseEngine's limitations clarifies its operational scope. We address the specific aspects raised in your questions (Q1–Q3) in detail below, and we have added a dedicated Limitations discussion in the revised manuscript.
In addition to the scope boundaries discussed in Q1–Q3, we also discuss Prefill-Decode (PD) disaggregation as a practical limitation for large-scale production deployment. SparseEngine currently has not yet implemented PD disaggregation, which limits direct deployment in distributed setups that decouple prefill and decode pools.
However, our lifecycle contract poses no fundamental architectural barrier to PD disaggregation, supporting it mainly requires defining a unified serialization and transfer protocol for method-owned KV layouts and metadata. Moreover, applying sparse compaction or compression during prefill offers a promising advantage by reducing inter-node KV bandwidth overhead between prefill and decode nodes.
Conclusion: We have added the composition experiments to the revised paper (Tables 8–9, Appendices B.3 and C.2). SparseEngine supports composing Dynamic Sparsity, KV Eviction, and KV Quantization within a single inference pipeline (
Quest/OmniKV + SnapKV + FP8/TurboQuant). On Qwen3-30B-A3B-Instruct-2507-FP8 (PRO5000), OmniKV + SnapKV achieves 3.3× decode speedup over Dense with BF16 KV, compared with 2.5× for OmniKV and 2.4× for SnapKV alone. Adding FP8 KV raises the speedup to 3.6×, with the same LongBench v1 score as Dense (53.2). On Llama-3.1-8B, the same three-family combination achieves 2.6× speedup with a score of 50.0 vs. 50.7 for Dense.TurboQuant also composes with dynamic selection and eviction while retaining comparable quality, but reduces throughput relative to BF16 in the evaluated H100 settings; vanilla decoding in both SparseEngine and vLLM shows the same trend.
Implementation: Combining dynamic selection with physical eviction introduces semantic conflicts across granularities and layers. SparseEngine's lifecycle hooks (
SparseController) andCacheManagerresolve these cleanly through lightweight adapters.Quest + SnapKV): SnapKV evicts individual tokens, whereas Quest selects fixed-size pages. In SparseEngine, SnapKV runs physical token compaction atfinish_stepduring prefill;CacheManagerthen constructs Quest's page summaries (min/maxkeys) directly over the compacted physical slots.OmniKV + SnapKV): OmniKV reuses token indices selected by observation layers across subsequent sparse layers, requiring identical retained positions across layers. Through hook configuration, SnapKV enforces a shared cross-layer eviction mask so OmniKV'son_layer_endindices map validly to every layer's physical cache.+ FP8 / TurboQuant):CacheManagerstores compacted KV entries in quantized layouts (FP8orTurboQuant) and exposes compatible payloads to attention kernels viaAttentionView.Integration effort: The evaluated combinations use method-specific adapters to coordinate cache updates and selection state. OmniKV + SnapKV and Quest + SnapKV require 101 and 190 effective lines of method-specific code, respectively, as reported in Table 6 and Appendix B.3 of the revised paper.