acceptodds
Under review as a conference paper at ICLR 2027

SparseEngine: Sparse-First Inference Engine

Abstract

Long-context LLM-based agents accumulate growing interaction histories, placing substantial pressure on KV cache memory and attention computation. Consequently, recent works have adopted sparse attention to reduce computational and memory overheads. However, due to the heterogeneous nature of sparse methods, they cannot be directly applied to existing inference engines. *Sparse inference engines* have therefore been proposed, but existing serving abstractions tie method integration to particular cache layouts or computational workflows, limiting support for heterogeneous sparse methods. We present **SparseEngine**, a ground-up sparse-first inference engine built around a shared lifecycle contract. The contract lets each method control its KV representation and computational workflow while coordinating state transitions with shared serving infrastructure. SparseEngine supports **15** sparse methods across four major categories, demonstrating its generality. Furthermore, built upon its lifecycle contract, it enables higher-level cross-request state management: we propose **Chain Cache** to allow KV eviction methods to resume from retained history, and Controllable **Prefix-Cache Pruning** to let applications prune KV from selected history regions while preserving logical prefix matching. While preserving the quality of sparse methods, SparseEngine achieves over **10×** higher throughput with KV eviction methods and over **2.5×** decode throughput speedup under identical concurrency, compared to vLLM. It also achieves over **2×** end-to-end speedup on agent benchmarks. The code is available at [https://anonymous.4open.science/r/SparseEngine](https://anonymous.4open.science/r/SparseEngine).

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Inference Engine, Infrastructure, Sparse Attention, KV Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eqsqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents SparseEngine, a ground-up sparse-first inference engine built around a shared lifecycle contract. It supports 15 exisiting sparse methods in four categories (Dynamic sparse attention, KV eviction, KV compression, KV quantization). Lifecycle hooks and method-customized Cachemanager in their design support the easy implementation of new sparse methods. Chain Cache and Sparsity-Based Prefix-Cache Pruning enable the sparse methods to improve the actual memory footprint and efficiency.

Critical strengths
  1. SparseEngine provides a well-defined shared interface for existing or future sparse attention methods to integrate with the capabilities of inference engines. This allows other researchers to integrate and test their sparse methods easily, which encourages new research in the sparse attention domain.

  2. The results of serving performance show that SparseEngine can translate the sparse methods into practical performance gains. This can encourage local LLM users and LLM serving providers to adopt these methods, turning the existing research to real-world benefit.

Critical weaknesses
  1. The proposed interface is well-suited for the sparse methods presented in the paper. However, the authors do not discuss its limitations, which makes the claim of broad compatibility with the sparse methods less convincing. Some examples of potential limitations are listed in Questions section.
Questions for authors
  1. Is it possible to apply two or more sparse methods from different families concurrently? For example, is it possible to serve SnapKV (KV Eviction) and TurboQuant (KV Quantization) combined in a single serving instance? Furthermore, what about Dynamic Sparsity family and KV Eviction family? If not, what additional work would be required (e.g. new implementation code for each combination)?

  2. Could you provide the throughput comparisons of KV Quantization or KV Compression (if possible) against other comparable engines?

  3. Are there examples of existing sparse methods or families (like KVComm, CacheBlend, or FastKV) which cannot be integrated with SparseEngine, or those which can be integrated but fail to translate their theoretical performance gains into practical gains?

4 comments
AuthorsOpenReview ↗

Response (1/2) to Reviewer eqsq

Thank you for your thoughtful review and for highlighting SparseEngine's unified interface for researchers and its ability to translate sparse attention methods into practical serving speedups.

Below, we address your limitation discussion request and three questions, and we have incorporated these additions into the updated PDF.

W1: Lack of limitation discussion.

Thank you for this valuable suggestion. We agree that explicitly discussing SparseEngine's limitations clarifies its operational scope. We address the specific aspects raised in your questions (Q1–Q3) in detail below, and we have added a dedicated Limitations discussion in the revised manuscript.

In addition to the scope boundaries discussed in Q1–Q3, we also discuss Prefill-Decode (PD) disaggregation as a practical limitation for large-scale production deployment. SparseEngine currently has not yet implemented PD disaggregation, which limits direct deployment in distributed setups that decouple prefill and decode pools.

However, our lifecycle contract poses no fundamental architectural barrier to PD disaggregation, supporting it mainly requires defining a unified serialization and transfer protocol for method-owned KV layouts and metadata. Moreover, applying sparse compaction or compression during prefill offers a promising advantage by reducing inter-node KV bandwidth overhead between prefill and decode nodes.

Q1: Is it possible to apply two or more sparse methods from different families concurrently?

Conclusion: We have added the composition experiments to the revised paper (Tables 8–9, Appendices B.3 and C.2). SparseEngine supports composing Dynamic Sparsity, KV Eviction, and KV Quantization within a single inference pipeline (Quest/OmniKV + SnapKV + FP8/TurboQuant). On Qwen3-30B-A3B-Instruct-2507-FP8 (PRO5000), OmniKV + SnapKV achieves 3.3× decode speedup over Dense with BF16 KV, compared with 2.5× for OmniKV and 2.4× for SnapKV alone. Adding FP8 KV raises the speedup to 3.6×, with the same LongBench v1 score as Dense (53.2). On Llama-3.1-8B, the same three-family combination achieves 2.6× speedup with a score of 50.0 vs. 50.7 for Dense.

TurboQuant also composes with dynamic selection and eviction while retaining comparable quality, but reduces throughput relative to BF16 in the evaluated H100 settings; vanilla decoding in both SparseEngine and vLLM shows the same trend.

Implementation: Combining dynamic selection with physical eviction introduces semantic conflicts across granularities and layers. SparseEngine's lifecycle hooks (SparseController) and CacheManager resolve these cleanly through lightweight adapters.

  1. Granularity & Index Alignment (Quest + SnapKV): SnapKV evicts individual tokens, whereas Quest selects fixed-size pages. In SparseEngine, SnapKV runs physical token compaction at finish_step during prefill; CacheManager then constructs Quest's page summaries (min/max keys) directly over the compacted physical slots.
  2. Cross-Layer Consistency (OmniKV + SnapKV): OmniKV reuses token indices selected by observation layers across subsequent sparse layers, requiring identical retained positions across layers. Through hook configuration, SnapKV enforces a shared cross-layer eviction mask so OmniKV's on_layer_end indices map validly to every layer's physical cache.
  3. Orthogonal Payload Quantization (+ FP8 / TurboQuant): CacheManager stores compacted KV entries in quantized layouts (FP8 or TurboQuant) and exposes compatible payloads to attention kernels via AttentionView.

Integration effort: The evaluated combinations use method-specific adapters to coordinate cache updates and selection state. OmniKV + SnapKV and Quest + SnapKV require 101 and 190 effective lines of method-specific code, respectively, as reported in Table 6 and Appendix B.3 of the revised paper.

AuthorsOpenReview ↗

Response (2/2) to Reviewer eqsq

Q2: Throughput comparisons of KV Quantization or KV Compression against other comparable engines.

Conclusion: Under matched settings on Llama-3.1-8B (Table R13), SparseEngine matches vLLM on FP8 KV quantization (within 1.5%) and outperforms vLLM by 7.3% on TurboQuant. For KV Compression (e.g., Palu, LoRC, DeltaKV), no comparable serving engine currently supports payload-replacement contracts for direct throughput comparison.

Setup: All runs use 32K input tokens, batch size 8 and CUDA Graphs. Decoding throughput is measured over 256 consecutive steps, with 3 repetitions.

Table R13: Decoding throughput with KV cache quantization (Llama-3.1-8B).

GPUKV FormatSparseEngine (tokens/s)vLLM (tokens/s)SparseEngine / vLLM
PRO5000BF16188.1189.11.0×
PRO5000FP8274.2278.51.0×
H100BF16452.6447.11.0×
H100TurboQuant160.1149.31.1×

Q3: Are there examples of existing sparse methods or families (like KVComm, CacheBlend, or FastKV) which cannot be integrated with SparseEngine?

Conclusion: SparseEngine's abstraction boundary is defined by a per-request lifecycle contract anchored to native Transformer execution order (Sec. 3.1) and a prefix-aligned cross-request state contract (P′=PP' = P, Sec. 3.2 & App. B.2). Under these contracts, FastKV is directly supported (achieving 2.0× decode speedup), whereas CacheBlend faces bounded practical speedups and KVComm falls out of scope (Tables R14–R15):

  • FastKV (In-scope & fast): Fits natively via on_layer_end propagation and CacheManager compaction (172 LoC), achieving 2.0× decode speedup over Dense and +0.4 LongBench score over the original paper.
  • CacheBlend (Expressible, bounded speedup): Expressible within a single forward pass (113 LoC), but two boundaries limit latency gains: (1) it targets RAG non-prefix chunk splicing, whereas our cross-request reuse (ChainCache / Radix) assumes prefix-aligned turns (Sec. 3.2); and (2) mid-sequence recomputation introduces dynamic shapes that break static CUDA Graph replay (App. B).
  • KVComm (Out of scope): Violates our exact-prefix invariant (Sec. 3.2 & App. B.2) and single-request CacheManager ownership (Sec. 3.1) by transferring and adapting KV across divergent prefixes or distinct agents/models, requiring an inter-agent communication layer above the engine.

Setup: Decoding throughput is measured with batch size 8 and 32K input tokens. FastKV retains 10% of KV tokens and 20% of TSP tokens, with token selection propagated at layer 15.

Table R14: Quality on LongBench and decoding throughput of FastKV (Llama-3.1-8B, H100).

MethodLongBench ScoreΔ vs. Dense / Origin PaperDecode (tokens/s)Speedup vs. Dense
Dense50.7—452.7—
FastKV48.8−1.9 / +0.4918.72.0×

Table R15: Method-specific lines of code for integrating new and composite sparse methods.

MethodRuntime / AlgorithmCache ManagerActivation HooksMethod-specific KernelsTotal Effective LoC
FastKV49110130172
CacheBlend318200113
Quest + FastKV149700111
OmniKV + SnapKV812000101
Quest + SnapKV11125054190

Thank you for your efforts to address my concerns. I have reviewed the response and the revised PDF, and I believe the practical limitations and the potential of SparseEngine are now more clearly separated.

As I understand it, SparseEngine makes it possible to integrate per-request sparse methods into serving systems. This helps efficient LLM research achieve real speedups and be put into real-world use.

I will therefore keep my positive rating and raise my soundness score to 3.

Thank you very much for your helpful comments and suggestions. We sincerely appreciate your time and careful review, which have helped us improve the manuscript.

Reviewer_Ea63OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SparseEngine, a unified serving framework designed to support a diverse range of heterogeneous sparse attention mechanisms and KV-cache optimization techniques. Its core contribution is a lifecycle-based abstraction mechanism.

Furthermore, the paper presents "Chain Cache," which enables the reuse of compressed KV states across multiple conversational turns within eviction-based schemes, and "Controllable Prefix-Cache Pruning," which allows applications to selectively reclaim KV memory space while maintaining logical prefix matching.

Experimental results across various models and benchmarks demonstrate that the system offers broad coverage of optimization methods. It delivers highly competitive throughput—comparable to engines specifically engineered for sparse inference—and achieves up to a 2.24x speedup in end-to-end inference for agent trajectory replay tasks. The authors also evaluate whether the integrated sparse methods maintain output quality consistent with their original reference implementations.

Critical strengths
  1. Substantial engineering effort and extensive support for a wide range of heterogeneous sparsity methods.

This paper presents a comprehensive inference system capable of accommodating diverse sparsity methods that differ fundamentally in their KV cache storage and execution requirements. The system’s broad support for various sparsity techniques and model architectures underscores the significant engineering investment involved and highlights its practical utility.

  1. Competitive serving performance and robust system-level evaluation.

Experimental results demonstrate that the system achieves a significant throughput improvement over native vLLM, while also delivering highly competitive performance when compared against various specialized sparse inference systems. Furthermore, end-to-end evaluations in agent-based application scenarios validate the practical value of sparse inference and cache reuse for long-context applications.

Critical weaknesses
  1. There is a lack of evidence demonstrating that the proposed abstraction mechanism offers capabilities beyond merely supporting existing sparse methods.

Although SparseEngine accommodates various sparse methods, the evaluation primarily focuses on assessing these methods in isolation. It remains unclear whether the abstraction mechanism supports the combined execution of diverse, heterogeneous optimization techniques. For instance, integrating dynamic sparse attention, physical KV cache eviction, and quantization within a single inference pipeline.

  1. There is an insufficient number of controlled experiments to isolate and quantify the actual benefits of the proposed cache management mechanisms.

While "Chain Cache" and "Controllable Prefix-Cache Pruning" represent significant contributions, the individual gains attributable to each have not been adequately verified. Consequently, it is difficult to determine to what extent the reported end-to-end speedups stem from cross-turn cache reuse, as opposed to sparse computation, KV cache eviction, or increased concurrency.

Questions for authors
  1. Does SparseEngine currently support the concurrent execution of multiple heterogeneous sparsity methods within a single inference pipeline, rather than requiring the selection of only one method at a time? If so, could the authors clarify how the system reconciles conflicting requirements—such as those concerning KV representations, metadata updates, and the execution of attention mechanisms—and provide evidence of the benefits gained from this capability? If not, what are the intended operational boundaries for this abstraction?

  2. Compared to existing sparse inference frameworks, how much engineering effort is required to integrate a new sparsity method into SparseEngine? Such evidence would help demonstrate the practical advantages of this lifecycle abstraction, beyond simply the number of supported methods.

2 comments
AuthorsOpenReview ↗

Response (1/2) to Reviewer Ea63

Thank you for your constructive review and for recognizing SparseEngine's broad support across heterogeneous sparsity methods and model architectures, as well as its competitive serving throughput and end-to-end agent evaluations.

Below, we address your questions and provide new experimental evidence (incorporated into the updated PDF).

W1 & Q1: Unclear support for combining heterogeneous sparse methods within a single inference pipeline.

Conclusion: We have added the composition experiments to the revised paper (Tables 8–9, Appendices B.3 and C.2). SparseEngine supports composing Dynamic Sparsity, KV Eviction, and KV Quantization within a single inference pipeline (Quest/OmniKV + SnapKV + FP8/TurboQuant). On Qwen3-30B-A3B-Instruct-2507-FP8 (PRO5000), OmniKV + SnapKV achieves 3.3× decode speedup over Dense with BF16 KV, compared with 2.5× for OmniKV and 2.4× for SnapKV alone. Adding FP8 KV raises the speedup to 3.6×, with the same LongBench v1 score as Dense (53.2). On Llama-3.1-8B, the same three-family combination achieves 2.6× speedup with a score of 50.0 vs. 50.7 for Dense.

TurboQuant also composes with dynamic selection and eviction while retaining comparable quality, but reduces throughput relative to BF16 in the evaluated H100 settings; vanilla decoding in both SparseEngine and vLLM shows the same trend.

Implementation: Combining dynamic selection with physical eviction introduces semantic conflicts across granularities and layers. SparseEngine's lifecycle hooks (SparseController) and CacheManager resolve these cleanly through lightweight adapters.

  1. Granularity & Index Alignment (Quest + SnapKV): SnapKV evicts individual tokens, whereas Quest selects fixed-size pages. In SparseEngine, SnapKV runs physical token compaction at finish_step during prefill; CacheManager then constructs Quest's page summaries (min/max keys) directly over the compacted physical slots.
  2. Cross-Layer Consistency (OmniKV + SnapKV): OmniKV reuses token indices selected by observation layers across subsequent sparse layers, requiring identical retained positions across layers. Through hook configuration, SnapKV enforces a shared cross-layer eviction mask so OmniKV's on_layer_end indices map validly to every layer's physical cache.
  3. Orthogonal Payload Quantization (+ FP8 / TurboQuant): CacheManager stores compacted KV entries in quantized layouts (FP8 or TurboQuant) and exposes compatible payloads to attention kernels via AttentionView.

W2: Insufficient controlled experiments to isolate the individual benefits of Chain Cache and Prefix Cache Pruning.

Conclusion: We have added controlled w/ vs. w/o ablation experiments for both components to the revised paper (Section 4.3, Tables 3 and 12, Figures 6–10, and Appendices C.4–C.6). With hardware, precision, sparse configuration, and concurrency held fixed within each comparison, the results show:

  • Chain Cache: Reusing retained KV and method metadata across turns delivers 2.1×–2.8× end-to-end speedup, reduces mean TTFT by 2.1×–3.1×, and lowers peak occupied KV memory by 30.2%–49.0%, while preserving task performance for both SnapKV and H₂O.
  • Prefix-Cache Pruning: Pruning accelerates trajectory replay by 4.7× on AgentX and 1.7× on SWE-bench Lite, reduces mean TTFT by 31.2× and 14.1×, respectively, and reclaims up to 8.8 GiB per pruning operation. On SWE-bench Lite, delaying pruning (lag=2/4) matches or slightly exceeds the corresponding unpruned resolve rates (e.g., 26.7% vs. 26.0% for KVzip + OmniKV).
AuthorsOpenReview ↗

Response (2/2) to Reviewer Ea63

Q2: How much engineering effort is required to integrate a new sparsity method into SparseEngine?

Conclusion: We have integrated an additional method, FastKV, using 172 effective lines of method-specific code (Table R11), and evaluated FastKV's quality and decoding throughput (Table R12). FastKV achieves a LongBench score of 48.8, 0.4 points above the score reported in the original paper, with 2.0× decoding speedup over Dense.

We further extend support to combinations of methods, such as Quest + FastKV, OmniKV + SnapKV and Quest + SnapKV, which require 111, 101 and 190 effective lines of code, respectively.

Setup: Decoding throughput is measured with batch size 8 and 32K input tokens. FastKV retains 10% of KV tokens and 20% of TSP tokens, with token selection propagated at layer 15. FastKV uses window=8 and pool=7.

Table R11: Method-specific lines of code for integrating new and composite sparse methods.

MethodRuntime / AlgorithmCache ManagerActivation HooksMethod-specific KernelsTotal Effective LoC
FastKV49110130172
Quest + FastKV149700111
OmniKV + SnapKV812000101
Quest + SnapKV11125054190

Table R12: Quality on LongBench and decoding throughput of FastKV (Llama-3.1-8B, H100).

MethodLongBench ScoreΔ vs. Dense / Origin PaperDecode (tokens/s)Speedup vs. Dense
Dense50.7—452.71.0×
FastKV48.8−1.9 / +0.4918.72.0×
Reviewer_SpyROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SparseEngine, a sparse-first inference engine that integrates heterogeneous sparse attention and KV-cache optimization methods through a shared lifecycle abstraction. The framework allows individual sparse methods to manage their own KV representations and execution workflows while sharing common serving infrastructure. It supports 15 sparse methods across four categories and introduces Chain Cache and Controllable Prefix-Cache Pruning for cross-request cache management.

The paper reports substantial throughput improvements over vLLM and specialized sparse-serving systems, along with quality evaluations on long-context, reasoning, and agent benchmarks.

Critical strengths

Supporting many different sparse methods in one serving framework is of clear practical value. These methods usually need their own cache layout, metadata, and execution path, and SparseEngine handles this while keeping batching and prefix caching.

The design is reasonable. Separating method-specific logic, attention execution, and cache management into lifecycle hooks, CacheManager, and AttentionView seems like a clean way to add new methods without modifying the model code.

The throughput results are strong in the method-matched setting (1.24× vs. Vortex and 1.55× vs. HiSparse for Quest, 1.55× vs. Tangram for SnapKV). The evaluation also covers GQA and MLA models, 32K and 128K contexts, and agent workloads such as SWE-bench Lite.

It is nice to see the paper address agent-level serving problems, such as reusing compacted KV across turns, rather than only individual sparse attention operations.

Critical weaknesses
  1. Novelty vs. Sparse-vLLM

The novelty over Sparse-vLLM in DeltaKV is not clear to me. Sparse-vLLM already has a modular CacheManager, a Sparse Controller, and sparse KV lifecycle management, and SparseEngine seems to extend this design. It would help if the paper explicitly separated what is inherited from Sparse-vLLM and what is newly added, e.g., the finer-grained hooks, scheduling interfaces, and cross-request cache management.

  1. The effect of each component is not isolated

The contribution of Chain Cache is hard to tell. The 2.24× speedup in Table 6 is confounded with sparse KV retention and higher concurrency, so a w/ vs. w/o Chain Cache ablation with the same sparse method would help. The effect of Prefix-Cache Pruning is also unclear. Table 5 lacks a matched no-pruning baseline, and the lag=4 result uses different hardware and precision, so it is hard to separate the effect of pruning from the effect of the lag. The gains over specialized systems are interesting, but it is unclear where they come from, such as attention kernels, cache layouts, or scheduling.

Questions for authors
  1. What can SparseEngine support that Sparse-vLLM cannot?

  2. Could you provide w/ vs. w/o ablations for Chain Cache and Prefix-Cache Pruning under matched settings? Latency, memory, and quality would all be useful.

  3. Are the sparse configurations and kernels matched in the comparisons with Tangram, Vortex, and HiSparse? A latency breakdown would help.

6 comments
AuthorsOpenReview ↗

Response (1/3) to Reviewer SpyR

Thank you for your thorough review and for recognizing SparseEngine's clean architectural separation (lifecycle hooks, CacheManager, and AttentionView), strong method-matched throughput, and agent-level cross-turn KV reuse.

Below, we address each of your points with controlled experiments and updated analyses, all of which have been added to the revised PDF.

W1 & Q1: Novelty vs. Sparse-vLLM

Thank you for raising this comparison. SparseEngine represents a substantial architectural redesign and extension. While Sparse-vLLM serves as a basic experimental prototype, it lacks essential serving capabilities such as prefix caching and cross-turn state management, making it impractical for real-world deployment. Specifically, SparseEngine introduces four core architectural capabilities that go beyond Sparse-vLLM's design:

  1. Finer-Grained Lifecycle Hooks and Execution (Sec. 3.1): SparseEngine replaces coarse step-level callbacks with fine-grained hooks spanning prefill chunks, decode steps, and layer boundaries (before/after attention, on_layer_end, finish_step), paired with backend-agnostic AttentionView (supporting MLA and CUDA Graphs). This scales support to 15 methods across 4 families and enables cross-family composition (revised paper, Appendix B.3 and Tables 8–9).
  2. Cross-Request Cache Management (Sec. 3.2): Unlike Sparse-vLLM's single-request execution, SparseEngine introduces Chain Cache for cross-turn eviction reuse (2.1×–2.8× speedup, Tables R2–R4) and Prefix Cache Pruning (up to 4.7× speedup, Tables R5–R8).
  3. Method-Aware Scheduling Interfaces (App. B.1): SparseEngine introduces MemoryOracle and compatibility-grouped step reservations to expose multi-pool capacity constraints and temporary reconstruction budgets to the scheduler.
  4. Higher System Efficiency: Streamlined runtime design, optimized kernels, and more efficient cache and view management yield 1.11×–1.32× higher decode throughput than Sparse-vLLM under identical hardware (RTX PRO 6000) and settings from DeltaKV Table 4, as shown in Table R1.

Setup: Testing Qwen2.5-7B-Instruct-1M on a single PRO6000 GPU using BF16 weights and KV cache, with TP set to 1 and CUDA Graph enabled. Configurations include 128K context length with a batch size of 8, and 256K context length with a batch size of 4, both with a 30% sparsity budget. Data for Sparse-vLLM is referenced from Table 4 of the original PDF.

Table R1: Decoding throughput compared with Sparse-vLLM (RTX PRO 6000, settings from DeltaKV Table 4).

ContextBSMethodSparse-vLLM (tok/s)SparseEngine (tok/s)Speedup
128K8Dense135.0171.71.27×
128K8OmniKV216.7286.91.32×
128K8SnapKV338.8376.81.11×
256K4Dense69.585.91.24×
256K4OmniKV115.9142.81.23×
256K4SnapKV168.8188.21.11×

W2 & Q2: Effect of each component (Chain Cache, Prefix Cache Pruning)

Conclusion: We have added the key ablation results to the revised paper (Section 4.3, Tables 3 and 12, Figures 6–10). Under strictly matched settings, Chain Cache independently delivers 2.1×–2.8× end-to-end speedup and 30.2%–49.0% peak KV memory savings (Tables R2–R4), while Prefix Cache Pruning independently achieves up to 4.7× end-to-end speedup and 31.2× lower TTFT, with lag=4 better preserving the model's original task performance (Tables R5–R8).


Setup: We use 32 multi-turn SWE-agent trajectories (415K total output tokens) for efficiency experiments. Without Chain Cache, each turn re-prefills the conversation history and re-executes sparse eviction. With Chain Cache, retained KV entries and method metadata are reused across turns.

Results: Enabling Chain Cache reduces end-to-end latency by 2.1×–2.8× and mean TTFT by 2.1×–3.1× across concurrencies 8–24 (Table R2), cuts peak occupied KV cache memory by up to 49.0% (Table R3), and preserves task score (Table R4).

AuthorsOpenReview ↗

Response (2/3) to Reviewer SpyR

Table R2: Effect of Chain Cache on inference latency (Qwen3-30B-A3B-Instruct-2507-FP8, H100).

MethodConcurrencyTime w/o Chain Cache (s)Time w/ Chain Cache (s)E2E SpeedupMean TTFT w/o → w/ (s)TTFT Speedup
SnapKV82057.1914.42.3×5.7→2.52.3×
SnapKV161868.5708.72.6×9.1→3.22.8×
SnapKV241818.1653.92.8×12.0→3.93.1×
H₂O81910.1921.42.1×5.3→2.52.1×
H₂O161718.7726.02.4×8.5→3.32.5×
H₂O241671.5674.12.5×11.3→4.12.7×

Table R3: Effect of Chain Cache on peak occupied KV cache memory (Qwen3-30B-A3B-Instruct-2507-FP8, H100).

MethodConcurrencyPeak KV w/o Chain Cache (GiB)Peak KV w/ Chain Cache (GiB)Reduction
SnapKV812.96.649.0%
SnapKV2427.319.130.2%

Table R4: Effect of Chain Cache on task performance (Qwen3-30B-A3B-Instruct-2507-FP8, H100).

MethodScore w/o Chain CacheScore w/ Chain Cache
SnapKV7.38.3
H₂O6.07.0

Setup: Efficiency is evaluated via trajectory replay on AgentX (20 sessions, 1,108 requests, 1.1M output tokens, concurrency 20) and SWE-bench Lite (64 trajectories, 4,253 requests, concurrency 64). Task quality is evaluated on SWE-bench Lite under matched hardware and precision, triggering tool-result pruning when cumulative tool tokens reach 8K and compacting to 2K.

Results: Prefix Cache Pruning speeds up replay by 4.7× on AgentX and 1.7× on SWE-bench Lite, reducing mean TTFT by 31.2× and 14.1× and cutting preemption/recomputation events by 92% and 90% (Tables R5–R6), while reclaiming up to 8.8 GiB per pruning operation (Table R7).

On SWE-bench Lite (Table R8), immediate pruning (lag=0) drops resolve rates by 0.3–1.0 points, whereas lag=2/4 restores or slightly surpasses unpruned baselines (e.g., 26.7% vs. 26.0% for KVzip + OmniKV). Because agents actively consume fresh tool outputs over subsequent turns before distilling them, delaying pruning protects this active working window and reclaims only stale context noise, remaining orthogonal to sparse decoding.

Table R5: Effect of Prefix Cache Pruning on AgentX replay efficiency (GLM-4.7-Flash-FP8, H100).

Metricw/o Pruningw/ PruningImprovement
Replay Time (min)246.452.04.7×
Output Throughput (tokens/s)79.1374.64.7×
Mean TTFT (s)129.44.131.2×
Mean Request E2E Latency (s)200.828.87.0×
Preemptions / Recomputation Events25292% reduction

Table R6: Effect of Prefix Cache Pruning on SWE-lite replay efficiency (GLM-4.7-Flash-FP8, H100).

Metricw/o Pruningw/ PruningImprovement
Replay Time (min)81.949.71.7×
Output Throughput (tokens/s)204.3336.71.7×
Mean TTFT (s)20.11.414.1×
Mean Request E2E Latency (s)36.213.42.7×
Preemptions / Recomputation Events40490% reduction

Table R7: KV cache memory freed per pruning operation (GLM-4.7-Flash-FP8, H100).

StatisticKV Memory Freed (GiB)
Mean0.6
Median0.4
Maximum8.8
AuthorsOpenReview ↗

Response (3/3) to Reviewer SpyR

Table R8: Effect of pruning method, decoding method, token budget, and lag on SWE-bench Lite resolve rate (GLM-4.7-Flash-FP8, H100).

Pruning MethodDecoding MethodTrigger → BudgetLagResolve Rate (%)
NoneVanilla——25.0
NoneQuest——25.3
NoneOmniKV——26.0
SnapKVOmniKV8K → 2K025.3
SnapKVOmniKV8K → 2K226.0
SnapKVOmniKV8K → 2K426.3
KVzipVanilla8K → 2K225.3
KVzipQuest8K → 2K024.3
KVzipQuest8K → 2K225.7
KVzipQuest8K → 2K425.3
KVzipOmniKV8K → 2K025.7
KVzipOmniKV8K → 2K226.7
KVzipOmniKV8K → 2K426.7

W2 & Q3: Latency breakdown compared to Tangram, Vortex, and HiSparse

Conclusion: We have added the latency comparisons and breakdowns to the revised paper (Tables 13–14, Appendix C.9). Under matched sparse budgets and configurations, SparseEngine achieves 1.2×–1.6× lower per-step decoding latency than Vortex, HiSparse, and Tangram (Table R9). The latency breakdown (Table R10) is consistent with the benefit of our flexible lifecycle abstraction, which enables method-specific execution optimizations without imposing unnecessary overhead from a uniform computational workflow.

Setup: All systems are evaluated under identical hardware, model precision, and strictly matched sparse budgets (e.g., 2K active tokens for Quest and 8K retained tokens for SnapKV), with each baseline running its official default high-performance attention kernels (FlashAttention, FlashInfer, or Triton).

Note: Table R9 reports actual decoding latency measured without profiling, while Table R10 reports component-level timings from separate profiling runs. In Table R10, Other Compute includes non-attention model forward passes (FFN/MoE, projections, and norms) and framework overhead, explaining Tangram's higher latency in this category despite comparable attention execution times.

Table R9: Decoding latency compared with Vortex, HiSparse, and Tangram.

MethodConcurrencyBaselineSparseEngine (ms/step)Baseline (ms/step)Speedup
Quest2Vortex5.87.21.2×
Quest2HiSparse5.89.41.6×
SnapKV16Tangram10.015.01.5×

Table R10: Decoding latency breakdown (ms/step).

SystemMethodConcurrencyAttentionSparse SelectionKV Layout / MetadataOther Compute
SparseEngineQuest20.951.000.163.68
VortexQuest21.271.060.134.22
HiSparseQuest20.664.280.243.84
SparseEngineSnapKV165.02—0.073.83
TangramSnapKV165.13—0.129.24

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.