- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents AVSG (Accelerated Vectorized Sparse Gather), an operator designed to reduce KV-cache offloading overhead in long-context LLM serving with DeepSeek Sparse Attention (DSA). Although DSA reduces attention computation by selecting a subset of historical tokens, the full KV cache remains necessary, and transferring the selected entries from host memory can introduce substantial decoding latency.
To address this problem, AVSG combines vectorized hash matching for efficient KV reuse, balanced distribution of cache-miss transfers across NPU cores, lifetime-based cache management, and shared-buffer reuse across speculative decoding tokens. These techniques aim to reduce redundant host-to-device transfers without changing the tokens selected by the attention indexer.
The authors evaluate AVSG using operator-level microbenchmarks and an end-to-end serving system running GLM-5.3 on Ascend 910B NPUs. On a production-derived workload containing 2,000 chat requests, AVSG reportedly reduces average time per output token by approximately 39% and improves output throughput by 1.27× compared with a KV-offloading baseline without device-buffer reuse.
Critical strengths
1. The paper addresses a practical and relevant systems bottleneck.
KV-cache offloading is an important problem for serving long-context LLMs, especially when sparse attention reduces computation but still requires access to historical KV entries. The idea of exploiting repeated KV selections across decoding steps is reasonable and practically useful. I also appreciate that AVSG operates after token selection, so it does not require modifying the attention indexer or introducing an additional approximation to the selected token set.
2. The paper provides encouraging performance results in a realistic serving environment.
The evaluation goes beyond isolated kernel benchmarks and includes a multi-node serving setup with production-derived requests. The reported reductions in TPOT and improvements in output throughput are meaningful compared with the naive offloading baseline. These results suggest that the proposed optimizations have practical engineering value, even though stronger comparisons are still needed to establish the significance of the contribution.
Critical weaknesses
1. Limited novelty and insufficient comparison with existing methods.
My main concern is that the proposed approach appears to be largely a combination of existing systems techniques rather than a substantially new algorithmic idea. Vectorized hash lookup, cache-miss compaction and load balancing, and lifetime-based cache replacement are all fairly standard optimization strategies. Their combination may be useful in practice, but I do not find the paper's explanation of its technical novelty particularly convincing.
More importantly, the evaluation does not adequately establish how much AVSG improves upon existing optimized solutions. The paper discusses GatherSelection, ECHO, SparseServe, and other related approaches, but the end-to-end evaluation primarily compares AVSG against a KV-offloading implementation that transfers all selected entries without reuse. This is a relatively weak baseline for a method whose central motivation is exploiting KV reuse. In particular, GatherSelection is presented as the direct predecessor of AVSG, yet there is no full-system comparison against it.
The large H2D bandwidth improvements reported in Figure 4 also need to be interpreted carefully. For example, the headline 34.44× improvement is obtained against request-level assignment at batch size 1, where most accelerator cores are necessarily idle. While this demonstrates the value of finer-grained parallelism, it does not establish a similarly large advantage over a well-optimized competing implementation.
Overall, I am not convinced that the paper sufficiently distinguishes its contribution from existing work or demonstrates a clear performance advantage over strong, relevant baselines.
2. The evaluation is too narrow to support the broader significance of the proposed approach.
The experimental validation is heavily tied to a particular hardware and model configuration. All measurements are conducted on Ascend NPUs, and the end-to-end results are limited to one GLM-5.3 serving setup using DSA. The proposed implementation relies on Ascend-specific SIMD and DMA capabilities, but the paper provides little evidence that its benefits would carry over to other accelerator architectures, particularly NVIDIA GPUs.
The limitation is not only hardware-related. Although the general idea of reusing selected KV entries could potentially extend to other selection-based sparse-attention mechanisms, the paper evaluates only DSA. Given the increasing diversity of sparse-attention designs, additional validation on mechanisms such as Qwen Sparse Attention (QSA) or MiniMax Sparse Attention (MSA) would make the contribution considerably more convincing. For substantially different mechanisms, such as the compressed attention designs used in DeepSeek-V4, the paper should at least discuss which parts of AVSG remain applicable and which assumptions no longer hold.
I am also concerned that the end-to-end experiments do not adequately characterize the performance trade-offs. According to Table 2, AVSG increases decode HBM utilization from 91.6% to 95.5% compared with KV Offload, while average TTFT increases from 5.94 s to 8.69 s. The paper therefore needs a more careful discussion of memory overhead and the latency-throughput trade-off, preferably under matched resource budgets. Additionally, there is no complete end-to-end ablation showing the contribution of each optimization.
The results on one private production-derived workload are encouraging, but they are not sufficient to establish that AVSG consistently outperforms existing methods across different attention mechanisms, workloads, or hardware platforms. Considering the primarily engineering-oriented nature of the contribution, I believe broader and more rigorous experimental validation is particularly important.
Questions for authors
1. How does AVSG compare with stronger existing baselines in an end-to-end serving environment?
Since GatherSelection is the direct predecessor discussed in the paper, could the authors provide an end-to-end comparison against its original implementation under the same hardware and workload configurations? Comparisons with other relevant KV-offloading approaches, such as ECHO or SparseServe, would also be useful where technically feasible. Additionally, a component-wise end-to-end ablation would help clarify the individual contributions of vectorized matching, miss-only transfer balancing, lifetime-based caching, and MTP buffer sharing.
2. How generalizable is AVSG across hardware platforms and sparse-attention mechanisms?
Could the authors clarify which parts of AVSG depend specifically on Ascend NPU hardware and which could be implemented efficiently on GPUs? Beyond DSA, it would be valuable to demonstrate applicability to at least one other selection-based sparse-attention mechanism, such as QSA or MSA. For compressed or hybrid attention mechanisms such as CSA/HCA, a discussion of the compatibility limitations would help establish the scope of the proposed method.
3. What are the performance benefits under matched memory budgets and different serving workloads?
Table 2 shows that AVSG consumes additional HBM and has higher average TTFT than the KV Offload baseline, despite improving throughput and TPOT. Could the authors provide a more complete latency-throughput comparison under matched HBM budgets, different concurrency levels, and varying prompt lengths or cache-reuse distributions? It would also be helpful to report variability across repeated end-to-end runs.
4. Could the authors provide further clarification and validation of the hash-state correctness guarantees?
The implementation uses a bounded probing budget and allows nondeterministic hash-bucket write collisions. Although Appendix B explains why unsuccessful mapping installation should not lead to false cache hits, it would be useful to report collision rates, probe-exhaustion rates, and their effects on actual cache efficiency. I would also appreciate clarification of how the 8-bit lifetime counters handle negative values and potential underflow, together with correctness tests covering speculative-token rejection and cache replacement.
11 comments
Response to W2-1. End-to-End experiments
We added a matched end-to-end GS comparison in the same 5P1D GLM-5.3-w4a8c16 serving stack. The comparison uses the same 2,000-request corpus (48,196.7 average prompt tokens), concurrency 64, maximum output length 32,768, and MTP-3 setting as §4.5. Both reuse methods reserve 8,192 selected-KV slots per request: GS uses four independent 2,048-entry pools for the MTP tokens, whereas AVSG shares one 8,192-entry pool across those tokens. Their sampled decode HBM usage is correspondingly close (95.47% for GS and 95.57% for AVSG).
Here, AVSG denotes the original method reported in §4.5. AVSG+ is an enhanced variant that exploits GLM-5.3's shared indexer. A skip layer uses the same Top- indices as the preceding layer. In our cold-start setup, every layer's hash pool starts empty, with no retained frequent-entry records. AVSG+ reuses the preceding layer's hash-matching result, slot remapping, and miss-copy plan for the skip layer, while each layer still copies its own KV values for misses. We report this cross-layer matching enhancement separately from the original AVSG result.
Each method has four end-to-end measurements. Latency and throughput are across-run means ± sample standard deviations. Device KV capacity is fixed within each method across runs, decode HBM usage is reported as a mean, and TPS counts generated output tokens per second.
| Metric | No Offload | KV Offload | GS | AVSG | AVSG+ |
|---|---|---|---|---|---|
| Device KV capacity (tokens) | 145,536 | 762,496 | 762,496 | 762,496 | 762,496 |
| Decode HBM usage (%) | 91.78 | 91.61 | 95.47 | 95.57 | 95.59 |
| AVG TTFT (s) | 6.58 ± 0.99 | 5.57 ± 0.27 | 6.77 ± 0.58 | 7.74 ± 0.61 | 9.58 ± 0.79 |
| AVG TPOT, token 2+ (ms) | 42.50 ± 0.58 | 91.00 ± 1.41 | 62.75 ± 0.96 | 54.40 ± 0.55 | 45.25 ± 0.50 |
| P99 TPOT, token 2+ (ms) | 61.50 ± 2.08 | 128.50 ± 3.00 | 89.50 ± 3.11 | 76.60 ± 1.14 | 65.25 ± 2.63 |
| AVG E2E latency (s) | 116.13 ± 2.31 | 115.51 ± 5.86 | 85.43 ± 4.12 | 74.77 ± 1.58 | 65.95 ± 1.44 |
| TPS | 489.1 ± 14.3 | 441.5 ± 13.0 | 531.7 ± 21.1 | 593.9 ± 18.3 | 691.5 ± 36.4 |
| Achieved QPS (req/s) | 0.46 ± 0.02 | 0.40 ± 0.01 | 0.48 ± 0.02 | 0.54 ± 0.01 | 0.62 ± 0.03 |
Relative to GS, original AVSG reduces average and P99 steady-state TPOT by 13.3% and 14.4%, respectively, while increasing output throughput by 11.7% and achieved request rate by 13.2%. AVSG+ reaches 45.25 ms average TPOT, 65.25 ms P99 TPOT, and 691.5 output tok/s.
TTFT and throughput. As described in §4.5, our closed-loop benchmark evaluates sustained serving throughput. When the concurrency cap is active, faster decoding allows replacement requests to enter sooner. AVSG achieves 0.54 requests/s versus GS's 0.48 on the same workload, supporting increased prefill pressure from the approximately 48K-token prompts as an explanation for the higher TTFT (7.74 vs. 6.77 s). At this higher achieved request rate, AVSG increases output throughput by 11.7% and reduces E2E latency by 12.5%. TTFT should therefore be read alongside the throughput achieved at each operating point.
Response to W2-2. Breadth of the evaluation
Testing other model families and platforms. The repeated 5P1D comparison above shows AVSG's gain over GS in our GLM service. We currently lack the resources to build matched multi-node serving stacks and conduct full end-to-end tests for every model and platform. To assess AVSG across model families, attention types (DSA, QSA, and CSA/HCA), and NPU/GPU platforms, we added attention-layer measurements across three model families on NPU and GPU. Within each configuration, methods use the same pretrained weights, synthetic inputs, finalized Top-K IDs, approximate real-workload locality, with about 80% overlap between successive steps and page tables. All GPU comparisons use the same device (Due to regulations from our affiliation, we are unable to disclose specific GPU models). TPS counts verification tokens per second, timed from finalized IDs to attention output. Each step covers batch × (MTP + 1) tokens. ECHO adopts the default settings of the ECHO paper. Values are means ± sample SD across five seed means, with three repetitions per seed. A dash indicates an unmeasured combination.
| Device | Model | Layer Type | MTP | Context | Batch | No Offload | KV Offload | GS | ECHO | AVSG |
|---|---|---|---|---|---|---|---|---|---|---|
| NPU | GLM-5.3 | 1 DSA | 3 | 16,384 | 8 | 180,532.45 ± 129.29 | 9,970.38 ± 8.25 | 27,650.04 ± 78.52 | — | 52,548.78 ± 67.98 |
| NPU | GLM-5.3 | 1 DSA | 3 | 51,200 | 2 | 77,489.08 ± 95.66 | 9,651.01 ± 47.98 | 16,675.75 ± 217.44 | — | 17,757.69 ± 20.29 |
| NPU | GLM-5.3 | 1 DSA | 3 | 51,200 | 4 | 146,810.08 ± 452.31 | 9,845.70 ± 7.12 | 23,549.33 ± 91.35 | — | 32,257.10 ± 48.67 |
| NPU | GLM-5.3 | 1 DSA | 3 | 51,200 | 8 | 178,201.40 ± 1,392.51 | 9,970.90 ± 6.74 | 27,607.53 ± 42.85 | — | 49,464.31 ± 355.16 |
| NPU | GLM-5.3 | 1 DSA | 3 | 51,200 | 16 | 200,731.88 ± 162.91 | 10,030.20 ± 4.02 | 29,457.58 ± 57.48 | — | 67,753.20 ± 68.01 |
| NPU | GLM-5.3 | 1 DSA | 3 | 131,072 | 8 | 176,779.00 ± 186.62 | 9,970.63 ± 5.97 | 27,628.14 ± 53.97 | — | 48,903.45 ± 109.02 |
| NPU | DeepSeek-V4-Flash | 1 CSA + 1 HCA | 7 | 51,200 | 4 | 250,463.50 ± 822.64 | 39,860.54 ± 52.72 | 61,809.95 ± 145.71 | — | 65,014.68 ± 171.27 |
| NPU | Qwen3.8-Flash-Next | 1 QSA | 3 | 51,200 | 4 | 18,577.45 ± 61.42 | 4,447.40 ± 3.18 | 10,227.00 ± 24.06 | — | 12,189.93 ± 31.23 |
| GPU | Qwen3.8-Flash-Next | 1 QSA | 3 | 51,200 | 4 | 269,568.15 ± 2,367.86 | 9,900.51 ± 832.23 | — | — | 142,378.63 ± 3,215.39 |
| GPU | GLM-5.3 | 1 DSA | 3 | 51,200 | 4 | 142,723.33 ± 688.20 | 9,706.46 ± 556.69 | — | 65,112.88 ± 607.51 | 103,050.36 ± 1,506.59 |
| GPU | GLM-5.3 | 1 DSA + 1 shared-index layer | 3 | 51,200 | 4 | 76,267.77 ± 308.20 | 4,733.73 ± 263.52 | — | 32,549.60 ± 275.58 | 62,206.03 ± 512.07 |
At context 51,200 and batch 4 on NPU platform, AVSG achieved 1.37× the throughput of GS for GLM DSA, 1.19× for Qwen QSA and 1.05× for the DeepSeek CSA+HCA pair. On the GPU platform, AVSG achieved 14.38× the throughput of direct host reads in Qwen. In GLM, AVSG achieved 1.58× and 1.91× the throughput of ECHO on one layer and two layers sharing indices, respectively.
AVSG is lossless with respect to native sparse attention. Holding the selection and KV values fixed lets us measure how much sooner the same attention computation finishes, including overlap with KV transfers. Time saved on the decode critical path can reduce full-model TPOT. The GLM layer and service results show gains in the same direction.
We thank the reviewer for the detailed feedback. Below, we clarify AVSG's contribution and report additional comparisons and analyses.
Response to W1. Contribution and strength of the baseline
AVSG introduces an exact post-selection gather algorithm for offloaded sparse attention. It replaces GS's repeated sort–merge matching with bounded vectorized hash probing over a persistent, lifetime-managed KV pool. Explicit slot remapping allows the pool to grow beyond the current Top-K selection and retain entries across decoding steps; shared slot reservations extend reuse across MTP tokens. Only misses require H2D transfers, which AVSG distributes evenly across accelerator cores. The indexer's selection remains unchanged.
The table below compares AVSG with existing KV-reuse and offloading methods, including FreeKV, LiteCache, and ECHO.
Relative to GS, AVSG uses a different post-selection gather design. GS does not depend on SIMT; its AscendC implementation uses vector sorting followed by scalar dual-pointer matching. AVSG instead vectorizes the matching hot path with bounded hash probes on SIMD-oriented AICores.
Coordinated contribution under a fixed KV budget. Vectorized matching keeps lookup cost low as the shared pool grows. Lifetime-based residency retains frequently selected KV in that pool. Miss-balanced copy tiling distributes the remaining transfers across AICores, while sharing the pool across MTP tokens avoids redundant KV residency and improves capacity efficiency for concurrency. These components reinforce one another; our contribution is their integrated design for DSA/(other dynamic sparse attention) offloading, rather than any one primitive in isolation.
Relative to ECHO, ECHO's published latency-hiding path fuses predictive prefetch into a CUDA indexer kernel using warp specialization and TMA. Porting that fused pipeline to SIMD-oriented hardware without SIMT warps would require redesign. AVSG consumes finalized Top-K IDs through an independent gather operator. Its matching hot path uses bounded SIMD hash probing without SIMT warp primitives, and its per-ID probe logic has a natural lane-per-ID mapping to SIMT.