acceptodds
Under review as a conference paper at ICLR 2027

AVSG: Accelerated Vectorized Sparse Gather for Efficient KV Cache Offload in Sparse-Attention LLM Serving

Abstract

Dynamic sparse attention reduces long-context attention computation by selecting only a subset of tokens, but still requires access to the full key-value (KV) cache, leaving serving memory-bound. Offloading the KV cache to host memory reduces device memory pressure but places host-to-device (H2D) transfers on the decoding critical path. In DeepSeek Sparse Attention (DSA), substantial overlap in selected KV entries across decoding steps creates an opportunity for device-resident reuse. Exploiting this reuse efficiently, however, presents three critical challenges: (1) costly matching of selected tokens against entries retained in the HBM buffer, (2) uneven distribution of the remaining H2D transfers across accelerator cores, and (3) retaining frequently accessed KV entries within limited HBM capacity. We present AVSG (Accelerated Vectorized Sparse Gather), an operator that addresses these challenges for DSA while preserving its exact token selections. AVSG uses vectorized hash matching to identify reusable HBM-buffer slots and reserve slots for entries requiring transfer, then evenly partitions these H2D transfers across accelerator cores. Lifetime-based buffer management retains frequently accessed KV entries in the device buffer across decoding steps, and shared slot reservations extend reuse across tokens within a multi-token prediction iteration. On a single NPU, vectorized hash matching is faster than scalar dual-pointer matching, miss-only transfer raises effective H2D bandwidth by up to over request-level assignment, and an 8K-entry buffer reaches a % hit rate on real requests. These gains translate into end-to-end improvement: on a serving stack processing a real-world production dataset, AVSG reduces time per output token by % and increases output throughput by relative to the same KV offload layout without HBM-buffer reuse, demonstrating the benefit of efficient matching, balanced transfer, and effective HBM residency in production-scale serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model Inference, Sparse Attention, KV Cache Offloading, Vectorized Hash Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_thcYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AVSG (Accelerated Vectorized Sparse Gather), an operator designed to reduce KV-cache offloading overhead in long-context LLM serving with DeepSeek Sparse Attention (DSA). Although DSA reduces attention computation by selecting a subset of historical tokens, the full KV cache remains necessary, and transferring the selected entries from host memory can introduce substantial decoding latency.

To address this problem, AVSG combines vectorized hash matching for efficient KV reuse, balanced distribution of cache-miss transfers across NPU cores, lifetime-based cache management, and shared-buffer reuse across speculative decoding tokens. These techniques aim to reduce redundant host-to-device transfers without changing the tokens selected by the attention indexer.

The authors evaluate AVSG using operator-level microbenchmarks and an end-to-end serving system running GLM-5.3 on Ascend 910B NPUs. On a production-derived workload containing 2,000 chat requests, AVSG reportedly reduces average time per output token by approximately 39% and improves output throughput by 1.27× compared with a KV-offloading baseline without device-buffer reuse.

Critical strengths

1. The paper addresses a practical and relevant systems bottleneck.

KV-cache offloading is an important problem for serving long-context LLMs, especially when sparse attention reduces computation but still requires access to historical KV entries. The idea of exploiting repeated KV selections across decoding steps is reasonable and practically useful. I also appreciate that AVSG operates after token selection, so it does not require modifying the attention indexer or introducing an additional approximation to the selected token set.

2. The paper provides encouraging performance results in a realistic serving environment.

The evaluation goes beyond isolated kernel benchmarks and includes a multi-node serving setup with production-derived requests. The reported reductions in TPOT and improvements in output throughput are meaningful compared with the naive offloading baseline. These results suggest that the proposed optimizations have practical engineering value, even though stronger comparisons are still needed to establish the significance of the contribution.

Critical weaknesses

1. Limited novelty and insufficient comparison with existing methods.

My main concern is that the proposed approach appears to be largely a combination of existing systems techniques rather than a substantially new algorithmic idea. Vectorized hash lookup, cache-miss compaction and load balancing, and lifetime-based cache replacement are all fairly standard optimization strategies. Their combination may be useful in practice, but I do not find the paper's explanation of its technical novelty particularly convincing.

More importantly, the evaluation does not adequately establish how much AVSG improves upon existing optimized solutions. The paper discusses GatherSelection, ECHO, SparseServe, and other related approaches, but the end-to-end evaluation primarily compares AVSG against a KV-offloading implementation that transfers all selected entries without reuse. This is a relatively weak baseline for a method whose central motivation is exploiting KV reuse. In particular, GatherSelection is presented as the direct predecessor of AVSG, yet there is no full-system comparison against it.

The large H2D bandwidth improvements reported in Figure 4 also need to be interpreted carefully. For example, the headline 34.44× improvement is obtained against request-level assignment at batch size 1, where most accelerator cores are necessarily idle. While this demonstrates the value of finer-grained parallelism, it does not establish a similarly large advantage over a well-optimized competing implementation.

Overall, I am not convinced that the paper sufficiently distinguishes its contribution from existing work or demonstrates a clear performance advantage over strong, relevant baselines.

2. The evaluation is too narrow to support the broader significance of the proposed approach.

The experimental validation is heavily tied to a particular hardware and model configuration. All measurements are conducted on Ascend NPUs, and the end-to-end results are limited to one GLM-5.3 serving setup using DSA. The proposed implementation relies on Ascend-specific SIMD and DMA capabilities, but the paper provides little evidence that its benefits would carry over to other accelerator architectures, particularly NVIDIA GPUs.

The limitation is not only hardware-related. Although the general idea of reusing selected KV entries could potentially extend to other selection-based sparse-attention mechanisms, the paper evaluates only DSA. Given the increasing diversity of sparse-attention designs, additional validation on mechanisms such as Qwen Sparse Attention (QSA) or MiniMax Sparse Attention (MSA) would make the contribution considerably more convincing. For substantially different mechanisms, such as the compressed attention designs used in DeepSeek-V4, the paper should at least discuss which parts of AVSG remain applicable and which assumptions no longer hold.

I am also concerned that the end-to-end experiments do not adequately characterize the performance trade-offs. According to Table 2, AVSG increases decode HBM utilization from 91.6% to 95.5% compared with KV Offload, while average TTFT increases from 5.94 s to 8.69 s. The paper therefore needs a more careful discussion of memory overhead and the latency-throughput trade-off, preferably under matched resource budgets. Additionally, there is no complete end-to-end ablation showing the contribution of each optimization.

The results on one private production-derived workload are encouraging, but they are not sufficient to establish that AVSG consistently outperforms existing methods across different attention mechanisms, workloads, or hardware platforms. Considering the primarily engineering-oriented nature of the contribution, I believe broader and more rigorous experimental validation is particularly important.

Questions for authors

1. How does AVSG compare with stronger existing baselines in an end-to-end serving environment?

Since GatherSelection is the direct predecessor discussed in the paper, could the authors provide an end-to-end comparison against its original implementation under the same hardware and workload configurations? Comparisons with other relevant KV-offloading approaches, such as ECHO or SparseServe, would also be useful where technically feasible. Additionally, a component-wise end-to-end ablation would help clarify the individual contributions of vectorized matching, miss-only transfer balancing, lifetime-based caching, and MTP buffer sharing.

2. How generalizable is AVSG across hardware platforms and sparse-attention mechanisms?

Could the authors clarify which parts of AVSG depend specifically on Ascend NPU hardware and which could be implemented efficiently on GPUs? Beyond DSA, it would be valuable to demonstrate applicability to at least one other selection-based sparse-attention mechanism, such as QSA or MSA. For compressed or hybrid attention mechanisms such as CSA/HCA, a discussion of the compatibility limitations would help establish the scope of the proposed method.

3. What are the performance benefits under matched memory budgets and different serving workloads?

Table 2 shows that AVSG consumes additional HBM and has higher average TTFT than the KV Offload baseline, despite improving throughput and TPOT. Could the authors provide a more complete latency-throughput comparison under matched HBM budgets, different concurrency levels, and varying prompt lengths or cache-reuse distributions? It would also be helpful to report variability across repeated end-to-end runs.

4. Could the authors provide further clarification and validation of the hash-state correctness guarantees?

The implementation uses a bounded probing budget and allows nondeterministic hash-bucket write collisions. Although Appendix B explains why unsuccessful mapping installation should not lead to false cache hits, it would be useful to report collision rates, probe-exhaustion rates, and their effects on actual cache efficiency. I would also appreciate clarification of how the 8-bit lifetime counters handle negative values and potential underflow, together with correctness tests covering speculative-token rejection and cache replacement.

11 comments

We thank the reviewer for the detailed feedback. Below, we clarify AVSG's contribution and report additional comparisons and analyses.

Response to W1. Contribution and strength of the baseline

AVSG introduces an exact post-selection gather algorithm for offloaded sparse attention. It replaces GS's repeated sort–merge matching with bounded vectorized hash probing over a persistent, lifetime-managed KV pool. Explicit slot remapping allows the pool to grow beyond the current Top-KK selection and retain entries across decoding steps; shared slot reservations extend reuse across MTP tokens. Only misses require H2D transfers, which AVSG distributes evenly across accelerator cores. The indexer's selection remains unchanged.

The table below compares AVSG with existing KV-reuse and offloading methods, including FreeKV, LiteCache, and ECHO.

MethodHow KV rows are suppliedEffect on the current query's selection
FreeKVReuses pages recalled at the preceding step. A KV head triggers correction when its group's mean query similarity falls below the threshold.Approximate. An uncorrected head can use the preceding query's selection.
LiteCacheReuses the cached Top-KK when query similarity exceeds the threshold. The reference query is refreshed on a cache miss.Approximate. The reused set can differ from a fresh selection.
ECHOPrefetches during indexing, then recalls entries still missing from the final Top-KK.Lossless relative to native sparse attention. Attention reads the final selection even if prefetch predictions miss.
AVSGMatches finalized Top-KK IDs against a shared resident pool and copies every miss before attention.Lossless relative to native sparse attention. Reuse changes where each selected row is read.

Relative to GS, AVSG uses a different post-selection gather design. GS does not depend on SIMT; its AscendC implementation uses vector sorting followed by scalar dual-pointer matching. AVSG instead vectorizes the matching hot path with bounded hash probes on SIMD-oriented AICores.

Coordinated contribution under a fixed KV budget. Vectorized matching keeps lookup cost low as the shared pool grows. Lifetime-based residency retains frequently selected KV in that pool. Miss-balanced copy tiling distributes the remaining transfers across AICores, while sharing the pool across MTP tokens avoids redundant KV residency and improves capacity efficiency for concurrency. These components reinforce one another; our contribution is their integrated design for DSA/(other dynamic sparse attention) offloading, rather than any one primitive in isolation.

Relative to ECHO, ECHO's published latency-hiding path fuses predictive prefetch into a CUDA indexer kernel using warp specialization and TMA. Porting that fused pipeline to SIMD-oriented hardware without SIMT warps would require redesign. AVSG consumes finalized Top-KK IDs through an independent gather operator. Its matching hot path uses bounded SIMD hash probing without SIMT warp primitives, and its per-ID probe logic has a natural lane-per-ID mapping to SIMT.

Response to W2-1. End-to-End experiments

We added a matched end-to-end GS comparison in the same 5P1D GLM-5.3-w4a8c16 serving stack. The comparison uses the same 2,000-request corpus (48,196.7 average prompt tokens), concurrency 64, maximum output length 32,768, and MTP-3 setting as §4.5. Both reuse methods reserve 8,192 selected-KV slots per request: GS uses four independent 2,048-entry pools for the MTP tokens, whereas AVSG shares one 8,192-entry pool across those tokens. Their sampled decode HBM usage is correspondingly close (95.47% for GS and 95.57% for AVSG).

Here, AVSG denotes the original method reported in §4.5. AVSG+ is an enhanced variant that exploits GLM-5.3's shared indexer. A skip layer uses the same Top-KK indices as the preceding layer. In our cold-start setup, every layer's hash pool starts empty, with no retained frequent-entry records. AVSG+ reuses the preceding layer's hash-matching result, slot remapping, and miss-copy plan for the skip layer, while each layer still copies its own KV values for misses. We report this cross-layer matching enhancement separately from the original AVSG result.

Each method has four end-to-end measurements. Latency and throughput are across-run means ± sample standard deviations. Device KV capacity is fixed within each method across runs, decode HBM usage is reported as a mean, and TPS counts generated output tokens per second.

MetricNo OffloadKV OffloadGSAVSGAVSG+
Device KV capacity (tokens)145,536762,496762,496762,496762,496
Decode HBM usage (%)91.7891.6195.4795.5795.59
AVG TTFT (s)6.58 ± 0.995.57 ± 0.276.77 ± 0.587.74 ± 0.619.58 ± 0.79
AVG TPOT, token 2+ (ms)42.50 ± 0.5891.00 ± 1.4162.75 ± 0.9654.40 ± 0.5545.25 ± 0.50
P99 TPOT, token 2+ (ms)61.50 ± 2.08128.50 ± 3.0089.50 ± 3.1176.60 ± 1.1465.25 ± 2.63
AVG E2E latency (s)116.13 ± 2.31115.51 ± 5.8685.43 ± 4.1274.77 ± 1.5865.95 ± 1.44
TPS489.1 ± 14.3441.5 ± 13.0531.7 ± 21.1593.9 ± 18.3691.5 ± 36.4
Achieved QPS (req/s)0.46 ± 0.020.40 ± 0.010.48 ± 0.020.54 ± 0.010.62 ± 0.03

Relative to GS, original AVSG reduces average and P99 steady-state TPOT by 13.3% and 14.4%, respectively, while increasing output throughput by 11.7% and achieved request rate by 13.2%. AVSG+ reaches 45.25 ms average TPOT, 65.25 ms P99 TPOT, and 691.5 output tok/s.

TTFT and throughput. As described in §4.5, our closed-loop benchmark evaluates sustained serving throughput. When the concurrency cap is active, faster decoding allows replacement requests to enter sooner. AVSG achieves 0.54 requests/s versus GS's 0.48 on the same workload, supporting increased prefill pressure from the approximately 48K-token prompts as an explanation for the higher TTFT (7.74 vs. 6.77 s). At this higher achieved request rate, AVSG increases output throughput by 11.7% and reduces E2E latency by 12.5%. TTFT should therefore be read alongside the throughput achieved at each operating point.

Response to W2-2. Breadth of the evaluation

Testing other model families and platforms. The repeated 5P1D comparison above shows AVSG's gain over GS in our GLM service. We currently lack the resources to build matched multi-node serving stacks and conduct full end-to-end tests for every model and platform. To assess AVSG across model families, attention types (DSA, QSA, and CSA/HCA), and NPU/GPU platforms, we added attention-layer measurements across three model families on NPU and GPU. Within each configuration, methods use the same pretrained weights, synthetic inputs, finalized Top-K IDs, approximate real-workload locality, with about 80% overlap between successive steps and page tables. All GPU comparisons use the same device (Due to regulations from our affiliation, we are unable to disclose specific GPU models). TPS counts verification tokens per second, timed from finalized IDs to attention output. Each step covers batch × (MTP + 1) tokens. ECHO adopts the default settings of the ECHO paper. Values are means ± sample SD across five seed means, with three repetitions per seed. A dash indicates an unmeasured combination.

DeviceModelLayer TypeMTPContextBatchNo OffloadKV OffloadGSECHOAVSG
NPUGLM-5.31 DSA316,3848180,532.45 ± 129.299,970.38 ± 8.2527,650.04 ± 78.52—52,548.78 ± 67.98
NPUGLM-5.31 DSA351,200277,489.08 ± 95.669,651.01 ± 47.9816,675.75 ± 217.44—17,757.69 ± 20.29
NPUGLM-5.31 DSA351,2004146,810.08 ± 452.319,845.70 ± 7.1223,549.33 ± 91.35—32,257.10 ± 48.67
NPUGLM-5.31 DSA351,2008178,201.40 ± 1,392.519,970.90 ± 6.7427,607.53 ± 42.85—49,464.31 ± 355.16
NPUGLM-5.31 DSA351,20016200,731.88 ± 162.9110,030.20 ± 4.0229,457.58 ± 57.48—67,753.20 ± 68.01
NPUGLM-5.31 DSA3131,0728176,779.00 ± 186.629,970.63 ± 5.9727,628.14 ± 53.97—48,903.45 ± 109.02
NPUDeepSeek-V4-Flash1 CSA + 1 HCA751,2004250,463.50 ± 822.6439,860.54 ± 52.7261,809.95 ± 145.71—65,014.68 ± 171.27
NPUQwen3.8-Flash-Next1 QSA351,200418,577.45 ± 61.424,447.40 ± 3.1810,227.00 ± 24.06—12,189.93 ± 31.23
GPUQwen3.8-Flash-Next1 QSA351,2004269,568.15 ± 2,367.869,900.51 ± 832.23——142,378.63 ± 3,215.39
GPUGLM-5.31 DSA351,2004142,723.33 ± 688.209,706.46 ± 556.69—65,112.88 ± 607.51103,050.36 ± 1,506.59
GPUGLM-5.31 DSA + 1 shared-index layer351,200476,267.77 ± 308.204,733.73 ± 263.52—32,549.60 ± 275.5862,206.03 ± 512.07

At context 51,200 and batch 4 on NPU platform, AVSG achieved 1.37× the throughput of GS for GLM DSA, 1.19× for Qwen QSA and 1.05× for the DeepSeek CSA+HCA pair. On the GPU platform, AVSG achieved 14.38× the throughput of direct host reads in Qwen. In GLM, AVSG achieved 1.58× and 1.91× the throughput of ECHO on one layer and two layers sharing indices, respectively.

AVSG is lossless with respect to native sparse attention. Holding the selection and KV values fixed lets us measure how much sooner the same attention computation finishes, including overlap with KV transfers. Time saved on the decode critical path can reduce full-model TPOT. The GLM layer and service results show gains in the same direction.

Reviewer_iStVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposers AVSG which is a gather operator for DSA serving with host-offloaded KV. It keeps selected KV entries in an HBM buffer and transfers only misses, using vectorized hash matching, miss-only load balancing across AICores, lifetime-based retention, and a buffer shared across MTP tokens. On a 5P1D GLM-5.3 deployment, it cuts TPOT by 39% and raises throughput 1.27× over offloading without reuse.

Critical strengths
  1. The paper addresses an important problem of H2D on the decode critical path, without changing token selection.
  2. The proposed method is hardware-aware which includes correctly handling MTP sharing and speculative invalidation, making it useful in practical scenario.
  3. The empirical results are promising, showing substantial throughput and time reduction over baseline method.
Critical weaknesses
  1. Cross-step KV reuse already applied in several methods such as FreeKV, LiteCache. The components are standard techniques, the authors need to clarify the novelty over previous methods.
  2. The end-to-end baseline is offloading without reuse, so the gain over prior reuse methods is not shown. Also, the end-to-end evaluation is weak, only one model, one workload and no repeated runs.
  3. The paper is not very easy to follow. Although the overall structure is clear, Section 3.2-3.3 present too many bit-level encoding details, and some terms are not consistent.
Questions for authors

How does AVSG compare end to end with GatherSelection on the same stack?

4 comments

Thank you for the constructive review. We address your three weaknesses and end-to-end comparison question below.

W1. Novelty beyond prior cross-step KV reuse

We acknowledge that cross-step KV reuse has been explored in prior work. Our contribution is a coordinated gather design that makes exact KV reuse efficient under a fixed device-memory budget. Vectorized matching identifies which selected Top-KK IDs are already in the device cache. Hits reuse their existing KV slots, while misses are assigned available slots and their H2D transfers are evenly distributed across accelerator cores. The shared buffer retains useful KV entries across decoding steps and MTP tokens. Like GatherSelection (GS), AVSG operates on DSA’s finalized Top-KK IDs without changing the model, indexer, or selected-token set.

The table below compares AVSG with existing KV-reuse and offloading methods, including FreeKV, LiteCache, and ECHO.

MethodHow KV rows are suppliedEffect on the current query's selection
FreeKVReuses pages recalled at the preceding step. A KV head triggers correction when its group's mean query similarity falls below the threshold.Approximate. An uncorrected head can use the preceding query's selection.
LiteCacheReuses the cached Top-KK when query similarity exceeds the threshold. The reference query is refreshed on a cache miss.Approximate. The reused set can differ from a fresh selection.
ECHOPrefetches during indexing, then recalls entries still missing from the final Top-KK.Lossless relative to native sparse attention. Attention reads the final selection even if prefetch predictions miss.
AVSGMatches finalized Top-KK IDs against a shared resident pool and copies every miss before attention.Lossless relative to native sparse attention. Reuse changes where each selected row is read.

Relative to GS, AVSG uses a different post-selection gather design. GS does not depend on SIMT; its AscendC implementation uses vector sorting followed by scalar dual-pointer matching. AVSG instead vectorizes the matching hot path with bounded hash probes on SIMD-oriented AICores.

Coordinated contribution under a fixed KV budget. Vectorized matching keeps lookup cost low as the shared pool grows. Lifetime-based residency retains frequently selected KV in that pool. Miss-balanced copy tiling distributes the remaining transfers across AICores, while sharing the pool across MTP tokens avoids redundant KV residency and improves capacity efficiency for concurrency. These components reinforce one another; our contribution is their integrated design for DSA offloading, rather than any one primitive in isolation.

Relative to ECHO, ECHO's published latency-hiding path fuses predictive prefetch into a CUDA indexer kernel using warp specialization and TMA. Porting that fused pipeline to SIMD-oriented hardware without SIMT warps would require redesign. AVSG consumes finalized Top-KK IDs through an independent gather operator. Its matching hot path uses bounded SIMD hash probing without SIMT warp primitives, and its per-ID probe logic has a natural lane-per-ID mapping to SIMT.

W2-1. Stronger end-to-end baseline, repeated runs

We agree that the original comparison did not isolate AVSG's incremental gain over an existing reuse method. We integrated the legacy GS operator into the same 5P1D GLM-5.3-w4a8c16 serving stack. The comparison uses the same 2,000-request corpus (48,196.7 average prompt tokens), concurrency 64, maximum output length 32,768, and MTP-3 setting as §4.5. Both reuse methods reserve 8,192 selected-KV slots per request: GS uses four independent 2,048-entry pools for the MTP tokens, whereas AVSG shares one 8,192-entry pool across those tokens. Their sampled decode HBM usage is correspondingly close (95.47% for GS and 95.57% for AVSG).

Here, AVSG denotes the original method reported in §4.5. AVSG+ is an enhanced variant that exploits GLM-5.3's shared indexer. A skip layer uses the same Top-KK indices as the preceding layer. In our cold-start setup, every layer's hash pool starts empty, with no retained frequent-entry records. AVSG+ reuses the preceding layer's hash-matching result, slot remapping, and miss-copy plan for the skip layer, while each layer still copies its own KV values for misses. We report this cross-layer matching enhancement separately from the original AVSG result.

Each method has four end-to-end measurements. Latency and throughput are across-run means ± sample standard deviations. Device KV capacity is fixed, decode HBM usage is reported as a mean, and TPS counts generated output tokens per second.

MetricNo OffloadKV OffloadGSAVSGAVSG+
Device KV capacity (tokens)145,536762,496762,496762,496762,496
Decode HBM usage (%)91.7891.6195.4795.5795.59
AVG TTFT (s)6.58 ± 0.995.57 ± 0.276.77 ± 0.587.74 ± 0.619.58 ± 0.79
AVG TPOT, token 2+ (ms)42.50 ± 0.5891.00 ± 1.4162.75 ± 0.9654.40 ± 0.5545.25 ± 0.50
P99 TPOT, token 2+ (ms)61.50 ± 2.08128.50 ± 3.0089.50 ± 3.1176.60 ± 1.1465.25 ± 2.63
AVG E2E latency (s)116.13 ± 2.31115.51 ± 5.8685.43 ± 4.1274.77 ± 1.5865.95 ± 1.44
TPS489.1 ± 14.3441.5 ± 13.0531.7 ± 21.1593.9 ± 18.3691.5 ± 36.4
Achieved QPS (req/s)0.46 ± 0.020.40 ± 0.010.48 ± 0.020.54 ± 0.010.62 ± 0.03

Relative to GS, original AVSG reduces average and P99 steady-state TPOT by 13.3% and 14.4%, respectively, while increasing output throughput by 11.7% and achieved request rate by 13.2%. AVSG+ reaches 45.25 ms average TPOT, 65.25 ms P99 TPOT, and 691.5 output tok/s.

W2-2. Generalization

Testing other model families and platforms. The repeated 5P1D comparison above shows AVSG's gain over GS in our GLM service. We currently lack the resources to build matched multi-node serving stacks and conduct full end-to-end tests for every model and platform. To assess AVSG across model families, attention types (DSA, QSA, and CSA/HCA), and NPU/GPU platforms, we added attention-layer measurements across three model families on NPU and GPU. Within each configuration, methods use the same pretrained weights, synthetic inputs, finalized Top-K IDs, approximate real-workload locality, with about 80% overlap between successive steps and page tables. All GPU comparisons use the same device (Due to regulations from our affiliation, we are unable to disclose specific GPU models). TPS counts verification tokens per second, timed from finalized IDs to attention output. Each step covers batch × (MTP + 1) tokens. ECHO adopts the default settings of the ECHO paper. Values are means ± sample SD across five seed means, with three repetitions per seed. A dash indicates an unmeasured combination.

DeviceModelLayer TypeMTPContextBatchNo OffloadKV OffloadGSECHOAVSG
NPUGLM-5.31 DSA316,3848180,532.45 ± 129.299,970.38 ± 8.2527,650.04 ± 78.52—52,548.78 ± 67.98
NPUGLM-5.31 DSA351,200277,489.08 ± 95.669,651.01 ± 47.9816,675.75 ± 217.44—17,757.69 ± 20.29
NPUGLM-5.31 DSA351,2004146,810.08 ± 452.319,845.70 ± 7.1223,549.33 ± 91.35—32,257.10 ± 48.67
NPUGLM-5.31 DSA351,2008178,201.40 ± 1,392.519,970.90 ± 6.7427,607.53 ± 42.85—49,464.31 ± 355.16
NPUGLM-5.31 DSA351,20016200,731.88 ± 162.9110,030.20 ± 4.0229,457.58 ± 57.48—67,753.20 ± 68.01
NPUGLM-5.31 DSA3131,0728176,779.00 ± 186.629,970.63 ± 5.9727,628.14 ± 53.97—48,903.45 ± 109.02
NPUDeepSeek-V4-Flash1 CSA + 1 HCA751,2004250,463.50 ± 822.6439,860.54 ± 52.7261,809.95 ± 145.71—65,014.68 ± 171.27
NPUQwen3.8-Flash-Next1 QSA351,200418,577.45 ± 61.424,447.40 ± 3.1810,227.00 ± 24.06—12,189.93 ± 31.23
GPUQwen3.8-Flash-Next1 QSA351,2004269,568.15 ± 2,367.869,900.51 ± 832.23——142,378.63 ± 3,215.39
GPUGLM-5.31 DSA351,2004142,723.33 ± 688.209,706.46 ± 556.69—65,112.88 ± 607.51103,050.36 ± 1,506.59
GPUGLM-5.31 DSA + 1 shared-index layer351,200476,267.77 ± 308.204,733.73 ± 263.52—32,549.60 ± 275.5862,206.03 ± 512.07

At context 51,200 and batch 4 on NPU platform, AVSG achieved 1.37× the throughput of GS for GLM DSA, 1.19× for Qwen QSA and 1.05× for the DeepSeek CSA+HCA pair. On the GPU platform, AVSG achieved 14.38× the throughput of direct host reads in Qwen. In GLM, AVSG achieved 1.58× and 1.91× the throughput of ECHO on one layer and two layers sharing indices, respectively.

AVSG is lossless with respect to native sparse attention. Holding the selection and KV values fixed lets us measure how much sooner the same attention computation finishes, including overlap with KV transfers. Time saved on the decode critical path can reduce full-model TPOT. The GLM layer and service results show gains in the same direction.

W3. Presentation of Sections 3.2–3.3 and terminology

Thank you for pointing this out. We will make the algorithmic data flow explicit before introducing the packed-state representation.

We will revise these sections to first present the overall algorithm more intuitively. Matching the selected Top-K IDs against the device cache identifies which entries are hits and which are misses. Hits reuse their existing KV slots, while misses are assigned new slots for H2D transfer. We will move the bit-level encoding, field layouts, and associated arithmetic to the appendix.

We will also standardize terminology throughout, particularly the distinction between hash buckets and KV buffer slots, and correct the identified inconsistencies and typos.

Q1. How does AVSG compare end to end with GatherSelection on the same stack?

The W2-1 table above reports four end-to-end measurements per method on the same 5P1D GLM-5.3 stack and 2,000-request trace, with the same 8,192-slot device KV capacity for GS and AVSG. Original AVSG reduces average token-2+ TPOT from 62.75 to 54.40 ms (13.3%) and raises output throughput from 531.7 to 593.9 tok/s (11.7%) relative to GS. The table also reports AVSG+ separately.

I appreciate the authors' efforts in reporting the new experimental results. I am inclined to keep my original score, though I would have no objection if the AC recommends this paper for acceptance.

Reviewer_HvrkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the decoding-time cost of KV-cache offloading under DeepSeek Sparse Attention (DSA), where the indexer selects a top-KK subset of tokens per step but the full KV history must remain addressable, forcing host-to-device (H2D) transfers onto the critical path. The authors observe that consecutive decoding steps select heavily overlapping token sets (81.61% adjacent-step overlap on their trace) and propose AVSG, a post-selection gather operator that keeps a device-resident buffer of SS slots and transfers only the misses, without altering the indexer or its selections. AVSG has four components: a vectorized open-addressing hash table in on-chip memory that classifies each selected index as hit/miss and allocates destination slots in three SIMD passes (§3.2–3.3); an even partition of the resulting miss list across AI-Cores for H2D DMA (§3.4); a per-slot lifetime counter with global decay and hit-triggered refresh that retains frequently reselected entries (§3.5); and a buffer shared across the tokens of one multi-token-prediction (MTP) iteration, with end-of-step invalidation of speculative entries (§3.6). Validation is split between single-NPU component studies — matching latency vs. a scalar dual-pointer baseline (Figure 3), effective H2D bandwidth vs. request-level and equal-top-KK core binding (Figure 4, Table 3), and buffer hit rates on real and synthetic traces (Tables 1, 4) — and one end-to-end run on a 12-node Ascend 910B 5P1D stack serving GLM-5.3-w4a8c16 over 2000 production agent requests (Table 2). The headline end-to-end result is a 39% TPOT reduction (91 → 55 ms) and 1.27× output throughput relative to the same offload layout without buffer reuse. Appendices A–H give the lifetime ablation, consolidated algorithms with a slot-preservation argument, profiling protocols, full bandwidth tables, the synthetic reuse sweep, and the serving configuration.

Critical strengths

Each claimed bottleneck is isolated and measured, with unusually transparent protocols. Sec. 4.2 fixes the working set so both matchers face identical conditions and reports median-of-1000 with an explicit note that median, trimmed mean and mean agree within 2%. Appendix D defines all three core-binding strategies, gives the bandwidth metric as V/t with V held equal across strategies (so the ratios are transfer-time ratios), and reports the full 24-cell table including the cases where AVSG loses (S3/S2 = 0.96-0.98x at zero reuse). Appendix E states outright that the Table 1 synthetic rows are gain-maximizing configurations rather than averages. This makes the component-level claims checkable rather than merely asserted, which is where most of the paper’s evidential weight sits.

The scaling property the design actually depends on is demonstrated directly. Figure 3(b,c) shows an 8x increase in K at fixed S costs 1.27x latency and a 16x increase in S at fixed K costs 1.60x, against a scalar baseline whose cost grows 7.2x over an 8x working set and which structurally admits only S = K. This decoupling is the precondition for the paper’s central move, namely enlarging the buffer beyond the adjacent-step working set to 8K entries to raise the hit rate from 81.61% to 94.83%; without it the extra hit rate would be paid for in matching time. Appendix C is appropriately explicit that the matched S = K = N protocol is dictated by the baseline’s limitation rather than chosen to favour AVSG.

The end-to-end evaluation is at production scale with the confounders controlled where it matters: a 12-node disaggregated 5P1D deployment, 2000 real agent requests with 48K average prompts, MTP-3, and a memory layout verified from server logs rather than assumed (Appendix G). AVSG and KV Offload share an identical device pool and host DRAM layout, so the gap between those two columns isolates the access strategy. Operator-level papers rarely carry their claims this far, and the fact that the microbenchmark gains survive integration is the paper’s strongest evidence that the mechanisms matter.

Critical weaknesses

The end-to-end comparison omits the one baseline the paper is positioned against. The entire framing (Figure 1(3), the GatherSelection paragraph in Sec. 2, the footnote to the public operator, the scalar dual-pointer baseline in Sec. 4.2, and the GatherSelection line in Figure 5a) says the state of practice already performs device-resident reuse via sort-merge matching and already attains 81.61% reuse. Table 2 nevertheless compares AVSG only against No Offload and KV Offload, neither of which reuses anything. Transfer volume drops from 100% to roughly 18.4% under the prior method and to roughly 5.2% under AVSG, so the 39% TPOT and 1.27x throughput figures measure “reuse versus no reuse”, which is not in dispute, rather than the delta attributable to vectorized matching, miss-level core balancing and lifetime aging. This is the most consequential gap because it is exactly what determines whether the paper’s own contributions pay off at the system level, and it looks feasible to close given that Sec. 4.2 already runs the scalar operator in the authors’ harness.

The claim of preserving exact token selections is argued but never verified. Exactness is load-bearing, since it is what separates AVSG from the speculative-correction methods cited in Sec. 2, yet no output-equivalence or accuracy measurement appears anywhere. The implementation has several places where the argument is subtle: the packed scatter in Algorithm 2 line 10 explicitly retains one nondeterministic writer for same-bucket writes; stale bucket-to-slot pointers are reclaimed lazily (Eq. 4); and the speculative invalidation in Sec. 3.6 sets c_{o_i} to -1 while leaving the slot’s lifetime at L. The safety argument (full 32-bit index comparison prevents false hits) is sound as far as it goes, but the slot-preservation guarantee in Appendix B rests on every allocated slot carrying lifetime L, and one experiment, a bitwise output comparison against the KV Offload column on the same 2000 requests, would settle the question at negligible cost.

The headline component numbers are best-case cells, and the deployed operating point is outside the measured range. The abstract and contributions lead with 34.44x, which Table 3 shows is B = 1 against one-core-per-request binding, a number explained almost entirely by core count; at B = 32 the same comparison gives 1.22-1.30x, and against the stronger equal-top-K baseline AVSG is 0.96-0.98x at zero reuse. Table 1’s synthetic rows are, by Appendix E’s own admission, the mu_hist that maximizes the gain: at gamma = 0.5 the reported +44.15 pp becomes +18.19 pp at mu_hist = 0.5 (Table 4). Separately, Figure 4 and Table 3 stop at 75% reuse while Sec. 4.5 runs at roughly 94.8% hit rate, so the regime that actually produces the end-to-end result is never measured in Sec. 4.3. None of this is hidden, and the appendices are candid, but the framing systematically overstates what the data support, and recalibrating it would change how a reader weighs the contribution.

Table 2 is a single uncharacterized run with missing configuration and at least one internal inconsistency. No repeats, seeds or variance are reported for the headline result. The buffer size S used in Sec. 4.5 is never stated, despite Eq. 7 requiring S >= N_q K = 8192 under MTP-3 and despite S being the main knob trading HBM for hit rate. The stated cost of about 4 GB more HBM does not follow from the table’s own numbers: 95.5% minus 91.6% of 64 GB per chip is about 2.5 GB. And the No Offload row does not reconcile internally: TTFT 7.92 s plus 43 ms times roughly 1074 output tokens (472.6/0.44) gives about 54 s against a reported AVG E2E of 119.1 s, a 65 s gap that presumably reflects admission queueing excluded from TTFT but included in E2E, which is never defined. Since Table 2 carries the paper’s main claim, these matter.

Questions for authors

Can you add a GatherSelection column (or an equivalent scalar-matching, adjacent-step-reuse configuration) to Table 2 on the same stack and trace?

Have you verified that AVSG’s generated tokens are identical to the KV Offload configuration, or measured accuracy on any benchmark? A positive result on the Sec. 4.5 trace would turn the exactness claim from an argument into evidence and would raise my soundness assessment; its absence is the main reason soundness is not a 3.

In Algorithm 2, can a slot in o_miss coincide with a probe bucket in h[v]? If so, does the packed write at line 10 overwrite the lifetime L installed at line 8 with the value read earlier, and could that slot then be returned by CollectDead for a later MTP token while an earlier token’s output vector o still points at it? This bears directly on the preservation guarantee in Appendix B and hence on soundness; either a statement that the gather at line 10 reads post-line-8 state, or an aliasing argument, would resolve it.

3 comments

Thank you for the detailed review and constructive questions. We address each weakness and question below.

W1. Missing GatherSelection baseline in the end-to-end comparison

We agree that the original comparison did not isolate AVSG's incremental gain over an existing reuse method. We integrated the legacy GS operator into the same 5P1D GLM-5.3-w4a8c16 serving stack. The comparison uses the same 2,000-request corpus (48,196.7 average prompt tokens), concurrency 64, maximum output length 32,768, and MTP-3 setting as §4.5. Both reuse methods reserve 8,192 selected-KV slots per request: GS uses four independent 2,048-entry pools for the MTP tokens, whereas AVSG shares one 8,192-entry pool across those tokens. Their sampled decode HBM usage is correspondingly close (95.47% for GS and 95.57% for AVSG).

Here, AVSG denotes the original method reported in §4.5. AVSG+ is an enhanced variant that exploits GLM-5.3's shared indexer. A skip layer uses the same Top-KK indices as the preceding layer. In our cold-start setup, every layer's hash pool starts empty, with no retained frequent-entry records. AVSG+ reuses the preceding layer's hash-matching result, slot remapping, and miss-copy plan for the skip layer, while each layer still copies its own KV values for misses. We report this cross-layer matching enhancement separately from the original AVSG result.

Each method has four end-to-end measurements. Latency and throughput are across-run means ± sample standard deviations. Device KV capacity is fixed, decode HBM usage is reported as a mean, and TPS counts generated output tokens per second.

MetricNo OffloadKV OffloadGSAVSGAVSG+
Device KV capacity (tokens)145,536762,496762,496762,496762,496
Decode HBM usage (%)91.7891.6195.4795.5795.59
AVG TTFT (s)6.58 ± 0.995.57 ± 0.276.77 ± 0.587.74 ± 0.619.58 ± 0.79
AVG TPOT, token 2+ (ms)42.50 ± 0.5891.00 ± 1.4162.75 ± 0.9654.40 ± 0.5545.25 ± 0.50
P99 TPOT, token 2+ (ms)61.50 ± 2.08128.50 ± 3.0089.50 ± 3.1176.60 ± 1.1465.25 ± 2.63
AVG E2E latency (s)116.13 ± 2.31115.51 ± 5.8685.43 ± 4.1274.77 ± 1.5865.95 ± 1.44
TPS489.1 ± 14.3441.5 ± 13.0531.7 ± 21.1593.9 ± 18.3691.5 ± 36.4
Achieved QPS (req/s)0.46 ± 0.020.40 ± 0.010.48 ± 0.020.54 ± 0.010.62 ± 0.03

Relative to GS, original AVSG reduces average and P99 steady-state TPOT by 13.3% and 14.4%, respectively, while increasing output throughput by 11.7% and achieved request rate by 13.2%. AVSG+ reaches 45.25 ms average TPOT, 65.25 ms P99 TPOT, and 691.5 output tok/s.

W2. Exactness and cache-state correctness

Our exactness reference is the same DSA model under No Offload or KV Offload. AVSG remaps a selected ID xix_i to an HBM slot oio_i but does not change the KV data associated with that ID. As specified by Eq. (10) in the main paper, a lookup counts as a hit only if the ID in the probed HBM slot exactly matches xix_i. If a prior hash mapping was overwritten by a collision or could not be installed within the probe budget, Algorithm 1 reports mi=0m_i=0. It then assigns a reclaimable empty slot returned by CollectDead to oio_i. Stage Two copies the KV for xix_i from host memory into oio_i before attention reads it. Lost mappings therefore add transfers, not incorrect KV substitutions.

For same-bucket writes in Algorithm 2, the packed scatter retains one writer's complete fingerprint and destination slot, rather than mixing fields from different writers. Every miss keeps its allocated slot and receives the selected KV for the current attention operation. A mapping that loses the write may cause a later miss and another host transfer, but the full-ID comparison in Eq. (10) prevents it from producing a false hit.

For the aliasing case in Q3, suppose a newly assigned slot oio_i and a hash bucket hjh_j share the table index kk, so oi=hj=ko_i=h_j=k. The lifetime assignment in Algorithm 2 line 8 sets qk.lifetime=Lq_k.\mathrm{lifetime}=L before line 10 executes. Since hj=kh_j=k, line 10 reads qhj.lifetime=qk.lifetime=Lq_{h_j}.\mathrm{lifetime}=q_k.\mathrm{lifetime}=L when packing the bucket word.

After all MTP tokens in a step have been served, AVSG sets cj=−1c_j=-1 for Hash-State entries whose stored ID cj≥Nc_j\geq N refers to a rejected speculative position, where NN is the verified context length. A slot may still have lifetime LL, but its invalidated ID cannot pass the next step's full-ID comparison. If that ID is selected again, it is treated as a miss and its current KV is fetched from host memory; the old slot is reclaimed after its lifetime becomes negative.

Empirical checks. We evaluated the 1,319-question GSM8K benchmark on the 5P1D service three times per method, with temperature 0 and up to 256 concurrent requests:

MetricNo OffloadKV OffloadGSAVSGAVSG+
Acc (%)97.14 ± 0.2397.25 ± 0.0997.27 ± 0.2097.02 ± 0.0497.17 ± 0.31

We also evaluated two API endpoints with Terminal-Bench 2.1:

API endpointPass@1 (%)
GLM Official Coding Plan API82.75
Our 5P1D AVSG API server82.02
W3. Best-case component numbers and the deployed operating point

We agree that the component maxima need their conditions stated. The 34.44× result is at B=1B=1 and 25% reuse against request-level binding (S1); at B=32B=32, the same comparison is 1.22–1.30×. Low batch sizes matter for our target workload: long inputs consume substantial KV capacity, and our 5P1D deployment caps the decode batch at four. Keeping the transfer engines busy at B=1B=1–44 was therefore a practical serving concern, while 34.44× remains a conditional component result.

At zero reuse, miss compaction incurs a small cost: AVSG's effective bandwidth is 0.96–0.98× that of equal-Top-KK partitioning (S2) in the cited cells. Zero reuse can occur at cold start; the measured 2,000-request trace instead has a 94.83% buffer hit rate. To cover this high-reuse regime, we extended Table 3 at every original batch size to 90% and 95% reuse, using the same effective-bandwidth columns and speedup definitions:

Batch sizeReuse (%)S1 (GB/s)S2 (GB/s)S3 (GB/s)S3/S1S3/S2
1902.609410.647121.36668.19×2.01×
1952.63515.302512.14254.61×2.29×
2904.920010.588633.23196.75×3.14×
2955.14125.282921.09534.10×3.99×
49010.528010.511752.39974.98×4.98×
49510.18069.022235.65513.50×3.95×
89019.849618.292171.46673.60×3.91×
89520.218117.684953.01192.62×3.00×
169042.051435.709784.72392.01×2.37×
169540.556932.619168.65401.69×2.10×
329081.006763.273193.27931.15×1.47×
329577.943354.253182.86951.06×1.53×

We will label the selection of gain-maximizing synthetic configurations explicitly and qualify the abstract and contributions with the batch size, reuse rate, and comparator behind the 34.44× headline, alongside representative higher-batch and high-reuse results.

W4. Table 2 repeats, buffer size, HBM usage, and E2E definition

The W1 table now reports repeated end-to-end measurements as means and sample standard deviations. In §4.5, the AVSG buffer size was S=8192S=8192, satisfying Eq. (7) for MTP-3. The maximum decode batch size was four. Across 78 KV layers, each slot holds 512 KV and 64 RoPE elements in BF16, so the configured full-batch buffer payload is 4×78×8192×(512+64)×2=2,944,401,4084\times78\times8192\times(512+64)\times2=2{,}944{,}401{,}408 bytes (2.94 GB, or 2.74 GiB). Table 2 reports a sampled per-chip decode HBM average, which rises from 91.6% to 95.5%, or 3.9 percentage points. The original “~4 GB more HBM” statement was a typo; it should say approximately 4% more HBM utilization.

AVG E2E measures time from request submission through completion and includes any wait to enter the decode running queue when device KV capacity is insufficient. Achieved QPS counts completed requests per benchmark wall time, so such waiting can reduce it. The token-2+ TPOT reported above excludes the first-to-second-token interval, which can contain decode admission waiting. Across the latest three valid runs per method, the mean first-to-second interval and directly measured mean output length, with sample standard deviations, were:

MethodFirst-to-second interval (s)Output length (tokens)
No Offload61.05 ± 1.901071.9 ± 12.5
KV Offload6.16 ± 0.091086.3 ± 63.0
GS5.62 ± 0.151099.2 ± 67.5
AVSG4.49 ± 0.161111.3 ± 39.1
AVSG+3.63 ± 0.071113.4 ± 40.7

Your estimate of 472.6/0.44≈1074472.6/0.44\approx1074 output tokens per request, obtained by dividing Output TPS by achieved QPS, agrees with our directly measured No Offload output length above. At the request level, E2E comprises TTFT, the first-to-second-token interval, and all later token intervals. The measured 61.05 s first-to-second interval in No Offload accounts for most of the apparent 65 s gap you identified in the original Table 2.

Q1. Can you add GatherSelection to Table 2 on the same stack and trace?

Yes. The W1 comparison above includes a GS column on the same 5P1D GLM-5.3 serving stack and 2,000-request corpus. GS and AVSG have the same device KV capacity. Original AVSG improves average TPOT by 13.3% and output TPS by 11.7% relative to GS.

Q2. Have you verified identical generated tokens or measured benchmark accuracy?

We measured benchmark accuracy or pass rate. The W2 results above show three GSM8K runs per method on the 5P1D service, with AVSG at 97.02 ± 0.04% and KV Offload at 97.25 ± 0.09%. Terminal-Bench 2.1 gave 82.75% for the GLM Official Coding Plan API and 82.02% for the 5P1D AVSG API.

Q3. Can a miss slot alias a probe bucket, and could it be reclaimed prematurely?

Yes, their indices can coincide. If oi=hj=ko_i=h_j=k, Algorithm 2 line 8 sets qk.lifetime=Lq_k.\mathrm{lifetime}=L, and line 10 reads that updated value when packing the bucket word. It therefore does not restore an earlier lifetime.

For MTP, Algorithm 1 line 3 first decrements every slot's lifetime when g=0g=0. Token 0 then refreshes its hits and initializes newly allocated output slots to LL; slots unused by token 0 remain at most L−1L-1. Each later global decay in line 13 decrements both groups equally, preserving this gap, while newly allocated output slots for later tokens are initialized to LL. Thus, slots not yet used by an MTP token reach a negative lifetime before any earlier token's output slot does. By Eq. (7), S≥NqKS\geq N_qK. After token 0 occupies at most ∣o∣=K|\mathbf{o}|=K slots, at least S−∣o∣≥NqK−∣o∣S-|\mathbf{o}|\geq N_qK-|\mathbf{o}| slots remain for later tokens. CollectDead (Eq. (12)) can allocate these unused slots even if all MTP tokens' Top-KK sets are disjoint. The same reasoning applies after each token, so no earlier token's output slot is reallocated within the step.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for their constructive feedback. Below, we summarize AVSG's methodological contribution and explain how the additional experimental evidence strengthens the paper.

Core contribution

We identify three key challenges in turning KV reuse into efficient sparse-attention serving: matching overhead, fragmented transfers, and effective HBM residency. These challenges motivate AVSG, an exact sparse-gather algorithm that maps selected IDs to reusable slots, balances the remaining transfers, and maintains shared KV state across decoding steps and MTP tokens.

  1. Matching overhead. AVSG uses bounded vectorized hash probing and a three-pass refresh–classify–allocate algorithm to map selected IDs to persistent KV slots. Full-ID validation preserves exact retrieval, while pool capacity can grow independently of Top-K size to capture longer-range reuse.

  2. Fragmented transfers. Cache-hit filtering leaves scattered misses with unequal counts across requests and selected-position partitions. AVSG constructs the transfer schedule from these actual misses, bounding the workload difference between cores by one KV entry. This directly addresses the residual imbalance after reuse.

  3. Effective HBM residency. Lifetime refresh and aging retain useful entries across steps and reclaim inactive slots while protecting the current working set. MTP tokens share both retained entries and pending reservations, allowing later tokens to reuse an earlier destination before its transfer occurs. Speculative-entry invalidation preserves correctness across steps.

These designs advance the compared methods in concrete ways. Vectorized matching replaces GS's sort-and-scalar-merge procedure, achieving 2.80× faster matching at K = S = 2,048. Miss-based partitioning achieves 3.50×/3.95× the effective bandwidth of request-level/equal-Top-K assignment at batch 4 and 95% reuse. In the matched-budget MTP-3 comparison, AVSG makes one 8,192-slot pool available across four tokens, whereas GS uses four independent 2,048-slot pools. The end-to-end GS comparison at the same total KV-slot budget yields 11.7% higher throughput and 13.3% lower TPOT. We have also expanded the comparison with prior methods in Sec. 2 of the revised manuscript.

Evidence and clarifications provided during rebuttal

End-to-end comparison with a stronger baseline and repeated runs (Hvrk W1/Q1/W4; iStV Q1; thcY W2/Q1). We added GatherSelection (GS) to the same 5P1D GLM-5.3 serving stack and 2,000-request workload. With each method measured four times and GS and AVSG assigned the same total selected-KV slot capacity, original AVSG increases output throughput from 531.7 to 593.9 tokens/s (11.7%) and reduces token-2+ TPOT from 62.75 to 54.40 ms (13.3%) relative to GS.

Generalization across models, attention types, and hardware (iStV W2; thcY W2/Q2). We added module-level comparisons spanning GLM DSA, Qwen QSA, and DeepSeek CSA+HCA under different MTP settings. At a 51,200-token context and batch 4 on NPU, AVSG achieves 1.37×, 1.19×, and 1.05× GS throughput, respectively. On GPU, the GLM attention-layer tests achieve 1.58× and 1.91× ECHO throughput for one DSA and one DSA plus one indices share DSA layer, respectively. The Qwen QSA test achieves 14.38× the throughput of direct host loading.

Moreover, we added performance results under matched memory budgets (thcY Q3), a component ablation study (Hvrk W3; thcY Q1), exactness analysis (Hvrk W2/Q2/Q3; thcY Q4), and probe-budget results (thcY Q4) to address the reviewers' questions and concerns. AVSG's core method remains consistent with the submitted work. We have incorporated the stronger-baseline comparison and repeated measurements into Sec. 4.5 and added the remaining experiments to Appendices I–L. Many of the issues raised during review have already been addressed in the public OpenReview discussion board, where we presented our solutions and the corresponding experimental results. In the final version, we will incorporate the key clarifications and cite the relevant discussion threads for supporting details, keeping changes to the submitted manuscript focused and limited in scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.