W1. Limited Algorithmic Novelty and Insufficient Differentiation from Existing Methods
The central contribution of StrideKV is to replace per-token historical KV retrieval with a fixed periodic retrieval schedule. Specifically, the method retrieves KV entries every s decoding steps and reuses the previously selected GPU-resident working set during intermediate steps. While this design is simple and practically motivated, the algorithmic contribution appears rather limited.
The general idea of exploiting temporal stability in KV selection and reusing selected entries across decoding steps has already been explored in prior work, including LServe, TokenSelect, and RefreshKV. Although StrideKV applies periodic reuse to a CPU-offloading setting and emphasizes batch-level synchronization, the proposed method largely amounts to a scheduling modification of the existing ShadowKV framework rather than a fundamentally new retrieval or attention mechanism.
The paper does not sufficiently demonstrate why the proposed fixed-stride strategy constitutes a substantial technical advancement over existing reuse-based approaches. More direct comparisons with these methods, or adaptations of their scheduling strategies to the same KV-offloading backend, would help establish the novelty of StrideKV.
W2. Insufficient Technical Depth and Justification of the Fixed Retrieval Schedule
The technical mechanism underlying StrideKV is relatively straightforward. Its core operation is to perform retrieval when tmods=0 and otherwise reuse the previous KV working set. The method does not introduce a new KV representation, retrieval algorithm, or attention computation mechanism.
Although the authors identify synchronization problems in adaptive retrieval under synchronous batched decoding, the proposed solution is simply to enforce a shared retrieval schedule across all requests. The paper does not provide a sufficiently detailed investigation of alternative scheduling strategies or demonstrate that this simple periodic policy is particularly well suited to heterogeneous workloads.
In particular, a fixed retrieval interval does not account for changes in query relevance. It may perform unnecessary retrievals when the selected KV set remains stable, while delaying retrieval when the query changes substantially. The paper does not sufficiently analyze these limitations or compare against stronger batch-aware adaptive scheduling strategies that could potentially reduce synchronization overhead without relying on a fixed interval.
Consequently, the technical contribution appears closer to an empirical scheduling heuristic than a sufficiently developed algorithmic framework.
W3. Limited Experimental Generalization and Incomplete Baseline Comparisons
The experimental evaluation is relatively narrow and does not sufficiently establish the generality of the proposed approach.
First, the model coverage is limited. The accuracy experiments primarily use Llama and GLM models, while the throughput experiments are restricted to Llama-3-8B-1M and GLM-4-9B-1M. The absence of evaluations on other widely used model families, particularly Qwen, makes it difficult to determine whether the reported benefits generalize across different architectures and attention configurations. The evaluation also does not establish scalability to substantially larger models.
Second, the throughput comparisons primarily focus on ShadowKV and a ShadowKV-Adaptive variant constructed by the authors. Although FreeKV is included in the accuracy evaluation, the proposed adaptive baseline is not the complete FreeKV system. Therefore, the results do not convincingly establish the advantage of StrideKV over more comprehensively optimized adaptive retrieval implementations.
Third, the performance evaluation is conducted on a single A100 GPU and mainly reports aggregate decoding throughput. The paper provides limited evidence concerning heterogeneous serving workloads, asynchronous scheduling, per-request latency, and the behavior of the proposed strategy under different hardware and memory-transfer configurations.
Finally, the accuracy evaluation is concentrated on RULER. While average accuracy degradation is relatively small for s=2, several individual tasks exhibit more noticeable differences. The paper does not systematically investigate scenarios involving abrupt changes in retrieval requirements or longer generation trajectories, where reusing outdated KV selections could have a greater impact.
Overall, more comprehensive comparisons across model families, workloads, retrieval policies, and serving configurations are needed to support the general applicability of StrideKV.
W4. Presentation and Experimental Visualization Could Be Improved
Although the overall organization is understandable, the presentation of several experimental results could be clearer.
Some figures contain dense annotations and relatively small labels, making detailed comparisons unnecessarily difficult. In particular, Figures 1, 4, and 5 would benefit from clearer visual organization and more readable labeling.
Furthermore, Figure 4 reports speedups over full attention using different feasible batch sizes, while comparisons against ShadowKV use matched batch sizes. Although the paper explains this distinction, the presentation could more clearly separate throughput improvements resulting from larger feasible batches and KV offloading from those specifically attributable to periodic retrieval reuse.
A more systematic visualization of the accuracy–throughput tradeoff, alongside a clearer decomposition of retrieval and computation costs, would make the empirical contributions easier to assess.
We sincerely appreciate the reviewer’s time and feedback.Below, we address each question individually:
Q1: Can throughput be measured during the actual evaluation?
A: We thank the reviewer for suggesting this measurement. We measured decoding throughput on 128K RULER QA2 using the same greedy decoding and stopping rules as in the accuracy evaluation. Each prompt contained 131,040 tokens, and generation stopped at EOS or after 32 output tokens. All three methods ran on the same 80 GB A100 using Gradient Llama-3-8B-Instruct-1048k in BF16. ShadowKV and StrideKV used the same retrieval settings, with a budget of 2,048 tokens, a rank of 160, and a chunk size of 8 tokens.
We measured one batch per method. ShadowKV and StrideKV processed the same 24 questions. Due to GPU memory constraints, we ran full attention at B=3 using the first three of those questions. Timing began when the first output token was ready and ended after the final GPU synchronization. We calculated decoding throughput by dividing the number of valid tokens generated during that interval by the elapsed batch decoding time.
At the matched batch size of B=24, StrideKV achieved 87.0% higher decoding throughput than ShadowKV on this QA2 workload.
Q2: Decoding throughput comparison with FreeKV.
A: We thank the reviewer for pointing out this missing comparison. We measured the decoding throughput of StrideKV (s=2) and the complete FreeKV system on the same 80 GB A100. Both used Llama-3-8B-Instruct-Gradient-1048k, the same 48 distinct requests with 61,440 input tokens each, and a 1,024-token dynamic KV retrieval budget. We recorded one timed run per method after warm-up.
Under these matched conditions, StrideKV achieved 4.32× the decoding throughput of FreeKV.
Q3: Accuracy–throughput tradeoff.
A: We thank the reviewer for suggesting a single plot. We have added Figure 8 to Appendix C (“Accuracy–Throughput Tradeoff,” page 19). The plot shows mean decoding throughput on the x-axis and mean 128K RULER accuracy on the y-axis, with one point each for ShadowKV, StrideKV (s=2), and StrideKV (s=3).
Both axes use the two models shared by Table 4 and Figure 5. We weight the six model–context configurations equally for throughput and the two models equally for accuracy. Compared with ShadowKV, StrideKV (s=2) improves mean decoding throughput by 54.0% with a 0.255 percentage-point decrease in mean accuracy. Increasing the stride to 3 raises mean throughput by a further 24.0% and decreases mean accuracy by another 0.750 percentage points.
Q4: Methodological novelty.
A: We appreciate the reviewer’s comment. Cross-step KV reuse is an established technique. Our analysis shows why it can yield limited throughput gains at high concurrency. In a synchronous batch, even when most requests can reuse their cached KV, one request that needs a refresh can make the entire batch wait.
StrideKV makes reuse a batch-wide decision by scheduling refreshes on the same steps for all requests. With s=2, every request attends to its own cached historical KV on alternating steps, removing both selection and loading from the batch’s critical path on those steps.
In the paired experiment with 32 distinct requests, this schedule increased decoding throughput by 49.25% over ShadowKV. In a separate 128K RULER evaluation on Llama-3-8B-1M, the two methods achieved comparable mean accuracy (86.05% for StrideKV and 85.76% for ShadowKV).