- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SurroKV, a training-free method for improving non-prefix KV cache reuse by modifying the offline cache construction process. The authors observe that conventional position-independent caching typically constructs reusable document chunks in isolation, leading to mismatches between their cached representations and those obtained from full-context prefilling.
To address this issue, SurroKV prepends a fixed, request-independent surrogate context when constructing each reusable chunk's KV cache. The surrogate context is discarded afterward, and only the chunk's KV states are stored. During online inference, these caches are assembled according to the actual context, and selected tokens are recomputed using query-guided attention scores.
The authors further analyze the impact of cache construction on attention routing and provide theoretical conditions under which surrogate conditioning can reduce the mismatch relative to full prefilling. Experiments on RULER, LongBench, and HELMET across three language models demonstrate improvements over several existing cache-reuse baselines, particularly under limited recomputation budgets, without introducing additional online inference latency.
Critical strengths
S1. Simple and effective design with relatively comprehensive experimental validation.
The proposed method is simple and training-free, requiring only a modification to the offline cache construction process without increasing the stored KV cache size or changing the online recomputation pipeline. The authors provide relatively comprehensive experiments across multiple models, benchmarks, and recomputation budgets. The consistent improvements, particularly under limited recomputation budgets, and the compatibility with several existing recomputation strategies support the effectiveness of the proposed approach.
Critical weaknesses
W1. Limited methodological novelty.
The core idea of constructing reusable KV caches with a shared prefix has already been explored in prior work, particularly CacheClip. Although SurroKV differs by discarding the surrogate context entirely after offline construction, the paper does not sufficiently establish the independent contribution of this design choice. The proposed method appears to be a relatively incremental modification of existing cache construction strategies, and a more direct comparison with closely related approaches would be helpful.
W2. Limited evaluation of architectural generality and practical serving benefits.
All the experiments in this paper are restricted to conventional attention architectures, such as Llama-3.1 and Qwen2.5, without considering more recent hybrid linear/full-attention or native sparse-attention models (these architectural advances may reduce the marginal benefits of conventional KV cache reuse or introduce additional cache compatibility challenges). Moreover, the efficiency evaluation is limited to single-request inference using Hugging Face Transformers and PyTorch SDPA, without integration into production-oriented serving engines. Consequently, the applicability of the method to modern LLM architectures and its practical benefits under realistic serving workloads remain insufficiently demonstrated.
Questions for authors
Q1. Novelty relative to prior work.
Could the authors clarify the key methodological differences between SurroKV and CacheClip, particularly regarding their shared-prefix cache construction strategies? A controlled comparison using the same online recomputation method and budget, while varying whether the shared prefix is retained or discarded, would help establish the independent contribution of SurroKV.
Q2. Applicability to more recent attention architectures.
Could the authors discuss whether SurroKV can be directly applied to more recent architectures, such as hybrid linear/full-attention models (e.g., Qwen3.5) and native sparse-attention models (e.g., DeepSeek-V3.2 or above)? In particular, would the surrogate-conditioned KV states remain reusable when attention routing or recurrent-state dependencies differ from those of conventional causal attention?
Q3. Practical serving performance.
Could the authors provide further evidence of SurroKV's practical benefits when integrated into realistic serving systems, such as vLLM or SGLang? How would the method affect end-to-end TTFT, request throughput, and latency under continuous batching and concurrent requests, considering cache transfer and offline construction overhead? This would be very helpful in assessing the method's potential for real-world deployment.
4 comments
Response 2/2
Q2: Hybrid and native sparse-attention models
Response: Thank you for the question. SurroKV works with both architectures: it conditions HYPIC's composable cached states for the hybrid model and the per-token KV for the sparse model, and the conditioned caches remain reusable and improve reuse accuracy on both.
Hybrid linear/full attention: Qwen3.5-35B-A3B. We use HYPIC's open-source SGLang implementation (arXiv:2607.01299), which caches composable linear-layer transitions and full-attention KV. SurroKV conditions both on the surrogate, then discards it while preserving composability. Both methods recompute a fixed fraction of each chunk's initial tokens, so absolute scores are not comparable to our main table.
Table: Hybrid Qwen3.5-35B-A3B. RULER / LongBench by recomputation budget.
| Construction | 5% | 10% | 15% |
|---|---|---|---|
| HYPIC | 36.1 / 16.4 | 37.0 / 16.6 | 38.8 / 17.8 |
| SurroKV (1024) | 54.5 / 44.7 | 57.4 / 45.1 | 59.1 / 46.5 |
Native sparse attention: MiniCPM4-8B. DeepSeek-V3.2 (671B) exceeds our compute budget, so we use MiniCPM4-8B, which also adopts native sparse attention (InfLLM v2) at a scale we can run. Its per-token KV remains reusable. We keep its sparse attention active throughout, so the results reflect the sparse model itself (details in revised Appendix A.15). ProphetKV and SurroKV use 10% recomputation, and CacheClip uses a nominal 15% (about 11.6% actual at 8K). LongBench is evaluated at its original lengths. SurroKV is the best at every length, and its gain over all baselines grows with context length; it also leads at the other budgets (revised Appendix A.15).
Table: Native sparse-attention MiniCPM4-8B. 10% recomputation.
| Method | RULER 8K | RULER 16K | RULER 32K | LongBench |
|---|---|---|---|---|
| CacheClip | 71.1 | 49.9 | 37.5 | 45.8 |
| ProphetKV | 71.9 | 57.4 | 42.5 | 49.0 |
| SurroKV (1024) | 82.3 | 78.5 | 64.6 | 52.5 |
Q3: Serving engines
Response: Thank you for the question. In SGLang, SurroKV reduces TTFT by about 3.0–4.8× versus full prefill with GPU-resident caches. Setup. We integrate SurroKV into SGLang via HYPIC (Q2) and serve Llama-3.1-8B-Instruct on one L40S with concurrent closed-loop clients (16 examples per task, 32 output tokens).
Throughput and latency. Naive reuse is faster but far less accurate. On RULER 8K, as concurrent clients increase from one to eight, SurroKV's throughput rises from 1.16 to 3.99 requests/s and its end-to-end request latency from 0.86 to 2.0 s (revised Appendix A.16).
Table: TTFT (s) in SGLang. Averaged over tasks; each cell gives TTFT at the two concurrency levels listed in the Setting column.
| Setting (concurrent requests) | Full prefill | Naive reuse (no recomp.) | SurroKV (10%) |
|---|---|---|---|
| RULER 8K (1 / 8) | 0.71 / 3.85 | 0.06 / 0.20 | 0.17 / 0.96 |
| RULER 16K (1 / 4) | 1.60 / 4.81 | 0.07 / 0.19 | 0.34 / 1.12 |
| RULER 32K (1 / 2) | 3.84 / 5.75 | 0.12 / 0.19 | 0.80 / 1.54 |
| LongBench (1 / 2) | 1.10 / 2.17 | 0.08 / 0.12 | 0.36 / 0.66 |
Transfer and offline cost. TTFT excludes loading caches from CPU memory; measured separately, pinned-CPU-to-GPU loading takes a median of 54 ms for one 8K request and 364 ms for eight. The surrogate is prefilled once, shared across chunks, and adds modest offline encoding cost. The serving results are in revised Sec. 6.3 with Table 5 and Appendix A.16.
Table: Offline encoding cost per chunk. Without → with the 1024-token surrogate (revised Table 15).
| Model | Encoding time per chunk | Peak memory |
|---|---|---|
| Llama-3.2-3B | 23.1 → 27.7 ms (1.20×) | 6.53 → 6.66 GB |
| Llama-3.1-8B | 48.9 → 55.8 ms (1.14×) | 16.20 → 16.40 GB |
| Qwen2.5-14B | 97.2 → 108.7 ms (1.12×) | 29.87 → 30.11 GB |
Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.
I thank the authors for the detailed responses and additional experiments.
The retained vs discarded control experiment addresses of my question regarding the independent contribution of SurroKV.
The added evaluations on hybrid attention and NSA, as well as concurrent serving in SGLang also alleviate my original concerns about the generality and real-world serving value.
Although it seems that the methodological design relatively incremental, I think the added ablation experiments provide much stronger support for its effectiveness. I am inclined to raise my rating from 2 to 3.
Thank you for taking the time to carefully consider our responses, for your thoughtful suggestions, and for raising your rating. We are grateful for your positive impression and support for our work.
In the final version, we will make SurroKV's contribution clearer, namely content-conditioned cache construction, the attention-routing analysis, and the discard design, and include the new ablations and the experiments on hybrid and sparse-attention models and SGLang serving.
Response 1/2
We sincerely thank Reviewer kdnX for the constructive review. We address each concern below.
W1 & Q1: Novelty relative to CacheClip; retaining vs. discarding the prefix
Response: Thank you for raising this concern and for the question. SurroKV contributes content-conditioned cache construction, attention-routing theory and a discard design compatible with existing selectors.
APE and CacheClip also encode chunks after a shared prefix, but the prefix serves a different purpose. The table summarizes the difference, and the three results after it show that this difference matters.
Table: Shared-prefix designs.
The benefit goes beyond sink suppression. On Llama-3.1-8B, masking each chunk's first four tokens leaves lower KL to full prefill for SurroKV than for the CacheClip prefix (0.216 vs. 0.281 with each method's own queries); the gap persists with fixed full-prefill queries. Random-token surrogates also suppress sinks but score only 30.6 on LongBench without recomputation (average of two seeds), versus 40.6 for SurroKV (revised Appendix A.17, full evaluation).
Surrogate content matters, as the theory predicts. All tested prose satisfies both measured theory conditions on disjoint calibration data and improves both benchmarks without recomputation; random tokens violate the construction-gap condition (response to Reviewer c3tw, W2/W3).
SurroKV outperforms both methods directly. At 10% recomputation, SurroKV beats APE (no recomputation) and CacheClip (nominal 15%) on all 8 model–benchmark averages, and wins all 18 matched-budget comparisons (≈5/10/15%) on the three paper models (response to Reviewer c3tw, W1). Without recomputation, it gives the best average in 8 of 10 model–benchmark cases (response to Reviewer ep3d, W3).
Controlled comparison requested in Q1. We vary only whether one shared copy of the surrogate's KV remains before the first chunk. Both variants use identical conditioned caches, keys rotated to realized positions, and the same online recomputation method and budget, counted over context tokens only. Discarding matches retention on RULER and improves LongBench in 10 of 12 model–budget settings across 0/5/10/15%. Despite nearly identical token selection, retention adds KV storage and 5–6% TTFT overhead on Llama-3.1-8B. Discarding thus preserves the construction benefit at lower online cost. We have added these comparisons and the ablation to revised Sec. 6.3 with Table 4 and Appendices A.12–A.13.
Table: Surrogate retained vs. discarded. RULER / LongBench at 10% recomputation.
W2: New architectures and serving
Response: Thank you for raising this concern. SurroKV improves reuse on hybrid and native sparse-attention models and achieves about 3.0–4.8× lower TTFT than full prefill in SGLang with GPU-resident caches. Q2 describes the architecture-specific adaptations and results; Q3 gives the serving setup and measurements.