acceptodds
Under review as a conference paper at ICLR 2027

Construct Before You Repair: Surrogate Contexts for Non-Prefix KV Cache Reuse

Abstract

Large language model serving increasingly reuses previously computed KV caches to avoid repeatedly prefilling long or retrieved contexts. While prefix caching is exact, more flexible reuse requires cached text chunks to be composed under different future contexts, creating a mismatch between offline cache construction and online use. Existing approaches typically encode each chunk in isolation and repair the resulting cache through online selective recomputation, effectively treating unknown future context as empty. We revisit this default and introduce SurroKV, a training-free surrogate-context cache construction method that encodes each reusable chunk behind a fixed, request-independent surrogate context and stores only the resulting chunk KV caches. The surrogate is discarded after construction and never appears at inference time, preserving both cache size and the online execution path. Our analysis shows that independent construction introduces substantial attention-routing mismatch and that surrogate conditioning can reduce this mismatch without knowing the realized future context. Across RULER, LongBench, and HELMET on multiple models, SurroKV improves accuracy by 2.1%–12.7% relative to existing methods, most notably under limited recomputation budgets and longer contexts, with no additional online latency. These results highlight cache construction itself as an important design dimension for position-independent caching.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Machine Learning System, Context Caching, KV Cache

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kdnXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SurroKV, a training-free method for improving non-prefix KV cache reuse by modifying the offline cache construction process. The authors observe that conventional position-independent caching typically constructs reusable document chunks in isolation, leading to mismatches between their cached representations and those obtained from full-context prefilling.

To address this issue, SurroKV prepends a fixed, request-independent surrogate context when constructing each reusable chunk's KV cache. The surrogate context is discarded afterward, and only the chunk's KV states are stored. During online inference, these caches are assembled according to the actual context, and selected tokens are recomputed using query-guided attention scores.

The authors further analyze the impact of cache construction on attention routing and provide theoretical conditions under which surrogate conditioning can reduce the mismatch relative to full prefilling. Experiments on RULER, LongBench, and HELMET across three language models demonstrate improvements over several existing cache-reuse baselines, particularly under limited recomputation budgets, without introducing additional online inference latency.

Critical strengths

S1. Simple and effective design with relatively comprehensive experimental validation.

The proposed method is simple and training-free, requiring only a modification to the offline cache construction process without increasing the stored KV cache size or changing the online recomputation pipeline. The authors provide relatively comprehensive experiments across multiple models, benchmarks, and recomputation budgets. The consistent improvements, particularly under limited recomputation budgets, and the compatibility with several existing recomputation strategies support the effectiveness of the proposed approach.

Critical weaknesses

W1. Limited methodological novelty.

The core idea of constructing reusable KV caches with a shared prefix has already been explored in prior work, particularly CacheClip. Although SurroKV differs by discarding the surrogate context entirely after offline construction, the paper does not sufficiently establish the independent contribution of this design choice. The proposed method appears to be a relatively incremental modification of existing cache construction strategies, and a more direct comparison with closely related approaches would be helpful.

W2. Limited evaluation of architectural generality and practical serving benefits.

All the experiments in this paper are restricted to conventional attention architectures, such as Llama-3.1 and Qwen2.5, without considering more recent hybrid linear/full-attention or native sparse-attention models (these architectural advances may reduce the marginal benefits of conventional KV cache reuse or introduce additional cache compatibility challenges). Moreover, the efficiency evaluation is limited to single-request inference using Hugging Face Transformers and PyTorch SDPA, without integration into production-oriented serving engines. Consequently, the applicability of the method to modern LLM architectures and its practical benefits under realistic serving workloads remain insufficiently demonstrated.

Questions for authors

Q1. Novelty relative to prior work.

Could the authors clarify the key methodological differences between SurroKV and CacheClip, particularly regarding their shared-prefix cache construction strategies? A controlled comparison using the same online recomputation method and budget, while varying whether the shared prefix is retained or discarded, would help establish the independent contribution of SurroKV.

Q2. Applicability to more recent attention architectures.

Could the authors discuss whether SurroKV can be directly applied to more recent architectures, such as hybrid linear/full-attention models (e.g., Qwen3.5) and native sparse-attention models (e.g., DeepSeek-V3.2 or above)? In particular, would the surrogate-conditioned KV states remain reusable when attention routing or recurrent-state dependencies differ from those of conventional causal attention?

Q3. Practical serving performance.

Could the authors provide further evidence of SurroKV's practical benefits when integrated into realistic serving systems, such as vLLM or SGLang? How would the method affect end-to-end TTFT, request throughput, and latency under continuous batching and concurrent requests, considering cache transfer and offline construction overhead? This would be very helpful in assessing the method's potential for real-world deployment.

4 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank Reviewer kdnX for the constructive review. We address each concern below.


W1 & Q1: Novelty relative to CacheClip; retaining vs. discarding the prefix

Response: Thank you for raising this concern and for the question. SurroKV contributes content-conditioned cache construction, attention-routing theory and a discard design compatible with existing selectors.

APE and CacheClip also encode chunks after a shared prefix, but the prefix serves a different purpose. The table summarizes the difference, and the three results after it show that this difference matters.

Table: Shared-prefix designs.

APECacheClipSurroKV
Prefixa few shared tokens (e.g., newlines)shared system prompt (chat-template header in our reimplementation)512/1024-token natural-language passage
Purposeavoid repeated attention sinks; attention is also rescaled onlineavoid repeated attention sinks; guided token selection and grouped recomputation onlinecorrect the chunk's attention routing toward full prefill (Sec. 5, Theorems 5.1 and 5.3)
Onlineone shared copy kept in every requestone shared copy kept in every requestdiscarded: no extra tokens, no position shift, same cache size
  • The benefit goes beyond sink suppression. On Llama-3.1-8B, masking each chunk's first four tokens leaves lower KL to full prefill for SurroKV than for the CacheClip prefix (0.216 vs. 0.281 with each method's own queries); the gap persists with fixed full-prefill queries. Random-token surrogates also suppress sinks but score only 30.6 on LongBench without recomputation (average of two seeds), versus 40.6 for SurroKV (revised Appendix A.17, full evaluation).

  • Surrogate content matters, as the theory predicts. All tested prose satisfies both measured theory conditions on disjoint calibration data and improves both benchmarks without recomputation; random tokens violate the construction-gap condition (response to Reviewer c3tw, W2/W3).

  • SurroKV outperforms both methods directly. At 10% recomputation, SurroKV beats APE (no recomputation) and CacheClip (nominal 15%) on all 8 model–benchmark averages, and wins all 18 matched-budget comparisons (≈5/10/15%) on the three paper models (response to Reviewer c3tw, W1). Without recomputation, it gives the best average in 8 of 10 model–benchmark cases (response to Reviewer ep3d, W3).

Controlled comparison requested in Q1. We vary only whether one shared copy of the surrogate's KV remains before the first chunk. Both variants use identical conditioned caches, keys rotated to realized positions, and the same online recomputation method and budget, counted over context tokens only. Discarding matches retention on RULER and improves LongBench in 10 of 12 model–budget settings across 0/5/10/15%. Despite nearly identical token selection, retention adds KV storage and 5–6% TTFT overhead on Llama-3.1-8B. Discarding thus preserves the construction benefit at lower online cost. We have added these comparisons and the ablation to revised Sec. 6.3 with Table 4 and Appendices A.12–A.13.

Table: Surrogate retained vs. discarded. RULER / LongBench at 10% recomputation.

ModelSurrogate retainedSurrogate discarded (SurroKV)
Llama-3.2-3B80.6 / 36.180.5 / 39.0
Llama-3.1-8B85.9 / 60.785.7 / 62.1
Qwen2.5-14B87.0 / 56.886.7 / 57.9

W2: New architectures and serving

Response: Thank you for raising this concern. SurroKV improves reuse on hybrid and native sparse-attention models and achieves about 3.0–4.8× lower TTFT than full prefill in SGLang with GPU-resident caches. Q2 describes the architecture-specific adaptations and results; Q3 gives the serving setup and measurements.

AuthorsOpenReview ↗

Response 2/2

Q2: Hybrid and native sparse-attention models

Response: Thank you for the question. SurroKV works with both architectures: it conditions HYPIC's composable cached states for the hybrid model and the per-token KV for the sparse model, and the conditioned caches remain reusable and improve reuse accuracy on both.

Hybrid linear/full attention: Qwen3.5-35B-A3B. We use HYPIC's open-source SGLang implementation (arXiv:2607.01299), which caches composable linear-layer transitions and full-attention KV. SurroKV conditions both on the surrogate, then discards it while preserving composability. Both methods recompute a fixed fraction of each chunk's initial tokens, so absolute scores are not comparable to our main table.

Table: Hybrid Qwen3.5-35B-A3B. RULER / LongBench by recomputation budget.

Construction5%10%15%
HYPIC36.1 / 16.437.0 / 16.638.8 / 17.8
SurroKV (1024)54.5 / 44.757.4 / 45.159.1 / 46.5

Native sparse attention: MiniCPM4-8B. DeepSeek-V3.2 (671B) exceeds our compute budget, so we use MiniCPM4-8B, which also adopts native sparse attention (InfLLM v2) at a scale we can run. Its per-token KV remains reusable. We keep its sparse attention active throughout, so the results reflect the sparse model itself (details in revised Appendix A.15). ProphetKV and SurroKV use 10% recomputation, and CacheClip uses a nominal 15% (about 11.6% actual at 8K). LongBench is evaluated at its original lengths. SurroKV is the best at every length, and its gain over all baselines grows with context length; it also leads at the other budgets (revised Appendix A.15).

Table: Native sparse-attention MiniCPM4-8B. 10% recomputation.

MethodRULER 8KRULER 16KRULER 32KLongBench
CacheClip71.149.937.545.8
ProphetKV71.957.442.549.0
SurroKV (1024)82.378.564.652.5

Q3: Serving engines

Response: Thank you for the question. In SGLang, SurroKV reduces TTFT by about 3.0–4.8× versus full prefill with GPU-resident caches. Setup. We integrate SurroKV into SGLang via HYPIC (Q2) and serve Llama-3.1-8B-Instruct on one L40S with concurrent closed-loop clients (16 examples per task, 32 output tokens).

Throughput and latency. Naive reuse is faster but far less accurate. On RULER 8K, as concurrent clients increase from one to eight, SurroKV's throughput rises from 1.16 to 3.99 requests/s and its end-to-end request latency from 0.86 to 2.0 s (revised Appendix A.16).

Table: TTFT (s) in SGLang. Averaged over tasks; each cell gives TTFT at the two concurrency levels listed in the Setting column.

Setting (concurrent requests)Full prefillNaive reuse (no recomp.)SurroKV (10%)
RULER 8K (1 / 8)0.71 / 3.850.06 / 0.200.17 / 0.96
RULER 16K (1 / 4)1.60 / 4.810.07 / 0.190.34 / 1.12
RULER 32K (1 / 2)3.84 / 5.750.12 / 0.190.80 / 1.54
LongBench (1 / 2)1.10 / 2.170.08 / 0.120.36 / 0.66

Transfer and offline cost. TTFT excludes loading caches from CPU memory; measured separately, pinned-CPU-to-GPU loading takes a median of 54 ms for one 8K request and 364 ms for eight. The surrogate is prefilled once, shared across chunks, and adds modest offline encoding cost. The serving results are in revised Sec. 6.3 with Table 5 and Appendix A.16.

Table: Offline encoding cost per chunk. Without → with the 1024-token surrogate (revised Table 15).

ModelEncoding time per chunkPeak memory
Llama-3.2-3B23.1 → 27.7 ms (1.20×)6.53 → 6.66 GB
Llama-3.1-8B48.9 → 55.8 ms (1.14×)16.20 → 16.40 GB
Qwen2.5-14B97.2 → 108.7 ms (1.12×)29.87 → 30.11 GB

Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.

I thank the authors for the detailed responses and additional experiments.

The retained vs discarded control experiment addresses of my question regarding the independent contribution of SurroKV.

The added evaluations on hybrid attention and NSA, as well as concurrent serving in SGLang also alleviate my original concerns about the generality and real-world serving value.

Although it seems that the methodological design relatively incremental, I think the added ablation experiments provide much stronger support for its effectiveness. I am inclined to raise my rating from 2 to 3.

Thank you for taking the time to carefully consider our responses, for your thoughtful suggestions, and for raising your rating. We are grateful for your positive impression and support for our work.

In the final version, we will make SurroKV's contribution clearer, namely content-conditioned cache construction, the attention-routing analysis, and the discard design, and include the new ablations and the experiments on hybrid and sparse-attention models and SGLang serving.

Reviewer_c3twOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SurroKV, an offline cache-construction method for non-prefix KV-cache reuse. Each chunk is encoded after a fixed surrogate context, whose own KV states are subsequently discarded while the conditioned chunk cache is retained. At inference time, these caches are assembled at their request-specific positions and combined with existing selective recomputation. The paper analyzes construction and attention-routing errors and evaluates the resulting quality improvements across multiple models, benchmarks, and recomputation budgets.

Critical strengths
  1. Clear motivation. The paper highlights offline cache construction as a source of reuse error and demonstrates that improving the initial cached states can complement selective online recomputation.
  2. Surrogate conditioning requires no model training, and no runtime burden on memory.
  3. Comprehensive controlled evaluations. Experiments cover multiple models, surrogate lengths, and recomputation budgets, while integration with four existing token selectors helps isolate the effect of the proposed construction strategy.
Critical weaknesses
  1. Missing comparisons with the closest prior work. To my understanding, APE and CacheClip already introduce shared prefixes during chunk encoding, but neither is directly evaluated.
  2. Missing ablation of surrogate removal and surrogate prompt content.
  3. Gap between the theoretical analysis and surrogate selection. The theory establishes conditional improvements in intermediate errors but does not explain why the selected Wikipedia surrogate should reliably satisfy those conditions.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank Reviewer c3tw for the constructive review. We address each concern below.


W1: Comparison with APE and CacheClip

Response: Thank you for raising this concern. SurroKV outperforms APE and CacheClip on every model–benchmark average below. We use APE's official code and reimplement CacheClip, whose code is not released (details in revised Appendix A.12). All methods use the same 512-token chunks. For a fair comparison, we set CacheClip's budget to 15%, which recomputes about 10% of tokens in practice because its grouping step drops isolated tokens, matching the 10% budget of the other recomputation methods. APE does not recompute tokens by design.

Because CacheClip's actual budget differs from its nominal one, we also compare at matched actual budgets (≈5/10/15%) on the three paper models: SurroKV wins all 18 comparisons. Tuning APE on held-out tasks trades RULER for LongBench and still stays below SurroKV. Details are in revised Sec. 6.3 (Table 4) and Appendix A.12.

Table: Comparison with APE and CacheClip. RULER / LongBench.

ModelAPE (no recomp.)CacheClipProphetKVSurroKV (1024)
Llama-3.2-3B50.4 / 29.567.7 / 31.574.7 / 35.380.5 / 39.0
Llama-3.1-8B52.3 / 39.078.4 / 54.284.0 / 59.885.7 / 62.1
Qwen2.5-14B40.1 / 32.682.1 / 50.883.0 / 55.086.7 / 57.9
Qwen3-8B41.1 / 35.564.1 / 44.582.5 / 58.386.6 / 60.4

W2: Ablation of surrogate removal and content

Response: Thank you for raising this concern. We add both ablations: the surrogate improves accuracy during encoding but is not needed afterwards, and a wide range of fluent text works as the surrogate.

Surrogate removal. We compare three variants with the same selector and budget: no surrogate, the surrogate retained online after encoding, and the surrogate discarded after encoding (SurroKV). Encoding without a surrogate lowers accuracy on every model. Removing it after encoding, as SurroKV does, keeps RULER on par with keeping it online and improves LongBench, while keeping it adds KV storage and 5–6% TTFT overhead on Llama-3.1-8B; across all budgets, discarding is better on LongBench in 10 of 12 model–budget settings.

Table: Surrogate removal. RULER / LongBench at 10% recomputation.

ModelNo surrogateSurrogate retainedSurrogate discarded (SurroKV)
Llama-3.2-3B74.7 / 35.380.6 / 36.180.5 / 39.0
Llama-3.1-8B84.0 / 59.885.9 / 60.785.7 / 62.1
Qwen2.5-14B83.0 / 55.087.0 / 56.886.7 / 57.9

Surrogate content. We test 17 candidates of 1024 tokens from Wikipedia, news, web pages, books, essays and code, plus random and repeated tokens, on Llama-3.1-8B; none shares a 16-gram with any evaluation task. To cover all 17 candidates within our compute budget, we evaluate them on the first 50 examples per task; representative surrogates are evaluated on the full benchmarks in our response to Reviewer ep3d, W2. All fluent text improves both benchmarks without recomputation, whereas random tokens mainly help RULER, code gives small and unstable gains, and repeated tokens collapse. Rows marked “avg. of 2” average two candidates from the same source (per-candidate results in revised Appendix A.17).

Table: Surrogate content study. Llama-3.1-8B, first 50 examples per task.

SurrogateRULER 0%RULER 10%LongBench 0%LongBench 10%
Independent construction43.785.328.357.0
Wikipedia (paper)63.086.441.360.3
Wikipedia, another HotpotQA passage64.286.442.159.1
Wikipedia, other topics (avg. of 2)54.589.738.458.1
News, 2026 (avg. of 2)57.688.238.158.0
Web pages (avg. of 2)59.388.538.259.6
Books (avg. of 2)55.589.138.258.9
Essays, 2026 (avg. of 2)65.689.436.557.9
Code (avg. of 2)55.390.930.356.8
Random tokens (avg. of 2 seeds)61.288.130.055.7
" the" repeated2.83.04.82.7
AuthorsOpenReview ↗

Response 2/2

W3: Gap between theory and surrogate selection

Response: Thank you for raising this concern. The theory turns "a good surrogate" into two measurable conditions, and our tests show that all tested fluent text satisfies them: the Wikipedia passage is simply our choice among many valid surrogates, not the only text that fits the theory. Theorem 5.1 favors a surrogate whose effect on each chunk is close to the average effect of real preceding contexts, and Theorem 5.3 favors one that brings the chunk's attention closer to full prefill.

We measure both conditions for 17 candidates, spanning five types of fluent text, code, random tokens and a repeated token, on calibration data disjoint from evaluation and without any accuracy labels. For each surrogate, the table below reports the two theory quantities and the resulting accuracy gain:

  • S_route (Theorem 5.3): how far the chunk's attention pattern is from full prefill, relative to encoding without a surrogate; lower is better, and <1 means improvement.

  • R (Theorem 5.1): construction gap, i.e., how far the surrogate's effect on the chunk is from that of real preceding context, relative to encoding without a surrogate; lower is better, and <1 means improvement.

  • Gain on LongBench, 0%: improvement over independent encoding without recomputation; higher is better.

Every tested fluent passage, from Wikipedia, news, web pages, books and essays, satisfies both conditions and improves LongBench, whereas random tokens fail the construction-gap condition and the repeated token fails both. Because the conditions are computed without accuracy labels, their agreement with the downstream gains across candidates (revised Appendix A.17, Table 28 and Figure 8) is a genuine test rather than a post-hoc fit. SurroKV is therefore not tied to a particular surrogate text; the Wikipedia passage is one representative of this broad class.

Table: Theory conditions and LongBench gain. Llama-3.1-8B, no recomputation.

SurrogateS_route (↓)R (↓)Gain on LongBench, 0% (↑)
Wikipedia (paper)0.0870.346+13.0
Wikipedia, another HotpotQA passage0.0800.313+13.8
Wikipedia, other topics (avg. of 2)0.0860.362+10.1
News, 2026 (avg. of 2)0.0920.364+9.8
Web pages (avg. of 2)0.0880.299+9.9
Books (avg. of 2)0.0860.337+9.9
Essays, 2026 (avg. of 2)0.0800.384+8.2
Code (avg. of 2)0.1010.667+2.0
Random tokens (avg. of 2 seeds)0.1281.85+1.7
" the" repeated2.1534.469−23.5

In practice, the conditions serve to rule out unsuitable text such as random or repeated tokens, and every fluent passage we tested gives a clear gain. We therefore use one fixed, task-independent passage and set its length by the length ablation, where gains largely saturate at 512–1024 tokens (Table 9); see Sec. 5 and revised Appendix A.17 for the full analysis, including selection by S_route alone.


Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer c3tw,

As the discussion period is coming to an end, we would greatly appreciate it if you could take a look at our responses. If you have any remaining questions or concerns, we would be happy to provide further clarification and discussion.

Thank you again for your time and consideration.

Best,

The Authors

Reviewer_ep3dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose an efficient KV cache reuse mechanism in the RAG system. Prefix caching requires exact matching while preceding contexts often differ in RAG systems. However, computing reusable chunks in isolation without future context causes residual and accumulative errors. This paper finds out that the attention-routing error dominates the mismatch across layers under independent reuse. Therefore, this paper proposes SurroKV, to use a surrogate-context to obtain the chunk KV cache, and removing the surrogate context during online inference. Experiments on multiple benchmarks show strong performances.

Critical strengths
  1. This paper targets on an important and practical question. KV cache reusing is necessary in LLM inference while RAG is widely used in real-world scenarios, so enhancing KV reusing in RAG is very important.

  2. The findings of this paper about attention-routing error is novel and interesting.

  3. The proposed SurroKV method is well-motivated and effective in experiments.

Critical weaknesses
  1. No analysis show that the selected Wiki surrogate matches the conditions when during decomposition in the theory. Inappropriate surrogate can also cause over correction according to the theory.

  2. I doubt that the selected surrogate is quite related with the pretrained corpus, the tested domains and more OOD tasks and the problems need to be further tested.

  3. APE (https://arxiv.org/pdf/2502.05431v1) and CacheClip (https://arxiv.org/pdf/2510.10129v1) needs to be compared via experiments. They says that the benefit primarily come from suppressing repeated attention sinks, which looks different from the current paper and more discussions are needed here.

Questions for authors

Can the proposed methods be extended to more latest base models, like Qwen-3.x series?

What is the incremental contribution over the closest prior works, like APE/CacheClip?

3 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank Reviewer ep3d for the constructive review. We address each concern below.


W1: Theory conditions and over-correction

Response: Thank you for raising this concern. We add this analysis: on calibration data disjoint from evaluation, the paper's Wikipedia surrogate satisfies both theory conditions. This is not specific to our passage: all tested prose from five text types (Wikipedia, news, web pages, books and essays) satisfies both conditions, whereas random tokens violate the R condition and a repeated token violates both. Fluent text thus generally meets the theory's conditions, so SurroKV does not depend on a carefully chosen surrogate.

For each surrogate, the table below reports the two theory quantities and the resulting accuracy gain:

  • S_route (Theorem 5.3): how far the chunk's attention pattern is from full prefill, relative to encoding without a surrogate; lower is better, and <1 means improvement.

  • R (Theorem 5.1): construction gap of the decomposition, i.e., how far the surrogate's effect on the chunk is from that of real preceding context, relative to encoding without a surrogate; lower is better, and <1 means improvement.

  • Gain on LongBench, 0%: improvement over independent encoding without recomputation; higher is better.

Table: Theory conditions of different surrogates. Llama-3.1-8B; gain is over independent encoding without recomputation.

SurrogateS_route (↓)R (↓)Gain on LongBench, 0% (↑)
Wikipedia (paper)0.0870.346+13.0
Wikipedia, another HotpotQA passage0.0800.313+13.8
Wikipedia, other topics (avg. of 2)0.0860.362+10.1
News, 2026 (avg. of 2)0.0920.364+9.8
Web pages (avg. of 2)0.0880.299+9.9
Books (avg. of 2)0.0860.337+9.9
Essays, 2026 (avg. of 2)0.0800.384+8.2
Code (avg. of 2)0.1010.667+2.0
Random tokens (avg. of 2 seeds)0.1281.85+1.7
" the" repeated2.1534.469−23.5

R also captures the over-correction risk you mention: R > 1 means the surrogate pushes the chunk further from real context than using no surrogate (Appendix B.3). Random and repeated tokens fall into this regime, whereas every prose passage stays well below 1. These results are in Sec. 5 and revised Appendix A.17.


W2: Pretraining corpus, domains and OOD tasks

Response: Thank you for raising this concern. To test whether the gain comes from the surrogate being seen during pretraining or matching the test domains, we replace the Wikipedia passage with text the model could not have seen and with text from other domains, keeping all other settings fixed.

A NASA press release published in November 2026, after Llama-3.1's knowledge cutoff (December 2023), performs comparably to Wikipedia with recomputation; without recomputation, LongBench is similar and RULER is lower. Our content study also covers 17 candidates from six sources, none sharing a 16-gram with any evaluation task, and all fluent text improves both benchmarks (response to Reviewer c3tw, W2). The gain therefore depends neither on the model having seen the surrogate during pretraining nor on the surrogate matching the test domains.

Table: Post-cutoff and code surrogates. Llama-3.1-8B, full evaluation, RULER / LongBench.

SurrogateNo recomp.5%10%15%
Wikipedia, 1024 (paper)61.5 / 40.681.4 / 59.085.7 / 62.188.0 / 63.2
NASA (post-cutoff), 102456.4 / 40.882.7 / 58.986.7 / 62.588.4 / 63.0
Code (sorting source), 102457.8 / 26.085.0 / 53.088.4 / 60.289.3 / 62.9

We would also like to clarify that many of our evaluation tasks are already unrelated to the Wikipedia surrogate: the synthetic RULER tasks (key–value retrieval and variable tracking), meeting summarization (QMSum) and Chinese passage retrieval (PR-zh). On all three paper models, SurroKV improves over the baselines on these tasks under the same 10% budget (Table 1), so its gain does not rely on the surrogate matching the target domain. Together with the surrogate-replacement results above, this addresses the concern from both directions. The surrogate-replacement results are in revised Appendix A.17.

AuthorsOpenReview ↗

Response 2/2

W3 & Q2: Comparison with APE and CacheClip; attention sinks; incremental contribution

Response: Thank you for raising this concern and for the question. As the reviewer notes, APE and CacheClip attribute their benefit mainly to suppressing repeated attention sinks, and this is the key difference from SurroKV. Their short shared prefix absorbs the attention that independently encoded chunks would otherwise place on their first tokens. SurroKV instead encodes each chunk after a content-bearing passage, which corrects the chunk's attention routing toward full prefill (Sec. 5); it also suppresses sinks, but this is not its main purpose. We therefore expect gains beyond sink suppression.

We support this with two experiments. First, we measure attention directly, comparing the attention of the last prompt token with full prefill on Llama-3.1-8B (100 examples, no recomputation; details in revised Appendix A.14). We report the attention mass on the first four tokens of each chunk (the sink positions) and the KL divergence to full prefill, with and without these tokens masked.

Both shared prefixes remove the sink peak, but SurroKV stays closer to full prefill even with the sink positions masked, and this gap persists with fixed full-prefill queries (both difference intervals exclude zero). SurroKV thus also improves routing over the remaining tokens, consistent with Theorem 5.3. The analysis is in revised Sec. 6.3 and Appendices A.12 and A.14.

Table: Attention diagnostics. Llama-3.1-8B, no recomputation.

QuantityFull prefillNaive reuseCacheClip prefixSurroKV
Attention mass on the first 4 tokens of each chunk0.0020.5220.0140.005
KL to full prefill01.9410.2960.222
KL to full prefill, first 4 tokens masked00.9880.2810.216

Second, we compare the methods without recomputation; with recomputation, SurroKV outperforms both methods (response to Reviewer c3tw, W1). Here, CacheClip prefix uses its shared chat-template prefix, kept online, without selection or recomputation. SurroKV gives the best average in 8 of 10 cases. Random-token prefixes also suppress the boundary peaks (Fig. 3) but recover little of the LongBench gain, indicating that sink suppression alone does not explain the gain.

Table: Comparison without recomputation. RULER / LongBench.

ModelNaive reuseAPECacheClip prefixSurroKV (1024)
Llama-3.2-3B23.5 / 13.150.4 / 29.551.3 / 26.748.2 / 32.6
Llama-3.1-8B44.3 / 28.952.3 / 39.055.1 / 40.961.5 / 40.6
Qwen2.5-14B47.6 / 17.240.1 / 32.655.0 / 35.064.6 / 38.0
Qwen3-8B46.2 / 30.041.1 / 35.552.0 / 37.763.6 / 45.9
MiniCPM4-8B36.8 / 31.041.6 / 30.144.3 / 33.750.6 / 34.6

Incremental contribution (Q2). SurroKV adds content-conditioned construction, an attention-routing analysis with measurable conditions (Sec. 5; response to Reviewer c3tw, W3), and a discard design that keeps no surrogate tokens online and works with existing selectors. The measurements above show that its benefit is not explained by sink suppression alone, and a controlled ablation shows that keeping the surrogate online adds cost without a consistent accuracy gain. A fuller discussion is in our response to Reviewer kdnX, W1 & Q1.


Q1: Compatibility with Qwen3.x models

Response: Thank you for the question. SurroKV only changes how chunk caches are built offline, so it is compatible with newer model architectures. We evaluate two Qwen3.x models: Qwen3-8B (standard attention, thinking disabled), where SurroKV is applied as is and leads all baselines at every budget (revised Appendix A.15, Table 22), and the hybrid linear/full-attention Qwen3.5-35B-A3B, where it conditions both the cached linear-layer states and the full-attention KV through HYPIC (details in our response to Reviewer kdnX, Q2). On both models, SurroKV improves over the corresponding reuse baseline at every recomputation budget.

Table: Qwen3.x models. RULER / LongBench by recomputation budget. Qwen3.5 follows HYPIC's recomputation policy, so its scores are not comparable to Qwen3-8B.

ModelMethod5%10%15%
Qwen3-8BProphetKV75.8 / 54.882.5 / 58.384.5 / 59.3
Qwen3-8BSurroKV (1024)84.1 / 58.786.6 / 60.487.7 / 61.5
Qwen3.5-35B-A3BHYPIC36.1 / 16.437.0 / 16.638.8 / 17.8
Qwen3.5-35B-A3BSurroKV (1024)54.5 / 44.757.4 / 45.159.1 / 46.5

Thank you again for your valuable feedback, which has helped us significantly improve the paper. We hope these responses address your concerns, and we would be happy to clarify any remaining questions during the discussion phase.

AuthorsOpenReview ↗

Follow-up on Rebuttal Discussion

Dear Reviewer ep3d,

As the discussion period is coming to an end, we would greatly appreciate it if you could take a look at our responses. If you have any remaining questions or concerns, we would be happy to provide further clarification and discussion.

Thank you again for your time and consideration.

Best,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Area Chairs and Reviewers,

We sincerely appreciate all your time and effort in reviewing our submission. We have uploaded our revised manuscript with all changes highlighted in blue. Below is a summary of the revisions:

1. Comparison with APE and CacheClip (Reviewers kdnX, c3tw, ep3d): Sec. 6.3 (Table 4), Appendices A.12 and A.14. SurroKV outperforms both methods, including at matched recomputation budgets, and its attention stays closer to full prefill even after chunk-boundary sinks are masked.

2. Retaining vs. discarding the surrogate (Reviewers kdnX, c3tw): Sec. 6.3 (Table 4) and Appendix A.13. Discarding the surrogate matches or improves accuracy while avoiding extra KV storage and TTFT overhead.

3. Surrogate content and theory conditions (Reviewers c3tw, ep3d): Sec. 5, Appendix A.17 (Figures 8–10) and Appendix B.9. Across 17 candidate surrogates, including text published after the model's knowledge cutoff, all fluent text satisfies both theory conditions and improves accuracy, while random and repeated tokens violate at least one condition and give little or no LongBench gain.

4. Newer architectures and serving (Reviewers kdnX, ep3d): Sec. 6.3 (Table 5) and Appendices A.15–A.16. SurroKV improves reuse on Qwen3-8B, hybrid Qwen3.5-35B-A3B and sparse-attention MiniCPM4-8B, and reduces TTFT by about 3.0–4.8× in SGLang.

Please note that all table references above correspond to the revised manuscript. For table numbering in the rebuttal response, please refer to the initial submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.