acceptodds
Under review as a conference paper at ICLR 2027

FastPIC: Fast Position Independent Caching for Multi-Tenant LLM Serving

Abstract

As LLM context windows continue to expand, prefill overhead has become major bottlenecks in long-context serving. Existing context-caching systems rely on prefix matching and cannot effectively reuse repeated content that appears at different positions across multi-tenant requests. Position-Independent Caching (PIC) supports non-prefix reuse, but it remains limited by boundary shifts caused by fixed chunking and attention deviations caused by cross-context KV reuse. This paper presents FastPIC, a position independent KV-cache reuse mechanism for multi-tenant LLM serving. FastPIC uses Token-Defined Chunking to derive boundaries and cache identities from token content, reducing the impact of prefix variations on matching identical token sequences. It further uses Dual-Path Token Selection to recompute attention sinks and high KV-deviation tokens under a limited budget, improving both cache reuse and generation quality. Experiments across multiple models and datasets show that FastPIC reduces TTFT and improves serving throughput while preserving generation quality.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, KV Cache Reuse, Position-Independent Caching, Token-Defined Chunking, Selective Recompute

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DDfLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In LLM serving, KV cache reuse plays a critical role in minimizing Time to First Token (TTFT). However, current approaches to reusing cached chunks are rigid and explicitly require that the prompt prefix also match. Position-Independent Caching (PIC) removes this constraint and allows a request to reuse the cache from requests with a different prefix. To that end, the authors propose FastPIC, which introduces two complementary techniques. First, rather than chunking at arbitrary boundaries, it uses Token-Defined Chunking to find chunk boundaries that maximize chunk matches. Second, it proposes a dual-path token selection mechanism that decides, for each incoming request, whether to recompute or reuse the cached chunks. When an exact cached chunk does not exist, it identifies the tokens whose attention matters most and recomputes only those. Their evaluation on an array of datasets shows that FastPIC improves both accuracy and the throughput of the serving system.

Critical strengths

The paper correctly identifies the boundary shift problem in PIC and proposes a simple solution with Token-Defined Chunking.

Critical weaknesses
  • Some key details are missing, and the manuscript reads like a set of instructions rather than an explanation of the intuition behind why the system was designed this way. For example, what are the implications of using a ww-bit fingerprint? What are practical values of ww, and how would an operator choose one? Similarly, the paper leaves the intuition behind the effectiveness of M(li)M(l_i) for the reader to work out.
  • The novelty of the dual-path token selection seems thin. Prior work, some of which the authors already cite in that section, has done most of this. It would help to state the delta explicitly.
Questions for authors
  1. How many chunks does Token-Defined Chunking generate compared to simple count-based chunking?
  2. For content-driven deviation, how is Z⋆Z^{\star} computed from the content alone, without any forward passes?
  3. Could you share TTFT and throughput as a function of QPS for any of the datasets where you report F1 scores?
3 comments
AuthorsOpenReview ↗

We clarify TDC chunk counts, early-layer KV proxies, and complementary quality and serving evaluations.

FastPIC: Response to Reviewer DDfL

We thank the reviewer for the careful reading. We address the questions below.

1. How many chunks does Token-Defined Chunking generate compared to simple count-based chunking?

For a fixed hash mapping, TDC deterministically partitions a given prompt; chunk counts across inputs can be estimated using a probabilistic model. We illustrate this with Token-Defined Chunking (256).

The boundary condition (f_i & M) == 0 requires all fingerprint bits selected by the mask to be zero. Assuming approximately uniform fingerprints, a mask with mm set bits yields a boundary probability of approximately 2−m2^{-m} per check:

MaskBinary representationSet bitsBoundary probability
Strict mask 0x2FF0010 1111 111191/5121/512
Relaxed mask 0x03F0000 0011 111161/641/64

With (Bmin⁡,Bnorm,Bmax⁡)=(64,256,512)(B_{\min}, B_{\mathrm{norm}}, B_{\max})=(64,256,512), boundary checks are skipped below 64 tokens. The strict mask applies at lengths 64–255, the relaxed mask at 256–511, and a cut is forced at 512.

Ignoring truncation at the end of the prompt and assuming independent boundary checks, the chunk length LL satisfies:

P(64≤L<256)=1−(511512)192≈31.3%,P(64 \leq L < 256) = 1 - \left(\frac{511}{512}\right)^{192} \approx 31.3\%,

P(256≤L<512)=(511512)192[1−(6364)256]≈67.5%,P(256 \leq L < 512) = \left(\frac{511}{512}\right)^{192} \left[1 - \left(\frac{63}{64}\right)^{256}\right] \approx 67.5\%,

P(L=512)=(511512)192(6364)256≈1.2%.P(L = 512) = \left(\frac{511}{512}\right)^{192} \left(\frac{63}{64}\right)^{256} \approx 1.2\%.

Let a=511/512a=511/512 and b=63/64b=63/64. The expected chunk length is:

\mathbb{E}[L] &=64+\sum_{j=64}^{511}P(L>j)\\ &=64+\sum_{r=1}^{192}a^r+a^{192}\sum_{r=1}^{256}b^r\\ &=64+\frac{a(1-a^{192})}{1-a} +a^{192}\frac{b(1-b^{256})}{1-b}\\ &\approx266.44\ \text{tokens}. \end{aligned}$$ Thus, for sufficiently long prompts of length $n$, the two methods produce comparable chunk counts under this model: $$N_{\mathrm{fixed}}=\left\lceil\frac{n}{256}\right\rceil, \qquad \mathbb{E}[N_{\mathrm{TDC}}]\approx\frac{n}{266.44}.$$ These are idealized estimates; actual counts depend on token content, fingerprint correlations, and the final partial chunk. **TDC’s primary benefit is facilitating local boundary resynchronization through content-driven chunking, thereby mitigating boundary shifts and improving cache reuse.** ## 2. For content-driven deviation, how is $Z^{*}$ computed from the content alone, without any forward passes? **The full-prefill KV states in Equation (3) serve only as an ideal reference; online selection does not require full prefill across all layers.** FastPIC uses early-layer KV deviations as a proxy to select tokens for recomputation in subsequent layers. This approach relies on the cross-layer correlation of high-KV-deviation tokens. Transformers progressively update token hidden representations through residual connections, with input representations changing gradually across adjacent layers. Since KV states are linear projections of these representations, tokens with large KV deviations in an early layer may retain relatively large deviations in the next layer, providing candidates for subsequent-layer recomputation. Specifically, FastPIC fully computes the first two layers and applies the normalized deviation measure in Equation (3) to compare the current and cached KV states at the second layer. It then ranks candidates and allocates the remaining recomputation budget accordingly. We use the second layer because, after positional alignment, first-layer KV states for identical tokens do not yet reflect differences in preceding context. Second-layer inputs already incorporate contextual information from first-layer attention, providing a signal of context-induced deviations. ## 3. Could you share TTFT and throughput as a function of QPS for any of the datasets where you report F1 scores? In principle, the datasets used to evaluate the F1–TTFT trade-off can also be organized into asynchronous workloads. We use two complementary settings to evaluate **single-request quality–latency trade-offs** and **concurrent multi-tenant serving performance**. For question-answering and long-context retrieval benchmarks, we process requests sequentially to minimize queuing and concurrent resource contention, allowing us to more directly assess how KV reuse and selective recomputation affect generation quality and TTFT. For concurrent serving, we use instructions from real user interactions in ShareGPT, generate request arrivals according to a Poisson process, and vary the arrival rate to examine queuing, resource contention, and throughput under different loads. This design uses benchmark reference answers to assess quality preservation and the diversity of real user instructions to evaluate serving performance, providing complementary assessments of FastPIC.

Thank you for your response.

About the last question, I meant could you share the TTFT and throughput values for a pair of datasets as a function of QPS? You current response details the experimental setup instead.

AuthorsOpenReview ↗

Additional MuSiQue results validate FastPIC’s TTFT and throughput gains across QPS levels.

FastPIC: Response to Reviewer DDfL

We thank the reviewer for clarifying the experimental request and for your continued guidance in improving our work. We are pleased to have clarified the first two questions in our previous responses. We now provide TTFT and input throughput results at varying QPS and will incorporate them into the revised PDF, with all changes highlighted.

4. Dataset Serving Performance at Different QPS Levels

Reviewer comment: Could you share TTFT and throughput as a function of QPS for any of the datasets where you report F1 scores?

Response: We use MuSiQue, a dataset for which we report F1 scores, to construct an asynchronous workload on Llama 3.1-8B. By varying the request arrival rate, we measure end-to-end TTFT and input throughput for FastPIC-15%, FR, CacheBlend-15%, and EPIC-32 at different QPS levels. The results are shown below.

Table 1: TTFT (s) ↓ and input throughput (tokens/s) ↑ on MuSiQue at different QPS levels

QPSFR (TTFT / input throughput)CacheBlend-15% (TTFT / input throughput)EPIC-32 (TTFT / input throughput)FastPIC-15% (TTFT / input throughput)
12.91 / 76682.82 / 76452.75 / 77022.69 / 7715
85.18 / 90514.96 / 92034.83 / 90014.71 / 9316
169.18 / 102128.83 / 103249.05 / 105588.25 / 10808
1810.24 / 1042010.57 / 1045210.08 / 106179.23 / 11305
2011.87 / 1046811.64 / 1029612.05 / 1051110.31 / 11447
2212.24 / 1045614.29 / 1068613.12 / 1079011.27 / 12041
2413.96 / 1069216.48 / 1080215.18 / 1082812.56 / 12391
2615.78 / 1045218.27 / 1072917.09 / 1078913.32 / 12241
2817.97 / 1058020.22 / 1077919.69 / 1088214.80 / 12513
3021.27 / 1022222.55 / 1069222.15 / 1071315.78 / 12453
3224.03 / 1040425.68 / 1064824.69 / 1074917.01 / 12478

Results analysis:

  1. FastPIC's TTFT grows more slowly as QPS increases. Table 1 shows that TTFT rises for all methods, with the gap widening at high QPS. At QPS=32, FastPIC's TTFT is 17.01 s, 29.2%, 33.8%, and 31.1% lower than that of FR, CacheBlend-15%, and EPIC-32, respectively. TDC improves cross-request cache reuse, reducing prefill work and limiting TTFT growth.
  2. FastPIC achieves higher throughput at saturation. At QPS=32, FastPIC reaches 12,478 input tokens/s, corresponding to 1.20×, 1.17×, and 1.16× the throughput of FR, CacheBlend-15%, and EPIC-32, respectively.
  3. FastPIC approaches throughput saturation at a higher QPS. Under fixed compute and KV-cache memory capacity, baseline throughput largely plateaus beyond QPS≈16, whereas FastPIC continues to increase up to approximately QPS=24. This is consistent with TDC increasing cross-request KV reuse, reducing prefill demand, and alleviating resource contention.
  4. Effective cache reuse is a key contributor to end-to-end performance. TDC mitigates chunk-boundary shifts, increasing cross-request cache hits and reducing prefill computation. Although EPIC-32 recomputes fewer tokens, its fixed-length chunking misses boundary-shifted content and limits cache reuse; its TTFT therefore remains higher than FastPIC's. This shows that reducing recomputation alone does not necessarily lower end-to-end latency; improving cache reuse is also essential. Together with the ShareGPT results reported in the original submission, the MuSiQue results indicate that FastPIC's serving benefits extend across workloads from different task domains.
Reviewer_cqXaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies position-independent KV-cache reuse for multi-tenant LLM serving. Existing approaches face two challenges: fixed-length chunking is susceptible to boundary shifts when request prefixes change, reducing cache hit rates; directly reusing KV states across contexts can introduce attention deviations and degrade generation quality.

The authors propose FastPIC with two components. Token-Defined Chunking (TDC) determines variable-length chunk boundaries from token content to improve matching across requests. Dual-Path Token Selection (DPTS) jointly selects attention sinks and tokens with large KV-representation deviations for recomputation under a limited budget. Implemented on vLLM, the system is evaluated on three models and six datasets for the generation quality–TTFT trade-off, asynchronous serving throughput, and long-context scalability.

Critical strengths
  1. The paper presents a coherent system design addressing practical barriers to cache reuse. TDC mitigates chunk-boundary shifts caused by prefix changes, while DPTS addresses contextual representation deviations that remain even after a cache hit. The two components address cache matching and reuse quality, respectively.

  2. The evaluation covers both generation quality and serving performance. The authors compare quality and TTFT across multiple models on question answering, summarization, and long-context retrieval tasks, and further evaluate asynchronous concurrent serving. At QPS=32, the reported TTFT of FastPIC is approximately 53%–55% of that of the evaluated PIC baselines, with higher input throughput, suggesting practical potential.

Critical weaknesses
  1. Missing ablations prevent attribution of the gains to chunking versus token selection. Section 5 mainly compares complete systems, without isolating the benefits of TDC and DPTS or establishing that dual-path selection outperforms either path alone under the same recomputation budget.

  2. It is unclear how KV deviations are obtained online. Section 3.2, Equation (3), defines deviation using full-prefill KV states and states that this reference need not be computed online. However, Section 4.2 directly uses this deviation for ranking without explaining how the reference is obtained in practice. This determines whether selection requires additional recomputation and whether the reported TTFT includes the full overhead.

Questions for authors
  1. Regarding Weakness 1: Could you isolate the benefits of TDC and DPTS through component ablations and compare dual-path selection with each path alone under a fixed chunking strategy and recomputation budget?

  2. Regarding Weakness 2: How does the implementation obtain the reference values needed for KV-deviation ranking without performing a full prefill? Are the costs of obtaining these values and selecting tokens fully included in TTFT?

5 comments
AuthorsOpenReview ↗

TDC reduces latency, DPTS improves quality, and TTFT includes selection overhead.

FastPIC: Response to Reviewer cqXa

We thank the reviewer for the careful reading and constructive feedback. We address the two main concerns below.

1. Could you isolate the benefits of TDC and DPTS through component ablations and compare dual-path selection with each path alone under a fixed chunking strategy and recomputation budget?

Response:

Table 1: Component-Level Ablation

We evaluate Llama-70B on SAMSum, measuring ROUGE-L and TTFT. All configurations use the same workload, cache capacity, and hardware. The baseline configurations are CacheBlend-15% and EPIC-32, while FastPIC uses FastPIC-15%.

Chunking strategyToken-selection strategyROUGE-L ↑TTFT (s) ↓
Fixed-length chunkingCacheBlend-15%0.3793.58
Fixed-length chunkingEPIC-320.3873.39
TDCCacheBlend-15%0.3753.23
TDCEPIC-320.3823.15
Fixed-length chunkingFastPIC-15% (DPTS)0.4093.67
TDCFastPIC-15% (DPTS)0.4063.26

The component ablations support three conclusions:

  1. With token selection held fixed, TDC reduces TTFT, consistent with its design goal of mitigating boundary shifts and improving cache reuse.
  2. With chunking held fixed, DPTS improves generation quality by jointly targeting anomalous attention concentration and large KV-representation deviations.
  3. FastPIC lies on the quality–latency Pareto frontier among the evaluated configurations: none offers better quality without increasing latency, or lower latency without reducing quality.

Table 2: Token-Selection Strategy Comparison

All configurations use TDC and the same total recomputation budget as FastPIC-15%, varying only the token-selection strategy.

Token-selection strategyROUGE-L ↑TTFT (s) ↓
Attention-sink path only0.3913.19
KV-deviation path only0.3753.23
Dual-Path Token Selection (DPTS)0.4063.26

The comparison supports two conclusions:

  1. DPTS jointly recomputes tokens exhibiting anomalous attention concentration and those with large KV-representation deviations, addressing both attention routing and contextual representation errors to achieve higher generation quality.
  2. Single-path selection achieves slightly lower TTFT but recovers less generation quality after position-independent KV reuse. This supports the complementary benefits of the two DPTS paths.

2. How does the implementation obtain the reference values required for KV-deviation ranking without full prefill, and are the associated costs included in TTFT?

Response:
The full-prefill KV states in Equation (3) serve only as an ideal reference; online selection does not require full prefill across all layers. FastPIC instead uses early-layer KV deviations as a proxy for selecting tokens to recompute in subsequent layers.

This design relies on the cross-layer correlation of high-KV-deviation tokens. Residual connections cause token representations to evolve gradually across adjacent layers, while KV states are obtained by linearly projecting these representations. Thus, tokens with large deviations in an early layer may retain relatively large deviations in the next layer, providing useful candidates for subsequent-layer recomputation.

Specifically, FastPIC fully computes the first two layers and applies the normalized form of Equation (3) to compare the current and cached KV states at the second layer. It then ranks the cache-hit tokens and allocates the remaining recomputation budget accordingly. We use the second layer because, after positional alignment, first-layer KV states for identical tokens may not yet reflect differences in preceding context, whereas second-layer representations already incorporate contextual information from first-layer attention.

The computation of the first two layers, KV-deviation estimation, ranking, and token selection are all included in TTFT. Therefore, all latency results reported in the paper and supplementary material correspond to end-to-end latency from request arrival to the first generated token. TDC offsets these costs by reusing more cached chunks and reducing prefill computation.

Thank you for the additional ablations and clarification. The component ablations and comparisons with each selection path alone under the same budget help clarify the respective contributions of TDC and DPTS. The explanation of using second-layer KV deviations as an online proxy, with the computation of the first two layers and selection overhead included in TTFT, also clarifies the implementation.

My remaining concern is the reliability of the cross-layer proxy. The response provides a qualitative motivation, but does not quantify how reliably second-layer deviations identify tokens requiring recomputation in later layers. Please incorporate the new results and implementation details into the revised paper, distinguish the proxy from the ideal reference in Equation (3), and clarify its empirical support and limitations.

AuthorsOpenReview ↗

Layer-2 KV deviations provide a favorable quality–latency trade-off.

FastPIC: Response to Reviewer cqXa

We thank the reviewer for continuing to examine the reliability of the cross-layer proxy. The previous revision clarified TDC, DPTS, and the online selection procedure. In this revision, we add a proxy-layer ablation and discuss its results. The corresponding changes are included in the revised manuscript.

3. Reliability of the Second-Layer KV-Deviation Proxy

Reviewer comment: Although layer 2 is qualitatively motivated, can you provide quantitative ablation evidence that its KV deviations reliably approximate the ideal reference in Equation (3) for selecting tokens to recompute in later layers?

Response: The reviewer correctly notes that cross-layer correlation induced by residual connections provides qualitative motivation, but does not establish that the online proxy is equivalent to the ideal deviation in Equation (3), which is defined using full-prefill KV states. To evaluate the proxy’s practical effectiveness, we add an ablation that varies the layer used to estimate KV deviation and measures its impact on generation quality and latency. We fix the attention-statistics layers for the attention-sink path to layers 1–2 and keep TDC, DPTS, and the total recomputation budget unchanged. Only the KV-deviation proxy layer varies. TTFT includes the computation of the corresponding layer and online token-selection overhead.

Table 3: Ablation of the KV-Deviation Proxy Layer

Proxy layerROUGE-L ↑TTFT (s) ↓
Layer 10.3373.05
Layer 2 (FastPIC)0.4063.26
Layer 30.4083.44

The proxy-layer ablation shows that:

  1. Layer 1 does not yet capture preceding-context differences. Using its KV deviations yields a ROUGE-L score of 0.337.
  2. Layer 2 provides a more informative context-sensitive signal. It improves ROUGE-L by 0.069 over layer 1.
  3. Adding layer 3 yields little additional benefit. ROUGE-L remains nearly unchanged, increasing by only 0.002, while TTFT rises by 0.18 s. Layer 2 therefore offers a favorable quality–overhead trade-off. These results support its use as an online proxy for selecting tokens for subsequent-layer recomputation, without implying equivalence to the ideal deviation in Equation (3).

As a complementary response to another reviewer’s questions about parameter choices, we also report ablations of the attention-sink count kk and the number of attention-statistics layers ll. These experiments use Llama-70B on SAMSum with TDC and a 15% recomputation budget.

Supplementary Ablation: Number of Attention-Sink Tokens kk

kkROUGE-L ↑TTFT (s) ↓
40.3883.32
80.4063.26
160.4113.37

Compared with k=4k=4, k=8k=8 improves both quality and latency. Increasing kk to 16 yields only a 0.005 ROUGE-L gain over k=8k=8, while increasing TTFT by 0.11 s.

Supplementary Ablation: Number of Attention-Statistics Layers ll

llLayers used for attention statisticsROUGE-L ↑TTFT (s) ↓
1Layer 10.3593.07
2Layers 1–20.4063.26
3Layers 1–30.4143.41

From l=1l=1 to l=2l=2, ROUGE-L increases substantially; adding layer 3 yields a smaller gain and increases latency. In this ablation, the KV-deviation proxy remains fixed at layer 2.

Reviewer_wCjyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work focuses on PIC in LLM inference and proposes FastPIC, a position independent KV-cache reuse mechanism for multi-tenant LLM serving. FastPIC uses TDC to derive boundaries and cache identities from token content and further uses Dual-Path Token Selection to recompute attention sinks and high KV-deviation tokens under a limited budget.

Critical strengths
  1. The topic is well worth attention. PIC is currently an important optimization technique for LLM inference.
  2. TDC enables better cache reuse in multi-tenant LLM serving.
Critical weaknesses
  1. Hyperparameter choices lack clear justification.
  2. Some straightforward ideas are presented with unnecessary formulas, which makes this paper hard to follow.
  3. Online computation of KV-deviation scores is unclearly explained.
  4. The experimental results contradict the EPIC paper: EPIC only recalculates k tokens, a number much smaller than that of FastPIC, yet its TTFT is higher than that of FastPIC.
  5. The vLLM version used is too old -- it’s a version from two years ago. The current version has reached 0.31.0, and the benefits of FastPIC may diminish on a newer vLLM version.
Questions for authors
  1. How were TDC’s chunk lengths and masks chosen? Why 64/256/512 and 0x2FF/0x03F?
  2. How does DPTS compute KV-deviation scores online without full recomputation? (illustrate with eq. 3)
  3. Does a higher attention score necessarily indicate that a token should be recomputed firstly? How was (k=8) chosen and why did you focus only on the first two layers?
1 comment
AuthorsOpenReview ↗

FastPIC’s parameter choices balance quality and efficiency.

FastPIC: Response to Reviewer wCjy

We thank the reviewer for the careful reading and constructive feedback. We address the questions below.

1. Why were the chunk lengths 64/256/512 and masks 0x2FF/0x03F chosen?

The masks determine boundary probabilities, while the length constraints define the span of each phase. The boundary condition (f_i & M) == 0 requires all fingerprint bits selected by the mask to be zero. Assuming approximately uniform fingerprints, a mask with bb set bits yields a boundary probability of approximately 2−b2^{-b} per check:

MaskBinary representationSet bitsBoundary probability
Strict mask 0x2FF0010 1111 11119qS=1/512q_{\mathrm S}=1/512
Relaxed mask 0x03F0000 0011 11116qL=1/64q_{\mathrm L}=1/64

The length configuration keeps the strict phase shorter than its ideal mean waiting length and allows the relaxed phase to span at least one such waiting length, suppressing premature cuts and encouraging longer chunks to terminate:

\qquad B_{\max}-B_{\mathrm{norm}}\geq1/q_{\mathrm L}.$$ For **64/256/512**, boundary checks are skipped below 64 tokens; the strict phase spans $192<512$ tokens, the relaxed phase spans $256=4\times64$ tokens, and a cut is forced at 512. This configuration balances matching granularity and management overhead, while content-driven boundaries enable local resynchronization within repeated spans, mitigating the boundary shifts caused by fixed-length chunking. ## 2. How does DPTS compute the KV-deviation scores in Equation (3) online without full recomputation? **The full-prefill KV states in Equation (3) serve only as an ideal reference; online selection does not require full prefill across all layers.** FastPIC uses early-layer KV deviations as a proxy to select tokens for recomputation in subsequent layers. This approach relies on the cross-layer correlation of high-KV-deviation tokens. Transformers progressively update token hidden representations through residual connections, with input representations changing gradually across adjacent layers. Since KV states are linear projections of these representations, tokens with large KV deviations in an early layer may retain relatively large deviations in the next layer, providing candidates for subsequent-layer recomputation [2]. Specifically, FastPIC fully computes the first two layers and applies the normalized deviation measure in Equation (3) to compare the current and cached KV states at the second layer. It then ranks candidates and allocates the remaining recomputation budget accordingly. We use the second layer because, after positional alignment, first-layer KV states for identical tokens do not yet reflect differences in preceding context. Second-layer inputs already incorporate contextual information from first-layer attention, providing a signal of context-induced deviations. ## 3. Does a higher attention score necessarily justify recomputation? Why choose k=8 and use only the first two layers? **(1) High attention alone does not justify recomputation.** Cross-context reuse may concentrate attention on a few tokens, but high scores do not necessarily indicate KV errors. DPTS combines position-normalized cumulative attention and KV deviation to address both routing and representation errors within a shared budget. **(2) $k=8$ balances sink coverage and budget allocation.** Following observations that attention concentrates on few tokens [1], we vary $k$ on **Llama-70B with SAMSum**, keeping TDC and the FastPIC-15% budget fixed: | $k$ | ROUGE-L ↑ | TTFT (s) ↓ | | ----- | --------- | ---------- | | 4 | 0.388 | 3.32 | | **8** | **0.406** | **3.26** | | 16 | 0.411 | 3.37 | $k=8$ improves both metrics over $k=4$; increasing $k$ to 16 slightly improves quality but raises latency. **(3) The first two layers provide early signals while reusing existing computation.** Early-layer attention informs sink selection [2], and aggregating two layers reduces reliance on one layer’s distribution. FastPIC already fully computes these layers for second-layer KV deviations, so no additional full-layer computation is needed for selection. Under the same setup with $k=8$, we vary the number of attention layers $l$ while retaining second-layer KV deviations: | $l$ | Attention layers used | ROUGE-L ↑ | TTFT (s) ↓ | | ----- | --------------------- | --------- | ---------- | | 1 | Layer 1 | 0.359 | 3.07 | | **2** | **Layers 1–2** | **0.406** | **3.26** | | 3 | Layers 1–3 | 0.414 | 3.41 | ### References [1] EPIC: Efficient Position-Independent Caching for Serving Large Language Models, ICML. [2] QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global response to the shared KV-deviation concern and revised-manuscript updates.

Global Response: Online KV-Deviation Proxy and Its Computational Overhead

We thank the three reviewers for raising this shared question about online KV-deviation estimation. We have addressed each reviewer’s specific comments separately and provide a consolidated response to this common concern here. We carefully considered the reviewers’ feedback and have incorporated the clarifications and new experiments into the revised manuscript. The revised manuscript has been uploaded, with all changes highlighted for ease of review.

Shared concern: How does FastPIC compute KV-deviation scores online and select tokens for recomputation without full prefill? Can the ideal reference in Equation (3) be obtained from token content alone? Are the associated costs included in TTFT?

Response: The full-prefill KV states in Equation (3) serve only as an ideal reference; online selection does not require full prefill across all layers. FastPIC uses early-layer KV deviations as a proxy to select tokens for recomputation in subsequent layers.

This design is motivated by cross-layer correlations in KV deviations. Transformer residual connections update token representations incrementally across layers. When representations change gradually between adjacent layers, large KV deviations in an early layer may persist into subsequent layers because KV states are linear projections of these representations. Early-layer deviations can therefore guide the selection of candidates for later-layer recomputation.

Specifically, FastPIC fully computes the first two layers for the current request. It applies the normalized form of Equation (3) to compare the current second-layer KV states with the cached KV states after RoPE position correction, obtaining online deviation scores d^t\widehat{d}_t. It then uses these scores to allocate the remaining recomputation budget. We use the second layer because, after positional alignment, first-layer KV states for identical tokens do not yet reflect differences in preceding context, whereas second-layer inputs incorporate contextual information from first-layer attention. TTFT includes the computation of the relevant layers, deviation estimation, and online ranking and selection.

We conduct a KV-deviation proxy-layer ablation on Llama-70B with SAMSum, reporting ROUGE-L and end-to-end TTFT. All variants use FastPIC-15%, with TDC, DPTS, and attention statistics from layers 1–2 fixed; only the proxy layer varies.

Table 3: Ablation of the KV-Deviation Proxy Layer

Proxy layerROUGE-L ↑TTFT (s) ↓
Layer 10.3373.05
Layer 2 (FastPIC)0.4063.26
Layer 30.4083.44

The ablation shows that:

  1. Layer 2 provides a key context-sensitive signal. Using its deviations improves ROUGE-L by 0.069 over layer 1, whose KV states do not yet capture preceding-context differences.
  2. Adding layer 3 yields limited benefit. ROUGE-L increases by only 0.002, while TTFT rises by 0.18 s, indicating diminishing returns.
  3. Layer 2 offers a favorable quality–overhead trade-off. Although its proxy is not equivalent to the ideal deviation in Equation (3), the results show that it can efficiently guide token recomputation in subsequent layers.

We thank the reviewers again for their careful reading and constructive feedback. We hope this response addresses their concerns and would be pleased to clarify any remaining questions.

Area_Chair_6gZpOpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

Thank you to those who have already engaged with the authors' responses. The author-reviewer discussion closes on November 18 (AoE).

If you have not yet replied, please read the authors' response to your review and post a reply here before then: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews and the ongoing discussion is also welcome.

Thank you, Area Chair

Area_Chair_6gZpOpenReview ↗

Final reminder: discussion closes today (Nov 18 AoE)

The author-reviewer discussion closes today, November 18 (AoE).

Dear reviewers,

Thank you to those who have already responded. If you have not yet replied: Please read the authors' response and post a brief reply here before the deadline: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Your response is important for the final recommendation.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.