- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In LLM serving, KV cache reuse plays a critical role in minimizing Time to First Token (TTFT). However, current approaches to reusing cached chunks are rigid and explicitly require that the prompt prefix also match. Position-Independent Caching (PIC) removes this constraint and allows a request to reuse the cache from requests with a different prefix. To that end, the authors propose FastPIC, which introduces two complementary techniques. First, rather than chunking at arbitrary boundaries, it uses Token-Defined Chunking to find chunk boundaries that maximize chunk matches. Second, it proposes a dual-path token selection mechanism that decides, for each incoming request, whether to recompute or reuse the cached chunks. When an exact cached chunk does not exist, it identifies the tokens whose attention matters most and recomputes only those. Their evaluation on an array of datasets shows that FastPIC improves both accuracy and the throughput of the serving system.
Critical strengths
The paper correctly identifies the boundary shift problem in PIC and proposes a simple solution with Token-Defined Chunking.
Critical weaknesses
- Some key details are missing, and the manuscript reads like a set of instructions rather than an explanation of the intuition behind why the system was designed this way. For example, what are the implications of using a -bit fingerprint? What are practical values of , and how would an operator choose one? Similarly, the paper leaves the intuition behind the effectiveness of for the reader to work out.
- The novelty of the dual-path token selection seems thin. Prior work, some of which the authors already cite in that section, has done most of this. It would help to state the delta explicitly.
Questions for authors
- How many chunks does Token-Defined Chunking generate compared to simple count-based chunking?
- For content-driven deviation, how is computed from the content alone, without any forward passes?
- Could you share TTFT and throughput as a function of QPS for any of the datasets where you report F1 scores?
3 comments
Thank you for your response.
About the last question, I meant could you share the TTFT and throughput values for a pair of datasets as a function of QPS? You current response details the experimental setup instead.
Additional MuSiQue results validate FastPIC’s TTFT and throughput gains across QPS levels.
FastPIC: Response to Reviewer DDfL
We thank the reviewer for clarifying the experimental request and for your continued guidance in improving our work. We are pleased to have clarified the first two questions in our previous responses. We now provide TTFT and input throughput results at varying QPS and will incorporate them into the revised PDF, with all changes highlighted.
4. Dataset Serving Performance at Different QPS Levels
Reviewer comment: Could you share TTFT and throughput as a function of QPS for any of the datasets where you report F1 scores?
Response: We use MuSiQue, a dataset for which we report F1 scores, to construct an asynchronous workload on Llama 3.1-8B. By varying the request arrival rate, we measure end-to-end TTFT and input throughput for FastPIC-15%, FR, CacheBlend-15%, and EPIC-32 at different QPS levels. The results are shown below.
Table 1: TTFT (s) ↓ and input throughput (tokens/s) ↑ on MuSiQue at different QPS levels
| QPS | FR (TTFT / input throughput) | CacheBlend-15% (TTFT / input throughput) | EPIC-32 (TTFT / input throughput) | FastPIC-15% (TTFT / input throughput) |
|---|---|---|---|---|
| 1 | 2.91 / 7668 | 2.82 / 7645 | 2.75 / 7702 | 2.69 / 7715 |
| 8 | 5.18 / 9051 | 4.96 / 9203 | 4.83 / 9001 | 4.71 / 9316 |
| 16 | 9.18 / 10212 | 8.83 / 10324 | 9.05 / 10558 | 8.25 / 10808 |
| 18 | 10.24 / 10420 | 10.57 / 10452 | 10.08 / 10617 | 9.23 / 11305 |
| 20 | 11.87 / 10468 | 11.64 / 10296 | 12.05 / 10511 | 10.31 / 11447 |
| 22 | 12.24 / 10456 | 14.29 / 10686 | 13.12 / 10790 | 11.27 / 12041 |
| 24 | 13.96 / 10692 | 16.48 / 10802 | 15.18 / 10828 | 12.56 / 12391 |
| 26 | 15.78 / 10452 | 18.27 / 10729 | 17.09 / 10789 | 13.32 / 12241 |
| 28 | 17.97 / 10580 | 20.22 / 10779 | 19.69 / 10882 | 14.80 / 12513 |
| 30 | 21.27 / 10222 | 22.55 / 10692 | 22.15 / 10713 | 15.78 / 12453 |
| 32 | 24.03 / 10404 | 25.68 / 10648 | 24.69 / 10749 | 17.01 / 12478 |
Results analysis:
- FastPIC's TTFT grows more slowly as QPS increases. Table 1 shows that TTFT rises for all methods, with the gap widening at high QPS. At QPS=32, FastPIC's TTFT is 17.01 s, 29.2%, 33.8%, and 31.1% lower than that of FR, CacheBlend-15%, and EPIC-32, respectively. TDC improves cross-request cache reuse, reducing prefill work and limiting TTFT growth.
- FastPIC achieves higher throughput at saturation. At QPS=32, FastPIC reaches 12,478 input tokens/s, corresponding to 1.20×, 1.17×, and 1.16× the throughput of FR, CacheBlend-15%, and EPIC-32, respectively.
- FastPIC approaches throughput saturation at a higher QPS. Under fixed compute and KV-cache memory capacity, baseline throughput largely plateaus beyond QPS≈16, whereas FastPIC continues to increase up to approximately QPS=24. This is consistent with TDC increasing cross-request KV reuse, reducing prefill demand, and alleviating resource contention.
- Effective cache reuse is a key contributor to end-to-end performance. TDC mitigates chunk-boundary shifts, increasing cross-request cache hits and reducing prefill computation. Although EPIC-32 recomputes fewer tokens, its fixed-length chunking misses boundary-shifted content and limits cache reuse; its TTFT therefore remains higher than FastPIC's. This shows that reducing recomputation alone does not necessarily lower end-to-end latency; improving cache reuse is also essential. Together with the ShareGPT results reported in the original submission, the MuSiQue results indicate that FastPIC's serving benefits extend across workloads from different task domains.
We clarify TDC chunk counts, early-layer KV proxies, and complementary quality and serving evaluations.
FastPIC: Response to Reviewer DDfL
We thank the reviewer for the careful reading. We address the questions below.
1. How many chunks does Token-Defined Chunking generate compared to simple count-based chunking?
For a fixed hash mapping, TDC deterministically partitions a given prompt; chunk counts across inputs can be estimated using a probabilistic model. We illustrate this with
Token-Defined Chunking (256).The boundary condition
(f_i & M) == 0requires all fingerprint bits selected by the mask to be zero. Assuming approximately uniform fingerprints, a mask with m set bits yields a boundary probability of approximately 2−m per check:0x2FF0010 1111 11110x03F0000 0011 1111With (Bmin,Bnorm,Bmax)=(64,256,512), boundary checks are skipped below 64 tokens. The strict mask applies at lengths 64–255, the relaxed mask at 256–511, and a cut is forced at 512.
Ignoring truncation at the end of the prompt and assuming independent boundary checks, the chunk length L satisfies:
P(64≤L<256)=1−(512511)192≈31.3%,
P(256≤L<512)=(512511)192[1−(6463)256]≈67.5%,
P(L=512)=(512511)192(6463)256≈1.2%.
Let a=511/512 and b=63/64. The expected chunk length is:
\mathbb{E}[L] &=64+\sum_{j=64}^{511}P(L>j)\\ &=64+\sum_{r=1}^{192}a^r+a^{192}\sum_{r=1}^{256}b^r\\ &=64+\frac{a(1-a^{192})}{1-a} +a^{192}\frac{b(1-b^{256})}{1-b}\\ &\approx266.44\ \text{tokens}. \end{aligned}$$ Thus, for sufficiently long prompts of length $n$, the two methods produce comparable chunk counts under this model: $$N_{\mathrm{fixed}}=\left\lceil\frac{n}{256}\right\rceil, \qquad \mathbb{E}[N_{\mathrm{TDC}}]\approx\frac{n}{266.44}.$$ These are idealized estimates; actual counts depend on token content, fingerprint correlations, and the final partial chunk. **TDC’s primary benefit is facilitating local boundary resynchronization through content-driven chunking, thereby mitigating boundary shifts and improving cache reuse.** ## 2. For content-driven deviation, how is $Z^{*}$ computed from the content alone, without any forward passes? **The full-prefill KV states in Equation (3) serve only as an ideal reference; online selection does not require full prefill across all layers.** FastPIC uses early-layer KV deviations as a proxy to select tokens for recomputation in subsequent layers. This approach relies on the cross-layer correlation of high-KV-deviation tokens. Transformers progressively update token hidden representations through residual connections, with input representations changing gradually across adjacent layers. Since KV states are linear projections of these representations, tokens with large KV deviations in an early layer may retain relatively large deviations in the next layer, providing candidates for subsequent-layer recomputation. Specifically, FastPIC fully computes the first two layers and applies the normalized deviation measure in Equation (3) to compare the current and cached KV states at the second layer. It then ranks candidates and allocates the remaining recomputation budget accordingly. We use the second layer because, after positional alignment, first-layer KV states for identical tokens do not yet reflect differences in preceding context. Second-layer inputs already incorporate contextual information from first-layer attention, providing a signal of context-induced deviations. ## 3. Could you share TTFT and throughput as a function of QPS for any of the datasets where you report F1 scores? In principle, the datasets used to evaluate the F1–TTFT trade-off can also be organized into asynchronous workloads. We use two complementary settings to evaluate **single-request quality–latency trade-offs** and **concurrent multi-tenant serving performance**. For question-answering and long-context retrieval benchmarks, we process requests sequentially to minimize queuing and concurrent resource contention, allowing us to more directly assess how KV reuse and selective recomputation affect generation quality and TTFT. For concurrent serving, we use instructions from real user interactions in ShareGPT, generate request arrivals according to a Poisson process, and vary the arrival rate to examine queuing, resource contention, and throughput under different loads. This design uses benchmark reference answers to assess quality preservation and the diversity of real user instructions to evaluate serving performance, providing complementary assessments of FastPIC.