- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes Inlay, a decoding runtime for position independent KV caching. Existing PIC systems reuse cached chunk KV across different prefixes through selective repair, but they still materialize a private KV copy per request and read these copies independently at every decode step. Inlay instead separates each request's KV into shared immutable Base KV, per request Repair KV, and private Decode KV. Queries that reference the same Base KV are grouped to share loads and batch attention, with per request masks and position offsets, and the partial outputs are merged. A CPU side planner chooses between shared and private execution for each chunk under a KV memory budget.
As I read it, the contributions are the storage decomposition, the cost aware hybrid decode execution, and the calibrated cost model with plan reuse. The evaluation runs on an A100 with Qwen3-8B and Llama-3.1-8B configurations: attention kernel latency against FlashInfer and PAT, integration with four PIC methods on single layer workloads, and estimated resident request capacity under KV budgets.
Critical strengths
-
The motivation is measured rather than assumed. Fig. 1 shows that at a target batch size of 64, non-prefix reuse accounts for roughly 22 to 74% of reusable logical KV accesses across the five workloads, and that the fraction generally grows with the batch. I found the proposed split into a shared Base KV, a token level Repair KV, and a private Decode KV a complete and consistent answer to this setting.
-
The planner is treated as a first class component. It selects execution modes over the full batch using access rates and task costs calibrated with microbenchmarks, and Appendix Fig. 6 checks its predictions against measured GPU times. I appreciated that execution mode selection is driven by a tested cost model instead of heuristics.
Critical weaknesses
Weakness 1: The performance gains are not demonstrated under serving conditions
-
Fig. 1c shows that shared KV reads can be slower than private reads once partitioning overhead kicks in, which is why the planner can fall back to Private mode and pack per-request buffers that duplicate shared data. But the 48.5% capacity gain in Fig. 5a is computed under a budget that excludes packed execution copies and temporary workspace, while Sec. 3.3 says the planner counts packed capacity at run time. The same system gets two different footprints depending on the convention, and the gap depends on how often the planner falls back, which is not reported. The memory gain is also smallest where sharing already exists: for MEPIC the authors note resident memory is nearly identical with and without Inlay, benefits shrink as the repair budget grows, and the capacity number sits at the smallest repair ratio of 5%.
-
The latency results have a scope problem. Everything in the main text is one layer: synthetic Q/K/V for the kernel benchmarks, and a single layer Qwen3-8B configuration reporting attention kernel latency only. I could not find an end to end result, a multi layer run, or a serving engine integration in the main text. The CacheBlend TPOT comparison in Appendix F is the one full model experiment, and the main text never points the reader to it.
Weakness 2: The identical output claim is not checked
- The merge is stated to be equivalent to attention over the effective KV cache up to floating point and positional transformation error. I agree with the merge argument in principle, but the positional transformation error appears exactly once, in the equivalence statement itself, and is never bounded or measured. I also looked for a direct check, for example a logit comparison against the private baseline or an output quality evaluation, and could not find one. Lossless sharing is one of the main reasons to prefer this design, so an equivalence measurement is the minimum evidence the claim needs.
Questions for authors
Question 1. Fig. 5a computes the capacity gain under a budget that excludes packed execution copies and temporary workspace, while Section 3.3 states that the planner counts packed capacity at run time. What fraction of chunks does the planner assign to Private or hybrid mode under the evaluated workloads, and what is the resident capacity when packed copies are included?
Question 2. The reuse statistics in Fig. 1 count logical accesses, but shared reads only pay off when concurrent requests in the same batch reference the same Base chunk. How often does such co-occurrence arise under dynamic request arrival, for example in a continuous batching simulation over the same workloads?
Question 3. The equivalence statement is qualified by a positional transformation error that does not appear anywhere else in the paper. What produces this error, how large is it, and could you provide a direct equivalence check against the private baseline, for example maximum logit difference or perplexity on a shared prompt set?
Question 4. The capacity number uses the smallest repair ratio of 5%, and the paper does not discuss output quality at any repair ratio. How should this gain be read against the output quality of the underlying PIC method at such an aggressive repair setting?
Question 5. The measurements in the main text are single layer, and the only full model result is the CacheBlend TPOT comparison in Appendix F. Could you provide an end to end result with the full model and multiple layers, even for one workload and one PIC method?
14 comments
(Due to the response length limit, this is Part 2 of 3.)
[W2, Q3: Position-transform error and numerical agreement]
Section 4.1 explicitly states that this experiment covers all model layers. The mean reported below averages the errors measured separately for each layer. The two position transforms are equivalent in exact arithmetic, and their errors are of the same order of magnitude after BF16 rounding. We will add this experiment to the revised paper.
The experimental setup and detailed results follow.
Setup. The private path rotates keys to their target positions. Inlay’s shared path leaves Base KV unchanged and applies the inverse offset rotation to queries. We use identical PIC inputs for both paths across two models and three datasets, round the transformed values to BF16, and then compute attention in FP64. We compute the reference output entirely in FP64. We calculate the relative error (\lVert o-o_{\mathrm{ref}}\rVert_2/\lVert o_{\mathrm{ref}}\rVert_2) for each layer and pool the error samples across all layers, models, and datasets to compute the mean, P90, and P99.
| Path | Mean | P90 | P99 |
|---|---|---|---|
| Private baseline | 3.49 × 10⁻⁴ | 7.32 × 10⁻⁴ | 1.42 × 10⁻³ |
| Inlay shared path | 3.96 × 10⁻⁴ | 7.02 × 10⁻⁴ | 1.26 × 10⁻³ |
Results. Inlay has a slightly higher mean error and lower P90 and P99 errors. We also compared Inlay and FlashInfer’s BF16 attention outputs layer by layer on Qwen3-8B with CacheBlend. The mean cosine similarity was 0.99999383, and the minimum was 0.99992826. We report generation quality in Q4.
[Q1: Capacity including execution buffers]
W1(a) reports the fraction of shareable Base chunks that the planner assigns to Private execution across all 72 configurations in Fig. 3. With packed KV copies included, the largest successful batch size on the same 40 GB A100 was 64 for Inlay and 48 for both FlashInfer and PAT, a 33.3% increase.
Setup. We swept batch size until OOM using the full Qwen3-8B model and 4,096-token ToolBench inputs. Each run loads the KV cache and executes six decode steps across all 36 layers. Peak PyTorch allocation includes model weights, KV caches, execution buffers (including packed KV copies), and workspace. It excludes unused allocator reservations and allocations outside PyTorch.
| Method | Largest successful batch | Peak PyTorch allocation (GiB) | Next batch tested |
|---|---|---|---|
| FlashInfer | 48 | 36.30 | 64 (OOM) |
| PAT | 48 | 36.17 | 64 (OOM) |
| Inlay | 64 | 32.54 | 96 (OOM) |
Inlay uses hybrid execution in this sweep, with packed KV copies occupying 2.835 GiB on average.
[Q2: Same-batch sharing under dynamic arrivals]
Same-batch sharing persists under dynamic arrivals. Even at a load of 0.25, more than 78% of cache-unit accesses on ToolBench and MultiDoc2Dial are shared within the batch.
Setup. We simulated continuous batching on traces from all three workloads with Poisson arrivals, FIFO admission, and a maximum batch size of 64. Load is (\rho=\lambda/\mu), where (\lambda) is the arrival rate and (\mu) is the measured request throughput at batch size 64. Each result is averaged over three random seeds.
Metric. The sharing rate is the fraction of cache-unit accesses for which the unit is referenced by at least two distinct active requests. A cache unit is a 16-token block of tool or document content for ToolBench and MultiDoc2Dial, and a content-hashed prompt component for RAGPulse.
| Workload | Load = 0.25 | 0.50 | 0.90 |
|---|---|---|---|
| ToolBench | 78.25% | 81.95% | 84.19% |
| MultiDoc2Dial | 91.77% | 94.68% | 96.34% |
| RAGPulse | 38.56% | 42.90% | 46.92% |
Results. Sharing increases with load on all three workloads, as concurrent requests have more opportunities to reference the same cache units. RAGPulse has the lowest sharing rate at every tested load. We will update Figure 1 in the revision to include these results under dynamic arrivals.
(Due to the response length limit, this is Part 3 of 3.)
[Q4: Repair ratio and output quality]
Figure 4 already reports how attention-kernel latency and resident KV usage vary with the repair setting. Figure 5(a) uses 5%, the smallest repair ratio in that sweep, for its resident-capacity estimate. To quantify output quality at the same repair ratios, we added a comparison against full prefill. At 5% repair, CacheBlend and QCFuse retain 87.93–96.63% of full-prefill F1 across the three datasets, with layer-averaged attention cosine similarity of at least 96.88%. We will include this quality comparison in the revised paper.
Performance across repair settings. Figure 4 sweeps repair ratios from 5% to 50% for CacheBlend and QCFuse and repair windows from 4 to 128 tokens for EPIC. Larger repair budgets make more KV private, so Inlay’s latency approaches that of the baseline. At the largest tested setting for each method, Inlay achieves comparable latency while reducing resident KV usage by 21.1% for CacheBlend, 40.92% for QCFuse, and 33.09% for EPIC. At both 15% and 30% repair, the largest successful batch tested was 64 for Inlay and 48 for both baselines.
Effect of Inlay on output quality. The PIC method determines which tokens to repair and how many. Given the same PIC decomposition, Inlay and the request-private baseline use identical logical KV. In a full-model, teacher-forced comparison, their top-1 predictions differed on 0/512, 2/512, and 0/512 tokens across the three datasets. These disagreement counts were no higher than those between PAT and FlashInfer.
Quality evaluation. We evaluate CacheBlend and QCFuse on Qwen3-8B in BF16 using full prefill as the reference. Following the PIC configuration, we retain full KV in the first three layers and compute the mean attention cosine similarity over the remaining layers.
Table 1. Generation quality (F1 as a percentage of full-prefill F1)
| Method | Repair setting | ToolBench | WikiMQA | MultiDoc2Dial |
|---|---|---|---|---|
| CacheBlend | 1% | 88.37% | 82.74% | 90.73% |
| CacheBlend | 3% | 90.51% | 88.40% | 90.30% |
| CacheBlend | 5% | 94.82% | 94.14% | 89.96% |
| CacheBlend | 10% | 96.24% | 95.36% | 91.87% |
| CacheBlend | 20% | 96.80% | 96.15% | 94.81% |
| CacheBlend | 30% | 92.87% | 96.88% | 94.92% |
| CacheBlend | 50% | 94.08% | 96.57% | 96.39% |
| QCFuse | 1% | 85.65% | 89.47% | 92.82% |
| QCFuse | 3% | 93.02% | 96.77% | 87.83% |
| QCFuse | 5% | 96.12% | 96.63% | 87.93% |
| QCFuse | 10% | 96.97% | 97.13% | 86.42% |
| QCFuse | 20% | 97.59% | 97.44% | 92.31% |
| QCFuse | 30% | 97.08% | 98.99% | 95.09% |
| QCFuse | 50% | 98.84% | 97.58% | 99.24% |
Table 2. Cosine similarity to full-prefill attention outputs, averaged over layers after the first three
| Method | Repair setting | ToolBench | WikiMQA | MultiDoc2Dial |
|---|---|---|---|---|
| CacheBlend | 1% | 96.66% | 97.09% | 90.49% |
| CacheBlend | 3% | 98.11% | 97.55% | 96.37% |
| CacheBlend | 5% | 98.11% | 97.77% | 96.88% |
| CacheBlend | 10% | 98.15% | 98.26% | 97.25% |
| CacheBlend | 20% | 98.78% | 98.98% | 97.48% |
| CacheBlend | 30% | 99.21% | 99.34% | 97.89% |
| CacheBlend | 50% | 99.69% | 99.76% | 98.78% |
| QCFuse | 1% | 99.63% | 99.07% | 95.94% |
| QCFuse | 3% | 99.72% | 99.16% | 97.82% |
| QCFuse | 5% | 99.78% | 99.23% | 98.35% |
| QCFuse | 10% | 99.83% | 99.37% | 98.76% |
| QCFuse | 20% | 99.83% | 99.55% | 98.93% |
| QCFuse | 30% | 99.88% | 99.69% | 99.51% |
| QCFuse | 50% | 99.86% | 99.87% | 99.59% |
(Due to the response length limit, this is Part 1 of 3.)
Thank you for your insightful comments and detailed feedback on our evaluation. We hope the additional evaluations and responses help address your concerns. We would be grateful if you would reconsider your rating in light of these results.
[W1(a): Memory accounting and private fallback]
Figure 5(a) estimates capacity from resident KV, and the execution-memory budget in §3.3 also includes packed KV copies and temporary workspace. We will make both scopes explicit in the revision. Figure 4 already reports resident KV usage and attention-kernel latency across the tested repair settings. We additionally report private fallback rates for all configurations in Fig. 3.
Benefits across repair settings. Figure 4(a)–(c) covers selection ratios of 5%–50% for CacheBlend and QCFuse and window sizes of 4–128 tokens for EPIC. Even at the largest tested setting for each method, Inlay reduces resident KV usage relative to request-private KV by 21.08% for CacheBlend, 33.09% for EPIC, and 40.92% for QCFuse. MEPIC already shares chunk KV, so its resident KV usage remains nearly identical with and without Inlay, and its main gain is lower decode attention latency.
Private fallback. We analyzed all 72 configurations in Fig. 3, spanning three datasets, two models, three batch sizes, and four attention settings. For each configuration, we computed the fraction of shareable Base chunks that the planner assigned to Private mode, then averaged these fractions by dataset.
RAGPulse has the highest fallback ratio, consistent with its lower within-batch sharing rate (see Q2).
[W1(b), Q5: Full-model evaluation]
The main-text capacity estimate already uses resident KV measurements summed across all model layers (§4.4, Fig. 5(a)), and the submitted paper reports full-model decoding TPOT for CacheBlend with and without Inlay across three workloads and two batch sizes (Appendix F, Fig. 8(c)). We will clarify the scope of the single-layer benchmarks and full-model measurements in the revision.
The additional full-model experiment and its results are detailed below.
Setup. We integrated Inlay into vLLM 0.20.0 and ran all 36 layers of Qwen3-8B on ToolBench using a 40 GB A100, with a batch size of 32, 4,096 input tokens, and 128 output tokens per request. Both systems use the same precomputed PIC caches, PIC decomposition, and fixed output token sequence. TTFT includes CPU planning, KV preparation, and first-step initialization. TPOT measures average latency per output token after the first token. We compute the mean, P90, and P99 across requests.
Results. Inlay reduces both mean and P99 TTFT and TPOT while using 32.55% less peak device memory. The reported latencies include per-step CPU preparation, which takes 8.77 ms at the median and 14.58 ms at P99.
Memory. At the device peak, Inlay’s KV cache and workspace account for 7.614 GiB, including Base KV (0.439 GiB), Repair/private prompt KV (4.661 GiB), Decode KV (0.563 GiB), packed KV copies (0.732 GiB), pool slack (0.719 GiB), and workspace (0.500 GiB). Peak device memory includes model weights, KV pools, workspace, and the CUDA context. Each system uses its own memory-pool configuration.
Capacity. In the separate capacity sweep on the same A100, with execution buffers included, the largest successful batch size was 64 for Inlay and 48 for both FlashInfer and PAT (see Q1).