acceptodds
Under review as a conference paper at ICLR 2027

Inlay: Accelerating Position-Independent KV Caching with Sparse Repairs and Shared Decode Kernels

Abstract

Position-independent KV caching (PIC) reduces repeated computation by reusing cached text chunks across different contexts. However, this reuse does not automatically yield compact KV storage or efficient attention execution during decoding. Request-specific repairs can lead to duplicated KV states, while request-oriented attention repeatedly loads the same shared data. We present Inlay, a serving runtime that combines compact KV storage with shared attention execution for PIC decoding. Inlay separates immutable shared Base KV, token-granular request- and layer-specific Repair KV, and private append-only Decode KV without changing the underlying PIC method’s repair policy. Its attention kernels reuse loaded Base KV across compatible queries, apply request-specific repair masks and position adjustments, and merge partial attention outputs. A cost-aware CPU planner selects between shared execution and private KV packing for each chunk under a memory budget. Valid plans, metadata, and packed prompt KV are reused across decode steps to reduce preparation overhead. On NVIDIA A100 attention microbenchmarks using Qwen3-8B and Llama-3.1-8B configurations, Inlay achieves median kernel speedups of 1.24× over FlashInfer and 1.32× over PAT across the tested workloads and attention geometries. In single-layer integrations with CacheBlend, EPIC, and QCFuse, Inlay achieves median kernel speedups of 1.43–1.98× and median resident KV memory reductions of 50.1–63.6% relative to their request-private FlashInfer baselines, while preserving the same logical inputs and PIC repair results. Resident KV measurements exclude additional packed execution copies.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
gpu kernel, HBM, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NY1eOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes Inlay, a decoding runtime for position independent KV caching. Existing PIC systems reuse cached chunk KV across different prefixes through selective repair, but they still materialize a private KV copy per request and read these copies independently at every decode step. Inlay instead separates each request's KV into shared immutable Base KV, per request Repair KV, and private Decode KV. Queries that reference the same Base KV are grouped to share loads and batch attention, with per request masks and position offsets, and the partial outputs are merged. A CPU side planner chooses between shared and private execution for each chunk under a KV memory budget.

As I read it, the contributions are the storage decomposition, the cost aware hybrid decode execution, and the calibrated cost model with plan reuse. The evaluation runs on an A100 with Qwen3-8B and Llama-3.1-8B configurations: attention kernel latency against FlashInfer and PAT, integration with four PIC methods on single layer workloads, and estimated resident request capacity under KV budgets.

Critical strengths
  • The motivation is measured rather than assumed. Fig. 1 shows that at a target batch size of 64, non-prefix reuse accounts for roughly 22 to 74% of reusable logical KV accesses across the five workloads, and that the fraction generally grows with the batch. I found the proposed split into a shared Base KV, a token level Repair KV, and a private Decode KV a complete and consistent answer to this setting.

  • The planner is treated as a first class component. It selects execution modes over the full batch using access rates and task costs calibrated with microbenchmarks, and Appendix Fig. 6 checks its predictions against measured GPU times. I appreciated that execution mode selection is driven by a tested cost model instead of heuristics.

Critical weaknesses

Weakness 1: The performance gains are not demonstrated under serving conditions

  • Fig. 1c shows that shared KV reads can be slower than private reads once partitioning overhead kicks in, which is why the planner can fall back to Private mode and pack per-request buffers that duplicate shared data. But the 48.5% capacity gain in Fig. 5a is computed under a budget that excludes packed execution copies and temporary workspace, while Sec. 3.3 says the planner counts packed capacity at run time. The same system gets two different footprints depending on the convention, and the gap depends on how often the planner falls back, which is not reported. The memory gain is also smallest where sharing already exists: for MEPIC the authors note resident memory is nearly identical with and without Inlay, benefits shrink as the repair budget grows, and the capacity number sits at the smallest repair ratio of 5%.

  • The latency results have a scope problem. Everything in the main text is one layer: synthetic Q/K/V for the kernel benchmarks, and a single layer Qwen3-8B configuration reporting attention kernel latency only. I could not find an end to end result, a multi layer run, or a serving engine integration in the main text. The CacheBlend TPOT comparison in Appendix F is the one full model experiment, and the main text never points the reader to it.

Weakness 2: The identical output claim is not checked

  • The merge is stated to be equivalent to attention over the effective KV cache up to floating point and positional transformation error. I agree with the merge argument in principle, but the positional transformation error appears exactly once, in the equivalence statement itself, and is never bounded or measured. I also looked for a direct check, for example a logit comparison against the private baseline or an output quality evaluation, and could not find one. Lossless sharing is one of the main reasons to prefer this design, so an equivalence measurement is the minimum evidence the claim needs.
Questions for authors

Question 1. Fig. 5a computes the capacity gain under a budget that excludes packed execution copies and temporary workspace, while Section 3.3 states that the planner counts packed capacity at run time. What fraction of chunks does the planner assign to Private or hybrid mode under the evaluated workloads, and what is the resident capacity when packed copies are included?

Question 2. The reuse statistics in Fig. 1 count logical accesses, but shared reads only pay off when concurrent requests in the same batch reference the same Base chunk. How often does such co-occurrence arise under dynamic request arrival, for example in a continuous batching simulation over the same workloads?

Question 3. The equivalence statement is qualified by a positional transformation error that does not appear anywhere else in the paper. What produces this error, how large is it, and could you provide a direct equivalence check against the private baseline, for example maximum logit difference or perplexity on a shared prompt set?

Question 4. The capacity number uses the smallest repair ratio of 5%, and the paper does not discuss output quality at any repair ratio. How should this gain be read against the output quality of the underlying PIC method at such an aggressive repair setting?

Question 5. The measurements in the main text are single layer, and the only full model result is the CacheBlend TPOT comparison in Appendix F. Could you provide an end to end result with the full model and multiple layers, even for one workload and one PIC method?

14 comments

(Due to the response length limit, this is Part 1 of 3.)

Thank you for your insightful comments and detailed feedback on our evaluation. We hope the additional evaluations and responses help address your concerns. We would be grateful if you would reconsider your rating in light of these results.

[W1(a): Memory accounting and private fallback]

Figure 5(a) estimates capacity from resident KV, and the execution-memory budget in §3.3 also includes packed KV copies and temporary workspace. We will make both scopes explicit in the revision. Figure 4 already reports resident KV usage and attention-kernel latency across the tested repair settings. We additionally report private fallback rates for all configurations in Fig. 3.

Benefits across repair settings. Figure 4(a)–(c) covers selection ratios of 5%–50% for CacheBlend and QCFuse and window sizes of 4–128 tokens for EPIC. Even at the largest tested setting for each method, Inlay reduces resident KV usage relative to request-private KV by 21.08% for CacheBlend, 33.09% for EPIC, and 40.92% for QCFuse. MEPIC already shares chunk KV, so its resident KV usage remains nearly identical with and without Inlay, and its main gain is lower decode attention latency.

Private fallback. We analyzed all 72 configurations in Fig. 3, spanning three datasets, two models, three batch sizes, and four attention settings. For each configuration, we computed the fraction of shareable Base chunks that the planner assigned to Private mode, then averaged these fractions by dataset.

DatasetPrivate fallback ratio
ToolBench11.87%
MultiDoc2Dial4.34%
RAGPulse30.08%

RAGPulse has the highest fallback ratio, consistent with its lower within-batch sharing rate (see Q2).

[W1(b), Q5: Full-model evaluation]

The main-text capacity estimate already uses resident KV measurements summed across all model layers (§4.4, Fig. 5(a)), and the submitted paper reports full-model decoding TPOT for CacheBlend with and without Inlay across three workloads and two batch sizes (Appendix F, Fig. 8(c)). We will clarify the scope of the single-layer benchmarks and full-model measurements in the revision.

The additional full-model experiment and its results are detailed below.

Setup. We integrated Inlay into vLLM 0.20.0 and ran all 36 layers of Qwen3-8B on ToolBench using a 40 GB A100, with a batch size of 32, 4,096 input tokens, and 128 output tokens per request. Both systems use the same precomputed PIC caches, PIC decomposition, and fixed output token sequence. TTFT includes CPU planning, KV preparation, and first-step initialization. TPOT measures average latency per output token after the first token. We compute the mean, P90, and P99 across requests.

MetricFlashInfer (private KV)InlayReduction
TTFT mean (s)10.9659.9049.68%
TTFT P90 (s)18.27216.19511.37%
TTFT P99 (s)20.40617.81512.70%
TPOT mean (ms)105.9089.3215.66%
TPOT P90 (ms)106.4592.7112.91%
TPOT P99 (ms)128.8799.9722.43%
Device peak memory (GiB)38.17825.75232.55%

Results. Inlay reduces both mean and P99 TTFT and TPOT while using 32.55% less peak device memory. The reported latencies include per-step CPU preparation, which takes 8.77 ms at the median and 14.58 ms at P99.

Memory. At the device peak, Inlay’s KV cache and workspace account for 7.614 GiB, including Base KV (0.439 GiB), Repair/private prompt KV (4.661 GiB), Decode KV (0.563 GiB), packed KV copies (0.732 GiB), pool slack (0.719 GiB), and workspace (0.500 GiB). Peak device memory includes model weights, KV pools, workspace, and the CUDA context. Each system uses its own memory-pool configuration.

Capacity. In the separate capacity sweep on the same A100, with execution buffers included, the largest successful batch size was 64 for Inlay and 48 for both FlashInfer and PAT (see Q1).

(Due to the response length limit, this is Part 2 of 3.)

[W2, Q3: Position-transform error and numerical agreement]

Section 4.1 explicitly states that this experiment covers all model layers. The mean reported below averages the errors measured separately for each layer. The two position transforms are equivalent in exact arithmetic, and their errors are of the same order of magnitude after BF16 rounding. We will add this experiment to the revised paper.

The experimental setup and detailed results follow.

Setup. The private path rotates keys to their target positions. Inlay’s shared path leaves Base KV unchanged and applies the inverse offset rotation to queries. We use identical PIC inputs for both paths across two models and three datasets, round the transformed values to BF16, and then compute attention in FP64. We compute the reference output entirely in FP64. We calculate the relative error (\lVert o-o_{\mathrm{ref}}\rVert_2/\lVert o_{\mathrm{ref}}\rVert_2) for each layer and pool the error samples across all layers, models, and datasets to compute the mean, P90, and P99.

PathMeanP90P99
Private baseline3.49 × 10⁻⁴7.32 × 10⁻⁴1.42 × 10⁻³
Inlay shared path3.96 × 10⁻⁴7.02 × 10⁻⁴1.26 × 10⁻³

Results. Inlay has a slightly higher mean error and lower P90 and P99 errors. We also compared Inlay and FlashInfer’s BF16 attention outputs layer by layer on Qwen3-8B with CacheBlend. The mean cosine similarity was 0.99999383, and the minimum was 0.99992826. We report generation quality in Q4.

[Q1: Capacity including execution buffers]

W1(a) reports the fraction of shareable Base chunks that the planner assigns to Private execution across all 72 configurations in Fig. 3. With packed KV copies included, the largest successful batch size on the same 40 GB A100 was 64 for Inlay and 48 for both FlashInfer and PAT, a 33.3% increase.

Setup. We swept batch size until OOM using the full Qwen3-8B model and 4,096-token ToolBench inputs. Each run loads the KV cache and executes six decode steps across all 36 layers. Peak PyTorch allocation includes model weights, KV caches, execution buffers (including packed KV copies), and workspace. It excludes unused allocator reservations and allocations outside PyTorch.

MethodLargest successful batchPeak PyTorch allocation (GiB)Next batch tested
FlashInfer4836.3064 (OOM)
PAT4836.1764 (OOM)
Inlay6432.5496 (OOM)

Inlay uses hybrid execution in this sweep, with packed KV copies occupying 2.835 GiB on average.

[Q2: Same-batch sharing under dynamic arrivals]

Same-batch sharing persists under dynamic arrivals. Even at a load of 0.25, more than 78% of cache-unit accesses on ToolBench and MultiDoc2Dial are shared within the batch.

Setup. We simulated continuous batching on traces from all three workloads with Poisson arrivals, FIFO admission, and a maximum batch size of 64. Load is (\rho=\lambda/\mu), where (\lambda) is the arrival rate and (\mu) is the measured request throughput at batch size 64. Each result is averaged over three random seeds.

Metric. The sharing rate is the fraction of cache-unit accesses for which the unit is referenced by at least two distinct active requests. A cache unit is a 16-token block of tool or document content for ToolBench and MultiDoc2Dial, and a content-hashed prompt component for RAGPulse.

WorkloadLoad = 0.250.500.90
ToolBench78.25%81.95%84.19%
MultiDoc2Dial91.77%94.68%96.34%
RAGPulse38.56%42.90%46.92%

Results. Sharing increases with load on all three workloads, as concurrent requests have more opportunities to reference the same cache units. RAGPulse has the lowest sharing rate at every tested load. We will update Figure 1 in the revision to include these results under dynamic arrivals.

(Due to the response length limit, this is Part 3 of 3.)

[Q4: Repair ratio and output quality]

Figure 4 already reports how attention-kernel latency and resident KV usage vary with the repair setting. Figure 5(a) uses 5%, the smallest repair ratio in that sweep, for its resident-capacity estimate. To quantify output quality at the same repair ratios, we added a comparison against full prefill. At 5% repair, CacheBlend and QCFuse retain 87.93–96.63% of full-prefill F1 across the three datasets, with layer-averaged attention cosine similarity of at least 96.88%. We will include this quality comparison in the revised paper.

Performance across repair settings. Figure 4 sweeps repair ratios from 5% to 50% for CacheBlend and QCFuse and repair windows from 4 to 128 tokens for EPIC. Larger repair budgets make more KV private, so Inlay’s latency approaches that of the baseline. At the largest tested setting for each method, Inlay achieves comparable latency while reducing resident KV usage by 21.1% for CacheBlend, 40.92% for QCFuse, and 33.09% for EPIC. At both 15% and 30% repair, the largest successful batch tested was 64 for Inlay and 48 for both baselines.

Effect of Inlay on output quality. The PIC method determines which tokens to repair and how many. Given the same PIC decomposition, Inlay and the request-private baseline use identical logical KV. In a full-model, teacher-forced comparison, their top-1 predictions differed on 0/512, 2/512, and 0/512 tokens across the three datasets. These disagreement counts were no higher than those between PAT and FlashInfer.

Quality evaluation. We evaluate CacheBlend and QCFuse on Qwen3-8B in BF16 using full prefill as the reference. Following the PIC configuration, we retain full KV in the first three layers and compute the mean attention cosine similarity over the remaining layers.

Table 1. Generation quality (F1 as a percentage of full-prefill F1)

MethodRepair settingToolBenchWikiMQAMultiDoc2Dial
CacheBlend1%88.37%82.74%90.73%
CacheBlend3%90.51%88.40%90.30%
CacheBlend5%94.82%94.14%89.96%
CacheBlend10%96.24%95.36%91.87%
CacheBlend20%96.80%96.15%94.81%
CacheBlend30%92.87%96.88%94.92%
CacheBlend50%94.08%96.57%96.39%
QCFuse1%85.65%89.47%92.82%
QCFuse3%93.02%96.77%87.83%
QCFuse5%96.12%96.63%87.93%
QCFuse10%96.97%97.13%86.42%
QCFuse20%97.59%97.44%92.31%
QCFuse30%97.08%98.99%95.09%
QCFuse50%98.84%97.58%99.24%

Table 2. Cosine similarity to full-prefill attention outputs, averaged over layers after the first three

MethodRepair settingToolBenchWikiMQAMultiDoc2Dial
CacheBlend1%96.66%97.09%90.49%
CacheBlend3%98.11%97.55%96.37%
CacheBlend5%98.11%97.77%96.88%
CacheBlend10%98.15%98.26%97.25%
CacheBlend20%98.78%98.98%97.48%
CacheBlend30%99.21%99.34%97.89%
CacheBlend50%99.69%99.76%98.78%
QCFuse1%99.63%99.07%95.94%
QCFuse3%99.72%99.16%97.82%
QCFuse5%99.78%99.23%98.35%
QCFuse10%99.83%99.37%98.76%
QCFuse20%99.83%99.55%98.93%
QCFuse30%99.88%99.69%99.51%
QCFuse50%99.86%99.87%99.59%
Reviewer_xrZzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Inlay, a decode-side runtime for position-independent KV caching (PIC). It separates immutable shared Base KV, token-granular request- and layer-specific Repair KV, and private append-only Decode KV. Its attention execution reuses Base KV loads across compatible queries while accounting for repair masks and positional offsets. A CPU-side planner selects shared or privately packed execution per chunk using a cost model and memory constraints, with plans and packed prompt KV reused when valid. The evaluation includes A100 attention microbenchmarks, integrations with several PIC methods, and storage and planning ablations.

Critical strengths
  1. The paper addresses an important systems problem by distinguishing prefill KV reuse from efficient KV storage and shared execution during decoding. Inlay's Base-Repair-Decode representation supports request- and layer-specific repairs without modifying the underlying PIC provider's repair policy. This design allows the system to support multiple PIC methods while reducing redundant KV storage and memory accesses during decoding.

  2. The evaluation demonstrates consistent performance improvements across several workloads and attention configurations. Inlay achieves median attention-kernel speedups of 1.24× over FlashInfer and 1.32× over PAT, and its integration with multiple PIC methods further demonstrates its applicability. The additional full-model decoding results also suggest that the benefits extend beyond isolated attention-kernel optimizations.

Critical weaknesses
  1. The novelty of Inlay appears somewhat limited. While the paper distinguishes its approach from prior work, many of its key components, such as shared KV storage, sparse KV representations, query aggregation, and cost-aware scheduling, have already been explored. The main contribution seems to lie in combining and adapting these existing techniques for PIC decoding, rather than introducing substantially new mechanisms.

  2. The evaluation does not fully establish the practical benefits of Inlay, as several results exclude CPU planning and preparation overheads or additional memory usage from packed buffers. Although these exclusions are acknowledged, the net runtime improvement and actual peak memory consumption remain unclear, making it difficult to assess the practical benefits of the proposed system.

Questions for authors
  1. While Inlay targets a different setting, its storage and execution designs appear closely related to existing approaches such as TokenDance and PAT. Could the authors further clarify the key technical contributions beyond adapting and combining these existing techniques?

  2. Several evaluations report latency improvements using prepared execution plans while excluding CPU planning and initial preparation costs. Could the authors provide end-to-end runtime comparisons that account for these overheads to demonstrate the actual net benefit?

  3. Could the authors also report peak memory usage, including packed KV copies and temporary buffers? Evaluating the number of requests that can actually be served under a fixed memory budget would help demonstrate the practical memory benefits.

  4. More details on the MEPIC reproduction would be helpful. In particular, how was MEPIC implemented, and what motivated the choice of a 32-token block size and the selected number of chunks?

  5. MiniPIC and TokenDance are also closely related to PIC-based KV optimization, but they are not included in the experimental comparisons. Why were these methods excluded?

  6. It would be helpful to include numerical correctness results comparing Inlay with a reference attention implementation using the same PIC repair results, to verify that the proposed optimizations preserve the computation within numerical tolerances.

5 comments

(Due to the response length limit, this is Part 1 of 2.)

Thank you for your thoughtful comments. We hope the clarifications and additional experiments below address your concerns about Inlay’s technical contributions and practical benefits. We would appreciate it if you would reconsider your rating in light of our responses.

[W1, Q1: Novelty and relation to prior work]

Prior PIC work primarily targets prefill acceleration. To our knowledge, Inlay is the first to exploit non-prefix KV sharing enabled by PIC for cross-request shared decode attention while preserving request-specific repairs. This requires mechanisms beyond existing shared storage and query aggregation.

PAT aggregates queries over common prefixes. Inlay enables queries with different position offsets and repair masks to share Base KV loads, combining attention over visible Base entries, token-granular Repair, and private Decode KV without modifying the shared Base.

TokenDance compresses KV through Master–Mirror storage and applies corrections during transfer before attention. Inlay additionally reuses loaded Base KV tiles across queries during decoding, reducing both storage duplication and repeated HBM reads while improving computational efficiency through query aggregation.

Because shared execution introduces partitioning overhead, Inlay selects shared or private execution per chunk using full-batch cost estimates, amortized preparation costs, and a peak-memory budget including packed copies. These mechanisms extend PIC’s benefits into decoding and open a future direction for co-designing PIC methods, KV layouts, and decode kernels.

[W2, Q2, Q3: Full-model latency and peak memory]

To address your questions about these metrics, we ran additional experiments.

On ToolBench with a 40 GiB A100, Inlay reached batch size 64 with a peak PyTorch allocation of 32.54 GiB, while FlashInfer and PAT both reached 48. Packed KV averaged 2.835 GiB in the capacity sweep. At batch size 32, Inlay reduced mean and P99 TTFT by 9.68% and 12.70%, and mean and P99 TPOT by 15.66% and 22.43%, versus FlashInfer. Median CPU preparation took 8.77 ms per step (9.82% of mean TPOT) and ran asynchronously with GPU execution, so any preparation time on the critical path is already reflected in the reported latencies.

The experimental setup and detailed results follow:

Setup. We integrated Inlay into vLLM v0.20.0 and evaluated all 36 layers of Qwen3-8B on ToolBench using a 40 GB A100, with batch size 32, 4,096 input tokens, and 128 output tokens per request. Both systems use identical precomputed PIC states and decode the same fixed token sequence. TTFT includes CPU planning, KV preparation, and first-step initialization. TPOT measures subsequent output-token latency. Latency statistics are computed across requests.

MetricFlashInfer (private KV)InlayReduction
TTFT mean (s)10.9659.9049.68%
TTFT P90 (s)18.27216.19511.37%
TTFT P99 (s)20.40617.81512.70%
TPOT mean (ms)105.9089.3215.66%
TPOT P90 (ms)106.4592.7112.91%
TPOT P99 (ms)128.8799.9722.43%
Device peak memory (GiB)38.17825.75232.55%

CPU overhead. The reported latencies include per-step CPU preparation of 8.77 ms (median) and 14.58 ms (P99).

Memory. Inlay’s KV cache and workspace account for 7.614 GiB at peak (all values in GiB): Base 0.439, Repair/private prompt 4.661, Decode 0.563, packed KV 0.732, pool slack 0.719, and workspace 0.500. Each system uses its own memory-pool configuration.

Capacity. Each run on the same A100 loads KV and executes six decode steps across all 36 layers. Inlay reaches batch size 64 with hybrid execution and averages 2.835 GiB of packed KV in the sweep. FlashInfer and PAT reach 48 and run out of memory at 64. The sweep uses the full device memory. The paper’s 32 GiB budget covers resident KV only.

We will add these comparisons and the implementation details to the main text of the revised paper.

(Due to the response length limit, this is Part 2 of 2.)

[Q4: MEPIC implementation and parameter sensitivity]

Our MEPIC-style FlashInfer baseline shares reusable chunk KV across requests, keeps each chunk’s first block private for repair, and applies positional adjustments within attention, using CacheBlend’s 512-token chunk size (14 chunks in our setup) and a 32-token block size common in vLLM paged attention. To further address your questions, we varied the chunk and block sizes and reran the experiments.

The experimental setup and detailed results follow:

Setup. We measure mean single-layer attention-kernel latency at batch size 64,with 14-chunk(chunk size is 512token), 32-token-block default, we vary block size and chunk count one at a time. A 32-token repair block fits exactly one physical KV page in our setup.

Implementation. We implemented the baseline on FlashInfer. It stores one reusable KV copy per shared chunk, recomputes each chunk’s first block for each request and stores it as private repair KV, and applies positional adjustments within attention.

SettingBlock size (tokens)ChunksMEPIC-style (ms)Inlay (ms)Speedup
Default32142.6970.9422.86×
Block size sweep128142.7041.2912.09×
64142.7021.0552.56×
Chunk count sweep3272.6930.7563.56×
32282.7031.3252.04×
32562.7052.1511.26×

Results. Inlay achieves 1.26–3.56× speedups across the tested settings. The MEPIC-style baseline stays at 2.693–2.705 ms because each request reads the shared KV separately. At a fixed block size of 32 tokens, increasing the chunk count from 7 to 56 lowers Inlay’s speedup from 3.56× to 1.26×. At a fixed count of 14 chunks, increasing the block size from 32 to 128 tokens lowers it from 2.86× to 2.09×. Both changes increase the number of private repair tokens per request, leaving less Base KV for Inlay to read jointly.

[Q5: MiniPIC and TokenDance]

We compared Inlay with MiniPIC and TokenDance on ToolBench. Compared with MiniPIC, Inlay reduces per-layer decode attention latency by 42.7% and peak GPU allocation by 15.5%. Compared with TokenDance, the reductions are 26.7% and 53.1%, respectively.

The experimental setup and detailed results follow.

Setup. We ran Qwen3-8B on a 40 GB A100 at batch size 32 with 4,096-token prompts and one decode step, using real model activations and ToolBench PIC decompositions. We measured attention latency and peak GPU allocation for each layer and averaged both metrics over all 36 layers. The memory measurements exclude model weights.

ImplementationLatency (ms/layer)Peak allocation (MiB/layer)
MiniPIC1.8551,105.5
TokenDance1.4511,992.3
Inlay1.063934.6

Thank you for pointing us to these related works and helping us broaden our evaluation. We will add these comparisons and the implementation details to the main text of the revised paper.

[Q6: Numerical correctness]

Using the setup in [W2, Q2, Q3], we compared Inlay and FlashInfer’s BF16 attention outputs layer by layer on Qwen3-8B with identical queries and logical KV, including repairs. The mean and minimum cosine similarities across layers were:

  • CacheBlend: 0.99999383 and 0.99992826
  • EPIC: 0.99999386 and 0.99992741
  • MEPIC: 0.99999386 and 0.99992741

We will add these comparisons and the implementation details to the main text of the revised paper.

Thank you for reviewing our paper and for your thoughtful feedback. With just over three days left in the discussion period, please review our author response and let us know if any specific concerns remain. This will give us time to address them before the discussion closes. If our response has addressed your concerns, we will be grateful if you can raise the scores accordingly.

Reviewer_bdwCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an execution system for position-independent KV reuse. It addresses the cost of materializing private cache copies when requests share content but require sparse repairs. Inlay separates shared Base storage, sparse Repair entries and private Decode entries. Its attention execution combines their contributions. A planner selects how shared computation is organized. Validation includes synthetic kernel tests, several repair providers, planner experiments and full-model measurements. Inlay accelerates attention execution relative to the tested kernels. It also lowers storage requirements for the evaluated repair providers. The full-model tests show a consistent token-latency benefit.

Critical strengths

The representation supports sharing with repairs. Shared Base entries coexist with sparse repairs and private decode state. The experiments include MEPIC, which already shares storage. Its results show speed gains with nearly unchanged storage. This indicates that the contribution extends beyond eliminating duplicate storage. It also supports a computational benefit for an already sharing provider.

Critical weaknesses

The capacity result describes resident-storage savings rather than measured execution capacity. The 48.5% capacity increase uses a 32 GiB resident-storage estimate. It excludes packed copies and temporary workspace. The paper explicitly acknowledges this limitation. The estimate does not establish the largest batch that actually executes within that memory budget. This limits the strength of the usable-capacity evidence.

Questions for authors

What maximum batch or concurrency actually runs within the stated budget? Does this include peak packed buffers, workspace and coexisting allocations? How does it compare with the strongest baseline?

2 comments

Thank you for pointing out the need to measure how many requests Inlay can serve within a fixed memory budget. We hope the new measurements address this concern, and we would be grateful if you would reconsider your rating in light of these results.

[W1: Resident-storage estimate and measured device memory]

The 48.5% capacity gain in Fig. 5(a) is an estimate based on resident KV storage. In a separate full-model measurement at batch size 32, Inlay reduced peak device memory from 38.178 to 25.752 GiB, a 32.55% reduction relative to FlashInfer with request-private KV.

Setup. We profiled the full Qwen3-8B model in vLLM 0.20.0 on ToolBench using a 40 GB A100, with 4,096 input tokens and 128 output tokens per request. Each system used its own memory-pool configuration. We recorded the device peak and its components in the same batch-32 runs and averaged them across runs. The last row includes overhead that could not be attributed to a specific component.

Memory breakdown (GiB)FlashInfer (private KV)Inlay (shared)
Device peak memory38.17825.752
Model weights15.26815.268
Reserved vLLM KV pool21.0060.598
Separate PIC-provider KV0.4080
Native FlashInfer workspace0.2500.250
Inlay cache and workspace—7.614
Other tensors, allocator overhead, CUDA context, and residual1.2462.022

Inlay memory breakdown. The 7.614 GiB comprises Base KV (0.439 GiB), Repair/private prompt KV (4.661 GiB), Decode KV (0.563 GiB), packed KV copies (0.732 GiB), pool slack (0.719 GiB), and workspace (0.500 GiB).

[Q1: Executable batch size and baseline comparison]

With execution buffers included, Inlay successfully ran at batch size 64 on the same 40 GB A100 where FlashInfer and PAT both reached 48, a 33.3% increase in the largest successful batch tested.

Setup. Using the full Qwen3-8B model and 4,096-token ToolBench inputs, we tested progressively larger batch sizes until each method ran out of GPU memory (OOM). Each run loads the KV cache and executes six decode steps across all 36 layers. Peak PyTorch allocation covers model weights, KV caches, execution buffers including packed KV copies, and workspace. It excludes unused allocator reservations and allocations outside PyTorch.

MethodLargest successful batchPeak PyTorch allocation (GiB)Next batch tested
FlashInfer4836.3064 (OOM)
PAT4836.1764 (OOM)
Inlay6432.5496 (OOM)

Inlay used hybrid execution, with packed KV copies occupying 2.835 GiB on average in this sweep. The sweep uses the full 40 GB of device memory. The 32 GiB budget in Fig. 5(a) applies only to resident KV.

Thank you for your thoughtful review and constructive feedback. We will make further improvements to the paper in the next revision.

Comments on the paper

1 comment

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

Thank you for the time and care that you and the reviewers have devoted to evaluating our paper and engaging with us during the discussion.

Inlay is the first system to extend cross-request reuse in position-independent KV caching (PIC) from prefill to decoding. Even when the same text chunk appears at different positions across requests and each request requires repairs to different tokens, Inlay allows these requests to share a single copy of the reusable KV. During decode attention, a single GPU memory load can therefore serve multiple requests. We have integrated Inlay with MEPIC, CacheBlend, QCFuse, and EPIC while preserving each method's repair policy.

During the rebuttal and follow-up discussion, we analyzed the extent of KV sharing within decode batches under dynamic request arrivals, added full-model end-to-end generation experiments on ToolBench and RAGPulse, and evaluated numerical agreement across several PIC configurations.

We compared Inlay with two state-of-the-art baselines, FlashInfer for paged decoding with request-private KV and PAT for shared-prefix decode attention. On ToolBench, Inlay reduced peak GPU memory usage by 32.55% and P99 time per output token (TPOT) by 22.43% relative to FlashInfer. On RAGPulse, Inlay reduced the mean, P90, and P99 of both time to first token (TTFT) and TPOT relative to both baselines. In direct comparisons with MiniPIC and TokenDance, Inlay achieved lower per-layer decode attention latency and lower peak GPU memory allocation than either method.

Our rebuttal responses describe the setup for each additional experiment and reference the relevant sections and figures to show how the new results build on the original evaluation.

The reviewers' questions helped us broaden our evaluation and strengthen the evidence supporting our conclusions. In response to their feedback, we further evaluated generation quality, quantified the benefits of sharing, and measured planner overhead.

We are sincerely grateful to all the reviewers for their constructive feedback throughout the discussion. As the discussion draws to a close, two reviewers have raised their ratings, and all reviewers now support acceptance.

As you make your final decision, we hope you will consider Inlay's contribution alongside the additional evidence and clarifications provided during the discussion. We would be grateful for your support in accepting our paper.

Best regards,
The authors of Paper #19698

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.