- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PrefillShare enables task-specific models derived from a common base model to reuse prompt KV caches. The base model performs prefill, while specialized decoders are fine-tuned to generate responses conditioned on its cache. During serving, a prefix-aware router keeps session requests on the same prefill worker, incrementally processes new context, and transfers the resulting cache to the selected decoder. The paper reports accuracy close to conventional fine-tuning across math, coding, and tool-calling benchmarks, together with up to 4.5× lower p95 latency and 2.0× higher throughput. The central idea is useful: make cache compatibility a training objective rather than trying to recover it after models have been independently fine-tuned. My main concerns are how much of the serving improvement comes from cache sharing, what the physical memory savings actually are, and whether accuracy holds in the multi-turn workloads used to demonstrate performance.
Critical strengths
- The training procedure directly addresses the representation mismatch behind cross-model KV reuse. Instead of assuming that independently fine-tuned models can consume the same cache, PrefillShare trains each decoder against the cache it will receive during inference. Figure 2 provides a useful comparison with naive sharing and supports the need for this adaptation.
- The accuracy evaluation covers several tasks, model sizes, and two model families. Results are generally close to conventional fine-tuning, including the additional benchmarks and out-of-domain evaluation in Appendix D.3. This makes the approach more convincing than a demonstration on a single task or model.
- The serving evaluation includes several worthwhile controls. Appendix C separates prefill pooling from cross-model reuse, examines predictive prefill, and studies concurrency and input length. The low-overlap workload is also informative because it shows that pooling can remain useful when cross-model prefix reuse contributes little.
- The comparison with ICaRus includes a matched LoRA configuration in Appendix E.1. This is an important control because the main comparison otherwise combines differences in architecture and adaptation capacity. PrefillShare retains a smaller but consistent TPOT advantage under matched adaptation. The multi-node 70B experiment also shows that the implementation extends beyond a single-node 8B deployment.
Critical weaknesses
- Much of the headline serving improvement comes from pooling rather than cross-model cache reuse. In Table 6, pooling alone raises throughput from 28.68k to 52.71k tokens/s; adding cross-model reuse raises it further to 63.13k. The additional improvement is meaningful, but substantially smaller than the comparison against isolated per-model pipelines suggests. This distinction should be central to the main evaluation. In particular, the low-overlap experiment needs a pooling-only control, since its baseline suffers from severe worker imbalance.
- The memory accounting does not clearly distinguish one reusable cache representation from one physical copy. Table 11 reports a reduction from four prompt-cache copies to one, but Appendix A states that caches are transferred to decode workers and retained in GPU memory. It is unclear when these decode-side copies are freed, whether returning to a previous decoder reuses them, and whether transfer buffers and CPU-staged caches are included. The calculation establishes the size of a cache, but not the total resident footprint of the deployed system.
- The accuracy tests and serving workloads leave an important gap. Appendix A describes the main serving traces as fixed-length, sequential invocations with prescribed agent transitions. These traces are useful for measuring system behavior, but they do not establish that an actual multi-agent workflow preserves task success. The reported training setup also uses a maximum sequence length of 1,024, whereas serving includes substantially longer and growing contexts. Accuracy under those conditions needs direct evaluation. No uncertainty estimates are provided for the small differences between PrefillShare and conventional fine-tuning.
- The serving measurement protocol is insufficiently specified, especially under overload. The paper does not clearly define whether throughput counts generated tokens, input tokens, or both, including cached input tokens. Experiment duration, warm-up, repeated trials, and treatment of unfinished sessions are also missing. These details matter when the baseline has saturated and queues continue to grow: a finite-run p95 can depend strongly on how long the experiment runs. Reporting the “best-performing” concurrency setting also requires a precise selection criterion.
- The boundary between prefill and decode needs a more explicit specification. Equation 5 discards the base model’s output, while Equation 6 assumes a preceding generated token. The paper never explains how the specialized decoder produces its first token. In a standard decoder-only model, that prediction comes from the final prompt position, so the implementation must define which model processes that position and exactly which tokens are already in the transferred cache. The corresponding teacher-forcing inputs and label alignment should also be stated. This is necessary to reproduce the method.
Questions for authors
- Can you extend the pooling-only comparison to the main arrival-rate curves and the low-overlap workload? For that baseline, which model weights reside on each prefill worker, and how are model switching costs and available KV memory accounted for?
- What is the measured aggregate cache footprint across prefill GPUs, decode GPUs, and CPU memory? When a session switches models, are previous decode-side copies released? Does each handoff transfer the entire prefix or only blocks missing at the destination?
- Does PrefillShare preserve task success in an actual multi-turn agent workflow with generated outputs, tool results, and variable stopping decisions? How does accuracy change as the context grows beyond the training length?
- What exactly does the throughput metric count? Please specify the measurement duration, warm-up, treatment of unfinished sessions, and whether latency includes time waiting outside the admission limit. A comparison of maximum sustainable arrival rate under a fixed latency target would help.
- How much variation is there across training seeds and repeated serving runs? Confidence intervals would help distinguish accuracy parity from small, potentially incidental improvements.
- How is the first response token generated after shared prefill? Please provide the exact prompt/cache boundary, decoder input, position indices, and training label alignment, ideally with a short training and inference algorithm.
11 comments
Response to Reviewer Tju3 (2/7): W1/Q1 — Prefill pooling and cross-model reuse, part 2
W1 / Q1 — Prefill pooling and cross-model reuse (continued)
Pooling balances the load. In a multi-agent workload, requests are not spread evenly over the models at any moment, so with per-model workers the worker of the busiest model saturates while the others idle. Pooling spreads the requests over all prefill workers. Pooling gives 1.76–1.99× at the arrival rate of Table 6, where one agent is busier than the others, and 2.09× on the low-overlap workload, where pooling accounts for the whole gain. At the highest throughput, where every per-model worker is already busy, pooling gives 1.02–1.23×.
Cross-model reuse removes repeated computation and memory. Without cross-model reuse, every model computes and stores a separate copy of the shared history, and once the cache fills, evicted history is computed again. The gain of cross-model reuse is largest near capacity, a further 1.32–1.45× at the highest throughput, where cross-model reuse is the larger of the two effects. At the arrival rate of Table 6 cross-model reuse adds 1.09–1.22× in throughput and lowers mean TTFT by 1.7×, from 3.61 to 2.10 s at 4K. On the low-overlap workload there is little history to share, so the gain there comes from pooling, as expected.
Since the share of each effect changes along each curve, a pooled curve would show the same trade-off point by point, and the two ends of the range separate the two effects. We have added this table and the discussion to Appendix C.1.
Response to Reviewer Tju3 (3/7): W2/Q2 — Aggregate cache footprint
W2 / Q2 — Aggregate cache footprint Table 11 reports a reduction from four prompt-cache copies to one, while the caches transferred to decode workers are also retained in GPU memory. What is the measured aggregate cache footprint across prefill GPUs, decode GPUs, and CPU memory? When a session switches models, are previous decode-side copies released? Does each handoff transfer the entire prefix or only blocks missing at the destination?
We agree that Table 11 conflated one reusable cache with one physical copy. In our disaggregated serving setup, Table 11 counted only the persistent cache on the prefill side and left out the copies that decode workers receive through handoff. Each decode worker keeps the copy received from the prefill worker in a local prefix cache for later reuse. For a session with models, prefill GPUs therefore hold copies in the baseline and one in PrefillShare, decode GPUs hold copies in both systems, and neither system keeps cache in CPU memory. The aggregate is versus , or 8 versus 5 copies, and we have corrected the memory table and text of Appendix D.2 accordingly.
We measured the footprint in vLLM with 4P+4D, LLaMA-3.1-8B, and 40 sessions, each with a 4K-token prefix shared by 4 agents, with 8 sessions running concurrently. Resident blocks were tracked on every worker through the KV-cache event stream of vLLM, and neither system evicted any block, as shown in Table R3.2.
Table R3.2. KV footprint by location with LLaMA-3.1-8B on 4P+4D, for 40 sessions with a 4K-token prefix shared by 4 agents and 8 sessions running concurrently.
| Location | Baseline | PrefillShare |
|---|---|---|
| Prefill GPUs | 80.0 GiB | 20.0 GiB |
| Decode GPUs | 80.3 GiB | 80.3 GiB |
| CPU memory (cache) | 0 | 0 |
| Aggregate | 160.3 GiB | 100.3 GiB |
The measured aggregate matches the versus count, 4.0 GiB versus 2.5 GiB per session. Of this, only the KV of running calls is actively referenced, which peaked at 4.0 GiB in both systems. The rest is cache kept for reuse.
Transfer buffers are fixed allocations of the same size in both systems and are not part of the KV pool. Each decode worker has a 1 GB GPU receive buffer, and the KV connector allocates a fixed-size pinned host buffer at startup. The host buffer is used only to stage a handoff that does not fit in the GPU receive buffer. The staged data is moved into GPU blocks and freed when the request completes, and is never reused as cache. Several handoffs arrive at a decode worker at once and share the receive buffer, so about 90% of handoffs in both systems passed through the host buffer in the run above. The CPU staging described in Appendix A.2 is this handoff transfer, not KV offloading, and we have revised that paragraph to state this. Neither system offloads KV to CPU memory, so when GPU cache runs short, blocks are evicted and later recomputed.
When a session switches models, the previous decoder releases the blocks as soon as the call finishes. The blocks return to the free pool of vLLM but stay in the prefix cache until they are overwritten. A session that later returns to that decoder therefore reuses whatever blocks remain. At each handoff, the destination decoder loads only the blocks missing from the local prefix cache and reuses the rest. A decoder copy that is evicted is restored by the next handoff, whereas an evicted prefill copy must be recomputed. The two systems therefore differ in the copies that avoid recomputation.
Response to Reviewer Tju3 (1/7): W1/Q1 — Prefill pooling and cross-model reuse, part 1
We thank the reviewer for the detailed review and for recognizing that making cache compatibility a training objective is a useful idea.
W1 / Q1 — Prefill pooling and cross-model reuse Much of the serving gain comes from pooling rather than cross-model reuse, and the low-overlap experiment needs a pooling-only control. Can you extend the pooling-only comparison to the main arrival-rate curves and the low-overlap workload? For that baseline, which model weights reside on each prefill worker, and how are model switching costs and available KV memory accounted for?
The two effects should indeed be reported separately. Please note that the arrival-rate curves of Fig. 4 sweep the load from a moderate rate to saturation, and the share of each effect changes along that range. We therefore measured the separation at both ends of the range, a moderate fixed rate and the load of highest throughput, on the setup of Table 6, together with a pooling-only control for the low-overlap workload. The two mechanisms are complementary, and both are gains of the PrefillShare serving system. Which one dominates depends on how evenly the load is spread and how much history the models share. In particular, the setting of Table 6 is below saturation, where pooling dominates. Near saturation, cross-model reuse adds a further 1.32–1.45× over pooling alone and becomes the larger of the two.
Effective prefill pooling across models is a capability that the PrefillShare serving system adds. In existing prefill/decode-disaggregated serving, each fine-tuned model runs a separate pipeline, so a prefill worker serves only the model whose weights it holds and the prefill GPUs are split per model. PrefillShare instead places a single base-only prefill tier in front of all models. Session-affine routing returns each session to the prefill worker that holds the history of the session, KV handoff delivers the cache to the decode workers of the requested model, and the session limit keeps the active KV within memory.
The pooling-only configuration in Table R3.1 is therefore not an existing serving system but PrefillShare with only cross-model KV reuse disabled. Every prefill worker holds the same frozen base model, so a request of any model can go to any worker without switching weights, and no model switching cost arises. This configuration runs with the same routing, session limit, and KV memory budget per worker as PrefillShare. The KV pool of each worker is shared across models, but prefix-cache entries are kept separately per model. Each model thus computes and stores a separate copy of the shared history. The gap between this configuration and the per-model pipelines measures the gain of the base-only prefill tier, and the gap between this configuration and PrefillShare measures the gain of cross-model reuse.
All runs use Qwen3-14B on 4 prefill and 4 decode H100 GPUs. The first two columns repeat the fixed arrival rate of Table 6 with 4K and 8K prompts, the next two are at the load where each system reaches the highest throughput, and the last is the low-overlap workload of Table 4 at 0.032 sessions/s, as shown in Table R3.1.
Table R3.1. Throughput (Ktok/s) of per-model pipelines, pooling only, and PrefillShare on Qwen3-14B with 4 prefill and 4 decode H100 GPUs. The ratio to the row above is in parentheses.