acceptodds
Under review as a conference paper at ICLR 2027

PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Serving

Abstract

Multi-agent systems increasingly orchestrate multiple specialized language models to solve complex real-world problems, often invoking them over a shared context. This execution pattern repeatedly processes the same prompt prefix across models. Consequently, each model redundantly executes the prefill stage and maintains its own key–value (KV) cache, increasing aggregate prefill load and worsening tail latency by intensifying prefill–decode interference in existing LLM serving stacks. Disaggregated serving reduces such interference by placing prefill and decode on separate GPUs, but disaggregation does not fundamentally eliminate inter-model redundancy in computation and KV storage for the same prompt. To address this issue, we propose PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple fine-tuned models in a disaggregated setting. PrefillShare factorizes the model into prefill and decode modules, freezes the prefill module, and fine-tunes only the decode module. This design allows multiple task-specific models to share a prefill module and the KV cache generated for the same prompt. We further introduce a routing mechanism that enables effective prefill sharing in vLLM-based disaggregated system. PrefillShare not only matches full fine-tuning accuracy on a broad range of tasks and models, but also delivers 4.5 lower p95 latency and 2.0 higher throughput in multi-model agent workloads.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, disaggregated serving, prefill-decode disaggregation, KV cache reuse, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tju3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrefillShare enables task-specific models derived from a common base model to reuse prompt KV caches. The base model performs prefill, while specialized decoders are fine-tuned to generate responses conditioned on its cache. During serving, a prefix-aware router keeps session requests on the same prefill worker, incrementally processes new context, and transfers the resulting cache to the selected decoder. The paper reports accuracy close to conventional fine-tuning across math, coding, and tool-calling benchmarks, together with up to 4.5× lower p95 latency and 2.0× higher throughput. The central idea is useful: make cache compatibility a training objective rather than trying to recover it after models have been independently fine-tuned. My main concerns are how much of the serving improvement comes from cache sharing, what the physical memory savings actually are, and whether accuracy holds in the multi-turn workloads used to demonstrate performance.

Critical strengths
  1. The training procedure directly addresses the representation mismatch behind cross-model KV reuse. Instead of assuming that independently fine-tuned models can consume the same cache, PrefillShare trains each decoder against the cache it will receive during inference. Figure 2 provides a useful comparison with naive sharing and supports the need for this adaptation.
  2. The accuracy evaluation covers several tasks, model sizes, and two model families. Results are generally close to conventional fine-tuning, including the additional benchmarks and out-of-domain evaluation in Appendix D.3. This makes the approach more convincing than a demonstration on a single task or model.
  3. The serving evaluation includes several worthwhile controls. Appendix C separates prefill pooling from cross-model reuse, examines predictive prefill, and studies concurrency and input length. The low-overlap workload is also informative because it shows that pooling can remain useful when cross-model prefix reuse contributes little.
  4. The comparison with ICaRus includes a matched LoRA configuration in Appendix E.1. This is an important control because the main comparison otherwise combines differences in architecture and adaptation capacity. PrefillShare retains a smaller but consistent TPOT advantage under matched adaptation. The multi-node 70B experiment also shows that the implementation extends beyond a single-node 8B deployment.
Critical weaknesses
  1. Much of the headline serving improvement comes from pooling rather than cross-model cache reuse. In Table 6, pooling alone raises throughput from 28.68k to 52.71k tokens/s; adding cross-model reuse raises it further to 63.13k. The additional improvement is meaningful, but substantially smaller than the comparison against isolated per-model pipelines suggests. This distinction should be central to the main evaluation. In particular, the low-overlap experiment needs a pooling-only control, since its baseline suffers from severe worker imbalance.
  2. The memory accounting does not clearly distinguish one reusable cache representation from one physical copy. Table 11 reports a reduction from four prompt-cache copies to one, but Appendix A states that caches are transferred to decode workers and retained in GPU memory. It is unclear when these decode-side copies are freed, whether returning to a previous decoder reuses them, and whether transfer buffers and CPU-staged caches are included. The calculation establishes the size of a cache, but not the total resident footprint of the deployed system.
  3. The accuracy tests and serving workloads leave an important gap. Appendix A describes the main serving traces as fixed-length, sequential invocations with prescribed agent transitions. These traces are useful for measuring system behavior, but they do not establish that an actual multi-agent workflow preserves task success. The reported training setup also uses a maximum sequence length of 1,024, whereas serving includes substantially longer and growing contexts. Accuracy under those conditions needs direct evaluation. No uncertainty estimates are provided for the small differences between PrefillShare and conventional fine-tuning.
  4. The serving measurement protocol is insufficiently specified, especially under overload. The paper does not clearly define whether throughput counts generated tokens, input tokens, or both, including cached input tokens. Experiment duration, warm-up, repeated trials, and treatment of unfinished sessions are also missing. These details matter when the baseline has saturated and queues continue to grow: a finite-run p95 can depend strongly on how long the experiment runs. Reporting the “best-performing” concurrency setting also requires a precise selection criterion.
  5. The boundary between prefill and decode needs a more explicit specification. Equation 5 discards the base model’s output, while Equation 6 assumes a preceding generated token. The paper never explains how the specialized decoder produces its first token. In a standard decoder-only model, that prediction comes from the final prompt position, so the implementation must define which model processes that position and exactly which tokens are already in the transferred cache. The corresponding teacher-forcing inputs and label alignment should also be stated. This is necessary to reproduce the method.
Questions for authors
  1. Can you extend the pooling-only comparison to the main arrival-rate curves and the low-overlap workload? For that baseline, which model weights reside on each prefill worker, and how are model switching costs and available KV memory accounted for?
  2. What is the measured aggregate cache footprint across prefill GPUs, decode GPUs, and CPU memory? When a session switches models, are previous decode-side copies released? Does each handoff transfer the entire prefix or only blocks missing at the destination?
  3. Does PrefillShare preserve task success in an actual multi-turn agent workflow with generated outputs, tool results, and variable stopping decisions? How does accuracy change as the context grows beyond the training length?
  4. What exactly does the throughput metric count? Please specify the measurement duration, warm-up, treatment of unfinished sessions, and whether latency includes time waiting outside the admission limit. A comparison of maximum sustainable arrival rate under a fixed latency target would help.
  5. How much variation is there across training seeds and repeated serving runs? Confidence intervals would help distinguish accuracy parity from small, potentially incidental improvements.
  6. How is the first response token generated after shared prefill? Please provide the exact prompt/cache boundary, decoder input, position indices, and training label alignment, ideally with a short training and inference algorithm.
11 comments
AuthorsOpenReview ↗

Response to Reviewer Tju3 (1/7): W1/Q1 — Prefill pooling and cross-model reuse, part 1

We thank the reviewer for the detailed review and for recognizing that making cache compatibility a training objective is a useful idea.

W1 / Q1 — Prefill pooling and cross-model reuse Much of the serving gain comes from pooling rather than cross-model reuse, and the low-overlap experiment needs a pooling-only control. Can you extend the pooling-only comparison to the main arrival-rate curves and the low-overlap workload? For that baseline, which model weights reside on each prefill worker, and how are model switching costs and available KV memory accounted for?

The two effects should indeed be reported separately. Please note that the arrival-rate curves of Fig. 4 sweep the load from a moderate rate to saturation, and the share of each effect changes along that range. We therefore measured the separation at both ends of the range, a moderate fixed rate and the load of highest throughput, on the setup of Table 6, together with a pooling-only control for the low-overlap workload. The two mechanisms are complementary, and both are gains of the PrefillShare serving system. Which one dominates depends on how evenly the load is spread and how much history the models share. In particular, the setting of Table 6 is below saturation, where pooling dominates. Near saturation, cross-model reuse adds a further 1.32–1.45× over pooling alone and becomes the larger of the two.

Effective prefill pooling across models is a capability that the PrefillShare serving system adds. In existing prefill/decode-disaggregated serving, each fine-tuned model runs a separate pipeline, so a prefill worker serves only the model whose weights it holds and the prefill GPUs are split per model. PrefillShare instead places a single base-only prefill tier in front of all models. Session-affine routing returns each session to the prefill worker that holds the history of the session, KV handoff delivers the cache to the decode workers of the requested model, and the session limit keeps the active KV within memory.

The pooling-only configuration in Table R3.1 is therefore not an existing serving system but PrefillShare with only cross-model KV reuse disabled. Every prefill worker holds the same frozen base model, so a request of any model can go to any worker without switching weights, and no model switching cost arises. This configuration runs with the same routing, session limit, and KV memory budget per worker as PrefillShare. The KV pool of each worker is shared across models, but prefix-cache entries are kept separately per model. Each model thus computes and stores a separate copy of the shared history. The gap between this configuration and the per-model pipelines measures the gain of the base-only prefill tier, and the gap between this configuration and PrefillShare measures the gain of cross-model reuse.

All runs use Qwen3-14B on 4 prefill and 4 decode H100 GPUs. The first two columns repeat the fixed arrival rate of Table 6 with 4K and 8K prompts, the next two are at the load where each system reaches the highest throughput, and the last is the low-overlap workload of Table 4 at 0.032 sessions/s, as shown in Table R3.1.

Table R3.1. Throughput (Ktok/s) of per-model pipelines, pooling only, and PrefillShare on Qwen3-14B with 4 prefill and 4 decode H100 GPUs. The ratio to the row above is in parentheses.

ConfigurationTable 6 rate, 4KTable 6 rate, 8KHighest throughput, 4KHighest throughput, 8KLow overlap
Per-model pipelines28.130.146.457.017.1
Pooling only49.5 (1.76×)59.8 (1.99×)57.1 (1.23×)58.3 (1.02×)35.7 (2.09×)
PrefillShare60.6 (1.22×)65.3 (1.09×)75.1 (1.32×)84.4 (1.45×)35.8 (1.00×)
AuthorsOpenReview ↗

Response to Reviewer Tju3 (2/7): W1/Q1 — Prefill pooling and cross-model reuse, part 2

W1 / Q1 — Prefill pooling and cross-model reuse (continued)

Pooling balances the load. In a multi-agent workload, requests are not spread evenly over the models at any moment, so with per-model workers the worker of the busiest model saturates while the others idle. Pooling spreads the requests over all prefill workers. Pooling gives 1.76–1.99× at the arrival rate of Table 6, where one agent is busier than the others, and 2.09× on the low-overlap workload, where pooling accounts for the whole gain. At the highest throughput, where every per-model worker is already busy, pooling gives 1.02–1.23×.

Cross-model reuse removes repeated computation and memory. Without cross-model reuse, every model computes and stores a separate copy of the shared history, and once the cache fills, evicted history is computed again. The gain of cross-model reuse is largest near capacity, a further 1.32–1.45× at the highest throughput, where cross-model reuse is the larger of the two effects. At the arrival rate of Table 6 cross-model reuse adds 1.09–1.22× in throughput and lowers mean TTFT by 1.7×, from 3.61 to 2.10 s at 4K. On the low-overlap workload there is little history to share, so the gain there comes from pooling, as expected.

Since the share of each effect changes along each curve, a pooled curve would show the same trade-off point by point, and the two ends of the range separate the two effects. We have added this table and the discussion to Appendix C.1.

AuthorsOpenReview ↗

Response to Reviewer Tju3 (3/7): W2/Q2 — Aggregate cache footprint

W2 / Q2 — Aggregate cache footprint Table 11 reports a reduction from four prompt-cache copies to one, while the caches transferred to decode workers are also retained in GPU memory. What is the measured aggregate cache footprint across prefill GPUs, decode GPUs, and CPU memory? When a session switches models, are previous decode-side copies released? Does each handoff transfer the entire prefix or only blocks missing at the destination?

We agree that Table 11 conflated one reusable cache with one physical copy. In our disaggregated serving setup, Table 11 counted only the persistent cache on the prefill side and left out the copies that decode workers receive through handoff. Each decode worker keeps the copy received from the prefill worker in a local prefix cache for later reuse. For a session with N=4N=4 models, prefill GPUs therefore hold NN copies in the baseline and one in PrefillShare, decode GPUs hold NN copies in both systems, and neither system keeps cache in CPU memory. The aggregate is 2N2N versus N+1N+1, or 8 versus 5 copies, and we have corrected the memory table and text of Appendix D.2 accordingly.

We measured the footprint in vLLM with 4P+4D, LLaMA-3.1-8B, and 40 sessions, each with a 4K-token prefix shared by 4 agents, with 8 sessions running concurrently. Resident blocks were tracked on every worker through the KV-cache event stream of vLLM, and neither system evicted any block, as shown in Table R3.2.

Table R3.2. KV footprint by location with LLaMA-3.1-8B on 4P+4D, for 40 sessions with a 4K-token prefix shared by 4 agents and 8 sessions running concurrently.

LocationBaselinePrefillShare
Prefill GPUs80.0 GiB20.0 GiB
Decode GPUs80.3 GiB80.3 GiB
CPU memory (cache)00
Aggregate160.3 GiB100.3 GiB

The measured aggregate matches the 2N2N versus N+1N+1 count, 4.0 GiB versus 2.5 GiB per session. Of this, only the KV of running calls is actively referenced, which peaked at 4.0 GiB in both systems. The rest is cache kept for reuse.

Transfer buffers are fixed allocations of the same size in both systems and are not part of the KV pool. Each decode worker has a 1 GB GPU receive buffer, and the KV connector allocates a fixed-size pinned host buffer at startup. The host buffer is used only to stage a handoff that does not fit in the GPU receive buffer. The staged data is moved into GPU blocks and freed when the request completes, and is never reused as cache. Several handoffs arrive at a decode worker at once and share the receive buffer, so about 90% of handoffs in both systems passed through the host buffer in the run above. The CPU staging described in Appendix A.2 is this handoff transfer, not KV offloading, and we have revised that paragraph to state this. Neither system offloads KV to CPU memory, so when GPU cache runs short, blocks are evicted and later recomputed.

When a session switches models, the previous decoder releases the blocks as soon as the call finishes. The blocks return to the free pool of vLLM but stay in the prefix cache until they are overwritten. A session that later returns to that decoder therefore reuses whatever blocks remain. At each handoff, the destination decoder loads only the blocks missing from the local prefix cache and reuses the rest. A decoder copy that is evicted is restored by the next handoff, whereas an evicted prefill copy must be recomputed. The two systems therefore differ in the copies that avoid recomputation.

Reviewer_HfuXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PrefillShare allows multiple fine-tuned models to share the same prompt KV cache. Instead of running prefill separately for each model, a frozen base model handles the prefill once, and each task-specific model is trained to decode using that shared cache. The authors also modify vLLM to support cache sharing and routing between agents, allowing different models to reuse the same cache and share prefill GPUs. Experiments on math, coding, and tool calling show that this approach achieves similar accuracy to full fine-tuning and it reduces latency and improves throughput.

Critical strengths
  • The method is simple and practical. Cache-conditioned fine-tuning stays within about 1 point of full fine-tuning across tasks and model scales. The system is implemented in vLLM and tested up to 70B on multiple nodes.
Critical weaknesses
  • The core idea is very close to Activated LoRA, which is not cited. aLoRA (Greenewald et al., 2025) already trains adapters that act only on tokens after they are invoked, so that they can reuse the base model's KV cache for the context. A follow-up (Li et al., 2025) builds cross-model KV-cache reuse between the base model and aLoRA models into vLLM. PrefillShare's main difference is that it fully fine-tunes the decoder instead of using LoRA, which is incremental, and the Qwen3 experiments (Table 2) use LoRA anyway. This leaves limited novelty for cache-conditioned fine-tuning.
  • Most of the speedup does not come from sharing the KV cache. The baseline gives each model its own prefill GPU, while PrefillShare lets all models share a pool of prefill GPUs. Appendix C.1 shows that this pooling alone accounts for most of the gain, and KV sharing adds a smaller improvement on top. The headline numbers mix the two effects, so the gain from the paper's main idea is smaller than it looks.
Questions for authors
  1. How does PrefillShare differ from aLoRA and its vLLM cross-model KV reuse? How do they compare under the same adaptation budget?
  2. Can you report the main serving results against the pooled-prefill baseline?
  3. Are there tasks where freezing the prompt-side computation hurts, e.g. ones that need the input read differently (long context, new formats, system or safety instructions)?

References:

  • Greenewald, K., et al. Activated LoRA: Fine-tuned LLMs for Intrinsics. NeurIPS 2025.
  • Li et al. Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA. arXiv:2512.17910, 2025.
  • Woo, S., et al. ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference. ICLR 2026.
8 comments
AuthorsOpenReview ↗

Response to Reviewer HfuX (1/5): W1/Q1 — Relation to Activated LoRA, part 1

We appreciate that the reviewer found the method simple and practical, and we address the three concerns in turn below.

W1 / Q1 — Relation to Activated LoRA How does PrefillShare differ from aLoRA and its vLLM cross-model KV reuse? How do they compare under the same adaptation budget?

We are grateful for the pointer to Activated LoRA (aLoRA; Greenewald et al., 2025) and the follow-up vLLM study (Li et al., 2025), and we have added both to the paper. aLoRA is closely related work, and the two start from a similar idea, a specialized model that reuses the KV cache of the base model. PrefillShare takes this idea further in two ways. First, to the best of our knowledge, PrefillShare is the first system to overcome the prefix-cache misses that model switches cause in disaggregated serving, which prior KV-sharing methods such as aLoRA still incur, and reaches 1.33–1.59× the peak throughput of aLoRA on the standard vLLM prefill/decode proxy. Second, PrefillShare removes the adapter-capacity limit of aLoRA, so the decoder can be fully fine-tuned and exceeds aLoRA on all four benchmarks.

How PrefillShare differs. aLoRA and the follow-up vLLM study switch an adapter on at an invocation sequence, so that the adapter can reuse the cache of the base model inside a single engine. PrefillShare is the first prefill/decode-disaggregated serving system designed around shared prefill, so that prefix-cache reuse holds across model switches even when consecutive calls of a session go to different models on different workers. PrefillShare consists of a base-only prefill tier shared by all models, session-affine routing that returns a session to the prefill worker holding the history of the session, KV handoff to the decode workers of the requested model, and a session limit that keeps the active KV within memory. In this design the decoder is an entire model that runs on dedicated workers, so the decoder can be a LoRA, a merged LoRA or a fully fine-tuned model. The difference is therefore more than LoRA versus full fine-tuning, and full fine-tuning is a capability that this serving system adds, at no cost in decoding speed.

What the serving system adds. To separate the system from the adapter, we served the same aLoRA adapters with and without the PrefillShare serving system. All runs use Qwen3-14B on 4 prefill and 4 decode H100 GPUs, with four agents sharing one history and an 8K-token initial prompt. Both aLoRA rows are our implementation on the same base-only prefill workers, since Li et al. serve aLoRA in a single engine, and the first row uses the prefill/decode proxy shipped with vLLM, which picks a prefill worker per request. We compare the peak throughput of the three configurations and the P90 TTFT at that peak, as shown in Table R2.1.

Table R2.1. Peak throughput (Ktok/s) and P90 TTFT at the peak of aLoRA and PrefillShare on Qwen3-14B with 4 prefill and 4 decode H100 GPUs, four agents sharing one history, and an 8K-token initial prompt. The ratio to the first row is in parentheses.

MethodSession-affine routingDedicated decode workerDecoderSupervisor orderSequential orderP90 TTFT at peak (s)
aLoRA with the vLLM P/D proxy✗✗LoRA63.662.42.5–3.3
aLoRA with PrefillShare routing✓✗LoRA73.9 (1.16×)77.4 (1.24×)4.5–4.9
PrefillShare✓✓Full-FT84.4 (1.33×)99.3 (1.59×)7.2–8.6

The gain comes from the serving system. With identical adapters, keeping the history of a session on one prefill worker alone gives 1.16–1.24×. Giving each model dedicated decode workers as well, so that the decoder can be merged or fully fine-tuned, raises the gain to 1.33–1.59×. At a matched P90 TTFT, PrefillShare is 1.23–1.27× ahead of aLoRA with the vLLM P/D proxy, with 76.9K against 62.4K and 80.7K against 63.6K tok/s. The first 1.16–1.24× is available to aLoRA-style adapters as they are, and the rest comes from the dedicated decode workers, which allow a fully fine-tuned decoder.

AuthorsOpenReview ↗

Response to Reviewer HfuX (2/5): W1/Q1 — Relation to Activated LoRA, part 2

W1 / Q1 — Relation to Activated LoRA (continued)

Accuracy under the same adaptation budget. We trained aLoRA on LLaMA-3.1-8B with the same rank (128), modules and learning-rate grid as our LoRA decoder.

Table R2.2. Accuracy of aLoRA and PrefillShare on LLaMA-3.1-8B under the same adaptation budget, with the same rank (128), modules, and learning-rate grid.

MethodGSM8KGSM-PlusHumanEvalHumanEval+
aLoRA70.6648.4748.1742.68
PrefillShare (LoRA)70.7449.2648.1743.29
PrefillShare (Full-FT)71.2749.0948.3745.12

Under the same budget, the PrefillShare LoRA decoder matches aLoRA, as shown in Table R2.2. PrefillShare (LoRA) is equal or higher on all four benchmarks, within 0.8 points. A fully fine-tuned decoder, which can share base-model KV only in PrefillShare, exceeds aLoRA on all four, by up to 2.4 points on HumanEval+.

We have added aLoRA and Li et al. to the related work in Section 5, stated the relation to aLoRA in the introduction in Section 1, added an aLoRA row to the accuracy table of Appendix E.1 (Table 21), and added the serving comparison to a new Appendix E.3 (Table 25). Section 4.4 now refers to Appendix E.3.

References. K. Greenewald et al. Activated LoRA: Fine-tuned LLMs for Intrinsics. NeurIPS 2025. A. Li et al. Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA. arXiv:2512.17910, 2025.

AuthorsOpenReview ↗

Response to Reviewer HfuX (3/5): W2/Q2 — Prefill pooling and KV sharing, part 1

W2 / Q2 — Prefill pooling and KV sharing Most of the speedup comes from pooling prefill GPUs rather than from KV sharing, and the headline numbers mix the two effects. Can you report the main serving results against the pooled-prefill baseline?

The headline numbers do combine two effects, and we now report the two effects separately against the pooled-prefill baseline. The two mechanisms are complementary, and both are gains of the PrefillShare serving system. Which one dominates depends on how evenly the load is spread and how much history the models share, so we measured both at a moderate fixed rate, at the load of highest throughput, and on the low-overlap workload. In particular, the setting of Table 6 is below saturation, where pooling dominates. Near saturation, cross-model reuse adds a further 1.32–1.45× over pooling alone and becomes the larger of the two.

Effective prefill pooling across models is a capability that the PrefillShare serving system adds. In existing prefill/decode-disaggregated serving, each fine-tuned model runs a separate pipeline, so a prefill worker serves only the model whose weights it holds and the prefill GPUs are split per model. PrefillShare instead places a single base-only prefill tier in front of all models. Session-affine routing returns each session to the prefill worker that holds the history of the session, KV handoff delivers the cache to the decode workers of the requested model, and the session limit keeps the active KV within memory. A KV-compatible adapter such as aLoRA can also share base-only prefill workers, but the standard vLLM prefill/decode proxy treats each call as an independent request, without session-level routing or admission control. It picks a prefill worker per call, so consecutive calls of a session miss the cached history, and every decode worker batches all four adapters together. With the same aLoRA adapters, this proxy reaches 62.4–63.6K tok/s, and the PrefillShare routing raises this by 1.16–1.24×, as shown in Table R2.1.

The pooling-only configuration in Table R2.3 is therefore not an existing serving system but PrefillShare with only cross-model KV reuse disabled. It runs on the same base-only prefill tier, routing, session limit, and KV memory budget as PrefillShare, and only the prefix cache is split per model. Each model thus computes and stores a separate copy of the shared history. The gap between this configuration and the per-model pipelines measures the gain of the base-only prefill tier, and the gap between this configuration and PrefillShare measures the gain of cross-model reuse.

All runs use Qwen3-14B on 4 prefill and 4 decode H100 GPUs. The first two columns repeat the fixed arrival rate of Table 6 with 4K and 8K prompts, the next two are at the load where each system reaches the highest throughput, and the last is the low-overlap workload of Table 4 at 0.032 sessions/s, as shown in Table R2.3.

Table R2.3. Throughput (Ktok/s) of per-model pipelines, pooling only, and PrefillShare on Qwen3-14B with 4 prefill and 4 decode H100 GPUs. The ratio to the row above is in parentheses.

ConfigurationTable 6 rate, 4KTable 6 rate, 8KHighest throughput, 4KHighest throughput, 8KLow overlap
Per-model pipelines28.130.146.457.017.1
Pooling only49.5 (1.76×)59.8 (1.99×)57.1 (1.23×)58.3 (1.02×)35.7 (2.09×)
PrefillShare60.6 (1.22×)65.3 (1.09×)75.1 (1.32×)84.4 (1.45×)35.8 (1.00×)
Reviewer_jjmFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrefillShare trains specialized decoders to consume prompt KV states from a frozen base model. Related models can then share prefill workers and reuse cached prefixes across model switches. The paper combines this training procedure with a serving implementation and evaluates both model accuracy and serving performance.

Critical strengths

Making cache compatibility part of training is a sensible way to address the mismatch between independently fine-tuned models. The accuracy results span several tasks, model sizes, and two model families, and generally stay close to ordinary fine-tuning. This supports the basic feasibility of the approach.

The ablation study separates the benefit of pooling from the additional benefit of cache reuse. Pooling raises throughput from 28.68k to 52.71k tokens/s, while adding reuse raises it to 63.13k. The additional improvement of about 20% is more informative than the headline comparison alone. The low-overlap and multi-node experiments also help establish where the system remains useful.

Critical weaknesses
  • The comparison with ICaRus is incomplete. The matched-LoRA comparison mainly measures decode cost. PrefillShare avoids running the base model at every decoding step, but must later prefill a specialist's generated output before another model can reuse it. Please compare complete multi-model workflows, including this work, cache handoff, and time to first token.

  • The memory accounting needs clarification. Table 11 counts one shared-prefix copy, while the serving implementation also keeps GPU-resident copies on decode workers. Both descriptions may be consistent, but they measure different things. Please distinguish persistent prefill-cache storage from active decoder copies and report the total live KV footprint at representative concurrency.

  • The quality and serving evaluations could be better connected. Serving uses prescribed invocation lengths, while the main training setup limits sequences to 1,024 tokens. A quality evaluation on long, growing multi-turn contexts would show whether the reported serving gains apply when model switching changes the actual generated history.

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer jjmF (1/3): W1 — Complete-workflow

We thank the reviewer for the careful reading and the concrete suggestions.

W1 — Complete-workflow comparison with ICaRus Please compare complete multi-model workflows, including the prefill of generated output, cache handoff, and time to first token.

A complete-workflow comparison is the right way to settle this question, and we have run it with the re-prefill of generated output, cache handoff, and TTFT all included. PrefillShare stays ahead of ICaRus over the complete workflow. It finishes 1.19–1.27× as many sessions per second with a fully fine-tuned decoder, and 1.11–1.13× with the same LoRA adapter. The re-prefill that the reviewer points to is counted in full, and it is small, about 6% of the prefill work.

The two designs pay for cache compatibility at different places. ICaRus keeps the cache compatible by running the base model next to the adapter at every decoding step, so it pays for every generated token in the sequential decoding phase. PrefillShare pays once per generated output, when that output is prefilled again at the next call in a single parallel prefill pass. In our measurements, the cost that ICaRus pays at every token outweighs the re-prefill that PrefillShare pays once per output.

Complete workflow. We implemented ICaRus in the same serving stack as PrefillShare, with the same prefill tier, routing, and decode layout as PrefillShare (LoRA). All runs use Qwen3-14B on 4 prefill and 4 decode H100 GPUs, with four agents sharing one history and an 8K-token initial prompt. To give ICaRus its advantage in full, we also return the KV of its generated tokens to the prefill worker, so that it re-prefills almost none of its output. Even so, PrefillShare is ahead in throughput, and its TTFT is no higher at the median and lower at P90, as shown in Table R1.1. In contrast, the second decode path of ICaRus raises the time per output token from 22.1 to 25.5 ms with the same LoRA adapter. The prefill-to-decode cache handoff is on the same path for all three methods and is included in every TTFT of Table R1.1. The additional KV return of ICaRus is asynchronous.

Table R1.1. Complete-workflow comparison with ICaRus on Qwen3-14B with 4 prefill and 4 decode H100 GPUs, four agents sharing one history, an 8K-token initial prompt, and a concurrency limit of 48. At the best concurrency limit of each method, ICaRus reaches 0.475 sessions/s and PrefillShare (LoRA) 0.528.

MethodDecoderDecode paths per tokenRe-prefilled tokens / sessionThroughput (Ktok/s)Sessions/sTTFT p50 / P90 (s)TPOT (ms)
ICaRus + KV returnLoRA210978.30.4463.7 / 11.325.5
PrefillShare (LoRA, unmerged)LoRA188288.70.5063.6 / 9.322.1
PrefillShare (Full-FT)Full-FT187699.30.5672.9 / 8.619.1

Longer outputs. The advantage of PrefillShare grows with the output length. Since the re-prefill grows with the output length, one may expect ICaRus to win for long outputs. However, the cost of ICaRus grows as well, because every additional token goes through two decode paths. The order stays the same from 58 to 1,024 output tokens per call, and the ratio grows from 1.26× to 1.37×, or from 1.11× to 1.19× with the same LoRA adapter, as shown in Table R1.2.

Table R1.2. Throughput (Ktok/s) by output length per call, with a 4K-token initial prompt and the same load in every cell.

Output tokens per callICaRus + KV returnPrefillShare (LoRA, unmerged)PrefillShare (Full-FT)Full-FT / ICaRus
5860.867.676.71.26×
25628.431.636.31.28×
1,02415.518.521.21.37×

We have replaced the single-GPU TPOT panel of Fig. 5 with the output-length comparison above, and Section 4.4 now reports the complete-workflow result in place of the single-GPU TPOT. Appendix E.1 gives both tables, and the single-GPU TPOT columns have been removed from its accuracy table.

AuthorsOpenReview ↗

Response to Reviewer jjmF (2/3): W2 — Memory accounting

W2 — Memory accounting Please distinguish persistent prefill-cache storage from active decoder copies and report the total live KV footprint at representative concurrency.

The reviewer is right that the two counts measure different things. In our disaggregated serving setup, Table 11 counted only the persistent cache on the prefill side and left out the copies that decode workers receive through handoff. Each decode worker keeps the copy received from the prefill worker in a local prefix cache for later reuse. For a session with N=4N=4 models, the baseline therefore holds NN prefill copies and NN decoder copies, while PrefillShare holds one prefill copy and NN decoder copies. The reduction is 2N2N to N+1N+1, or 8 to 5 copies, rather than 4 to 1. Because calls within a session are sequential, only one decoder copy per session is in active use at any moment in either system. The two kinds of copy also save different work. A decoder copy that is evicted is restored by the next handoff, which transfers only the missing blocks, whereas an evicted prefill copy must be recomputed. The two systems therefore differ in the copies that avoid recomputation.

We measured the footprint in vLLM with 4P+4D, LLaMA-3.1-8B, and 40 sessions, each with a 4K-token prefix shared by 4 agents, with 8 sessions running concurrently. Resident blocks were tracked on every worker through the KV-cache event stream of vLLM, and neither system evicted any block.

Table R1.3. KV footprint with LLaMA-3.1-8B on 4P+4D, for 40 sessions with a 4K-token prefix shared by 4 agents and 8 sessions running concurrently.

BaselinePrefillShare
Persistent prefill cache80.0 GiB20.0 GiB
Decoder copies (resident)80.3 GiB80.3 GiB
Total live KV160.3 GiB100.3 GiB
Actively referenced KV (peak)4.0 GiB4.0 GiB

The measured totals match the 2N2N versus N+1N+1 count, 4.0 GiB versus 2.5 GiB per session. The two parts scale differently with load. Actively referenced KV grows by 0.5 GiB per concurrent session and is the same in both systems. The cached part grows with the number of sessions kept for reuse, and the two systems differ in this cached part.

Since vLLM preallocates the KV pool, the saving appears as cache capacity rather than as lower memory use. Fig. 8 shows this effect. Under the same pool, the prefix hit rate of the baseline drops beyond about 40 concurrent sessions, while PrefillShare stays near 89%. We have revised the memory table of Appendix D.2 to separate prefill-side and decode-side copies and to report the measured total, and the text of Appendix D.2 to state the eight-versus-five count.

AuthorsOpenReview ↗

Response to Reviewer jjmF (3/3): W3 — Accuracy on long, growing multi-turn contexts

W3 — Accuracy on long, growing multi-turn contexts A quality evaluation on long, growing multi-turn contexts would show whether the reported serving gains apply when model switching changes the actual generated history.

We agree that this connection matters, and we ran the quality evaluation that the reviewer describes on an actual multi-agent workflow. We evaluated PrefillShare in α-UMi, a multi-agent tool-use workflow in which a planner, a caller, and a summarizer take turns on one shared history (Shen et al., 2024). We fine-tuned the three roles of LLaMA-3.1-8B-Instruct on the α-UMi ToolBench data with 2K-token training examples, once with Full-FT and once with PrefillShare, and ran all 1,095 test tasks. The agents call tools, receive the tool results, and decide themselves when to stop, give up, or answer. We ran each task in two settings. In a single-task episode, the task starts from an empty context. In a multi-task session, tasks run back-to-back in one growing context, so the trajectories of earlier tasks stay in the prompt, up to 28K tokens or 14 times the training length. In PrefillShare, the frozen base model processes this whole history, including the outputs of the other roles.

Table R1.4. Accuracy of Full-FT and PrefillShare in the α-UMi workflow on all 1,095 ToolBench test tasks.

ContextMethodTool use (strict success)Answer quality (Rouge-L)
Single-task episodeFull-FT25.331.8
PrefillShare28.834.0
Multi-task session (up to 28K tokens)Full-FT33.433.7
PrefillShare32.735.1

PrefillShare keeps the accuracy of Full-FT in this workflow. PrefillShare is higher than Full-FT in single-task episodes and on par in multi-task sessions, as shown in Table R1.4. Both methods score higher in multi-task sessions, because the trajectories of earlier tasks act as in-context examples.

We split the multi-task session by the context length at the start of each task, as shown in Table R1.5.

Table R1.5. Accuracy in the multi-task session by the context length at the start of each task.

MetricMethodUp to 10K (≤ 5×)10K–28K (5–14×)
Tool use (strict success)Full-FT31.035.1
PrefillShare29.934.7
Answer quality (Rouge-L)Full-FT30.735.9
PrefillShare32.237.2

Neither method degrades as the context grows to 28K tokens, 14 times the training length, and PrefillShare tracks Full-FT in both ranges. The decoders were trained only on 2K-token examples, yet the decoders keep the same accuracy when the frozen base model reads histories many times longer, written by other agents and filled with tool results. As in our serving workloads, the history is written by the agents themselves and changes at every model switch, so the serving gains apply in a regime where accuracy holds. We have added this evaluation to a new Appendix D.5, and Section 4.2 now refers to Appendix D.5.

References. W. Shen et al. Small LLMs Are Weak Tool Learners: A Multi-LLM Agent. EMNLP 2024.

I've read the response and I have no further questions. I will keep my score as-is.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.