acceptodds
Under review as a conference paper at ICLR 2027

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

Abstract

In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes the execution of a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per decode GPU by up to 2.0 over conventional disaggregation while keeping time-per-output-token (TPOT) degradation within 10%. In eight-model multi-node serving, SUN retains 98.7% of baseline throughput with 25% fewer total GPUs at moderate load and increases throughput by 29.1% on the same hardware at higher load. SUN also outperforms popularity-aware allocation and GPU colocation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Serving, Disaggregated Inference, Multi-LLM Serving, Prefill-Decode Disaggregation, KV cache

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qXbdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses inter-model isolation in disaggregated multi-LLM serving: conventional disaggregation separates prefill from decode, but decode workers remain partitioned per model, causing GPU underutilization under skewed workloads (popular models queue while idle capacity behind less-requested models cannot be reused). The authors propose SUN (Shared Use of Next-token Prediction), which:

  1. Formalizes prefill–decode decoupling: decomposes decoder-only Transformer execution into a prefill module PθpP_{\theta_p} (producing first-token distribution and prompt KV cache) and a decode module DθdD_{\theta_d}, which need not share parameters (unlike conventional models where θp=θd=θ\theta_p = \theta_d = \theta).
  2. Prefill-only tuning: each task-specific prefill module is fine-tuned against the frozen shared decoder (the same decoder used at inference), so its prompt KV cache is directly interpretable by the shared decoder. This avoids the train–inference mismatch of naive sharing (GSM8K drops 71.3%→51.2% without it) and the accuracy/efficiency trade-offs of Multi-LoRA serving.
  3. Model-agnostic decode routing: since all decode requests use identical frozen weights, requests from different prefill modules can enter one decode batch in a unified vLLM-based worker pool, enabling cross-model batching and GPU consolidation. Results: Accuracy is comparable to Full-FT across LLaMA3.1-8B and Qwen3-1.7B/8B/14B on math (GSM8K/GSM+), coding (HumanEval/HumanEval+), and tool calling (BFCL). System-wise, SUN achieves up to 2.0× throughput per decode GPU with ≤10% TPOT degradation; in eight-model multi-node serving it retains 98.7% of baseline throughput with 25% fewer GPUs at moderate load and gains 29.1% throughput on the same hardware at higher load. It outperforms popularity-aware allocation (+7.0%) and MPS-based GPU colocation (+23.6%, extending to +32.2% in a twelve-service evaluation).
Critical strengths
  1. First formalization of prefill–decode parameter decoupling for cross-model decode sharing. The paper explicitly formulates the prefill and decode modules as separately parameterized components (Eqs. 1–4), turning an implicit systems intuition into a clean mathematical model that enables shared decoding — a genuinely novel framing relative to Splitwise/DistServe/Mooncake.
  2. Prefill-only tuning elegantly resolves the KV-cache compatibility problem. Training the task-specific prefill module against the inference-time frozen decoder aligns the prompt KV cache with shared decoding (Fig. 2), avoiding the naive-sharing accuracy collapse while retaining full-parameter adaptation capacity (superior to LoRA's low-rank restriction) and eliminating adapter computation from the decode path (no per-step adapter overhead, unlike Punica/S-LoRA).
  3. Model-agnostic decode routing converts underutilized capacity into real resource savings. Pooling decode workers enables cross-model batching: up to 2.0× per-GPU throughput with ≤10% TPOT degradation, 25% total GPU reduction at moderate load (Table 2), and robustness to workload skew (interactivity +44.1% at α=3.0, Fig. 4/Fig. 6).
  4. Comprehensive and honest evaluation design. The paper sweeps decode pool sizes, Zipf skew α, model scales (1.7B–14B), single-node and multi-node (TP=2, cross-node KV transfer) settings, and reports sanity checks (achieved/offered load ratios in Appendix B, one-time training cost quantification in Appendix C.2).
  5. Well-motivated baselines addressing the obvious rebuttals. The comparison against popularity-aware allocation and workload-informed MPS colocation demonstrates that SUN's gains come from cross-model batching (breaking the model-specific batching boundary), not merely from demand-aware placement or physical GPU sharing — the batching-opportunity analysis (λ/16 vs. pᵢλ/rᵢ arrivals) makes the mechanism explicit.
Critical weaknesses
  1. Fairness of the MPS baseline is questionable. In Table 3, MPS p95 TPOT (23.90 ms) is worse than isolated serving (19.36 ms), and in Appendix C.1 MPS TPOT explodes to 43.56 ms. While Appendix A.5/C.1 grant MPS favorable assumptions (full workload knowledge, 100% thread cap, resident workers, no loading costs), the configurations (e.g., batch caps, the hot model A occupying all 16 GPUs) may have been artificially placed at a disadvantage. The +23.6%/+32.2% comparisons would be more credible with justification that the MPS configurations are at least reasonably tuned.
  2. Cross-node KV transfer overhead is not quantified. Sections 3.2.3 and A.4 acknowledge that cross-node KV transfer is a necessary cost of shared decoding, but the paper never reports the bandwidth/latency cost of KV transfers. It is unclear how much of the p95 TPOT increase from 10.6→14.3 ms (Table 2, 2P/1D vs. isolated) is attributable to KV transfer versus queueing on fewer decode nodes — an ablation is needed.
  3. Consolidation boundary conditions are insufficiently discussed. Fig. 3 shows that at OSL=512, α=1.5, total throughput drops from 6,323 (4D) to 3,910 (1D), indicating a single decode node saturates. The paper presents sweeps but gives no criterion for when it is safe to cut GPUs (e.g., a load/SLA-aware admission rule), nor any auto-scaling policy for the shared pool — which limits practical applicability of the claimed TCO savings.
  4. Narrow compatibility requirement limits generality. SUN requires an identical backbone, tokenizer, attention configuration, and KV-cache layout across all shared models (Section 3.2.3, Appendix D). This restricts sharing to fine-tunes of a common base model; heterogeneous fleets (the realistic production scenario) are out of scope, and the paper offers no discussion of bridging mechanisms.
  5. Accuracy gaps are task-dependent and not fully explained. On Qwen3-14B, SUN trails Full-FT by 2.7/2.9 points on GSM8K/GSM+ and LLaMA3.1-8B shows a 4.2-point gap on HumanEval+. The paper does not analyze why certain task–model combinations degrade (e.g., whether long-form reasoning chains are more sensitive to prefill–decode parameter mismatch), nor whether hyperparameter search could close these gaps.
Questions for authors
  1. MPS baseline configuration: Can you demonstrate that the MPS results in Table 3 and Table 4 are not artificially weakened? Specifically, how were the MPS batch caps, thread caps, and engine placements (e.g., A on all 16 GPUs) selected, and would alternative MPS configurations (e.g., different per-engine batch limits or replica placements) change the +23.6%/+32.2% throughput gap?
  2. KV transfer cost: Please quantify the bandwidth and latency cost of cross-node KV transfer in the multi-node experiments. In Table 2, how much of the p95 TPOT increase (10.6→14.3 ms for 2P/1D at 8 RPS) is attributable to KV transfer versus decode-node consolidation?
  3. Safe consolidation policy: Given that Fig. 3 shows single-decode-node saturation at OSL=512, α=1.5 (throughput 6,323→3,910), what operational criterion (e.g., queue depth, KV occupancy, offered-load threshold) should practitioners use to decide when shrinking the decode pool is safe? Could such a rule be automated?
7 comments

W1 + Q1 (1)

Even against the best of 27 MPS configurations swept along the parameters the reviewer raised, SUN keeps its advantage: 14–26% higher throughput per GPU on one node, and at least 37% lower p95 TPOT than every MPS configuration on four nodes.

How the batch caps, thread caps, and engine placements were selected. All three follow the principle described in Appendices A.5 and C.1: give MPS the most favorable setting, with the demand distribution known in advance.

  • Engine placement. The hot model A is placed on every decode GPU, so that A can use the capacity of the whole pool, and requests are balanced so that every physical decode GPU receives the same load.

  • Thread cap. Every MPS client has a 100% active-thread cap, so each engine can use all SMs whenever the colocated engine is idle, instead of being limited to a fixed share.

  • Batch cap. On one node, no batch cap was set beyond the vLLM default. All 120 requests were in flight at once, so each decode GPU held 30 concurrent sequences (12 and 18 for the two engines of a shared GPU), and no engine reached a batch limit. On four nodes, every engine in every system, including SUN and isolated serving, admits at most 16 concurrent decode requests. On the original cluster, we first used a limit of 128, but two colocated MPS engines then stalled for more than two minutes at 92–94% KV occupancy with requests still waiting. The limit of 16 avoided this stall while still allowing each shared MPS GPU to hold up to 32 concurrent sequences, more than the 30 per GPU of the single-node setting. On the cluster used for the re-measurement below, the same MPS configuration with a limit of 128 completes without stalls, at 7,633 tok/s and a p95 TPOT of 24.65 ms (single run), compared with 7,419 tok/s and 24.70 ms with a limit of 16. Raising the limit therefore increases MPS throughput by 2.9% and leaves its p95 TPOT unchanged, far above the 14.97 ms of SUN. Whether alternative MPS configurations change the +23.6%/+32.2% gap. On one node, they do not: SUN outperforms every MPS configuration. On four nodes, the +32.2% throughput gap disappears after re-measurement, but the latency gap remains under every MPS configuration. Tuning improves MPS throughput by at most 4.4% on one node and 6.6% on four nodes, so the MPS results in the paper were not artificially weakened. To show this, we re-ran both comparisons on H100 with a sweep of 27 MPS configurations per setting, covering every parameter the reviewer mentions:

  • Placement: A/A/AB/AC as in the paper, A/A/A/BC, and AB/AC/AB/AC on one node; A on all 16 decode GPUs as in the paper, 8 decode GPUs dedicated to A, and 10 dedicated to A on four nodes.

  • Per-engine batch cap: 8, 16, and unlimited on one node; 8, 16, and 32 on four nodes.

  • MPS thread share: 100%, 50%, and proportional to the demand of each engine. Single node (Table 3 setting). One H100 node with 8 GPUs, serving three LLaMA3.1-8B services (A, B, C, with 70/15/15% of the requests) on four prefill and four decode GPUs. In Table R3.1, each cell shows throughput per GPU (tok/s) / p95 TPOT (ms).

Table R3.1: Throughput per GPU and p95 TPOT on one H100 node (Table 3 setting).

ConfigurationGPUs4 RPS8 RPS
Popularity-aware isolated8645.7 / 9.66861.6 / 10.70
MPS, paper configuration8574.0 / 14.14693.2 / 14.88
MPS, range over 27 configurations8366–574 / 12.9–16.6374–724 / 13.0–17.0
SUN, 4 shared decoders8656.4 / 8.99908.8 / 9.60
SUN, 3 shared decoders7737.0 / 9.55990.0 / 10.53

Four nodes (Table 4 setting). Four H100 nodes with 32 GPUs, serving twelve Qwen3-14B services. Table R3.2 shows total throughput (tok/s) and p95 TPOT (ms).

Table R3.2: Total throughput and p95 TPOT on four H100 nodes (Table 4 setting).

ConfigurationThroughput (tok/s)p95 TPOT (ms)
Popularity-aware isolated7,87615.92
MPS, paper configuration7,41924.70
MPS, range over 27 configurations4,771–7,91324.0–31.9
SUN, 16 shared decoders7,91114.97

W1 + Q1 (2)

  • On one node, the gap remains under every MPS configuration. We re-measured this setting on H100 instead of the A100 used in the paper, and the trend observed on A100 holds. In Table R3.1, every one of the 27 MPS configurations has lower throughput per GPU and higher p95 TPOT than SUN. With the same 8 GPUs, SUN has 14–26% higher throughput per GPU than the best MPS configuration, and a p95 TPOT 26–31% lower than the lowest p95 TPOT of any MPS configuration. Against the configuration in the paper, the throughput gain of SUN is 14.4% at 4 RPS, the load used in the paper, and 31.1% at 8 RPS, compared with the 23.6% reported in the paper.
  • On four nodes, the throughput gap closes, but the latency gap remains under every MPS configuration. In Table R3.2, the best MPS configuration reaches the throughput of SUN, 7,913 versus 7,911 tok/s, so the +32.2% reported in Appendix C.1 does not hold. The p95 TPOT of SUN is 14.97 ms, compared with at least 24.0 ms for every MPS configuration. It is therefore at least 37% lower than that of any MPS configuration, and at least 39% lower than those that reach the throughput of SUN. In addition, every MPS configuration relies on the demand being known in advance, whereas SUN routes requests to any shared decoder without knowing the demand. SUN also matches the throughput of popularity-aware isolated serving with a 6% lower p95 TPOT.
  • Best MPS configurations. On one node, the best MPS configuration is the configuration in the paper at 4 RPS, and the same placement and batch cap with a demand-proportional thread share at 8 RPS. On four nodes, it dedicates 8 decode GPUs to A, with a batch cap of 32 and a demand-proportional thread share; the variants with a batch cap of 16 or a 100% thread share are within 0.02% of its throughput. Why no MPS configuration closes the gap. The gap does not depend on the configuration, because it comes from colocation itself. On one node, even the best of the 27 configurations has a p95 TPOT of 12.9 ms, above the 9.66 ms of isolated serving, and neither removing the batch cap nor changing the thread share closes this gap. When two model-specific engines share one GPU, the GPU reads the weights of both models in every decode step, and the requests of each model form separate batches. The GPU memory also holds two copies of the weights, so however the memory is divided, each engine has less KV cache space than an engine on a dedicated GPU. In the four-node re-measurement, the KV cache of the MPS engines is 90–93% full at p95 in the configurations that reach the throughput of SUN, compared with 13% for SUN. No batch cap, thread share, or placement removes these costs, because MPS keeps a separate engine for each model. SUN avoids all three costs, because every request decodes with the same weights in one batch.

For the same reason, MPS does not outperform popularity-aware isolated serving in this comparison. The isolated baseline is also configured with the demand known in advance, so its GPUs are already balanced, and colocation adds the costs above without raising utilization. MPS helps when isolated GPUs sit idle because some models receive few requests, which an allocation based on the known demand already avoids.

Why we re-measured. When we re-ran the comparison on another available H100 cluster, the submitted four-node numbers did not reproduce. We no longer have access to the original cluster, so we could not identify the exact cause. On our cluster, disabling GPUDirect RDMA for the KV transfer brings isolated serving to 4,059 tok/s, close to the submitted 3,598 tok/s, whereas with GPUDirect RDMA enabled it reaches 7,876 tok/s. We ran all re-measurements, including the MPS configuration sweep, with GPUDirect RDMA enabled. Every system in the four-node comparison transfers KV from the prefill nodes to the decode nodes through the same cross-node path, so a difference in this path affects all systems; we therefore re-measured every system rather than MPS alone. The single-node comparison leads to the same conclusion as in the paper.

Revisions. We will replace Tables 3 and 4 with these measurements, revise the +23.6% and +32.2% statements and the other four-node comparisons in Appendix C.1 accordingly, and add the full sweep data to the appendix. We thank the reviewer for this question, which gave us the opportunity to re-examine these measurements.

W2 + Q2

In our re-measurement, the TPOT increase that the reviewer asks about no longer appears. At 8 RPS, SUN 2P/1D uses 25% fewer GPUs than isolated serving and still has a 17% lower p95 TPOT, 9.4 versus 11.3 ms. With the RDMA path configured correctly, KV transfer does not contribute to TPOT. A request’s KV cache is transferred once, before decoding starts, so its cost appears in TTFT and not in TPOT. We re-ran Table 2 with per-request logging on four H100 nodes (3 runs each) after correcting the RDMA configuration described in our response to W1, to measure both.

  • Latency. One transfer takes 16–19 ms per request in all three configurations, so their p95 TTFT at 8 RPS is the same (234–237 ms).
  • Bandwidth. The KV cache of Qwen3-14B is 160 KiB per token, and the mean prompt has 1,712 tokens. The decode side receives 2.25 GB/s at 8 RPS and 4.4 GB/s at 16 RPS, which is about 4.5% and 9% of one 400 Gb/s link. TPOT follows the decode batch size. Measuring one decoder at different batch sizes gives TPOT = 8.16 + 0.145 × batch (ms). This predicts the TPOT of each configuration at 8 RPS (median for SUN, p95 for the hot-model worker of isolated serving), as shown in Table R3.3.

Table R3.3: Decode batch size, TPOT predicted from the batch size, and measured TPOT at 8 RPS.

ConfigurationDecode batchPredicted TPOT (ms)Measured TPOT (ms)
SUN 2P/2D2.1 (mean)8.58.51 (p50)
SUN 2P/1D4.4 (mean)8.88.90 (p50)
Isolated, hot-model worker21 (p95)11.211.25 (p95)

This decomposition answers the question of attribution. The decode batch alone predicts the measured TPOT of every configuration within 0.1 ms, so no part of TPOT remains to be attributed to KV transfer.

Re-measured Table 2. The p95 TPOT of 14.3 ms for SUN 2P/1D came from the run with the misconfigured RDMA path and does not reproduce with the corrected configuration. In Table R3.4, consolidating from two decode nodes to one now raises p95 TPOT by about 0.5 ms at 8 RPS (9.0 → 9.4 ms). This matches the 0.58 ms predicted from the larger batch (p95 batch 4 → 8), so the increase comes from consolidation and not from KV transfer.

Table R3.4: Re-measured Table 2 on four H100 nodes (mean of 3 runs).

RPSConfigurationGPUsThroughput (tok/s)Throughput/GPUp95 TPOT (ms)p95 TTFT (ms)
8Isolated 2×(1P/1D)321,94460.811.3237
8SUN 2P/2D321,96761.59.0236
8SUN 2P/1D241,96581.99.4234
16Isolated 2×(1P/1D)323,312103.513.044,899*
16SUN 2P/2D323,827119.69.5339
16SUN 2P/1D243,815159.010.5343

* Not a steady-state value: at 16 RPS the hot model’s single isolated decoder is past its capacity (11.2 req/s offered, about 10.1 req/s served at its admission cap of 32), so its queue grows throughout the 300 s run. TTFT of the other models stays at 0.2 s.

Revision. We will replace Table 2 with the values in Table R3.4 and update the statements in the abstract and Section 4.3 and Appendix C.2 that cite it.

Reviewer_J9MGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SUN, a training-serving co-design for disaggregated serving of task-specialized LLMs. SUN trains a separate prefill model for each task while keeping a common decoder frozen, allowing KV caches from different prefill models to be routed to a shared decode pool and batched together. Experiments on LLaMA and Qwen models show accuracy generally close to full fine-tuning, decode-GPU consolidation under skewed demand, and improvements over isolated serving, MPS colocation, and multi-LoRA.

Critical strengths
  1. The idea of training task-specific prefill models to produce caches compatible with one frozen decoder is interesting and enables cross-task decode batching with standard kernels and serving frameworks.

  2. The evaluation covers multiple tasks, model configurations, demand distributions, and both single-node and multi-node deployments.

  3. The paper reports useful accuracy, throughput, GPU-efficiency, TTFT, and TPOT trade-offs and includes comparisons with multi-LoRA and demand-aware placement.

Critical weaknesses
  1. The multi-LLM framing is broader than the supported setting. All models in a SUN pool must share the same decoder weights, tokenizer, attention configuration, dimensions, and KV layout. They are therefore variants of the same base checkpoint and size that must be retrained using SUN. The evaluations cover several model families and sizes separately, but never share decoding across them. SUN does not address the common multi-model setting containing heterogeneous architectures and sizes.

  2. The closest alternatives are not adequately evaluated. MuxServe supports serving models of different sizes, while DeltaZip directly targets full-model-tuned variants of a common base. Neither is used as a serving baseline, even though MuxServe is cited as motivation and the evaluation uses a DeltaZip workload. Given that SUN requires training specialists, comparisons with DeltaZip or other PEFT settings are needed to establish when this design is preferable.

  3. Long-horizon output quality is not established. After prefill, all new tokens and KV states are produced by the frozen base decoder, with task specialization represented only through the prompt KV cache. The accuracy evaluation does not test whether this specialization remains effective for long contexts or extended tool-use trajectories. The longer-sequence experiments measure serving performance rather than output quality.

  4. The agentic and end-to-end serving benefits remain unclear. Prompt KVs are specific to the prefill model that produced them, so common prefixes cannot generally be shared across specialists. This can duplicate caching and prefill work for shared system prompts and tool schemas, while agent handoffs may require recomputation. The paper does not report cache hit rates, cache duplication, or an actual multi-agent workload. Moreover, the main popularity-aware comparison improves throughput marginally by only 7.0% at the same GPU count.

Questions for authors
  1. What is the most general set of models that can share one SUN decoder? Must all specialists start from exactly the same base checkpoint and model size and be retrained using SUN?

  2. How does SUN compare with MuxServe and DeltaZip under the same hardware, workload, and latency SLO? How do stronger LoRA ranks change the quality-efficiency comparison?

  3. Does the quality gap from full fine-tuning increase with input length, output length, or number of reasoning steps?

  4. Since prefixes may not be shared across specialists, what are the TTFT, memory, and recomputation costs for shared tool schemas, multi-turn sessions, and agentic scenarios?

8 comments

W1 + Q1

As the reviewer points out, SUN requires all specialists to start from the same base checkpoint and model size. Because a SUN pool uses one frozen decoder checkpoint, all models in the pool share the architecture, model size, tokenizer, attention configuration, and KV of that checkpoint. Each specialist's prefill module must also be trained with that decoder as demonstrated in Section 3.2.2 of the manuscript. An existing Full-FT checkpoint cannot join a pool without retraining, because reusing its prefill with the frozen decoder degrades accuracy, as shown in Fig. 2. For a new specialist this is not an extra step: prefill-only tuning replaces Full-FT rather than following it (3.0–5.9 GPU-hours per Qwen3-14B specialist, Appendix C.2).

SUN targets models that share a backbone, and we will state this scope clearly. SUN does not share decoding across different architectures or sizes. We will revise the abstract and Introduction to describe SUN as serving multiple task-specialized variants of a shared backbone, and move this limitation from Appendix D into the main text.

The supported setting is nevertheless common. Multi-agent systems fine-tune one backbone into several role-specific models and run them together for each task. Shen et al. (2024) fine-tune three copies of one backbone as a planner, a caller, and a summarizer, and Subramaniam et al. (2025) specialize a group of models that all start from the same base model. In a fleet with several backbones, SUN still applies within each backbone: models of the same backbone share one decoder pool.

Each specialist’s prefill module must be trained with SUN, but this one-time cost is amortized by the serving savings. As described in Section 3.2.2, each specialist's prefill module must indeed be trained together with the frozen shared decoder. This training, however, is not an extra step, because it replaces Full-FT rather than following it. In addition, the training cost is paid only once per model version and is amortized by the serving savings. As shown in Appendix C.2, training takes 3.0–5.9 GPU-hours per Qwen3-14B specialist, whereas the serving savings of SUN recur for every hour of operation and reach 192 GPU-hours per day in the eight-model deployment.

W2 + Q2 (1)

Comparison to DeltaZip and MuxServe. Under the same hardware, workload, and latency SLO, SUN sustains 4 req/s, twice the isolated baseline, while DeltaZip and MuxServe sustain at most 1 req/s.

Setup. We ran the original implementations of DeltaZip (Yao et al., 2025) and MuxServe (Duan et al., 2024), which are built on vLLM 0.3.3 and vLLM 0.2.0 respectively, and changed only what H100 and LLaMA3.1 require. All systems serve four LLaMA3.1-8B specialists on one 8×H100 node, and SUN and the isolated baseline run on vLLM 0.11. All systems serve the same Azure trace with a skewed model popularity of 23:4:3:3. A request meets the SLO when its TTFT and TPOT are within 5× the unloaded values, and goodput is the highest load at which at least 99% of requests meet the SLO. For MuxServe, we report the best placement. Table R2.1 shows the results.

Table R2.1: Goodput and p95 TPOT of SUN, DeltaZip, and MuxServe under the same hardware, workload, and SLO.

ConfigurationGoodput (req/s)p95 TPOT at 4 req/s (ms)TPOT gap with SUNTPOT gap explained by vLLM engine diff.
Isolated 4×(1P/1D)29.11.4 ms– (same engine)
DeltaZip, 8 instances × 1 GPU17264.4 ms1.7 ms (3%)
DeltaZip, 4 instances × 2 GPUs (TP=2)09385.3 ms7.8 ms (9%)
MuxServe, best placement (4 meshes × TP2)12921.0 ms2.7 ms (13%)
SUN 4P/4D47.7––

The gap comes from how each system handles decoding.

  • DeltaZip adds per-model work to decoding. Every decode step adds a compressed-delta product (SBMM) for each distinct model in the batch. In the DeltaZip engine, base-only decode takes 8.9 ms per token, one delta 31.6 ms, and four mixed deltas 63.8 ms. The older engine of DeltaZip explains only 1.7 ms of the 64.4 ms gap, as shown in the last column. In contrast, SUN adds no per-model work to decoding, because all specialists decode with the same weights in one batch, and SUN keeps p95 TPOT at 7.7 ms.

  • MuxServe shares GPUs but not decoding. Each model decodes a separate batch with separate weights, so the best placement gives each model a dedicated 2-GPU mesh, as in isolated serving; colocating models only trades TTFT against TPOT. The older engine of MuxServe explains only 2.7 ms of the 21.0 ms gap, and most of the gap comes from running prefill and decode on the same GPUs, which share the SMs. SUN instead merges the requests of all specialists into one decode batch and runs prefill and decode on separate GPUs, so decoding neither splits into per-model batches nor competes with prefill for SMs. DeltaZip and MuxServe fit other settings.

  • DeltaZip: many rarely used variants. DeltaZip trades decode speed for capacity. The compressed deltas take less memory than the full prefill modules of SUN, so under light load DeltaZip can keep more rarely used variants resident, but every decode step incurs the cost of the deltas in the batch. The DeltaZip authors note in Section 8 that "when there are only a handful of models to serve and they fit in GPU memory, DeltaZip may not be suitable". When the specialists fit in GPU memory and the load is heavy or skewed, as in our setting, SUN is the better fit.

  • MuxServe: models of different sizes. MuxServe colocates models of any size, which SUN cannot pool, because all models in a SUN pool share one decoder. The two can be used together: when models of other sizes join the fleet, SUN pools the specialists of each backbone, and MuxServe places these pools alongside the other models.

W2 + Q2 (2)

Stronger LoRA ranks. We trained LoRA at ranks 16, 128, and 1,024 on EnvSimBench, where Qwen3-4B reads a tool environment and the outputs of previous tool calls, and predicts the state change and feedback of the next call, scored by Config Match and Feedback Match respectively. Each configuration serves requests of EnvSimBench length, averaging 9.5K input and 344 output tokens, at 5 req/s on one H100 node with 4P/4D, and Full-FT models are served in isolation. In Table R2.2, each α column shows throughput in tok/s and p95 TPOT in ms, against an offered load of about 1.7K tok/s.

Table R2.2: Accuracy and serving performance of LoRA at different ranks on EnvSimBench (Qwen3-4B). Each α column shows throughput (tok/s) / p95 TPOT (ms).

ConfigurationConfig MatchFeedback Matchα=0α=1.5α=3
Full-FT39.5882.291,766 / 14.41,611 / 46.81,153 / 53.2
SUN39.5880.211,767 / 10.51,679 / 10.21,753 / 10.0
LoRA r=1628.1369.791,768 / 14.61,677 / 13.91,745 / 12.8
LoRA r=12839.5869.791,769 / 21.51,680 / 19.91,536 / 16.2
LoRA r=1,02439.5875.001,633 / 70.81,100 / 61.6687 / 49.4
  • Accuracy: higher ranks narrow the gap but stay below SUN. Config Match reaches the Full-FT level from rank 128, but Feedback Match, which requires the largest change from the base model, rises from 69.79 at ranks 16 and 128 to only 75.00 at rank 1,024, compared with 80.21 for SUN and 82.29 for Full-FT. Config Match is the same for all methods from rank 128 (39.58). We attribute this gap to capacity: reproducing exact tool feedback over long contexts needs a larger update than a low-rank adapter provides, whereas SUN updates every parameter of the prefill module.
  • Efficiency: higher ranks slow decoding, and merging does not help. Every decode step computes the adapter of each request in the batch, so decoding slows as rank grows. At rank 1,024, p95 TPOT is nearly seven times that of SUN, and LoRA cannot keep up with the offered load under skew. Merging adapter removes the adapter overhead, but the merged model becomes a separate full model per task, which is served like the isolated Full-FT models and slows down just as much under skew, while keeping the LoRA accuracy.
Reviewer_LvcoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SUN, a system designed to improve Decode GPU utilization in P/D-disaggregated serving of task-specific LLMs. SUN fine-tunes the Prefill component for each task while keeping the Decoder frozen and shared across models. It further supports KV Cache compatibility, cross-model batching, and model-independent routing to improve resource efficiency. Experiments on mathematical reasoning, code generation, and tool-calling tasks using LLaMA and Qwen show accuracy comparable to full fine-tuning. Under skewed workloads, SUN improves single-GPU Decode throughput by up to 2.0×. In an eight-model deployment, it maintains approximately 98.7% of the original throughput while using only 75% of the GPU resources.

Critical strengths
  1. Training design addresses the compatibility challenges of KV Cache.
  2. The system experiment covers key loads and competing schemes. The paper compares request skewing, heat perception allocation, MPS and multi-node deployment.
Critical weaknesses
  1. The corresponding relationship between the actual load and the sharing conditions remains unclear. SUN only supports task models that are of the same origin and compatible with the KV interface. However, the public model popularity statistics in the motivation do not indicate the request distribution of such models in real deployments. The multi-node experiments, which used proxy popularity and the set hotspot ratio, verified the effectiveness under skewed load, but this is not sufficient to demonstrate the actual representativeness of this scenario.
  2. Lack of more targeted training baselines. Fig. 2 demonstrates that the Prefill feature of the ordinary Full-FT cannot directly be combined with the frozen Decoder, but it does not explain why the Prefill must be fully fine-tuned. The absence of alternatives such as Prefill-only LoRA, Prefix-Tuning, etc., makes it difficult to determine whether the additional training and storage costs are worthwhile. The relationship between the selective freezing and sharing paradigm in this study and those in ICaRus (ICLR 2026) and PrefillShare (2026 arxiv) also requires a more explicit discussion.
Questions for authors
  1. Is full Prefill fine-tuning necessary? Under the same conditions of fully identical frozen Decoder, service implementation, and task training data, can Prefill-only LoRA or Prefix/Deep Prompt Tuning achieve a task accuracy rate close to that of SUN? Can it also report training GPU-hours, the storage/deployment occupancy of each task weights, TTFT, and TPOT?

  2. How to quantitatively isolate the benefits of cross-model batch processing and load balancing? Is it possible to compare (a) model-specific Decode Worker, (b) shared Worker but prohibiting cross-task access to the same Decode Batch, and (c) full SUN under the same SUN fine-tuning model and the same GPU/request load? Please provide evidence such as actual Batch Size, GPU memory bandwidth, or Kernel time to confirm the main source of benefits.

  3. How much GPU is actually saved under the same latency SLO? Can the SLO-aware Goodput/GPU or the minimum number of Gpus required to reach the target Goodput be supplemented when the p95 TTFT and TPOT thresholds are fixed?

5 comments

W1

Mismatch between statistics in motivation and SUN’s real target scenario. We thank the reviewer for raising this point. We agree that the OpenRouter statistics in our motivation show demand heterogeneity across heterogeneous models, not across the KV-compatible variants that SUN targets. Using them to motivate skew among same-base variants was an imprecise link, and we will replace them with more direct evidence below.

  • Task specialists released from one backbone. Models fine-tuned from one general-purpose backbone differ widely in how often they are used. Qwen3Guard-Gen-8B and Fin-o1-8B are both fine-tuned from Qwen3-8B: the former moderates user prompts and model responses (Alibaba, 2025), and the latter is specialized for financial reasoning (TheFinAI, 2025). A safety model is applied to every prompt and response, while a task specialist is called only for its own task, so their demand is uneven by design.
  • Role-specific models in multi-agent systems. One backbone is fine-tuned into several specialized models that are served together to complete each task (Shen et al., 2024; Subramaniam et al., 2025). In Shen et al., for example, the planner and the caller run at every step, while the summarizer runs once per task, so the demand differs across models that are served at the same time. We will revise the motivation (Sections 1 and 2.2) to rely on this evidence instead, and state the scope as serving multiple fine-tuned specialists that share a common backbone.

Representativeness of proxy data. To our knowledge, no public trace records per-model request rates for full-model-tuned variants of a common base. We therefore followed the workload methodology of DeltaZip (Yao et al., 2025), which targets the same setting.

Because a proxy cannot establish representativeness by itself, we evaluated SUN across a range of conditions rather than a single workload, with demand ranging from perfectly uniform to heavily concentrated on a single model. We varied demand skew (Zipf α from 0 to 3 in Fig. 4, a 70/15/15 split in Table 3, and a 70% hot model over eight models in Table 2), request lengths (fixed ISL/OSL with OSL from 128 to 2,048, and variable-length prompts sampled from LMSYS Chatbot Arena), offered load (8 and 16 RPS), and deployment scale (three to twelve models, single-node and multi-node).

SUN does not need to know the skew in advance. The results in manuscripts show that the gain of SUN comes from sharing the decoder, not from matching an assumed popularity distribution. Because all specialists run on the same decode weights, SUN batches requests from every specialist together, regardless of which model each request belongs to. As shown in Fig. 3 of the manuscript, even under uniform demand, SUN improves throughput per decode GPU by 33%. Skewed demand extends the gain to 2.0×, but SUN does not need to know which model will be popular to obtain the larger gain. The benefit of SUN therefore does not hinge on how accurately our proxy reflects real-world popularity.

W2 + Q1

We thank the reviewer for this suggestion. We trained prefill-only LoRA (r=32) and prefix tuning with the same frozen decoder, training data, and recipe as SUN (Qwen3-8B-Base, 1 epoch, learning rate tuned separately for each method). We then measured accuracy, training cost, storage, and serving latency. Prefix tuning learns a per-layer KV prefix of 64 tokens.

Accuracy. In Table R1.1, the Full-FT and SUN rows are from Table 1 of the paper.

Table R1.1: Task accuracy of each training method with the frozen shared decoder (Qwen3-8B-Base).

MethodDecodeGSM8KGSM+HEHE+BFCL Simple PythonBFCL Multiple
Full-FTIndependent85.865.783.574.393.392.0
SUN (full prefill)Shared85.564.584.176.293.391.5
Prefill-only LoRAShared74.355.183.576.890.090.0
Prefix tuningShared37.128.682.976.285.887.5

Cost. Table R1.2 compares the costs. Training is on 8×H200. Serving uses one H100 node with four prefill and four decode GPUs (ISL 1024 / OSL 256, 8 RPS, Zipf α=3), averaged over 3 seeds. Full-FT is served with model-specific decoders, 4×(1P/1D), while the other methods share the decoders (4P/4D). Prefill-only LoRA is served with an adapter unmerged; merging removes the TTFT overhead but also the storage saving, so a merged LoRA has the same serving and storage cost as SUN with lower accuracy.

Table R1.2: Training cost, storage, and serving latency of each training method.

MethodTrainable paramsStorage per taskWeights per prefill GPUTraining GPU-h (math / code / tool)TTFT p95 (ms)TPOT mean (ms)
Full-FT8.19B16.4 GB16.4GB1.9 / 3.6 / 2.8100.19.13
SUN6.95B13.9 GB13.9GB1.9 / 3.7 / 2.891.27.74
Prefill-only LoRA87.3M175 MB13.9 GB + 175 MB2.2 / 4.1 / 3.2116.97.73
Prefix tuning4.72M9.4 MB13.9 GB + 9.4 MB1.4 / 5.8 / 2.494.67.81

Summary.

  • Prefix tuning and prefill-only LoRA are not sufficient. Prefix tuning trails SUN by 48.4 and 35.9 points on GSM8K and GSM+, and by 7.5 and 4.0 points on BFCL Simple and Multiple. Prefill-only LoRA trails SUN by 11.2 and 9.4 points on GSM8K and GSM+, and by 3.3 and 1.5 points on BFCL Simple and Multiple. Both are comparable to SUN only on coding (within 1.2 points). We attribute this to capacity: the prefill module steers the frozen decoder only through the KV cache it writes, and a low-rank or prefix-only update is not enough for tasks such as math that need the largest change in output behavior. Prefill-only LoRA also raises p95 TTFT from 91.2 to 116.9 ms (+28%) due to adapter overhead; TPOT is unchanged because the adapter runs only in the prefill engines.
  • Prefill-only LoRA saves storage but not training cost. It reduces per-task storage by about 80×. Although prefill-only LoRA skips the weight gradients of the prefill module, the adapters add small low-rank matrix multiplications to every layer in both the forward and backward passes, and these run inefficiently on GPUs, so training is 11–16% slower than SUN in wall-clock time. Relation to ICaRus and PrefillShare. ICaRus (Woo et al., 2026a) and PrefillShare (Woo et al., 2026b) make the opposite design choice from SUN. They freeze and share the prefill side (the module that produces the KV cache) and adapt the decode side per task. This removes redundant prefill computation and duplicate KV caches for shared contexts, but decode execution remains model-specific, so decode requests still cannot be batched across models. SUN instead freezes and shares the decoder and adapts the prefill side. The two designs address different bottlenecks: prefill-sharing designs suit workloads dominated by shared long contexts, and SUN suits decode-heavy workloads.

Q2

We ran the requested ablation on one H100 node with four LLaMA3.1-8B specialists (ISL 1024, OSL 2048, 4 req/s, Zipf α ∈ {0, 1.5, 3}, 3 runs). All configurations use the same SUN models and load: (a) model-specific decode workers; (b) shared workers with cross-task batching disabled: a worker accepts requests of any task, but each forward passes batches requests of a single task; (c) SUN. (a)→(b) isolates load balancing and (b)→(c) isolates cross-task batching. Table R1.3 shows the results.

Table R1.3: Throughput (tok/s) and TPOT (ms) of the three decode configurations under different demand skews (one H100 node, four LLaMA3.1-8B specialists).

ConfigurationDecode GPUsα=0: tok/s / TPOT (ms)α=1.5: tok/s / TPOT (ms)α=3: tok/s / TPOT (ms)
(a) Model-specific48,187 / 9.48,116 / 12.97,748 / 23.9
(b) Shared, no cross-task batch47,861 / 36.97,798 / 37.17,797 / 31.8
(c) SUN48,185 / 9.38,188 / 9.48,187 / 9.3
(b) Shared, no cross-task batch27,317 / 50.37,278 / 51.17,187 / 50.0
(c) SUN28,144 / 12.98,143 / 12.98,143 / 12.9

(1) Load balancing removes the skew penalty, and cross-task batching keeps TPOT low. With four workers, (a) loses 5% of its throughput from α=0 to α=3, while (b) and SUN stay within 1%. Load balancing alone ((a)→(b)) thus removes the skew penalty, but it raises TPOT to 32–37 ms at every skew level and costs 4% of throughput at α ≤ 1.5. Without cross-task batching, a forward pass serves only one task, so the requests of the other tasks have to wait. Each decoder therefore holds 3.2–3.8× more requests at a time (61–73 vs. 19) and about 3× more KV cache. Adding cross-task batching ((b)→(c)) removes this wait: TPOT falls to 9.3 ms (−71% to −75%) and throughput rises another 4–5%. Overall, SUN raises throughput by 6% and reduces TPOT by 61% relative to (a) at α=3.

(2) With half the decode workers, only SUN keeps its throughput. Reducing the pool from four to two workers costs SUN less than 1% of its throughput, while (b) loses 7–8% because each worker must hold 175–181 requests and exhausts its KV memory. TPOT shows the same gap: (b) reaches 50–51 ms (2–5× that of (a)), while SUN is at 13 ms. As a result, SUN with two workers serves the load of the four model-specific workers in (a) at every skew level (−0.5% at α=0, +0.3% at α=1.5, +5% at α=3). Cross-task batching makes this possible: at α=3, SUN runs 52 requests per forward pass and reads 307 MB of weights per output token, compared with 44 requests and 364 MB for (b).

In summary, the gain comes from both sources, and they play different roles: load balancing removes the throughput loss under skew, and cross-task batching keeps TPOT low and lets SUN serve the same load with fewer decode GPUs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.