- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses inter-model isolation in disaggregated multi-LLM serving: conventional disaggregation separates prefill from decode, but decode workers remain partitioned per model, causing GPU underutilization under skewed workloads (popular models queue while idle capacity behind less-requested models cannot be reused). The authors propose SUN (Shared Use of Next-token Prediction), which:
- Formalizes prefill–decode decoupling: decomposes decoder-only Transformer execution into a prefill module (producing first-token distribution and prompt KV cache) and a decode module , which need not share parameters (unlike conventional models where ).
- Prefill-only tuning: each task-specific prefill module is fine-tuned against the frozen shared decoder (the same decoder used at inference), so its prompt KV cache is directly interpretable by the shared decoder. This avoids the train–inference mismatch of naive sharing (GSM8K drops 71.3%→51.2% without it) and the accuracy/efficiency trade-offs of Multi-LoRA serving.
- Model-agnostic decode routing: since all decode requests use identical frozen weights, requests from different prefill modules can enter one decode batch in a unified vLLM-based worker pool, enabling cross-model batching and GPU consolidation. Results: Accuracy is comparable to Full-FT across LLaMA3.1-8B and Qwen3-1.7B/8B/14B on math (GSM8K/GSM+), coding (HumanEval/HumanEval+), and tool calling (BFCL). System-wise, SUN achieves up to 2.0× throughput per decode GPU with ≤10% TPOT degradation; in eight-model multi-node serving it retains 98.7% of baseline throughput with 25% fewer GPUs at moderate load and gains 29.1% throughput on the same hardware at higher load. It outperforms popularity-aware allocation (+7.0%) and MPS-based GPU colocation (+23.6%, extending to +32.2% in a twelve-service evaluation).
Critical strengths
- First formalization of prefill–decode parameter decoupling for cross-model decode sharing. The paper explicitly formulates the prefill and decode modules as separately parameterized components (Eqs. 1–4), turning an implicit systems intuition into a clean mathematical model that enables shared decoding — a genuinely novel framing relative to Splitwise/DistServe/Mooncake.
- Prefill-only tuning elegantly resolves the KV-cache compatibility problem. Training the task-specific prefill module against the inference-time frozen decoder aligns the prompt KV cache with shared decoding (Fig. 2), avoiding the naive-sharing accuracy collapse while retaining full-parameter adaptation capacity (superior to LoRA's low-rank restriction) and eliminating adapter computation from the decode path (no per-step adapter overhead, unlike Punica/S-LoRA).
- Model-agnostic decode routing converts underutilized capacity into real resource savings. Pooling decode workers enables cross-model batching: up to 2.0× per-GPU throughput with ≤10% TPOT degradation, 25% total GPU reduction at moderate load (Table 2), and robustness to workload skew (interactivity +44.1% at α=3.0, Fig. 4/Fig. 6).
- Comprehensive and honest evaluation design. The paper sweeps decode pool sizes, Zipf skew α, model scales (1.7B–14B), single-node and multi-node (TP=2, cross-node KV transfer) settings, and reports sanity checks (achieved/offered load ratios in Appendix B, one-time training cost quantification in Appendix C.2).
- Well-motivated baselines addressing the obvious rebuttals. The comparison against popularity-aware allocation and workload-informed MPS colocation demonstrates that SUN's gains come from cross-model batching (breaking the model-specific batching boundary), not merely from demand-aware placement or physical GPU sharing — the batching-opportunity analysis (λ/16 vs. pᵢλ/rᵢ arrivals) makes the mechanism explicit.
Critical weaknesses
- Fairness of the MPS baseline is questionable. In Table 3, MPS p95 TPOT (23.90 ms) is worse than isolated serving (19.36 ms), and in Appendix C.1 MPS TPOT explodes to 43.56 ms. While Appendix A.5/C.1 grant MPS favorable assumptions (full workload knowledge, 100% thread cap, resident workers, no loading costs), the configurations (e.g., batch caps, the hot model A occupying all 16 GPUs) may have been artificially placed at a disadvantage. The +23.6%/+32.2% comparisons would be more credible with justification that the MPS configurations are at least reasonably tuned.
- Cross-node KV transfer overhead is not quantified. Sections 3.2.3 and A.4 acknowledge that cross-node KV transfer is a necessary cost of shared decoding, but the paper never reports the bandwidth/latency cost of KV transfers. It is unclear how much of the p95 TPOT increase from 10.6→14.3 ms (Table 2, 2P/1D vs. isolated) is attributable to KV transfer versus queueing on fewer decode nodes — an ablation is needed.
- Consolidation boundary conditions are insufficiently discussed. Fig. 3 shows that at OSL=512, α=1.5, total throughput drops from 6,323 (4D) to 3,910 (1D), indicating a single decode node saturates. The paper presents sweeps but gives no criterion for when it is safe to cut GPUs (e.g., a load/SLA-aware admission rule), nor any auto-scaling policy for the shared pool — which limits practical applicability of the claimed TCO savings.
- Narrow compatibility requirement limits generality. SUN requires an identical backbone, tokenizer, attention configuration, and KV-cache layout across all shared models (Section 3.2.3, Appendix D). This restricts sharing to fine-tunes of a common base model; heterogeneous fleets (the realistic production scenario) are out of scope, and the paper offers no discussion of bridging mechanisms.
- Accuracy gaps are task-dependent and not fully explained. On Qwen3-14B, SUN trails Full-FT by 2.7/2.9 points on GSM8K/GSM+ and LLaMA3.1-8B shows a 4.2-point gap on HumanEval+. The paper does not analyze why certain task–model combinations degrade (e.g., whether long-form reasoning chains are more sensitive to prefill–decode parameter mismatch), nor whether hyperparameter search could close these gaps.
Questions for authors
- MPS baseline configuration: Can you demonstrate that the MPS results in Table 3 and Table 4 are not artificially weakened? Specifically, how were the MPS batch caps, thread caps, and engine placements (e.g., A on all 16 GPUs) selected, and would alternative MPS configurations (e.g., different per-engine batch limits or replica placements) change the +23.6%/+32.2% throughput gap?
- KV transfer cost: Please quantify the bandwidth and latency cost of cross-node KV transfer in the multi-node experiments. In Table 2, how much of the p95 TPOT increase (10.6→14.3 ms for 2P/1D at 8 RPS) is attributable to KV transfer versus decode-node consolidation?
- Safe consolidation policy: Given that Fig. 3 shows single-decode-node saturation at OSL=512, α=1.5 (throughput 6,323→3,910), what operational criterion (e.g., queue depth, KV occupancy, offered-load threshold) should practitioners use to decide when shrinking the decode pool is safe? Could such a rule be automated?
7 comments
W1 + Q1 (2)
- On one node, the gap remains under every MPS configuration. We re-measured this setting on H100 instead of the A100 used in the paper, and the trend observed on A100 holds. In Table R3.1, every one of the 27 MPS configurations has lower throughput per GPU and higher p95 TPOT than SUN. With the same 8 GPUs, SUN has 14–26% higher throughput per GPU than the best MPS configuration, and a p95 TPOT 26–31% lower than the lowest p95 TPOT of any MPS configuration. Against the configuration in the paper, the throughput gain of SUN is 14.4% at 4 RPS, the load used in the paper, and 31.1% at 8 RPS, compared with the 23.6% reported in the paper.
- On four nodes, the throughput gap closes, but the latency gap remains under every MPS configuration. In Table R3.2, the best MPS configuration reaches the throughput of SUN, 7,913 versus 7,911 tok/s, so the +32.2% reported in Appendix C.1 does not hold. The p95 TPOT of SUN is 14.97 ms, compared with at least 24.0 ms for every MPS configuration. It is therefore at least 37% lower than that of any MPS configuration, and at least 39% lower than those that reach the throughput of SUN. In addition, every MPS configuration relies on the demand being known in advance, whereas SUN routes requests to any shared decoder without knowing the demand. SUN also matches the throughput of popularity-aware isolated serving with a 6% lower p95 TPOT.
- Best MPS configurations. On one node, the best MPS configuration is the configuration in the paper at 4 RPS, and the same placement and batch cap with a demand-proportional thread share at 8 RPS. On four nodes, it dedicates 8 decode GPUs to A, with a batch cap of 32 and a demand-proportional thread share; the variants with a batch cap of 16 or a 100% thread share are within 0.02% of its throughput. Why no MPS configuration closes the gap. The gap does not depend on the configuration, because it comes from colocation itself. On one node, even the best of the 27 configurations has a p95 TPOT of 12.9 ms, above the 9.66 ms of isolated serving, and neither removing the batch cap nor changing the thread share closes this gap. When two model-specific engines share one GPU, the GPU reads the weights of both models in every decode step, and the requests of each model form separate batches. The GPU memory also holds two copies of the weights, so however the memory is divided, each engine has less KV cache space than an engine on a dedicated GPU. In the four-node re-measurement, the KV cache of the MPS engines is 90–93% full at p95 in the configurations that reach the throughput of SUN, compared with 13% for SUN. No batch cap, thread share, or placement removes these costs, because MPS keeps a separate engine for each model. SUN avoids all three costs, because every request decodes with the same weights in one batch.
For the same reason, MPS does not outperform popularity-aware isolated serving in this comparison. The isolated baseline is also configured with the demand known in advance, so its GPUs are already balanced, and colocation adds the costs above without raising utilization. MPS helps when isolated GPUs sit idle because some models receive few requests, which an allocation based on the known demand already avoids.
Why we re-measured. When we re-ran the comparison on another available H100 cluster, the submitted four-node numbers did not reproduce. We no longer have access to the original cluster, so we could not identify the exact cause. On our cluster, disabling GPUDirect RDMA for the KV transfer brings isolated serving to 4,059 tok/s, close to the submitted 3,598 tok/s, whereas with GPUDirect RDMA enabled it reaches 7,876 tok/s. We ran all re-measurements, including the MPS configuration sweep, with GPUDirect RDMA enabled. Every system in the four-node comparison transfers KV from the prefill nodes to the decode nodes through the same cross-node path, so a difference in this path affects all systems; we therefore re-measured every system rather than MPS alone. The single-node comparison leads to the same conclusion as in the paper.
Revisions. We will replace Tables 3 and 4 with these measurements, revise the +23.6% and +32.2% statements and the other four-node comparisons in Appendix C.1 accordingly, and add the full sweep data to the appendix. We thank the reviewer for this question, which gave us the opportunity to re-examine these measurements.
W2 + Q2
In our re-measurement, the TPOT increase that the reviewer asks about no longer appears. At 8 RPS, SUN 2P/1D uses 25% fewer GPUs than isolated serving and still has a 17% lower p95 TPOT, 9.4 versus 11.3 ms. With the RDMA path configured correctly, KV transfer does not contribute to TPOT. A request’s KV cache is transferred once, before decoding starts, so its cost appears in TTFT and not in TPOT. We re-ran Table 2 with per-request logging on four H100 nodes (3 runs each) after correcting the RDMA configuration described in our response to W1, to measure both.
- Latency. One transfer takes 16–19 ms per request in all three configurations, so their p95 TTFT at 8 RPS is the same (234–237 ms).
- Bandwidth. The KV cache of Qwen3-14B is 160 KiB per token, and the mean prompt has 1,712 tokens. The decode side receives 2.25 GB/s at 8 RPS and 4.4 GB/s at 16 RPS, which is about 4.5% and 9% of one 400 Gb/s link. TPOT follows the decode batch size. Measuring one decoder at different batch sizes gives TPOT = 8.16 + 0.145 × batch (ms). This predicts the TPOT of each configuration at 8 RPS (median for SUN, p95 for the hot-model worker of isolated serving), as shown in Table R3.3.
Table R3.3: Decode batch size, TPOT predicted from the batch size, and measured TPOT at 8 RPS.
| Configuration | Decode batch | Predicted TPOT (ms) | Measured TPOT (ms) |
|---|---|---|---|
| SUN 2P/2D | 2.1 (mean) | 8.5 | 8.51 (p50) |
| SUN 2P/1D | 4.4 (mean) | 8.8 | 8.90 (p50) |
| Isolated, hot-model worker | 21 (p95) | 11.2 | 11.25 (p95) |
This decomposition answers the question of attribution. The decode batch alone predicts the measured TPOT of every configuration within 0.1 ms, so no part of TPOT remains to be attributed to KV transfer.
Re-measured Table 2. The p95 TPOT of 14.3 ms for SUN 2P/1D came from the run with the misconfigured RDMA path and does not reproduce with the corrected configuration. In Table R3.4, consolidating from two decode nodes to one now raises p95 TPOT by about 0.5 ms at 8 RPS (9.0 → 9.4 ms). This matches the 0.58 ms predicted from the larger batch (p95 batch 4 → 8), so the increase comes from consolidation and not from KV transfer.
Table R3.4: Re-measured Table 2 on four H100 nodes (mean of 3 runs).
| RPS | Configuration | GPUs | Throughput (tok/s) | Throughput/GPU | p95 TPOT (ms) | p95 TTFT (ms) |
|---|---|---|---|---|---|---|
| 8 | Isolated 2×(1P/1D) | 32 | 1,944 | 60.8 | 11.3 | 237 |
| 8 | SUN 2P/2D | 32 | 1,967 | 61.5 | 9.0 | 236 |
| 8 | SUN 2P/1D | 24 | 1,965 | 81.9 | 9.4 | 234 |
| 16 | Isolated 2×(1P/1D) | 32 | 3,312 | 103.5 | 13.0 | 44,899* |
| 16 | SUN 2P/2D | 32 | 3,827 | 119.6 | 9.5 | 339 |
| 16 | SUN 2P/1D | 24 | 3,815 | 159.0 | 10.5 | 343 |
* Not a steady-state value: at 16 RPS the hot model’s single isolated decoder is past its capacity (11.2 req/s offered, about 10.1 req/s served at its admission cap of 32), so its queue grows throughout the 300 s run. TTFT of the other models stays at 0.2 s.
Revision. We will replace Table 2 with the values in Table R3.4 and update the statements in the abstract and Section 4.3 and Appendix C.2 that cite it.
W1 + Q1 (1)
Even against the best of 27 MPS configurations swept along the parameters the reviewer raised, SUN keeps its advantage: 14–26% higher throughput per GPU on one node, and at least 37% lower p95 TPOT than every MPS configuration on four nodes.
How the batch caps, thread caps, and engine placements were selected. All three follow the principle described in Appendices A.5 and C.1: give MPS the most favorable setting, with the demand distribution known in advance.
Engine placement. The hot model A is placed on every decode GPU, so that A can use the capacity of the whole pool, and requests are balanced so that every physical decode GPU receives the same load.
Thread cap. Every MPS client has a 100% active-thread cap, so each engine can use all SMs whenever the colocated engine is idle, instead of being limited to a fixed share.
Batch cap. On one node, no batch cap was set beyond the vLLM default. All 120 requests were in flight at once, so each decode GPU held 30 concurrent sequences (12 and 18 for the two engines of a shared GPU), and no engine reached a batch limit. On four nodes, every engine in every system, including SUN and isolated serving, admits at most 16 concurrent decode requests. On the original cluster, we first used a limit of 128, but two colocated MPS engines then stalled for more than two minutes at 92–94% KV occupancy with requests still waiting. The limit of 16 avoided this stall while still allowing each shared MPS GPU to hold up to 32 concurrent sequences, more than the 30 per GPU of the single-node setting. On the cluster used for the re-measurement below, the same MPS configuration with a limit of 128 completes without stalls, at 7,633 tok/s and a p95 TPOT of 24.65 ms (single run), compared with 7,419 tok/s and 24.70 ms with a limit of 16. Raising the limit therefore increases MPS throughput by 2.9% and leaves its p95 TPOT unchanged, far above the 14.97 ms of SUN. Whether alternative MPS configurations change the +23.6%/+32.2% gap. On one node, they do not: SUN outperforms every MPS configuration. On four nodes, the +32.2% throughput gap disappears after re-measurement, but the latency gap remains under every MPS configuration. Tuning improves MPS throughput by at most 4.4% on one node and 6.6% on four nodes, so the MPS results in the paper were not artificially weakened. To show this, we re-ran both comparisons on H100 with a sweep of 27 MPS configurations per setting, covering every parameter the reviewer mentions:
Placement: A/A/AB/AC as in the paper, A/A/A/BC, and AB/AC/AB/AC on one node; A on all 16 decode GPUs as in the paper, 8 decode GPUs dedicated to A, and 10 dedicated to A on four nodes.
Per-engine batch cap: 8, 16, and unlimited on one node; 8, 16, and 32 on four nodes.
MPS thread share: 100%, 50%, and proportional to the demand of each engine. Single node (Table 3 setting). One H100 node with 8 GPUs, serving three LLaMA3.1-8B services (A, B, C, with 70/15/15% of the requests) on four prefill and four decode GPUs. In Table R3.1, each cell shows throughput per GPU (tok/s) / p95 TPOT (ms).
Table R3.1: Throughput per GPU and p95 TPOT on one H100 node (Table 3 setting).
Four nodes (Table 4 setting). Four H100 nodes with 32 GPUs, serving twelve Qwen3-14B services. Table R3.2 shows total throughput (tok/s) and p95 TPOT (ms).
Table R3.2: Total throughput and p95 TPOT on four H100 nodes (Table 4 setting).