acceptodds
Under review as a conference paper at ICLR 2027

SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

Abstract

No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independent requests favor data-parallel attention, and long prompts favor context parallelism. Reason- ing, agentic, and RL- rollout workloads make a fixed choice untenable: a batch that begins as many short requests ends as a few very long ones, so the best lay- out changes while the same requests run. Serving engines nevertheless fix one layout at launch, because changing it has meant draining requests and restarting workers. We present SPLASH, a serving system that switches the parallel layout of attention while requests are running. It builds on one observation: modern at- tention, with few or no KV heads, decouples where a request’s KV cache lives from how attention weights are sharded. This has two consequences. First, lay- outs differ only in who owns the weights and the cache, and most of that state already sits where the next layout needs it; SPLASH reuses it, moves the rest in the background of ongoing inference, and hands off at a batch boundary, making a switch nearly free: its median overhead is under 0.51% of the step it runs in. Second, the decoupling exposes a layout that existing engines lack: Decoupled Ownership Parallelism (DOP) shards attention weights as tensor parallelism does while keeping each request’s cache on a single owner as data-parallel attention does. DOP replicates neither, offers 27–60% more KV capacity than data-parallel attention, and gives the scheduler a choice when KV memory limits admission. A transition-aware scheduler follows the best of the four layouts as load changes. On B200 GPUs serving GLM-5.3, SPLASH improves end-to-end serving throughput by 1.3–1.73× over fixed-layout deployments, and the same layout regimes appear with DeepSeek-V3.2 on H200 and GLM-5.3-Flash on DCU.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM Infrastructure, Attention Optimization, MLA, Parallelism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nuiaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes a system that dynamically adjusts GPU parallelization strategies according to workload characteristics. It integrates several attention parallelization layouts into a unified serving system and introduces a layout designed to reduce redundant storage.

Critical strengths

The system has a unified design and addresses practical resource utilization concerns.

Critical weaknesses
  1. Adaptive parallelization, reuse of existing states, and reduction of redundant storage are all directions that prior work has already explored. The contribution of this paper appears to lie primarily in implementation and system integration rather than in the discovery of a fundamentally new layout. This could certainly still make for a strong paper, but it makes the following limitation more consequential.
  2. The paper provides reasonably convincing evidence that different parallelization layouts have different strengths and that switching between them can be inexpensive. However, it does not demonstrate equally clearly how much overall benefit is achieved when the complete system operates under dynamically changing workloads. Given the substantial overlap between its underlying techniques and prior work, this missing system-level validation makes it difficult to assess whether SPLASH delivers a sufficiently significant new contribution.
Questions for authors
  1. Could the authors clarify the specific technical advances of DOP over the parallelization scheme described by Pope et al. (Efficiently Scaling Transformer Inference)? If the main differences are changes to the cache representation and operator implementation, I would still consider the conceptual novelty of the layout itself limited.
  2. Could the authors provide a clear experiment demonstrating that SPLASH achieves a significant throughput improvement over the best reasonable fixed-layout deployment, and that this improvement actually comes from dynamic layout switching?
8 comments

Thank you for the clear feedback on how DOP relates to Pope et al. and on isolating the throughput gain from dynamic switching.

Relation to Pope et al., DOP's MLA/hybrid extensions, and SPLASH's system novelty (§1–§3, §6, Appendix E). DOP shares the high-level idea of Pope et al. (2023, §3.3, Fig. 5b): sharding attention projections across ranks while keeping shared KV request-local via activation all-to-all. We removed the imprecise "new layout" phrasing in §1 and discuss Pope et al. directly in §6 and Appendix E.

However, Pope et al. apply batch-sharded attention only to dense single-head MQA. Supporting modern long-context models required two technical adaptations (detailed in §3 and Appendices A–C, E):

  1. Two-stage absorbed MLA (§2, §3, Appendices A and E): Unlike explicit single-head MQA, MLA uses two-stage projections (Q-A/Q-B, KV-A/KV-B) over a shared latent cache plus a RoPE key, where materializing KV-B over long histories would blow up memory. DOP operates directly on absorbed MLA: sharding Q-B, KV-B, and O, folding KV-B into queries with fused latent/RoPE query packing, and reducing across ranks before owner-row selection (plus DOP-P's optional owner O-weight cache).
  2. Sparse indexers and hybrid recurrent state (Appendices B, C, and E): DOP co-partitions sparse MLA indexer caches and FP8 scale metadata (GLM-5.3, DeepSeek-V3.2) plus interleaved KDA linear/recurrent state (GLM-5.3-Flash) alongside each owner's KV pages.

These adaptations free 12.69 GiB/GPU for KV on B200 (+27.3% over DP-attention, fitting 16 vs. 12 requests at 512K; +59.7% on H200 with DeepSeek-V3.2, Table 3) and let DOP beat TP, CP, DP-attention, DP2+TP4, and DP4+TP2 by 3.4–11.0% across all 256K–512K and B=128,256B=128, 256 points (Figure 4, Table 4).

More broadly, we agree that our main contribution is at the system level rather than claiming sharded-weight/request-owned-KV as a new parallelization principle. To our knowledge, SPLASH is the first general, composable live-switching system for attention layouts that switches in-flight batches at near-zero cost without halting active requests, unlike Flying Serving (MHA/GQA DP↔TP only, pausing or recomputing active DP requests), Shift Parallelism (TP↔SP on one KV layout), Seesaw (phase stalls), or concurrent preprints Moebius, ReMP, and PipeLive (coarse step/transaction commits in tens of ms to seconds; §6, Appendix H). Aligning DOP's shards with TP and owner pages with DP-attention under pre-captured CUDA graphs makes TP → DOP and DP-attention → DOP zero-network-byte switches (0.02 ms median), covering all 30 switches across TP, CP, DP-attention, DOP, and DP×TP in 0.02–11.76 ms (< 0.51% of a step, Tables 1 and 7).

Throughput gain from dynamic switching (§5.2, §5.6, Figures 4 and 6). As a 64K batch drains from 256 requests to 1 on B200 (Figure 4c, Table 4), five layouts take turns leading (DOP → CP → DP-attention → DP4+TP2 → TP), and any fixed choice loses 35–57% outside its best regime. Replaying 24-hour AgentX and production traces (§5.6, Figure 6; 24 h cumulative speedup, with instantaneous min–max in parentheses):

Trace and metric (24 h window)SPLASH vs. fixed TPSPLASH vs. fixed DPSPLASH vs. fixed CPSPLASH vs. fixed DOPSPLASH vs. fixed DP2+TP4SPLASH vs. fixed DP4+TP2
AgentX request throughput (RPS)1.69× (0.98–2.49×)1.30× (1.01–1.90×)1.16× (1.00–1.29×)1.16× (1.02–1.49×)1.33× (0.98–1.61×)1.25× (1.01–1.35×)
AgentX total token throughput1.71× (1.01–2.51×)1.30× (0.98–1.91×)1.15× (1.03–1.30×)1.15× (1.01–1.51×)1.33× (1.03–1.60×)1.25× (1.02–1.37×)
Production trace daily request rate1.63× (0.99–2.20×)1.29× (1.01–2.01×)1.15× (0.99–1.32×)1.15× (1.02–1.54×)1.34× (1.03–1.61×)1.24× (0.99–1.36×)

With all six fixed baselines running the same SPLASH engine with online switching disabled under identical arrivals and GPU budgets, SPLASH achieves 1.16–1.69× higher cumulative RPS (1.15–1.71× token throughput) on AgentX and 1.15–1.63× higher full-day throughput on the production trace, beating even the strongest static choices (CP and DOP) by 15–16% directly from online switching.

Thank you for the authors' response. You have demonstrated the contributions of your work. SPLASH's performance improvements may be sufficient to reach the borderline standard of a top-tier conference. However, as Reviewer NLGu pointed out, the presentation remains problematic, even after the revisions, and the paper's novelty and soundness are not strong enough to overlook this issue.

If the rating scale were out of 6, I would consider raising my rating from 2/6 to 3/6. Overall, however, I do not believe the paper meets the standard for acceptance. I therefore maintain my current rating.

Thank you for your reply and for acknowledging that we have "demonstrated the contributions of [our] work" and that SPLASH's performance gains reach the borderline standard of a top-tier conference. Before the discussion period closes, we would appreciate your brief clarification on two points:

  1. Are your original Weaknesses (W1–W2) and Questions (Q1–Q2) fully addressed? In our previous response and the Nov 11 revision, we clarified DOP's technical adaptations beyond Pope et al. and SPLASH's system-level live-switching contribution (W1/Q1; §§1–3, §5.5, §6, Appendices E and H), and added the controlled 24-hour end-to-end trace evaluations isolating the throughput gain of dynamic switching over all six fixed layouts (W2/Q2; §5.6, Figure 6). Could you confirm whether you had a chance to check the end-to-end results in §5.6 / Figure 6, and whether any technical aspect of W1–W2 or Q1–Q2 remains unresolved?

  2. Specific feedback on the revised manuscript Because ICLR supports interactive revisions (open through Nov 18 AoE), we substantially rewrote the Abstract, §1, Figure 1 caption, and the passages flagged by Reviewer NLGu across §§3–5.5 in the Nov 11 PDF. We hope each reviewer can assess the revised manuscript independently. If there are specific sections, sentences, or technical claims in the current revised PDF where presentation or soundness still needs improvement, could you kindly point them out so we can refine them before the revision deadline? Otherwise, if your technical questions and the end-to-end evaluation are satisfactorily addressed, we respectfully ask that you reconsider your rating.

Reviewer_NLGuOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on parallelism in LLM inference. It analyzes that for a given model, different sequence lengths and batch sizes have different optimal parallelism techniques. Moreover, each parallelism technique has a cap on the maximum context length or batch size. Based on that, it proposes:

  • Decoupled Ownership Parallelism (DOP): a parallelism approach that splits along weight tensor dimension as well as data batch size dimension. It is sort of a combination of tensor parallelism (TP) and data parallelism (DP).
  • Splash: a serving system that switches from different parallelism techniques on the fly as context length and batch size increase. It minimizes overhead by overlapping preparation of a layer with computation of its previous layer.

Results show that DOP has a higher cap on sequence length, and performs better than other parallelism techniques on larger sequence lengths, while Splash has a low overhead in switching between parallelisms.

Critical strengths
  • Mathematically formulates different types of parallelisms, and formulates the overhead to switch between them (not sure though if this was covered in previous papers)
  • Detailed results covering different scenarios of parallelisms, batch sizes, and sequence lengths
  • Results show that the proposed solution outperforms all other techniques especially on long context
Critical weaknesses
  • Did not compare overhead of Splash to other parallelism switching techniques in literature
  • The writing is unclear, with lots of AI slop and grammatically incorrect phrases. I don't think this paper should be published in any conference with its style of writing. I strongly recommend that it gets re-writen then re-submitted to another conference. Examples of issues:
    • Line 016 in the Abstract: "ends as a few very long ones," is grammatically incorrect
    • Lines 040 and 041: "must turn a fixed set of GPUs into as many answered requests as it can" is grammatically incorrect
    • Line 051: "What stands in the way is state that cannot be rebuilt cheaply." Sounds like AI slop
    • Line 052: "or receive them once, and be done". Sounds like a blog and not an academic paper
    • Figure 1 caption: "the inset enlarges the first." Not sure what that means
    • Line 077: "which satisfies all three". Three what?
    • Line 427: "The missing state determines the remaining cost." Not sure what the sentence means.
Questions for authors
  • Lines 089 and 090: "Multi-query and grouped-query attention cut the number of KV heads to a handful, and MLA removed them". Is that correct? Does MLA remove KV heads or compress them?
  • By "projections" you mean the Wk, Wv, Wq weights?
  • Line 290: "One layer’s communication costs 0.29–0.30 ms for weights...", which model does this apply to?
  • Table 1: "All twelve directed switches..." What are the 12 switches in the table? I see 6 rows.
3 comments

Thank you for the detailed feedback. We rewrote all flagged passages (Abstract, §1–§5), added a direct Flying Serving benchmark (§5.5, Appendix H), and expanded §1 and §6 on prior and concurrent switching systems (original line numbers below).

Comparison with other switching systems (§1, §5.5, §6, Appendix H). To our knowledge, SPLASH is the first general, composable live-switching system for attention layouts that switches in-flight batches at near-zero cost without halting active requests. Existing and concurrent systems cover narrower settings:

  1. Flying Serving (ICS 2026): Switches DP↔TP on MHA/GQA models via resident weight views and lazy column reordering, but omits CP, DOP, composed DP×TP, and MLA/MoE models (GLM-5.3, DeepSeek-V3.2); drains, pauses, or recomputes active DP requests on merge; and switches across the whole model without per-layer overlap.
  2. Shift Parallelism (ASPLOS 2026): Toggles TP↔SP on a shared head-sharded KV layout, but cannot change KV ownership (DP-attention, DOP, CP, or DP×TP) and frees no replicated MLA cache.
  3. LoongServe (SOSP 2024) and Seesaw (MLSys 2025): Scale sequence parallelism across instances or re-shard weights across buffered phases; neither switches attention weight and KV ownership for an active batch inside one worker group without stalling.
  4. Concurrent 2026 preprints (Moebius, ReMP, PipeLive): Reshard MoE TP/EP, model TP/PP, or pipeline stages, but commit coarsely between steps or at transaction boundaries in tens of milliseconds to seconds (§6). SPLASH instead overlaps layer ii's transfer with layer i−1i-1's compute, handling all 30 switches across TP, CP, DP-attention, DOP, DP2+TP4, and DP4+TP2 in 0.02–11.76 ms median overhead on eight B200 GPUs (< 0.51% of a step, Tables 1 and 7).

Because Flying Serving cannot run MLA/MoE models, we benchmarked its open-source code directly on Llama-3.1-8B-Instruct (BF16) on four H200 GPUs (DP4×TP1 to TP4 at 4K, 100 paired runs; §5.5, Appendix H, Tables 8 and 9):

  • Empty initial KV (Table 8): SPLASH finishes the switch and first TP4 batch in 38.47 ms versus 72.53 ms for Flying Serving (1.88× faster), cutting switch overhead from 34.17 ms to 0.02 ms (using 2.125 GiB/GPU for prepacked weight views).
  • Active DP requests (Table 9): Flying Serving cannot merge active DP KV caches into TP4 (suspending and recomputing them instead), whereas SPLASH migrates four active 4K requests online into the first merged TP4 batch in 1.24 ms (3.07% only because an 8B model has ~1.2 ms compute per layer to hide transfers, vs. < 0.51% on 78-layer GLM-5.3).

Since your review did not name specific papers, we covered both published systems and concurrent preprints; please let us know if you had another work in mind.

Writing and phrasing (Abstract, §1–§5). We replaced informal phrasing throughout the paper with clear technical English:

  • Abstract (L016) and §1 (L040–041, L051–052): Rewrote the Abstract and §1 to state how concurrency and context lengths shift mid-execution, define the fixed-GPU throughput goal, and contrast immutable weights (reused or sent once) with growing KV caches.
  • Figure 1 caption: Clarified the cost-model wording and stated that the inset magnifies the TP vs. DP-attention crossover (18.1k tokens).
  • L077: Listed the three invariants right before "satisfies all three requirements": preserving (i) cached histories, (ii) KV-update order, and (iii) collective ordering across ranks.
  • L427 (§5.5): Rewrote to state that each switch sends only missing weights and KV entries, leaving mainly the first layer's transfer exposed.

Technical questions (Abstract, §1–§3, §4.2, §5.4–§5.5).

  1. MLA heads vs. KV cache (L089–090): Fixed imprecise wording ("removed") in the Abstract, §1, and §3. MQA and GQA reduce KV heads, whereas MLA keeps multiple query heads and compresses keys and values into a shared low-rank latent vector plus a separate RoPE key (§1, §2); without a KV-head axis, TP still replicates that cache across all GPUs.
  2. "Projections" (WK,WV,WQW_K, W_V, W_Q): Yes, for MHA/GQA this refers to WQ,WK,WV,WOW_Q, W_K, W_V, W_O. In our absorbed MLA models, the layout-dependent matrices are Q-B, KV-B, and O (§2, Appendix A Eq. 9), while Q-A and KV-A stay replicated.
  3. Model for 0.29–0.30 ms (L290): GLM-5.3 (78 layers, FP8) on eight B200 GPUs, now stated directly at that sentence in §4.2 (and in §1, §5.4, Figure 5b). It measures one layer's weight all-gather (145.73 MiB/GPU); the paired KV numbers (3.47–3.81 ms/layer) use B=16B=16 at 256K.
  4. Table 1 rows: Table 1 places two Direction column groups side by side, reporting 2 switches per row across 6 rows (6×2=126\times 2 = 12).

I thank the reviewers for taking the time to write a concise rebuttal and to re-write the paper. I have read all the other reviews and their rebuttals. While I highly appreciate the authors re-writing the paper, I believe it is not practical to ask reviewers to re-read the full paper, as some reviewers have upto 4 other papers to read their rebuttals in detail. In fact, it is not practical to ask reviewers to read a submission for the second time. Nevertheless, I did read the Abstract and I do feel it is way clearer and more pleasant to read than before. One concern I have after reading the Review by nuia, is that the proposed DOP is an extension to Pope et al. (2023) from MHA to GQA / MLA. This narrows the contribution slightly, while it seems that the contribution of SPLASH is more significant compared to other papers.

Overall, I am going to increase my score from Clear Rejection to Weak Rejection, but I feel it may be difficult to Accept it. I iterate my appreciation to the authors in re-writing the paper.

Dear Reviewer NLGu,

Thank you very much for raising your score and checking our rewritten Abstract; we completely understand reviewer workload constraints across multiple submissions, and we are glad the new Abstract already read much more clearly (the rest of the paper was manually rewritten to the same standard). Finally, since you acknowledged that you felt SPLASH's contribution is more significant compared to other papers and found the rewritten prose much clearer, and we would be very grateful if you might consider a positive rating. Thank you again for your candid feedback and time!

Reviewer_MYH7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper addresses the problem that agentic and RL-rollouts have workload patterns that change, resulting in a dynamic workload. Existing TP, CP, or DP parallelism strategies are a poor fit for changing workloads, as a change would require you to drain the serving engine, switch the data layouts, before resuming computations. Additionally, since recent attention mechanisms use few or no KV heads, a uniform tensor-parallel strategy across both attention and FFN is not feasible. The authors therefore propose DOP, which is a sharded layout for both weights and KV state, and on-demand realizes the desired parallelization using a collective. The movement is performed one layer ahead of the computation, thus not adding significant overhead to the system. The system is thus able to exploit the most efficient strategy for the current state of the workload.

Critical strengths

The paper proposes an elegant approach where the following two decisions are independent: where do attention weights live and who owns each request’s cache. With this approach, the system can provide a memory allocation scheme for weights/KV that incurs no duplication and can transform from one form of parallelism to another using simple collectives or discards.

Critical weaknesses

The system needs to be evaluated end-to-end in the context of a real changing workload. Evaluation also does not compare against hybrid parallelism -- for example, what happens when you use DP+TP for the attention layer; in this case, the duplication of the weight matrices is reduced by a TP factor, while the attention values (e.g., MLA) might incur some duplication.

Questions for authors

What would be the behavior of the system (throughput and tail latency) on a real changing workload, such as an RL rollout trace or an agentic trace? Does DOP win against hybrid schemes (e.g., DP of 2 with TP of 4, etc.), which cut weight duplication without full cache duplication in the context of real traces? What are empirically measured wins over prior systems that do some form of runtime switching?

3 comments

Thank you for raising end-to-end trace evaluation, hybrid DP×TP baselines, and comparisons with prior runtime-switching systems; we added all three to the revised paper (§3, §5.2, §5.5–§5.6, §6, Appendices D, G, H).

End-to-end performance under changing load (§5.6, Figure 6). We evaluate SPLASH on 24-hour AgentX and production traces against all six fixed layouts (TP, DP-attention, CP, DOP, DP2+TP4, DP4+TP2) running in the same engine with switching disabled under identical arrivals and GPU budgets (24 h cumulative speedup, instantaneous min–max in parentheses):

Trace and metric (24 h window)SPLASH vs. fixed TPSPLASH vs. fixed DPSPLASH vs. fixed CPSPLASH vs. fixed DOPSPLASH vs. fixed DP2+TP4SPLASH vs. fixed DP4+TP2
AgentX request throughput (RPS)1.69× (0.98–2.49×)1.30× (1.01–1.90×)1.16× (1.00–1.29×)1.16× (1.02–1.49×)1.33× (0.98–1.61×)1.25× (1.01–1.35×)
AgentX total token throughput1.71× (1.01–2.51×)1.30× (0.98–1.91×)1.15× (1.03–1.30×)1.15× (1.01–1.51×)1.33× (1.03–1.60×)1.25× (1.02–1.37×)
Production trace daily request rate1.63× (0.99–2.20×)1.29× (1.01–2.01×)1.15× (0.99–1.32×)1.15× (1.02–1.54×)1.34× (1.03–1.61×)1.24× (0.99–1.36×)

Online switching gives SPLASH 1.16–1.69× higher cumulative RPS (1.15–1.71× token throughput) on AgentX and 1.15–1.63× higher full-day throughput on the production trace, while p95 step-switch overhead stays within 0.07–16.79 ms on base switches (Table 1) and 0.89–17.27 ms on hybrid switches (Table 7), under 0.7% of a prefill step.

DOP versus hybrid DP×TP (§3, §5.2, Figure 4, Appendices D and G). Your point on the memory tradeoff is right. Under MLA, a hybrid layout with TP size gg splits TT GPUs into T/gT/g groups: each gg-GPU group shards attention weights gg ways while keeping gg copies of MLA's shared KV cache, holding WA/g+(g/T)BksW_A/g + (g/T)Bks bytes/GPU (WA/4+Bks/2W_A/4 + Bks/2 for DP2+TP4 and WA/2+Bks/4W_A/2 + Bks/4 for DP4+TP2 on 8 GPUs) versus WA/8+Bks/8W_A/8 + Bks/8 for DOP.

Across GLM-5.3 FP8 sweeps on eight B200 GPUs (1K–512K input, B=1B=1–256256; Figure 4, Table 4), both sides appear clearly: smaller TP groups help hybrids at short contexts or small batches (DP2+TP4 leads at 1K–4K, B=16B=16; DP4+TP2 at 64K/B=4B=4 and 128K/B=8B=8), whereas extra KV copies hurt both hybrids at long contexts or large batches, where DOP wins every 256K–512K (B=16,32B=16, 32) and B=128,256B=128, 256 (64K, 128K) point by 3.4–11.0% (e.g., 30,724 tokens/s at 128K/B=256B=256 vs. 24,943 for DP4+TP2, 1.23×, and 18,993 for DP2+TP4, 1.62×). Across all 18 live switches involving both hybrids (Table 7), SPLASH adds just 0.48–9.27 ms median overhead (< 0.40% of a step, 92.93–98.72% below blocking).

Comparison with prior runtime switching (§1, §5.5, §6, Appendix H). To our knowledge, SPLASH is the first general, composable live-switching system for attention layouts that switches in-flight batches at near-zero cost without halting active requests:

  • Flying Serving (ICS 2026): Switches DP↔TP on MHA/GQA models via weight views and lazy column reordering, but omits CP, DOP, composed DP×TP, and MLA/MoE models; suspends or recomputes active DP requests on merge; and activates layouts across the whole model without per-layer overlap.
  • Shift Parallelism (ASPLOS 2026): Toggles TP↔SP on one shared head-sharded KV layout, but cannot change KV ownership (DP-attention, DOP, CP, or DP×TP) and frees no replicated MLA cache.
  • LoongServe (SOSP 2024) and Seesaw (MLSys 2025): Scale sequence parallelism across instances or re-shard weights across buffered phases; neither switches attention weight and KV ownership for an active batch inside one group without stalling.
  • Concurrent 2026 preprints (Moebius, ReMP, PipeLive): Reshard MoE TP/EP, model TP/PP, or pipeline stages coarsely at step or transaction boundaries (tens of ms to seconds) without intra-step per-layer overlap.

We also benchmarked Flying Serving's open-source code directly on a model it supports, Llama-3.1-8B-Instruct on four H200 GPUs (DP4×TP1 to TP4 at 4K, 100 paired runs, Appendix H). Even on an 8B model with little per-layer compute (~1.2 ms vs. 25–70 ms on GLM-5.3) to hide transfers, SPLASH finishes the empty-KV switch and first TP4 batch in 38.47 ms versus 72.53 ms for Flying Serving (1.88× faster; 0.02 ms vs. 34.17 ms switch overhead, Table 8), and migrates four active 4K DP requests online into the first merged TP4 step in 1.24 ms (3.07%, Table 9) instead of suspending and recomputing them.

Dear Reviewer MYH7,

Thank you again for your time and constructive feedback on our submission. As the author–reviewer discussion deadline (Nov 18 AoE) approaches, we wanted to gently check in to see whether our rebuttal, additional experiments (24-hour trace evaluation, hybrid DP×TP baselines, and Flying Serving comparison), and revised manuscript have addressed your concerns and might impact your rating. Your support is much appreciated if the end2end results and revised presentation are satisfactory.

Please let us know if any question remains or if there is anything else we can clarify before the discussion period closes. We would greatly appreciate your feedback!

As the authors have updated the paper to include end-to-end evaluations for serving (as opposed to stitching together the optimal estimates) and have also provided comparisons with hybrid parallelism strategies, I have updated my score.

Comments on the paper

2 comments

Dear Reviewers,

As we are approaching the end of reviewer-author discussions, please go through authors' responses and raise questions that you would like authors to reply.

Thanks,

AC

AuthorsOpenReview ↗

Additional Writing Revision: Full Manual Rewrite of the Manuscript

Dear AC and Reviewers,

Thank you for your questions and constructive feedback during the discussion period. To address the presentation issues raised in the reviews, we built on our previous rebuttal version and manually rewrote the entire manuscript from scratch, while keeping all technical designs, formulations, and experimental results completely unchanged.

At a high level, this writing revision makes three main changes:

  • We reworked the narrative flow across Sections 1–4 to explain the motivation behind each design choice upfront, removing logical jumps and cutting unnecessary digressions into minor details.
  • We streamlined Section 5 and refined the figures, captions, and float layout so the main empirical takeaways read clearly without repetitive prose, while keeping Related Work in the main paper (Section 6) and placing Discussion and Limitations at the start of the appendix (Appendix A).
  • We unified notation and terminology consistently across the main text and Appendices A–K.

Because almost every paragraph in the paper was rewritten, marking changes in color would turn nearly the entire PDF blue and make it harder to read, so we uploaded a clean uncolored PDF.

We hope presentation and readability are much improved in this major writing revision, and we would greatly appreciate your feedback on the updated paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.