- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes a system that dynamically adjusts GPU parallelization strategies according to workload characteristics. It integrates several attention parallelization layouts into a unified serving system and introduces a layout designed to reduce redundant storage.
Critical strengths
The system has a unified design and addresses practical resource utilization concerns.
Critical weaknesses
- Adaptive parallelization, reuse of existing states, and reduction of redundant storage are all directions that prior work has already explored. The contribution of this paper appears to lie primarily in implementation and system integration rather than in the discovery of a fundamentally new layout. This could certainly still make for a strong paper, but it makes the following limitation more consequential.
- The paper provides reasonably convincing evidence that different parallelization layouts have different strengths and that switching between them can be inexpensive. However, it does not demonstrate equally clearly how much overall benefit is achieved when the complete system operates under dynamically changing workloads. Given the substantial overlap between its underlying techniques and prior work, this missing system-level validation makes it difficult to assess whether SPLASH delivers a sufficiently significant new contribution.
Questions for authors
- Could the authors clarify the specific technical advances of DOP over the parallelization scheme described by Pope et al. (Efficiently Scaling Transformer Inference)? If the main differences are changes to the cache representation and operator implementation, I would still consider the conceptual novelty of the layout itself limited.
- Could the authors provide a clear experiment demonstrating that SPLASH achieves a significant throughput improvement over the best reasonable fixed-layout deployment, and that this improvement actually comes from dynamic layout switching?
8 comments
Thank you for the authors' response. You have demonstrated the contributions of your work. SPLASH's performance improvements may be sufficient to reach the borderline standard of a top-tier conference. However, as Reviewer NLGu pointed out, the presentation remains problematic, even after the revisions, and the paper's novelty and soundness are not strong enough to overlook this issue.
If the rating scale were out of 6, I would consider raising my rating from 2/6 to 3/6. Overall, however, I do not believe the paper meets the standard for acceptance. I therefore maintain my current rating.
Thank you for your reply and for acknowledging that we have "demonstrated the contributions of [our] work" and that SPLASH's performance gains reach the borderline standard of a top-tier conference. Before the discussion period closes, we would appreciate your brief clarification on two points:
-
Are your original Weaknesses (W1–W2) and Questions (Q1–Q2) fully addressed? In our previous response and the Nov 11 revision, we clarified DOP's technical adaptations beyond Pope et al. and SPLASH's system-level live-switching contribution (W1/Q1; §§1–3, §5.5, §6, Appendices E and H), and added the controlled 24-hour end-to-end trace evaluations isolating the throughput gain of dynamic switching over all six fixed layouts (W2/Q2; §5.6, Figure 6). Could you confirm whether you had a chance to check the end-to-end results in §5.6 / Figure 6, and whether any technical aspect of W1–W2 or Q1–Q2 remains unresolved?
-
Specific feedback on the revised manuscript Because ICLR supports interactive revisions (open through Nov 18 AoE), we substantially rewrote the Abstract, §1, Figure 1 caption, and the passages flagged by Reviewer NLGu across §§3–5.5 in the Nov 11 PDF. We hope each reviewer can assess the revised manuscript independently. If there are specific sections, sentences, or technical claims in the current revised PDF where presentation or soundness still needs improvement, could you kindly point them out so we can refine them before the revision deadline? Otherwise, if your technical questions and the end-to-end evaluation are satisfactorily addressed, we respectfully ask that you reconsider your rating.
Thank you for the clear feedback on how DOP relates to Pope et al. and on isolating the throughput gain from dynamic switching.
Relation to Pope et al., DOP's MLA/hybrid extensions, and SPLASH's system novelty (§1–§3, §6, Appendix E). DOP shares the high-level idea of Pope et al. (2023, §3.3, Fig. 5b): sharding attention projections across ranks while keeping shared KV request-local via activation all-to-all. We removed the imprecise "new layout" phrasing in §1 and discuss Pope et al. directly in §6 and Appendix E.
However, Pope et al. apply batch-sharded attention only to dense single-head MQA. Supporting modern long-context models required two technical adaptations (detailed in §3 and Appendices A–C, E):
These adaptations free 12.69 GiB/GPU for KV on B200 (+27.3% over DP-attention, fitting 16 vs. 12 requests at 512K; +59.7% on H200 with DeepSeek-V3.2, Table 3) and let DOP beat TP, CP, DP-attention, DP2+TP4, and DP4+TP2 by 3.4–11.0% across all 256K–512K and B=128,256 points (Figure 4, Table 4).
More broadly, we agree that our main contribution is at the system level rather than claiming sharded-weight/request-owned-KV as a new parallelization principle. To our knowledge, SPLASH is the first general, composable live-switching system for attention layouts that switches in-flight batches at near-zero cost without halting active requests, unlike Flying Serving (MHA/GQA DP↔TP only, pausing or recomputing active DP requests), Shift Parallelism (TP↔SP on one KV layout), Seesaw (phase stalls), or concurrent preprints Moebius, ReMP, and PipeLive (coarse step/transaction commits in tens of ms to seconds; §6, Appendix H). Aligning DOP's shards with TP and owner pages with DP-attention under pre-captured CUDA graphs makes
TP → DOPandDP-attention → DOPzero-network-byte switches (0.02 ms median), covering all 30 switches across TP, CP, DP-attention, DOP, and DP×TP in 0.02–11.76 ms (< 0.51% of a step, Tables 1 and 7).Throughput gain from dynamic switching (§5.2, §5.6, Figures 4 and 6). As a 64K batch drains from 256 requests to 1 on B200 (Figure 4c, Table 4), five layouts take turns leading (
DOP → CP → DP-attention → DP4+TP2 → TP), and any fixed choice loses 35–57% outside its best regime. Replaying 24-hour AgentX and production traces (§5.6, Figure 6; 24 h cumulative speedup, with instantaneous min–max in parentheses):With all six fixed baselines running the same SPLASH engine with online switching disabled under identical arrivals and GPU budgets, SPLASH achieves 1.16–1.69× higher cumulative RPS (1.15–1.71× token throughput) on AgentX and 1.15–1.63× higher full-day throughput on the production trace, beating even the strongest static choices (CP and DOP) by 15–16% directly from online switching.