- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies when additional GPUs help LLM serving over slow interconnects. It proposes a decision layer that checks memory feasibility and uses calibrated latency models to choose between single GPU execution, replication, and layer splitting. Two fitted coefficients summarize the fixed cost difference and how latency grows with context length, providing an estimated switching point between modes. Runtime measurements update the estimates. The implementation integrates with llama.cpp and includes a custom staged execution path. Evaluation covers consumer GPUs, several engines, models, context lengths, and concurrency levels. The authors report high agreement with measured mode choices and demonstrate 65K token contexts on two 12 GB GPUs, with automated configuration as the main practical benefit.
Critical strengths
-
The paper addresses a practical problem for consumer GPU users. Slow links and limited memory make execution choices difficult, so automatic mode selection could reduce manual setup and avoid configurations that run out of memory.
-
The evaluation spans several hosts, link speeds, engines, and workloads. These measurements provide useful evidence about when each mode works well, while matched configurations help separate mode choice from differences in engine speed.
Critical weaknesses
-
The presentation reads like an engineering report, with dense inline formulas and many disconnected implementation details. The decision rule, calibration procedures, and fallback rules are not organized into a clear argument. Important corrections and limits are scattered across appendices, making the core contribution and its supporting evidence very difficult to assess.
-
The central explanation of the split cost conflicts with the implementation. The abstract describes remote KV transfers at every decoding step, while Appendix 24 reports no such transfers and attributes the cost mainly to synchronization. This undermines the physical meaning of the fitted coefficient and the claimed relationship between bandwidth and the mode switching point.
Questions for authors
-
What is the core decision rule used at runtime? Please distinguish the roles of the fitted latency model, memory checks, and empirical fallback rules in producing the reported decisions.
-
How do you reconcile the abstract's remote KV transfer explanation with Appendix 24’s finding that KV remains local? What does the fitted cost coefficient actually measure, and what evidence supports its dependence on link bandwidth?
-
Based on the existing measurements, what does the calibrated selector add over a simple rule with the same memory checks? Table S55 shows it matching always splitting. Please identify where the fitted model leads to a better choice.