- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents NebulaSD, a speculative decoding system that organizes draft and target workers into independent resource pools. It removes fixed worker assignments and cross-stage batch affinity, allowing each stage to form batches independently. A centralized scheduler plans upcoming execution, while a persistent host-side KV store and overlapped state preparation support request migration. Experiments on four RTX 4090 GPUs show higher speculative-round processing rates than co-located and fixed-affinity distributed baselines, complemented by larger-scale simulation.
Critical strengths
• The paper addresses a practical inefficiency: draft and target models have different batching characteristics, and fixed assignments can leave resources idle.
• The prototype includes concrete mechanisms for KV management and overlapping state preparation with computation.
• The appendix provides useful sensitivity studies on KV growth, request concurrency, proposal depth, and draft-model size.
Critical weaknesses
- Scheduling and coordination between drafting and verification have been studied in both co-located and disaggregated speculative decoding systems. In particular, StarSD already shares draft computation across multiple target instances through asynchronous execution. NebulaSD extends resource sharing to both stages, but its main mechanisms are resource pooling, batch reconstruction, and overlapped KV preparation. The current manuscript does not make a convincing case that this extension introduces a sufficiently new scheduling insight or system design beyond existing approaches.
- The benefit for long-context workloads is uncertain. Preparing KV states on different workers introduces transfer costs that grow with context length. Appendix G.1 already shows the advantage over co-located execution shrinking to 7.1% when the maximum generation length reaches 512 tokens. With larger KV states, retaining requests on the same GPUs may become preferable, substantially limiting the opportunities for dynamic pooling.
- The main performance metric, i.e., rounds per second, does not measure the serving performance completely. Requests require different numbers of rounds depending on their output lengths and the number of tokens accepted per round. A higher round-processing rate therefore does not directly establish higher request throughput or useful output-token throughput.
- I am also concerned about the evaluation. The real deployment uses only four GPUs, while the larger-scale simulations assume zero communication and control-plane latency. These assumptions exclude important costs of the proposed migration and scheduling mechanisms. The comparisons are also limited to basic co-located and fixed-affinity distributed deployments, without a direct comparison against recent optimized systems such as StarSD.
- In addition, the evaluation uses fixed proposal depths, including the additional depth settings in the appendix. Dynamic draft trees and adaptive speculation lengths can change the balance between drafting and verification, as well as the predictability of batch execution times. Since these factors directly affect NebulaSD’s scheduling decisions, the reported gains may not carry over unchanged when such optimizations are enabled.
Questions for authors
- What prompt dataset and input/output length distributions are used in the real-system experiments?
- How many output tokens are committed per verification round on average, and does this differ across the compared systems?
- How frequently do requests change workers? When a request stays on the same GPU, can its resident KV state be reused without importing it again from HostKV?
- What is the measured overhead of online batch reconstruction, and how does it change as the shared request pool grows?
Thank you for the additional experiments and discussion; most of my concerns have been addressed, so I will maintain my positive score.