- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes StarSD, a distributed speculative decoding framework in which multiple distributed target models share a single draft model. StarSD aims to improve draft model utilization and aggregate throughput by enabling asynchronous drafting and verification across multiple target instances through asynchronous request scheduling and per-target state management. The paper also presents an analytical model that characterizes the trade-off between resource utilization and throughput, and evaluates the proposed framework across different models and speculative decoding backends.
Critical strengths
- The proposed framework is conceptually clear, simple, and straightforward.
Critical weaknesses
-
Unclear problem definition and practical motivation
- The paper does not clearly establish which serving problem it primarily aims to address. In the motivation, the authors emphasize limited serving capacity under memory-constrained deployment settings. However, the proposed framework primarily focuses on improving draft model utilization and aggregate throughput, while the experiments also highlight improvements in SLO-aware goodput under high-concurrency workloads. These are related but distinct objectives, and the paper does not sufficiently explain their relationship or clarify which serves as the primary optimization target.
- Furthermore, the reported improvements appear to be concentrated in high-concurrency settings. It remains unclear to what extent these workloads are representative of the resource-constrained deployment scenarios emphasized in the motivation. A more explicit connection between the motivating deployment constraints, the proposed optimization objectives, and the evaluated workloads would help establish the practical relevance of the framework.
-
Insufficient justification of the benefits of the proposed architecture
-
The paper does not provide a sufficiently convincing justification for dedicating a separate accelerator to a shared draft model. While separating the draft model from the target models can free up memory on the target accelerators (e.g., for larger KV caches), it also consumes an additional accelerator that could otherwise be allocated to target inference. Moreover, StarSD introduces communication and shared-draft queueing overheads without eliminating the sequential dependency between drafting and verification.
-
Notably, Table 1 shows that StarSD achieves lower per-target throughput than co-located EAGLE across all evaluated settings. Although the paper emphasizes improvements in aggregate throughput and draft utilization, these results do not clearly establish whether the proposed architecture improves overall serving efficiency when accounting for the additional hardware resources and associated overheads.
-
This concern also extends to the scaling experiments. Aggregate throughput naturally increases as more target GPUs are added, making it difficult to isolate the benefits attributable to StarSD itself. A more carefully controlled comparison under matched hardware budgets would be necessary to demonstrate the practical advantages of dedicating resources to a shared drafter rather than alternative deployment configurations.
-
-
Limited technical novelty and insufficient differentiation from existing approaches
-
The core mechanisms of StarSD, namely asynchronous request multiplexing, work-conserving scheduling, and per-session state isolation, are well-established principles in distributed systems and inference serving. Similarly, the proposed analytical model appears to be a relatively straightforward formalization of pipeline utilization dynamics. While applying these techniques to speculative decoding is reasonable, the paper does not sufficiently clarify what constitutes the key technical novelty beyond a direct adaptation of existing resource-sharing and scheduling principles.
-
Additionally, the experimental evaluation does not appear to use production-grade inference engines such as vLLM or SGLang. In particular, it is unclear whether common optimizations in modern inference systems, including continuous batching, CUDA graph replay, efficient request scheduling, and optimized KV cache management, are incorporated into the evaluated implementations. These optimizations can substantially affect throughput, accelerator utilization, and scheduling overheads, potentially changing the relative benefits of the proposed architecture.
-
Given that StarSD primarily targets serving throughput and resource utilization, evaluating the framework without clearly accounting for these optimizations raises concerns about the practical significance, scalability, and generalizability of the reported improvements. Overall, the paper does not yet provide sufficient evidence that the proposed framework offers meaningful advantages under realistic, production-level serving conditions.
-
Questions for authors
See weaknesses.