acceptodds
Under review as a conference paper at ICLR 2027

StarSD: One-for-Many Speculative Decoding

Abstract

Speculative decoding accelerates autoregressive generation by introducing a draft-then-verify procedure, but most existing approaches are designed for single-node execution and do not scale well to multi-accelerator clusters used for serving modern Large Language Models (LLMs). We present StarSD, a one-for-many speculative decoding framework that uses a single draft model to serve multiple target models across distributed nodes via a star topology. StarSD decouples drafting and verification, enabling effective sharing of draft computation, and preventing distributed accelerators from remaining idle under bursty workloads. We provide a system-level analysis that characterizes when and why a single draft model can remain fully utilized by multiple verifiers, yielding predictable latency and utilization gains. Extensive experiments in real-world distributed inference settings demonstrate that StarSD simplifies deployment and supports flexible resource allocation across heterogeneous accelerators, while maintaining output quality. These results indicate that StarSD is a practical and scalable framework for bringing speculative decoding to memory-constrained multi-session serving.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Speculative Decoding, LLM Infra

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qTkKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes StarSD, a distributed speculative decoding framework in which multiple distributed target models share a single draft model. StarSD aims to improve draft model utilization and aggregate throughput by enabling asynchronous drafting and verification across multiple target instances through asynchronous request scheduling and per-target state management. The paper also presents an analytical model that characterizes the trade-off between resource utilization and throughput, and evaluates the proposed framework across different models and speculative decoding backends.

Critical strengths
  • The proposed framework is conceptually clear, simple, and straightforward.
Critical weaknesses
  • Unclear problem definition and practical motivation

    • The paper does not clearly establish which serving problem it primarily aims to address. In the motivation, the authors emphasize limited serving capacity under memory-constrained deployment settings. However, the proposed framework primarily focuses on improving draft model utilization and aggregate throughput, while the experiments also highlight improvements in SLO-aware goodput under high-concurrency workloads. These are related but distinct objectives, and the paper does not sufficiently explain their relationship or clarify which serves as the primary optimization target.
    • Furthermore, the reported improvements appear to be concentrated in high-concurrency settings. It remains unclear to what extent these workloads are representative of the resource-constrained deployment scenarios emphasized in the motivation. A more explicit connection between the motivating deployment constraints, the proposed optimization objectives, and the evaluated workloads would help establish the practical relevance of the framework.
  • Insufficient justification of the benefits of the proposed architecture

    • The paper does not provide a sufficiently convincing justification for dedicating a separate accelerator to a shared draft model. While separating the draft model from the target models can free up memory on the target accelerators (e.g., for larger KV caches), it also consumes an additional accelerator that could otherwise be allocated to target inference. Moreover, StarSD introduces communication and shared-draft queueing overheads without eliminating the sequential dependency between drafting and verification.

    • Notably, Table 1 shows that StarSD achieves lower per-target throughput than co-located EAGLE across all evaluated settings. Although the paper emphasizes improvements in aggregate throughput and draft utilization, these results do not clearly establish whether the proposed architecture improves overall serving efficiency when accounting for the additional hardware resources and associated overheads.

    • This concern also extends to the scaling experiments. Aggregate throughput naturally increases as more target GPUs are added, making it difficult to isolate the benefits attributable to StarSD itself. A more carefully controlled comparison under matched hardware budgets would be necessary to demonstrate the practical advantages of dedicating resources to a shared drafter rather than alternative deployment configurations.

  • Limited technical novelty and insufficient differentiation from existing approaches

    • The core mechanisms of StarSD, namely asynchronous request multiplexing, work-conserving scheduling, and per-session state isolation, are well-established principles in distributed systems and inference serving. Similarly, the proposed analytical model appears to be a relatively straightforward formalization of pipeline utilization dynamics. While applying these techniques to speculative decoding is reasonable, the paper does not sufficiently clarify what constitutes the key technical novelty beyond a direct adaptation of existing resource-sharing and scheduling principles.

    • Additionally, the experimental evaluation does not appear to use production-grade inference engines such as vLLM or SGLang. In particular, it is unclear whether common optimizations in modern inference systems, including continuous batching, CUDA graph replay, efficient request scheduling, and optimized KV cache management, are incorporated into the evaluated implementations. These optimizations can substantially affect throughput, accelerator utilization, and scheduling overheads, potentially changing the relative benefits of the proposed architecture.

    • Given that StarSD primarily targets serving throughput and resource utilization, evaluating the framework without clearly accounting for these optimizations raises concerns about the practical significance, scalability, and generalizability of the reported improvements. Overall, the paper does not yet provide sufficient evidence that the proposed framework offers meaningful advantages under realistic, production-level serving conditions.

Questions for authors

See weaknesses.

Reviewer_VTbEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StarSD lets several target instances share one draft model. While a target verifies, the drafter can serve other requests instead of waiting. The paper studies the gains and queueing costs of sharing and compares serving performance under different loads.

Critical strengths
  1. The experiments compare systems using the same number of GPUs and report both throughput and request latency. This helps show whether sharing a drafter is worthwhile.

  2. The paper also shows the limits of sharing. More targets reduce draft idle time, but queueing eventually becomes a bottleneck.

Critical weaknesses
  1. I am not sure the systems handle multiple requests in the same way. Request assignment and whether targets can interleave requests could affect the results.

  2. It is still unclear when dedicating a GPU to drafting is worthwhile. The main results do not clearly separate the benefits of memory savings, faster drafting, and request scheduling.

  3. Some throughput numbers use different measurement methods, and the verification capacity in the scaling analysis needs checking. It is unclear whether these differences affect the conclusions.

Questions for authors
  1. When requests outnumber target instances, how does each system assign and serve them? Are any differences required by the design or choices in the implementation?

  2. How much do memory savings and faster drafting each help in the main experiment? When is a dedicated draft GPU worthwhile?

  3. Why do the two single-target throughput numbers differ? Should the scaling analysis use the total verification capacity of all targets, and which conclusions would this affect?

Reviewer_qDcqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes StarSD, an approach that distributes the draft from a single drafter across many other workers (GPUs). The core idea is to design a scheduler that supports this one-for-many concept and delivers real speedup. The paper addresses some practical challenges from disaggregating the drafter and verifier, such as drafter idling due to burstiness. The strategies to mitigate these challenges are adequate, and as a result, StarSD shows promising results as the number of sharing instances increases, though the experimental setups are quite limited.

Critical strengths

[S1] Under the authors' assumption regarding the scenario, the one-for-many concept could be plausible.

[S2] The strategies to address the practical challenges of the one-for-many concept make sense.

Critical weaknesses

[W1] The motivation for the target scenario is quite limited.

[W2] The experimental setup is quite limited, shown on small models and with a limited number/types of GPUs.

[W3] The paper's writing can be improved further.

For details on the weaknesses, I explained them in the questions.

Questions for authors

Thank the authors for submitting the paper to ICLR. I liked the concept of one-for-many drafting; however, I had critical concerns as follows, mainly regarding the motivation for utilizing the one-for-many concept.

[Q1] What is the main deployment scenario of the one-for-many concept?

  • In the introduction, the paper mentions the KV cache issue when one-for-one deployment is adopted because the drafter and target model live together on GPUs. The authors further emphasize long-context cases. However, the reviewer did not fully agree with this scenario. Recent models are large and often deployed across multiple GPUs. Compared to model size, the drafter KV cache is very limited; especially when deployed across multiple GPUs, the drafter KV cache share is small. For example, for GLM5.3, the drafter size is only a few billion, and when the context length is long, the drafter KV cache share is small.
  • In the introduction, the authors also talk about the resource-constrained scenario. One of the goals of edge inference is to address privacy issues. However, using one-for-many drafting breaks that purpose. Also, the authors discuss a resource-constrained scenario, but one-for-many requires a separate GPU for the drafter. Of course, the paper can target an intermediate between severely edge and non-edge, but then the target scenario is too small to meet the ICLR bar.

[Q2] Can the authors deploy and benchmark the one-for-many concept on typically deployed environments?

  • The current setup is too complex and very limited to a few consumer-grade GPUs with very small and outdated models for both the target and drafter.
  • Here's the reviewer's suggestion: can the authors benchmark on A100 (or more recent H100/H200/B200) nodes with 8 GPUs, setting one GPU as the drafter GPU and the others as draft consumers (target deployers)? The reviewer believes this is a practical deployment hardware environment.
  • Also, please use recent models, not outdated models like EAGLE-2/DFlash. Please use EAGLE-3, a recent model. Also, for the target model, please test MoE models and larger models.
  • I enjoyed the one-for-many concept, but I cannot fully agree with its usability before it is adequately validated on a typical deployment setup.

[Q3] Can the authors improve the writing?

  • Despite the reviewer currently working on this domain with deployment, the writing was a bit hard to follow. The reviewer thinks this is partly because I cannot fully agree with the scenario's practicality, but I believe the authors can significantly improve the flow.
  • For instance, putting an overview design (framework) structure figure will help readers understand the concept of StarSD easily.
  • Also, while addressing Q2/3, the reviewer believes that the authors can improve the introduction significantly to make the target scenario very convincing.

Since I believe the concept itself is valid if the above questions are addressed, I'll raise my score if they are fully addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.