acceptodds
Under review as a conference paper at ICLR 2027

NebulaSD: Many-for-Many Speculative Decoding

Abstract

Speculative decoding accelerates Large Language Model (LLM) inference by using a lightweight draft model to propose candidate tokens for parallel verification by a target model. Drafting and verification, however, exhibit different service characteristics and favor different batch configurations, making fixed draft-target coupling inefficient under concurrent workloads. Existing distributed designs can physically separate the two stages, but often retain request or batch affinities that prevent their capacities from being shared globally. We present NebulaSD, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools. Such dynamic reassignment removes fixed worker locality, requiring request states to be made available at newly selected workers without introducing migration stalls. NebulaSD addresses this challenge through worker-triggered batch reconstruction and asynchronous KV-state preparation overlapped with model execution. We evaluate NebulaSD from both system and scaling perspectives, showing that dynamic pooling improves request-round processing rate by over a physically disaggregated baseline and over co-located execution on a four-GPU deployment while substantially increasing effective GPU utilization. Profile-driven simulations further show approximately proportional compute-side capacity scaling under idealized state movement.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, LLM Infra

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xRUyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NebulaSD, a speculative decoding system that organizes draft and target workers into independent resource pools. It removes fixed worker assignments and cross-stage batch affinity, allowing each stage to form batches independently. A centralized scheduler plans upcoming execution, while a persistent host-side KV store and overlapped state preparation support request migration. Experiments on four RTX 4090 GPUs show higher speculative-round processing rates than co-located and fixed-affinity distributed baselines, complemented by larger-scale simulation.

Critical strengths

• The paper addresses a practical inefficiency: draft and target models have different batching characteristics, and fixed assignments can leave resources idle.

• The prototype includes concrete mechanisms for KV management and overlapping state preparation with computation.

• The appendix provides useful sensitivity studies on KV growth, request concurrency, proposal depth, and draft-model size.

Critical weaknesses
  1. Scheduling and coordination between drafting and verification have been studied in both co-located and disaggregated speculative decoding systems. In particular, StarSD already shares draft computation across multiple target instances through asynchronous execution. NebulaSD extends resource sharing to both stages, but its main mechanisms are resource pooling, batch reconstruction, and overlapped KV preparation. The current manuscript does not make a convincing case that this extension introduces a sufficiently new scheduling insight or system design beyond existing approaches.
  2. The benefit for long-context workloads is uncertain. Preparing KV states on different workers introduces transfer costs that grow with context length. Appendix G.1 already shows the advantage over co-located execution shrinking to 7.1% when the maximum generation length reaches 512 tokens. With larger KV states, retaining requests on the same GPUs may become preferable, substantially limiting the opportunities for dynamic pooling.
  3. The main performance metric, i.e., rounds per second, does not measure the serving performance completely. Requests require different numbers of rounds depending on their output lengths and the number of tokens accepted per round. A higher round-processing rate therefore does not directly establish higher request throughput or useful output-token throughput.
  4. I am also concerned about the evaluation. The real deployment uses only four GPUs, while the larger-scale simulations assume zero communication and control-plane latency. These assumptions exclude important costs of the proposed migration and scheduling mechanisms. The comparisons are also limited to basic co-located and fixed-affinity distributed deployments, without a direct comparison against recent optimized systems such as StarSD.
  5. In addition, the evaluation uses fixed proposal depths, including the additional depth settings in the appendix. Dynamic draft trees and adaptive speculation lengths can change the balance between drafting and verification, as well as the predictability of batch execution times. Since these factors directly affect NebulaSD’s scheduling decisions, the reported gains may not carry over unchanged when such optimizations are enabled.
Questions for authors
  1. What prompt dataset and input/output length distributions are used in the real-system experiments?
  2. How many output tokens are committed per verification round on average, and does this differ across the compared systems?
  3. How frequently do requests change workers? When a request stays on the same GPU, can its resident KV state be reused without importing it again from HostKV?
  4. What is the measured overhead of online batch reconstruction, and how does it change as the shared request pool grows?
Reviewer_UfkPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NebulaSD places draft and target models in separate worker pools and shares requests across them. Each stage can form its own batches, reducing waiting and idle time.

Critical strengths
  1. The paper addresses a practical problem. Drafting and verification favor different batch sizes, and the experiments show that sharing resources reduces idle time caused by fixed pairing.

  2. The paper also shows its limits. Gains shrink with longer outputs or fewer requests, helping explain where the method is useful.

Critical weaknesses
  1. I would like to see output tokens per second and request latency, as well as a comparison with AR. Rounds/s alone makes the practical gain hard to judge, and the workload needs a clearer description.

  2. Why does drafting take so long, and why does a larger batch add so little time? A faster drafter might leave less room for pooling to help.

  3. The systems keep different numbers of requests in memory, so the comparison may not be fair. The simulations also skip communication and scheduling costs, so it is hard to tell how well the system would scale in practice.

Questions for authors
  1. I would like to see output tokens per second and request latency, as well as a comparison with AR. Rounds/s alone makes the practical gain hard to judge, and the workload needs a clearer description.

  2. Why does drafting take so long, and why does a larger batch add so little time? A faster drafter might leave less room for pooling to help.

  3. The systems keep different numbers of requests in memory, so the comparison may not be fair. The simulations also skip communication and scheduling costs, so it is hard to tell how well the system would scale in practice.

1 comment

Thank you for the additional experiments and discussion; most of my concerns have been addressed, so I will maintain my positive score.

Reviewer_kAkXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a disaggregated service for LLM serving with a draft model and a target model. Its dynamic pooling improves request-round processing rate by 50.4% over a physically disaggregated baseline and 72.6% over co-located execution on a four-GPU deployment, while substantially increasing effective GPU utilization.

Critical strengths
  1. The system is carefully designed, and the implementation is solid.
  2. Interesting mathematical analysis.
  3. Propose a promising way to disaggregate draft and target models.
Critical weaknesses
  1. Too limited an evaluation scale; the largest experiment is only 8B on 4 GPUs. Most of the results come from simulation.
  2. Does not compare or show compatibility with more advanced speculative decoding algorithms, like DFlash, DSpark, etc.
Questions for authors
  1. Is it possible to show a larger-scale experiment? (like a> 30 B target model)
  2. Is the proposed method compatible with DFlash/DSpark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.