acceptodds
Under review as a conference paper at ICLR 2027

SCD: Enabling Speculative Overlap for Target-Conditioned Draft Models

Abstract

A stronger drafter can improve acceptance in speculative decoding, but its extra computation may offset the gain. Target–draft disaggregation places the models on separate devices, yet each request still waits for verification before the next draft. Speculative speculative decoding (SSD) predicts the next starting point to overlap these steps, but target-conditioned drafters must also wait for target hidden states. We propose Staged Conditional Drafting (SCD) to shorten this wait. The first draft layers use earlier target features to compute predicted next-round branches during verification. The remaining layers reuse these intermediate activations and continue with deeper target features. For batched requests, a small drafter handles prediction misses in parallel. We implement SCD with Two-stage DSpark and KDA + Target-KV. Both preserve accepted length on a 4B target; the KDA version also does so on 14B. With the same models, batch, and two-GPU layout, using target features earlier reduces total decode time by 15.47%. Under greedy decoding in native SGLang, two-GPU overlap improves batch-size-one decode rate by 6.86–9.32% over single-GPU serial execution across two draft architectures and three tasks. It also improves throughput in fully occupied batched rounds by 8.50–17.99% over single-GPU DSpark. KDA + Target-KV accommodates 13% more concurrent sessions within the same GPU memory budget, at a 2.17% lower single-GPU decode rate.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, speculative speculative decoding, draft-target disaggregation, blockwise draft models, inference acceleration, model-system co-design

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_W2vUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Staged Conditional Drafting (SCD), a framework for overlapping target verification with next-round drafting in speculative decoding. Building on existing cross-round speculation methods, SCD splits a target-conditioned drafter into stages: earlier layers use intermediate target hidden states, while later layers incorporate deeper target features after they become available. The method additionally introduces joint prediction of the next accepted boundary and anchor token, a fallback drafter for prediction misses, and a KDA + Target-KV variant for reducing draft-history memory. Experiments demonstrate reduced decoding latency and competitive acceptance lengths across several tasks.

Critical strengths
  1. Practical staged drafting: The proposed approach allows target-conditioned draft models to begin computation before verification completes. The controlled experiment showing a 15.47% reduction in decode time with the same models and two-GPU configuration provides useful evidence of its effectiveness.

  2. Detailed systems evaluation: The paper evaluates acceptance length, prediction accuracy, decoding throughput, and memory usage. The KDA + Target-KV variant also demonstrates useful memory savings for longer-context inference.

Critical weaknesses
  1. Limited novelty and missing comparisons with prior cross-round speculation: The main idea of overlapping next-round drafting with target verification has already been explored in prior work, including Mirror-SD and SSD. SCD's primary distinguishing contribution is staged execution of target-conditioned draft layers using earlier and later target hidden states. While this is a useful architectural extension, the paper does not include direct empirical comparisons with these prior cross-round speculation approaches, making it difficult to assess the incremental benefit of the proposed mechanism.

  2. Additional training complexity and limited scalability evidence: SCD modifies the drafter architecture and requires training the staged drafter, joint predictor, and fallback model. Although the authors demonstrate largely preserved acceptance in their candidate-quality experiments, the deployed system shows some accepted-length degradation, particularly on Arena. The incremental training and compute costs are not quantified. Furthermore, decoding speedups are only measured with 4B targets, leaving the performance benefits on larger models unclear.

Questions for authors
  1. How does SCD compare with existing cross-round speculative decoding methods such as Mirror-SD and SSD under comparable hardware and drafting budgets? In particular, what incremental benefit comes specifically from staged target conditioning?

  2. Could the authors quantify the additional training cost, including GPU-hours or compute requirements for the staged drafter, joint predictor, and fallback model, relative to the original DSpark setup?

  3. The proposed staged architecture changes the conditioning inputs used by the drafter and introduces a smaller fallback model. Although candidate-quality experiments show largely preserved acceptance, how does this trade-off behave on harder workloads or larger target models?

  4. The paper reports accepted-length results for the 14B target, but I could not find corresponding decoding latency or throughput measurements. If these results are already included, could the authors point me to the relevant section? Otherwise, it would be helpful to understand whether the proposed overlap benefits extend to larger models beyond the 4B configuration.

1 comment

Thank you for recognizing the practical value of staged drafting and the controlled scheduling evidence. We have clarified the distinction from prior cross-round speculation and added direct SSD comparisons, training-cost information, and end-to-end 14B results.

1. Novelty and comparison with prior cross-round speculation

SCD’s contribution is not cross-round overlap itself, but enabling strong target-conditioned drafters to overlap while consuming fresh, current-round target representations.

SSD advances drafting before verification finishes, when the latest target features are unavailable. Mirror-SD uses early-exit token predictions to guide speculative expansion. SCD instead feeds real target hidden states into successive stages of the same draft forward, while joint prediction limits early work to a small set of likely starting points.

The revision adds:

  • A direct comparison with the official SSD implementation: 2.35×/1.73× throughput on MATH500/Arena under the same two-GPU budget (Table 6).
  • A fixed-model, fixed-resource scheduling comparison: 15.47% shorter decoding completion time, with identical outputs and acceptance trajectories (Table 2).
  • A fixed-drafter predictor comparison: 8.18% higher full-slot throughput than SSD-style selection (Table 3).

These distinguish complete-system performance from the scheduling and prediction contributions.

2. Additional training cost

Main and Small can be trained together using the same generated target hidden states, without duplicating target feature generation—the dominant training cost.

For the 14B system, Main and Small were trained together for approximately 60 hours on six A3 nodes. Main retains DSpark’s draft-training cost, while Small adds approximately half of Main’s draft-training computation.

The joint predictor has approximately 1/165 of Main’s trainable parameters. Predictor training took approximately one hour on eight A100 GPUs, or 8 GPU-hours.

Small supports efficient batched recovery but is not required for overlap: the single-request implementation reruns Main on misses and still achieves 7.27–8.53% decode-rate improvements without Small (Table 25).

3. Acceptance on difficult workloads

We now distinguish staged Main’s candidate quality from the deployed Main/Small mixture. Staged Main achieves acceptance comparable to original DSpark on both 4B and 14B (Table 1). Arena’s larger deployed acceptance loss is associated with its higher fallback rate.

The same-prefix experiment explains this behavior: Main itself accepts fewer tokens on missed starting points, showing that misses select difficult prefixes. On those prefixes, Small trails Main by only 0.29 token on average (Table 13). Lightweight recovery therefore substitutes for Main where Main’s own useful-token yield is already lower.

Small runs alongside Main’s late stages, so a higher fallback rate primarily affects acceptance rather than necessarily introducing additional waiting.

4. End-to-end performance on 14B

The revised Table 4 reports Qwen3-14B TPOT and throughput across all three datasets at batch sizes 16 and 32. SCD improves throughput by 5.50–14.11%, while staged Main preserves acceptance comparable to the original 14B DSpark.

The smaller speedups relative to 4B also reflect our chosen configuration: the 14B setup leaves five target layers after early-feature readiness, compared with six at 4B. SCD does not require this shorter window; feature depths and stage boundaries can be selected to balance candidate quality and overlap across deployments.

Reviewer_KvKVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the DSpark approach to a draft-verification disaggregation (DVD) setting, where the draft and target models run on separate GPUs. A predictor estimates the number of draft tokens expected to be accepted, enabling partial overlap between drafting and verification to reduce inference latency. The reported results show 6.86–9.32% speedups at batch size 1 and 8.50–17.99% improvements over DSpark at batch sizes 16 and 32.

Critical strengths
  1. Staged conditioning preserves draft quality while enabling overlap. The design enables overlapping drafting and verification without relying solely on an independently speculative drafter.

  2. The method accounts for prediction misses. When the prediction is incorrect, the main drafter can resume drafting after receiving the verification results, providing a recovery mechanism rather than assuming perfect predictions.

Critical weaknesses
  1. Insufficient comparison with existing DVD methods. The evaluation primarily compares the proposed method against DSpark, without sufficient comparisons against existing approaches designed for draft-verification disaggregation. This makes it difficult to assess the method's incremental contribution over prior DVD systems and determine whether the reported gains arise from the proposed prediction mechanism or other system design choices.

  2. Limited evaluation of scalability and realistic serving scenarios. The end-to-end evaluation primarily focuses on Qwen3-4B, while the Qwen3-14B results appear limited to the draft model. This leaves the scalability of the complete system insufficiently validated across model sizes and serving configurations.

  3. Insufficient characterization of predictor overhead and overlap efficiency. Figure 3 reports a predictor execution time of 0.770 ms, while the target GPU appears to exhibit idle gaps around the prediction stage. Although Appendix C.2 provides a careful analysis of prediction misses and their relationship with accepted draft length, the system-level impact of the predictor remains unclear. In particular, it would be useful to clarify whether the observed idle gaps result from synchronization or scheduling dependencies and quantify how much of the predictor's execution cost is hidden by overlap versus remaining on the critical path. This would help establish the net benefit of the proposed overlap strategy.

  4. The performance contribution of KDA and Target-KV remains unclear. Although these techniques are incorporated into the system, their reported benefits appear primarily related to memory reduction rather than inference acceleration. The paper should better clarify their role in the overall design and distinguish their memory-efficiency benefits from the speedup attributable to the proposed overlap strategy.

Questions for authors

Q1. How does the proposed method compare with existing DVD approaches under comparable hardware, model configurations, and serving workloads?

Q2. Figure 3 shows that the predictor takes 0.770 ms, while the target GPU appears to have idle gaps around the prediction stage. Could the authors clarify whether these gaps are caused by synchronization or scheduling dependencies introduced by the predictor, and quantify their impact on end-to-end latency? In particular, how much of the predictor's execution cost is hidden by computation overlap, and how much remains on the critical path?

Q3. What are the respective contributions of the overlap strategy, KDA, and Target-KV to latency, throughput, and memory consumption?

3 comments

Thank you for recognizing that staged conditioning preserves draft quality while enabling overlap. We have clarified the contribution beyond disaggregation and added direct baselines, controlled prediction experiments, and 14B system results.

1. Contribution beyond draft–verification disaggregation

Conventional disaggregation places target and draft on separate devices but leaves their data dependencies unchanged: within each request, drafting still follows verification serially.

SCD breaks this serialization through joint starting-point prediction and staged consumption of current-round target features. It therefore shortens the decoding process itself and reduces end-to-end generation latency, rather than merely separating resource placement.

The revised experiments distinguish three contributions:

  • With models, devices, branch computation, and outputs fixed, overlapped SCD completes the same workload 15.47% sooner than delayed SCD (Table 2).
  • With drafters and schedule fixed, joint prediction improves starting-point coverage from 20.99% to 89.88% and full-slot throughput by 8.18% over SSD-style selection (Table 3).
  • Against the official SSD implementation under the same two-GPU budget, SCD achieves 2.35×/1.73× throughput on MATH500/Arena (Table 6).

The first two experiments isolate scheduling and prediction; the last measures complete-system performance.

2. Predictor overhead and apparent idle gaps

The apparent gaps in Figure 3 contain feature preparation, candidate selection, and branch scheduling; they should not be interpreted as the target simply waiting for the predictor.

The 0.770 ms predictor interval includes input packing and finishes before verification completes. Preparation, prediction, and early drafting use the remaining verification window, leaving 0.30 ms of total early-path spill. Main continuation and Small recovery then execute in parallel.

All these costs are included in the controlled experiment showing a 15.47% reduction in decoding completion time. This measures the complete early-release schedule, which advances preparation, prediction, and early drafting together.

3. Larger targets and shared serving

The revision adds Qwen3-14B TPOT and throughput results across three datasets and two batch sizes, with improvements of 5.50–14.11% (Table 4).

The smaller gains reflect, in part, the chosen staging configuration: 14B leaves five target layers after early-feature readiness, versus six at 4B. Feature depths and stage boundaries are configurable; the shorter window is not an inherent requirement at larger scales.

We also demonstrate two targets sharing one Main: three GPUs retain 97.1% of the aggregate throughput of four-GPU dedicated deployment (Table 7).

4. Separate roles of overlap, KDA, and Target-KV

The core SCD speed results use DSpark and do not require KDA or Target-KV. Staged conditioning addresses execution latency; KDA + Target-KV is a separate history-capacity extension, now presented in Appendix E.

At 16K context, that extension reduces draft-owned history by 99.22% and increases resumable session capacity by 13.64%, with a 2.17% reduction in single-GPU decode rate. Its demonstrated benefit is memory capacity, not standalone acceleration. The revision separates these results from the speedup attributable to overlap.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Thank the authors for the thoughtful rebuttal. Your responses address my concerns and provide meaningful additional evidence that strengthens the paper. I will raise my rating to Weak Accept.

Reviewer_RBDROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCD proposes a novel speculative decoding method for target-conditioned draft models that can execute asynchronously with the target model. In contrast to existing target-conditioned draft models that generally must run synchronously with the target model (since they rely both on complete target hidden states as well as the verification outcome), SCD uses a small predictor to estimate verification outcomes from earlier target hidden states so that the drafter can proceed on these estimated outcomes directly without waiting for the target model to finish. The drafter is split into two stages, the first of which overlaps with the target model and the second of which is a small component that runs synchronously with the target. The first stage can execute on a small number of likely verification outcomes at once (under a fixed drafting budget) while the second stage can finalize the draft on just the true verification outcome’s branch. If the true verification outcome was not included in the drafting budget, then a small drafter fallback is used to produce the next draft from scratch. SCD achieves competitive draft acceptance length with DSpark (synchronous SD), allowing it to realize some practical efficiency gains.

Critical strengths

The work addresses an interesting problem of enabling target-conditioning in asynchronous SD, which can make asynchronous drafters significantly more capable and viable. Using different target layer conditioning for different drafter stages is clever. The problem and method are very well-defined and articulated. I appreciate the figures that make it clear how the execution is scheduled.

Critical weaknesses
  • The argument that target-conditioning is needed to improve async SD is only stated in prose, and no baseline for non-target-conditioned async SD (e.g., SSD) is provided in the experiments. If non-target-conditioned async SD enables substantially more overlap so that the drafting budget can be scaled up significantly, this would limit the effectiveness of SCD since a higher drafting budget could anyway compensate for lower acceptance length in regular async SD. Could these baselines be provided?
  • No results are provided for a no-speculative-decoding baseline under the same hardware budget, like TP-2. Also, always placing the target model on one GPU does not always match realistic deployments. Can additional ablations be provided over TP-2, TP-4, etc. for the target model to evaluate the gain under these settings, including a comparison against DSpark also deployed at TP-2/4?
  • The choice of late-stage features is not properly justified: if Main’s late stage anyway runs after verification completes so that missed branches can be dropped, why is the late stage conditioned on layers only up to 33 (4B) / 37 (14B) instead of up to the last layer’s hidden states, which would already be available and would arguably contain the richest representation to improve acceptance length?
  • One primary benefit of async SD is that the draft budget and/or model can be substantially scaled up to improve acceptance length without affecting the critical path. However, this does not seem to carry over to SCD, since the experiments only demonstrate fairly modest overlap between the drafter and target of only ~10-20% of the target layers, and Section 3.4 already specifies how Top-2 can outperform Top-4. Without enabling more overlap, it does not seem plausible that the drafting budget or the draft model itself can be substantially scaled up in practice. Could SCD’s layer selection be more carefully justified in this context, including with the additional experiments from Q1?
  • Along with the missing baselines, the evaluation appears incomplete. The most complete table (Table 1) only shows acceptance length, which does not necessarily correlate with practical efficiency gains. Could you provide complete latency / throughput results in all settings to fully evaluate the design?
  • The ablations on different drafter designs is appreciated, but they are conducted in an ad hoc manner that makes it difficult to gain any insight. Could you conduct the ablations more systematically, and evaluate the same set of designs for both 4B and 14B?
  • Could an ablation be conducted without the fallback during batched settings to more clearly isolate the gain of the Main drafter?
Questions for authors

N/A

3 comments

Thank you for the detailed feedback. We have added asynchronous baselines, budget analyses, controlled predictor experiments, and complete 14B system results.

1. Can an independent drafter compensate through a larger overlap budget?

We now test this directly using the official SSD implementation with Qwen3-0.6B and 1.7B drafters. We search draft lengths and branch allocations, then evaluate selected operating points on the complete datasets. SCD achieves 2.35×/1.73× the best tested SSD throughput on MATH500/Arena (Table 6).

The independent drafters can achieve substantial acceptance. The constraint is the cost of obtaining it: longer AR rollouts increase serial execution and speculative branch work. Our budget scans show that higher acceptance does not necessarily improve decoding speed.

Two mechanisms make SCD’s budget effective:

  • Fresh target features support high-quality block-parallel drafts.
  • Hidden-informed joint prediction concentrates early computation on two likely starting points.

With the same drafters, schedule, and two-branch budget, replacing SSD-style selection with joint prediction raises starting-point coverage from 20.99% to 89.88% and improves full-slot throughput by 8.18% (Table 3).

We also distinguish execution time from total draft computation: even when two methods fully hide drafting and obtain the same acceptance, the one requiring less draft work leaves more service capacity for other targets.

2. Parallel target deployment and hardware allocation

Our deployment principle is latency co-design between target and draft. Current DSpark drafters retain target-scale layers: shallow depth reduces total model capacity, but not per-layer computation. A target using TP/EP should therefore be paired with a draft configuration that preserves the required layer execution speed. For these architectures, matching the target’s parallel configuration is the direct starting point.

The draft need not be dedicated to one target. Its shorter service time allows multiple target replicas to share it; our two-target experiment retains 97.1% of dedicated throughput using three rather than four GPUs.

This explains how SCD extends to parallel deployments. Our measurements do not constitute TP2/TP4 target-only benchmarks. The fixed-resource scheduling experiment instead isolates overlap, completing identical decoding work 15.47% sooner (Table 2).

Replicating synchronous instances increases serving capacity, whereas SCD shortens the decoding path within a request. These are complementary deployment objectives.

3. Why retain H33/H37 for late conditioning?

H33/H37 follow the original DSpark feature configurations. We retain them to preserve the baseline design and avoid introducing an additional feature-search variable.

The late stage runs after verification, when the deeper features are already available. Choosing a different late feature is therefore a quality-design choice rather than a prerequisite for overlap. SCD supports different feature depths and stage boundaries; its demonstrated result is that staged conditioning preserves the original drafter’s acceptance while enabling earlier execution.

4. Overlap window, draft scaling, and stage selection

The objective is not to maximize the fraction of target layers overlapped, but to reduce exposed drafting while maintaining useful-token yield.

Our architecture experiments show that middle-to-late target information matters for candidate quality. Staging allows early draft layers to begin with these features and later layers to incorporate deeper information, without requiring the deepest features before any computation starts.

Larger branch budgets are not automatically beneficial: Top4 increases early computation without materially shortening late drafting or recovery. Conversely, adding model capacity can improve quality: our eight-layer, three-stage drafter increases acceptance across all three tasks (Table 1). This extends the design beyond the two-stage example.

The 14B configuration provides a shorter window—five remaining target layers versus six at 4B. This is a configuration choice, not a fixed scaling restriction of SCD. Feature depths and stage boundaries can be tuned jointly for quality and overlap.

5. Complete latency and throughput results

Table 4 now reports TPOT and throughput for both Qwen3-4B and Qwen3-14B, across all three datasets and batch sizes 16 and 32. Throughput improvements range from 5.50% to 17.99%.

The controlled scheduling experiment additionally measures completion of the same decoding workload, showing a 15.47% reduction with identical outputs, acceptance, and routing (Table 2).

6. More systematic architecture evidence

We reorganized the evaluation around separate questions:

  • Does staged conditioning preserve acceptance? Table 1 and Appendix A.
  • Does earlier execution reduce completion time? Table 2.
  • Does joint prediction improve the same asynchronous system? Table 3.
  • Does the complete system improve decoding performance at both scales? Table 4.
  • How do recovery and speculative budgets affect the result? Appendices B and C.

KDA + Target-KV is separated into Appendix E as a history-capacity extension, rather than mixed into the central staging argument.

7. Isolating Main from Small recovery

We evaluate overlap without Small at batch size one: prediction misses rerun Main, and overlap improves decode rate by 7.27–8.53% using the same draft weights (Table 25).

For batched execution, suppressing fallback would remove speculative decoding on misses, changing the comparison. We instead compare Main and Small on the same true prefixes. On missed starting points, Main itself has lower acceptance, and Small trails Main by only 0.29 token on average (Table 13). This separates recovery quality from the greater difficulty of missed prefixes.

Small executes alongside Main’s late stages, so its primary effect is on acceptance rather than necessarily adding a separate waiting period.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Reviewer_ow27OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Staged Conditional Drafting (SCD), a method that enables speculative overlap between target verification and parallel drafting by placing the target and drafter on separate GPUs. The authors adapt two draft architectures, DSpark and KDA + Target-KV, to support staged conditioning. They also introduce a lightweight trainable joint predictor to predict the acceptance boundary and anchor token for the next drafting round. The experimental results show that SCD reduces decoding latency in both single-request and batched settings using a two-GPU deployment.

Critical strengths
  • The proposed method provides a practical design for applying cross-round speculative execution to target-conditioned parallel drafters in a disaggregated deployment.
  • The staged conditioning design effectively reduces the synchronization delay between target verification and drafting by overlapping their execution. The authors also investigate the trade-off between target hidden-state depth, draft quality, and available overlap time through ablation studies.
  • The integration of KDA + Target-KV substantially reduces the memory overhead of draft-owned KV history while preserving competitive acceptance lengths.
Critical weaknesses
  • The proposed design still leaves substantial GPU idle time. The main drafter must wait for early target hidden states, while the target GPU remains idle during late-stage drafting in the BS1 setting. The paper does not address these inefficiencies.
  • The main experiments compare a two-GPU SCD deployment against a single-GPU DSpark baseline, raising concerns about resource fairness. A comparison against two independent DSpark instances under the same GPU budget is missing. The reported throughput gains are relatively modest given the additional GPU.
  • A direct comparison with the SSD extension using EAGLE-3 is missing. Without this baseline, it is difficult to assess the advantage of SCD over an existing target-conditioned SSD approach.
  • The evaluation is limited to the Qwen3 family, with decoding speed measured only on the 4B model. This leaves generalization to other models and larger scales unclear.
  • All overlap experiments use a two-GPU A100 setup. The effectiveness of SCD under different hardware configurations remains unexplored.
Questions for authors
  • The hidden-state selection and branch budget appear hardware-dependent. Would deploying SCD on different GPUs require retraining the drafter and predictor, or could these be adjusted at runtime?
3 comments

Thank you for recognizing the value of staged conditioning and the quality–overlap trade-off. We address your concerns about idle time, resource allocation, asynchronous baselines, and deployment below.

1. Idle time and shared draft resources

SCD’s objective is to shorten each request’s generation path, not to keep a dedicated Main GPU continuously occupied. At a fixed decoding speed and candidate quality, more idle capacity on Main makes it easier to serve additional targets. These intervals need not be wasted resources.

We now demonstrate this directly: two targets sharing one Main require three GPUs instead of four, while retaining 97.1% of dedicated deployment’s aggregate throughput (Table 7). Main therefore need not be dedicated to each target.

The target-side wait is different: exposed drafting delays the next verification and must be reduced. Staged execution moves early drafting into verification, while Small recovery runs alongside Main’s late stages.

2. Resource fairness and the latency objective

Replicating synchronous DSpark increases aggregate serving capacity, but does not by itself shorten the decoding path of an individual request. SCD addresses this latter objective.

Our controlled experiment keeps models, devices, and branch computation fixed and produces identical outputs and acceptance trajectories. Changing feature-release scheduling reduces decoding completion time by 15.47% (Table 2). This demonstrates a latency benefit beyond simply assigning more hardware.

We also add a same-two-GPU comparison against SSD and separately report shared deployment. These comparisons distinguish decoding speed, asynchronous-system performance, and resource sharing.

3. SSD and SSD–EAGLE-3 comparisons

The revision adds the official SSD implementation with Qwen3-0.6B and 1.7B drafters, including budget searches and complete-set evaluation. SCD achieves 2.35×/1.73× the best tested SSD throughput on MATH500/Arena (Table 6).

We also evaluate SSD–EAGLE-3 using the released Qwen3-4B head (Appendix C). The released implementation lacks the FAST recovery path available for independent drafters, so our adapter uses whole-batch JIT recovery. We report its measured runtime alongside separate candidate-quality experiments, rather than attributing recovery overhead to EAGLE-3’s candidate quality.

A same-prefix paired experiment further shows that stale conditioning reduces acceptance even when the true starting point is supplied: accepted length falls from 5.233 to 3.067 on MATH500 and 2.460 to 1.842 on Arena (Table 17). This directly motivates preserving fresh target conditioning.

4. Larger models and hardware-dependent choices

The revision adds Qwen3-14B system results across three datasets and two batch sizes, with throughput improvements of 5.50–14.11% (Table 4).

Hardware changes do not inherently require retraining. Branch budget, scheduling, and device placement can be adjusted while retaining the trained conditioning scheme. Changing feature sources changes the model’s inputs and may require training or fine-tuning.

For current DSpark architectures, target and draft have comparable per-layer computation. Matching their parallel configurations is therefore a direct deployment strategy, while multiple targets can share the shallow draft service. Appendix D discusses deadline-based provisioning and includes a MoE placement microbenchmark.

Thank you for the detailed response and additional experiments.

1. Idle time and shared draft resources

The shared deployment is a meaningful attempt to improve resource utilization. However, the 2-target/1-Main setup demonstrates better efficiency relative to dedicated SCD, rather than eliminating the remaining GPU idle periods. Table 7 only reports aggregate throughput. It would be useful to examine how sharing affects per-target latency and whether the 2-target/1-Main ratio is optimal.

2. Resource fairness and the latency objective

I agree that running two independent DSpark replicas does not reduce single-request latency. However, my concern was about using two GPUs to accelerate a single request through model parallelism, rather than serving two independent requests. A comparison against a two-GPU model-parallel DSpark deployment would help assess single-request latency under the same hardware budget. I appreciate the newly added two-GPU SSD comparison, but it does not directly address this concern.

3. SSD and SSD–EAGLE-3 comparisons

The additional SSD comparisons and budget tuning significantly strengthen the evaluation.

4. Larger models and hardware-dependent choices

The additional 14B experiments address an important limitation of the original submission. Regarding hardware adaptation, my original question concerned cases where different hardware configurations favor different target hidden-state layers or stage boundaries. Since the drafter and predictor are trained with specific target features, changing these inputs may require retraining or fine-tuning. The authors' clarification is helpful, although the adaptation cost remains unclear.

Overall Assessment

The rebuttal and revised experiments address several of my major concerns, particularly the missing SSD comparisons and larger-model evaluations. While I still have concerns about resource-matched latency comparisons and the efficiency of shared deployment, I acknowledge that the revision substantially strengthens the experimental evidence. Therefore, I have decided to update my rating to Weak Reject.

Thanks for clarifying the TP2 comparison. On MATH500 at BS16, steady-state TPOT was 2.08 ms/token for SCD and 2.09 ms/token for synchronous TP2 DSpark. Both used two GPUs, with SCD’s target at TP1. SCD’s overlap can also be combined with TP2 and other execution optimizations, and multiple targets can share Main’s GPU capacity.

The deployment question is how much extra compute and GPU capacity SCD uses, and what that buys in decoding latency.

  1. Extra compute. On Qwen3-4B, we estimate that SCD adds 8–12% decoding FLOPs per output token, mainly from prediction and the second early branch. Our two-GPU SCD deployment delivered 8.5–18.0% higher steady-state decode throughput than synchronous TP1 DSpark. This gives users another way to spend extra compute for faster decoding alongside their existing parallel configuration.

  2. Shared scheduling. With two targets sharing Main, mean request decode time rose from 2.13 to 2.21 s. The current overlap window leaves little time to batch work across targets. Earlier target features or a different stage split could give the scheduler more room. The best sharing ratio depends on the workload and how much accepted length we can retain as we adjust the window.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their constructive feedback. We have substantially revised the manuscript to clarify SCD’s contribution and strengthen the experimental evidence.

SCD shortens the decoding process itself. Conventional draft–verification disaggregation separates resource placement but leaves target-conditioned drafting and verification serial within each request. SCD resolves two dependencies: staged conditioning enables draft layers to consume current-round target features progressively, while joint boundary–anchor prediction and lightweight recovery control the cost of advancing uncertain next-round computation.

The revision adds the following evidence:

  • Quality preservation: Staged DSpark achieves acceptance comparable to the original drafter on both Qwen3-4B and Qwen3-14B (Table 1).
  • Controlled scheduling: With models, devices, branch computation, and outputs unchanged, overlapped SCD completes the same decoding workload 15.47% sooner than delayed SCD (Table 2).
  • Prediction contribution: With drafters and schedule fixed, joint prediction improves starting-point coverage from 20.99% to 89.88% and full-slot throughput by 8.18% over SSD-style selection (Table 3).
  • End-to-end performance: SCD improves throughput by 5.50–17.99% across 4B/14B targets and reaches 2.35×/1.73× the best tested SSD throughput on MATH500/Arena (Tables 4 and 6).
  • Shared deployment: Two targets sharing one Main use three GPUs instead of four while retaining 97.1% of dedicated deployment’s aggregate throughput (Table 7).

We distinguish these results throughout: the controlled experiment measures SCD’s scheduling benefit; the SSD comparison measures complete-system performance; and sharing demonstrates how dedicated draft resources can be amortized. KDA + Target-KV is now presented separately as a memory-capacity extension.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.