acceptodds
Under review as a conference paper at ICLR 2027

SAS: Scaling Speculative Drafting for Efficient 1T LLM Serving with Semi-Autoregressive Decoding

Abstract

Speculative decoding accelerates LLM serving by verifying multiple tokens in parallel through a drafter model. When target verification is much more expensive than drafting, there is room to scale draft computation for longer accepted prefixes. We study depth scaling, which increases backbone capacity by adding layers, and round scaling, which reapplies a shared backbone after extending the selected prefix. In our study, round scaling improves acceptance length more than depth scaling, at the cost of multiple sequential forwards. We introduce Semi-Autoregressive Scaling (SAS) to turn the acceptance gains of round scaling into faster generation while controlling sequential overhead. Its in-block draft combines shallow readout and candidate hedge to produce multiple tokens per forward with early predecessor conditioning. Dynamic partition training supports different round schedules with shared weights, while execution specialization compacts layouts and fuses state handling. On the open-weight 1T-parameter MiMo-v2.5-Pro target evaluated on 32 NVIDIA H20 GPUs across four nodes with 48 concurrent requests in total, the two SAS schedules improve overall throughput by 14.56%-17.08% over native multi-token prediction (MTP), 8.11%-9.88% over DFlash, and 4.17%-7.92% over DSpark across both greedy decoding and sampling, reaching up to autoregressive throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, LLM serving, LLM scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oxvFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SAS improves speculative decoding by allocating more computation to drafting when target-model verification is expensive. It generates draft tokens over multiple rounds, allowing later groups to condition on previously selected tokens. Within each round, a shallow prediction head proposes predecessor candidates, enabling parallel computation of candidate-conditioned continuations that are selected using the full-depth prediction. Dynamic partition training supports different round schedules with shared weights, while execution optimizations reduce overhead. On the approximately 1T-parameter MiMo-v2.5-Pro model deployed across 32 NVIDIA H20 GPUs, SAS achieves up to 2.30× autoregressive throughput and improves throughput over DFlash and DSpark by 8.11%–9.88% and 4.17%–7.92%, respectively.

Critical strengths
  1. SAS improves drafter performance by scaling draft computation, which is practically valuable when verification latency substantially exceeds drafting latency. The experiments also demonstrate end-to-end speedups.

  2. SAS investigates speculative decoding for trillion-parameter target models, offering greater practical relevance than studies focused on smaller-scale models.

Critical weaknesses
  1. The overall novelty is limited, and the work appears primarily to be an engineering optimization with modest gains—less than 8% over the state-of-the-art baseline, DSpark. The pipeline is also complex: additional training and the integration of shallow readout and candidate mixing reduce SAS’s practicality.

  2. The evaluation is limited to a specific hardware configuration. For models at the scale of MiMo-v2.5-Pro, Blackwell GPUs are a more mainstream deployment choice. SAS demonstrates performance gains only on comparatively less powerful H20 GPUs, where high verification latency particularly favors its design. Consequently, the results represent a narrow subset of deployments rather than mainstream LLM serving configurations.

Questions for authors
  1. I could not find an ablation study isolating the effects of shallow readout and candidate mixing. Could the authors add one? Without it, it is difficult to determine whether the increased acceptance length comes from multi-round drafting or these mechanisms, since multi-round drafting alone would already be expected to improve acceptance length.

  2. Could the authors explain the data flow in the right panel of Figure 2 in more detail? The current illustration is difficult to follow, particularly because the attention visibility relationships are not clearly indicated.

5 comments

We sincerely thank you for recognizing the practical value of studying trillion-parameter LLM serving. We appreciate your constructive feedback on the method’s contribution, ablation, and hardware coverage. We address your concerns below.

Q1: Effects of shallow readout and candidate conditioning

We add ablation study to show that shallow supervision and candidate conditioning raise AL by 9.0% over naive multi-round drafting and candidate conditioning adds 7.61% over shallow supervision alone. We compare three Qwen3-8B variants all with N=5N=5, B=8B=8, and π=�0�\pi=�0�:

  • A multi−roundbaselinemulti-round baseline: apply the parallel drafter to groups of 2, 2, and 3 tokens. Each round uses the selected prefix from earlier rounds, with no shallow loss or candidate conditioning.

  • B A+shallowsupervisionA + shallow supervision: add the layer-1 shallow loss, keeping candidate conditioning disabled.

  • C B+candidateconditioning,ie˙,˙SASB + candidate conditioning, i\.e\., SAS: propose shallow leader candidates and create a committed slot for each. Followers attend to their branch's candidate slot, and candidate hedge selects the branch matching the full-depth leader.

All variants train from scratch for 4,000 updates. Shallow supervision raises overall AL by 1.29% from A to B. Candidate conditioning then raises it by 7.61% from B to C. This shows the benefit of predecessor information within a round, alongside the prefix updates between rounds.

VariantMATH-500HumanEvalMT-BenchOverall AL
A4.4523.7913.1913.743
B4.5043.8113.2573.791
C4.8034.1283.5084.080

Q2: Figure 2 data flow and attention visibility

We add a clearer figure in the revised PDF (Figure 3 in Appendix B) to show how one draft forward produces two draft tokens with K=2K=2 shallow candidates.

11 In Figure 3bb, layer 1 processes the leader, initialized from the last verified token, and a masked follower, producing h1h_1 and h2h_2. Shallow readout uses h1h_1 to propose “inference” and “training” as example candidate tokens.

22 Each candidate embedding is fused with verified-context features to form a committed state h1(k)h_1^{(k)}. Candidate mix gives h2(k)=h2+h1(k)h_2^{(k)}=h_2+h_1^{(k)}; the original h2h_2 remains the fallback. The committed candidate slots supply attention states and have no token readout. In layers 2:N2{:}N, all queries can read the verified context, earlier-round prefix, and their own state. Figure 3cc shows the within-group connections: each candidate follower also reads its matching committed slot. Branches remain isolated; the leader and fallback read no other current-group slots.

33 Figure 3dd selects the first follower branch if the full-depth leader predicts “inference”, the second if it predicts “training”, and the fallback otherwise, as in Equation 55. Figure 3aa shows the selected tokens extending the prefix across rounds; the target verifies the completed linear proposal once.

W1: Contribution, throughput gains, and training cost

Compared to current SoTA DSpark, SAS consistently improves throughput on all nine H20 workloads, with 6.18%–8.83% overall gains; the gains reach 10.29% on the harder H20 tasks and 21.20% on B300. Training takes 29.8 versus 24.1 s/step with the same GPUs and data, and one checkpoint supports both schedules.

We show that round scaling improves acceptance more than depth scaling in our study, and design SAS to turn these gains into higher throughput through in-block draft, dynamic partition training, and execution specialization. Besides existing evaluation on six workloads, we also test AIME25, LiveCodeBench, and DeepSWE on MiMo-v2.5-Pro to show the effectiveness on harder and long-context tasks. The table reports decoding throughput in tok/s.

WorkloadDFlashDSparkSAS 2+2+32+2+3SAS 2+52+5
AIME253416.43447.43690.53702.8
LCB2675.92776.23044.13061.9
DeepSWE2837.92961.43118.33141.5

Both schedules outperform DFlash and DSpark on all three workloads. SAS 2+52+5 gains 7.41%, 10.29% and 6.08% over DSpark.

With the same data budget and GPU allocation, training on MiMo-v2.5-Pro takes about 29.8 s/step for SAS and 24.1 s/step for DSpark. Dynamic partition training produces one checkpoint for both π=(2,2,3)\pi=(2,2,3) and π=(2,5)\pi=(2,5), avoiding separate training for each schedule. The extra training cost can be spread over many serving requests.

The serving gains matter because they recur over the target model's deployment lifetime. Our experiments evaluate end-to-end throughput for a real 1T LLM deployment on 32 H20 GPUs. At fixed workload and GPU pricing, higher throughput directly reduces GPU time per served token. Therefore, a persistent gain approaching 8% can still have substantial absolute value when applied to a large volume of served tokens.

W2: Blackwell measurements

We add evaluation on real B300 deployment and show that both SAS schedules improve decoding throughput over DFlash and DSpark on all workloads. We evaluate MiMo-v2.5-Pro on 16 NVIDIA B300 GPUs with attention DP2×TP8, MoE EP16, batch size 12 per attention DP group, and total concurrency 24. All methods use temperature 1, top-p 0.95, and thinking enabled. The workloads include the five public benchmarks in the paper, AIME25, LiveCodeBench and DeepSWE. Decoding throughput excludes prefill and is reported as mean±sample standard deviation across three repetitions.

WorkloadDFlashDSparkSAS 2+2+3SAS 2+5
AIME251084.7±32.41117.0±41.61210.6±54.71203.3±20.1
LCB840.8±31.8858.3±22.3946.9±39.2945.8±11.6
GSM8K1261.3±39.11284.3±52.61419.8±78.81409.8±23.1
HumanEval1148.1±44.91127.2±20.51366.2±33.71356.0±59.3
MATH-5001266.7±15.61286.5±43.21398.2±21.01389.1±12.2
MBPP1236.4±30.71257.1±45.11446.8±47.61426.3±23.0
MT-Bench999.9±69.61022.3±10.81142.7±18.31138.5±17.0
DeepSWE1085.1±5.01138.2±25.01254.5±45.01226.6±24.8

SAS 2+2+32+2+3 improves mean throughput by 8.38%–21.20% over DSpark and 10.38%–19.00% over DFlash. SAS 2+52+5 improves it by 7.73%–20.30% and 9.66%–18.11%, respectively. Faster GPUs can reduce both verification time τverify\tau_{\mathrm{verify}} and drafting time τdraft\tau_{\mathrm{draft}}.

As H20 is a cheaper deployment option for LLM inference, we report results on H20 in the main paper. However, results below show SAS gain consistent improvements across devices. This is because SAS is guided by the measured balance between target verification and drafting instead of the capability of GPUs, as lower draft latency can support additional rounds whenever verification remains much more expensive than drafting.


In the revised manuscript (marked in green and purple), we add the mechanism ablation, evaluations on harder and long-context tasks, and B300 throughput measurements. We also clarified the in-block data flow, attention visibility, and drafting-budget criterion. We hope these updates could resolve your concerns and sincerely ask if you might consider raising the score.

Thank you again for your thoughtful suggestions. We added ablations separating the effects of shallow supervision and candidate conditioning from multi-round drafting, harder-task evaluations, and B300 measurements showing throughput gains over DFlash and DSpark across all tested B300 workloads. We would appreciate your thoughts on these additions and would be happy to address any remaining concerns about the method or its practical value.

Reviewer_GTgEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Given verification using a large model is expensive, this paper explores a better drafting algorithm such that each verification step is more fruitful, ie. improves the probability of drafted tokens to be accepted. In order to do this the authors explore 2 directions of increasing draft compute.

  1. Depth scaling the drafter and draft multiple tokens in parallel
  2. Round scaling, ie iterative causal decoding

The main contribution of the paper is to enable efficient causal round scaling with an adjustable computation budget through in-block draft, dynamic partition training, and execution specialization. The idea is to use initial layers of the draft model to propose multiple token candidates which are then fed to the remaining backbone to generated next token predictions. This basically scales the sequence axis and modifies the attention mask accordingly to propose multiple candidate paths per draft forward.

Overall the idea looks promising and is able to illustrate the impact of it through experimental validation. However, similar work has been done in the past in quadratic decoding proposed in [1]. The results look promising especially given the benefit on SoTA methods like DFlash. [1]. Samragh, Mohammad, et al. "Your llm knows the future: Uncovering its multi-token prediction potential." arXiv preprint arXiv:2507.11851 (2025).

Critical strengths
  1. Good improvements over most SoTA methods
  2. Problem motivation is clear on why depth scaling didn't work and round scaling might work
Critical weaknesses
  1. Figure 2 is difficult to follow
  2. The training part is not clear. I could not understand the objective.
Questions for authors
  1. I was not able to understand if the draft model weights are different for each round.
  2. Is there any analysis on the depth where the leaders are chosen?
  3. I could not understand how this is combined with DFlash?
3 comments

We sincerely thank you for recognizing the motivation behind round scaling and the improvements over strong speculative decoding baselines. We appreciate your constructive questions about the method, which have helped us strengthen the presentation. Below, we address each question and explain the corresponding results.

Q1: Are the weights different for each round?

No. All rounds and candidate branches share the same drafter weights. The stack D{1:N}D_\{1:N\} has N=5N=5 layers in the main experiments. Different layers have separate parameters, but each round reuses the full stack with the prefix extended by earlier rounds. Dynamic partition training also shares these weights across schedules π\pi.

Q2: Shallow readout depth

We add a study on shallow readout depth to show layer 1 gives the highest overall AL among the three tested readout depths. We train three Qwen3-8B SAS drafters from scratch for 4,000 updates, varying the shallow readout from layer 1 to 3. All use N=5N=5, B=8B=8, K=4K=4, and π=�0�\pi=�0�. We compare AL and Hit@K averaged over three repetition. Hit@K is the fraction of full-depth leader predictions contained in the shallow top-K candidates.

Shallow readout layerMATH-500HumanEvalMT-BenchOverall ALOverall Hit@1 / 4 / 8 %
14.7964.1543.4664.06772.87 / 91.77 / 95.20
24.7544.1163.4644.04478.69 / 95.03 / 97.46
34.7604.0833.4694.03784.04 / 97.25 / 98.72

Moving the readout from layer 1 to 3 raises Hit@4 from 91.77% to 97.25%, while overall AL falls from 4.067 to 4.037. This supports early injection at this training budget: it leaves more Transformer layers for followers to use the candidate information.

Q3: Combining SAS with DFlash

SAS uses DFlash's parallel block drafter as its backbone and applies it once per causal group. Within a round, shallow readout, candidate mix, and candidate hedge supply early predecessor information. Across rounds, selected tokens extend the prefix. With B=8B=8 and π=(2,2,3)\pi=(2,2,3), three forwards produce seven proposal tokens, followed by one target verification.

The distinction from Quadratic Decoding is where predecessor conditioning enters the computation. Quadratic Decoding uses gated LoRA in the target to predict future representations, followed by a lightweight sampler conditioned on previously sampled tokens. SAS supplies candidate tokens before the remaining layers of a separate drafter and reuses its backbone across rounds with an extended prefix. This organizes causal computation within the drafter while preserving the target architecture and verification block.

W1: Figure 2 data flow

Shallow candidates condition separate follower branches; the full-depth leader selects which follower prediction to keep. We add Figure 3 in Appendix B of the revised manuscript to illustrate this computation.

11 In Figure 3bb, for the context “We use speculative decoding to accelerate”, shallow readout proposes x^1(1)=\hat{x}_1^{(1)}= “inference” and x^1(2)=\hat{x}_1^{(2)}= “training”. These are example candidate tokens. The leader starts from E(x0)E(x_0), where x0x_0 is the last verified token, and the follower starts from a mask embedding. Both are fused with verified-context features hch_c. Layer 1 produces h1h_1 and h2h_2, and the shallow head uses h1h_1 to propose the candidates.

22 The candidate mix in Figure 3bb constructs

$h_1^{(k)}=\phi(E(\hat{x}_1^{(k)}),h_c), \qquad h_2^{(k)}=h_2+h_1^{(k)},\quad k\in{1,2}.$

The committed states h1(1)h_1^{(1)} and h1(2)h_1^{(2)} supply attention states and have no token readout. Each follower branch retains its layer-1 state and adds its candidate state; the original h2h_2 remains the fallback. Layers 2:N2{:}N process the leader and all branches together. All queries can read the verified context, earlier-round prefix, and their own state. Figure 3cc shows the remaining attention connections within this two-token group:

QueryOther current-group state it can read
Leader h1h_1None
Fallback follower h2h_2None
Follower h2(1)h_2^{(1)}Its committed candidate state h1(1)h_1^{(1)}
Follower h2(2)h_2^{(2)}Its committed candidate state h1(2)h_1^{(2)}

33 Figure 3dd shows candidate hedge after the shared LM head produces the full-depth leader and follower predictions. If x^1\hat{x}_1 is “inference”, candidate hedge keeps x^2(1)\hat{x}_2^{(1)}; if it is “training”, it keeps x^2(2)\hat{x}_2^{(2)}; otherwise, it keeps fallback x^2\hat{x}_2. The leader remains x^1\hat{x}_1 in every case. Figure 3aa shows how each round extends the selected prefix before reusing the backbone. The target verifies one linear proposal after all rounds, as in Equation 55 and Algorithm 1.

W2: Training objective

Like DFlash, we train the drafter using supervision from a frozen target. Our objective jointly trains the final token predictions and the shallow leader proposals:

L=Eπ∼U(Π)[ℓdeep(π)+0.3 ℓsh(π)].\mathcal L=\mathbb E_{\pi\sim U(\Pi)}[\ell_{\mathrm{deep}}(\pi)+0.3\,\ell_{\mathrm{sh}}(\pi)].

The deep loss supervises all valid proposal positions; the shallow loss supervises only leaders, teaching the early head to provide candidates for the followers. We sample a partition π\pi from the training set Π\Pi and use its causal mask, so the shared parameters learn several round schedules. Both terms use weighted distribution matching:

ℓu(π)=∑i∈Ωu(π)wiH2(pi,qiu)∑i∈Ωu(π)wi,u∈{deep,sh}.\ell_u(\pi)=\frac{\sum_{i\in\Omega_u(\pi)}w_i H^2(p_i,q_i^u)}{\sum_{i\in\Omega_u(\pi)}w_i},\qquad u\in\{\mathrm{deep},\mathrm{sh}\}.

Here pip_i is the target distribution, qiuq_i^u the corresponding draft prediction, H2H^2 squared Hellinger distance, and Ωu\Omega_u the positions supervised by that term. Training uses teacher-forced prefixes and leader tokens. The target, embeddings and final LM head remain frozen. Earlier draft errors invalidate later proposals, so we use detached, acceptance-aware position weights:

ρ~ℓ=(ρℓ+1)/2,wi=∑m=iB−1∏ℓ=1mρ~ℓ,\widetilde\rho_\ell=(\rho_\ell+1)/2,\qquad w_i=\sum_{m=i}^{B-1}\prod_{\ell=1}^{m}\widetilde\rho_\ell,

where ρℓ\rho_\ell is the full-depth drafter's probability of the target token at offset ℓ\ell, and B−1B-1 is the proposal count.


Overall, in the revised manuscript (marked in brown and purple), we clarified the DFlash backbone, weight sharing, training objective, and in-block data flow. We also added an explanatory figure and the shallow readout depth study. We are glad to provide further clarifications, and we would be very grateful if you could consider raising your final rating based on the clarifications and revisions outlined above.

Thank you again for your helpful questions. In our rebuttal and revised manuscript, we clarified weight sharing across rounds, how SAS builds on DFlash, and the joint training objective for deep and shallow predictions. We also added a detailed data-flow illustration, together with experiments comparing shallow readout depths. We wanted to follow up and invite your feedback on whether these additions make the method clearer. Please let us know if any aspect of the design or training procedure would benefit from further explanation.

Reviewer_BUj6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effects of Depth Scaling and Round Scaling for draft models on a 1T-parameter target model (MiMo-v2.5-Pro). Through motivation experiments, the authors demonstrate that, under a fixed draft length, increasing causal rounds (Round Scaling) yields higher Acceptance Length (AL) compared to increasing network depth (Depth Scaling), and they propose the Semi-Autoregressive Scaling (SAS) method. SAS combines in-block draft, candidate hedge, and dynamic partition training. On MiMo-v2.5-Pro, it achieves up to a 2.30x improvement in overall throughput, outperforming DFlash and DSpark by 8.11% - 9.88% and 4.17% - 7.92%, respectively. Furthermore, when composed with DFlash-2 on Qwen3.8-27B, it further improves the acceptance length by 9.37%.

Critical strengths
  1. This paper targets the pain point of extremely high verification costs in large target models, the idea of round scaling resembles a loop transformer [1]. Sharing a 5-layer draft model and adopting an in-block semi-autoregressive format like (2,2,3) is intuitive and effective. By sharing the backbone to increase inference rounds rather than parameter counts, it effectively mitigates the risk of "over-capacity/overfitting" that might arise from merely increasing depth. Moreover, dynamic partition training enables flexible adjustment of the computation budget.

  2. On the scale of the experiments, this paper conducts end-to-end tests on a frontier 1T-parameter model (MiMo-v2.5-Pro) is very rare in prior speculative decoding literature. Combined with a realistic 32x H20 GPU deployment, continuous batching, and 48-concurrency testing, this work demonstrates highly valuable practical reference for industrial deployment.

  3. The presentation is mostly clear and accessible. The motivation (Figure 1) intuitively compares the effects of depth and round scaling on conditional acceptance rates. The explanation of the mechanisms for candidate mix and candidate hedge (Figure 2 and Equations 4-5) is logically rigorous. Additionally, the execution specialization section clearly describes the underlying operator fusion and memory layout optimizations.

Critical weaknesses
  1. Capacity Constraints of the Shared Backbone needs more justification. Round scaling has currently only proven its superiority on several relatively simple and short datasets. I suspect that sharing the same backbone parameters across rounds might encounter a capacity bottleneck when dealing with more difficult datasets with more complex distributions.

  2. Lack of Convincing Target Generalization Experiments and some more justification is needed. In the target experiments on Qwen3.8-27B, the authors did not report actual throughput metrics. Considering that round scaling inevitably increases draft latency, this latency overhead is non-negligible in scenarios with a 27B target where verification costs are relatively lower. Therefore, reporting only the AL improvement represents an unfair comparison. The authors should provide more experiments to determine the optimal backbone depth and round configurations for targets like Qwen3.8-27B, thereby providing theoretical guidance for parameter selection to demonstrate the method's generalizability.

  3. Semi-Autoregressive Approach Reintroduces Training-time Mismatch Issues: In the in-block draft, follower positions cannot receive the true target hidden states from preceding positions, which equates to reintroducing the training-time mismatch problem similar to that in EAGLE-3 [2]. However, I do not see how the authors consider or mitigate this potential issue in the paper.

Questions for authors
  1. Experiments on More Datasets: The reported acceptance lengths on these datasets (e.g., GSM8K, HumanEval) are inherently unstable in existing literature. The authors need to further isolate the dataset difficulty factor and prove the true effect of round scaling on more challenging or longer-text datasets.

  2. More In-depth Target Generalization Experiments: Clear theoretical guidance and experimental evidence are needed for selecting backbone layers and round numbers for general target models (e.g., different scales and architectures). Otherwise, the paper appears more like an internal technical report for the MiMo model rather than a universally applicable acceleration method.

[1] Dehghani M, Gouws S, Vinyals O, et al. Universal transformers[J]. arXiv preprint arXiv:1807.03819, 2018. [2] Li Y, Wei F, Zhang C, et al. Eagle-3: Scaling up inference acceleration of large language models via training-time test[J]. Advances in Neural Information Processing Systems, 2026, 38: 136737-136756.

2 comments

We sincerely thank you for recognizing the value of round scaling and the practical relevance of our trillion-parameter serving experiments. We appreciate your thoughtful comments on backbone capacity and generalization, which have helped us strengthen the evaluation on harder tasks and clarify the selection of depth and round configurations for different target LLMs. Below, we address your concerns point by point.

W1/Q1: Shared-backbone capacity on harder and longer tasks

We evaluate harder and long-context tasks (AIME25, LiveCodeBench, DeepSWE) with MiMo-v2.5 Pro and Qwen3-8B, showing that (1) SAS improves throughput by 6.08%–10.29% over DSpark and (2) round scaling improves AL by 10.2%–13.2%, compared with 2.9%–4.0% from depth scaling. Sharing the five-layer backbone keeps its parameter count fixed, separating round scaling from adding backbone parameters.

(1) We evaluate MiMo-v2.5-Pro on AIME25, LiveCodeBench, and DeepSWE using 32 NVIDIA H20 GPUs, attention DP4×TP8, MoE EP32, and concurrency 48, with at most 12 requests per attention DP group. The table reports sampling throughput in tok/s from three independent repetitions, with temperature 1, top-p 0.95, and thinking enabled. AIME25 and LiveCodeBench use steady-state end-to-end throughput including prefill. DeepSWE uses a decode throughput estimate excluding prefill.

WorkloadDFlashDSparkSAS 2+2+32+2+3SAS 2+52+5
AIME253416.43447.43690.53702.8
LCB2675.92776.23044.13061.9
DeepSWE2837.92961.43118.33141.5

Both SAS schedules outperform DFlash and DSpark on all three workloads. SAS 2+52+5 is the faster schedule in each case, gaining 7.41% on AIME25, 10.29% on LiveCodeBench, and 6.08% on DeepSWE over DSpark. These results extend the throughput gains to harder reasoning, code generation, and long-context software engineering.

(2) We compare depth and round scaling on Qwen3-8B using AIME25 and LiveCodeBench. The checkpoints use the same training recipe and data budget.

DrafterAIME25LCBOverall ALGain over N=5N=5, J=1J=1
N=5N=5, J=1J=1 (DFlash)5.2255.5175.469—
N=10N=10, J=1J=15.5635.6435.630+2.9%
N=15N=15, J=1J=15.3555.7505.689+4.0%
N=5N=5, J=3J=35.7096.0866.025+10.2%
N=5N=5, J=4J=45.6826.2326.143+12.3%
N=5N=5, J=7J=75.8256.2606.191+13.2%

Every five-layer Round-Scale configuration exceeds both deeper single-pass drafters on both datasets. Additional layers process the same verified context, while additional rounds receive tokens selected by earlier rounds. These results support the value of extending the conditioning prefix alongside increasing backbone capacity.

W3: Concerns on training and inference mismatch

SAS uses target hidden features only from the verified context during both training and inference to prevent mismatch. Followers receive states constructed by the drafter, rather than true target hidden states at unverified proposal positions. The source of target features therefore stays the same across training and inference.

(1) During training, round jj receives verified context cc, its target features hch_c, and the teacher-forced prefix Pj−1P_{j-1}. The leader is at sjs_j and followers are at positions Fj\mathcal F_j. A masked follower produces layer-1 state hih_i. Equation (4) constructs its candidate-conditioned state as

hsj(k)=ϕ(E(xsj^(k)),hc),hi(k)=hi+hsj(k),i∈Fj.h_{s_j}^{(k)}=\phi\left(E(\hat{x_{s_j}}^{(k)}),h_c\right), h_i^{(k)}=h_i+h_{s_j}^{(k)}, i\in\mathcal F_j.

Here EE is the token embedding and ϕ\phi is input fusion. Training uses the true leader token xsjx_{s_j} in place of the shallow candidate x^sj(k)\hat{x}_{s_j}^{(k)}. The resulting state initializes a committed slot and is added to the branch's followers. The target supplies supervision distributions, but its hidden states at unverified positions are not follower inputs.

(2) At inference, the selected prefix and shallow candidates enter through the same input construction. Layers 2:N2{:}N retain the within-round attention mask, and the full-depth leader selects the matching follower branch. Teacher forcing changes the conditioning tokens but does not change follower inputs from true in-block target features to draft-generated features.

W2/Q2: Throughput and selection of depth and rounds for different target LLMs

We report actual decoding throughput comparison on Qwen3.8-27B and conduct hyper-parameter studies using Qwen3.8-27B and Qwen3.6-35B-A3B to show generalizability and principles on selection of optimal configurations. In summary, (1) SAS improves decoding throughput by 3.69%–14.22% over DFlash-2 on Qwen3.8-27B; (2) optimal depth NN and partition π\pi depend on specific targets and should be selected by measured throughput. SAS makes this selection more flexiable by training one checkpoint per depth supporting multiple partitions.

(1) Actual throughput on Qwen3.8-27B.

We compare the official DFlash-2 checkpoint from huggingface with SAS (2,5)(2,5) under the same target and sampling settings on the five datasets. Both approaches are deployed on two GPUs at TP2 and evaluated with B=8B=8, temperature=1, top-p=0.95 and thinking enabled. The table reports decode throughput excluding prefill. Overall, SAS improves the decode throughput by 3.69%–14.22%.

DatasetDFlash-2SAS (2,5)(2,5)Gains
GSM8K2522.622881.28+14.22%
HumanEval2006.942097.53+4.51%
MATH-5002368.952480.24+4.70%
MBPP2059.912173.61+5.52%
MT-Bench1683.181745.30+3.69%
Overall2285.402480.41+8.53%

(2) Selecting depth and rounds across targets.

Let AA be the accepted draft-prefix length excluding the target bonus, and let τverify\tau_{\mathrm{verify}} and τdraft\tau_{\mathrm{draft}} denote verification and drafting time. At fixed target, block budget BB, and batch size, decode throughput is approximately proportional to

1+E[A]τverify+τdraft.\frac{1+\mathbb E[A]}{\tau_{\mathrm{verify}}+\tau_{\mathrm{draft}}}.

With verification time fixed, additional drafting improves this ratio when

<(\tau_{\mathrm{verify}}+\tau_{\mathrm{draft}}) \frac{\Delta\mathbb E[A]}{1+\mathbb E[A]}.$$ This suggests a selection procedure: **train a few backbone depths with the same data budget, compare supported partitions using one checkpoint per depth, and prioritize acceptance gains that cover the added step time.** Under this principle, we study how different $N$ and $\pi$ affect the final performance on Qwen3.8-27B and Qwen3.6-35B-A3B by training multiple SAS checkpoints with the same budget and comparing the end-to-end throughput. For Qwen3.8-27B, we compare $N\in\{3,4\}$ and seven partition strategies. We find that changing $(2,5)$ to $(2,2,3)$ raises AL by 1.73%/2.04% for $N=3/4$, but lowers throughput by 1.01%/1.66%. Among these 14 configurations, $N=4$, $\pi=(2,5)$ has the highest mean throughput. | Partition $\pi$ | $J$ | $N=3$ AL | $N=3$ E2E tok/s | $N=4$ AL | $N=4$ E2E tok/s | |---|---:|---:|---:|---:|---:| | $(2,5)$ | 2 | 4.0291 | 970.17±5.85 | 4.1379 | 976.80±5.05 | | $(3,4)$ | 2 | 4.0141 | 972.38±4.48 | 4.1045 | 969.48±4.07 | | $(4,3)$ | 2 | 3.9459 | 949.84±1.93 | 4.0520 | 948.59±2.54 | | $(5,2)$ | 2 | 3.8804 | 942.93±3.14 | 3.9715 | 942.19±4.09 | | $(2,2,3)$ | 3 | 4.0990 | 960.42±2.31 | 4.2222 | 960.54±1.15 | | $(2,3,2)$ | 3 | 4.0704 | 955.76±4.55 | 4.1837 | 950.33±3.55 | | $(3,2,2)$ | 3 | 4.0332 | 952.30±3.93 | 4.1454 | 952.00±3.44 | we compare $N\in\{3,4\}$ and seven partition strategies. We find that changing $(2,5)$ to $(2,2,3)$ raises AL by 1.73%/2.04% for $N=3/4$, but lowers throughput by 1.01%/1.66%. Among these 14 configurations, $N=4$, $\pi=(2,5)$ has the highest mean throughput. For Qwen3.6-35B-A3B, we compare $N\in\{3,6\}$ and two partition strategies. We find that both 35B-A3B backbones obtain higher AL with three rounds but higher throughput with two. The highest mean among the shown configurations uses $N=3$, FFN width 10240, and $(2,5)$, whereas the 27B study favors $N=4$, $(2,5)$. | $N$ | FFN width | Partition $\pi$ | $J$ | AL | E2E tok/s | |---:|---:|---|---:|---:|---:| | 3 | 10240 | $(2,5)$ | 2 | 4.3566 | 2708.01±5.51 | | 3 | 10240 | $(2,2,3)$ | 3 | 4.4375 | 2601.28±17.04 | | 6 | 6144 | $(2,5)$ | 2 | 4.4719 | 2649.49±19.14 | | 6 | 6144 | $(2,2,3)$ | 3 | 4.5555 | 2491.30±13.83 | Overall, these experiments extend SAS to both dense-FFN and MoE targets. The backbone and full partition should be selected by measured throughput for each target to achieve the optimal performance. --- In the revised manuscript, the changes addressing these points are marked in blue and purple. We added harder-task evaluations, actual throughput and configuration studies on more targets, and clarified follower inputs during training and inference. Should you have any remaining concerns, please let us know. We are glad to provide further clarifications and would be grateful if you could consider raising your final score.

Comments on the paper

1 comment

We thank the reviewers for their careful reading and constructive feedback, and the Area Chair for overseeing the review process. We appreciate the reviewers' recognition of the clear motivation for round scaling, the improvements over strong speculative decoding baselines, and the practical value of serving LLMs with a trillion parameters. We expanded the evaluation and discussion to address concerns raised by the reviewers:

  1. Harder tasks and shared backbone capacity: We evaluated AIME25, LiveCodeBench, and DeepSWE for MiMo-v2.5-Pro. Under sampling, SAS improves 7.41%, 10.29%, and 6.08% throughput over SoTA method DSpark on the three benchmarks, respectively. A Qwen3-8B study using the same training recipe and data budget further shows 10.2%–13.2% overall AL gains from round scaling, compared with 2.9%–4.0% from depth scaling, on AIME25 and LiveCodeBench. These results support the effectiveness of the shared backbone on more challenging tasks.

  2. Different targets and configuration selection: Measurements on Qwen3.8-27B show that SAS improves decode throughput over the official DFlash-2 checkpoint by 3.69%–14.22% across workloads and 8.53% overall. Backbone and partition studies on Qwen3.8-27B and Qwen3.6-35B-A3B extend the evaluation to dense and MoE targets. They show that higher AL does not always yield higher throughput, supporting selection by measured throughput for each target. One checkpoint supports multiple partitions, allowing this tradeoff to be adjusted without separate training for each schedule.

  3. Hardware generalization: We added measurements of the same 1T target on 16 NVIDIA B300 GPUs. Both SAS schedules achieve higher decode throughput than DFlash and DSpark on all eight evaluated workloads. Gains over DSpark range from 7.73% to 21.20% across the two schedules and workloads. These measurements show that the benefits of SAS extend to the Blackwell deployment.

  4. Mechanism attribution and hyper-parameter study: we add a Qwen3-8B ablation, which shows shallow supervision increases overall AL by 1.29%, and candidate conditioning yields a further 7.61%, providing evidence of benefits beyond increasing the number of rounds alone. A study of readout depth finds that layer 1 gives the highest overall AL among layers 1–3 at the evaluated 3 SAS variations.

  5. Training and inference clarity: We clarified the DFlash backbone, weight sharing across rounds and branches, and the deep/shallow training objective. We add figures to explicitly illustrate candidate conditioning, attention visibility and branch selection. We also clarify that target features come only from verified context during both training and inference and thus SAS does not exist training-inference mismatch.

In summary, our expanded experiments show across nine workloads on H20 and B300, SAS reaches overall throughput gains of 8.76%–12.01% over DFlash and 6.18%–8.83% over SoTA DSpark under both greedy decoding and sampling. This improvement could be transfered to more targets, like Qwen3.6-35B-A3B and Qwen3.8-27B, and could be integrated with advanced draft optimization techniques like DFlash-2. We hope this evidence, together with the detailed replies and revised manuscript, addresses the reviewers' concerns and assists the Area Chair's assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.