- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
SAS improves speculative decoding by allocating more computation to drafting when target-model verification is expensive. It generates draft tokens over multiple rounds, allowing later groups to condition on previously selected tokens. Within each round, a shallow prediction head proposes predecessor candidates, enabling parallel computation of candidate-conditioned continuations that are selected using the full-depth prediction. Dynamic partition training supports different round schedules with shared weights, while execution optimizations reduce overhead. On the approximately 1T-parameter MiMo-v2.5-Pro model deployed across 32 NVIDIA H20 GPUs, SAS achieves up to 2.30× autoregressive throughput and improves throughput over DFlash and DSpark by 8.11%–9.88% and 4.17%–7.92%, respectively.
Critical strengths
-
SAS improves drafter performance by scaling draft computation, which is practically valuable when verification latency substantially exceeds drafting latency. The experiments also demonstrate end-to-end speedups.
-
SAS investigates speculative decoding for trillion-parameter target models, offering greater practical relevance than studies focused on smaller-scale models.
Critical weaknesses
-
The overall novelty is limited, and the work appears primarily to be an engineering optimization with modest gains—less than 8% over the state-of-the-art baseline, DSpark. The pipeline is also complex: additional training and the integration of shallow readout and candidate mixing reduce SAS’s practicality.
-
The evaluation is limited to a specific hardware configuration. For models at the scale of MiMo-v2.5-Pro, Blackwell GPUs are a more mainstream deployment choice. SAS demonstrates performance gains only on comparatively less powerful H20 GPUs, where high verification latency particularly favors its design. Consequently, the results represent a narrow subset of deployments rather than mainstream LLM serving configurations.
Questions for authors
-
I could not find an ablation study isolating the effects of shallow readout and candidate mixing. Could the authors add one? Without it, it is difficult to determine whether the increased acceptance length comes from multi-round drafting or these mechanisms, since multi-round drafting alone would already be expected to improve acceptance length.
-
Could the authors explain the data flow in the right panel of Figure 2 in more detail? The current illustration is difficult to follow, particularly because the attention visibility relationships are not clearly indicated.
5 comments
W2: Blackwell measurements
We add evaluation on real B300 deployment and show that both SAS schedules improve decoding throughput over DFlash and DSpark on all workloads. We evaluate MiMo-v2.5-Pro on 16 NVIDIA B300 GPUs with attention DP2×TP8, MoE EP16, batch size 12 per attention DP group, and total concurrency 24. All methods use temperature 1, top-p 0.95, and thinking enabled. The workloads include the five public benchmarks in the paper, AIME25, LiveCodeBench and DeepSWE. Decoding throughput excludes prefill and is reported as mean±sample standard deviation across three repetitions.
| Workload | DFlash | DSpark | SAS 2+2+3 | SAS 2+5 |
|---|---|---|---|---|
| AIME25 | 1084.7±32.4 | 1117.0±41.6 | 1210.6±54.7 | 1203.3±20.1 |
| LCB | 840.8±31.8 | 858.3±22.3 | 946.9±39.2 | 945.8±11.6 |
| GSM8K | 1261.3±39.1 | 1284.3±52.6 | 1419.8±78.8 | 1409.8±23.1 |
| HumanEval | 1148.1±44.9 | 1127.2±20.5 | 1366.2±33.7 | 1356.0±59.3 |
| MATH-500 | 1266.7±15.6 | 1286.5±43.2 | 1398.2±21.0 | 1389.1±12.2 |
| MBPP | 1236.4±30.7 | 1257.1±45.1 | 1446.8±47.6 | 1426.3±23.0 |
| MT-Bench | 999.9±69.6 | 1022.3±10.8 | 1142.7±18.3 | 1138.5±17.0 |
| DeepSWE | 1085.1±5.0 | 1138.2±25.0 | 1254.5±45.0 | 1226.6±24.8 |
SAS improves mean throughput by 8.38%–21.20% over DSpark and 10.38%–19.00% over DFlash. SAS improves it by 7.73%–20.30% and 9.66%–18.11%, respectively. Faster GPUs can reduce both verification time and drafting time .
As H20 is a cheaper deployment option for LLM inference, we report results on H20 in the main paper. However, results below show SAS gain consistent improvements across devices. This is because SAS is guided by the measured balance between target verification and drafting instead of the capability of GPUs, as lower draft latency can support additional rounds whenever verification remains much more expensive than drafting.
In the revised manuscript (marked in green and purple), we add the mechanism ablation, evaluations on harder and long-context tasks, and B300 throughput measurements. We also clarified the in-block data flow, attention visibility, and drafting-budget criterion. We hope these updates could resolve your concerns and sincerely ask if you might consider raising the score.
Thank you again for your thoughtful suggestions. We added ablations separating the effects of shallow supervision and candidate conditioning from multi-round drafting, harder-task evaluations, and B300 measurements showing throughput gains over DFlash and DSpark across all tested B300 workloads. We would appreciate your thoughts on these additions and would be happy to address any remaining concerns about the method or its practical value.
We sincerely thank you for recognizing the practical value of studying trillion-parameter LLM serving. We appreciate your constructive feedback on the method’s contribution, ablation, and hardware coverage. We address your concerns below.
Q1: Effects of shallow readout and candidate conditioning
We add ablation study to show that shallow supervision and candidate conditioning raise AL by 9.0% over naive multi-round drafting and candidate conditioning adds 7.61% over shallow supervision alone. We compare three Qwen3-8B variants all with N=5, B=8, and π=�0�:
A multi−roundbaseline: apply the parallel drafter to groups of 2, 2, and 3 tokens. Each round uses the selected prefix from earlier rounds, with no shallow loss or candidate conditioning.
B A+shallowsupervision: add the layer-1 shallow loss, keeping candidate conditioning disabled.
C B+candidateconditioning,ie˙,˙SAS: propose shallow leader candidates and create a committed slot for each. Followers attend to their branch's candidate slot, and candidate hedge selects the branch matching the full-depth leader.
All variants train from scratch for 4,000 updates. Shallow supervision raises overall AL by 1.29% from A to B. Candidate conditioning then raises it by 7.61% from B to C. This shows the benefit of predecessor information within a round, alongside the prefix updates between rounds.
Q2: Figure 2 data flow and attention visibility
We add a clearer figure in the revised PDF (Figure 3 in Appendix B) to show how one draft forward produces two draft tokens with K=2 shallow candidates.
1 In Figure 3b, layer 1 processes the leader, initialized from the last verified token, and a masked follower, producing h1 and h2. Shallow readout uses h1 to propose “inference” and “training” as example candidate tokens.
2 Each candidate embedding is fused with verified-context features to form a committed state h1(k). Candidate mix gives h2(k)=h2+h1(k); the original h2 remains the fallback. The committed candidate slots supply attention states and have no token readout. In layers 2:N, all queries can read the verified context, earlier-round prefix, and their own state. Figure 3c shows the within-group connections: each candidate follower also reads its matching committed slot. Branches remain isolated; the leader and fallback read no other current-group slots.
3 Figure 3d selects the first follower branch if the full-depth leader predicts “inference”, the second if it predicts “training”, and the fallback otherwise, as in Equation 5. Figure 3a shows the selected tokens extending the prefix across rounds; the target verifies the completed linear proposal once.
W1: Contribution, throughput gains, and training cost
Compared to current SoTA DSpark, SAS consistently improves throughput on all nine H20 workloads, with 6.18%–8.83% overall gains; the gains reach 10.29% on the harder H20 tasks and 21.20% on B300. Training takes 29.8 versus 24.1 s/step with the same GPUs and data, and one checkpoint supports both schedules.
We show that round scaling improves acceptance more than depth scaling in our study, and design SAS to turn these gains into higher throughput through in-block draft, dynamic partition training, and execution specialization. Besides existing evaluation on six workloads, we also test AIME25, LiveCodeBench, and DeepSWE on MiMo-v2.5-Pro to show the effectiveness on harder and long-context tasks. The table reports decoding throughput in tok/s.
Both schedules outperform DFlash and DSpark on all three workloads. SAS 2+5 gains 7.41%, 10.29% and 6.08% over DSpark.
With the same data budget and GPU allocation, training on MiMo-v2.5-Pro takes about 29.8 s/step for SAS and 24.1 s/step for DSpark. Dynamic partition training produces one checkpoint for both π=(2,2,3) and π=(2,5), avoiding separate training for each schedule. The extra training cost can be spread over many serving requests.
The serving gains matter because they recur over the target model's deployment lifetime. Our experiments evaluate end-to-end throughput for a real 1T LLM deployment on 32 H20 GPUs. At fixed workload and GPU pricing, higher throughput directly reduces GPU time per served token. Therefore, a persistent gain approaching 8% can still have substantial absolute value when applied to a large volume of served tokens.