- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper identify a meaningful problem in the training of diffusion-style draft model in speculative decoding, and come up with the idea that the weight of training loss on different position could be varies based on its accept rates. Based on this idea, the author come up with the EDR method and evaluate it, the evaluation shows improvement over baselines and other methods.
Critical strengths
- The paper identifies a real gap in how parallel and semi-autoregressive drafters are trained: the draft distribution at a position depends on where the decoding round starts, so existing block-local objectives do not directly optimize the number of decoding rounds.
- It formulates speculative decoding, conditioned on the output sequence, as a Markov reward process. This captures the dependence across decoding rounds and gives an exact expression for the expected number of rounds, with no additional hyperparameters.
- The same framework yields an exact offline evaluator of round counts, which allows paired comparison of drafters on shared target rollouts. This is useful independently of the training objective.
- The modeling and the proofs are presented clearly and in detail.
Critical weaknesses
- The contribution beyond the E2E objective is incremental. E2E already accounts for the multiplicative acceptance structure within a block; EDR adds the cross-round term. The paper does not show that this term matters in practice.
- The gain over the original checkpoints is confounded with additional training. The only controlled comparison is EDR against E2E. There is no baseline that continues training with each drafter's original objective for one more epoch on the same data, so the improvement over DSpark and DFly cannot be attributed to the objective.
Questions for authors
- If DFly's final training stage already used the E2E objective, is the E2E row for DFly equivalent to one more epoch of the original training? If not, can you provide comparsion to baselines with one more epoch?
4 comments
Appreciation and final question
Thank you for the detailed rebuttal and additional experiments.
My primary concern regarding the additional training confound has been addressed. I also appreciate the theoretical contribution of this work, it extends previous theoretical analyses of speculative decoding (e.g., Leviathan et al., 2023; Yin et al., 2024) to parallel and semi-AR drafters by explicitly modelling cross-round dependencies. I believe that EDR provides a theoretically more accurate objective than block-local optimization.
My final question is about the practical significance of this objective mismatch. Although EDR demonstrates consistent overall improvements, the gains remain modest (approximately 0.5% MAL on average), while requiring over 30% additional training time. One possible explanation is that E2E is already close to the global optimum in realistic LLM settings, making the benefit of cross-round optimization relatively limited. Besides experiments, could authors assess the severity of this mismatch in practice? Are there particular settings (e.g., larger block sizes, more restricted drafter architectures, or different target distributions) where EDR could yield substantially larger improvements?
- Leviathan et al. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
- Yin et al. (2024). A Theoretical Perspective for Speculative Decoding Algorithm. NeurIPS 2024. https://arxiv.org/abs/2411.00841
Thank you for your insightful questions.
Is E2E already close to the global optimum? We agree with this explanation for low-entropy tasks. On GSM8K and MATH, E2E and EDR stay within about 0.01 MAL of each other, which suggests that E2E is already close to the global optimum there. However, this is not the case for high-entropy tasks. To examine this, we continued both E2E and EDR finetuning on the Qwen3-4B/DSpark pair and report the average MAL over the three chat benchmarks (MT-Bench, Alpaca, and Arena-Hard).
| Training Effort | + E2E | + EDR | Gain of EDR over E2E |
|---|---|---|---|
| 1 epoch | 3.694 | 3.726 | +0.89% |
| More training steps | 3.738 | 3.789 | +1.36% |
The advantage of EDR over E2E for chat benchmarks grows with training while the performance differences for math benchmarks are about the same.
Measuring the mismatch in theory. Our framework provides a natural way to characterize this mismatch through a Bellman residual of a block-local optimal drafter. At a given round-start prefix, we define the Bellman residual as the maximum reduction in the expected number of remaining decoding rounds that can be achieved by deviating from this block-local choice for the current round only. Intuitively, a positive residual means that sacrificing some accepted length in the current round can steer decoding toward a more favorable next-round start.
These residuals therefore measure the local/global objective mismatch directly in units of expected decoding rounds. Occupancy-weighted residuals indicate where the practically relevant optimization headroom lies. The suboptimality criterion added in the revision (Proposition 2 and Corollary 1 in Appendix B.7) is the first-order version of this quantity.
We note that the residual is hard to estimate exactly for real LLMs, as it requires both the block-local optimal drafter and an optimization over the drafter class at every round-start prefix. We therefore use it as a conceptual characterization that identifies where the mismatch comes from.
Settings with a larger mismatch. This characterization also suggests when the mismatch should be larger. The key factor is the amount of useful target dependence that is unavailable to the drafter within the current block.
- Block size. Larger block sizes can expose more such unavailable dependencies. A draft position deeper in the block is separated from the committed prefix by more tokens that the drafter has not observed, as a first-order drafter sees only the preceding draft token. The part of the target context that is hidden from the drafter therefore grows with the block size. In addition, a larger block offers a wider range of continuation values to trade against the accepted length of the current round.
- Drafter architecture. More restricted drafters should generally leave more room for cross-round optimization. For example, fully parallel drafters cannot condition later draft positions on earlier sampled tokens in the same block, whereas first-order semi-AR drafters can recover part of this information.
- Target distribution. Targets with stronger dependencies that are hidden from the drafter, and with more heterogeneous continuation difficulty across round starts, should exhibit larger Bellman residuals. Chat-style distributions may be one practical setting of this kind, which is consistent with the larger gains of EDR on the chat benchmarks.
Training cost.
- For a single training step, EDR is indeed more expensive than block local surrogates. However, the occupancy weighting also concentrates the optimization effort on the anchors that have the greatest effect on MAL, rather than treating all anchors uniformly. Empirically, this improves training efficiency. As shown below for the Qwen3-4B/DSpark pair, 0.5 epoch of EDR (18 hours) already reaches a higher MAL than 1 epoch of E2E (26 hours) when pooled over all nine benchmarks.
| Drafter | Training time | MAL (all nine benchmarks) |
|---|---|---|
| + E2E (1 epoch) | 26 hours | 4.869 |
| + EDR (0.5 epoch) | 18 hours | 4.881 |
Thus, even after accounting for the higher per-step cost, EDR reaches a better overall MAL with less total training time. This suggests that the additional computation should be viewed in terms of optimization efficiency rather than per-step cost alone. We also expect this advantage to be larger when starting from a less-optimized drafter, since correct credit assignment across round-start states matters more earlier in training.
- Training cost versus serving benefit. Finally, we believe the additional offline training cost is well justified in practice. Drafter training is a one-time cost, whereas inference cost is incurred repeatedly over the entire lifetime of a deployed model. Even a modest but systematic reduction in decoding rounds is therefore amortized over a very large number of inference requests, as training-time overhead and serving-time savings operate on very different scales.
Thanks for Author's final explaination
Thanks for your in-depth explanation and I agree that high entropy tasks are currently one limitation of SD, I believe you can emphases it more.
I am happy to raise my recommendation.
Question 1
DFly. Yes. DFly was originally trained with the E2E objective, so the E2E row for DFly is exactly the released checkpoint trained for one additional epoch with its original objective, using the same data and training budget as EDR. The DFly comparison therefore directly controls for the effect of additional finetuning.
DSpark. The original DSpark objective is 0.1CE+0.9L1, which differs from E2E. In the revised paper, we have trained a baseline with this objective for one additional epoch under the same data and training budget, and added the results to the revised experimental table (Table 1), as shown below. This baseline performs worse than E2E on all nine benchmarks, while EDR remains the strongest objective. These results show that the improvement is attributable to the objective rather than to additional training.
Weakness 1
Cross-round optimization matters. We agree that E2E already captures the within-block multiplicative acceptance structure, and this is precisely why we use E2E as the controlled baseline for the cross-round term. EDR and E2E share the same drafter architecture, training data, and training budget. Across two drafter architectures (DSpark and DFly), two target models (Qwen3-4B and Qwen3-8B), and nine benchmarks spanning math, code, and chat, EDR matches or exceeds E2E in all 18 drafter–benchmark combinations. Therefore, we believe that an improvement that holds in the same direction across drafter architectures, target models, and task domains, shows that the cross-round term matters in practice.
Gradient decomposition. To further examine whether the cross-round term matters in practice, we also performed a gradient decomposition analysis on the Qwen3-4B/DSpark pair. Let gEDR=glocal+gdownstream, where glocal is the occupancy-weighted gradient of the local rejection cost and gdownstream is the gradient of the downstream value-gap term (Theorem 2). Across 10 optimizer steps, the relative norm of the downstream component is ∥gdownstream∥/∥gEDR∥=0.764±0.036. Moreover, the local-only gradient has a cosine similarity of 0.812±0.015 with the full EDR gradient, corresponding to an average angular deviation of approximately 36∘, while the local and downstream components are strongly anti-aligned, with a cosine similarity of −0.645±0.020. These results show that the downstream term substantially corrects the optimization direction.
Weakness 2
Confounding with additional training. Please see our response to Question 1 regarding the additional baseline.
Closing Remarks
Contributions beyond the empirical results. We would also like to highlight that the contributions of this paper go beyond the empirical comparison with E2E. First, we formulate speculative decoding, conditioned on the output sequence, as a Markov reward process. Second, the same framework yields an exact offline estimator of the expected number of decoding rounds, which enables paired comparisons of drafters on shared target rollouts. It evaluates all nine benchmarks in approximately 11 minutes, compared with more than 60 minutes for online speculative decoding at a comparable statistical error. As the reviewer noted in the strengths, this estimator is useful independently of the training objective.
Request for re-evaluation. We believe we have addressed questions raised by the reviewer. The gain over the original checkpoints is not confounded with additional training, since the DFly E2E row corresponds to one additional epoch of DFly's original objective and we have added a one-additional-epoch baseline with the original DSpark objective. Together with the theoretical framework and the offline estimator, whose clarity and usefulness the reviewer acknowledged, we would greatly appreciate it if the reviewer could reconsider the Informativeness and Soundness ratings in light of the full set of contributions.