acceptodds
Under review as a conference paper at ICLR 2027

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

Abstract

Speculative decoding accelerates large language model inference by using a low-cost draft model to propose tokens that the full-size target model verifies in parallel. Parallel and semi-autoregressive (semi-AR) drafters improve drafting efficiency by proposing an entire block in a single forward passes, but training them raises a new difficulty: the draft distribution for a given position depends on where the decoding round starts, and where rounds start depends on how many tokens earlier rounds accepted. Existing training objectives typically rely on block-local surrogates that ignore this cross-round coupling, and therefore do not directly optimize the global decoding efficiency. In this work, we develop a theoretical framework for training and evaluating these drafters by representing speculative decoding as a Markov reward process. This formulation yields the Expected Decoding Rounds (EDR) objective, which weights local rejection costs by state occupancies and exactly equals the expected number of decoding rounds. Unlike prior surrogate objectives, EDR introduces no auxiliary hyperparameters. We then derive an exact temporal-difference gradient that supports unbiased stochastic optimization from target-model rollouts. The same framework also yields an exact offline evaluator for round counts, enabling paired drafter comparisons on shared target rollouts without running speculative decoding. Finetuning two state-of-the-art drafters, DSpark and DFly, with EDR consistently improves mean accepted length and outperforms existing training objectives across nine benchmarks spanning math reasoning, code generation, and chat.

Primary area:
learning theory
Keywords:
Large language models, Speculative decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uDc3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper identify a meaningful problem in the training of diffusion-style draft model in speculative decoding, and come up with the idea that the weight of training loss on different position could be varies based on its accept rates. Based on this idea, the author come up with the EDR method and evaluate it, the evaluation shows improvement over baselines and other methods.

Critical strengths
  • The paper identifies a real gap in how parallel and semi-autoregressive drafters are trained: the draft distribution at a position depends on where the decoding round starts, so existing block-local objectives do not directly optimize the number of decoding rounds.
  • It formulates speculative decoding, conditioned on the output sequence, as a Markov reward process. This captures the dependence across decoding rounds and gives an exact expression for the expected number of rounds, with no additional hyperparameters.
  • The same framework yields an exact offline evaluator of round counts, which allows paired comparison of drafters on shared target rollouts. This is useful independently of the training objective.
  • The modeling and the proofs are presented clearly and in detail.
Critical weaknesses
  • The contribution beyond the E2E objective is incremental. E2E already accounts for the multiplicative acceptance structure within a block; EDR adds the cross-round term. The paper does not show that this term matters in practice.
  • The gain over the original checkpoints is confounded with additional training. The only controlled comparison is EDR against E2E. There is no baseline that continues training with each drafter's original objective for one more epoch on the same data, so the improvement over DSpark and DFly cannot be attributed to the objective.
Questions for authors
  • If DFly's final training stage already used the E2E objective, is the E2E row for DFly equivalent to one more epoch of the original training? If not, can you provide comparsion to baselines with one more epoch?
4 comments

Question 1

DFly. Yes. DFly was originally trained with the E2E objective, so the E2E row for DFly is exactly the released checkpoint trained for one additional epoch with its original objective, using the same data and training budget as EDR. The DFly comparison therefore directly controls for the effect of additional finetuning.

DSpark. The original DSpark objective is 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1}, which differs from E2E. In the revised paper, we have trained a baseline with this objective for one additional epoch under the same data and training budget, and added the results to the revised experimental table (Table 1), as shown below. This baseline performs worse than E2E on all nine benchmarks, while EDR remains the strongest objective. These results show that the improvement is attributable to the objective rather than to additional training.

DrafterGSM8KMATHAIME25MBPPHumanEvalLCBMT-BenchAlpacaArena-Hard
DSpark6.236.125.465.285.525.463.703.623.59
+ 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1} (original)6.276.135.465.325.565.513.723.643.61
+ LK6.256.115.435.315.555.493.703.633.59
+ D-PACE6.266.155.505.255.525.463.673.593.57
+ E2E6.326.205.575.345.585.543.753.663.67
+ EDR (Ours)6.326.205.595.345.595.563.783.703.70

Weakness 1

Cross-round optimization matters. We agree that E2E already captures the within-block multiplicative acceptance structure, and this is precisely why we use E2E as the controlled baseline for the cross-round term. EDR and E2E share the same drafter architecture, training data, and training budget. Across two drafter architectures (DSpark and DFly), two target models (Qwen3-4B and Qwen3-8B), and nine benchmarks spanning math, code, and chat, EDR matches or exceeds E2E in all 18 drafter–benchmark combinations. Therefore, we believe that an improvement that holds in the same direction across drafter architectures, target models, and task domains, shows that the cross-round term matters in practice.

Gradient decomposition. To further examine whether the cross-round term matters in practice, we also performed a gradient decomposition analysis on the Qwen3-4B/DSpark pair. Let gEDR=glocal+gdownstream,g_{\mathrm{EDR}} = g_{\mathrm{local}} + g_{\mathrm{downstream}}, where glocalg_{\mathrm{local}} is the occupancy-weighted gradient of the local rejection cost and gdownstreamg_{\mathrm{downstream}} is the gradient of the downstream value-gap term (Theorem 2). Across 10 optimizer steps, the relative norm of the downstream component is ∥gdownstream∥/∥gEDR∥=0.764±0.036.\lVert g_{\mathrm{downstream}} \rVert / \lVert g_{\mathrm{EDR}} \rVert = 0.764 \pm 0.036. Moreover, the local-only gradient has a cosine similarity of 0.812±0.0150.812 \pm 0.015 with the full EDR gradient, corresponding to an average angular deviation of approximately 36∘36^\circ, while the local and downstream components are strongly anti-aligned, with a cosine similarity of −0.645±0.020-0.645 \pm 0.020. These results show that the downstream term substantially corrects the optimization direction.

Weakness 2

Confounding with additional training. Please see our response to Question 1 regarding the additional baseline.

Closing Remarks

Contributions beyond the empirical results. We would also like to highlight that the contributions of this paper go beyond the empirical comparison with E2E. First, we formulate speculative decoding, conditioned on the output sequence, as a Markov reward process. Second, the same framework yields an exact offline estimator of the expected number of decoding rounds, which enables paired comparisons of drafters on shared target rollouts. It evaluates all nine benchmarks in approximately 11 minutes, compared with more than 60 minutes for online speculative decoding at a comparable statistical error. As the reviewer noted in the strengths, this estimator is useful independently of the training objective.

Request for re-evaluation. We believe we have addressed questions raised by the reviewer. The gain over the original checkpoints is not confounded with additional training, since the DFly E2E row corresponds to one additional epoch of DFly's original objective and we have added a one-additional-epoch baseline with the original DSpark objective. Together with the theoretical framework and the offline estimator, whose clarity and usefulness the reviewer acknowledged, we would greatly appreciate it if the reviewer could reconsider the Informativeness and Soundness ratings in light of the full set of contributions.

Reviewer_uDc3OpenReview ↗

Appreciation and final question

Thank you for the detailed rebuttal and additional experiments.

My primary concern regarding the additional training confound has been addressed. I also appreciate the theoretical contribution of this work, it extends previous theoretical analyses of speculative decoding (e.g., Leviathan et al., 2023; Yin et al., 2024) to parallel and semi-AR drafters by explicitly modelling cross-round dependencies. I believe that EDR provides a theoretically more accurate objective than block-local optimization.

My final question is about the practical significance of this objective mismatch. Although EDR demonstrates consistent overall improvements, the gains remain modest (approximately 0.5% MAL on average), while requiring over 30% additional training time. One possible explanation is that E2E is already close to the global optimum in realistic LLM settings, making the benefit of cross-round optimization relatively limited. Besides experiments, could authors assess the severity of this mismatch in practice? Are there particular settings (e.g., larger block sizes, more restricted drafter architectures, or different target distributions) where EDR could yield substantially larger improvements?

Thank you for your insightful questions.

Is E2E already close to the global optimum? We agree with this explanation for low-entropy tasks. On GSM8K and MATH, E2E and EDR stay within about 0.01 MAL of each other, which suggests that E2E is already close to the global optimum there. However, this is not the case for high-entropy tasks. To examine this, we continued both E2E and EDR finetuning on the Qwen3-4B/DSpark pair and report the average MAL over the three chat benchmarks (MT-Bench, Alpaca, and Arena-Hard).

Training Effort+ E2E+ EDRGain of EDR over E2E
1 epoch3.6943.726+0.89%
More training steps3.7383.789+1.36%

The advantage of EDR over E2E for chat benchmarks grows with training while the performance differences for math benchmarks are about the same.

Measuring the mismatch in theory. Our framework provides a natural way to characterize this mismatch through a Bellman residual of a block-local optimal drafter. At a given round-start prefix, we define the Bellman residual as the maximum reduction in the expected number of remaining decoding rounds that can be achieved by deviating from this block-local choice for the current round only. Intuitively, a positive residual means that sacrificing some accepted length in the current round can steer decoding toward a more favorable next-round start.

These residuals therefore measure the local/global objective mismatch directly in units of expected decoding rounds. Occupancy-weighted residuals indicate where the practically relevant optimization headroom lies. The suboptimality criterion added in the revision (Proposition 2 and Corollary 1 in Appendix B.7) is the first-order version of this quantity.

We note that the residual is hard to estimate exactly for real LLMs, as it requires both the block-local optimal drafter and an optimization over the drafter class at every round-start prefix. We therefore use it as a conceptual characterization that identifies where the mismatch comes from.

Settings with a larger mismatch. This characterization also suggests when the mismatch should be larger. The key factor is the amount of useful target dependence that is unavailable to the drafter within the current block.

  • Block size. Larger block sizes can expose more such unavailable dependencies. A draft position deeper in the block is separated from the committed prefix by more tokens that the drafter has not observed, as a first-order drafter sees only the preceding draft token. The part of the target context that is hidden from the drafter therefore grows with the block size. In addition, a larger block offers a wider range of continuation values to trade against the accepted length of the current round.
  • Drafter architecture. More restricted drafters should generally leave more room for cross-round optimization. For example, fully parallel drafters cannot condition later draft positions on earlier sampled tokens in the same block, whereas first-order semi-AR drafters can recover part of this information.
  • Target distribution. Targets with stronger dependencies that are hidden from the drafter, and with more heterogeneous continuation difficulty across round starts, should exhibit larger Bellman residuals. Chat-style distributions may be one practical setting of this kind, which is consistent with the larger gains of EDR on the chat benchmarks.

Training cost.

  • For a single training step, EDR is indeed more expensive than block local surrogates. However, the occupancy weighting also concentrates the optimization effort on the anchors that have the greatest effect on MAL, rather than treating all anchors uniformly. Empirically, this improves training efficiency. As shown below for the Qwen3-4B/DSpark pair, 0.5 epoch of EDR (18 hours) already reaches a higher MAL than 1 epoch of E2E (26 hours) when pooled over all nine benchmarks.
DrafterTraining timeMAL (all nine benchmarks)
+ E2E (1 epoch)26 hours4.869
+ EDR (0.5 epoch)18 hours4.881

Thus, even after accounting for the higher per-step cost, EDR reaches a better overall MAL with less total training time. This suggests that the additional computation should be viewed in terms of optimization efficiency rather than per-step cost alone. We also expect this advantage to be larger when starting from a less-optimized drafter, since correct credit assignment across round-start states matters more earlier in training.

  • Training cost versus serving benefit. Finally, we believe the additional offline training cost is well justified in practice. Drafter training is a one-time cost, whereas inference cost is incurred repeatedly over the entire lifetime of a deployed model. Even a modest but systematic reduction in decoding rounds is therefore amortized over a very large number of inference requests, as training-time overhead and serving-time savings operate on very different scales.
Reviewer_uDc3OpenReview ↗

Thanks for Author's final explaination

Thanks for your in-depth explanation and I agree that high entropy tasks are currently one limitation of SD, I believe you can emphases it more.

I am happy to raise my recommendation.

Reviewer_S37aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies training objectives for parallel and semi-autoregressive speculative drafters, whose predictions depend on where each decoding round begins. It conditions on a target-generated output sequence and represents verification as a Markov reward process over round starts and token positions. This yields an Expected Decoding Rounds (EDR) objective, an occupancy-weighted temporal-difference gradient, and an offline estimator of expected round counts. A constructed example demonstrates that optimizing accepted length independently within each round can be globally suboptimal. One-epoch finetuning of DSpark and DFly with Qwen3 targets produces small but consistent improvements in offline mean accepted length across nine benchmarks.

Critical strengths
  1. The objective addresses a real distinction between local acceptance and global decoding efficiency. The state construction in Section 3.1 makes the dependence on round starts explicit, and Theorems 1-2 connect this dependence to an exact expected-round objective and a usable gradient. The EOS treatment is carefully accounted for. Theorem 3 supplies a concrete counterexample to block-local optimality rather than relying solely on intuition. Together, these provide a useful theoretical foundation for training this class of drafters.

  2. The same formulation provides a useful evaluation method and controlled empirical evidence. Section 3.4 integrates verification randomness conditional on target rollouts, permitting paired comparisons on identical outputs. Section 4 holds architectures, training data, and update counts fixed when comparing EDR with E2E. Table 1 shows nonnegative gains across both drafter configurations, and Appendix D.2 reports Holm-corrected significance for all nine DFly comparisons. This supports the relevance of the theoretical objective beyond the constructed example.

Critical weaknesses
  1. The practical value of the additional optimization cost remains unclear. The DFly improvements over E2E in Table 3 are approximately 0.005-0.040 tokens per round, or about 0.08%-1.24% relative MAL. These are statistically resolved under the reported fixed-prompt analysis, but the paper does not measure whether they yield meaningful end-to-end latency reductions. The conclusion acknowledges that EDR is more expensive, and Appendix C.2 still computes the full occupancy and value recursions even when only sampled anchors are differentiated. Without training-time/memory measurements and an online decoding check, it is difficult to assess the tradeoff between a better objective and the cost of obtaining these small gains.

  2. The empirical study does not isolate which part of EDR produces the improvement. The comparison changes the local EOS-aware cost, occupancy weighting, downstream gradient term, and anchor-selection procedure together. There is no controlled ablation of the continuation-value term or comparison with an EOS-aware local objective under matched anchor sampling. All experiments also use block size seven and one target family. This leaves the central empirical claim about the benefit of cross-round optimization less directly established than the theoretical result.

Questions for authors
  1. What are the training time and peak-memory overheads of EDR relative to E2E, including the full no-gradient recursions? On the same trained checkpoints, do measured online round counts agree with the offline evaluator, and how large is the end-to-end latency change?
  2. With the local EOS treatment and anchor sampling held fixed, how much of the gain remains when the continuation-value term is removed? This would clarify whether the observed improvement is specifically due to cross-round optimization.
  3. Do the small improvements persist across independent finetuning seeds? Appendix D.2 addresses rollout uncertainty conditional on fixed prompts and checkpoints; variation from training would materially affect the interpretation of the empirical gains.
4 comments
AuthorsOpenReview ↗

Respond to Questions 1

Question 1

Training time and peak memory. We have added reports of training time, peak memory, and evaluation cost (Appendix D.4, Table 7). For the Qwen3-4B/DSpark pair on a single NVIDIA H200, one epoch of finetuning on the first-round user prompts of Open-PerfectBlend takes approximately 26 hours with E2E and 36 hours with EDR. Peak memory usage is approximately 100 GB for E2E and 120 GB for EDR. The occupancy and value-function dynamic programs themselves, including the full no-gradient recursions, add negligible wall-clock overhead, since they are computed on the CPU and overlapped with GPU computation. The main additional cost of EDR is an extra drafter forward pass over the candidate anchors to compute the acceptance probabilities required by the occupancy recursion.

Training cost versus serving benefit. We believe this additional training cost is worthwhile in practice. Drafter training is a one-time offline cost, whereas inference cost is incurred repeatedly over the lifetime of the deployed model, so even a modest reduction in decoding rounds is amortized over a very large number of inference requests. In other words, training time and serving time operate on very different scales.

Online vs. offline round counts. We have also validated the offline evaluator against online speculative decoding. Over 10,000 online runs, the measured round count differs from our offline estimate by only 0.04%, with a 99% bootstrap confidence interval of [-0.145%, 0.222%]. This confirms that the offline estimator accurately predicts the actual number of decoding rounds.

End-to-end latency. First, we would like to note that, as observed in the literature, in the low-concurrency (memory-bound) regime the end-to-end latency of online speculative decoding scales linearly with the number of decoding rounds [1, 2]. The drafter architecture, block size, and verification procedure are identical across training objectives, so every decoding round incurs essentially the same execution cost, regardless of how many tokens are eventually accepted. DistillSpec [1] notes that each speculative decoding step takes a constant amount of time, so the wall-clock time scales linearly with the number of steps. The decoding latency data in the LK losses paper [2] also show that the Pearson correlation between the number of decoding rounds and the end-to-end latency is over 0.995 for dense models, confirming that the number of rounds is a reliable proxy for end-to-end latency.

Second, we report MAL because paired evaluation of round counts on identical target rollouts has substantially lower variance than wall-clock latency measurements, which are affected by system noise.

[1] Zhou et al. DistillSpec: Improving Speculative Decoding via Knowledge Distillation. [2] Samarin et al. LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding.

AuthorsOpenReview ↗

Respond to Questions (Part 2)

Question 2

Removing the continuation-value term. We fully agree that it would strengthen the paper to show the benefit of modeling cross-round dependence. However, we note that removing the continuation-value term (i.e., the downstream value-gap term in Theorem 2) alone does not fully ablate the cross-round effect, since the occupancy weights are still computed across rounds.

Regarding the concern about the lack of ablations, we would also like to clarify that the occupancy weights, the downstream value-gap term, and the local rejection cost are not independently introduced design modules. In particular, the occupancy weights and the downstream term both arise directly from differentiating the exact EDR objective. Removing either term therefore breaks the exact EDR gradient identity rather than cleanly removing an independent mechanism.

To illustrate this point directly, and following the suggestion in Weakness 2, we trained a variant of EDR under anchor sampling matched to the E2E baseline (uniform anchors), while keeping the local rejection cost and the within-block weights unchanged. As reported in the table below, this variant shows a clear deterioration in MAL.

LossGSM8KMATHAIME25MBPPHumanEvalLCBMT-BenchAlpacaArena-Hard
EDR6.326.205.595.345.595.563.783.703.70
EDR with uniform anchors5.244.383.562.853.422.541.791.651.66

With uniform anchors, the occupancy weights are dropped and the sum of local rejection costs no longer equals the expected number of decoding rounds. It penalizes whether a rejection occurs in a round but not where it occurs, since in EDR the cost of the rejection position is carried by the occupancy of the next round start. The variant therefore optimizes a different objective, which explains why it falls below even the initial checkpoint. This result supports the view that the components form the EDR objective as a whole and that capturing cross-round dependence correctly is necessary.

Gradient decomposition. To quantify the role of the downstream term, we also performed a gradient decomposition analysis on the Qwen3-4B/DSpark pair. Let gEDR=glocal+gdownstream,g_{\mathrm{EDR}} = g_{\mathrm{local}} + g_{\mathrm{downstream}}, where glocalg_{\mathrm{local}} is the occupancy-weighted gradient of the local rejection cost and gdownstreamg_{\mathrm{downstream}} is the gradient of the downstream value-gap term (Theorem 2). Across 10 optimizer steps, the relative norm of the downstream component is ∥gdownstream∥/∥gEDR∥=0.764±0.036.\lVert g_{\mathrm{downstream}} \rVert / \lVert g_{\mathrm{EDR}} \rVert = 0.764 \pm 0.036. Moreover, the local-only gradient has a cosine similarity of 0.812±0.0150.812 \pm 0.015 with the full EDR gradient, corresponding to an average angular deviation of approximately 36∘36^\circ, while the local and downstream components are strongly anti-aligned, with a cosine similarity of −0.645±0.020-0.645 \pm 0.020. These results show that the downstream term substantially corrects the optimization direction.

Question 3

Robustness across training seeds. To assess the robustness of the EDR improvement, we repeated EDR finetuning with a different training seed and observed no meaningful difference between the two seeds.

SeedGSM8KMATHAIME25MBPPHumanEvalLCBMT-BenchAlpacaArena-Hard
16.326.205.595.345.595.563.783.703.70
26.336.205.595.355.595.563.783.703.70
AuthorsOpenReview ↗

Respond to Weakness

Weakness 1

Practical value of the additional cost. Please see our response to Question 1.

Weakness 2

Isolating the source of improvement. Please see our response to Question 2.

Block size and target family. DSpark and DFly are state-of-the-art self-speculative drafters, so we finetune these well-trained released checkpoints to show the advantage of EDR. The main experiments of both DSpark and AngelSpec use Qwen3 target models with block size 7. We adopt the same configuration, which keeps our setting consistent with the released drafter checkpoints that we finetune and with the results reported in these works.

In addition, the EDR formulation itself does not depend on a specific block size or target model as the objective and its gradient (Theorems 1 and 2) are exact for any block size and any target. We leave experiments with other block sizes and target families to future work.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer,

With the discussion deadline approaching, we would like to gently check whether our responses have resolved your concerns or if any further clarification is needed.

Thank you again for your time and effort!

Reviewer_tj8nOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the mismatch between local training objectives for parallel/semi-auto-regressive drafters and their actual global decoding efficiency, which stems from cross-round dependencies. The authors propose to train drafters by directly minimizing the expected number of decoding rounds. They formulate speculative decoding, conditioned on a given output sequence, as a Markov reward process (MRP), introduce the EDR objective, and prove that it equals the expected number of decoding rounds.

Critical strengths
  1. The MRP formulation and the EDR identity are theoretically original. The paper casts speculative decoding for a fixed output sequence as an MRP over states (n,t)(n,t), expresses the expected number of decoding rounds exactly via occupancies and local rejection costs, and proves it. This explicitly captures cross-round dependencies for parallel/semi-AR drafters and meaningfully complements existing block-local objectives.

  2. Theorem 3 provides existence evidence that local optimality can be globally suboptimal. The paper constructs a first-order semi-AR counterexample showing that a block-local optimal drafter maximizing expected accepted length in every round can be strictly outperformed by another first-order drafter, with a MAL gap lower bound. This supports the claim that the next-round start effect cannot be ignored.

Critical weaknesses
  1. The theoretical scope is limited, and several definitions and statements need clarification. Theorem 3 is a constructive existence proof restricted to first-order semi-AR drafters and the specific local objective of maximizing expected accepted length per round; it does not generally imply that EDR dominates all block-local objectives. The uniqueness of “any locally optimal q^\hat{q},” the definition of accepted length (whether correction/bonus tokens are counted), and the first-order legality of q∗q^* require further formalization.

  2. The paper mainly compares against the E2E block-local objective and lacks strong drafter-training baselines such as D-PACE, LK Losses, and DFlash. Most ΔMAL values in Table 1 are below 0.04, and the significance analysis covers only one target–drafter pair (Qwen3-8B + DFly). There is no mechanism-level ablation (e.g., removing occupancy weights, removing the value gap, or using only the local cost) and no diagnostic of how the next-round start/rejection-position distribution changes after training. The paper also does not systematically report EDR training/evaluation overhead, long-sequence behavior, or scalability.

Questions for authors
  1. Please clarify the specific motivation for choosing E2E as the primary baseline: which class of block-local objectives does E2E represent, and does it cover the “maximize expected accepted length per round” objective targeted by Theorem 3? Can comparisons against E2E alone sufficiently validate the paper’s central claims that EDR outperforms existing block-local objectives and that cross-round modeling is necessary? If not, please add strong baselines such as D-PACE, LK Losses, and DFlash under the same target–drafter pairs and training budget, and explain how these baselines map to the theoretical analysis.

  2. Theorem 3 is currently a constructive existence proof: it applies only to a specially constructed target distribution, first-order semi-AR drafters, and the specific block-local objective of maximizing expected accepted length per round. Please clarify: (a) whether the theorem can be extended to other block-local objectives (e.g., E2E TV, D-PACE, LK Losses), or whether it applies only to the constructed local objective; (b) whether there exists a general construction or non-constructive condition, rather than relying on this special case; and (c) how strongly this theorem supports the paper’s main claim that EDR outperforms existing block-local objectives. If the theorem does not cover other baselines, please state its precise role in the paper and discuss how this limitation affects its theoretical value.

4 comments
AuthorsOpenReview ↗

Respond to Questions

Question 1

Motivation for choosing E2E. We chose E2E as our primary baseline for two reasons. Theoretically, E2E is designed as a surrogate for the accepted length under a fixed context, and it represents training objectives that aim to maximize the accepted length per round [1]. It is therefore directly covered by Theorem 3 since the block-local optimal drafter q^\hat{q} in Theorem 3 is a minimizer of the E2E loss. Practically, among D-PACE, LK losses, and the original DSpark objective 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1}, E2E is the strongest, both in our experiments and in the comparisons reported in AngelSpec [2].

[1] Li et al. Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling. [2] Liu et al. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding.

Additional baselines. We agree that additional comparisons are helpful, and we report below the MAL of D-PACE, LK, and 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1} on the Qwen3-4B/DSpark pair. Additional baselines are also added to the revised experimental table in the paper (Table 1).

DrafterGSM8KMATHAIME25MBPPHumanEvalLCBMT-BenchAlpacaArena-Hard
DSpark6.236.125.465.285.525.463.703.623.59
+ 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1} (original)6.276.135.465.325.565.513.723.643.61
+ LK6.256.115.435.315.555.493.703.633.59
+ D-PACE6.266.155.505.255.525.463.673.593.57
+ E2E6.326.205.575.345.585.543.753.663.67
+ EDR (Ours)6.326.205.595.345.595.563.783.703.70

In our experiments, E2E outperforms all three of these objectives.

DFlash. The DFlash training objective is a cross-entropy loss with fixed, exponentially decaying position weights. D-PACE replaces this fixed schedule with adaptive weights derived from a differentiable accepted-length surrogate, so we include D-PACE as the representative of this family of position-weighted cross-entropy objectives.

Mapping to the theoretical analysis. E2E is directly covered by Theorem 3, as discussed above, while all other block-local objectives, including D-PACE, LK, 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1}, and the DFlash objective, are covered by an extension of Theorem 3 in the revised paper (Theorem 4 in Appendix B.6); please see (a) in our response to Question 2.

Question 2

(a) Extension to other block-local objectives. Theorem 3 can be extended to any fixed block-local objective, establishing the suboptimality of every such objective. In the revised paper (Appendix B.6), we first give a formal definition of a block-local objective (Definition 2) and then show that, for any fixed block-local objective with any fixed tie-breaking rule among its minimizers, there exists a target distribution under which that objective is suboptimal, with a MAL ratio of at least 9/89/8 between the optimal drafter and the block-local optimal one (Theorem 4).

(b) Beyond the special construction. First, we would like to point out that the suboptimality of block-local objectives does not hinge on one specific target distribution. It is driven by a general mechanism. Whenever the target has a dependence between earlier and later tokens, within a block or across blocks, that the first-order drafter cannot observe at the later position, the block-local optimal drafter commits to the choice that maximizes the accepted length of the current round, while the choice that minimizes the total number of rounds also depends on where the next round starts. The construction in Theorem 3 is a minimal instance that isolates this mechanism.

Second, we have updated the paper to include a general criterion for the suboptimality of any fixed block-local objective (Proposition 2 and Corollary 1 in Appendix B.7). It applies to an arbitrary target distribution. The minimizer of a block-local objective is suboptimal whenever EDR admits a feasible descent direction at that minimizer.

(c) Coverage of existing objectives. E2E is directly covered by Theorem 3, and all other block-local objectives, including D-PACE, LK, 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1}, and the DFlash objective, are covered by the newly added Theorem 4 discussed in (a), which serves as an extension of Theorem 3.

Role of Theorem 3. Theorem 3, together with the newly added Theorem 4, serves as the theoretical justification for modeling cross-round dependence. They show that optimality under a fixed block-local objective does not guarantee optimality in the expected number of decoding rounds, so block-local objectives can be misaligned with the global goal of maximizing MAL.

AuthorsOpenReview ↗

Respond to weakness (Part 1)

Weakness 1

Scope of the first-order assumption. The first-order assumption covers the current state-of-the-art parallel and semi-AR drafter architectures, including DFlash 1 and 2, DSpark, and DFly (parallel drafters such as DFlash are the special case of first-order drafters that does not condition on any within-block draft). The setting of Theorem 3 is motivated by these drafter architectures used in practice, and we believe the first-order assumption faithfully describes practical scenarios.

Scope of Theorem 3. We have generalized Theorem 3 to cover any fixed block-local objective (Theorem 4 in Appendix B.6); please see our response to Question 2.

Uniqueness of q^\hat{q} and first-order legality of q∗q^{*}. We have updated the paper to prove the uniqueness of q^\hat{q} and to formally prove that q∗q^{*} is indeed a first-order drafter (Appendix B.5).

Definition of accepted length. We have restated the definition of accepted length explicitly and emphasized that correction and bonus tokens are counted (Appendix B.5).

Weakness 2

Additional baselines. We have added comparisons with D-PACE, LK, and 0.1 CE+0.9 L10.1\,\mathrm{CE} + 0.9\,\mathrm{L1} (Table 1); please see our response to Question 1.

Significance analysis. We have added a significance analysis for the Qwen3-4B/DSpark pair (Appendix D.2, Table 4).

Mechanism-level ablation. First, we would like to clarify that the occupancy weights, the downstream value-gap term, and the local rejection cost are not independently introduced design modules. In particular, the occupancy weights and the downstream term both arise directly from differentiating the exact EDR objective. Removing either term therefore breaks the exact EDR gradient identity rather than cleanly removing an independent mechanism.

Second, we agree that an ablation illustrates this point directly. Following the suggestion, we trained a variant of EDR that removes the occupancy weights while keeping the local rejection cost and the within-block weights unchanged. As reported in the table below, this variant shows a clear deterioration in MAL.

LossGSM8KMATHAIME25MBPPHumanEvalLCBMT-BenchAlpacaArena-Hard
EDR6.326.205.595.345.595.563.783.703.70
EDR w/o occupancy weights5.244.383.562.853.422.541.791.651.66

Without the occupancy weights, the sum of local rejection costs no longer equals the expected number of decoding rounds. It penalizes whether a rejection occurs in a round but not where it occurs, since in EDR the cost of the rejection position is carried by the occupancy of the next round start. The variant therefore optimizes a different objective, which explains why it falls below even the initial checkpoint. This result supports the view that the components form the EDR objective as a whole and that capturing cross-round dependence correctly is necessary.

Gradient decomposition. To further analyze the effect of the cross-round gradient, we performed a gradient decomposition analysis on the Qwen3-4B/DSpark pair. Let gEDR=glocal+gdownstream,g_{\mathrm{EDR}} = g_{\mathrm{local}} + g_{\mathrm{downstream}}, where glocalg_{\mathrm{local}} is the occupancy-weighted gradient of the local rejection cost and gdownstreamg_{\mathrm{downstream}} is the gradient of the downstream value-gap term (Theorem 2). Across 10 optimizer steps, the relative norm of the downstream component is ∥gdownstream∥/∥gEDR∥=0.764±0.036.\lVert g_{\mathrm{downstream}} \rVert / \lVert g_{\mathrm{EDR}} \rVert = 0.764 \pm 0.036. Moreover, the local-only gradient has a cosine similarity of 0.812±0.0150.812 \pm 0.015 with the full EDR gradient, while the local and downstream components are strongly anti-aligned, with a cosine similarity of −0.645±0.020-0.645 \pm 0.020. These results show that the downstream term substantially corrects the optimization direction.

AuthorsOpenReview ↗

Respond to weakness (Part 2)

Rejection-position diagnostic. We have also added an analysis (Appendix D.3, Tables 5 and 6) of how the rejection-position distribution changes before and after EDR training on the Qwen3-4B/DSpark pair (block size B=7B = 7), comparing the original DSpark checkpoint with the EDR-finetuned checkpoint. In each round, the accepted length k∈{1,…,8}k \in \{1, \dots, 8\} counts the accepted draft tokens plus the correction or bonus token. k≤7k \le 7 means that the first rejection occurs at draft position kk, and k=8k = 8 means that the whole block is accepted. Since the next round starts kk positions after the current round start, the distribution of kk characterizes both the rejection position and the next-round start. We compute these distributions exactly with our offline estimator on the same target rollouts.

Distribution of the accepted length kk , pooled over all nine benchmarks.

kkRejection positionDSpark (%)+ EDR (%)Δ (pp)
1Draft position 116.6115.83-0.78
2Draft position 213.1412.62-0.52
3Draft position 310.6710.33-0.34
4Draft position 48.618.49-0.12
5Draft position 57.007.03+0.03
6Draft position 65.926.08+0.16
7Draft position 75.185.46+0.28
8None (full block accepted)32.8734.16+1.29
Mean kk4.794.89+0.10

Early-rejection probability P(kk ≤ 2) and full-acceptance probability P(kk = 8) on each benchmark (DSpark → + EDR, in %).

BenchmarkP(kk ≤ 2)P(kk = 8)
GSM8K13.28 → 12.3657.77 → 58.97
MATH14.27 → 13.3755.00 → 56.25
AIME2520.17 → 18.8741.58 → 43.61
MBPP21.92 → 21.0738.07 → 38.51
HumanEval19.72 → 18.8843.35 → 43.99
LCB20.44 → 19.5042.46 → 43.79
MT-Bench42.02 → 40.8215.17 → 15.79
Alpaca42.15 → 41.0112.78 → 13.26
Arena-Hard47.53 → 45.8517.21 → 18.38
All29.75 → 28.4532.87 → 34.16

EDR shifts probability mass from early rejections to late rejections and full acceptance. Pooled over all benchmarks, the probability of each rejection position k≤4k \le 4 decreases, with the largest decrease at the first draft position (-0.78 pp), while the probability of each k≥5k \ge 5 increases, with the largest increase for fully accepted blocks (+1.29 pp).

Training and evaluation overhead. We now report this overhead explicitly in the revision (Appendix D.4, Table 7). For the Qwen3-4B/DSpark pair on a single NVIDIA H200, one epoch of finetuning on the first-round user prompts of Open-PerfectBlend takes approximately 26 hours with E2E and 36 hours with EDR. For evaluation, our offline estimator evaluates all nine benchmarks in approximately 11 minutes, whereas online speculative decoding requires more than 60 minutes to reach a comparable statistical error.

Long-sequence behavior and model-scale scalability. We would like to clarify that EDR does not rely on any short-horizon approximation. For a rollout of any length, the objective and its gradient are exact. Regarding scalability, the additional occupancy and value computations are independent of the target-model size, and the same training procedure is applied without modification to both Qwen3-4B/DSpark and Qwen3-8B/DFly. The additional cost therefore comes primarily from the extra drafter forward pass rather than from any component of the EDR algorithm that grows with the target model.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer,

With the discussion deadline approaching, we would like to gently check whether our responses have resolved your concerns or if any further clarification is needed.

Thank you again for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.