acceptodds
Under review as a conference paper at ICLR 2027

Multi-Draft On-Policy Distillation for Diffusion Drafters

Abstract

Speculative decoding is used to reduce the inference cost of language models. One-step diffusion drafters propose a block of multiple tokens in parallel, reducing the sequential cost of speculative decoding. On-policy distillation trains these drafters using verifier feedback on their own proposals. However, parallel drafting makes this feedback difficult to use: the drafter predicts each token without observing the preceding tokens sampled within the same block, while the verifier conditions its prediction on those tokens. Different sampled prefixes therefore give different verifier targets for the same drafter prediction. We show that distilling against these targets separately can suppress tokens that are plausible after some prefixes but unlikely after others, harming draft acceptance. We explain the effects of this mismatch and introduce Multi-Draft On-Policy Distillation (MDraft), a new parallel drafter distillation method which samples several draft blocks from the same context and aggregates the verifier's full-vocabulary conditionals across their prefixes. We use all the drafts to collect information from the verifier, allowing support from alternative prefixes to influence the training target. The resulting algorithm improves the acceptance length of a parallel drafter beyond the level of pre-training, and alleviates the effects of the distributional mismatch. MDraft changes only training, retaining the drafter's architecture and inference procedure, allowing the resulting drafter to be used as-is in place of the pretrained checkpoint. Experiments on a suite of mathematical, coding, and conversational tasks show longer accepted drafts than prior distillation methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, language models, distillation, diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7wuCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Multi-Draft On-Policy Distillation (MDraft) for training parallel diffusion drafters. It identifies conflicting teacher supervision caused by the drafter predicting tokens independently while the teacher conditions on sampled prefixes. MDraft aggregates teacher predictions across multiple drafts and blends them with individual-prefix predictions, improving acceptance length without changing the drafter architecture or inference procedure.

Critical strengths

The paper identifies a supervision mismatch in parallel drafter training

The paper provides some theoretical analysis explaining its effect on standard distillation objectives.

The evaluation shows some improved acceptance lengths on different benchmarks

Critical weaknesses

Narrow methodological scope. The paper focuses specifically on refining OPD for parallel diffusion drafters by aggregating teacher predictions across sampled prefixes, which represents a relatively narrow contribution.

Experimental attribution. The original DFlash checkpoint receives no additional training, and no continued-SFT baseline is provided. Furthermore, applying common optimization settings across different objectives does not establish that each baseline is comparably well tuned. Could you provide more ablations and hyperparameter sweeps for the competing objectives to isolate the benefit of the proposed method?

Limited evidence of general applicability. All experiments use a single Qwen3-4B target and the same DFlash drafter, with thinking disabled and a fixed block length. The evaluation does not establish whether the proposed improvement holds across model families, larger targets, reasoning configurations, or drafters that explicitly model dependencies between proposed tokens.

Modest gains over the strongest training baselines. In Table 1, the mean acceptance-length improvement over the strongest baseline is approximately 0.21%, 2.48%, and 3.09% at temperatures 0, 0.6, and 1, respectively. Moreover, Appendix D shows that AdaFlash with survival weighting slightly outperforms MDraft at temperature 0. These results indicate a relatively small incremental benefit over existing training objectives

Questions for authors

see weakness

3 comments

We thank the reviewer for their careful consideration of our work and for recognizing the conditioning mismatch that we identify, the accompanying analysis, and the improved acceptance lengths. We address the reviewer’s concerns below:

  • 1. Magnitude of the improvements and survival weighting (W4).

    MDraft delivers consistent gains under stochastic decoding and measurable inference speedups through a drop-in drafter checkpoint. In the main comparison in Table 1, MDraft ranks first in 15 of the 18 benchmark–temperature settings and second in the remaining three. It leads on all six benchmarks at both T=0.6T=0.6 and T=1T=1, improving mean acceptance length over pretrained DFlash by 7.3% and 11.9%, respectively. This consistency across tasks is a central practical benefit of the method.

    The gains persist with matched weighting: Table 6 in Appendix D directly addresses whether survival weighting accounts for the improvement. At T=1T=1, MDraft outperforms AdaFlash by 6.0% with survival weights for both methods (5.903 versus 5.568), and by 5.2% with uniform weights for both (5.836 versus 5.546). At T=0.6T=0.6, the corresponding gains are 1.6% and 1.5%. MDraft's advantage therefore holds under either matched weighting scheme. With uniform weighting throughout, its improvement over pretrained DFlash is 1.33× that of the strongest baseline at T=0.6T=0.6 (AdaFlash), and 1.24× at T=1T=1 (RKL). These ratios compare the improvements over the same pretrained drafter.

    At T=0T=0, survival-weighted AdaFlash's lead is only 0.006 accepted tokens (6.691 versus 6.685). With uniform weights for both methods, AdaFlash leads by 1.6% (6.671 versus 6.568). We acknowledge this greedy-decoding result; MDraft's strongest benefits occur under sampling, where it consistently leads in acceptance length under both weighting schemes.

    Measured inference benefits: Figure 2 shows that these improvements translate into higher throughput. At T=1T=1, MDraft is 6.9% faster than DFlash on GSM8K and 10.5% faster on MBPP, and 3.0% and 7.4% faster than AdaFlash, respectively. We measure tokens per second using the officially released DFlash testing codebase [5], with Transformers-based inference on a single NVIDIA H200 GPU and 128 samples per dataset, using the same checkpoints as in the acceptance-length evaluation. At T=0T=0, MDraft and AdaFlash achieve similar throughput. These speedups are obtained by replacing the drafter checkpoint, with no changes to the inference architecture or verification procedure.

  • 2. Scope and significance (W1).

    We respectfully disagree with the description of our contribution as too narrow. On-policy distillation is an established approach to language-model training [1, 2], and recent work has demonstrated its value for speculative drafters [3, 4]. This establishes drafter on-policy distillation of drafters as a valuable and promising research direction for LLM inference acceleration. Our paper identifies a structural obstacle to applying this approach to parallel drafters: the drafter must learn a shared prediction from conflicting, prefix-dependent verifier targets. We explain the consequences of this mismatch and introduce a simple method that uses cross-prefix information to improve distillation. This addresses an important training problem while preserving the architecture and inference efficiency that make parallel drafters attractive.

    The contribution also includes an explanation of what standard distillation objectives learn in this setting. Section 4.5 characterizes the optimal shared token distribution for a fixed context and position, a fixed draft-prefix distribution, and uniform position weights. Per-prefix forward KL yields the arithmetic average of verifier distributions, while reverse KL yields their normalized geometric average. We express reverse KL as matching average verifier support plus a penalty for disagreement across prefixes, and show how MDraft reduces that penalty. This explains why per-prefix reverse KL can suppress useful branch-specific tokens and provides a principled basis for the target construction.

    MDraft changes only the training target, so the resulting checkpoint can replace the original drafter without changing its architecture or inference procedure.

  • 3. General applicability (W3).

    We have expanded the evaluation to Qwen3-8B, training MDraft and AdaFlash starting from the published DFlash drafter for this model. The full results are included in Section 4.2 and Table 2 of our new revision. MDraft achieves the highest acceptance length on all six benchmarks at T=0.6T=0.6 and T=1T=1. At T=0T=0, its mean acceptance length is close to AdaFlash's (6.631 versus 6.655). The gains established in the 4B experiment broadly transfer to the larger model: at T=1T=1, MDraft’s mean relative acceptance-length improvement over pretrained DFlash reaches 13.4%, compared with 11.9% for Qwen3-4B.

    We acknowledge that experiments across verifier families, reasoning configurations, block lengths, and different drafter architectures would be useful. Within our constrained compute budget, these additional experiments establish that MDraft’s practical improvements extend to a larger verifier.

  • 4. Experimental attribution and baseline tuning (W2).

    Continued DFlash training: to isolate the benefit of on-policy training, we provide an off-policy baseline by continuing the training of the Qwen3-4B DFlash drafter with its original position-weighted cross-entropy objective [5], for the same number of steps used by the on-policy methods. We evaluate the resulting checkpoint using the same protocol and include the results in the main experiment table in Section 4.1, in the row labeled SFT. Relative to the pretrained drafter, continued SFT improves mean acceptance length by 1.9% at T=0T=0 and 0.7% at T=0.6T=0.6, but lowers it by 2.6% at T=1T=1. MDraft improves it by 3.9%, 7.3%, and 11.9%, respectively. MDraft’s substantially larger stochastic-decoding gains therefore cannot be explained simply by continuing training with the original objective.

    Baseline configurations: while we cannot apply full hyperparameter tuning for all competing baselines, we use the objective-specific hyperparameters specified in the original Draft-OPD [3] and AdaFlash [4] papers, including the λ\lambda coefficient in AdaFlash, and the combination of losses on accepted and rejected tokens in Draft-OPD. The objective configurations and optimization setup are described in more detail in Appendix E.

    Stronger AdaFlash baseline: we also evaluate AdaFlash with the same survival weighting used by MDraft (Table 6 in Appendix D). As detailed above, MDraft retains its advantage at T=0.6T=0.6 and T=1T=1, showing that survival weighting alone does not account for the gains.

References

  1. Rishabh Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. 2024.
  2. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024.
  3. Haodi Lei et al. Draft-OPD: On-Policy Distillation for Speculative Draft Models. 2026.
  4. Yu-Yang Qian et al. AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters. 2026.
  5. Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block Diffusion for Flash Speculative Decoding. 2026. Official code.

Dear reviewer 7wuC,

Thank you for your careful review and helpful feedback. As the discussion period draws to a close, we would appreciate your thoughts on whether our additional experiments and clarifications address your concerns. We would be happy to answer any remaining questions.

Reviewer_2ByKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation for one-step parallel diffusion drafters. Unlike an autoregressive student, a parallel drafter predicts every position in a draft block without observing the sampled in-block prefix, while the verifier evaluates each position conditioned on that prefix. Consequently, the same drafter distribution may receive conflicting verifier targets from different sampled prefixes. The paper proposes Multi-Draft On-Policy Distillation (MDraft). For each context, MDraft samples multiple draft blocks from one drafter forward pass, evaluates their prefix-conditioned distributions with the verifier, and averages these distributions across sampled prefixes. Each individual verifier distribution is geometrically blended with the average distribution to form a distillation target. The drafter is then trained with survival-weighted reverse KL. MDraft changes only training and retains the original DFlash architecture and exact speculative verification procedure at inference. On six mathematical, coding, and conversational benchmarks with Qwen3-4B, MDraft achieves the highest mean acceptance length among the evaluated post-training methods, particularly under stochastic decoding.

Critical strengths
  1. The paper identifies a meaningful structural mismatch in parallel-drafter distillation. The distinction between a prefix-blind drafter distribution and prefix-dependent verifier targets is clear and specific to parallel drafting.

  2. The proposed objective is well motivated. Aggregating verifier distributions across multiple sampled prefixes provides a reasonable way to preserve tokens that are useful under some likely branches but receive low probability under another sampled branch.

Critical weaknesses
  1. The experimental scope is too narrow. All main experiments use a single Qwen3-4B verifier and one 15-token DFlash checkpoint. It is unclear whether the observed prefix mismatch generalize across model sizes or architectures. Evaluation on larger Qwen models and a different family such as Llama is necessary to support the paper’s general claims.

  2. The DFlash baseline is the unchanged pretrained checkpoint, while MDraft and the other post-training methods receive three additional epochs on the new corpus. The paper does not include a matched control that continues training DFlash with its original position-weighted cross-entropy loss using the same initialization, data, steps, and checkpoint selection. It also lacks simpler aggregated-target baselines, such as cross-entropy, forward KL, or reverse KL directly against the prefix-averaged verifier distribution. These controls are needed to isolate the contribution of geometric target blending from additional training, multi-prefix marginalization, and divergence choice.

  3. The paper reports no wall-clock speedup. Mean acceptance length is a useful mechanism-level metric because the inference architecture is fixed, but it does not fully establish end-to-end acceleration. Relative to the strongest fine-tuned baseline, MDraft improves the mean acceptance length by only approximately 0.2%, 2.5%, and 3.1% at T=0, 0.6, and 1, respectively. These gains may translate into only modest or statistically insignificant latency improvements. End-to-end latency, tokens/s, and speedup over both DFlash and standard autoregressive decoding should be reported under matched hardware and runtime settings.

  4. The inference evaluation uses a single sampled chain; comparison with a strong multi-branch tree construction method such as DDTree would clarify whether MDraft remains beneficial when prefix diversity is already handled during inference.

Questions for authors
  1. Can the authors evaluate MDraft on additional verifier sizes and architectures, for example Qwen3-8B and at least one Llama-family model, using independently trained DFlash checkpoints?

  2. Can the authors add a continued-training baseline using the original DFlash position-weighted cross-entropy loss, matched in initialization, corpus, number of updates, optimizer, and checkpoint selection?

  3. How does MDraft compare with simpler multi-prefix targets, including cross-entropy, forward KL, and reverse KL directly against the averaged verifier distribution? Can the effects of multiple prefix sampling, geometric blending, reverse KL, and survival weighting be isolated separately?

  4. What are the end-to-end latency, output tokens/s, and speedup over the original DFlash checkpoint, the strongest fine-tuned baseline, and standard autoregressive decoding? Please report repeated measurements and uncertainty under the same hardware and inference stack.

  5. Does MDraft remain beneficial when combined with a multi-branch inference method such as DDTree? A useful evaluation would cross the original and MDraft-trained checkpoints with chain and tree-based inference under the same draft-node budget and system configuration.

6 comments

We thank the reviewer for their thorough analysis and assessment of our work and for recognizing the conditioning mismatch and the motivation for MDraft. We address the reviewer's questions below:

  • 1. Magnitude of gains and inference speed measurement (W3, Q4).

    MDraft and the strongest baseline in Table 1 are very close at T=0T=0: their mean acceptance lengths differ by only 0.2% (6.685 versus 6.671). However, at T=0.6T=0.6 and T=1T=1, the raw mean acceptance lengths are 2.5% and 3.1% higher than the strongest Table 1 baseline. To put these gains in context, MDraft's improvement over the pretrained drafter is 1.55×1.55\times AdaFlash's at T=0.6T=0.6 (+0.429+0.429 versus +0.276+0.276 tokens), and 1.39×1.39\times RKL's at T=1T=1 (+0.629+0.629 versus +0.452+0.452 tokens). The consistency of these gains also matters: in Table 1, MDraft ranks first in 15 of the 18 benchmark–temperature settings and second in the remaining three. In contrast, AdaFlash's mean ranks second at T=0T=0 but fourth at T=1T=1, while RKL ranks second at T=1T=1 but fifth at T=0T=0. MDraft's stronger performance under sampling is practically relevant: Qwen recommends non-thinking decoding with T=0.7T=0.7, top-p=0.8p=0.8, and top-k=20k=20 [4].

    We now also report measured inference speed in tokens per second (TPS) using the officially released DFlash testing codebase [2]. We evaluate the pretrained DFlash checkpoint and the MDraft and AdaFlash checkpoints used for acceptance-length evaluation, using Transformers-based inference on a single NVIDIA H200 GPU and 128 samples each from GSM8K and MBPP at T=0T=0 and T=1T=1. Figure 2 shows that the acceptance length gains generally translate to higher TPS; at T=1T=1, MDraft is 6.9% faster than DFlash on GSM8K and 10.5% faster on MBPP, and 3.0% and 7.4% faster than AdaFlash, respectively. At T=0T=0, it is 5.0% and 4.0% faster than DFlash and on par with AdaFlash. These measurements establish a throughput benefit under stochastic sampling. MDraft preserves the drafter architecture and inference procedure, so these gains introduce no additional inference operations.

  • 2. More model sizes and architectures (W1, Q1).

    To examine how the gains transfer to a larger verifier, we conducted additional experiments on Qwen3-8B, training MDraft and AdaFlash [1] from the published DFlash drafter [2] for this model. Section 4.2 and Table 2 of our revision provide the full results. MDraft achieves the highest acceptance length on all six benchmarks at T=0.6T=0.6 and T=1T=1. Its mean acceptance length exceeds AdaFlash's by 2.0% and 4.1%, respectively, and pretrained DFlash's by 8.0% and 13.4%. At T=0T=0, MDraft and AdaFlash remain close (6.631 versus 6.655). The mean gain over pretrained DFlash at T=1T=1 is larger than the 11.9% obtained with Qwen3-4B.

    We acknowledge that experiments with other verifier families, including Llama, and different drafter architectures would further strengthen the evidence. Within our constrained compute budget, the Qwen3-8B experiments establish that MDraft's practical improvements extend to a larger verifier.

  • 3. Matched continued-training baseline (W2, Q2).

    We now include continued training with the original DFlash position-weighted cross-entropy objective, reported as SFT in Table 1. The comparison uses the same pretrained Qwen3-4B DFlash initialization, teacher-generated corpus, 62,358 updates with a global batch size of 32, AdamW optimizer and learning-rate schedule, and final-checkpoint selection as the on-policy methods (Section 4.1 and Appendix E). Continued SFT changes mean acceptance length by +1.9%, +0.7%, and -2.6% at T=0,0.6,1T=0,0.6,1, respectively, while MDraft improves it by +3.9%, +7.3%, and +11.9%. MDraft's gains therefore are not explained by additional training with the original DFlash objective.

  • 4. Simpler aggregated targets and component ablations (W2, Q3).

    RKL against the averaged target and divergence choice: setting α=1\alpha=1 in MDraft recovers RKL directly against the aggregated distribution p^\widehat p, providing the simpler aggregated-target RKL baseline. We discuss the effects of α\alpha in the geometric blending section below. Unlike FKL, RKL depends on the target through its log probabilities, so averaging changes its expected objective, as characterized in Section 4.5 and Appendix C.

    Single-draft FKL experiment: we trained and evaluated a single-draft FKL baseline, and it reduced acceptance length relative to the pretrained DFlash checkpoint. On the GSM8K subset, it remained below initialization throughout the measured training at both T=0T=0 and T=1T=1 (Figure 6 in Appendix E.2).

    FKL against the averaged target: for a fixed context and position, with the sampled prefixes and verifier targets detached, soft-label cross-entropy and FKL against p^=K−1∑kp(k)\widehat p=K^{-1}\sum_k p^{(k)} have the same drafter gradient. With uniform or proposal-independent position weights, averaging the target also gives the same gradient as averaging per-prefix FKL over those drafts:

    =\nabla_\theta\mathrm{KL}(\widehat p\|q_\theta) =-\sum_y\widehat p(y)\nabla_\theta\log q_\theta(y) =\frac1K\sum_{k=1}^{K}\nabla_\theta\mathrm{KL}(p^{(k)}\|q_\theta).$$ In expectation over drafts from a fixed prefix law, this matches the single-draft FKL gradient.

Role of geometric blending: the parameter α\alpha provides a continuous interpolation between per-prefix RKL at α=0\alpha=0 and RKL against the aggregated distribution at α=1\alpha=1. To examine this choice jointly with proposal count, we first train a DFlash drafter for Qwen3-1.7B for four epochs on continuations from the Nemotron post-training dataset [3]. Starting from that checkpoint, we train each MDraft configuration for one additional epoch, sweeping K∈{2,4,8}K\in\{2,4,8\} and α∈{0.5,0.75,0.9,1.0}\alpha\in\{0.5,0.75,0.9,1.0\}. Table 4 in Appendix C reports the results, reproduced below as final mean acceptance length across the same six benchmarks used in the main evaluation. Bold marks the best value within each proposal count and temperature.

α\alphaK=2,T=0K=2, T=0K=2,T=0.6K=2, T=0.6K=2,T=1K=2, T=1K=4,T=0K=4, T=0K=4,T=0.6K=4, T=0.6K=4,T=1K=4, T=1K=8,T=0K=8, T=0K=8,T=0.6K=8, T=0.6K=8,T=1K=8, T=1
0.53.38333.26493.06163.40053.31613.08783.41363.31413.1223
0.753.39023.29173.08563.41233.31923.09263.42623.32973.1248
0.93.39263.26623.08493.41593.30483.10203.43623.32883.1308
1.03.39313.28863.09313.41583.28773.09613.43443.31563.1231

The best acceptance length increases with KK at every temperature, and all best-performing configurations use α>0.5\alpha>0.5. At K=2K=2, α=1\alpha=1 is best at T=0T=0 and T=1T=1, while α=0.75\alpha=0.75 is best at T=0.6T=0.6. At K=4K=4 and K=8K=8, intermediate values outperform α=1\alpha=1 at all three temperatures: α=0.9\alpha=0.9 is best at T=0T=0 and T=1T=1, and α=0.75\alpha=0.75 is best at T=0.6T=0.6.

Connection to the finite-KK analysis: Section 4.5 and Equation (10) in Appendix C show that the expected reverse-KL surrogate contains the disagreement penalty (1−α)J(y)+αJK(y)(1-\alpha)J(y)+\alpha J_K(y). At finite KK, JK(y)J_K(y) retains a penalty even when α=1\alpha=1. As KK increases, JK(y)J_K(y) decreases, and choosing α<1\alpha<1 retains an explicit contribution from J(y)J(y). The observed preference for α=1\alpha=1 at small KK in two decoding settings, and for intermediate values at larger KK, is consistent with this mechanism. These results support the complementary roles of the two parameters: KK improves estimation of the average verifier target, while α\alpha controls the remaining disagreement penalty.

Separating the components: Figure 4 varies the number of drafts KK while holding the other training settings fixed. Figure 5 compares per-prefix RKL with K=1K=1 and K=8K=8, both at α=0\alpha=0: the additional verifier evaluations leave both controls below the MDraft reference. This comparison complements the joint target-blending sweep above. Finally, Table 6 in Appendix D compares uniform and survival weighting: survival weights raise MDraft's mean acceptance length by 0.1170.117, 0.0610.061, and 0.0670.067 at T=0,0.6,1T=0,0.6,1, respectively, quantifying the additional contribution from position weighting.

  • 5. Compatibility with multi-branch inference (W4, Q5).

    MDraft is composable with DDTree [5]. We evaluate the original DFlash and MDraft-trained checkpoints for Qwen3-4B using the official DDTree codebase on a single NVIDIA H200 GPU, with matched system and evaluation settings across drafters. We compare node budgets of 16, 32, 64, 128, and 256 on GSM8K and MBPP, at decoding temperatures T=0T=0 and T=1T=1. Increasing the tree budget improves acceptance length for both drafters across all tested configurations. MDraft's gains therefore remain useful when inference also explores multiple branches.

    Adapting tree construction to the drafter: under the default tree construction algorithm, MDraft allocates nodes deeper in the tree, consistent with greater confidence in longer continuations. At a node budget of 256, the average depth of allocated nodes is:

    Drafter and tree temperatureGSM8KMBPP
    DFlash, τtree=1\tau_{\mathrm{tree}}=18.027.48
    MDraft, τtree=1\tau_{\mathrm{tree}}=110.089.77

    The deeper allocation favors long continuations at the expense of early branching. We adjust compare tree-construction temperatures, which we name τtree\tau_{\mathrm{tree}}, distinct from the decoding temperature TT. The default DDTree option is τtree=1\tau_{\mathrm{tree}}=1. Raising τtree\tau_{\mathrm{tree}} to 1.5 broadens the proposal distribution and shifts MDraft's allocation toward shallower branches. This improves MDraft's acceptance length at every tested tree budget on both datasets and at both decoding temperatures. Notably, the same change reduces DFlash's acceptance length and throughput throughout the tested tree settings, showing that the preferred tree-construction settings depend on the drafter.

    The effect is visible even at budget 32: on MBPP at T=1T=1, increasing MDraft's tree temperature from 1 to 1.5 raises throughput from 218.66 to 231.28 tokens/s, turning a 3.2% deficit into a 2.3% lead over DFlash with τtree=1\tau_{\mathrm{tree}}=1 (225.97 tokens/s).

    Acceptance and throughput at matched node budgets: MDraft with τtree=1.5\tau_{\mathrm{tree}}=1.5 achieves higher acceptance length than DFlash with any tested τtree\tau_{\mathrm{tree}} at every tested tree budget, on both datasets and at both T=0T=0 and T=1T=1. At T=1T=1, it also achieves higher throughput at every budget: gains range from 2.1% to 3.1% on GSM8K and 1.4% to 3.7% on MBPP. At budget 256, MDraft + DDTree achieves 271.88 versus 264.49 tokens/s on GSM8K (+2.8%) and 260.12 versus 253.86 tokens/s on MBPP (+2.5%) compared with DFlash + DDTree. At T=0T=0, MDraft's fastest tested configuration uses τtree=1\tau_{\mathrm{tree}}=1 on GSM8K and 1.51.5 on MBPP, and exceeds DFlash's throughput at every tree budget.

    We want to point out that DDTree's original configuration was tuned for DFlash. Our results show that adapting tree construction to the trained drafter matters: even our limited adjustment preserves MDraft's acceptance advantage across node budgets and its throughput advantage under stochastic decoding. Further tuning of DDTree for MDraft may yield additional gains. Our experiment shows that MDraft is compatible with DDTree and their gains can be effectively combined. Careful tuning of DDTree would be necessary to further establish the best achievable throughput for MDraft in this framework.

    We share the full results of this evaluation below for your reference. Each cell reports tokens/s / mean acceptance length. τ\tau in the column headers denotes τtree\tau_{\mathrm{tree}}. Bold marks the highest throughput and acceptance length separately for each dataset–decoding-temperature pair, across all budgets and tree settings. Comparisons between drafters use equal node budgets.

Reviewer_mSnhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Multi-Draft On-Policy Distillation (MDraft), a new training method designed to improve the acceptance length of parallel diffusion drafters. The authors identify a mismatch between parallel drafters, which predict tokens independently, and autoregressive verifiers, which condition on previously generated tokens. This mismatch can lead to conflicting supervision signals during on-policy distillation. To address this issue, MDraft samples multiple draft blocks from the same context and aggregates the verifier's probability distributions across different prefixes to construct more reliable distillation targets.

Critical strengths
  1. The motivation is very clear, the paper identifies the conditioning mismatch between the parallel drafter and the AR verifier, and analyzes how this problem affects on-policy distillation.
  2. The proposed method is simple and aligns well with the motivation.
  3. The paper gives a fairly thorough analysis of why the method works.
Critical weaknesses
  1. Although the proposed method makes sense, its practical effectiveness is very limited. Compared with AdaFlash and RKL, the improvement in acceptance length is really marginal.
  2. The experiments are not comprehensive. The paper only runs experiments in one setting, Qwen3-4B + DFlash. To demonstrate the effect of the proposed method, I think the authors need to provide results across different model sizes or model families.
  3. The experiments only report mean acceptance length and lack actual speedup comparison results. Since speculative decoding is an inference acceleration technique, reporting end-to-end latency and speedup is necessary.
  4. Multi-draft training is much more expensive than standard DFlash training, and the authors need to report how much the training cost increases. Given that the actual speedup gain is very marginal, I think this significantly higher training cost is not worth it.
  5. The presentation of Figure 1 needs to be improved.
Questions for authors

Please see the weaknesses.

5 comments

We thank the reviewer for their careful assessment of our work and for recognizing the conditioning mismatch identified in our work, the motivation for MDraft, and the accompanying analysis. We address the reviewer's questions below:

  • 1. Practical effectiveness and end-to-end inference speed (W1, W3).

    Gains over the strongest training baselines in Table 1: At T=0.6T=0.6, MDraft's improvement over the pretrained DFlash drafter [1] is 1.55× that of AdaFlash (+0.429 versus +0.276 accepted tokens); at T=1T=1, it is 1.39× that of RKL (+0.629 versus +0.452). These ratios compare the improvement over the pretrained drafter; the corresponding raw acceptance-length advantages over the strongest baseline in Table 1 are 2.5% and 3.1%. Relative to pretrained DFlash, MDraft improves mean acceptance length by 3.9%, 7.3%, and 11.9% at T=0T=0, 0.60.6, and 11, respectively. On Qwen3-8B, MDraft's improvement over the pretrained drafter is 1.37× and 1.50× that of AdaFlash at T=0.6T=0.6 and T=1T=1, respectively. MDraft and AdaFlash [2] are effectively tied in mean acceptance length at T=0T=0 (6.685 versus 6.671), while MDraft has a clear advantage under sampling.

    The consistency of these gains also matters: in Table 1, MDraft ranks first in 15 of the 18 benchmark–temperature settings and second in the remaining three. In contrast, AdaFlash's mean ranks second at T=0T=0 but fourth at T=1T=1, while RKL ranks second at T=1T=1 but fifth at T=0T=0. MDraft's strongest advantage is under stochastic decoding, a practically relevant setting: Qwen recommends sampling for non-thinking mode, with T=0.7T=0.7, top-p=0.8p=0.8, and top-k=20k=20 [3].

    Measured inference speed: with the architecture and inference procedure fixed, the methods perform comparable work per draft-and-verify step, so longer accepted blocks generally improve throughput. We now measure this benefit directly using the officially released DFlash testing codebase [1], with Transformers-based inference on a single NVIDIA H200 GPU. We evaluate 128 samples each from GSM8K and MBPP at T=0T=0 and T=1T=1, using the same DFlash, AdaFlash, and MDraft checkpoints as in the acceptance-length evaluation. Figure 2 reports the results. At T=1T=1, MDraft improves TPS over DFlash by 6.9% on GSM8K and 10.5% on MBPP, and over AdaFlash by 3.0% and 7.4%, respectively. At T=0T=0, MDraft improves TPS over DFlash by 5.0% and 4.0%, respectively, and achieves similar throughput to AdaFlash.

    MDraft's target construction applies to pretrained one-step parallel drafters. The trained checkpoint is a drop-in replacement: the measured gains require no changes to the drafter architecture or inference procedure.

  • 2. Model sizes and families (W2).

    We have expanded the evaluation to Qwen3-8B, training MDraft and AdaFlash from the published DFlash drafter for this model. Section 4.2 and Table 2 of our revision provide the full results. MDraft achieves the highest acceptance length on all six benchmarks at both T=0.6T=0.6 and T=1T=1. Its mean acceptance length is 2.0% and 4.1% higher than AdaFlash's, respectively, and 8.0% and 13.4% higher than pretrained DFlash's. At T=0T=0, MDraft and AdaFlash achieve similar means (6.631 versus 6.655). The 13.4% gain over pretrained DFlash at T=1T=1 also exceeds the 11.9% gain for Qwen3-4B.

    We acknowledge that experiments across different verifier families and architectures would further strengthen the evidence. Within our constrained compute budget, these additional experiments establish that MDraft's practical improvements extend to a larger verifier.

  • 3. Training cost (W4).

    Per-update cost: Table 5 in Appendix C reports update time and peak memory on a single NVIDIA H200 GPU at batch size 8. For K=8K=8, the setting used in our main Qwen3-4B experiments, an update takes 1.6652 seconds versus 0.3968 seconds for K=1K=1: 4.2× the update time, with approximately 3 GiB more peak memory (20.93 versus 17.94 GiB). Cost grows more slowly than the number of drafts: in the same profiling setup, K=2K=2 takes 0.5769 seconds, or 1.45× the single-draft update time, and K=4K=4 takes 0.9475 seconds, or 2.4×. Our Qwen3-8B experiments use K=4K=4.

    More compute for RKL: the Multiple-draft reverse-KL control in Appendix E (Figure 5) compares RKL with K=1K=1 and K=8K=8. Both controls remain below MDraft at the evaluated endpoint: increasing the number of drafts without blending their targets does not reproduce MDraft's gains in these runs.

    Continued DFlash training: we also include a matched continued-training baseline, labeled SFT in Table 1, which trains the Qwen3-4B drafter with the original DFlash position-weighted cross-entropy objective for the same number of updates. Relative to pretrained DFlash, SFT improves mean acceptance length by 1.9% at T=0T=0 and 0.7% at T=0.6T=0.6, but lowers it by 2.6% at T=1T=1, with decreases on five of the six benchmarks. MDraft instead improves acceptance by 3.9%, 7.3%, and 11.9%, respectively. The additional training investment therefore produces gains that continuing standard DFlash training does not.

    This is a one-time adaptation cost that can be amortized over repeated inference; the resulting checkpoint introduces no additional inference operations. Together with the measured throughput improvements, these results make MDraft an attractive adaptation for models serving repeated inference workloads.

  • 4. Presentation of Figure 1 (W5).

    We have reworked Figure 1 to showcase the main methodological idea of MDraft: the usage of multiple drafts during training to address the conditioning mismatch between the drafter and the verifier. The updated figure is present in our new revision.

References

  1. Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block Diffusion for Flash Speculative Decoding. 2026. Official code.
  2. Yu-Yang Qian et al. AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters. 2026.
  3. Qwen Team. Qwen3-4B model card: Best Practices.

Dear reviewer mSnh,

Thank you for your careful review and helpful feedback. As the discussion period draws to a close, we would appreciate your thoughts on whether our additional experiments and clarifications address your concerns. We would be happy to answer any remaining questions.

Comments on the paper

1 comment

We thank the reviewers for recognizing the challenge we identify in on-policy distillation for parallel drafters and our well-motivated solution. Their feedback helped strengthen our evidence and presentation.

Questions raised by the reviewers

  1. Practical significance: How substantial are the gains over DFlash and the baselines, and do the improvements translate into actual inference speedups?
  2. Generality: Do the gains extend beyond the original Qwen3-4B experiment to other settings?
  3. Experimental attribution: Can the gains be separated from additional training, multi-prefix sampling, geometric blending, and survival weighting?
  4. Training cost: How much additional time does multi-draft training require, and could giving a standard objective more compute recover the same gains?
  5. Tree-based inference: Does MDraft remain useful when combined with DDTree?

Experiments and evidence provided

  • Consistency (Question 1). In Table 1, MDraft ranks first in 15/18 settings and second in the rest, leading on all six benchmarks at T=0.6T=0.6 and T=1T=1. AdaFlash's mean rank falls from second at T=0T=0 to fourth at T=1T=1; RKL ranks fifth and second, respectively.

  • Bootstrap analysis (Question 1). For the evaluated checkpoints, bootstrap 95% CIs and adjusted pp-values for six-benchmark mean acceptance length show significant MDraft gains over AdaFlash and RKL at T=0.6T=0.6 and T=1T=1, and over RKL at T=0T=0.

  • Measured inference throughput (Question 1). We evaluated DFlash, AdaFlash, and MDraft on GSM8K and MBPP using the official DFlash codebase. At T=1T=1, MDraft improves tokens/s over DFlash by 6.9% and 10.5%, and over AdaFlash by 3.0% and 7.4%, respectively.

  • Qwen3-8B evaluation (Questions 1 and 2). MDraft achieves the highest acceptance length on all six benchmarks at both T=0.6T=0.6 and T=1T=1. At T=1T=1, its mean acceptance length exceeds DFlash's by 13.4% and AdaFlash's by 4.1%.

  • Matched continued-training control (Question 3). We continued DFlash training with its original position-weighted cross-entropy objective. At T=1T=1, this SFT baseline reduces mean acceptance length by 2.6%, whereas MDraft improves it by 11.9%. Additional training with the original objective therefore does not explain MDraft's gains.

  • Joint proposal-count and blending ablation (Question 3). On Qwen3-1.7B, we swept KK and α\alpha. This includes RKL directly against the averaged target at α=1\alpha=1. The best acceptance length increases with KK; at K=4K=4 and K=8K=8, intermediate blending outperforms α=1\alpha=1. These results support the complementary roles of multi-prefix aggregation and geometric blending.

  • FKL experiments (Question 3). We trained and evaluated a single-draft FKL baseline. On the evaluated subset, it remained below the pretrained checkpoint at both T=0T=0 and T=1T=1.

  • Matched position-weighting (Questions 1 and 3). We compare MDraft and AdaFlash with both uniform and survival weights. At T=1T=1, MDraft's mean acceptance length exceeds AdaFlash's by 5.2% with uniform weights and 6.0% with survival weights, showing that its advantage persists under either weighting scheme.

  • Training cost and RKL control (Questions 3 and 4). Relative to K=1K=1, K=4K=4 and K=8K=8 require approximately 2.4×2.4\times and 4.2×4.2\times the update time. RKL with either K=1K=1 or K=8K=8 remains below MDraft at the evaluated endpoint: additional verifier evaluations alone do not recover the gains.

  • DDTree composability (Question 5). Using the official DDTree codebase with Qwen3-4B, we compare node budgets from 16 to 256. Adapting the tree-construction temperature preserves MDraft's acceptance advantage across all budgets and its throughput advantage at T=1T=1. At budget 256, MDraft + DDTree is 2.8% faster on GSM8K and 2.5% faster on MBPP than DFlash + DDTree.

Revisions to the paper:

  • Section 4.1, Table 1: added the continued-SFT baseline to the main Qwen3-4B comparison.
  • Section 4.1, Figure 2, and Appendix E: added measured inference throughput and its evaluation setup.
  • Section 4.2, Table 2, and Appendix E.1: added the Qwen3-8B evaluation and data details.
  • Section 4.4 and Appendix C, Table 4: added the joint K/αK/\alpha sweep, complementing the experiment in Figure 4. Section 4.5 connects these findings to the finite-KK analysis in Appendix C.
  • Appendix C, Table 5: added training update-time measurements.
  • Appendix D, Table 6, and Appendix E.2, Figure 6: provide the matched position-weighting comparisons and single-draft FKL training dynamics.
  • Figure 1 and the accompanying explanation: revised the illustration.

These experiments and clarifications address the reviewers' concerns. We thank them again for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.