acceptodds
Under review as a conference paper at ICLR 2027

DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decoding

Abstract

Block diffusion speculative decoding improves LLM inference efficiency by proposing a block of future tokens in parallel and verifying them with a single forward pass through the target model. However, existing methods retain only the accepted prefix and discard the rejected suffix, preventing the computation spent on these positions from benefiting subsequent drafting rounds and forcing the drafter to repeatedly reconstruct representations for future tokens from scratch. We observe that rejection only determines whether a proposed token can be committed, while the verifier representations at rejected positions can still provide useful information for subsequent predictions. Based on this observation, we propose DFlow, a simple yet effective framework that enables verifier information to flow across drafting rounds. DFlow reuses the hidden states produced by the target verifier for the rejected suffix to guide subsequent drafting. To effectively learn this information flow across drafting rounds, we introduce a self-condition train strategy that feeds verifier representations from earlier predictions back into subsequent predictions. Experiments on Qwen3 models across diverse benchmarks demonstrate that DFlow consistently improves both acceptance length and end-to-end decoding speedup over DFlash. Our code will be openly released on GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KZ1NOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends DFlash-style block diffusion speculative decoding by reusing target-verifier hidden states from rejected speculative positions. DFlow projects these states using DFlash’s feature projection, aligns them with the drafter’s masked-token predictions, and injects them into the next drafting round through modulation conditioned on the bonus token and the first rejected token. Multi-Round Self-Conditioned Training unfolds an initial mask prediction round followed by two reject conditioned rounds. Experiments on Qwen3-1.7B, 4B, and 8B report approximately 10-13% higher average acceptance length and 9-12% higher relative decoding speed than DFlash under greedy decoding, with improvements also reported at temperature 1 and under concurrent SGLang serving. An integration with Domino further improves acceptance length.

Critical strengths
  1. DFlow offers a simple method to improve drafting by reusing verifier representations that would otherwise be discarded after rejection. The method aligns these representations with subsequent draft positions and conditions their reuse on the correction at the rejection boundary, without requiring additional target model forward passes at inference time. Table 3 supports the benefit of relay beyond multi-round mask-only training, and Figure 4 reports only 0.51 ms of additional latency in the profiled Qwen3 8B setting.

  2. Experimental results show consistent improvements in actual inference efficiency. Table 1 shows improvements over DFlash in acceptance length and end-to-end decoding speed across six benchmarks, three model sizes, and two temperatures, while Table 2 shows higher serving throughput across the evaluated concurrency levels. Combining DFlow with Domino further increases acceptance length from 4.02 to 4.50, suggesting that verifier reuse can complement stronger drafting mechanisms.

Critical weaknesses
  1. The additional training cost and performance under matched training budgets remain unclear. The DFlash + Self-Conditioning baseline in Table 3 controls for multi-round training, and its comparison with DFlow supports the benefit of verifier relay. However, this does not establish a comparison under matched computational budgets. DFlow constructs relay features through online verification by the frozen target model, but the associated verification cost and total training GPU hours are not reported. It remains unclear whether standard DFlash trained for more steps under a comparable compute budget could narrow the gap.

  2. Stochastic verification and runtime measurement are insufficiently specified. Section 3 describes exact token matching without specifying the sampling and acceptance/replacement procedure used at temperature 1 or reporting correctness checks. Conditioning on relayed states does not necessarily introduce bias, but the requirements for preserving the target distribution and managing the cache should be clearly specified. The evaluation protocol for Table 1 leaves some key settings unspecified, including the inference framework, batch/concurrency, and whether timing includes prefill or only decoding, and does not report the number of repeated measurements or their variability. These gaps limit independent assessment of the claimed lossless acceleration.

Questions for authors
  1. How is temperature=1 verification implemented, including target sampling, acceptance, bonus-token handling, and cache updates? Is the same verification scheme applied to EAGLE-3, DFlash, and DFlow at T=1?

  2. For the three training settings in Table 3, how do training GPU hours, target model verification costs, and the number of supervised draft blocks compare?

  3. To assess the training cost tradeoff, could the authors compare DFlow against a DFlash model trained with a computational budget comparable to DFlow’s?

  4. What implementation and timing settings were used for Table 1, including the inference framework, batch/concurrency, and whether timing includes prefill or only decoding? How many repeated measurements were taken and how variable are the results?

  5. In Figure 4, verify latency is reported as 37.55 ms for all three methods, and draft latency as 6.09 ms for both DFlash and DFlow. Were these component latencies measured separately for each method, or were single measurements reused across methods? Does the 0.51 ms relay component include all additional feature projection and modulation costs?

  6. Does the increase in acceptance length from 4.02 to 4.50 with Domino+DFlow in Table 4 also yield an end-to-end decoding speedup over Domino alone?

5 comments
AuthorsOpenReview ↗

Response to Reviewer KZ1N

Response to Reviewer KZ1N

1. Training Cost and Performance under Comparable Computational Budgets (Q2, Q3, W1)

All three settings in Table 3 use the same total number of supervised draft blocks. DFlash + Self-Conditioning and DFlow also share the same training procedure across multiple drafting rounds, with verifier replay disabled and enabled, respectively. This comparison controls the amount of supervision and isolates the contribution of verifier replay within the same training framework. The ratio of GPU hours consumed per training step is 1.00 : 1.75 : 3.50 for DFlash, DFlash + Self-Conditioning, and DFlow, respectively.

To compare performance under the same computational budget, we extended DFlash training to match DFlow’s total GPU hours. DFlash achieves an average acceptance length of 3.68, comparable to the DFlash + Self-Conditioning baseline in Table 3 but still below DFlow. Our extended training experiments further show that DFlash’s performance largely saturates, with additional training yielding no sustained improvement. These results indicate that DFlow’s gains under the evaluated setup cannot be attributed solely to additional training computation.

Although DFlow requires additional training computation, it delivers greater decoding speedups than DFlash. For a fixed target model, the extra training cost is incurred upfront, whereas the computational savings accumulate across subsequent inference requests. In deployments with sufficiently large inference volumes, these savings can outweigh the additional training cost, yielding a net reduction in overall computational cost.

2. Sampling and Verification at Temperature 1 (Q1, W2)

At temperature (T = 1), verification tokens are sampled from the target distribution, while draft tokens are selected by argmax. Draft tokens are accepted consecutively until the first mismatch, where the target sample is emitted as the bonus token. Relay conditioning only affects draft proposals.The emitted tokens remain target samples conditioned on the actual emitted prefix, preserving the target distribution. After verification, the target KV cache is truncated to the accepted prefix, while hidden states from rejected positions are retained only to guide the next drafting round.EAGLE-3, DFlash, and DFlow use the same inference framework, with identical target sampling, acceptance rules, bonus-token handling, and cache updates. The only difference is how draft tokens are generated.

3. Evaluation Settings and Overhead Measurement (Q4, Q5)

The results in Table 1 were obtained using the Transformers backend with a batch size of 1. Following DFlash, timing covers only the decoding stage and excludes prompt prefill. The serving experiments in Table 2 use SGLang with concurrency levels of 2, 4, 8, 16, and 32, and likewise measure decoding performance only. Timing measurements were repeated three times. The observed variation in decoding speedup is approximately 0.03×, which is small relative to the gains achieved by DFlow over DFlash.

Regarding Figure 4, target verification is primarily memory bound, leading to similar verification latencies across all methods. DFlash and DFlow also share the same draft backbone, with relay computation measured separately, so their draft execution latencies are expected to be similar. Our empirical measurements are consistent with this analysis. For simplicity, Figure 4 presents these shared components using common representative values. The reported relay latency of 0.51 ms accounts for all additional runtime costs introduced by DFlow relative to DFlash, including the additional feature projection and modulation costs.

4. End-to-End Speedup with Domino (Q6)

We additionally measured the decoding speed of Domino and Domino + DFlow on a PRO 5000 GPU. Both methods were evaluated under identical runtime settings, with results averaged across six benchmarks. DFlow provides an end-to-end decoding speed improvement of approximately 9.9% over Domino, demonstrating that the acceptance length improvement translates into practical acceleration.

MethodAccept LengthSpeedup
Domino4.022.62×
Domino + DFlow4.502.88×
AuthorsOpenReview ↗

Gentle reminder regarding our rebuttal

Dear Reviewer KZ1N,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them). Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Best regards,

The Authors

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Reviewer_3f1YOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DFlow, which reuses target-verifier hidden states from rejected draft suffixes to guide later rounds of block diffusion speculative decoding. It aligns these states with unresolved future positions and adjusts them using the first rejected token and its target correction. Multi-round self-conditioned training teaches the drafter to use this information across rounds. Experiments on three Qwen3 model scales and six benchmarks report longer acceptance lengths and higher end-to-end speedups than DFlash under greedy and sampling decoding, with further tests of concurrent serving and compatibility with Domino.

Critical strengths
  1. A useful role for rejected-suffix representations. DFlow separates the decision to reject a token from the usefulness of the verifier states computed at that position. Position alignment, rejection-conditioned modulation, and multi-round training give the drafter a way to reuse these states while leaving target verification unchanged.

  2. Ablations support the relay mechanism. In the same trained DFlow model, enabling relay raises average acceptance length from 3.66 to 3.93 (Table 3). This demonstrates the benefit of using verifier states from the effect of multi-round training. Table 1 shows gains across model scales, tasks, and both decoding settings. The serving results and the 0.51 ms relay latency measured on Qwen3-8B also support its practical value.

Critical weaknesses
  1. End-to-end evidence with newer block drafters is limited. The main comparisons use EAGLE-3 and DFlash. DFlash is a reasonable controlled base for isolating relay, as Appendix A.1 explains. Evidence that the gains carry over to newer drafters is limited: the Domino experiment reports only acceptance length on one model scale (Table 4), and DFlash 2 and DFlare are discussed without end-to-end comparisons.

  2. Training and memory costs are not quantified. Training unfolds an initial draft followed by two reject-conditioned rounds, with frozen-target processing of draft proposals (§4.3). The paper profiles inference-time relay latency at 0.51 ms (Figure 4), but does not report the additional training compute, peak memory, or verifier-state storage relative to DFlash. Table 2 also shows that, at concurrency 32, DFlow's HumanEval throughput is approximately 1.00× the autoregressive baseline on both model scales. It remains faster than DFlash, but the benefit over target-only decoding depends on the serving regime.

Questions for authors
  1. Throughput with other stronger drafters. Table 4 reports an increase in acceptance length from 4.02 to 4.50 when DFlow is added to Domino. Can you report the corresponding throughput and relay overhead, using existing measurements if available? How should this result be interpreted alongside newer parallel drafters such as DFlash 2 and DFlare?

  2. Training and memory costs. Compared with DFlash trained on the same data, how much additional training time or GPU computation do the rollout rounds require, including the frozen-target passes? What are the extra verifier-state storage and peak memory costs at inference? Which serving regimes benefit most, given the near-1.00× speedup over autoregressive decoding on HumanEval at concurrency 32?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3f1Y

Response to Reviewer 3f1Y

1. Compatibility and Speedup with Newer Drafters (Q1, W1)

We have conducted additional experiments with Domino, DFlare, and DFlash2 to examine whether DFlow’s benefits extend to newer drafting methods. For Domino, we supplemented the acceptance length results with decoding speedup measurements. For DFlare, we trained both DFlare and DFlare + DFlow on ShareGPT for one epoch. For DFlash2, we trained both variants on approximately 1.4M OpenPerfectBlend samples using a combination of CE and TV losses. Across these different architectures and training settings, adding DFlow consistently improves performance over the corresponding baseline on PRO 5000 GPU.

MethodAccept LengthSpeedup
Domino4.022.62×
Domino + DFlow4.502.88×
DFlare2.221.68×
DFlare + DFlow2.611.93×
DFlash26.374.40×
DFlash2 + DFlow6.704.60×

2. Inference and Training Costs (Q2, W2)

At inference, DFlow introduces only the additional relay computation, with approximately 40 MiB of weight memory for the relay module. It retains verifier representations for at most 16 positions per active request, and the measured increase in peak GPU memory is approximately 80 MiB. The additional inference memory overhead is therefore small.

For training, DFlow requires approximately 3.5× the training time of DFlash on Qwen3-1.7B. Peak training memory increases from 19.9 GiB to 42.4 GiB.Given this additional training cost, we further examined whether simply training DFlash for longer could achieve comparable gains. We therefore extended DFlash training to match DFlow’s total GPU hours. Under the same computation budget, DFlash achieves an average acceptance length of 3.68, compared with 3.93 for DFlow. The gains from further DFlash training have largely saturated in this setting, indicating that DFlow’s advantage cannot be explained solely by additional training computation.The additional training cost is incurred upfront and can be amortized over many subsequent inference requests for a fixed target model. As the trained drafter is reused, savings in inference GPU time accumulate across requests. In deployments with sufficiently large inference volumes, these savings can outweigh the additional training cost, yielding a net reduction in total GPU time across training and inference.

3. Performance across Serving Regimes (Q2, W2)

Speedup depends on the drafter, workload, and concurrency. Gains are generally more pronounced at low to moderate concurrency, where execution is less likely to be compute bound. With a stronger drafter, substantial acceleration can also be retained at higher concurrency. Our additional experiments show that DFlash2 + DFlow achieves speedups of 2.68× on GSM8K and 2.53× on HumanEval at concurrency 32.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

I thank authors for their rebuttal and clarification. I chose to keep my score unchanged. I have a small suggestion just for reference. Indeed, “Rejection does not imply uselessness” is a fairly good observation. The authors can try to refine this paper by add more analysis to answering: "From an information-theoretic perspective, how much information is actually retained?" I think this may help the paper to go deeper in its theoretical framing. It need not to be solved right now but authors can give a try. Good luck.

Thank you for the follow-up and for these insightful suggestion. Given the limited time remaining before the discussion deadline, we conducted a preliminary information-theoretic analysis to quantify how much verifier states from a rejected draft suffix help the drafter predict tokens in the next round. We present our findings below and plan to follow up with a more systematic analysis.

A rejected suffix retains useful verifier computation. Under greedy decoding with a fixed target model, the full committed context determines the target continuation. Given that context, the conditional Shannon mutual information between the continuation and the rejected suffix, including its verifier states, is therefore zero. The verifier can still help a shallow drafter by making predictive information implicit in the context easier to use. We quantify this benefit by measuring how much access to these states reduces the same drafter’s cross-entropy on the next round’s target tokens, in bits per slot.¹

Measurement. We evaluate Qwen3-4B on 500 prompts. For each relay round, we hold the committed context fixed and evaluate the same drafter under three input conditions. The first uses an all-mask block (no relay signal). The second uses the verifier states computed on the rejected draft suffix, as in standard DFlow inference. The third uses oracle states computed at the corresponding positions along the ground truth continuation and passed through the same relay channel. This oracle condition provides an empirical reference for how much the drafter can benefit from verifier states computed along the correct trajectory. We measured cross-entropy and top-1 accuracy on the same slots eligible for relay in all three conditions.

Drafter inputCross-entropy (bits/slot)Reduction vs. all-mask (bits/slot)Top-1 accuracy
All mask4.070.0049.9%
Rejected suffix states3.740.3353.0%
Oracle states2.901.1760.2%

Reusing verifier states from the rejected suffix reduces cross-entropy by 0.33 bits per relay slot, approximately 28% of the reduction observed with oracle states, and raises per-slot top-1 accuracy from 49.9% to 53.0%.

¹ Our analysis is motivated by A Theory of Usable Information Under Computational Constraints (Xu et al., ICLR 2020).

Reviewer_L5SoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DFlow, a method for improving block diffusion speculative decoding by reusing verifier information that is normally discarded after rejection. Hidden states associated with rejected draft tokens are aligned with overlapping positions in the next draft block and incorporated into its inputs through a correction module conditioned on the rejection-boundary tokens. The drafter is trained over multiple drafting and verification rounds to learn to use this feedback. Experiments on Qwen3 models report improvements in acceptance length for DFlash and Domino, along with end-to-end inference gains for the DFlash-based implementation. The central idea is to improve subsequent drafts using information already computed during verification, without requiring an additional verifier forward pass.

Critical strengths
  1. The robustness of the gains to baseline quality is not established. The main experiments use drafters trained on ShareGPT with block size 16. Their acceptance lengths are substantially below those reported in the original DFlash paper. For example, with Qwen3-8B and greedy decoding, the DFlash baseline in this submission achieves acceptance lengths of 4.36 on MATH500 and 3.82 on HumanEval, compared with 7.87 and 6.50 in the original DFlash results, also at block size 16. However, the submission does not sufficiently explain this gap or show how much of DFlow’s incremental benefit remains when the underlying drafter is more thoroughly optimized.
  2. Block size is also directly relevant to the mechanism: DFlow relies on rejected positions that overlap with the next block, whose availability changes with draft quality and block size. Evaluation only at block size 16 therefore leaves an important uncertainty about the method’s benefit at shorter, practically useful blocks such as 5 or 7.
Critical weaknesses
  1. What explains the gap between the reproduced DFlash baseline and the original reported results? Do DFlow’s gains persist with stronger training settings, such as target-regenerated OpenPerfectBlend and the loss used in DSpark, or with stronger drafters such as DSpark or DFlash2?
  2. How does DFlow perform at block sizes 5 or 7? Please compare the same drafter with and without DFlow, reporting the gains in both acceptance length and end-to-end decoding speed.
Questions for authors

The motivation is reasonable, and the ablations provide evidence that verifier feedback is useful within the reported setting. My main reservation concerns the robustness and attribution of the gains: the current experiments do not sufficiently establish their persistence with a stronger, adequately trained baseline, shorter blocks, and clearly documented training budgets. The current experiments do not sufficiently address these questions, leaving the practical scope of the improvement uncertain.

4 comments
AuthorsOpenReview ↗

Response to Reviewer L5So

Response to Reviewer L5So

1. Clarifying the DFlash Baseline Performance Gap (Q1, W1)

Thank you for pointing out the difference in acceptance length between our reproduced DFlash baseline and the results reported in the original DFlash paper. We attribute this gap primarily to differences in training data composition and scale. The original DFlash experiments use approximately 800K samples from NVIDIA Nemotron Post-Training Dataset V2 and CodeAlpaca. In contrast, following prior work such as EAGLE, we train all draft models, including DFlash, from scratch on the same 80K ShareGPT samples to ensure a fair comparison.

Our reproduced DFlash baseline also achieves results comparable to those reported in prior work using ShareGPT for training. Specifically, the DOMINO paper reports an acceptance length of 3.78 for its DFlash baseline on HumanEval with Qwen3-8B under greedy decoding (Table 3). This closely matches our reproduced result of 3.82 under the same target model, decoding mode, and drafting budget.

Although our training dataset is smaller than that used in the original DFlash experiments, using the same dataset for DFlash and DFlow ensures a fair comparison and eliminates the confounding effect of different training data. Under this setting, DFlow consistently improves both acceptance length and end-to-end decoding speed over DFlash.

2. Effectiveness with a Stronger Drafter and Larger Scale Training (Q1, W1)

We conducted additional experiments using DFlash2 trained on approximately 1.4M OpenPerfectBlend samples with a combination of CE loss and TV loss, following the loss design in DSpark. Compared with our original experiments on 80K ShareGPT samples, this setting uses substantially more training data, a stronger training objective, and a stronger drafting architecture. Evaluations on PRO 5000 GPU show that DFlow continues to improve acceptance length over DFlash2 consistently across all six benchmarks.

DatasetDFlash2 (Speedup / τ)DFlash2 + DFlow (Speedup / τ)
GSM8K6.67× / 9.346.89× / 9.70
MATH5005.89× / 8.386.11× / 8.72
HumanEval4.59× / 6.374.83× / 6.79
MBPP4.41× / 6.204.66× / 6.60
MT-Bench2.69× / 4.672.87× / 4.94
Alpaca2.12× / 3.262.24× / 3.47
Average4.40× / 6.374.60× / 6.70

3. Effectiveness with a Shorter Draft Block Size(Q2, W2)

To examine whether DFlow remains effective with shorter blocks, we additionally evaluated Qwen3-1.7B on PRO 5000 GPU with an inference block size of 8. The results show that DFlow’s gains in both acceptance length and decoding speed persist when the inference block size is reduced from 16 to 8.

MethodAccept LengthSpeedup
DFlash3.002.35×
DFlow3.592.78×
AuthorsOpenReview ↗

Gentle reminder regarding our rebuttal

Dear Reviewer L5So,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns (or at least part of them). Should there be any remaining questions or areas requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Best regards,

The Authors

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Dear Reviewer L5So,

Thank you again for your time and thoughtful review. As the discussion period closes in about five hours, we would like to briefly summarize how our response addresses your concerns.

  • Baseline comparison (W1 & Q1). We attribute the gap between our DFlash baseline and the original results primarily to differences in training data. We used 80K ShareGPT examples, while the original study used roughly 800K examples from Nemotron and CodeAlpaca. When we extended DFlash training to match DFlow’s total GPU-hours, DFlow still achieved a higher average acceptance length (3.93 vs. 3.68).
  • Stronger drafter (W1 & Q1). With DFlash2 trained on 1.4M OpenPerfectBlend examples using DSpark’s training objective, adding DFlow increased average acceptance length from 6.37 to 6.70 and average decoding speedup from 4.40× to 4.60×, with gains on all six benchmarks.
  • Shorter draft blocks (W2 & Q2). At block size 8 on Qwen3-1.7B, DFlow increased acceptance length from 3.00 to 3.59 and end-to-end decoding speedup from 2.35× to 2.78×.

We hope these results address your concerns, and we would be grateful if you could take them into account in your final assessment. We would be happy to clarify anything further within the remaining time.

Sincerely,
The Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your valuable feedback. We have conducted additional experiments to address your concerns and uploaded the revised manuscript with all changes highlighted in blue. We would be happy to discuss any remaining questions.

Thank you for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback. During the rebuttal, we evaluated DFlow with three stronger drafters, compared it with DFlash under a matched training budget, and tested a shorter draft block. We summarize the main results below:

  1. DFlow improves both acceptance length and end-to-end decoding speed with stronger drafters. For Domino we measured end-to-end decoding speed and found that adding DFlow increased the average speedup from 2.62× to 2.88×. Reviewer KZ1N confirmed that this result addressed their concern. We also evaluated DFlare on ShareGPT and DFlash2 on 1.4M OpenPerfectBlend. DFlow improved both metrics with both drafters. In particular, even with DFlash2 trained using DSpark’s combination of cross entropy and total variation losses, DFlow improved both metrics on all six benchmarks, increasing average acceptance length and average speedup. With DFlash2 as the stronger drafter, DFlash2 + DFlow achieved speedups of 2.68× on GSM8K and 2.53× on HumanEval over autoregressive decoding, even with 32 concurrent requests.

  2. DFlow’s gains persist under a matched training budget. Table 3 controls for the number of supervised draft blocks. In response to Reviewer KZ1N, we also extended DFlash training to match DFlow’s total GPU-hours. Under this matched budget, DFlow still achieved a higher average acceptance length (3.93 vs. 3.68). At inference, the relay added only 0.51 ms on Qwen3-8B, including feature projection.

  3. DFlow remains effective with a shorter draft block. With the inference block size reduced from 16 to 8 on Qwen3-1.7B, adding DFlow to DFlash increased average acceptance length from 3.00 to 3.59 and decoding speedup from 2.35× to 2.78×.

  4. A preliminary analysis of the predictive value retained after rejection. Following reviewer 3f1Y’s suggestion, we evaluated Qwen3-4B on 500 prompts. Reusing verifier states from the rejected suffix reduced cross-entropy by 0.33 bits per relay slot (about 28% of the reduction obtained with oracle states) and raised per-slot top-1 accuracy from 49.9% to 53.0%.

  5. We clarified verification, runtime measurements, and computational costs. We described how target sampling, exact-match acceptance, and cache updates work across methods. We also reported training overhead, peak training and inference memory, and the timing protocol for both single-request decoding and concurrent serving. We will incorporate these clarifications into the final version.

Thank you again for helping us improve the paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.