acceptodds
Under review as a conference paper at ICLR 2027

Fork and Regenerate: MTP-Guided Early Forking and Suffix Regeneration for Block Diffusion Speculative Decoding

Abstract

Block diffusion accelerates speculative decoding by predicting multiple draft tokens in parallel. However, verification accepts only a matching prefix, so an early mismatch prevents the remaining draft tokens from being accepted. Moreover, later predictions cannot use the tokens selected at earlier masked positions in the same pass. To address these issues, we introduce MTP-guided early forking and suffix regeneration (FoRE). Since earlier predictions determine how much of a draft can be accepted, we investigate whether the target's pretrained multi-token prediction (MTP) module can improve candidate selection at the first drafted position. Our analysis shows that MTP provides higher target-token coverage than DFlash at this position, motivating us to retain multiple MTP candidates and supply each to the block diffusion drafter before generating the remaining tokens. To further improve acceptance at later positions, draft refinement selects alternative tokens and regenerates the remaining tokens from each modified prefix. Training with visible prefixes of varying lengths enables the same block diffusion drafter to perform both stages, without an added correction head or a separately trained second drafter. The resulting drafts are verified together in one batched target-model forward pass, so the output follows the target's greedy decoding. Across seven benchmarks on Qwen3.5-9B and Qwen3.5-4B, our method achieves 1.53–1.55× the average acceptance length of DFlash and 1.28–1.32× its average decoding speedup, reaching 3.82–3.93× over autoregressive decoding. When applied to DFlash 2, our method also improves acceptance length and speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, large language models, LLM inference acceleration, multi-token prediction, block diffusion drafter

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6RKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FoRE combines MTP-guided first-position forking with later suffix regeneration for block diffusion speculative decoding. The target's pretrained multi-token prediction module proposes several alternatives at the first drafted position. A diffusion drafter completes those alternatives in one batched pass. Selected later tokens are then replaced, and the same drafter regenerates their suffixes in a second batched pass. Variable-length visible-prefix training enables one drafter to serve both stages. The target verifies the candidate rows and retains the longest prefix matching its greedy predictions.

Across seven workloads on Qwen3.5-9B/4B, the paper reports approximately 1.53-1.55x DFlash's accepted length and 1.28-1.32x its mean speedup factor. The DFlash 2 extension improves mean speedup from 2.97x to 3.12x. Validation includes matched fine-tuning and loss controls, fixed-replacement suffix comparisons, candidate-budget studies, and a second training seed.

Critical strengths
  1. A coherent reuse of a single conditioned drafter. The same variable-prefix-trained model completes MTP candidates and regenerates suffixes after replacements, avoiding a separate second drafter or correction head. The measured first-position MTP advantage provides a concrete motivation for using MTP selectively. This is a practical contribution to integrating candidate selection and conditional drafting.

  2. Controls establish that the gains are not explained solely by wider inference or extra fine-tuning. Table 3 and Appendix B include matched L=0 and loss controls. Appendix D fixes replacements and shows additional accepted tokens from regenerated rather than unchanged suffixes. Appendix E reports similar results under a second training seed. These controls are consequential because they support the method's ingredients within the evaluated pipeline.

Critical weaknesses
  1. The nearest algorithmic alternatives are described but not compared experimentally. The manuscript identifies DDTree/BASTION for multi-candidate drafting and D2SD/CURE for related conditional drafting or repair, yet the principal comparisons are with MTP, EAGLE-3, and single-draft DFlash. Internal ablations do not establish superiority over those close methods at a matched verification budget. Because the contribution combines related mechanisms, this omission limits the evidence of incremental significance and is the main reason for my negative recommendation.

  2. The final system does not isolate the value of MTP candidate selection, and the speed comparison needs complete accounting. Figure 3 studies top-one/top-three coverage, while the default system uses eight MTP candidates and eight refinements. An otherwise identical configuration taking its first-position candidates from DFlash would separate MTP's contribution from conditioned branching. In addition, the method verifies repeated complete rows and reconstructs accepted hybrid-model state (Appendix A.3); it should be explicit that reconstruction and all candidate-management costs are included in the reported timings. The modest DFlash 2 improvement makes this precision particularly consequential.

Questions for authors
  1. Evidence needed for assessment: How does FoRE compare with a close tree-drafting or suffix-repair method on the same target and implementation at a matched verification budget? A convincing advantage here could change my recommendation.

  2. Evidence needed for assessment: What happens when the K first-position candidates come from DFlash rather than MTP, with training, K, R, and verification fixed? This would quantify the special contribution of MTP-guided forking.

  3. Clarification: Does the measured decoding interval include hybrid-state reconstruction, MTP execution, replacement selection, regeneration, and cache management? Please provide the relevant timing boundaries or breakdown. EAGLE-3's normalization to a different engine's baseline should remain a reference comparison rather than an absolute latency ranking.

4 comments

Thank you for the review, and for stating what evidence would change the assessment. We ran both requested experiments.

W1, Q1. Comparison with a tree-drafting or suffix-repair method at a matched verification budget

How does FoRE compare with a close tree-drafting or suffix-repair method on the same target and implementation at a matched verification budget?

We thank the reviewer for highlighting the importance of comparisons with closely related methods. As detailed in General Response 1 (G1), we conducted additional experiments comparing FoRE against DDTree, BASTION, and D²SD under controlled evaluation settings.

FoRE achieves 7.7% and 8.2% higher average acceptance length than DDTree and BASTION's tree construction, respectively, under the same verification budget, ahead of both with their released drafters on all seven benchmarks. Even when DDTree and BASTION use our trained drafter, FoRE retains improvements of 2.4% and 1.9%, respectively.

For suffix regeneration, FoRE outperforms D²SD's design by 18.8% under matched training data and verification-row budgets. This is a controlled comparison of the two designs rather than a reproduction of D²SD's original system. Detailed experimental settings and per-benchmark results are provided in G1.

W2, Q2. First-position candidates from DFlash instead of MTP

What happens when the K first-position candidates come from DFlash rather than MTP, with training, K, R, and verification fixed?

We thank the reviewer for suggesting this control. To isolate the contribution of MTP-guided early forking, we replaced MTP's top-KK candidates with the drafter's own top-KK predictions at the first drafted position, while keeping training, draft refinement, verification, and evaluation settings fixed.

As shown in the table below, both candidate sources achieve comparable acceptance lengths, while consistently outperforming the original DFlash baseline (5.78 accepted tokens per step). At the default setting (K=8K=8, R=8R=8), MTP-guided and drafter-based forking achieve 8.84 and 8.88 accepted tokens per step, respectively. Although MTP provides higher first-position candidate coverage (99.2% vs. 98.6% at K=8K=8), both sources achieve nearly saturated coverage.

KKMTP, without draft refinementdrafter, without draft refinementMTP, with draft refinement (R=8R=8)drafter, with draft refinement (R=8R=8)
16.366.338.268.19
26.736.718.698.60
46.776.818.878.86
86.876.898.848.88

Importantly, MTP-guided forking offers an efficiency advantage. MTP provides the first-position candidates before conditional drafting, whereas using the drafter as the candidate source requires an additional drafter forward pass. At the default configuration (K=8K=8, R=8R=8), with both sources timed in the same batch, MTP-guided forking achieves 2.9–10.6% higher decoding throughput across all seven benchmarks. These results further clarify the role of MTP-guided early forking: MTP provides accurate early candidates with lower drafting overhead, while our prefix-conditioned drafting and suffix regeneration enable longer accepted sequences.

GSM8K to MT-Bench: tokens per second (↑)

sourcecost of the candidates (ms per step, ↓)GSM8KMATH-500AIME25HumanEvalMBPPLCBMT-Bench
drafter4.0260.6262.6220.6283.3230.4181.7138.6
MTP1.6270.9279.7243.9299.6237.1194.5147.1

W2, Q3. What the measured interval includes

Does the measured decoding interval include hybrid-state reconstruction, MTP execution, replacement selection, regeneration, and cache management?

Yes. All decoding speedups measured in our harness use a single wall-clock timer covering the entire decoding loop, including MTP execution, candidate selection, both drafter passes, target verification, recurrent-state reconstruction, and cache management. As detailed in General Response 2 (G2), FoRE increases the per-cycle latency from 32.4 ms to 38.7 ms compared with DFlash, but achieves substantially longer accepted prefixes (8.84 vs. 5.78 tokens per cycle), resulting in higher overall decoding throughput. The detailed timing breakdown is provided in G2. The DFlash 2 speedups in Table 2 (2.97× to 3.12×) are measured with the same timer.

We also clarify that EAGLE-3's speedup is measured relative to SGLang's own autoregressive baseline and is included only as a reference, rather than for direct absolute-latency comparison.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Thank you for running the requested experiments. The candidate-source ablation clarifies MTP’s contribution. At the default setting, both sources yield nearly identical acceptance lengths, and MTP improves decoding throughput by 3.2–10.6% across the seven benchmarks. Your clarification that the timer covers the full decoding loop, including recurrent-state reconstruction and cache management, also resolves my timing concern.

The matched-budget experiments address the comparison gap I raised. The 18.8% gain in the controlled D²SD-style comparison supports FoRE’s suffix regeneration design in that setting. With the same trained drafter, FoRE improves average acceptance length by 2.4% over DDTree and 1.9% over BASTION. These gains are modest, and it remains unclear whether they translate into higher end-to-end throughput against those tree-drafting baselines.

I am keeping my overall rating because the practical improvement over the closest alternatives is still uncertain.

We thank the reviewer for the follow-up and for confirming which points are resolved. To address the remaining question, we measured end-to-end decoding throughput against DDTree.

End-to-end throughput against tree drafting

These gains are modest, and it remains unclear whether they translate into higher end-to-end throughput against those tree-drafting baselines.

At the matched verification budget of 256 positions, FoRE achieves 26% higher decoding throughput than DDTree and 20% higher than DDTree given our trained drafter. FoRE is ahead on all seven benchmarks against both.

The two parts of the acceptance gain. FoRE's 7.7% gain in acceptance length over DDTree has two parts, and both are contributions of this paper. Our drafter training gives 5.2%, which is what DDTree gains when it uses our drafter. FoRE's prefix-conditioned candidate generation gives the remaining 2.4%. For BASTION the two parts are 6.2% and 1.9%. DDTree and BASTION are training-free, so the same-drafter comparison sets FoRE against a baseline that already includes the first part. Our drafter raises DDTree's own decoding throughput by 5%. Against that strengthened baseline, FoRE's decoding throughput is still 20% higher.

Source of the throughput gap. With the same drafter the two methods accept nearly the same number of tokens per step, so the cost of a step decides the comparison. FoRE's rows are ordinary sequences: they run with FlashAttention-2 and need no special kernel in the recurrent layers of the hybrid target. A tree needs an attention backend that takes its ancestor mask (PyTorch scaled-dot-product attention here) and a tree kernel in every recurrent layer. Verification takes 27 ms per step for FoRE's 16 rows and 38 ms for a 256-node tree in these runs. The difference more than covers the drafter pass of FoRE's draft refinement.

Results. We timed DDTree with the same tree verifier and kernels as the tree timings in G3, at one request on one H200 (three repetitions). BASTION's tree construction, which uses the same verifier and node count at this fixed size, was not timed. FoRE runs as in the paper, and its figures are those of the table in our first reply. With FoRE moved to the trees' attention backend, which mainly slows its drafter, its lead on the seven-benchmark mean is 12% over DDTree and 7% over DDTree with our drafter.

Decoding throughput in tokens per second:

methodGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-Benchmean
DDTree213.8224.7193.8229.6173.8164.0127.0189.5
DDTree, with our drafter222.3231.8201.6239.7195.0173.9128.9199.0
FoRE270.9279.7243.9299.6237.1194.5147.1239.0

A faster tree kernel. The seven-benchmark comparison uses the recurrence kernel of G1 and G3 (SGLang's). Weaver's release provides a second one, which computes its dot products at TF32 precision and is faster. With it, on GSM8K, HumanEval and MBPP, FoRE's decoding throughput is 4% above DDTree's, and DDTree's with our drafter is 1.5% above FoRE's.

Correction to our first reply. In the timing table under W2, Q2, the drafter row should read 4.0 ms per step (given as 3.5), 262.6 on MATH-500 (265.0), 283.3 on HumanEval (283.7) and 230.4 on MBPP (229.8), and the MTP row should read 147.1 on MT-Bench (148.4). MTP-guided forking therefore achieves 2.9–10.6% higher decoding throughput (given as 3.2–10.6%), and remains ahead on all seven benchmarks. These values have been corrected in the first reply and in the revised paper.

Reviewer_62ZeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes MTP-guided early forking and suffix regeneration (FoRE), which is an interesting drafting method in speculative decoding to enhance the acceptance length of block-diffusion speculative decoding like DFlash. FoRE has two major designs. First, it uses the target model’s built-in MTP module to pick the top-K candidates for the first draft token, and the diffusion drafter completes each candidate into a full draft in one batched pass. Second, it selects up to R later positions in these drafts, replaces the token at each position with an alternative, and runs the drafter again to regenerate the rest of the draft after the new token. The work also fine-tunes DFlash with randomly revealed ground-truth prefixes of different lengths to make the second design work. All K+R draft paths are then verified as separate rows in one target-model batch, and the longest match is accepted. The extensive experiments show that FoRE outperforms MTP, Eagle-3, and dFlash. It can also be integrated to dFlash2.

Critical strengths

The research question studied in this work is making sense. A wrong first token wastes the whole block, and tokens in the diffusion block cannot see each other theoretically. It is interesting to see the combination of MTP and diffusion drafter. Training the drafter with visible prefixes is also a simple idea that lets one drafter handle both drafting and regeneration. I am glad to see it works. Last, the method gives consistent gains over DFlash on all benchmarks and both model sizes, which is convincing.

Critical weaknesses

(1) I think the novelty of this work is somewhat limited. Early forking picks several candidates at the first drafted position is exactly the first level of a draft tree. In the second stage, draft refinement swaps one token at a later position for another candidate and completes the rest, which actually adds branches deeper in the tree. The only real difference is how the branches are built. FoRE uses an MTP pass and two drafter passes, while tree methods such as DDTree and PRESTO build the tree from one drafter pass. I do not see a fundamental difference from tree drafting, and this work does not show that this more complex way of building the tree works better. Moreover, because every branch is verified as a separate full row, the verification cost grows quickly with the number of branches. I am concerned that this design will not scale well to larger batch sizes.

(2) This manuscript cites DDTree, PRESTO, BASTION, Domino, Weaver, D2SD and CURE, but the main baselines are only MTP, EAGLE-3 and DFlash. Since FoRE verifies 16 drafts per step while DFlash verifies one, it is unclear whether the gains come from the proposed design or simply from verifying more drafts. A comparison with tree drafting under the same token budget is necessary in this case. The mauscript also misses some relevant citations, such as DSpark (Cheng et al., 2026, arXiv:2607.05147) and xPress (Wang et al., 2026, arXiv:2608.02438), which also address dependencies among tokens within a block.

Questions for authors

Please see weaknesses.

3 comments

Thank you for the review, and for the two references.

W1(a). FoRE as a draft tree

I do not see a fundamental difference from tree drafting, and this work does not show that this more complex way of building the tree works better.

We thank the reviewer for raising this important concern. While FoRE and existing tree-drafting methods both construct multiple candidate branches, the key distinction lies in how these branches are generated. DDTree and BASTION construct trees from token distributions computed in a single parallel drafter pass, without updating subsequent predictions based on selected alternative tokens. In contrast, FoRE drafts each branch with its selected prefix visible to the drafter, after MTP-guided early forking and after each later replacement, so subsequent predictions adapt to the selected tokens. Our variable-length visible-prefix training enables a single drafter to support both stages.

Importantly, our new experimental results support the effectiveness of this design. As detailed in General Response 1 (G1), FoRE achieves 7.7% and 8.2% higher average acceptance length than DDTree and BASTION's tree construction, respectively, under the same verification budget. Even when both methods use our trained drafter, FoRE retains improvements of 2.4% and 1.9%, demonstrating benefits beyond drafter training alone. Furthermore, our controlled ablation (Appendix D) shows that regenerating suffixes improves acceptance length by 0.44–1.07 tokens per step over retaining the original suffix under identical replacement choices.

W1(b). Verification cost and larger batch sizes

Because every branch is verified as a separate full row, the verification cost grows quickly with the number of branches. I am concerned that this design will not scale well to larger batch sizes.

Our experiments in the paper focus on the single-request setting, where verifying 16 candidate rows takes the same time as verifying one (about 27 ms). Under concurrent requests, verification latency grows with the total number of verified positions, as detailed in General Response 3 (G3).

Importantly, this behavior is shared by tree-based methods that verify a fixed node budget: a 256-node tree, the budget used for DDTree and BASTION in G1, costs within 0.5% of 16 rows at four and eight requests, and 4.4% less at sixteen. The cost can be reduced with existing system-level techniques that are orthogonal to FoRE. FoRE's rows can be verified as a single packed tree, following the tree verification used in DDTree, which is 17–24% cheaper than the 256-node tree at every batch size at the same acceptance length. BASTION's online budget controller further chooses the tree size at each decoding step, and with it BASTION's throughput stays above DFlash's up to 64 concurrent requests. We have not combined this controller with FoRE.

W2(a). Baselines at the same token budget, and whether the gain comes from verifying more drafts

it is unclear whether the gains come from the proposed design or simply from verifying more drafts. A comparison with tree drafting under the same token budget is necessary in this case.

The paper's matched controls hold the number of verified drafts fixed and still show the gain: with eight drafts each, our training improves acceptance length by 7.9–11.2% per benchmark over a control trained without visible prefixes (Table 3). In addition, as detailed in General Response 1 (G1), we conducted additional comparisons with DDTree and BASTION under the same verification budget of 256 token positions. FoRE achieves 7.7% and 8.2% higher average acceptance length, respectively, demonstrating that its improvements cannot be explained solely by verifying more candidates. PRESTO also builds its tree from a single drafter pass, adding a prefix-dependent nn-gram score to candidate ranking; we did not find a public implementation of it, so the tree comparison uses DDTree and BASTION. Furthermore, the matched analysis in Appendix D shows the additional benefit of suffix regeneration under identical replacement choices.

W2(b). DSpark and XPress

We thank the reviewer for pointing out these relevant studies. The revised paper cites and discusses both DSpark and XPress. Specifically, DSpark introduces an additional head to adjust token predictions based on preceding tokens, while XPress employs a refiner with parallel Jacobi iterations to model within-block dependencies. Unlike these approaches, which focus on refining a single draft, FoRE constructs multiple candidate branches and explicitly regenerates their suffixes conditioned on selected prefixes using the same trained diffusion drafter.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Hi authors of submission 50931,

Thank you for the response! However, my concerns are still there. When the tree methods use the same drafter, FoRE’s gain is only 1.9–2.4% in acceptance length. Appendix G also reports only acceptance length, although FoRE needs an extra MTP pass and a second drafter pass. Moreover, FoRE is slower than DFlash from two concurrent requests. Even as a packed tree, its verification still costs far more than DFlash’s at higher concurrency, and the mitigations are not evaluated end to end.

So I will keep my original score. This direction is very promising, and I encourage the authors to strengthen the novelty and the evaluation under batched serving in the future submission!

Thanks

Reviewer 62Ze

Reviewer_f4BBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FoRE, which uses a target model’s pretrained MTP module to select multiple candidates at the first drafted position and conditions a block-diffusion drafter on those candidates to generate continuations. The same drafter is trained with variable-length visible prefixes to regenerate suffixes after later replacements, and all candidate sequences are jointly verified under greedy decoding.

Critical strengths
  1. The focus on early-position candidate coverage is supported by a direct MTP–DFlash comparison: MTP improves both top-1 and top-3 coverage at the first drafted position.

  2. The ablations make a useful effort to isolate conditioning from extra fine-tuning and candidate expansion, including matched training controls, matched suffix replacements, and an additional training seed.

Critical weaknesses
  1. The experiments do not adequately position FoRE against the most closely related multi-candidate and suffix-regeneration methods. I suggest add comparisons withDDTree, BASTION, D²SD, and CURE.
  2. The serving implications of verifying many complete branches remain insufficiently established. The default verifies 16 sequences with repeated shared prefixes, while the reported speed excludes prefill and uses a single H200 NVL GPU, so the results do not yet show whether the advantage persists under concurrent requests and constrained KV-cache capacity.
  3. The speed comparison with EAGLE-3 is difficult to interpret across different execution engines. EAGLE-3 is measured in SGLang against its own autoregressive baseline, while FoRE is measured in Transformers, so the normalized speedup factors alone do not establish which system has lower absolute latency.
Questions for authors
  1. Could the authors compare against D²SD or CURE, and a tree-based candidate-selection baseline, under matched training and verification budgets?

  2. Could the authors report peak memory, absolute latency, and throughput across request batch sizes, including the cost of repeated branch prefixes and reconstruction of the accepted recurrent state?

  3. How does the method behave under stochastic decoding, and which parts of candidate verification would need modification to preserve the target sampling distribution?

4 comments

Thank you for the review, and for naming the measurements that would settle each point.

W1, Q1. Comparison with DDTree, BASTION, D²SD and CURE under matched training and verification budgets

Could the authors compare against D²SD or CURE, and a tree-based candidate-selection baseline, under matched training and verification budgets?

We thank the reviewer for emphasizing the importance of comparisons with closely related methods under matched training and verification budgets. As detailed in General Response 1 (G1), we conducted additional comparisons against tree-drafting methods (DDTree and BASTION) and a suffix-regeneration method (D²SD).

For tree-based methods, FoRE achieves 7.7% and 8.2% higher average acceptance length than DDTree and BASTION's tree construction, respectively, under the same verification budget. To further control for differences in drafter training, we evaluated both methods using our trained drafter while retaining their original tree-construction algorithms. FoRE still achieves improvements of 2.4% and 1.9%, respectively, demonstrating benefits beyond drafter training alone.

For suffix regeneration, we implemented D²SD's design and trained the drafters from scratch using the same 98K-example dataset. Under the same budget of 16 verified rows, FoRE achieves 18.8% higher average acceptance length (6.73 vs. 5.66). This is a controlled comparison of the two designs rather than a reproduction of D²SD's original system.

Regarding CURE, we selected D²SD for the additional comparison because its code is publicly available, whereas we did not find a public implementation of CURE. Detailed experimental settings, per-benchmark results, and comparison limitations are provided in G1.

W2, Q2. Peak memory, absolute latency, and throughput across request batch sizes

Could the authors report peak memory, absolute latency, and throughput across request batch sizes, including the cost of repeated branch prefixes and reconstruction of the accepted recurrent state?

We thank the reviewer for raising this important systems-level concern. As detailed in General Response 2 (G2), we conducted a comprehensive latency breakdown. This reply reports absolute decoding throughput and peak GPU memory on Qwen3.5-9B using a single NVIDIA H200 NVL GPU.

Single-request efficiency. As shown in the table below, FoRE achieves 29% higher decoding throughput than DFlash (239.5 vs. 186.0 tokens/s), with peak memory usage of 21.6–23.8 GiB versus 18.9–19.6 GiB. All reported decoding timings include MTP execution, drafting and regeneration, candidate selection, verification of repeated full-row prefixes, recurrent-state reconstruction, and cache management. Target prefill is the same single prompt pass for every method and is excluded for all of them (Section 4.1). It accounts for at most 2.8% of FoRE's total time and 2.1% of DFlash's on any of the seven benchmarks. Despite verifying 16 candidate rows rather than one, target verification latency increases by only 0.3 ms (26.7 to 27.0 ms) in this setting.

GSM8KMATH-500AIME25HumanEvalMBPPLCBMT-Benchmean
autoregressive61.560.961.361.061.060.760.260.9
DFlash215.6222.6193.3244.3164.1151.4110.7186.0
FoRE K=8,R=0K=8,R=0238.5257.7219.6271.6207.8175.1133.2214.8
FoRE K=8,R=8K=8,R=8269.6286.6245.8298.2236.0195.4145.0239.5

Concurrent-request scalability. We additionally measured target verification latency across 1–16 concurrent requests, as detailed in General Response 3 (G3). Verification latency remains at about 27 ms up to 16 rows in total and then grows by approximately 1.1 ms per additional row. In decoding throughput on a GSM8K subset, FoRE's default configuration is 26% faster than DFlash at one request and 6% slower at two, and DFlash scales further beyond that. This behavior is shared by tree-based methods that verify a fixed node budget, and it can be reduced with existing system-level techniques. Packed-tree verification of the same FoRE candidates, with the tree convolution kernel from Weaver's released implementation in the recurrent layers, removes the repeated prefixes (189 nodes instead of 256 positions on GSM8K). For example, at eight requests it reduces verification latency from 143.7 to 109.8 ms, although the two implementations use different attention backends. It peaks at 29.3 GiB of memory at sixteen requests, model included. BASTION's online budget controller further chooses the tree size at each decoding step. We have not combined it with FoRE.

W3. EAGLE-3 across engines

We thank the reviewer for raising this point. Due to the engineering complexity of reimplementing EAGLE-3 and FoRE across inference engines, we evaluated both methods using their respective implementations to preserve their original execution settings. As clarified in Table 1 and Section 4.2, EAGLE-3 was evaluated in SGLang, with its speedup normalized to SGLang's own autoregressive baseline, whereas FoRE and DFlash were evaluated in the same Transformers framework. We include EAGLE-3 as a reference comparison rather than an absolute latency ranking, while FoRE's decoding-speedup improvements over DFlash are measured under the same execution environment.

Q3. Stochastic decoding

How does the method behave under stochastic decoding, and which parts of candidate verification would need modification to preserve the target sampling distribution?

FoRE extends to stochastic decoding without modifying its MTP-guided early forking or prefix-conditioned suffix regeneration. The key modification lies in the verification procedure. Our current greedy verifier independently checks candidate branches and selects the longest prefix matching the target's greedy predictions. However, under stochastic decoding, independently sampling tokens for branches that share the same prefix and selecting the longest match would bias the output distribution.

To preserve the target's sampling distribution, we adopt prefix-shared sampling during verification, the rule SpecInfer calls naive sampling and DDTree also uses. Specifically, the target samples one token at each distinct candidate prefix, and all branches sharing that prefix use the same sampled token. Verification continues along the matching branch; if no candidate matches, the sampled token is emitted and the current decoding cycle terminates. This ensures that each generated token follows the target model's conditional sampling distribution.

We implemented this verification procedure and evaluated FoRE and DFlash at temperature 1 on all seven benchmarks. FoRE achieves 55% higher average acceptance length than DFlash (7.66 vs. 4.94 tokens per cycle), compared with 53% under greedy decoding, and is higher on every benchmark. The results in the paper use greedy decoding; the revision adds this experiment.

Thanks for the authors' detailed response, which have addressed my concerns. I raised my score to 3.

Thank you for reading our response carefully and for updating your assessment. We are glad it addressed your concerns, and the additional experiments and clarifications are included in the revised paper.

Comments on the paper

3 comments

We sincerely thank all three reviewers for their constructive feedback and insightful comments. We are encouraged that the reviewers recognize the well-motivated design (f4BB, 62Ze, 6RKX), effective conditional drafting with a single trained drafter (62Ze, 6RKX), consistent performance improvements (62Ze), and careful experimental validation (f4BB, 6RKX).

Below we address three common questions shared across reviewers.

G1. Comparison with tree-drafting and suffix-repair methods at a matched verification budget (all reviewers)

We thank the reviewers for highlighting the importance of comparisons with closely related methods. To address this concern, we conducted additional experiments comparing FoRE against (1) tree-drafting methods (DDTree and BASTION) and (2) a suffix-regeneration method (D²SD) under controlled training and verification budgets.

G1.1. Novelty and Experimental Comparison with Tree-Based Methods (DDTree and BASTION)

DDTreeBASTION¹DDTree, with our drafter (control²)BASTION¹, with our drafter (control²)FoRE
GSM8K9.038.999.539.469.81
MATH-5009.889.9410.4010.4310.63
AIME258.678.649.169.239.73
HumanEval10.019.8210.2610.5210.39
MBPP7.757.758.518.518.71
LiveCodeBench7.147.107.437.447.51
MT-Bench5.455.445.685.645.61
mean8.288.248.718.758.91

¹BASTION's tree construction at a fixed size, without its online budget controller, which relies on a latency model specified for full-attention targets.
²Their trees with our trained drafter. DDTree and BASTION are training-free, and their authors did not evaluate this configuration.
Percentages in the text are computed from the unrounded means.

Experimental Comparison. To provide a fair comparison, we conducted additional experiments against DDTree and BASTION under a matched verification budget of 256 forwarded token positions. Since their released implementations target full-attention Qwen3 models and do not directly support Qwen3.5-9B's hybrid architecture, we implemented a compatible tree verifier using SGLang's tree-aware kernels for recurrent layers and validated it against our original row-based verifier. We retained their original tree-construction algorithms, taken from their released code, and evaluated all methods using the same target model, prompts, random seed, decoding length, and attention backend (scaled-dot-product attention). As shown in the table above, FoRE achieves 7.7% and 8.2% higher average acceptance length than DDTree and BASTION's tree construction, respectively, ahead of both with their released drafters on all seven benchmarks.

Algorithmic Difference and Novelty. Although both FoRE and existing tree-drafting methods construct multiple candidate branches, the key difference is whether subsequent tokens are conditioned on the selected earlier tokens. DDTree and BASTION construct their trees from a single drafter forward pass. When an alternative token is selected at an earlier position, the predictions for subsequent positions remain unchanged because they were computed before that token was selected. In contrast, FoRE feeds the selected prefix back into the drafter and regenerates the remaining tokens, allowing each branch to have a continuation conditioned on its own prefix. To enable this, we train a single diffusion drafter with variable-length visible prefixes, which supports both early forking and later suffix regeneration. Our controlled ablation (Appendix D) confirms the benefit: with identical replacement choices, regeneration improves acceptance length by 0.44–1.07 tokens per step over keeping the original suffix.

Controlled Comparison with the Same Drafter. One potential concern is whether FoRE's improvements over DDTree and BASTION arise primarily from our additional drafter training rather than the proposed candidate-generation strategy. To isolate these factors, we further evaluated DDTree and BASTION using our visible-prefix-trained drafter, while retaining their original tree-construction algorithms and the same verification budget. As shown in the table, their average acceptance lengths increase from 8.28 to 8.71 and from 8.24 to 8.75, respectively. Nevertheless, FoRE still achieves 2.4% and 1.9% higher acceptance length, respectively. These results demonstrate that our training strategy improves the underlying drafter, while FoRE's prefix-conditioned candidate-generation procedure provides additional benefits beyond using the same trained drafter for one-pass tree construction.

The budget can also be counted in distinct nodes: FoRE's rows share prefixes and contain 196 distinct nodes per step on average. Against DDTree at 193 nodes, the nearest budget we ran below 196, FoRE achieves 8.7% higher average acceptance length, and 4.2% higher when DDTree uses our drafter, outperforming it on all seven benchmarks in both cases.

G1.2. Comparison with Suffix-Regeneration Methods (D²SD)

GSM8KMATH-500AIME25HumanEvalMBPPLCBMT-Benchmean
D²SD's design6.197.086.046.705.444.773.425.66
D²SD's design (our drafter)¹5.856.635.866.024.994.423.235.29
FoRE7.248.147.317.406.525.884.596.73

¹Our drafter in place of both of D²SD's drafters.
Percentages in the text are computed from the unrounded means.

Experimental Comparison. We further compare FoRE with D²SD, a closely related method that employs a separately trained second drafter for suffix regeneration. Since D²SD's released implementation does not support Qwen3.5's hybrid architecture and no pretrained D²SD drafter is available for Qwen3.5-9B, we implemented its candidate-selection and suffix-regeneration procedures. For a controlled comparison, we trained all drafters from scratch on the same 98K-example dataset and evaluated both methods with 16 verified candidate rows. As shown in the table above, FoRE achieves an average acceptance length of 6.73, outperforming D²SD's design (5.66) by 18.8% across the seven-benchmark average. With our drafter in both stages, D²SD's procedure reaches 5.29, against 6.73 for FoRE's, so the gain comes from FoRE's candidate generation and not from the drafter alone.

Novelty and Algorithmic Difference. While both methods regenerate suffixes conditioned on selected prefixes, D²SD employs two separately trained drafters for initial drafting and suffix regeneration. In contrast, FoRE uses a single visible-prefix-trained drafter for both stages, together with MTP-guided early candidate selection. This design avoids maintaining a separate regeneration drafter while improving acceptance length under the evaluated settings.

Scope of the comparison. This is a controlled comparison of the two designs rather than a reproduction of D²SD's original system. All drafters, including ours, are trained with one seed on the 98K-example dataset, whereas D²SD trains on the full PerfectBlend dataset (about 1.4M examples). Whether the ordering holds at that volume is not measured. To match FoRE's 16 rows, D²SD's design uses 15 branches, where its default is four.

G2. What the reported timings include (R-6RKX, R-f4BB)

To address the reviewers' concerns about computational overhead, we conducted a detailed latency breakdown of each decoding stage on Qwen3.5-9B using a single NVIDIA H200 NVL GPU. All speedups measured in our harness use a single wall-clock timer covering the entire decoding loop, including MTP execution, candidate selection, both drafter passes, target verification, recurrent-state reconstruction, and cache management. The table below reports the average latency of each component in milliseconds per decoding cycle (ms/cycle), averaged across seven benchmarks.

methodMTPdrafting¹verificationstate reconstruction²selectionother³step
DFlash–3.426.71.9–0.432.4
FoRE K=8,R=0K=8,R=0, 8 rows1.53.526.11.9–0.333.4
FoRE K=8,R=8K=8,R=8, 16 rows1.67.027.01.90.70.538.7

¹Includes draft refinement, the second drafter pass at R=8R=8.
²Keeping the accepted row's cache and rebuilding its recurrent state.
³Time inside the clock outside the named stages.

The key finding is that FoRE's substantial improvement in acceptance length outweighs its additional decoding overhead, resulting in higher overall throughput. Although FoRE verifies 16 candidate sequences instead of one, target verification latency increases by only 0.3 ms (26.7 to 27.0 ms) in the single-request setting, as the GPU can process multiple candidates in parallel within a single batched target-model forward pass. The additional drafting and candidate-selection operations increase total decoding-cycle latency by 19% (32.4 to 38.7 ms), while improving average acceptance length by 53% (5.78 to 8.84 tokens per cycle). By advancing more tokens per cycle, FoRE reduces the number of expensive target-verification cycles required for generation, achieving 29% higher measured decoding throughput (239.5 vs. 186.0 tokens/s) than DFlash.

G3. Verification cost above one request (R-62Ze, R-f4BB)

To address the reviewers' concerns about verification cost under concurrent requests, we measured the latency of the target verification forward pass on Qwen3.5-9B using a single NVIDIA H200 NVL GPU, for 1 to 16 concurrent requests with 1, 8 or 16 candidate rows each (512-token prefix). Verification latency is reported in milliseconds.

concurrent requests124816
verified as rows
1 row per request (DFlash)27.226.726.426.426.8
8 rows per request27.126.641.774.5143.8
16 rows per request (FoRE default)26.641.974.6143.7281.9
verified as a tree¹
FoRE's candidates (189 nodes on GSM8K)30.639.258.7109.8207.1
256 nodes (the budget of the tree methods in G1)²36.949.374.9143.7269.6

¹Trees use PyTorch scaled-dot-product attention, since a tree needs an ancestor mask, and the tree convolution kernel from Weaver's released implementation in the recurrent layers; rows use FlashAttention-2.
²A synthetic tree with the node count we ran DDTree at, not a tree DDTree built.

Single-request setting. Our experiments in the paper focus on the single-request setting. In this setting, verifying 16 candidate rows takes the same time as verifying one (about 27 ms at this prefix length), as the GPU processes all rows in parallel within a single batched forward pass.

Concurrent requests. As the number of concurrent requests increases, verification latency remains at about 27 ms up to 16 rows in total and then grows by approximately 1.1 ms per additional row. Verification accounts for most of each decoding cycle (27.0 of 38.7 ms, G2), so FoRE's default configuration loses its throughput advantage over DFlash, which verifies one row per request, at higher concurrency.

Comparison with fixed-budget tree methods. Importantly, this behavior is not specific to FoRE's row-based verification; it is shared by tree-based methods that verify a fixed node budget. A 256-node tree, the budget used for DDTree and BASTION in G1, shows the same growth: it costs within 0.5% of 16 rows at four and eight requests, and 4.4% less at sixteen. BASTION reports the same trend in end-to-end throughput on Qwen3-8B served with SGLang (Table 2 of its October 2026 arXiv version): the largest fixed tree in that table (64 nodes) is faster than DFlash at one request and slower at every higher concurrency level reported.

Reducing verification cost under concurrency. This cost can be reduced with existing system-level techniques for tree verification, which are orthogonal to FoRE. First, because FoRE's rows share prefixes, they can be verified as a single packed tree, following the tree verification used in DDTree. On GSM8K, the same 16 candidates then occupy 189 nodes instead of 256 positions, and they are 17–24% cheaper to verify than the 256-node tree at every batch size. The candidates are unchanged, so FoRE keeps its acceptance length, which is 7.7% above DDTree's at 256 nodes (G1). Second, BASTION's online budget controller chooses the tree size at each decoding step from a latency profile. With it, BASTION's throughput is above DFlash's at every concurrency level reported, up to 64 requests. These approaches could complement FoRE's prefix-conditioned candidate generation and potentially improve its scalability under concurrent serving. We discuss these integration opportunities in the revised manuscript while clarifying that our current end-to-end evaluation focuses on single-request decoding.

Changes in the revised PDF

  • A comparison section with the DDTree and BASTION tables and the D²SD comparison of G1, their qualifications, and the verifier used.

  • The early-forking control (reply to Reviewer 6RKX) with its table and clock.

  • The timing breakdown, the statement of what the clock includes, and peak memory.

  • Verification cost against concurrent requests, and a statement that the reported results are for one request.

  • Acceptance length under stochastic decoding at temperature 1, with the verification procedure it uses.

  • Citations of DSpark and XPress; a statement that EAGLE-3's speedup, measured against SGLang's own baseline, does not rank absolute latency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.