- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FoRE combines MTP-guided first-position forking with later suffix regeneration for block diffusion speculative decoding. The target's pretrained multi-token prediction module proposes several alternatives at the first drafted position. A diffusion drafter completes those alternatives in one batched pass. Selected later tokens are then replaced, and the same drafter regenerates their suffixes in a second batched pass. Variable-length visible-prefix training enables one drafter to serve both stages. The target verifies the candidate rows and retains the longest prefix matching its greedy predictions.
Across seven workloads on Qwen3.5-9B/4B, the paper reports approximately 1.53-1.55x DFlash's accepted length and 1.28-1.32x its mean speedup factor. The DFlash 2 extension improves mean speedup from 2.97x to 3.12x. Validation includes matched fine-tuning and loss controls, fixed-replacement suffix comparisons, candidate-budget studies, and a second training seed.
Critical strengths
-
A coherent reuse of a single conditioned drafter. The same variable-prefix-trained model completes MTP candidates and regenerates suffixes after replacements, avoiding a separate second drafter or correction head. The measured first-position MTP advantage provides a concrete motivation for using MTP selectively. This is a practical contribution to integrating candidate selection and conditional drafting.
-
Controls establish that the gains are not explained solely by wider inference or extra fine-tuning. Table 3 and Appendix B include matched L=0 and loss controls. Appendix D fixes replacements and shows additional accepted tokens from regenerated rather than unchanged suffixes. Appendix E reports similar results under a second training seed. These controls are consequential because they support the method's ingredients within the evaluated pipeline.
Critical weaknesses
-
The nearest algorithmic alternatives are described but not compared experimentally. The manuscript identifies DDTree/BASTION for multi-candidate drafting and D2SD/CURE for related conditional drafting or repair, yet the principal comparisons are with MTP, EAGLE-3, and single-draft DFlash. Internal ablations do not establish superiority over those close methods at a matched verification budget. Because the contribution combines related mechanisms, this omission limits the evidence of incremental significance and is the main reason for my negative recommendation.
-
The final system does not isolate the value of MTP candidate selection, and the speed comparison needs complete accounting. Figure 3 studies top-one/top-three coverage, while the default system uses eight MTP candidates and eight refinements. An otherwise identical configuration taking its first-position candidates from DFlash would separate MTP's contribution from conditioned branching. In addition, the method verifies repeated complete rows and reconstructs accepted hybrid-model state (Appendix A.3); it should be explicit that reconstruction and all candidate-management costs are included in the reported timings. The modest DFlash 2 improvement makes this precision particularly consequential.
Questions for authors
-
Evidence needed for assessment: How does FoRE compare with a close tree-drafting or suffix-repair method on the same target and implementation at a matched verification budget? A convincing advantage here could change my recommendation.
-
Evidence needed for assessment: What happens when the K first-position candidates come from DFlash rather than MTP, with training, K, R, and verification fixed? This would quantify the special contribution of MTP-guided forking.
-
Clarification: Does the measured decoding interval include hybrid-state reconstruction, MTP execution, replacement selection, regeneration, and cache management? Please provide the relevant timing boundaries or breakdown. EAGLE-3's normalization to a different engine's baseline should remain a reference comparison rather than an absolute latency ranking.
4 comments
As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.
Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.
Thank you all for your time and constructive engagement.
Thank you for running the requested experiments. The candidate-source ablation clarifies MTP’s contribution. At the default setting, both sources yield nearly identical acceptance lengths, and MTP improves decoding throughput by 3.2–10.6% across the seven benchmarks. Your clarification that the timer covers the full decoding loop, including recurrent-state reconstruction and cache management, also resolves my timing concern.
The matched-budget experiments address the comparison gap I raised. The 18.8% gain in the controlled D²SD-style comparison supports FoRE’s suffix regeneration design in that setting. With the same trained drafter, FoRE improves average acceptance length by 2.4% over DDTree and 1.9% over BASTION. These gains are modest, and it remains unclear whether they translate into higher end-to-end throughput against those tree-drafting baselines.
I am keeping my overall rating because the practical improvement over the closest alternatives is still uncertain.
We thank the reviewer for the follow-up and for confirming which points are resolved. To address the remaining question, we measured end-to-end decoding throughput against DDTree.
End-to-end throughput against tree drafting
These gains are modest, and it remains unclear whether they translate into higher end-to-end throughput against those tree-drafting baselines.
At the matched verification budget of 256 positions, FoRE achieves 26% higher decoding throughput than DDTree and 20% higher than DDTree given our trained drafter. FoRE is ahead on all seven benchmarks against both.
The two parts of the acceptance gain. FoRE's 7.7% gain in acceptance length over DDTree has two parts, and both are contributions of this paper. Our drafter training gives 5.2%, which is what DDTree gains when it uses our drafter. FoRE's prefix-conditioned candidate generation gives the remaining 2.4%. For BASTION the two parts are 6.2% and 1.9%. DDTree and BASTION are training-free, so the same-drafter comparison sets FoRE against a baseline that already includes the first part. Our drafter raises DDTree's own decoding throughput by 5%. Against that strengthened baseline, FoRE's decoding throughput is still 20% higher.
Source of the throughput gap. With the same drafter the two methods accept nearly the same number of tokens per step, so the cost of a step decides the comparison. FoRE's rows are ordinary sequences: they run with FlashAttention-2 and need no special kernel in the recurrent layers of the hybrid target. A tree needs an attention backend that takes its ancestor mask (PyTorch scaled-dot-product attention here) and a tree kernel in every recurrent layer. Verification takes 27 ms per step for FoRE's 16 rows and 38 ms for a 256-node tree in these runs. The difference more than covers the drafter pass of FoRE's draft refinement.
Results. We timed DDTree with the same tree verifier and kernels as the tree timings in G3, at one request on one H200 (three repetitions). BASTION's tree construction, which uses the same verifier and node count at this fixed size, was not timed. FoRE runs as in the paper, and its figures are those of the table in our first reply. With FoRE moved to the trees' attention backend, which mainly slows its drafter, its lead on the seven-benchmark mean is 12% over DDTree and 7% over DDTree with our drafter.
Decoding throughput in tokens per second:
| method | GSM8K | MATH-500 | AIME25 | HumanEval | MBPP | LCB | MT-Bench | mean |
|---|---|---|---|---|---|---|---|---|
| DDTree | 213.8 | 224.7 | 193.8 | 229.6 | 173.8 | 164.0 | 127.0 | 189.5 |
| DDTree, with our drafter | 222.3 | 231.8 | 201.6 | 239.7 | 195.0 | 173.9 | 128.9 | 199.0 |
| FoRE | 270.9 | 279.7 | 243.9 | 299.6 | 237.1 | 194.5 | 147.1 | 239.0 |
A faster tree kernel. The seven-benchmark comparison uses the recurrence kernel of G1 and G3 (SGLang's). Weaver's release provides a second one, which computes its dot products at TF32 precision and is faster. With it, on GSM8K, HumanEval and MBPP, FoRE's decoding throughput is 4% above DDTree's, and DDTree's with our drafter is 1.5% above FoRE's.
Correction to our first reply. In the timing table under W2, Q2, the drafter row should read 4.0 ms per step (given as 3.5), 262.6 on MATH-500 (265.0), 283.3 on HumanEval (283.7) and 230.4 on MBPP (229.8), and the MTP row should read 147.1 on MT-Bench (148.4). MTP-guided forking therefore achieves 2.9–10.6% higher decoding throughput (given as 3.2–10.6%), and remains ahead on all seven benchmarks. These values have been corrected in the first reply and in the revised paper.
Thank you for the review, and for stating what evidence would change the assessment. We ran both requested experiments.
W1, Q1. Comparison with a tree-drafting or suffix-repair method at a matched verification budget
We thank the reviewer for highlighting the importance of comparisons with closely related methods. As detailed in General Response 1 (G1), we conducted additional experiments comparing FoRE against DDTree, BASTION, and D²SD under controlled evaluation settings.
FoRE achieves 7.7% and 8.2% higher average acceptance length than DDTree and BASTION's tree construction, respectively, under the same verification budget, ahead of both with their released drafters on all seven benchmarks. Even when DDTree and BASTION use our trained drafter, FoRE retains improvements of 2.4% and 1.9%, respectively.
For suffix regeneration, FoRE outperforms D²SD's design by 18.8% under matched training data and verification-row budgets. This is a controlled comparison of the two designs rather than a reproduction of D²SD's original system. Detailed experimental settings and per-benchmark results are provided in G1.
W2, Q2. First-position candidates from DFlash instead of MTP
We thank the reviewer for suggesting this control. To isolate the contribution of MTP-guided early forking, we replaced MTP's top-K candidates with the drafter's own top-K predictions at the first drafted position, while keeping training, draft refinement, verification, and evaluation settings fixed.
As shown in the table below, both candidate sources achieve comparable acceptance lengths, while consistently outperforming the original DFlash baseline (5.78 accepted tokens per step). At the default setting (K=8, R=8), MTP-guided and drafter-based forking achieve 8.84 and 8.88 accepted tokens per step, respectively. Although MTP provides higher first-position candidate coverage (99.2% vs. 98.6% at K=8), both sources achieve nearly saturated coverage.
Importantly, MTP-guided forking offers an efficiency advantage. MTP provides the first-position candidates before conditional drafting, whereas using the drafter as the candidate source requires an additional drafter forward pass. At the default configuration (K=8, R=8), with both sources timed in the same batch, MTP-guided forking achieves 2.9–10.6% higher decoding throughput across all seven benchmarks. These results further clarify the role of MTP-guided early forking: MTP provides accurate early candidates with lower drafting overhead, while our prefix-conditioned drafting and suffix regeneration enable longer accepted sequences.
GSM8K to MT-Bench: tokens per second (↑)
W2, Q3. What the measured interval includes
Yes. All decoding speedups measured in our harness use a single wall-clock timer covering the entire decoding loop, including MTP execution, candidate selection, both drafter passes, target verification, recurrent-state reconstruction, and cache management. As detailed in General Response 2 (G2), FoRE increases the per-cycle latency from 32.4 ms to 38.7 ms compared with DFlash, but achieves substantially longer accepted prefixes (8.84 vs. 5.78 tokens per cycle), resulting in higher overall decoding throughput. The detailed timing breakdown is provided in G2. The DFlash 2 speedups in Table 2 (2.97× to 3.12×) are measured with the same timer.
We also clarify that EAGLE-3's speedup is measured relative to SGLang's own autoregressive baseline and is included only as a reference, rather than for direct absolute-latency comparison.