- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
Context and problem. The paper targets single-request (batch size 1) speculative decoding for local deployment, where compute is underused and verifying more candidate tokens per step is nearly free. Factorized (parallel) drafters such as DFlash predict all future positions in one forward pass. However, each position's distribution is a marginal that does not depend on which tokens were actually chosen before it. The authors show that acceptance therefore drops at later positions, even when the marginals are estimated directly from the verifier (Fig. 1). As a result, these drafters scale poorly to large tree budgets. A second, systems-level obstacle is that hybrid models such as Qwen3.6-27B contain Gated DeltaNet (GDN) layers. Verifying a tree through these recurrent layers needs branch-specific states, so a tree attention mask alone is not enough.
Method. The paper proposes Trees from Marginals (TfM), which has three parts:
- (1) Weaver: a 60M-parameter, single-layer autoregressive Transformer on top of a frozen DFlash. For each depth, DFlash's top-512 tokens define the candidate set. Weaver attends to the verifier and DFlash hidden states, which are prefilled once as a shared KV cache, and to the tokens on the current tree path. It outputs residual logits over the 512 candidates, using only the matching 512 rows of the vocabulary projection, and adds them to DFlash's logits. The draft distribution thus depends on the prefix, while the cost of a full-vocabulary projection is avoided.
- (2) Tree construction: a batched best-first grow-and-prune procedure. In each of rounds, it picks the frontier nodes with the highest cumulative log-probability, runs Weaver on them in one batch, and adds children to each. Children are chosen by top- or Gumbel top-. At the end, the highest-scoring nodes are kept.
- (3) GDN tree verification: an ancestor-masked extension of the chunked gated delta rule. It reduces tree verification to a triangular solve, implemented as fused CUDA and Metal kernels. The verify pass writes no state; the accepted path is replayed at commit time.
Verification uses naive sampling (TfM-NS) or Gumbel-max coupling (TfM-GMC), and both are lossless.
Contributions. (i) An analysis showing that marginal-based drafting is inherently limited at later positions. (ii) Weaver, a cheap prefix-conditioned corrector that turns a factorized drafter into a scalable tree drafter. (iii) A rollback-free tree-verification algorithm and kernels for GDN layers. (iv) Implementations in SGLang and in the Apple-Silicon runtime Uzu.
Validation. The paper evaluates four verifiers (Llama3.1-8B, Qwen3.5-9B, Qwen3.6-27B, Muse Glimmer) on 8 chat, math, and code benchmarks at batch size 1, against DFlash, DDTree, EAGLE-3, and (on Apple Silicon) MTPLX.
- On B200 with Qwen3.6-27B, TfM reaches 4.09× over AR on average, which is 1.22× over DFlash and 1.10× over DDTree, with about 8.1 tokens per step versus 6.65 for DDTree. Muse Glimmer shows similar gains. For the two smaller models (≤9B), TfM is slower than DFlash on B200, because tree construction costs more than the extra accepted tokens save.
- In local deployment with Qwen3.6-27B, TfM-GMC is 1.27× and 1.35× faster than DFlash on M5 Pro and M5 Max. On DGX Spark it reaches 6.29× over AR, 1.51× over DFlash and 1.16× over DDTree.
- TfM at matches DDTree at in tokens per step (7.43 vs. 7.30).
- Ablations show the gain comes mainly from prefix conditioning. With Weaver disabled, Algorithm 1 matches DDTree's tokens per step (5.25 vs. 5.31); with Weaver enabled it rises to 6.26. A marginalized Weaver loses about 65% of the acceptance gain.
- The fused GDN kernel is 1.17–4.33× faster than a recurrent baseline on B200.
- The latency breakdown shows tree construction costs about 2.2 ms per round on B200, versus about 0.4 ms for DDTree.
Critical strengths
1. A clear diagnosis with a targeted, well-isolated fix. The paper first shows that the limit of factorized drafting is not just a matter of drafter quality. Even marginals estimated directly from the verifier lose acceptance at later positions, because a marginal cannot depend on the realized prefix (Fig. 1). Weaver addresses exactly this gap at low cost. It only re-scores DFlash's top-512 candidates per position, so it avoids a full-vocabulary projection and keeps the drafter small (60M parameters). The ablations attribute the gain convincingly. With Weaver disabled, the same tree procedure only matches DDTree in tokens per step (5.25 vs. 5.31). With Weaver enabled it rises to 6.26, and a marginalized Weaver loses about 65% of the acceptance gain (Appendix B.4–B.5). With Weaver, TfM at already matches DDTree at in tokens per step. Why it matters: the core idea is well motivated, and its effect is cleanly separated from other factors such as extra parameters or the search procedure. The acceptance improvement is therefore credible.
2. Tree verification for Gated DeltaNet layers, a practical systems contribution. Hybrid attention/GDN models such as the Qwen3.5/3.6 family are now widely deployed. Tree speculation on them needs branch-specific recurrent states, so a tree attention mask alone is not enough. The paper derives an ancestor-masked form of the chunked gated delta rule. The verify pass writes no state, and only the accepted path is replayed at commit. The paper implements this as fused CUDA and Metal kernels, 1.17–4.33× faster than a recurrent baseline on B200. The kernel is also used for the DDTree and EAGLE-3 baselines, and the full system runs in both SGLang and Uzu. Why it matters: this contribution is independent of Weaver. It enables tree-based speculative decoding on an important and growing class of models, and other methods can reuse it directly.
Critical weaknesses
1. The comparison with sequential tree drafters is not adequate, which leaves the core design choice unjustified. Weaver is itself a small autoregressive drafter: TfM uses a sequential model to correct a parallel drafter's tree. The natural alternative is a sequential tree drafter alone, such as EAGLE-3 with dynamic trees. However, the EAGLE-3 baseline appears under-configured. For Qwen3.6-27B it uses a third-party compressed checkpoint and reaches only 3.23 tokens per step, below even chain DFlash (4.90). Its tree configuration (depth, width, shape) is not described, and it is evaluated on only two of the four verifiers. Meanwhile, most of TfM's gain over DFlash already comes from the tree itself. On Qwen3.6-27B, DDTree is 1.10× over DFlash, and Weaver adds a further 1.10× on top of DDTree. On Muse Glimmer the extra gain is 1.06×. Why it matters: without a properly tuned sequential tree drafter under a matched tree budget and comparable training effort, it is unclear whether correcting a parallel drafter is better than drafting the tree sequentially from the start. This is the central question for the paper's design.
2. The method reads more like a patch on DFlash than a standalone algorithm, and its gains are modest and not general. Weaver cannot work on its own. Its candidate sets, input hidden states, and base logits all come from a pre-trained DFlash, and it only learns residual corrections on top of them. Any change to the parallel drafter therefore requires retraining Weaver. The paper also evaluates only DFlash, although it claims the method applies to any parallel drafter. At the same time, Weaver gives back part of the advantage of parallel drafting. It needs sequential rounds per step, and on B200 with Qwen3.6-27B total drafting time rises from 1.22 ms for DFlash to 3.76 ms for TfM, about 3×. The paper itself shows this does not pay off for smaller verifiers: on Llama3.1-8B and Qwen3.5-9B, TfM is slower than both DFlash and DDTree on B200 (e.g., 2.39× vs. 3.35× and 2.46× on Llama3.1-8B). At a small budget (), it brings essentially no throughput gain over DDTree (1.01×, Table 9). Why it matters: as a patch tied to one specific drafter, with clear gains only on larger verifiers and hardware where verification dominates, the contribution has limited independence and generality.
Questions for authors
-
Matched comparison with a sequential tree drafter. Could the authors compare TfM with a properly configured EAGLE-3 using dynamic trees, at the same tree budget and with comparable training data and compute? Ideally this would use an official or retrained checkpoint for Qwen3.6-27B. Please also report the EAGLE-3 tree configuration (depth, width, shape) used in Table 1 and Figure 4, and explain why its tokens per step (3.23) is below chain DFlash (4.90). This is my main concern (Weakness 1).
-
Independence from DFlash. Weaver depends on DFlash's candidate sets, hidden states, and base logits. Has the method been tried on any other parallel drafter (e.g., I-DLM or PARD), as the paper suggests is possible? If DFlash is retrained or replaced, must Weaver be retrained from scratch? Could Weaver be made less tied to one specific drafter?
-
Sequential drafting cost. TfM needs sequential Weaver rounds per step and is slower than DFlash and DDTree for verifiers of 9B or smaller on B200. Can the number of rounds be reduced, for example with larger and fewer rounds, so that the method also pays off in these settings? Alternatively, is there a practical rule for deciding when Weaver should be turned off?
Based on the authors' responses, if the core concerns above, especially Question 1, can be resolved, I would consider raising my score. More broadly, I hope that both parallel and sequential drafting continue to develop, and that each finds the application settings it suits best.
5 comments
Response to Reviewer RTrE (Part 2/3)
(3.) Matched EAGLE-3 Benchmark
Setup. Since no official EAGLE-3 checkpoint is available for Qwen3.6-27B, we are training one ourselves, on 650K completions drawn from the training data (Appendix A.2), with rounds=15 (to match the depth of Weaver) and the LK loss. It uses the full vocabulary.
EAGLE-3 is known to scale with training data, so we believe the fair protocol is to keep training it until its acceptance length stops improving. Below, we report the intermediate results at 400K completions.
(A.) Depth sweep We fix the width to 8 (to match Weaver) and sweep the depth at B={16, 64}. MTBench.
| Depth | Tokens/step (B=16) | Tokens/s (B=16) | Tokens/step (B=64) | Tokens/s (B=64) |
|---|---|---|---|---|
| 2 | 2.74 | 179.7 | 2.79 | 166.1 |
| 3 | 3.29 | 206.0 | 3.47 | 196.0 |
| 4 | 3.67 | 219.1 | 3.99 | 215.2 |
| 5 | 3.87 | 220.1 | 4.35 | 225.7 |
| 6 | 4.05 | 220.3 | 4.66 | 232.0 |
| 7 | 4.18 | 218.6 | 4.82 | 230.2 |
| 14 | 4.42 | 178.8 | 5.34 | 203.0 |
| 15 | 4.38 | 173.1 | 5.42 | 200.3 |
| TfM-NS (Depth 15) | 5.13 | 266.1 | 6.26 | 300.5 |
(B.) Training progress
B=64, (depth, width) = (7, 8), MTBench.
| Training Completions | tokens/step | tokens/sec |
|---|---|---|
| 100K | 4.440 | 212.16 |
| 200K | 4.654 | 223.08 |
| 300K | 4.762 | 227.57 |
| 400K | 4.817 | 230.15 |
EAGLE-3 is still improving.
(C.) Matched-budget comparison with TfM
Below, EAGLE-3 uses B=64, depth=15, width=8. TfM-GMC uses B=64, R16W8C8.
| Dataset | Method | Tokens/step | Tokens/s |
|---|---|---|---|
| MTBench | EAGLE-3 (retrained, 400K) | 5.42 | 199.6 |
| MTBench | TfM-GMC | 6.32 | 299.5 |
| GSM8K | EAGLE-3 (retrained, 400K) | 8.39 | 304.2 |
| GSM8K | TfM-GMC | 9.17 | 427.4 |
| HumanEval | EAGLE-3 (retrained, 400K) | 8.55 | 311.0 |
| HumanEval | TfM-GMC | 9.30 | 439.9 |
| Avg. | EAGLE-3 (retrained, 400K) | 7.45 | 271.6 |
| Avg. | TfM-GMC | 8.26 | 388.9 |
(D.) Effect of retraining and of the GDN kernel
B=64, (depth, width) = (7, 8), MTBench.
| Model | GDN | tokens/step | tokens/sec |
|---|---|---|---|
| EAGLE-3 (retrained, 400K) | Recurrent | 4.863 | 152.23 |
| EAGLE-3 (retrained, 400K) | Fused (Ours) | 4.817 | 230.75 |
| EAGLE-3 (public, compressed) | Fused (Ours) | 3.315 | 188.92 |
We additionally note that efficient implementation of EAGLE-3 for Qwen3.6-27B requires a fast GDN tree verification algorithm, which is one of the core contributions of our paper.
Three observations:
- Retraining helps EAGLE-3 substantially. The reviewer’s concern about the original baseline was justified.
- TfM remains faster.
- Depth is costly for EAGLE-3.
We will replace the EAGLE-3 baseline in the paper once the acceptance length stops improving. This will take a few more days.
Response to Reviewer RTrE (Part 3/3)
Q2: Independence from DFlash
Other factorized drafters. We have not evaluated the current version of Weaver on other factorized drafters, but we see no reasons preventing its use with drafters like PARD or I-DLM, as weaver only requires the lookahead states and marginal logits at each draft position, which any factorized drafter provides. In fact, the first prototype of Weaver was developed before the DFlash release, and used Medusa heads as a factorized drafter.
Retraining. Yes, Weaver must be retrained when the factorized drafter changes. We consider Weaver an integral part of the drafter, the same way a decoder is an integral part of an encoder-decoder model. In our experiments, we reused existing DFlash models to save compute, but nothing prevents training DFlash and weaver trained jointly.
Less coupling. We do not think this is the right direction. Thanks to this coupling, Weaver never needs a full-vocabulary projection, even during training, and can be trained together with DFlash in a single run. DFlash is trained as usual with its own optimizer. Weaver reads DFlash’s lookahead states and marginal logits through a stop-gradient and is updated by a separate optimizer. We trained both from scratch in this way for Qwen3.8-27B and added the results to Appendix A.4 of the revised paper.
Q3: Sequential drafting cost.
We hope Figure 1 helps here. Drafting latency is roughly linear in depth D:
- Weaver: , with a fixed cost c (DFlash) and a small slope.
- EAGLE-3: , with no fixed cost and a steep slope.
The two lines cross at . Below this depth, Weaver is not needed: an autoregressive drafter is cheaper, and a factorized drafter still predicts well at early positions (Figure 2). Above it, Weaver is cheaper, and the gap grows with depth.
Can the number of rounds be reduced? In Algorithm 1, the tree depth is bounded by the number of rounds R. Fewer rounds therefore mean a shallower tree, which moves back to the shallow regime. Width within a round is nearly free.
When to turn Weaver off? When the relative gain in is smaller than the relative increase in cycle time. This follows from the speedup model (Section 2, Notation). On B200 with Qwen3.5-9B, Weaver needs 20.6% more tokens/step than DDTree but delivers 16.3%, so it should be off. With Qwen3.6-27B, it needs 8.3% and delivers 22.1%, so it should be on. On all local devices we tested, Weaver should be on for both verifiers (Table 2).
We thank the reviewer again for the constructive suggestions. We hope these clarifications address their concerns, and help inform future work on both autoregressive and factorized drafters.
Response to Reviewer RTrE (Part 1/3)
We thank the reviewer for their careful reading and for raising excellent questions that help clarify how autoregressive, parallel, and hybrid drafters behave across datacenter and consumer GPUs.
Our answers to Q1-Q3 all rest on the hardware premise in our general response. In short: on local hardware, verifying more tokens is nearly free, so the speedup is bounded by speculation depth. We therefore need a drafter that stays cheap and accurate as depth grows.
We address their questions below.
Q1: Matched comparison with a sequential tree drafter.
(1.) The rationale for our EAGLE-3 baseline configuration
We evaluated EAGLE-3 with its dynamic trees. For each budget, we swept depth and width over the grid below and report the configuration with the highest throughput on the evaluation data, which favors the baseline. The configurations selected for Table 1 are (depth, width) = (5, 4) for Llama3.1-8B and (5, 10) for Qwen3.6-27B.
(Each cell lists the widths evaluated at that depth.)
We agree that the submission should have described this configuration. We will revise the paper once the EAGLE-3 training described in (3) is complete.
(2.) Why EAGLE-3’s tokens/step is below DFlash.
The DFlash paper reports the same result ([1], Table 1). The reason is drafter capacity. EAGLE-3 uses a single transformer layer; when scaled to five layers, it matches or slightly exceeds DFlash ([1], Table 9). However, as an autoregressive drafter pays memory traffic costs for its capacity at every drafting iteration (see our general response), it cannot afford five layers. DFlash pays only once, so it can. This is the main consideration behind the design of Weaver.
[1] Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block Diffusion for Flash Speculative Decoding. ICML 2026.