acceptodds
Under review as a conference paper at ICLR 2027

Trees from Marginals: Autoregressive drafting with factorized priors

15 views

Abstract

On-device applications of autoregressive large language models (LLMs) require fast token generation for individual requests, yet sequential decoding limits generation speed while underutilizing available compute. Speculative decoding trades additional computation for more tokens per target-model forward pass. While factorized drafters have set new records in speculative decoding efficiency by predicting future-token marginals in parallel, their independence assumption causes acceptance rates to degrade sharply at later draft positions, limiting their scalability to large speculative budgets. In this paper, we analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top- predictions of a factorized drafter. Weaver scales to large tree budgets by restoring conditional dependencies while avoiding the cost of a full-vocabulary projection. To verify these trees efficiently in recent models with Gated DeltaNet layers, we derive a rollback-free tree-verification algorithm and implement GPU kernels in CUDA and Metal. By combining these model and systems contributions, we achieve average lossless speedups of over autoregressive decoding and over a DFlash baseline on NVIDIA B200 with Qwen3.6-27B. For local deployment with the same model, we achieve average speedups over autoregressive decoding of on Apple M5 Max and on NVIDIA DGX Spark, with corresponding speedups of and over DFlash.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Large Language Model, Inference, CUDA, Metal

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RTrEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Context and problem. The paper targets single-request (batch size 1) speculative decoding for local deployment, where compute is underused and verifying more candidate tokens per step is nearly free. Factorized (parallel) drafters such as DFlash predict all future positions in one forward pass. However, each position's distribution is a marginal that does not depend on which tokens were actually chosen before it. The authors show that acceptance therefore drops at later positions, even when the marginals are estimated directly from the verifier (Fig. 1). As a result, these drafters scale poorly to large tree budgets. A second, systems-level obstacle is that hybrid models such as Qwen3.6-27B contain Gated DeltaNet (GDN) layers. Verifying a tree through these recurrent layers needs branch-specific states, so a tree attention mask alone is not enough.

Method. The paper proposes Trees from Marginals (TfM), which has three parts:

  • (1) Weaver: a 60M-parameter, single-layer autoregressive Transformer on top of a frozen DFlash. For each depth, DFlash's top-512 tokens define the candidate set. Weaver attends to the verifier and DFlash hidden states, which are prefilled once as a shared KV cache, and to the tokens on the current tree path. It outputs residual logits over the 512 candidates, using only the matching 512 rows of the vocabulary projection, and adds them to DFlash's logits. The draft distribution thus depends on the prefix, while the cost of a full-vocabulary projection is avoided.
  • (2) Tree construction: a batched best-first grow-and-prune procedure. In each of RR rounds, it picks the ww frontier nodes with the highest cumulative log-probability, runs Weaver on them in one batch, and adds cc children to each. Children are chosen by top-cc or Gumbel top-cc. At the end, the BB highest-scoring nodes are kept.
  • (3) GDN tree verification: an ancestor-masked extension of the chunked gated delta rule. It reduces tree verification to a triangular solve, implemented as fused CUDA and Metal kernels. The verify pass writes no state; the accepted path is replayed at commit time.

Verification uses naive sampling (TfM-NS) or Gumbel-max coupling (TfM-GMC), and both are lossless.

Contributions. (i) An analysis showing that marginal-based drafting is inherently limited at later positions. (ii) Weaver, a cheap prefix-conditioned corrector that turns a factorized drafter into a scalable tree drafter. (iii) A rollback-free tree-verification algorithm and kernels for GDN layers. (iv) Implementations in SGLang and in the Apple-Silicon runtime Uzu.

Validation. The paper evaluates four verifiers (Llama3.1-8B, Qwen3.5-9B, Qwen3.6-27B, Muse Glimmer) on 8 chat, math, and code benchmarks at batch size 1, against DFlash, DDTree, EAGLE-3, and (on Apple Silicon) MTPLX.

  • On B200 with Qwen3.6-27B, TfM reaches 4.09× over AR on average, which is 1.22× over DFlash and 1.10× over DDTree, with about 8.1 tokens per step versus 6.65 for DDTree. Muse Glimmer shows similar gains. For the two smaller models (≤9B), TfM is slower than DFlash on B200, because tree construction costs more than the extra accepted tokens save.
  • In local deployment with Qwen3.6-27B, TfM-GMC is 1.27× and 1.35× faster than DFlash on M5 Pro and M5 Max. On DGX Spark it reaches 6.29× over AR, 1.51× over DFlash and 1.16× over DDTree.
  • TfM at B=32B=32 matches DDTree at B=256B=256 in tokens per step (7.43 vs. 7.30).
  • Ablations show the gain comes mainly from prefix conditioning. With Weaver disabled, Algorithm 1 matches DDTree's tokens per step (5.25 vs. 5.31); with Weaver enabled it rises to 6.26. A marginalized Weaver loses about 65% of the acceptance gain.
  • The fused GDN kernel is 1.17–4.33× faster than a recurrent baseline on B200.
  • The latency breakdown shows tree construction costs about 2.2 ms per round on B200, versus about 0.4 ms for DDTree.
Critical strengths

1. A clear diagnosis with a targeted, well-isolated fix. The paper first shows that the limit of factorized drafting is not just a matter of drafter quality. Even marginals estimated directly from the verifier lose acceptance at later positions, because a marginal cannot depend on the realized prefix (Fig. 1). Weaver addresses exactly this gap at low cost. It only re-scores DFlash's top-512 candidates per position, so it avoids a full-vocabulary projection and keeps the drafter small (60M parameters). The ablations attribute the gain convincingly. With Weaver disabled, the same tree procedure only matches DDTree in tokens per step (5.25 vs. 5.31). With Weaver enabled it rises to 6.26, and a marginalized Weaver loses about 65% of the acceptance gain (Appendix B.4–B.5). With Weaver, TfM at B=32B=32 already matches DDTree at B=256B=256 in tokens per step. Why it matters: the core idea is well motivated, and its effect is cleanly separated from other factors such as extra parameters or the search procedure. The acceptance improvement is therefore credible.

2. Tree verification for Gated DeltaNet layers, a practical systems contribution. Hybrid attention/GDN models such as the Qwen3.5/3.6 family are now widely deployed. Tree speculation on them needs branch-specific recurrent states, so a tree attention mask alone is not enough. The paper derives an ancestor-masked form of the chunked gated delta rule. The verify pass writes no state, and only the accepted path is replayed at commit. The paper implements this as fused CUDA and Metal kernels, 1.17–4.33× faster than a recurrent baseline on B200. The kernel is also used for the DDTree and EAGLE-3 baselines, and the full system runs in both SGLang and Uzu. Why it matters: this contribution is independent of Weaver. It enables tree-based speculative decoding on an important and growing class of models, and other methods can reuse it directly.

Critical weaknesses

1. The comparison with sequential tree drafters is not adequate, which leaves the core design choice unjustified. Weaver is itself a small autoregressive drafter: TfM uses a sequential model to correct a parallel drafter's tree. The natural alternative is a sequential tree drafter alone, such as EAGLE-3 with dynamic trees. However, the EAGLE-3 baseline appears under-configured. For Qwen3.6-27B it uses a third-party compressed checkpoint and reaches only 3.23 tokens per step, below even chain DFlash (4.90). Its tree configuration (depth, width, shape) is not described, and it is evaluated on only two of the four verifiers. Meanwhile, most of TfM's gain over DFlash already comes from the tree itself. On Qwen3.6-27B, DDTree is 1.10× over DFlash, and Weaver adds a further 1.10× on top of DDTree. On Muse Glimmer the extra gain is 1.06×. Why it matters: without a properly tuned sequential tree drafter under a matched tree budget and comparable training effort, it is unclear whether correcting a parallel drafter is better than drafting the tree sequentially from the start. This is the central question for the paper's design.

2. The method reads more like a patch on DFlash than a standalone algorithm, and its gains are modest and not general. Weaver cannot work on its own. Its candidate sets, input hidden states, and base logits all come from a pre-trained DFlash, and it only learns residual corrections on top of them. Any change to the parallel drafter therefore requires retraining Weaver. The paper also evaluates only DFlash, although it claims the method applies to any parallel drafter. At the same time, Weaver gives back part of the advantage of parallel drafting. It needs R=16R=16 sequential rounds per step, and on B200 with Qwen3.6-27B total drafting time rises from 1.22 ms for DFlash to 3.76 ms for TfM, about 3×. The paper itself shows this does not pay off for smaller verifiers: on Llama3.1-8B and Qwen3.5-9B, TfM is slower than both DFlash and DDTree on B200 (e.g., 2.39× vs. 3.35× and 2.46× on Llama3.1-8B). At a small budget (B=16B=16), it brings essentially no throughput gain over DDTree (1.01×, Table 9). Why it matters: as a patch tied to one specific drafter, with clear gains only on larger verifiers and hardware where verification dominates, the contribution has limited independence and generality.

Questions for authors
  1. Matched comparison with a sequential tree drafter. Could the authors compare TfM with a properly configured EAGLE-3 using dynamic trees, at the same tree budget BB and with comparable training data and compute? Ideally this would use an official or retrained checkpoint for Qwen3.6-27B. Please also report the EAGLE-3 tree configuration (depth, width, shape) used in Table 1 and Figure 4, and explain why its tokens per step (3.23) is below chain DFlash (4.90). This is my main concern (Weakness 1).

  2. Independence from DFlash. Weaver depends on DFlash's candidate sets, hidden states, and base logits. Has the method been tried on any other parallel drafter (e.g., I-DLM or PARD), as the paper suggests is possible? If DFlash is retrained or replaced, must Weaver be retrained from scratch? Could Weaver be made less tied to one specific drafter?

  3. Sequential drafting cost. TfM needs R=16R=16 sequential Weaver rounds per step and is slower than DFlash and DDTree for verifiers of 9B or smaller on B200. Can the number of rounds be reduced, for example with larger ww and fewer rounds, so that the method also pays off in these settings? Alternatively, is there a practical rule for deciding when Weaver should be turned off?

Based on the authors' responses, if the core concerns above, especially Question 1, can be resolved, I would consider raising my score. More broadly, I hope that both parallel and sequential drafting continue to develop, and that each finds the application settings it suits best.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RTrE (Part 1/3)

We thank the reviewer for their careful reading and for raising excellent questions that help clarify how autoregressive, parallel, and hybrid drafters behave across datacenter and consumer GPUs.

Our answers to Q1-Q3 all rest on the hardware premise in our general response. In short: on local hardware, verifying more tokens is nearly free, so the speedup is bounded by speculation depth. We therefore need a drafter that stays cheap and accurate as depth grows.

We address their questions below.

Q1: Matched comparison with a sequential tree drafter.
(1.) The rationale for our EAGLE-3 baseline configuration
We evaluated EAGLE-3 with its dynamic trees. For each budget, we swept depth and width over the grid below and report the configuration with the highest throughput on the evaluation data, which favors the baseline. The configurations selected for Table 1 are (depth, width) = (5, 4) for Llama3.1-8B and (5, 10) for Qwen3.6-27B.

Budget BBd=3d=3d=5d=5d=6d=6d=7d=7d=8d=8d=12d=12
16, 324, 8, 104, 8, 104, 8, 10–4, 8, 10–
648, 104, 8, 104, 8, 10–4, 8, 10–
128–8, 108, 10–8, 104, 8, 10
256–8, 108, 108, 10–8, 10
512––––108, 10

(Each cell lists the widths evaluated at that depth.)
We agree that the submission should have described this configuration. We will revise the paper once the EAGLE-3 training described in (3) is complete.
(2.) Why EAGLE-3’s tokens/step is below DFlash.
The DFlash paper reports the same result ([1], Table 1). The reason is drafter capacity. EAGLE-3 uses a single transformer layer; when scaled to five layers, it matches or slightly exceeds DFlash ([1], Table 9). However, as an autoregressive drafter pays memory traffic costs for its capacity at every drafting iteration (see our general response), it cannot afford five layers. DFlash pays only once, so it can. This is the main consideration behind the design of Weaver.

[1] Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash: Block Diffusion for Flash Speculative Decoding. ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer RTrE (Part 2/3)

(3.) Matched EAGLE-3 Benchmark
Setup. Since no official EAGLE-3 checkpoint is available for Qwen3.6-27B, we are training one ourselves, on 650K completions drawn from the training data (Appendix A.2), with rounds=15 (to match the depth of Weaver) and the LK loss. It uses the full vocabulary.

EAGLE-3 is known to scale with training data, so we believe the fair protocol is to keep training it until its acceptance length stops improving. Below, we report the intermediate results at 400K completions.

(A.) Depth sweep We fix the width to 8 (to match Weaver) and sweep the depth at B={16, 64}. MTBench.

DepthTokens/step (B=16)Tokens/s (B=16)Tokens/step (B=64)Tokens/s (B=64)
22.74179.72.79166.1
33.29206.03.47196.0
43.67219.13.99215.2
53.87220.14.35225.7
64.05220.34.66232.0
74.18218.64.82230.2
144.42178.85.34203.0
154.38173.15.42200.3
TfM-NS (Depth 15)5.13266.16.26300.5

(B.) Training progress

B=64, (depth, width) = (7, 8), MTBench.

Training Completionstokens/steptokens/sec
100K4.440212.16
200K4.654223.08
300K4.762227.57
400K4.817230.15

EAGLE-3 is still improving.

(C.) Matched-budget comparison with TfM

Below, EAGLE-3 uses B=64, depth=15, width=8. TfM-GMC uses B=64, R16W8C8.

DatasetMethodTokens/stepTokens/s
MTBenchEAGLE-3 (retrained, 400K)5.42199.6
MTBenchTfM-GMC6.32299.5
GSM8KEAGLE-3 (retrained, 400K)8.39304.2
GSM8KTfM-GMC9.17427.4
HumanEvalEAGLE-3 (retrained, 400K)8.55311.0
HumanEvalTfM-GMC9.30439.9
Avg.EAGLE-3 (retrained, 400K)7.45271.6
Avg.TfM-GMC8.26388.9

(D.) Effect of retraining and of the GDN kernel

B=64, (depth, width) = (7, 8), MTBench.

ModelGDNtokens/steptokens/sec
EAGLE-3 (retrained, 400K)Recurrent4.863152.23
EAGLE-3 (retrained, 400K)Fused (Ours)4.817230.75
EAGLE-3 (public, compressed)Fused (Ours)3.315188.92

We additionally note that efficient implementation of EAGLE-3 for Qwen3.6-27B requires a fast GDN tree verification algorithm, which is one of the core contributions of our paper.

Three observations:

  1. Retraining helps EAGLE-3 substantially. The reviewer’s concern about the original baseline was justified.
  2. TfM remains faster.
  3. Depth is costly for EAGLE-3.

We will replace the EAGLE-3 baseline in the paper once the acceptance length stops improving. This will take a few more days.

AuthorsOpenReview ↗

Response to Reviewer RTrE (Part 3/3)

Q2: Independence from DFlash
Other factorized drafters. We have not evaluated the current version of Weaver on other factorized drafters, but we see no reasons preventing its use with drafters like PARD or I-DLM, as weaver only requires the lookahead states and marginal logits at each draft position, which any factorized drafter provides. In fact, the first prototype of Weaver was developed before the DFlash release, and used Medusa heads as a factorized drafter.

Retraining. Yes, Weaver must be retrained when the factorized drafter changes. We consider Weaver an integral part of the drafter, the same way a decoder is an integral part of an encoder-decoder model. In our experiments, we reused existing DFlash models to save compute, but nothing prevents training DFlash and weaver trained jointly.

Less coupling. We do not think this is the right direction. Thanks to this coupling, Weaver never needs a full-vocabulary projection, even during training, and can be trained together with DFlash in a single run. DFlash is trained as usual with its own optimizer. Weaver reads DFlash’s lookahead states and marginal logits through a stop-gradient and is updated by a separate optimizer. We trained both from scratch in this way for Qwen3.8-27B and added the results to Appendix A.4 of the revised paper.

Q3: Sequential drafting cost.
We hope Figure 1 helps here. Drafting latency is roughly linear in depth D:

  • Weaver: Tdraft≈c+awDT_{\mathrm{draft}} \approx c + a_{\mathrm{w}} D, with a fixed cost c (DFlash) and a small slope.
  • EAGLE-3: Tdraft≈aeDT_{\mathrm{draft}} \approx a_{\mathrm{e}} D, with no fixed cost and a steep slope.

The two lines cross at D≈5D \approx 5. Below this depth, Weaver is not needed: an autoregressive drafter is cheaper, and a factorized drafter still predicts well at early positions (Figure 2). Above it, Weaver is cheaper, and the gap grows with depth.

Can the number of rounds be reduced? In Algorithm 1, the tree depth is bounded by the number of rounds R. Fewer rounds therefore mean a shallower tree, which moves back to the shallow regime. Width within a round is nearly free.

When to turn Weaver off? When the relative gain in τ\tau is smaller than the relative increase in cycle time. This follows from the speedup model (Section 2, Notation). On B200 with Qwen3.5-9B, Weaver needs 20.6% more tokens/step than DDTree but delivers 16.3%, so it should be off. With Qwen3.6-27B, it needs 8.3% and delivers 22.1%, so it should be on. On all local devices we tested, Weaver should be on for both verifiers (Table 2).

We thank the reviewer again for the constructive suggestions. We hope these clarifications address their concerns, and help inform future work on both autoregressive and factorized drafters.

Reviewer_xbU8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Trees from Marginals (TfM), a tree-based speculative decoding method that combines parallel marginal predictions from a factorized drafter with autoregressive refinement. Existing factorized drafters do not explicitly model dependencies among drafted tokens, resulting in degraded acceptance rates at later draft positions. To address this limitation, TfM uses marginal predictions from DFlash as priors and introduces a lightweight autoregressive adapter, called Weaver, to perform prefix-dependent corrections. The paper also develops an efficient tree verification algorithm and specialized GPU kernels for Gated Delta Networks (GDNs). Experiments demonstrate up to a 1.22× speedup over DFlash on Qwen3.6-27B.

Critical strengths
  • Clear problem definition and well-motivated methodological approach

    • The paper identifies the acceptance degradation problem in factorized drafters and analyzes it from the perspective of missing prefix dependencies. The authors further provide empirical evidence that even oracle marginal predictions cannot fully resolve this limitation. The proposed Weaver directly addresses these sequential dependencies through lightweight autoregressive refinement, establishing a clear connection between the identified problem and the proposed solution.
  • Technical contributions to GDN tree verification

    • The paper develops an efficient tree verification algorithm and specialized GPU kernels to handle branch-specific recurrent states in GDNs. These implementation-level contributions are valuable for enabling tree-based speculative decoding in architectures with recurrent state dependencies. The evaluation across different hardware platforms is also a positive aspect of the work.
Critical weaknesses
  • Insufficient differentiation from existing approaches

    • The core idea of Weaver, which applies lightweight autoregressive corrections to parallel marginal predictions, shares substantial similarities with Domino. Although the paper explicitly cites Domino, it does not provide a sufficiently detailed discussion of the methodological differences or a direct empirical comparison. Furthermore, the probability-based tree construction in Algorithm 1 appears conceptually similar to the dynamic tree construction strategy introduced in EAGLE-2. Given these similarities, the paper should more clearly articulate its distinct methodological contributions and explain how the proposed approach differs from existing methods.
  • Unclear fairness and optimization of baseline configurations

    • It is unclear whether the speculative budgets and hyperparameters of the baseline methods were adequately optimized for end-to-end throughput. In particular, DFlash is evaluated with draft lengths of 10 or 16, although shorter draft lengths might yield better throughput given the substantial acceptance degradation at later draft positions. While Figure 4 examines the sensitivity of DDTree and TfM to speculative budgets, no corresponding analysis is provided for DFlash or EAGLE-3. Without ensuring that each baseline is evaluated under its individually optimized configuration, it is difficult to determine whether the reported speedups reflect genuine methodological improvements or differences in configuration choices. A more detailed description of the tuning procedure, along with budget-sensitivity analyses for the baselines, would strengthen the fairness and credibility of the comparisons.
  • Lack of evaluation of batch-size scalability

    • All experiments are conducted with a batch size of one. Although the focus on single-request local inference is understandable, TfM introduces additional autoregressive drafting overhead, whose impact on efficiency may become more pronounced as batch size increases. Consequently, it remains unclear whether the reported advantages extend beyond the single-request setting. Even a limited batch-size scaling study comparing TfM against DFlash and EAGLE-3 would help establish the practical applicability and performance characteristics of the proposed method.
  • Limited characterization of favorable hardware and workload regimes

    • Tables 1 and 2 show that the relative performance of TfM varies substantially across models and hardware platforms. Although Section 4.2 provides a latency breakdown of drafting and verification, the analysis primarily explains individual experimental observations rather than establishing a general understanding of when TfM is advantageous. In particular, it remains unclear under which conditions the additional autoregressive drafting overhead is sufficiently offset by improved acceptance and verification efficiency. A more systematic analysis of the relevant model characteristics, hardware constraints, and workload properties would help clarify the regimes in which TfM is preferable to existing approaches, especially where performance crossovers with DFlash occur.
Questions for authors
  • What motivates the inclusion of both GMC and NS verification? Since they appear to achieve similar performance in the reported experiments, are there specific conditions under which one is preferable to the other? It would be helpful to clarify their respective advantages, limitations, and practical trade-offs.
4 comments
AuthorsOpenReview ↗

Response to Reviewer xbU8 (Part 1/2)

We thank the reviewer for their careful reading and constructive feedback. We address their concerns below.

Our answers to W2-W4 rest on one hardware fact, explained in our general response. At batch size 1, verification is bound by memory bandwidth, so its latency grows slowly with the speculative budget. Table 3 of the submission already shows this, and we added Figure 7 to show it directly.

W1 : Insufficient differentiation from existing approaches

Regarding Domino

Domino is contemporaneous work, developed independently of ours (Section 2.1). The two methods differ in three ways.

  • Prefix modeling. Domino’s head sees the backbone state at the current position and a GRU state over the draft tokens. Weaver is a Transformer that attends to the verifier state, all lookahead states, and the draft prefix.
  • Vocabulary projection. Domino learns a separate low-rank correction head. Weaver instead reuses the verifier’s projection, restricted to the top-K candidates given by the factorized prior.
  • Proposal structure. Domino drafts a chain. Weaver builds trees, and we provide tree verification for GDN layers. Efficient tree speculation on Qwen3.6-27B requires such an algorithm, and it is one of the core contributions of our paper.

We also benchmarked Domino on Qwen3.6-27B in SGLang. Domino reaches 5.69 tokens/step and 347.7 tokens/s at its best budget (16), while TfM-GMC reaches 8.26 tokens/step and 388.9 tokens/sec (details in our response to Reviewer cCjE, Q3).

Regarding EAGLE-2

We agree that Algorithm 1 is conceptually similar to the dynamic trees of EAGLE-2. However, it is very important to note that the EAGLE-2 algorithm is incorrect, at least in the form described in the paper! The crucial problem is that EAGLE-2 paper proposes to use the original speculative sampling procedure of Leviathan et al. for verifying the pruned tree. But this is invalid:

  1. Extending speculative sampling of Leviathan et al. to tree-shaped drafts requires adjusting the proposal distributions. EAGLE-2 paper cites Sequoia and SpecInfer, which describe the correct adjustments, but doesn’t mention those adjustments anywhere in the text.
  2. More importantly, the dynamic extension and pruning procedure of EAGLE-2 introduces selection bias, which is incompatible with any verification procedure which relies on likelihood ratios, including SpecInfer, RSSw, Traversal, Univer and others.

Most current implementations of EAGLE-2 and EAGLE-3 fix those issues by switching to the “target-only” verification regime, which is equivalent to our “Naive Sampling” procedure. Naive sampling is theoretically suboptimal: it can reject a token even when the probabilities of the drafter and verifier agree at every position.

We realized that incompatibility between can be alleviated by using a communication-free Gumbel-max coupling, which has a much higher theoretical acceptance rate than naive sampling (see https://arxiv.org/abs/2408.07978). Despite the empirical difference in performance between TfM-NS and TfM-GMC being low, we consider TfM-GMC to be mathematically elegant and simple to implement, which motivates its inclusion in the paper.

W2: Unclear fairness and optimization of baseline configurations

DFlash

A shorter draft has little to gain for DFlash, for two reasons. DFlash performs drafting in a single forward pass, so its latency does not depend on the draft length (Figure 1). Verification latency is flat up to B=16 (Figure 7). A shorter draft therefore saves almost no time, while lowering acceptance length. We confirmed this by sweeping the draft length over {8, 10, 16}: throughput changes by at most 1.9% (table below), far less than the 1.22x gap to TfM.

BBTokens/step (τ\tau)Tokens/sChange vs. B=16B=16
84.327 ± 0.018317.6 ± 2.9−1.4%
104.623 ± 0.032328.2 ± 3.2+1.9%
164.994 ± 0.036322.0 ± 1.9—

Qwen3.6-27B on B200 (BF16, batch size 1), averaged over MTBench, GSM8K, and HumanEval; mean ± SD over 3 seeds. DFlash uses the recurrent GDN kernel.

EAGLE-3

EAGLE-3 was tuned: for each budget, we swept depth and width and reported the best configuration. The submission did not describe this; the grid is given in our response to Reviewer RTrE, Q1. Its budget sensitivity is shown in Figure 5, together with DDTree and TfM. We are also training a stronger EAGLE-3 checkpoint for Qwen3.6-27B (see our response to Reviewer RTrE, Q1).

AuthorsOpenReview ↗

Response to Reviewer xbU8 (Part 2/2)

W3: Lack of evaluation of batch-size scalability
We acknowledge that evaluation across batch sizes could provide useful insights into the draft model research. However, as stated in the introduction, our work specifically targets local inference setups, where the overwhelming majority of queries cannot be batched. We believe that local inference is an underrepresented research area, and the hardware constraints of local inference are sufficiently different from the cloud to warrant special self-contained treatment.

A proper evaluation of our method in a cloud deployment would require targeting a large and sparse Mixture of Experts verifier model in a multi-GPU configuration (a dense 27B or 9B model is not a typical target for the cloud). Such evaluation would require a significant amount of resources, and is beyond the scope of this study.

Nevertheless, an interested reader can consult the DSpark paper (independent concurrent work), which addresses some of the same problems using similar techniques, and provides an excellent analysis of its performance in real-life production serving. Based on similarities between our methods, we believe that, when used as part of a well-configured cloud setup, our approach should provide performance gains similar to DSpark.

Lastly, we disagree that large batch sizes may make the costs of autoregressive drafting more pronounced. In fact, the opposite is true! Autoregressive drafting is memory-bound, which means that as the number of parallel requests grows, the cost of autoregressive iterations remains almost constant! We take full advantage of this fact by parallelizing the autoregressive drafting breadthwise during tree construction

W4: Limited characterization of favorable hardware and workload regimes

We give an analysis of hardware considerations in our general response.

The speedup model (Section 2, Notation) gives a simple rule: TfM is preferable when its relative gain in tokens/step exceeds its relative increase in cycle time. This rule predicts the crossovers in our tables.

  • Model. The larger the verifier, the smaller the share of tree construction in the cycle. On B200, TfM needs 20.6% more tokens/step than DDTree for Qwen3.5-9B but delivers 16.3%, so it loses. For Qwen3.6-27B, it needs 8.3% and delivers 22.1%, so it wins.
  • Hardware. The lower the memory bandwidth, the longer the verification, and the smaller this share becomes. For Qwen3.6-27B, tree construction is 11% of the cycle on B200 but 4% on DGX Spark and M5 Pro (Table 3). On all local devices we tested, TfM wins for both 9B and 27B verifiers (Table 2).
  • Workload. The gain is larger on domains where deep drafts are accepted. As we show in Section 4.1, our advantage over other methods is highest on coding and math prompts. Code, terminal commands and tool calls are more predictable, so agentic coding use cases should benefit from our method more. Additionally, longer context increases the cost of verification while keeping the Weaver cost constant, so on long context tasks we expect our method to have an even larger advantage over methods which trade acceptance rate for lower drafting latency. We are running additional long-context agentic coding benchmarks to prove this claim empirically.

Q1: The motivation of GMC and NS verification

As we described in our comment on the connection to EAGLE-2, tree extensions of speculative sampling require the children of each node to be sampled from the draft distribution. Algorithm 1 selects children by top-c and prunes by score, which breaks this requirement. NS and GMC remain lossless regardless of the tree construction procedure (Section 2.2), so we compare these two.

NS is the simplest but it rejects tokens even when the drafter and verifier are the same model. GMC provides a simple and elegant solution to this problem. We believe that for a good drafter, GMC is always preferable.

Lastly, in our original manuscript, PruneToBudget ranked nodes by their cumulative log-probabilities without accounting for the gumbel noise, which we found to be suboptimal after submission. We have fixed PruneToBudget to use the shared Gumbel noise and are rerunning the GMC experiments. We will post the updated results here in a few days.

We thank the reviewer for deeply engaging with our paper, we found the feedback very helpful for better articulating our motivation and results!

AuthorsOpenReview ↗

Update on EAGLE-3 and GMC

Regarding EAGLE-3 baseline (W2, W4)

We completed the EAGLE-3 training and updated the paper accordingly. Table 1 and Figure 5 now use our retrained checkpoint, and Appendix D (“Matched comparison with EAGLE-3”) describes its training setup and tree configuration.

The retrained EAGLE-3 reaches a speedup of 2.92x on average, compared with 3.35x for DFlash and 4.09x for TfM (Table 1). Retraining improves its tokens/step. However, as predicted by Figure 1, its throughput does not scale with depth (Figure 16).

Regarding GMC (Q1)

We fixed PruneToBudget to account for the shared Gumbel noise. Below, we compare TfM-GMC with TfM-NS on B200, under the same setting as Table 1.

(1.) Qwen3.5-9B

MethodMTBenchShareChatGSM8KMATH500AIME25HumanEvalMBPPLiveCodeBenchAvg.
TfM-GMC(2.05×, 6.09)(1.72×, 5.09)(2.57×, 7.65)(3.02×, 8.91)(2.91×, 8.61)(2.83×, 8.51)(2.89×, 8.70)(2.38×, 7.09)(2.55×, 7.58)
TfM-NS(1.85×, 5.39)(1.53×, 4.43)(2.34×, 6.81)(2.86×, 8.27)(2.75×, 7.96)(2.69×, 7.95)(2.79×, 8.22)(2.20×, 6.41)(2.37×, 6.93)

(2.) Qwen3.6-27B

MethodMTBenchShareChatGSM8KMATH500AIME25HumanEvalMBPPLiveCodeBenchAvg.
TfM-GMC(3.22×, 6.47)(2.49×, 4.98)(4.64×, 9.44)(4.99×, 10.01)(4.77×, 9.56)(4.77×, 9.58)(4.60×, 9.24)(3.91×, 7.87)(4.17×, 8.39)
TfM-NS(3.17×, 6.31)(2.44×, 4.80)(4.52×, 9.11)(4.85×, 9.59)(4.57×, 9.03)(4.65×, 9.23)(4.53×, 9.04)(3.78×, 7.51)(4.06×, 8.08)

(Each cell reports speedup over AR, and tokens/step)

With the fix, TfM-GMC improves average tokens/step over TfM-NS by 9.4% on Qwen3.5-9B and 3.9% on Qwen3.6-27B, and increases average throughput by 7.6% and 2.7%, respectively. To keep the current revision focused, we will incorporate the updated results in the final version.

We hope these clarifications resolve the reviewer’s concerns. We are happy to answer any remaining questions.

Thank you for the detailed response and additional experiments. The new baseline comparisons, tuning analyses, and hardware-related clarifications substantially address my main concerns. While some limitations regarding methodological novelty remain, I find the rebuttal convincing overall and have updated my rating.

Reviewer_cCjEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Trees from Marginals (TfM), a speculative decoding method that builds prefix-conditioned proposal trees from a parallel drafter's predictions. Its lightweight adapter, Weaver, refines DFlash logits over restricted candidate sets. The paper also develops efficient tree verification for Gated DeltaNet layers. Experiments across datacenter and local hardware show improved decoding speed, particularly when verification costs outweigh the added drafting overhead.

Critical strengths
  • The paper explains why accurate marginal predictions alone are not enough, with ablations showing that prefix conditioning accounts for much of Weaver's improvement.

  • Weaver combines parallel predictions with lightweight corrections over a restricted candidate set, improving the use of the tree budget while keeping drafting costs manageable.

  • The Gated DeltaNet formulation and GPU kernels support efficient tree verification where attention masking alone is insufficient.

Critical weaknesses

While the results are encouraging, two design choices would benefit from further empirical justification to clarify their contribution to the method's effectiveness.

  • It is not yet clear how the choice of Top-512 affects overall performance.

    Appendix B.1 shows that the candidate sets have good coverage, but this does not tell readers how changing their size affects acceptance length, tree-construction time, and decoding speed. Testing a few values of K would help explain why 512 is a good choice and whether nearby settings work similarly well.

  • The role of residual prediction could be clearer.

    The existing ablations show that Weaver benefits from prefix information. What remains unclear is how much it helps to add corrections to DFlash's logits, rather than predict candidate logits directly from the same inputs. Comparing these two versions under similar training conditions would help show whether the residual design makes training easier or produces better drafts.

It would help to clarify how Weaver differs from Domino and DSpark, particularly in its use of candidate restriction and path-conditioned tree construction, and how GDN verification supports the overall approach. Direct experimental comparisons would also add value by showing how these design choices translate into draft quality and decoding speed.

Questions for authors
  1. How sensitive is Weaver's end-to-end performance to the candidate-set size (K)? Could the authors evaluate a few values, such as 128, 256, 512, and 1024, on a representative model and hardware configuration, reporting tokens per step, tree-construction latency, and decoding throughput? If (K) is varied only at inference time, please clarify the candidate-set size used during training.

  2. What advantages does residual prediction offer over direct conditional prediction? Could the authors compare separately trained residual and direct-prediction variants while keeping the conditioning inputs, candidate sets, model capacity, and training budget as closely matched as possible? Comparing convergence behavior and final performance would help establish the role of DFlash logits as an explicit prior.

  3. Could the authors explain more clearly how Weaver differs from Domino and DSpark? In particular, how do prefix modeling, candidate restriction, and tree construction contribute to its advantages? If suitable implementations and checkpoints are available, a direct comparison of acceptance length and end-to-end latency would add value.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cCjE (Part 1/2)

We thank the reviewer for their careful and encouraging review, and for their concrete suggestions for additional ablations. We address their questions below.

Q1: How sensitive is Weaver's end-to-end performance to the candidate-set size (K)?
We first clarify that K is fixed at 512 during training and is not intended to be varied at inference time. Nevertheless, we show in additional experiments that end-to-end performance remains almost constant for K∈{128,256,512,1024}K \in \lbrace 128, 256, 512, 1024 \rbrace.

We ran TfM-GMC with B=64B=64 for Qwen3.6-27B on B200, varying K at inference with the same Weaver checkpoint. “Tree” is the tree-construction part of Tdraft(B)T_{\mathrm{draft}}(B). ± denotes the SD over 3 seeds.

DatasetKTokens/stepTree (ms/round)Tokens/s
MTBench1286.277 ± 0.0582.096 ± 0.039301.4 ± 2.0
MTBench2566.273 ± 0.0592.159 ± 0.045300.2 ± 3.9
MTBench5126.321 ± 0.0162.203 ± 0.016301.8 ± 1.0
MTBench10246.357 ± 0.0212.360 ± 0.016299.3 ± 2.4
GSM8K1289.026 ± 0.0832.086 ± 0.024425.7 ± 4.3
GSM8K2569.078 ± 0.0362.140 ± 0.040426.5 ± 3.3
GSM8K5129.169 ± 0.1132.198 ± 0.016430.7 ± 3.5
GSM8K10249.135 ± 0.2132.355 ± 0.018422.9 ± 10.4
HumanEval1289.290 ± 0.0262.078 ± 0.016443.3 ± 3.2
HumanEval2569.288 ± 0.0492.122 ± 0.016443.5 ± 1.5
HumanEval5129.302 ± 0.0402.202 ± 0.013442.4 ± 1.5
HumanEval10249.294 ± 0.0082.359 ± 0.018437.7 ± 3.1

From K = 128 to 1024, tree-construction latency grows by about 13%, while throughput stays within noise.

Q2: What advantages does residual prediction offer over direct conditional prediction?
We have added a training ablation on Qwen3-4B to Section 4.3 (“Ablation study: weaver architecture”) of the revised paper. We compare residual and direct prediction with and without conditioning on hdflashdh_{\mathrm{dflash}}^d, using 700K training completions and evaluating acceptance length on MTBench with an R16W4C4 tree.

We find that residual prediction accelerates convergence and stabilizes early training. We therefore retain the residual parameterization for faster convergence. The variants without conditioning on hdflashdh_{\mathrm{dflash}}^d also test whether Weaver benefits from the factorized drafter beyond candidate restriction. At 700K completions, adding this conditioning improves acceptance length as follows:

PredictionGain from hdflashdh_{\mathrm{dflash}}^d
Residual+0.208
Direct+1.623

The improvement under direct prediction shows that the factorized drafter’s hidden states provide useful information beyond candidate restriction. Overall, Weaver performs best with both components enabled.

AuthorsOpenReview ↗

Response to Reviewer cCjE (Part 2/2)

Q3: Could the authors explain more clearly how Weaver differs from Domino and DSpark?

We appreciate the suggestion of direct comparisons against Domino and DSpark. We first clarify that Weaver was developed independently of Domino and DSpark, which we discuss in Section 2.1 as contemporaneous work.

MethodPrefix ModelingVocabulary ProjectionTree Construction
DominoDominoHead(backbone_state[i], GRU(previous_token, recurrent_state))Sequential low-rank vocabulary projectionChain
DSpark (Markov)MarkovHead(previous_draft_token)Sequential low-rank vocabulary projectionChain
DSpark (RNN)RNN(backbone_state[i], previous_draft_token, recurrent_state)Sequential low-rank vocabulary projectionChain
WeaverTransformer(verifier_state, backbone_state[:], draft_prefix)Sequential truncated vocabulary projectionChain or Tree

Weaver uses a Transformer to condition jointly on the verifier state, all lookahead states, and the draft prefix. Its output side is also different. Domino and DSpark learn a separate low-rank correction head. Weaver instead reuses the verifier’s vocabulary projection, restricted to the top-K candidates given by the factorized prior. We believe Domino and DSpark could likewise be combined with our tree construction and verification, since their prefix-conditioned predictions can be applied along each tree path.

We could not benchmark against DSpark, because its released checkpoints target other verifiers and none is available for Qwen3.6-27B. We therefore compare only with Domino in SGLang, reporting acceptance length and tokens/s.

DatasetMethodTokens/stepTokens/s ± SD
MTBenchDomino (B=8)3.729265.2 ± 2.9
MTBenchDomino (B=10)3.921266.4 ± 2.6
MTBenchDomino (B=16)4.250261.0 ± 3.2
MTBenchTfM-GMC (B=64)6.321299.5 ± 1.4
GSM8KDomino (B=8)5.044356.1 ± 14.2
GSM8KDomino (B=10)5.628378.7 ± 2.8
GSM8KDomino (B=16)6.607399.8 ± 10.7
GSM8KTfM-GMC (B=64)9.169427.4 ± 4.9
HumanEvalDomino (B=8)4.933352.5 ± 6.7
HumanEvalDomino (B=10)5.406370.2 ± 4.5
HumanEvalDomino (B=16)6.198382.4 ± 6.0
HumanEvalTfM-GMC (B=64)9.302439.9 ± 0.8
Avg.Domino (B=8)4.569324.6 ± 7.2
Avg.Domino (B=10)4.985338.4 ± 3.1
Avg.Domino (B=16)5.685347.7 ± 6.5
Avg.TfM-GMC (B=64)8.264388.9 ± 2.0

(Qwen3.6-27B on B200, under the same settings as Table 1 of our paper)

Note that the two methods apply recurrent correctors in different ways. Domino targets chain drafts, while our work uses trees.

We thank the reviewer again for their constructive feedback. We hope the additional ablations and clarifications address their concerns raised.

Thank you for your response.

Based on the additional experimental results and my previous assessment of the paper, I am maintaining my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We are glad to see that our work received a thorough engagement from the reviewers, and we found the feedback to be very high quality. We want to thank all reviewers for putting their valuable time into it. We greatly appreciate it.

The submission left its hardware premise implicit, and several questions trace back to it. The revision now states it explicitly, and adds two figures: Figure 1 (drafting cost vs depth) and Figure 7 (end-to-end verification latency vs budget). We summarize it here; our individual responses refer to it.

Hardware Premise (to all reviewers and the AC)

Our work targets local inference, where queries cannot be batched. Consumer chips now include dedicated units for low-precision matrix multiplication (e.g., NVIDIA Tensor Cores, Apple Neural Accelerators, various NPUs). Their arithmetic throughput per byte of memory bandwidth is high and is growing with new generations.

ChipPrecision usedMeasured throughputMemory bandwidthOperations/byte
M4 ProBF168.0464 TFLOP/s273 GB/s29.47
M4 Max — 40-core GPUBF1616.0883 TFLOP/s546 GB/s29.47
M5 Max — 40-core GPUINT8132.6172 TOP/s614 GB/s215.99
M6 — 24/32 GB RAMINT842.6244 TOP/s170 GB/s250.73
M6 — 16 GB RAMINT842.6244 TOP/s153 GB/s278.59

As a result, for drafts whose size is less than the width of the matrix multiplication unit tile (*), the verification latency as a function of the draft size remains almost flat! Figure 7 clearly demonstrates this counterintuitive fact. This suggests that efficient local inference requires large speculative budgets.

The speedup from speculative decoding is limited by tokens/step (τ\tau), which is bounded by the depth of the draft tree: τ≤D+1\tau \le D + 1, therefore we need a drafter that can provide high-quality drafts at large depths. This is the starting point of our work.

How do existing drafters behave as depth grows? We added Figure 1 (Section 2.1) to answer this. It shows drafting memory traffic and latency as a function of draft depth. (**)

Purely autoregressive methods (e.g., EAGLE-3, MTP) scale poorly with depth. Every drafting iteration incurs the cost of the full DRAM->SRAM transfer for the entire draft model and the vocabulary projection matrix. The latter alone accounts for 36-75% of the traffic (EAGLE-3 with truncated vocabulary and native MTP, respectively). This effectively caps both the drafter capacity and speculation depth.

Factorized drafters (e.g., DFlash, PARD) avoid this cost by drafting in a single iteration. However, their effective depth is low, as their predictions at later positions are unlikely to get accepted (Figure 2 in the revised paper). Prior work that attempts to overcome this limitation by trading depth for width (namely DDTree) yields only limited improvements in practice: 1.10x over DFlash at 4x the budget (Table 1).

Weaver is designed to combine the best of both worlds. It pays for the large factorized drafter and the full vocabulary projection once, and then adds a small prefix-dependent correction at each depth (Figures 1 and 2), breaking the fundamental limitation of purely factorized drafters. Our adaptive tree construction algorithm produces long chain drafts in predictable domains like code and math, where deeper tokens are likely to be accepted, and hedges by increasing width in hard-to-predict cases.

The reviewers’ questions made the revision much clearer, and we are grateful for them.

(*) Theoretical roofline analysis suggests that the verification latency should remain flat for even longer inputs, but performance of the real-world kernels tends to be constrained by other factors like register use and occupancy.

(**) Latency is measured on a B200. On local hardware, where memory bandwidth is lower, the gap between drafters is even larger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.