acceptodds
Under review as a conference paper at ICLR 2027

TreeSpark: Learning to Draft the Trees You Verify for Speculative Decoding

10 views

Abstract

Parallel drafters such as DFlash and DSpark reduce proposal latency, but early mismatches limit how much of each block can be accepted. Retaining alternatives introduces parent states and candidate choices that reference-conditioned training does not directly supervise. We introduce TreeSpark, a unified training and inference framework extending the semi-autoregressive DSpark architecture. It constructs parent-conditioned candidate trees from one backbone pass and replays their expansion, pruning, and budgeted packing during training. Survival negative log-likelihood weights retained edges by reference-derived path mass, while a capped coverage hinge promotes target-important alternatives near local selection boundaries. Both objectives reuse reference-chain target distributions without additional target forward passes. We evaluate greedy decoding with Qwen3-4B and Qwen3-8B on nine general benchmarks and six writing domains. Across targets and suites, TreeSpark improves mean acceptance length over DSpark by 23.4% to 31.6% and mean single-request throughput by 15.1% to 25.0%. Batched Qwen3-4B throughput improves by 12.6% to 17.2%. At fixed attention capacity and inference policy, the proposed supervision further improves mean acceptance length by approximately 6.4% and mean system throughput by 4.2% to 5.0% on Qwen3-4B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative decoding, Large language model inference, Tree-structured drafting, Semi-autoregressive generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7APkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TreeSpark extends DSpark's semi-autoregressive drafter to propose a budgeted tree of candidate continuations rather than a single chain. It reuses the draft backbone's parallel predictions and applies lightweight, parent-conditioned corrections to expand branches, which the target model then verifies. To align training with this inference procedure, TreeSpark replays the drafter's tree-construction policy and adds survival and coverage objectives to the existing distillation losses. These objectives encourage useful branches to remain in the tree and target-favored candidates to be considered.

Experiments with Qwen3-4B and Qwen3-8B show that TreeSpark improves acceptance length over DSpark and raises throughput across single-request and concurrent serving settings. In concurrent serving, throughput improves by approximately 13--17%; on single-request writing benchmarks, gains reach 22--25%. The results indicate that training for tree construction can improve speculative decoding, although the increase in accepted tokens per round does not translate directly into an equally large throughput gain, since constructing and verifying a tree also incurs cost.

Critical strengths

The paper's strength is its use of survival and coverage losses to train the drafter for the tree it will construct at inference. The coverage loss helps target-favored candidates to be included at expanded parents, while the survival loss rewards branches that remain in the packed tree. The two objectives address candidate selection both before and after the tree budget is applied. The DSpark+Tree comparison uses the same tree-based inference setup as TreeSpark but omits the survival and coverage losses, helping show what these training objectives add beyond switching from a chain to a tree.

Critical weaknesses

W1. Reuse of reference-path target distributions may mis-supervise non-reference branches. During replay, TreeSpark reuses the same target distribution for every parent at a given depth. The paper notes that this distribution is exact for the reference context but only a proxy for alternative branches. Because the target’s preferred continuation may change as a branch diverges from the reference, this approximation may give the survival and coverage losses inaccurate signals, particularly at greater depths. The paper does not quantify the proxy error, analyze how it changes with branch depth or divergence, or evaluate the sensitivity of its results to this approximation.

W2. Modest throughput gains and limited evaluation at high concurrency. TreeSpark improves concurrent throughput over DSpark by about 13--17% at total concurrency C=6 and C=12, and by 12.6% at C=36. The latter result aggregates three model replicas, each serving up to 12 requests; it therefore does not show how TreeSpark performs with 32 or more concurrent requests per replica. Since speculative-decoding throughput can depend on batch size and hardware utilization, its gains in higher-throughput production settings could be much smaller. The paper does not evaluate this regime, and the practical benefit at larger per-replica batch sizes is unclear.

W3. The EAGLE-3 comparison needs clarification. The paper says that EAGLE-3 produces a single proposal path per round, while TreeSpark can pack up to 32 non-root candidates. It is unclear whether the EAGLE-3 baseline used its tree-based decoding configuration. If tree expansion was disabled, the comparison penalizes EAGLE-3; the paper should report the exact configuration and, also compare against a tree-enabled baseline.

W4. The evaluation provides limited evidence about its robustness and generalizability. Throughput results are averaged over three runs, but the paper reports no error bars or significance tests, making it difficult to assess the reliability of the more modest gains. The evaluation covers only two target sizes from one model family, Qwen3-4B and Qwen3-8B, and uses Ascend 910B hardware. It therefore leaves open whether the results generalize to larger models, other model families, or other accelerator platforms, such as NVIDIA and AMD GPUs.

Questions for authors

Q1. Can you quantify how the reused reference-path target distributions differ from branch-specific target distributions as branch depth or divergence increases? Do you have results showing how this approximation affects the survival and coverage losses or the resulting acceptance length?

Q2. Can you report TreeSpark's performance on a single replica with continuous batching and target batches of 32--64 requests? Please clarify how consistently the serving system maintains batches in this range, report the observed number of requests per model call, and show how TreeSpark’s throughput gains over DSpark change at these batch sizes.

Q3. The paper states that EAGLE-3 produces a single proposal path per round, while TreeSpark uses a tree with up to 32 non-root candidates. Was tree expansion enabled for the EAGLE-3 baseline? If not, could you provide results for a tree-enabled EAGLE-3 configuration with a comparable verification budget?

Q4. Can you report variability across the three throughput runs, such as standard deviations or confidence intervals? Do the results generalize to other target model families, larger model sizes, and NVIDIA or AMD hardware? Q4. Can you report variability across the three throughput runs, such as standard deviations or confidence intervals? Do the results hold for other target model families or (larger) sizes, and on NVIDIA and AMD platforms?

Reviewer_nVcTOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TreeSpark extends DSpark with parent-conditioned tree drafting and two auxiliary training losses. Training replays candidate expansion, pruning, and packing to better match the inference policy. Experiments on Qwen3-4B and Qwen3-8B report higher acceptance and throughput than chain-based baselines.

Critical strengths
  1. Reusing one backbone pass across branches makes sense. The method can keep alternatives without rerunning the whole drafter for each one.

  2. I appreciate the DSpark + Tree control. It gives a clearer picture of what the new training losses add once the tree and model capacity are held fixed.

  3. The benchmarks cover math, code, dialogue, and writing, so the reported gains are not limited to one type of output.

Critical weaknesses
  1. I am not convinced by the novelty. Training a drafter to match its inference-time tree policy has already been studied in GTO (ICLR 2026) [1], which is missing from the discussion. TreeSpark brings this direction to the DSpark architecture, but it is unclear whether the proposed losses offer a substantial advance over existing tree-aware training. A direct comparison with GTO would be more informative than framing training-inference alignment as the main contribution.

  2. Most headline comparisons give TreeSpark a much larger verification budget: up to 33 target input positions versus eight for the chain baselines. Its attention head dimension is also increased from 128 to 192. These comparisons mix the effects of branching, capacity, and training. The matched DSpark + Tree control is useful, but the remaining throughput gain is only about 4-5%. I would like the main evaluation to focus on matched-budget tree baselines, including tree-based EAGLE, with measurement uncertainty reported for these smaller gains.

  3. The high-concurrency case is not convincing yet. Extra branches increase target computation and KV storage, so longer accepted paths may stop paying off as the device becomes saturated. The reported concurrency of 36 is actually three independent workers with 12 requests each, rather than one instance serving 36 requests. Without a single-instance concurrency sweep and a clear account of the optimized serving implementation, I am unsure whether the tree remains worthwhile under realistic high load.

[1] Shijing Hu, Jingyang Li, Zhihui Lu, and Pan Zhou. Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding. International Conference on Learning Representations (ICLR), 2026.

Questions for authors

Same as the weaknesses above.

Reviewer_wCKpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TreeSpark extends DSpark [1] so that a small draft model can propose several continuations instead of one. A single pass predicts seven positions. A small correction head uses the previous proposed token to expand each branch. The target model checks up to 32 proposed tokens and follows the path that matches its own greedy choices. During training, the drafter replays how it expands, prunes, and packs the tree under the node limit. One loss rewards likely paths that make it into the tree. A second loss promotes tokens near the cutoff for a local branch. Both use target scores saved on one reference continuation. On Qwen3-4B and Qwen3-8B, the authors report 15.1% to 25.0% higher single-request throughput than chain DSpark, averaged separately over nine general benchmarks and six writing domains (Table 1).

Critical strengths
  1. The training method needs no extra target-model passes. Group Tree Optimization already trains on trees built with the inference policy, but it uses the target to score the proposed branches [2]. Table 3 of DARTree shows that a pretrained DSpark Markov head can build useful trees on Qwen3-4B without retraining [3]. What is useful here is the step between those results: TreeSpark trains that kind of drafter on its own branches using scores it already has. Replay adds work for the draft model, but not for the target model (Sections 3.2–3.4; Appendix D.2). I would make this distinction explicit in Section 2.

  2. The authors test whether training helps when the verifier budget stays fixed. The other methods in Table 1 propose one path each. The target checks up to eight input positions for a chain, versus 33 for TreeSpark. Figure 3 gives a stronger test: both tree models have the same attention size and the same 32 candidate slots. With both losses, Qwen3-4B commits 5.25 rather than 4.93 tokens per round, including the bonus token. Mean system throughput rises by 4.2% at 12 concurrent requests (283.54 to 295.34 tokens/s) and by 5.0% at 36 requests split across three workers. The model makes more progress per round without a larger tree.

Critical weaknesses
  1. After a branch leaves the reference, its label comes from a different history. Suppose the drafter proposes B where the reference says A. Equation 6 scores the children of B using the target's prediction after A. If the reference is greedy, that branch cannot itself be accepted at this point. If the reference was sampled, B could be the greedy choice, but its children would still get labels from A. The authors call these scores a proxy. The identity in Appendix C.1 is for sampled paths with true branch probabilities, and Appendix C.4 gives no measured agreement between the proxy and the verifier on alternative branches. This leaves open why the training helps; it does not invalidate greedy verification.

  2. The evidence is stronger for the survival loss than for the coverage loss. On Qwen3-4B with 12 concurrent requests, survival alone gives 293.36 tokens/s; adding coverage gives 295.34, a gain of 0.67%. Coverage alone gives 281.89 tokens/s, compared with 283.54 with neither loss (Table 4; Appendix E.5). Appendix E.2 gives means from three timing runs but no range. The matched tests are also limited to the 4B model on the general suite. I would credit the combined gain, but the data do not yet tell me that the smaller benefit from coverage is repeatable.

References:

[1] Xin Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

[2] Shijing Hu, Jingyang Li, Zhihui Lu, and Pan Zhou. Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding

[3] Tianyi Li, Yaxin Luo, Xinyi Shang, and Zhiqiang Shen. DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees

Questions for authors
  1. Were the saved reference continuations generated greedily or sampled? If you checked any held-out trees, how often did the saved top token agree with the target's actual top token after the first branch away from the reference?
  2. For the 12-request comparison, what were the three throughput readings for the matched control, survival alone, and both losses? If the existing logs include committed tokens per round for survival alone, did coverage improve that number as well?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.