- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
TreeSpark extends DSpark's semi-autoregressive drafter to propose a budgeted tree of candidate continuations rather than a single chain. It reuses the draft backbone's parallel predictions and applies lightweight, parent-conditioned corrections to expand branches, which the target model then verifies. To align training with this inference procedure, TreeSpark replays the drafter's tree-construction policy and adds survival and coverage objectives to the existing distillation losses. These objectives encourage useful branches to remain in the tree and target-favored candidates to be considered.
Experiments with Qwen3-4B and Qwen3-8B show that TreeSpark improves acceptance length over DSpark and raises throughput across single-request and concurrent serving settings. In concurrent serving, throughput improves by approximately 13--17%; on single-request writing benchmarks, gains reach 22--25%. The results indicate that training for tree construction can improve speculative decoding, although the increase in accepted tokens per round does not translate directly into an equally large throughput gain, since constructing and verifying a tree also incurs cost.
Critical strengths
The paper's strength is its use of survival and coverage losses to train the drafter for the tree it will construct at inference. The coverage loss helps target-favored candidates to be included at expanded parents, while the survival loss rewards branches that remain in the packed tree. The two objectives address candidate selection both before and after the tree budget is applied. The DSpark+Tree comparison uses the same tree-based inference setup as TreeSpark but omits the survival and coverage losses, helping show what these training objectives add beyond switching from a chain to a tree.
Critical weaknesses
W1. Reuse of reference-path target distributions may mis-supervise non-reference branches. During replay, TreeSpark reuses the same target distribution for every parent at a given depth. The paper notes that this distribution is exact for the reference context but only a proxy for alternative branches. Because the target’s preferred continuation may change as a branch diverges from the reference, this approximation may give the survival and coverage losses inaccurate signals, particularly at greater depths. The paper does not quantify the proxy error, analyze how it changes with branch depth or divergence, or evaluate the sensitivity of its results to this approximation.
W2. Modest throughput gains and limited evaluation at high concurrency. TreeSpark improves concurrent throughput over DSpark by about 13--17% at total concurrency C=6 and C=12, and by 12.6% at C=36. The latter result aggregates three model replicas, each serving up to 12 requests; it therefore does not show how TreeSpark performs with 32 or more concurrent requests per replica. Since speculative-decoding throughput can depend on batch size and hardware utilization, its gains in higher-throughput production settings could be much smaller. The paper does not evaluate this regime, and the practical benefit at larger per-replica batch sizes is unclear.
W3. The EAGLE-3 comparison needs clarification. The paper says that EAGLE-3 produces a single proposal path per round, while TreeSpark can pack up to 32 non-root candidates. It is unclear whether the EAGLE-3 baseline used its tree-based decoding configuration. If tree expansion was disabled, the comparison penalizes EAGLE-3; the paper should report the exact configuration and, also compare against a tree-enabled baseline.
W4. The evaluation provides limited evidence about its robustness and generalizability. Throughput results are averaged over three runs, but the paper reports no error bars or significance tests, making it difficult to assess the reliability of the more modest gains. The evaluation covers only two target sizes from one model family, Qwen3-4B and Qwen3-8B, and uses Ascend 910B hardware. It therefore leaves open whether the results generalize to larger models, other model families, or other accelerator platforms, such as NVIDIA and AMD GPUs.
Questions for authors
Q1. Can you quantify how the reused reference-path target distributions differ from branch-specific target distributions as branch depth or divergence increases? Do you have results showing how this approximation affects the survival and coverage losses or the resulting acceptance length?
Q2. Can you report TreeSpark's performance on a single replica with continuous batching and target batches of 32--64 requests? Please clarify how consistently the serving system maintains batches in this range, report the observed number of requests per model call, and show how TreeSpark’s throughput gains over DSpark change at these batch sizes.
Q3. The paper states that EAGLE-3 produces a single proposal path per round, while TreeSpark uses a tree with up to 32 non-root candidates. Was tree expansion enabled for the EAGLE-3 baseline? If not, could you provide results for a tree-enabled EAGLE-3 configuration with a comparable verification budget?
Q4. Can you report variability across the three throughput runs, such as standard deviations or confidence intervals? Do the results generalize to other target model families, larger model sizes, and NVIDIA or AMD hardware? Q4. Can you report variability across the three throughput runs, such as standard deviations or confidence intervals? Do the results hold for other target model families or (larger) sizes, and on NVIDIA and AMD platforms?