acceptodds
Withdrawn from ICLR 2027

TreeSpark: From Acceptance Scores to Useful Work in Conditional Speculative Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Speculative decoding accelerates autoregressive language models by using a small drafter to propose tokens that a larger target verifies in parallel. Diffusion-based drafters can produce a whole block in one forward pass, and lightweight conditional heads can model dependencies within that block. Extending these proposals to trees covers more possible continuations, but the extra construction and verification work can outweigh the benefit of accepting more tokens. We study this trade-off with TreeSpark, a conditional-tree decoding framework built on the DSpark drafter. TreeSpark preserves a primary chain and adds optional leaves using already computed parent distributions and the drafter’s existing confidence head. A threshold chosen from the active batch size controls leaf admission, while ordered rejection verifies the sampled tree. We evaluate this design against chain and fixed-tree controls in the same optimized inference engine. On Qwen3-4B, selected fixed trees improve throughput over chain by 5.0–8.6%; confidence-guided trees gain 5.8–6.7% in sampling and 2.4–3.3% under fixed-rate serving. Matched static and constant-threshold controls do not establish a consistent additional benefit from adaptation. The results show why useful tree allocation must account for executed work, alongside acceptance predictions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, large language model inference, conditional draft trees, inference efficiency, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pZZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TreeSpark studies conditional draft-tree allocation using DSpark, separating acceptance prediction, branch selection, and executed work. It preserves a primary chain and adds optional leaves using existing confidence scores, with batch-dependent thresholds controlling admission.

Critical strengths
  1. Careful comparisons against independently tuned static trees and constant-threshold controls, with paired confidence intervals.
  2. Useful accounting of retained nodes, prepared parent distributions, and executed work.
  3. Transparent reporting of negative results and limitations.
Critical weaknesses

1.Limited methodological novelty. The conditional drafter, tree topology, confidence head, and recursive rejection mechanism build on existing work. The additional policy primarily combines a heuristic confidence score with a two-bin threshold rule. The paper does not sufficiently establish a substantial algorithmic advance beyond these components.

  1. No consistent incremental benefit over simpler controls. Although confidence-guided trees improve throughput over chain decoding, their gains relative to the optimized static tree are −1.38% and +0.63% in the primary sampling comparisons, with both confidence intervals containing zero. Fixed-rate serving likewise does not establish an advantage over the constant high-bin threshold. The results therefore support efficient tree construction more strongly than the proposed adaptive allocation mechanisms.

  2. Limited generality of the diagnostic contribution. The study offers useful negative findings, but the evidence centers on one GPU platform and the DSpark architecture. Moreover, the load controller spends approximately 98–100% of measured request-rounds in its high-batch bin, limiting what these experiments establish about adaptation across changing operating conditions. The paper provides limited guidance on when adaptive selection should outperform a well-tuned static policy.

Questions for authors

1.What is the principal transferable insight beyond the established need to balance acceptance gains against execution costs? 2.Do any evaluated conditions demonstrate a reproducible advantage over both the strongest static tree and constant-threshold controls?

Reviewer_5wQ1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper builds on DSpark, and suggests building a tree instead of a single chain of tokens. The main chain is constructed just like in DSpark, and extra leaves along the chain tokens are added based on the confidence scores of DSpark. Compared to DSpark, TreeSpark achieves roughly 5-9% throughput improvement.

Critical strengths
  1. The method is simple and easy to implement.
  2. In the experiments, there is speedup compared to DSpark.
Critical weaknesses
  1. The paper is really hard to read.
  2. There is a limited novelty in the paper. Previous work already combined DSpark-like drafters with draft trees. Some were mentioned in the related work section.
  3. No experimental comparison to previous methods that uses draft trees.
  4. Experiments were not conducted on optimized inference frameworks such as SGLang or vLLM. Hence, the real world gains are not clear.
Questions for authors

Could you please share a comparison to other methods that build draft trees and are based on DSpark?

Reviewer_1u2aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the relationship between acceptance rate/length and end-to-end speedup in speculative decoding and proposes TreeSpark. TreeSpark keeps the drafter's primary chain and adds at most one second-draw leaf to each primary parent. The leaves reuse the conditional distributions already computed for the primary chain, so they require no additional parent projections. Leaves are admitted by a surrogate score built from DSpark's released confidence head, with a threshold chosen from the current active batch size. Verification uses ordered recursive rejection, and the paper states the conditions under which the target distribution is preserved.

The paper also splits draft-side cost into three counters: retained parents, prepared projection rows, and executed rows including padding. On Qwen3-4B with a single H100, fixed trees improve throughput over the chain by 5.0–8.6%, and confidence-guided trees by 5.8–6.7% under sampling. Calibrated best-first trees achieve better greedy calibration and longer accepted length, yet their throughput is lower than the chain's.

Critical strengths

• The paper measures node count, parent-projection cost and accepted length separately. Table 2 clearly shows that a policy with a longer accepted length can still have lower throughput, possibly because it executes more parent projections. • The theoretical conditions and the evaluation protocol are rigorous. The conditions under which sampled trees preserve the target distribution are stated explicitly, policies are frozen on validation before testing, results use paired 95% confidence intervals, and null results are reported honestly.

Critical weaknesses

• The static tree has higher point estimates than the confidence-guided tree in almost every setting (e.g., +8.2% vs. +6.7% at T=1/B4, and +7.3% vs. +5.9% on Qwen3-8B), and the authors acknowledge that this static topology is "not a new topology" (l.205–206). Yet the abstract still presents the 5.8–6.7% and 2.4–3.3% gains of confidence-guided trees over the chain as the main results (l.24–26). These numbers mainly reflect the benefit of the existing static tree, not of the proposed method. • The core message, that accepted length does not directly translate into throughput and that execution cost must be measured, is not new, and the paper does not explain what it adds. The title promises a path "from acceptance scores to useful work", but the paper proposes no cost-aware allocation rule and only arrives at the known conclusion that executed work should be measured. • PCTree, DominoTree, DARTree, CaDDTree and BASTION are closely related methods discussed in the related work, yet all comparisons in the paper are among the authors' own internal variants. • Abbreviations such as chain7, fixed14, B4/B8, native-batch and same-builder are used without definition.

Questions for authors

• Relative to the analysis of Liu et al. [1], which shows that acceptance rate and actual speedup can diverge, what new findings does this paper contribute? • How does TreeSpark compare with PCTree, DominoTree, CaDDTree and BASTION in the same engine? How do the results change at batch size 1 and at batch sizes of 32 or more? • Within the chain-plus-leaves builder and under the same leaf budget, how do calibrated scores compare with the confidence head? [1] Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, and Alvin Cheung. Speculative Decoding: Performance or Illusion? In Proceedings of Machine Learning and Systems (MLSys), 2026.

1 comment

Thanks for the response. I still believe the submission is below the bar for ICLR. It would be great if the authors could incorporate the explanation and add the discussion/experiments on the existing methods to the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.