acceptodds
Under review as a conference paper at ICLR 2027

From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding

Abstract

Speculative decoding accelerates LLM inference only when drafted continuations survive target-model verification. Semi-autoregressive drafters such as DSpark predict a token block with one backbone forward and refine it with a lightweight Markov head. However, DSpark decodes this block as a chain, so an early mismatch invalidates the remaining suffix and limits the benefit of large draft blocks. We show that the conditional structure already learned by DSpark can support multiple parent-consistent continuations without retraining or additional backbone passes. We introduce Parent-Conditioned Drafting Tree (PCTree), which uses the pretrained Markov head to score alternative children separately for each parent and allocates a fixed verification budget to the most probable paths. This converts DSpark's linear draft into a tree while preserving its one-pass parallel backbone. Across Qwen3-{4B,8B,14B} and nine benchmarks, at , measured speedup gains over autoregressive (AR) decoding, relative to matched DSpark, range from % to %. On Qwen3-4B GSM8K at , \method increases mean acceptance length from to and three-run mean AR speedup from to . These show that parent-conditioned branching can turn conditional capacity already present in a semi-autoregressive drafter into end-to-end inference gains through an inference-only change.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, Tree-Structured Decoding, LLM Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iqS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PCTree converts DSpark’s chain drafts into budgeted trees by applying its pretrained Markov head to multiple parents. It requires no additional training or backbone passes and improves acceptance length and end-to-end speed over matched DSpark baselines.

Critical strengths
  1. A practical inference-only modification with consistent improvements across three model sizes and nine benchmarks.
  2. The Shared-Markov control demonstrates that parent-specific conditioning improves acceptance beyond branching alone.
  3. The paper reports actual latency gains and acknowledges tree-construction overhead.
Critical weaknesses
  1. Limited methodological novelty. Parent conditioning is already provided by DSpark, while path-probability scoring, frontier pruning, and budgeted tree selection follow established dynamic-tree methods, particularly EAGLE-2. DDTree also establishes tree construction from parallel draft distributions. The main contribution therefore appears to be a straightforward adaptation of existing tree-search techniques to DSpark’s conditional head. The paper does not sufficiently establish a nontrivial algorithmic advance beyond this combination.

  2. The distinct contribution is not fully validated in wall-clock terms. Table 4 shows higher acceptance than the Shared-Markov tree but does not report their relative latency. Consequently, the end-to-end benefit specifically attributable to parent conditioning, beyond ordinary branching, remains unclear.

Questions for authors
  1. What technical obstacle requires a novel solution beyond applying established dynamic-tree construction to DSpark?
  2. Do existing timing measurements show a net speed advantage over the Shared-Markov tree?
1 comment

We thank the reviewer for precisely identifying the novelty boundary. We agree that PCTree is a direct adaptation of established dynamic-tree construction to DSpark, rather than a new dynamic-tree algorithm. In particular, frontier expansion, joint path scoring, pruning, and budgeted prefix-closed selection follow prior dynamic-tree methods such as EAGLE-2. We will revise the paper to make this relationship explicit and substantially narrow the novelty claim.

Our contribution is practical and empirical: DSpark’s pretrained Markov head can be reused, without an additional backbone pass or retraining for tree expansion, to score branches conditioned on their parent tokens. The controlled Shared-Markov comparison shows that this simple conversion yields a measurable end-to-end gain beyond ordinary tree branching. We do not claim a fundamentally new tree-search algorithm.

We added the requested wall-clock comparison. For the B=16B=16 study, DSpark chain, Shared-Markov, and PCTree all use our own target-matched DSpark B=16B=16 checkpoint, trained on the same training dataset as the corresponding DSpark setting; only the decoding/tree-construction policy differs. The target is Qwen3-4B; evaluation uses GSM8K, BF16, greedy decoding, k=4k=4, and N=32N=32.

MethodAcceptance τ\tauRounds/sampleTarget verify (ms/round)Total (ms/round)Tok/sAR speedup
DSpark chain9.12428.35022.15326.664330.875.975.97\times$$
Shared-Markov tree10.22425.27021.06629.656330.735.975.97\times$$
PCTree11.07023.37021.09729.607358.396.476.47\times$$

Controlled effect of parent-conditioned scoring. Under the same DSpark checkpoint, target, verifier, tree budget, and implementation, Shared-Markov and PCTree differ only in whether each tree parent is rescored by DSpark’s Markov head. Parent-conditioned scoring improves acceptance from 10.224 to 11.070, reduces rounds/sample from 25.270 to 23.370, and increases throughput from 330.73 to 358.39 tok/s (+8.4%). The target-verification and total per-round latencies remain essentially unchanged. This result does not claim a new tree-search algorithm; it validates the practical benefit of applying DSpark’s existing parent-conditioned scores to tree branches.

The result is consistent at B=7, where all controls use the same official dspark_qwen3_4b_block7 checkpoint: Shared-Markov versus PCTree improves acceptance from 6.800 to 7.247, reduces rounds/sample from 37.520 to 35.250, and improves throughput from 230.40 to 245.97 tok/s (+6.8%).

Reviewer_fndAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work extends DSpark to tree-based speculative decoding via the PCTree framework. The draft trees are constructed from DSpark's pretrained Markov head and backbone logits, and there is no retraining the drafter or additional backbone passes. The experiments show improved acceptance lengths and speedups over DSpark.

Critical strengths
  1. I find the idea of using DSpark's Markov scores for tree drafting to be simple and effective. Constructing draft trees without any need to retrain the drafter is a promising approach for improving acceptance length. The experimental results validate this, showing consistent improvements over DSpark.
Critical weaknesses
  1. The main weakness I see is the novelty of the score-based tree construction, which is similar to EAGLE-2's dynamic tree construction. Also, the EAGLE-3 and DFlash comparisons are not under matched settings, and DFlash+DDTree has a different drafter, so it is unclear how competitive PCTree is against existing tree-based approaches.
Questions for authors
  1. Can the authors provide matched latency comparisons with EAGLE-3?

  2. Can the authors provide a latency breakdown in Table 9 which includes the target verification overhead?

2 comments

We thank the reviewer for the constructive comments. We agree that the tree-search components follow established dynamic-tree methods; our claim is limited to the practical, training-free adaptation to DSpark and its controlled empirical benefit.

Matched EAGLE-3 comparison. We added a comparison using official dspark_qwen3_4b_block7 for PCTree and official eagle3_qwen3_4b_ttt7 for EAGLE-3. Both use Qwen3-4B target, BF16, batch size 1, greedy decoding, identical prompts/stopping criteria, 512-token cap, k=4k=4, and N=32N=32.

DatasetPCTree τ\tauEAGLE-3 tree τ\tauPCTree tok/sEAGLE-3 tree tok/s
GSM8K7.2476.820245.97167.74
HumanEval6.8195.953233.21147.73
MT-Bench4.9773.683169.5691.08

For reference, EAGLE-3 chain obtains 158.95 / 129.80 / 78.75 tok/s on these datasets. This is a matched inference-system comparison. The released DSpark and EAGLE-3 checkpoints were trained in the same DeepSpec framework and on the same Open-PerfectBlend corpus, but use different drafter architectures; thus, it does not isolate the effect of tree construction alone.

Target-verification overhead. We extended the phase-cost breakdown to include target verification. Under Qwen3-4B, GSM8K, B=16B=16, BF16, k=4k=4, and N=32N=32:

MethodBackboneMarkov/treeTarget verifyTotalRounds/sampleTok/s
DSpark3.4521.05922.15326.66428.350330.87
Shared-Markov3.4475.14321.06629.65625.270330.73
PCTree3.4525.05821.09729.60723.370358.39

All latency values are ms/round. PCTree has essentially the same target-verification and total per-round cost as Shared-Markov, but its higher acceptance reduces rounds/sample by 7.5%, yielding 8.4% higher throughput. Relative to DSpark chain, its AR speedup improves from (5.97$\times$) to (6.47$\times$). We will add this complete decomposition to the revised phase-cost table.

Thank you for the additional experiments. Excluding my point about novelty, these resolve my main concerns. I will keep my score, and recommend acceptance.

Reviewer_2MZGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies speculative decoding with diffusion-based drafters. The paper builds on DSpark, which uses an additional Markov head to refine the output distributions of a diffusion backbone, making the drafting process semi-autoregressive. The authors extend DSpark to tree-based drafting and propose PCTree, a simple expansion-and-pruning algorithm for constructing candidate draft trees. Experiments on Qwen3-{4B, 8B, 14B} show that PCTree improves both mean acceptance length and end-to-end decoding speed compared to DSpark.

Critical strengths

S1. The proposed algorithm shows consistent performance improvements over DSpark across different models and benchmarks.

Critical weaknesses

My main concerns are W2, W3, and W4. W1 is a relatively minor concern.

W1. I think the methodological novelty of the proposed method is limited.

  • The proposed method seems to be a direct, straightforward extension of DSpark to tree-based drafting. The main difference from DSpark is the batched parent-conditioned scoring mechanism. Other components largely follow existing methods.

W2. The comparisons with existing diffusion-based tree drafting methods is insufficient.

  • The paper does not provide sufficient comparisons with other diffusion-based tree drafting methods. Several relevant methods are missing or only partially considered. Some examples are listed below:
  • DART: The paper neither cites nor compares against DART.
  • DDTree: DDTree is evaluated only in Section 5.3 and is not included in the main experimental results in Table 2.
  • BASTION: BASTION was released on arXiv in May, so a direct comparison is not strictly necessary, but it is still relevant related work that could be discussed.

W3. PCTree seems to support only greedy decoding.

  • The current paper seems to only focus on greedy decoding (temperature = 0) scheme and extending PCTree to stochastic sampling does not seem straightforward. In particular, it is unclear how to ensure correct stochastic sampling when the draft tree is constructed using top-k pruning during Markov tree expansion. The paper does not discuss this limitation or how it could be addressed.

W4. The comparison in Table 2 is misleading.

  • Table 2 appears to reuse the EAGLE-3 and DFlash results from Table 1 of the original DSpark paper. I checked Table 1 of the original DSpark paper and found that the EAGLE-3 and DFlash acceptance lengths match those in Table 2 of this paper, while the DSpark results differ. To understand this discrepancy, I have checked the original DSpark paper carefully, and realized that the EAGLE-3 and DFlash results were measured under stochastic sampling (temperature = 1.0), while DSpark and PCTree in this paper were evaluated under greedy decoding. Thus, Table 2 directly compares acceptance lengths obtained under different decoding settings. Since acceptance length strongly depends on the decoding setting, these results are not directly comparable. Even though comparisons with EAGLE-3 and DFlash are not the main focus of this paper, presenting results obtained under fundamentally different protocols in the same table is nonsense.
Questions for authors

Q1. What does "stable node identifier" mean in line 260? If I am not mistaken, this term is not explained in the paper.

1 comment

We thank the reviewer for the careful reading and for identifying the mixed-protocol issue.

Novelty. We agree that PCTree is a direct adaptation of established dynamic-tree construction to DSpark, rather than a new dynamic-tree algorithm. Frontier expansion, joint path scoring, pruning, and budgeted prefix-closed selection follow prior work such as EAGLE-2. We will make this relationship explicit and narrow our novelty claim. Our contribution is practical and empirical: without retraining or additional backbone passes, we reuse DSpark’s pretrained Markov head to score tree branches conditioned on their parent tokens, and show through controlled experiments that this conversion provides a measurable end-to-end benefit beyond ordinary tree branching. DDTree constructs trees from position-wise block-diffusion distributions, whereas PCTree reuses DSpark’s existing parent-conditioned Markov scores. We will clarify this architectural distinction without claiming that generic tree construction or budgeted pruning is new.

External tree baselines and protocol. We added new measurements under a unified greedy protocol: Qwen3-4B target, BF16, batch size 1, T=0T=0, identical prompts/stopping criteria, 512-token cap, k=4k=4, N=32N=32, and the same hardware. PCTree uses official dspark_qwen3_4b_block7; DFlash+DDTree uses official dflash_qwen3_4b_block7; EAGLE-3 uses official eagle3_qwen3_4b_ttt7; and DART uses official fvliang/qwen4b-dart.

MethodGSM8K (tok/s)HumanEval (tok/s)MT-Bench (tok/s)
PCTree245.97233.21169.56
DFlash+DDTree234.88215.79158.98
EAGLE-3 tree167.74147.7391.08
EAGLE-3 chain158.95129.8078.75
DART72.0086.5772.64

PCTree’s acceptance lengths are 7.247 / 6.819 / 4.977 on GSM8K / HumanEval / MT-Bench, compared with 6.684 / 6.180 / 4.590 for DFlash+DDTree.

These are matched inference-system comparisons: PCTree, DFlash+DDTree, and EAGLE-3 use official checkpoints released through DeepSpec, and we did not retrain any drafter. In particular, DART is an external system baseline: its released Qwen3-4B checkpoint is trained on a different corpus and training recipe (ShareGPT+UltraChat, approximately 280K examples, 3 epochs), whereas the DeepSpec DSpark/DFlash/EAGLE-3 checkpoints are trained on target-regenerated Open-PerfectBlend prompts (1.3M examples, 10 epochs). We therefore report DART under the same inference protocol for system-level context, but do not attribute performance differences solely to the tree-construction policy. We will add DART and BASTION to related work.

Mixed stochastic/greedy results. We agree that the prior table was misleading: its published EAGLE-3/DFlash values used stochastic sampling, whereas DSpark/PCTree used greedy decoding. We will remove these mixed-protocol rows and replace them with the new matched greedy results above. We apologize for this presentation error.

Stochastic decoding. We agree that the current method is greedy-only. Deterministic top-kk pruning alone is not a complete exact stochastic-sampling procedure. Exact sampling would require a tree-aware proposal/correction rule, including treatment of probability mass outside the retained tree. We will state this limitation explicitly.

“Stable node identifier.” This is a deterministic unique insertion-order ID assigned when a node enters the candidate pool, serving only as a reproducibility tie-breaker in the final global top-NN selection; it has no modeling or scoring role. We will revise Algorithm 1 to match the released implementation: candidates are ranked by decreasing joint score, with remaining ties broken by this insertion-order ID; the “depth asc.” tier listed in Algorithm 1 is not separately enforced and will be corrected accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.