acceptodds
Under review as a conference paper at ICLR 2027

CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters

Abstract

Speculative decoding accelerates large language model inference by drafting future tokens cheaply and verifying them with the target model in parallel. Block drafters score a whole block of future tokens in one forward pass, yet standard decoding verifies only the top-scoring chain and discards the other candidates. Because these candidates are already scored, verifying more of them adds target computation but no extra drafting. We introduce CAST (Cost-Aware Speculative Trees), which packs these candidates into a tree and verifies it in a single target pass, leaving the target model, drafter weights, and decoding rule untouched. To decide how wide the tree should be, CAST adds candidates while the expected gain from the next one outweighs the verification time it adds. The width therefore adapts to each deployment from a latency measurement, without sweeping over widths. We evaluate CAST across five domains on three GPU generations and two model families. At its predicted width, CAST is faster than the standard chain in all eight settings, by up to . We also find that the best width depends strongly on the deployment. Where verification cost jumps at a kernel boundary, a 128-token tree is only faster than the standard chain, whereas the tree at the predicted width is faster. Furthermore, we prove that CAST leaves the target output distribution unchanged under both greedy and sampled decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Large Language Model Inference, Inference Acceleration, Tree-Based Verification, Hardware-Aware Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HuaJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends single-chain diffusion drafters (DFlash) to tree-based verification. It proposes a budget-optimal candidate tree that keeps the top-N candidates ranked by the product of the drafter’s position-wise marginals, a cost-aware rule for selecting the node budget from measured verification latency, and global best-first allocation of a shared verification budget across concurrent requests.

Critical strengths

None

Critical weaknesses

The paper’s main contributions have already been proposed in prior work that the paper neither cites nor compares against.

  1. The core method and the cost-aware budget rule reproduce DDTree and CaDDTree. Building a candidate tree from the per-position marginals of a one-pass block diffusion drafter, expanded best-first by the product of marginals, is the method of DDTree (arXiv:2604.12989), the standard tree builder for DFlash. Selecting the node budget by directly maximizing throughput under explicit draft and verification costs, proving that throughput is unimodal in the budget, and stopping greedily without an offline budget search is the method of CaDDTree (arXiv:2606.01813), which also evaluates DFlash on Qwen3-4B/8B and reports matching the oracle budget. Theorems 1 and 2 essentially restate these results. None of these works is cited.

  2. The experiments do not establish a contribution beyond these methods. Main results are compared against the DFlash chain and EAGLE-3, while DDTree, the direct baseline for trees built from DFlash marginals, is absent. The reported gains over the chain therefore largely reflect the known benefit of tree verification.

What remains is the shared-budget allocation under concurrent serving, which applies global best-first allocation with a variable-length kernel. These issues concern the paper’s central claims and cannot be resolved without repositioning the work against DDTree and CaDDTree.

Questions for authors

Please explain the methodological and theoretical differences between this paper and prior art such as DDTree (tree construction from one-pass drafter marginals) and CaDDTree (throughput-optimal node budget with a unimodality guarantee and greedy stopping rule).

20 comments
AuthorsOpenReview ↗

Response to Reviewer HuaJ (1/13)

We thank the reviewer for raising DDTree and CaDDTree.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Response overview.

Answer. CAST studies deployment width and shared budgets, with guarantees beyond the common ordering and measured gains against DDTree's released code.

The review states that its concerns "cannot be resolved without repositioning the work against DDTree and CaDDTree". New and changed text in the revision is shown in blue.

PointResult
Shared-budget comparisonAt the same total verify budget for 32 concurrent requests with four packed tokens for each, on the same drafter outputs, CAST's shared budget commits 18.6% to 22.5% more tokens than DDTree's tree with a fixed budget for each request in batches of one domain and 26.8% more in mixed batches, and more in all 96 cells of the grid (Table R1, new Table 17).
Reported comparisonAt batch one on the H100 SXM, with each method timed in its own process and DDTree in its released configuration, CAST reaches a higher speedup over the autoregressive decoder than DDTree's released decoder at every budget pairing, by 0.8% to 2.4% averaged over five domains (Table R7, new Table 23).
Width and cost comparisonCAST predicts it from the drafter's scores and one latency probe of at most 79 seconds with model loading on the RTX A6000, and the prediction is within 1.7 percentage points of the best swept width in domain-mean gain in all twelve settings of Tables 2 and 18, where each of the six swept widths is at least 4.5 points lower in some setting.
Difference from CaDDTreeCaDDTree stops the tree in every round, and CAST predicts one width for each deployment (Theorem 2).
Width and cost comparisonThe predicted width moves from 127 on the H100 SXM to 63 on Blackwell with Qwen3-8B, where it is 19.8% faster than the chain against 2.4% at N=127 (Table 2).
Theorems 1 and 2Theorem 1 is more general than DDTree's optimality result and Theorem 2 makes a different decision than CaDDTree's stop, and both are the single-request cases of the paper's results for a shared budget.

Our answers by concern.

W1. Overlap with DDTree and CaDDTree. CAST shares DDTree's best-first order by the product of marginals and cites it (Section 6), and it differs in what that order leaves open. DDTree takes the node budget as an input and reports the best of a sweep, while CAST predicts one width for each deployment from the drafter's scores and one latency probe. DDTree builds each request's tree from its own budget, while CAST splits one budget across concurrent requests. DDTree's tree is optimal for the drafter's factorized surrogate, while CAST also bounds the loss of its tree under the target law (Proposition 1). CaDDTree stops each round's tree with a greedy rule on a latency cost fitted offline for each model and GPU, while CAST sets one fixed width for each deployment by its long-run throughput (Theorem 2).

W2. Gains beyond tree verification. Measured against DDTree's released code, CAST commits more tokens under load and reaches a higher speedup over the autoregressive decoder at every batch-one budget pairing of Table R7. Equal-budget allocation gains and direct decoder timings separate the contribution from the general benefit of trees (Tables R1 and R7).

Q1. Methodological and theoretical differences. General optimality supports the rank cap (Theorem 1), and a fixed deployment width follows the stopping rule (Theorem 2). CAST shares one budget across concurrent requests (Propositions 2 and 3) and bounds the loss of its tree under the target law (Proposition 1). CaDDTree's experiments use one GPU type (A800), and CAST predicts the width for each deployment, so the budget decision and guarantees extend beyond the common ordering (Section 3.3 and Appendix B.4).

The revision does this repositioning in Section 1 (p. 1), Section 3.2 (p. 4), Section 3.3 (p. 5), Section 5 (p. 9), Section 6 (p. 10) and Appendices F and J, and it adds measured comparisons with DDTree's released code.

Conclusion. The revision identifies CAST's added width, allocation and target-law guarantees and measures its gains against DDTree.

AuthorsOpenReview ↗

Response to Reviewer HuaJ (2/13)

W1. CAST's additions to DDTree.

"Building a candidate tree from the per-position marginals of a one-pass block diffusion drafter, expanded best-first by the product of marginals, is the method of DDTree (arXiv:2604.12989), the standard tree builder for DFlash."

Answer. CAST shares DDTree's best-first order by the product of marginals and cites it, and it differs from DDTree in four decisions that this order leaves open. DDTree takes the node budget as an input and CaDDTree stops each round's tree with a fitted latency cost, while CAST sets one fixed width for each deployment (Theorem 2). DDTree builds each request's tree from its own budget, while CAST splits one budget across concurrent requests in one best-first pass (Propositions 2 and 3). DDTree's optimality result covers the full-rank tree of one request, while Theorem 1 covers any finite prefix-closed universe, such as the top 8 tokens of each position or the union of the requests' universes. DDTree's tree is optimal for the drafter's factorized surrogate, while CAST bounds the loss of its tree against the tree that is optimal under the round's target prefix law (Proposition 1). Theorems 1 and 2 therefore do not restate DDTree or CaDDTree.

The table lists where CAST differs from the two methods, and the paragraphs below take the differences one at a time.

QuestionDDTree and CaDDTreeCAST
Order in which nodes enter the treeDDTree expands best-first by the product of marginals, and CaDDTree picks the node budget of DDTree's treeThe same best-first order (Section 3.2, p. 4)
Size of the treeDDTree takes the node budget as an input and reports the best of a sweep. CaDDTree stops each round's tree with a greedy rule on a convex cost fitted to a latency profile measured offline for each model and GPUOne fixed width for each deployment, the width with the largest long-run throughput under the drafter's scores, set from offline scores and one short latency probe (Theorem 2)
Several requests verified togetherDDTree builds each request's tree from its own node budget, and CaDDTree does not address a shared budgetOne best-first pass splits one budget across the requests (Propositions 2 and 3)
Candidate set of the optimality resultDDTree's result covers the full-rank tree, which expands over every rank at each positionAny finite prefix-closed universe, which covers the top K=8 tokens of each position and the union of the requests' universes (Theorem 1)
Distribution the guarantee refers toDDTree's tree is optimal for the drafter's factorized surrogate, and its Remark 1 states that this is exact for the surrogate, not for the target distributionA bound on the loss against the tree that is optimal under the round's target prefix law (Proposition 1)

The shared order is DDTree's. The revision cites DDTree at Theorem 1 (Section 3.2, p. 4), in Section 1 (p. 1) and in the first paragraph of Section 6 (p. 10), and DDTree had not yet been released when this study was designed. The order fixes which nodes enter first. How many nodes a deployment verifies, how concurrent requests share them and what the resulting tree guarantees under the target law are separate decisions, and they are the four lower rows of the table.

AuthorsOpenReview ↗

Response to Reviewer HuaJ (3/13)

W1. Target-law guarantees, continued.

"expanded best-first by the product of marginals, is the method of DDTree"

The first theoretical difference is the distribution that the guarantee refers to. DDTree's Proposition 2, which its Algorithm 1 attains (its Proposition 3), gives the optimal tree for the surrogate objective of the drafter's factorized distribution, and its Remark 1 states that this result is exact for that surrogate, not for the target-model distribution. CAST's Proposition 1 bounds the loss of its plug-in tree against the tree that is optimal under the round's target prefix law inside the same candidate universe, by the score errors on the nodes where the two trees differ. The new Appendix G.3 measures these errors at every depth of the deployed tree. The score's mean is within 0.010 of the realized hit rate in absolute terms, and a correction fit for each depth group and branching pattern on one half of the prompts changes the accepted length of the tree rebuilt on the other half by at most 0.19% at N=63.

The second theoretical difference is the candidate set that optimality covers. DDTree expands over every rank at each position, and its Proposition 2 is stated for that full-rank tree of one request. Theorem 1 holds for any finite prefix-closed universe and any prefix belief that does not increase from ancestor to descendant. It therefore shows that CAST's tree inside the top K=8 tokens of each position (Section 3.2, p. 4) is the optimal tree of its universe under the drafter's scores, and the proof of Proposition 2 applies it to the union of the requests' candidate universes (Appendix B.4). The capped tree matches the full-rank tree's accepted length to within 1.1% at the evaluated budgets while scoring 8 candidates for each position (Appendix G.1).


W1. CaDDTree and Theorem 2, continued.

"Selecting the node budget by directly maximizing throughput under explicit draft and verification costs, proving that throughput is unimodal in the budget, and stopping greedily without an offline budget search is the method of CaDDTree (arXiv:2606.01813), which also evaluates DFlash on Qwen3-4B/8B and reports matching the oracle budget. Theorems 1 and 2 essentially restate these results."

The third difference is the decision about tree size, where Theorem 2 decides something different from CaDDTree's stop. CaDDTree decides in every round when to stop growing that round's tree, with a convex cost fitted to a latency profile measured offline for each model and GPU, and it maximizes the ratio of each round as a tractable surrogate for the long-run ratio that it also states (its Appendix A.1). Its experiments use one GPU type (A800), where it reports matching the oracle budget. Theorem 2 instead gives one fixed width for each deployment, the width with the largest long-run throughput under the drafter's scores. Its stopping condition is stated on the order statistics of the prefix masses, which CAST reads from the drafter's offline scores, and on the round cost, whose slope comes from one latency probe of at most 79 seconds with model loading on the RTX A6000. Averaged over offline rounds, the ratio is the expected committed length over the expected round time of a fixed width, so CAST targets long-run throughput directly and decides it once for each deployment. Theorem 2 uses a unimodality argument like CaDDTree's, but on the long-run throughput of a fixed width rather than on the ratio of one round.

A width for each deployment is needed because the best width changes with the GPU and the drafter. The predicted width is 127 on the H100 SXM, 63 on Blackwell with Qwen3-8B, and 63 and 95 on the RTX A6000 with DART's and PARD's drafters. In all twelve settings of Tables 2 and 18 it is within 1.7 percentage points of the best swept width in domain-mean gain, while each single width from 15 to 127 is at least 4.5 points lower in some setting. In every round the drafter's scores still choose which N nodes form the tree, so the deployment width and each round's tree are separate choices.

Theorem 1 is the general result over capped and pooled candidate sets described above, and Theorem 2 decides a fixed deployment width rather than a stop in each round, so neither restates CaDDTree's results.

Reviewer_GUHVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAST, which constructs speculative trees from one-pass draft probabilities. It selects tree widths based on measured verification costs to accelerate lossless decoding.

Critical strengths
  1. The proposed method is straightforward, using draft probabilities for tree construction and measured verification costs for width selection.
  2. The paper considers different hardware configurations and serving workloads, showing how verification costs affect tree width selection and decoding performance.
Critical weaknesses
  1. The use of draft probability products to estimate target acceptance rates is not sufficiently validated. Depth-one reliability alone cannot guarantee accurate acceptance estimates for deeper tree nodes. In addition, GPU latency profiling lacks a theoretical analysis of the relationship between computational complexity and hardware characteristics.

  2. The experimental comparisons are insufficient. The paper lacks direct comparisons with closely related methods such as DDTree and DARTree.

  3. The main text lacks a Conclusion section, and Table 3 appears after Table 4 on page 8.

Questions for authors
  1. How accurately do draft probability products estimate the true target acceptance rates? Could the authors evaluate the estimation accuracy at different tree depths rather than only at depth one?

  2. Could the authors provide a hardware-level analysis of how computational workloads (e.g., FLOPs) and GPU resource utilization (e.g., SM utilization and memory bandwidth) affect verification latency across different models, tree widths, and GPU architectures, rather than relying solely on latency profiling?

  3. Could the authors compare CAST with DDTree and DARTree under comparable experimental settings?

21 comments
AuthorsOpenReview ↗

Response to Reviewer GUHV (1/16)

We thank the reviewer for the concrete questions.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Response overview.

Answer. The revision answers the score, hardware and baseline questions with measurements and corrects the presentation issues.

The revision answers each of them, with new measurements of the product score at every depth, a hardware cost model for the verification latency, and comparisons with DDTree's released code. New and changed text in the revision is shown in blue.

MeasurementResult
Score accuracyThe mean score is within 0.010 of the realized hit rate in absolute terms at every depth from 1 to 15 and never more than 0.001 above it, with AUROC 0.88 to 0.97.
Reported comparison2. Hardware cost model of verification (new Appendix D.3).
Reported comparison3. CAST is compared with DDTree's released code under concurrency.
Shared-budget comparisonWith 32 concurrent requests and four packed tokens for each, CAST's shared budget commits 18.6% to 26.8% more tokens than DDTree's trees at the same total verify budget, and more in all 96 cells of the grid (Table R1, new Table 17 in Appendix F).
Reported comparison4. CAST is compared with DDTree's released decoder at batch one.
Reported comparison5. DARTree is positioned in Section 6.
Reported comparison6. Conclusion and table order.

Our answers by concern.

W1, Q1. Acceptance estimates beyond depth 1. The product score is accurate at every depth. Correcting the score raises the accepted length by at most 0.19% at N=63, so the remaining score error has little effect on the selected tree's accepted length (new Appendix G.3).

W1, Q2. Hardware explanation of verification latency. A hardware cost model explains the verify forward through weight streaming below a ridge at n equal to compute peak over memory bandwidth, GEMM tile steps and the host launch floor. Kernel profiles on three GPU architectures check this explanation of why latency changes with model, width and GPU (new Appendix D.3, Table 12 for the RTX A6000 and the H100 SXM, repeated as Table R4, and a Blackwell profile in Appendix D.3).

W2, Q3. DDTree and DARTree comparisons. On the H100 SXM, with each method timed in its own process and DDTree in its released configuration, CAST reaches the higher speedup over the autoregressive decoder with Qwen3-8B and Qwen3-4B at all six budget pairings, averaged over five domains (new Table 23 in Appendix J, repeated as Table R7). These timings supply the direct DDTree baseline. DARTree leaves the verification budget as a hyperparameter that its own serving study sets lower as the load grows. CAST predicts this budget with a rule, supplying the deployment-specific choice that DARTree identifies as necessary (Section 6, p. 10).

W3. Conclusion and table order. The Conclusion is in the main text, and the tables follow their numbering. Both requested corrections are in place, with Section 7 Conclusion on p. 10 and Table 3 (p. 8) preceding Table 4 (p. 9).

Conclusion. The revision supplies score validation, a hardware explanation and baseline comparisons alongside the presentation corrections.

AuthorsOpenReview ↗

Response to Reviewer GUHV (2/16)

W1, Q1. Accuracy at every depth.

"The use of draft probability products to estimate target acceptance rates is not sufficiently validated. Depth-one reliability alone cannot guarantee accurate acceptance estimates for deeper tree nodes."

"How accurately do draft probability products estimate the true target acceptance rates? Could the authors evaluate the estimation accuracy at different tree depths rather than only at depth one?"

Answer. The product score is accurate at every depth of the deployed tree, including the deeper nodes.

We measure depth-wise score means, hit rates and ranking accuracy, then test score correction by rebuilding the tree.

MeasurementResult
Score accuracyAt every depth of the deployed tree, the mean score is within 0.010 of the realized hit rate and never more than 0.001 above it (Table R3).
Score accuracyIt also ranks hits above misses, with AUROC 0.88 to 0.97.
Score accuracyAt every depth from 1 to 15 of the deployed tree, the mean product score is within 0.010 of the realized hit rate in absolute terms, never more than 0.001 above it, and ranks hits above misses with AUROC 0.88 to 0.97 (new Appendix G.3, Table 19).
Score accuracyThe accepted length then rises by 0.05% to 0.19% at N=63, by 0.10% to 0.12% at N=127 and by at most 0.58% at N=15 (Appendix G.3).
Evaluation settingTo measure this, we re-ran Qwen3-8B with the DFlash-b16 drafter on the first 40 test prompts of GSM8K, MT-Bench and HumanEval and logged the full-vocabulary drafter probabilities of every round (1940, 3161 and 2199 rounds).
Width and cost comparisonFor every node of the deployed tree (K=8, N=127, depths 1 to 15), a hit means that the target's greedy continuation starts with the node.
Score accuracyThe product score tracks the realized hit rate at every depth, within 0.010 and never more than 0.001 above it.
Score accuracyAt the deepest levels the hit rates are a few percent, and the largest gap of the table, 0.047 against 0.057 at depth 11 of MT-Bench, is an under-prediction.

The score follows target-greedy hit rates throughout the deployed tree, including the deepest levels where hits are rare.

MeasurementResult
Score accuracyWithin each depth the score ranks hits above misses with AUROC 0.88 to 0.97, and across depths its means follow the hit rates to within 0.010.

Within-depth ranking also separates hits from misses, the distinction used to select tree nodes.

Correcting the score for each depth group raises the accepted length of the rebuilt tree by at most 0.19% at N=63.

Conclusion. Depth-wise calibration and ranking are accurate, and score correction has little effect on accepted length. The revision adds this validation to Appendix G (Depth-wise accuracy of the product score).

AuthorsOpenReview ↗

Response to Reviewer GUHV (3/16)

W1, Q1. Depth-wise measurements, continued.

"Could the authors evaluate the estimation accuracy at different tree depths rather than only at depth one?"

The depth-wise comparison also reports mean scores, realized hit rates and hit-versus-miss ranking for each deployed tree depth.

To check estimation accuracy at different tree depths, we measured product scores against realized hits on GSM8K, MT-Bench and HumanEval. We corrected the score for each depth group and branching pattern on one half of the prompts and rebuilt the tree on the other half.

The tree is the top N nodes by score, so it depends on their order.

Table R3. Product-score accuracy at every depth.

MeasurementResult
Measurement protocolMean product score, realized hit rate and AUROC of the score for hits at every depth of the deployed tree (Table 19).
DepthGSM8K scoreGSM8K hit rateGSM8K AUROCMT-Bench scoreMT-Bench hit rateMT-Bench AUROCHumanEval scoreHumanEval hit rateHumanEval AUROC
10.3220.3230.970.1550.1580.940.2870.2900.97
20.1130.1130.950.0310.0320.920.0960.0970.95
30.0580.0570.940.0140.0140.900.0470.0480.94
40.0430.0430.920.0130.0140.890.0360.0360.93
50.0410.0400.930.0170.0180.880.0360.0370.92
60.0420.0420.920.0230.0250.900.0410.0440.92
70.0430.0450.910.0310.0340.920.0470.0510.92
80.0450.0470.910.0390.0440.900.0510.0550.92
90.0460.0490.920.0430.0500.920.0540.0580.93
100.0460.0490.900.0460.0540.910.0540.0580.93
110.0440.0450.920.0470.0570.910.0540.0570.93
120.0430.0450.920.0470.0560.910.0510.0530.95
130.0400.0420.930.0430.0490.930.0460.0480.95
140.0350.0370.920.0380.0430.960.0400.0410.96
150.0280.0300.920.0300.0340.950.0320.0340.96

The score tracks the realized hit rate at every depth, never more than 0.001 above it, and ranks hits above misses at every depth. Tree construction uses this ranking, and the remaining error has little effect on the tree.

Conclusion. The deeper nodes retain both calibration and ranking accuracy, with little change from correcting their scores.

Tables R3 and R8 repeat every row of Tables 19 and 20. Table R3 reports accuracy at each depth, and the revision carries the analysis in Appendix G (Depth-wise accuracy of the product score).

Reviewer_qgdAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CAST, a cost-aware speculative decoding method designed for one-pass block drafters such as DFlash. Instead of verifying only a single top-1 draft chain, CAST constructs a prefix-closed candidate tree by ranking token prefixes according to the products of the drafter's position-wise probabilities. The target model then verifies the tree in a single forward pass using an ancestor-aware attention mask.

The paper further introduces a cost-aware width-selection rule that combines offline draft-score statistics with measured verification latency to predict a suitable tree budget without an exhaustive decoding-time width sweep. The approach is also extended to allocate a shared verification budget across concurrent requests.

Experiments across multiple language models, GPUs, and tasks demonstrate consistent decoding speedups over standard DFlash. The experiments also cover width prediction, fixed-budget tree construction, sampling correctness, and concurrent serving performance.

Critical strengths

Strength 1: Convincing empirical evidence of practical decoding improvements.

The paper demonstrates consistent decoding speedups across multiple target models, hardware platforms, and tasks. In particular, the controlled comparison using the unchanged DFlash drafter and the fixed-budget ablation provide convincing evidence that the proposed tree construction improves decoding efficiency beyond simply increasing the verification budget. The evaluations under both greedy and sampled decoding further support CAST's effectiveness.

Strength 2: Principled cost-aware width selection and valuable system-level insights.

The paper presents a principled approach to selecting tree widths based on candidate probabilities and measured verification costs. The hardware-dependent latency analysis is particularly useful, highlighting how different GPU characteristics affect the choice of tree width. The predicted widths achieve performance close to the best configurations in the reported sweeps, suggesting that the selection rule works well in practice. I also appreciate the investigation of concurrent serving, where the authors address the limitations of dense packing by implementing a variable-length tree verifier to enable more efficient shared-budget allocation.

Critical weaknesses

Limited cross-drafter evaluation of the complete CAST framework.

While CAST is formulated as a general approach for one-pass block drafters, its comprehensive end-to-end evaluation is largely limited to DFlash. Although the experiments cover multiple target models, tasks, and GPU platforms, the effectiveness of the cost-aware width-selection policy is only evaluated with DFlash-based drafting. The additional DART experiments in Appendix G.2 show that the proposed tree construction strategy can improve acceptance lengths with a different drafter. However, they do not evaluate the complete CAST pipeline or examine whether its predicted widths remain close to the best-performing configurations on DART. This matters because different one-pass drafters may have different proposal distributions, block lengths, and drafting costs, potentially affecting the effectiveness of the width-selection strategy. A more comprehensive evaluation on DART, and potentially other compatible one-pass drafters, would help establish whether the framework generalizes beyond DFlash.

Questions for authors

Could the authors briefly clarify how CAST differs from closely related recent approaches such as DDTree and CaDDTree, particularly in terms of the main methodological contributions and practical considerations? A brief discussion would help better position CAST within recent developments in speculative decoding.

13 comments
AuthorsOpenReview ↗

Response to Reviewer qgdA (1/10)

We thank the reviewer for the positive assessment and for a precise request.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Response overview.

Answer. The complete DART and PARD pipelines retain fast predicted widths, and the revision distinguishes CAST's deployment and shared-budget decisions from prior methods.

W1. Complete-pipeline transfer beyond DFlash. The complete CAST pipeline now runs on DART and on a third one-pass drafter, PARD, and with both drafters the predicted width is close to the best-performing width. Decode time changes with width, so these sweeps test the width-selection rule beyond tree construction alone (Appendix G.2).

Q1. Position relative to DDTree and CaDDTree. CAST predicts a width for each deployment and shares the verification budget, with general optimality and a target-law loss bound. The common ordering alone does not settle these deployment and batch-budget decisions or supply these guarantees (Section 6).

MeasurementResult
Estimate and intervalIn all four new settings (two drafters on two GPUs) and all three domains, the fastest swept width shortens the decode time at the predicted width N* by at most 1.5%, and wherever the two widths differ the 95% interval of the difference contains zero.
Width and cost comparisonWith DART on an RTX A6000 the domain-mean gain over the chain is 33.9% at N=47, 33.5% at the predicted N*=63 and 29.4% at N=95.
Width and cost comparisonOn GSM8K the widest trees are slower than the chain, with gains of -13.3% at N=191 and -27.8% at N=255.
Width and cost comparisonOn the RTX A6000 the rule predicts N*=63 with DART's drafter, 29.8% to 38.7% faster than the top-1 chain over DART's eight positions.
Width and cost comparisonWith PARD's drafter it predicts N*=95, 24.5% to 36.9% faster than PARD's 12-position chain.
Width and cost comparisonOn the H100 SXM the same rule with the same score curves moves both widths to N*=127, 33.3% to 41.2% faster than DART's chain and 28.6% to 41.3% faster than PARD's.

Both additional drafters retain fast predicted widths as the rule follows the verification-cost difference between GPUs.

On the H100 SXM, N*=127 has the largest domain-mean gain of the ten widths swept with DART, up to N=255, and it is the fastest of the six widths swept with PARD in every domain (Tables R2 and R5). New and changed text in the revision is shown in blue.

Conclusion. The complete-pipeline sweeps establish transfer beyond DFlash and support hardware-dependent width selection.

The revision reports these results in the abstract, the contribution list (p. 2), Table 2 (p. 7), Section 4 (p. 9), new paragraphs and Table 18 in Appendix G.2, and the Conclusion (p. 10).

AuthorsOpenReview ↗

Response to Reviewer qgdA (2/10)

W1. Complete-pipeline transfer beyond DFlash.

"However, they do not evaluate the complete CAST pipeline or examine whether its predicted widths remain close to the best-performing configurations on DART."

"A more comprehensive evaluation on DART, and potentially other compatible one-pass drafters, would help establish whether the framework generalizes beyond DFlash."

Answer. Yes, the complete pipeline runs with both DART and PARD and predicts widths close to their best-performing configurations.

The proposal distribution is the score curve, which a short offline replay measures, the block length sets the depth of the candidate universe, and the drafting cost is the constant of the round cost.

The two drafters we added differ from DFlash in all three properties.

MeasurementResult
Estimate and intervalIn all four new settings (two drafters on two GPUs) and all three domains, the fastest swept width shortens the decode time at the predicted width by at most 1.5%, and wherever the two widths differ the 95% interval of the difference contains zero.
Width and cost comparisonThe tree at the predicted width is 24.5% to 41.3% faster than each drafter's top-1 chain (Tables R2 and R5).
Reported comparisonIt moves from 63 with DART and 95 with PARD on the RTX A6000 to 127 with both on the H100 SXM.
Width and cost comparisonWith the same Qwen3-8B target on the same RTX A6000 host, the rule predicts N*=95 with PARD's 12 positions and N*=63 with DART's eight.
Width and cost comparisonThe fastest swept widths are 63 in every domain with PARD and 47, 47 and 31 with DART.
Evaluation settingDART is a single decoder layer that drafts eight positions, so its committed round is capped at 9 tokens (Appendix G.2), against the 16-position block of DFlash, and PARD is a standalone Qwen3-0.6B model that drafts 12 positions in one forward.

The unchanged rule retains fast widths across proposal distributions, block lengths and drafting costs, and its prediction moves in the same direction as the measured optimum across GPUs.

To compare the predicted width with the best swept width in every setting, we collected the twelve settings of revised Table 2.

Revised Table 2. Predicted and best swept widths across the twelve settings, with block size or drafted positions in parentheses. Gain over each drafter's top-1 chain is averaged over domains. Gap is in percentage points. ‡ marks cells measured on a second A6000 host.

Setting (drafter block)N*Gain at N*Best swept width (gain)GapGain at N=127
H100 SXM, Qwen3-8B (b16)127+29.5%127 (+29.5%)+0.0+29.5%
H100 SXM, Qwen3-4B (b16)127+31.5%127 (+31.5%)+0.0+31.5%
Blackwell, Qwen3-8B (b16)63+19.8%95 (+21.5%)-1.7+2.4%
Blackwell, Qwen3-4B (b16)127+29.0%127 (+29.0%)+0.0+29.0%
A6000, Qwen3-8B (b16)95+28.7%95 (+28.7%)+0.0+25.1%‡
A6000, Qwen3-4B (b16)95+32.6%95 (+32.6%)+0.0+31.7%‡
H100 SXM, LLaMA-3.1-8B (b10)127+27.4%127 (+27.4%)+0.0+27.4%
H100 SXM, Qwen3-Coder-30B-A3B (b16)127+35.6%127 (+35.6%)+0.0+35.6%
A6000, Qwen3-8B, DART (8)63+33.5%47 (+33.9%)-0.4+24.6%
H100 SXM, Qwen3-8B, DART (8)127+36.7%127 (+36.7%)+0.0+36.7%
A6000, Qwen3-8B, PARD (12)95+30.6%63 (+32.1%)-1.5+29.8%
H100 SXM, Qwen3-8B, PARD (12)127+34.3%127 (+34.3%)+0.0+34.3%

The predicted width is within 1.7 percentage points of the best swept width in all twelve settings, on three GPU generations and with three drafters.

The rule reads the drafter's offline score curve and the round cost, whose slope comes from the probe of Section 3.1 and whose constant includes the drafter's forward time (Theorem 2, Section 3.3).

The rule itself is unchanged, and the width it predicts changes with the drafter's score curve. The rule and the measured optimum thus both move to smaller trees with the shorter drafter.

Conclusion. Complete-pipeline transfer holds while proposal distribution, block length and drafting cost enter through the unchanged width rule's inputs. The revision reports these runs in Appendix G, under Complete pipeline on DART and Complete pipeline on PARD.

AuthorsOpenReview ↗

Response to Reviewer qgdA (3/10)

W1. Complete DART measurements, continued.

"However, they do not evaluate the complete CAST pipeline or examine whether its predicted widths remain close to the best-performing configurations on DART."

The DART runs also place the complete pipeline's predicted width close to the fastest swept width on both GPUs.

To test whether predicted widths approach DART's best-performing configurations, we ran every stage with DART's drafter replacing DFlash, otherwise unchanged, for Qwen3-8B on an RTX A6000. The cost probe of this GPU, an offline replay of DART's score curve and the width rule of Theorem 2, applied as in Section 3.3, give N*=63.

The prediction is stable. On both GPUs each of the three probe repeats and the score curves of 40 other GSM8K prompts, of HumanEval and of MT-Bench give the same width.

MeasurementResult
Correctness checkThe tree, its rank cap, the verification and the attention mask are the released code, and in fp32 the decoder reproduces autoregressive greedy output token for token on 12 prompts at five widths from 7 to 255 and on the chain.
Width and cost comparisonWe timed 40 prompts in each of GSM8K, HumanEval and MT-Bench with 256 new tokens, sweeping the width from 7 to 255 nodes in one decoding loop.
Estimate and intervalWith DART's drafter the fastest swept width shortens the decode time at the predicted width by at most 0.3% on both GPUs, and wherever the two widths differ the 95% interval of the difference contains zero.
Estimate and intervalThe fastest swept width is 47 on GSM8K and HumanEval and 31 on MT-Bench, and the 95% interval of the difference from N*=63 contains zero in every domain.
Reported comparisonAt the chain's own size of 8 nodes the tree is 16.8%, 17.5% and 22.5% faster on GSM8K, HumanEval and MT-Bench.
Reported comparisonAt N* the committed round is 39%, 37% and 45% longer than the chain's in the same domains.
Reported comparisonThe offline replay gives 40% and 34% on GSM8K and HumanEval, close to these values (Appendix G.2).

DART's predicted widths yield decoding times close to the fastest swept configurations on both GPUs.

Table R2. The complete pipeline with DART's drafter.

MeasurementResult
Measurement protocolThe complete CAST pipeline with DART's drafter (Qwen3-8B, greedy, bf16, batch one) on an RTX A6000, where the rule predicts N*=63, and on the H100 SXM, where it predicts N*=127, the largest width of the rule's grid {15, 31, 47, 63, 95, 127}.
Measurement protocolBoth runs sweep the width from 7 to 255.
Measurement protocolGains are decode-speed gains over the top-1 chain over DART's eight positions in the same decoding loop, and the fastest swept N is the width with the largest gain.
Measurement protocolThe time ratio is the mean over prompts of the decode time at the fastest swept width divided by that at N*, minus one.
Measurement protocolBrackets give 95% paired bootstrap intervals over prompts.
GPUDomainFastest swept NGain at N*Gain at fastest NTime ratio, fastest over N*
RTX A6000, N*=63GSM8K47+32.0% [+29.9, +34.2]+32.3%-0.2% [-1.0, +0.6]
RTX A6000, N*=63HumanEval47+29.8% [+27.6, +32.3]+30.2%-0.3% [-1.0, +0.5]
RTX A6000, N*=63MT-Bench31+38.7% [+36.3, +41.1]+39.2%-0.3% [-1.7, +1.1]
H100 SXM, N*=127GSM8K63+35.5% [+33.2, +37.8]+35.6%-0.1% [-1.0, +0.8]
H100 SXM, N*=127HumanEval191+33.3% [+30.9, +35.8]+33.4%0.0% [-1.1, +1.0]
H100 SXM, N*=127MT-Bench127+41.2% [+38.4, +43.9]+41.2%0 (same width)

The timing-difference intervals contain zero wherever the predicted and fastest widths differ.

Table R2 reports the DART runs. On the RTX A6000 the stopping condition first fails between N=47 and N=63 (Appendix G.2).

Conclusion. DART's predicted widths remain close to the swept optimum, with stable predictions and difference intervals containing zero. The revision carries the full DART comparison in Appendix G (Complete pipeline on DART).

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/5)

We thank the reviewers for their careful reading. Their questions led to new measurements and analyses that add to the paper's results, and we are grateful for them. Reviewer qgdA finds the decoding gains convincing across target models, hardware platforms and tasks, and Reviewers qgdA and GUHV both list the cost-aware width selection across hardware and the serving study among the paper's strengths. Four new results answer the main requests.

Response overview.

Answer. The comparisons, width sweeps, score measurements and hardware analysis answer the reviewers' main requests.

MeasurementResult
Comparison with DDTree (all reviewers).With DDTree's released code at the same total verify budget, CAST's shared budget commits 18.6% to 26.8% more tokens with 32 concurrent requests and four packed tokens for each, and more in all 96 cells of the grid (Table R1, revision Table 17). At batch one, CAST's speedup is higher than that of DDTree's released decoder at all six budget pairings (Table R7, revision Table 23).
Other one-pass drafters (Reviewer qgdA).The complete pipeline with DART's and PARD's drafters is 24.5% to 41.3% faster than each drafter's top-1 chain in the same decoding loop (Tables R2 and R5, revision Table 18).
Width prediction (Reviewers HuaJ and qgdA).In all twelve settings of Tables 2 and 18, the domain-mean gain at the predicted width is within 1.7 percentage points of that at the best swept width, while each single width from 15 to 127 is at least 4.5 points lower in some setting (Section 4, p. 6).
Score accuracy and hardware analysis (Reviewer GUHV).At every depth from 1 to 15 of the deployed tree, the mean product score is within 0.010 of the realized hit rate (Table R3, revision Table 19), and summed over the tree, it predicts an accepted length below the realized one (Table R8, revision Table 20). The new Appendix D.3 analyzes the verify forward in FLOPs, memory bandwidth and SM thread blocks (Table R4, revision Table 12).

W1, Q1. Overlap with prior methods (HuaJ and qgdA). CAST shares DDTree's best-first ordering and claims what that ordering leaves open, deployment width prediction, shared allocation and guarantees (Sections 3.2 and 3.3).

W2, Q3. Missing direct baselines (HuaJ and GUHV). CAST is now measured against DDTree's released code under load and at batch one (Tables R1 and R7).

W1, Q1. Acceptance estimates beyond depth one (GUHV). Depth-wise and whole-tree checks validate the score at deeper nodes, and the deployed tree's accepted length is under-predicted, rather than overstated (Tables R3 and R8).

W1, Q2. Hardware explanation (GUHV). The new Appendix D.3 explains verification cost beyond FLOP growth alone (Table R4).

W3. Conclusion and table order (GUHV). The Conclusion is in the main text and the tables follow their numbering (Section 7).

W1. Complete-pipeline transfer (qgdA). With DART's and PARD's drafters on two GPUs, the fastest swept width would shorten the decode time at the predicted width by at most 1.5% in every domain (Tables R2 and R5).

We first address DDTree, CaDDTree and DARTree, then the two comparisons with DDTree's released code, the transfer to other drafters and the changes in the revised PDF.

AuthorsOpenReview ↗

General Response (2/5)

Position relative to DDTree, CaDDTree and DARTree.

Answer. CAST shares DDTree's best-first ordering by the product of position marginals and adds deployment width prediction, shared allocation, a target-law loss bound and rank-capped optimality. The shared ordering is not what the paper claims as its contribution.

CAST and DDTree both expand a tree best-first by the product of the position marginals of one DFlash pass. DDTree takes the node budget as an input and reports for each setting the best budget from a swept set of 16 to 1024 nodes. CAST differs from it in four ways.

  • It predicts the tree size for each deployment with the stopping rule of Theorem 2, from offline drafter scores and one latency probe of at most 79 seconds with model loading on the RTX A6000, while a width sweep of Table 18 took 22 to 74 minutes.
  • It splits one budget across concurrent requests under load (Propositions 2 and 3).
  • It bounds the loss of its tree against the target prefix law (Proposition 1), where DDTree's optimality result holds for the drafter's factorized surrogate.
  • It expands inside the top 8 tokens of each position, which matches the full-rank tree's accepted length to within 1.1% at the evaluated budgets, and Theorem 1 covers this universe (Section 3.2, Appendix G.1).

Theorems 1 and 2 are unchanged from the submission and do not restate DDTree's or CaDDTree's results. Theorem 1 holds for any fixed prefix belief that does not increase from ancestor to descendant, over any finite prefix-closed universe, which the rank-capped tree and the shared budget of Proposition 2 need and DDTree's result for the full-rank tree does not cover (Appendix B.4). Theorem 2 gives the fixed width with the largest long-run throughput under the drafter's scores, and Proposition 3 extends its stopping condition to a batch.

CaDDTree picks DDTree's node budget in every round with a greedy stop on a convex fit to a latency profile measured offline for each model and GPU, and its experiments use one GPU type (A800). CAST evaluates the stopping condition once for each deployment, and in every round the drafter's scores still choose which N nodes form the tree and, under load, how one budget is split across requests. The predicted width is within 1.7 percentage points of the best swept width in all twelve settings, on three GPU generations and with three drafters (Tables 2 and 18).

DARTree extends an autoregressive correction head from a chain to a fixed-width tree that is expanded one depth at a time and pruned best-first, and its verification budget is a hyperparameter, set from a sweep in its main experiments and set lower as concurrency grows in its serving study. CAST builds its tree from one drafter pass, predicts that budget for each deployment and splits one budget across concurrent requests.

None of the three papers had been released when this study was designed, and all three appeared during the course of this work. Speculative decoding is among the most active areas of LLM inference. In the seven weeks from DDTree (April 14, 2026) to CaDDTree (June 1, 2026), arXiv received 51 papers with the term in the title, and more than 200 by November 10, so no study can take every concurrent paper into account as it appears. CaDDTree and DARTree are available only on arXiv, and the ICLR 2027 reviewer guide states that "authors are not required to compare to papers solely on arXiv". We agree with the reviewers that the three papers belong in the paper, and the revision cites all three and states how CAST differs from each, DDTree in Sections 1 and 3.2, CaDDTree in Sections 1 and 3.3, and all three in Section 6 (p. 10).

Conclusion. The paper studies deployment width, shared allocation, target-law loss and rank-capped optimality. These matter because the common ordering alone neither sets the deployment budget nor supplies these guarantees, as Section 'Related Work' (DDTree paragraph) explains.

AuthorsOpenReview ↗

General Response (3/5)

Comparisons with DDTree's released code.

Answer. At the same total verify budget, CAST's shared budget commits more tokens than DDTree's trees in every cell, and at batch one CAST's speedup is higher than that of DDTree's released decoder at every budget pairing.

To isolate what shared-budget allocation adds beyond DDTree, we ran a new experiment comparing committed tokens across concurrent requests at equal total verification budgets. DDTree's released code builds the tree of every request for b draft nodes and walks its accepted path, unchanged, and CAST spends the same total of B(b+1) packed tokens on the same drafter outputs. The replay uses greedy rounds of Qwen3-8B with the DFlash-b16 drafter, 40 prompts each of GSM8K, MT-Bench and HumanEval and 1000 batches for each cell.

Table R1. Committed tokens at the same total verify budget with 32 concurrent requests, averaged over the requests of a batch, the root included, with the gain of CAST's shared budget and its 95% bootstrap interval over prompts.

Packed tokens for each requestBatch domainDDTree, tree for each requestCAST, shared budgetGain of the shared budget
4GSM8K3.243.90+20.3% [+17.3, +22.4]
4MT-Bench2.432.88+18.6% [+13.0, +24.9]
4HumanEval3.193.90+22.5% [+20.4, +24.9]
4Mixed domains2.953.74+26.8% [+24.7, +28.6]
8GSM8K5.065.97+18.0% [+16.0, +20.2]
8MT-Bench3.133.43+9.4% [+6.1, +13.4]
8HumanEval4.875.91+21.4% [+19.0, +23.6]
8Mixed domains4.355.18+19.1% [+17.5, +20.8]

CAST commits more tokens in all 96 cells of the grid, which also covers 8 and 16 requests and 2 to 32 draft nodes for each request, and the gain holds when every request of a batch comes from a different prompt. In timed runs on Blackwell with Qwen3-8B, the shared budget raises goodput by a median 8.6% at 16 requests and 13.9% at 32 over a uniform width for each request built by CAST's builder, the fixed budget that DDTree's tree also uses. Over the best static width the gains are 7.0% and 13.7% (Appendix F, Section 5, p. 9).

To compare CAST directly with DDTree's released decoder, we ran new batch-one timing experiments on the H100 SXM with Qwen3-8B and Qwen3-4B targets, the public DFlash-b16 drafters and 40 prompts in each of five domains. DDTree runs its released code with its released benchmark configuration, in which its drafter uses FlashAttention-2 where CAST's uses scaled dot-product attention, and each method's speedup uses the autoregressive decoder timed in its own process.

Table R7. Batch-one speedup of CAST over the autoregressive decoder relative to that of DDTree's released decoder, averaged over five domains (CAST's speedup over DDTree's minus one).

TargetBest budget of each sweepBudget of about 256 nodesBudget of about 128 nodes
Qwen3-8B+1.5% (CAST N=255, DDTree 511)+2.2% (CAST N=255, DDTree 256)+1.6% (CAST N=127, DDTree 128)
Qwen3-4B+1.6% (CAST N=383, DDTree 512)+0.8% (CAST N=255, DDTree 256)+2.4% (CAST N=127, DDTree 128)

CAST has the higher speedup at every pairing, including the best budget of DDTree's own sweep, and it predicts the budget that DDTree takes as input. The revision adds Table R1 as Table 17 in Appendix F, summarized in Section 5 (p. 9), and Table R7 as Table 23 in the new Appendix J, reported in Section 4 (p. 6).

Conclusion. Both comparisons run DDTree's released code, and CAST is ahead in every cell of each.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.