- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Tsubame proposes a two-pass speculative decoding framework for diffusion-based drafters that separates tree topology planning from candidate generation. The first pass constructs a deterministic, context-aware tree, while the second freezes its structure and repopulates its nodes using a chosen sampling strategy. Parallel backbone outputs and lightweight conditional updates make this replay inexpensive. The framework preserves the target distribution when paired with compatible verification algorithms. Experiments with Domino, DSpark, and DFlash2 across six datasets report improvements in acceptance length and throughput over deterministic trees, with suitable configurations also outperforming sampled-chain baselines.
Critical strengths
- The method fully decouples context-aware tree construction from stochastic candidate sampling.
Critical weaknesses
-
Tsubame’s Domino Chain Top1 baseline reports substantially lower speedups across all six benchmarks than the original Domino paper (Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding, with T = 1, draft budget = 16). For example, on GSM8K in qwen3-8B , the baseline drops from 6.47× to 5.54×, while Tsubame achieves 6.44×—even below the original reported result.
-
Although the two-pass design decouples tree construction from sampling, the frozen topology may not suit the replayed tokens. For example, a deep subtree planned for token a may become inefficient when replay replaces it with token b. It is therefore unclear whether replay preserves the benefits of context-aware budget allocation.
Questions for authors
- Why are your Domino baseline results lower than the original paper’s, and do Tsubame’s gains persist under matched evaluation settings?
5 comments
Response to Reviewer voBZ (2/3)
W2. Where Is the Mismatch, and How Much Does It Matter?
For where Tsubame's gains come from (tree structure, sampling, and verification), please see Part 1 of our response to Reviewer 3m2g.
We analyze the tree in two parts: the single-chain case and the branching points.
First, for a single chain, the mismatch does not invalidate the planned budget allocation.
Using draft probabilities as a proxy for acceptance to guide tree construction is the basic premise of dynamic tree methods such as EAGLE-2 and DDTree. The same premise naturally extends to OT-based verifiers such as RS and Block Verification: at a given prefix, the expected acceptance of RS is the overlap between p and q, which does not depend on which token is actually sampled. A sampled path does change the subsequent prefixes, so two questions remain: whether the proxy itself changes along the new prefixes, and whether the planned prediction is realized on the sampled path. We verify both directly.
We run replay on a single chain with Domino. We use the top-1 draft sequence to predict the acceptance length (the planned prediction), then replay (resample and apply AR correction to obtain another draft sequence). As a post hoc diagnostic only (it is never used to decide the structure), we recompute the same prediction along the replayed prefixes, i.e., using the top-1 draft probability under each new prefix. Finally, we compute the expected acceptance length of the sampled sequence under RS or Block Verification, and of the planning top-1 draft under direct verification. We take the first 16 prompts of each of GSM8K, HumanEval, and MT-Bench.
The mean proxy prediction barely changes after replay.
| Dataset | Planned prediction (top-1 path) | Recomputed along the replayed prefixes | Change |
|---|---|---|---|
| GSM8K | 10.452 | 10.368 | −0.80% |
| HumanEval | 7.889 | 7.807 | −1.04% |
| MT-Bench | 3.374 | 3.322 | −1.52% |
Although the replayed path differs from the planned one, the mean predicted acceptance length changes by only about 1%. The confidence-based proxy that dynamic trees use to estimate acceptance length remains stable before and after replay.
After replay, the gap between predicted and actual acceptance becomes smaller.
Values are expected acceptance lengths computed with the target distribution at matched starting prefixes (a paired counterfactual diagnostic).
| Dataset | Planned prediction (basis of budget allocation) | Top1, direct verification (no mismatch) | Sample + RS | Sample + Block Verification |
|---|---|---|---|---|
| GSM8K | 10.452 | 7.806 (overestimated by 33.9%) | 8.106 (overestimated by 28.9%) | 8.410 (overestimated by 24.3%) |
| HumanEval | 7.889 | 5.870 (overestimated by 34.4%) | 6.013 (overestimated by 31.2%) | 6.286 (overestimated by 25.5%) |
| MT-Bench | 3.374 | 2.754 (overestimated by 22.5%) | 2.756 (overestimated by 22.4%) | 2.855 (overestimated by 18.2%) |
The planned prediction is closest to Block Verification, then RS, and farthest from Top1, the case with no mismatch at all.
Taken together: the mean proxy prediction is essentially unchanged after replay, and the planned prediction is realized even more accurately on the sampled sequence than on the top-1 sequence itself. Perfect alignment with the plan is therefore not required; even when the actual trajectory differs from the planned one, the basis for allocating budget along this path remains valid in our measurements.
As for why we do not use sampled nodes to estimate acceptance length: this is exactly what we deliberately avoid. If the probability of a realized sampled node itself is used as the proxy, the sampling outcome affects whether the node is retained, which breaks losslessness.
Response to Reviewer voBZ (3/3)
Second, at branching points, the mismatch exists, but it can be greatly mitigated by the choice of sampling scheme.
Replay can use any valid sampling scheme with its matching verifier, not only standard sampling without replacement. Here we use Mixed sampling, the scheme proposed by Hu et al. (ICLR 2025): for a branch with m slots, the first m−1 slots are filled deterministically with the highest-probability tokens, and the last one is sampled from the renormalized distribution over the remaining tokens.
The benefits are clear:
- Acceptance: Mixed sampling achieves higher acceptance than TopK or WoR with the same number of candidates (see the single-layer counterfactual analysis in our response to Reviewer 3m2g).
- Branch control: given the same prefix, the leading tokens that carry the most mass at a branch are exactly the same as in planning; only the tail slot explores additional acceptance.
We measure, under Mixed sampling and sampling without replacement (WoR), the fraction of branching points whose prefix remains consistent with the plan. A branching point is consistent if the node and all its ancestor tokens are identical to those in the first pass, weighted by the probability that the node is included in the accepted prefix. Each dataset uses 16 prompts, with 3 samples per starting point.
| Budget | Dataset | Mixed | WoR |
|---|---|---|---|
| B = 16 | GSM8K | 94.0% | 87.5% |
| B = 16 | HumanEval | 95.1% | 89.9% |
| B = 16 | MT-Bench | 99.3% | 88.5% |
| B = 64 | GSM8K | 99.4% | 89.7% |
| B = 64 | HumanEval | 99.2% | 81.6% |
| B = 64 | MT-Bench | 99.5% | 77.3% |
The trend is clear. At B = 16, the prefix consistency of Mixed reaches 94.0%–99.3%; WoR is somewhat lower but still 87.5%–89.9%, which is why WoR/Traversal remains competitive at small budgets in the paper. At B = 64, WoR drops to 77.3% on MT-Bench, whereas Mixed rises above 99%. For Mixed, prefix mismatch is therefore rare on the accepted path.
The results reported in our paper agree well with this experiment. At small budgets, the tree has few branches, so WoR and Mixed sampling, each paired with its advanced verifier, perform similarly, each winning in some settings; here the gain from pure WoR sampling outweighs the negative effect of the mismatch. At B = 64, frequent mismatch under WoR disrupts the budget allocation across branches, and in some cases the acceptance length can fall below that of the deterministic tree, whereas Mixed sampling retains substantial gains.
To summarize:
- Single chain: replay sampling a different path does not invalidate the planned budget.
- Branching points: the mismatch exists, but a suitable sampling scheme (Mixed sampling) makes it rare on the accepted path. At large budgets in particular, over 99% of the branching points (weighted by their probability of being accepted) have prefixes identical to the plan.
For the reviewer's specific example of a deep subtree planned for token a: if a is among the top m−1 children at a branching point, Mixed sampling retains it as is during replay (given the same prefix), so the subtree planned under it is not replaced. Tokens change only at the last (sampled) slot of a branching point and at single-child nodes. The latter is exactly the single-chain case analyzed above, and the prefix-consistency results above already account for both, weighted by each node's probability of being on the accepted path.
Regarding whether replay preserves the benefit of context-aware budget allocation, please see the static-tree comparison in our response to Reviewer 3m2g. It is a 2×2 comparison at the same budget B = 37: static vs. dynamic topology, each with TopK and with Mixed/UniVer. Under the same Mixed/UniVer sampling and verification, the replayed dynamic topology outperforms the static topology on every dataset (e.g., τ 10.005 vs. 9.114 on GSM8K). The benefit of context-aware allocation is therefore retained after replay.
Response to Reviewer voBZ (1/3)
We thank the reviewer for the review. We address the reviewer's concerns below.
Q1/W1. On Comparing Absolute Speedups Across Hardware
It is well established in speculative decoding that speedup ratios measured under different hardware conditions cannot be directly compared.
Even with identical code, the hardware (CPU/GPU specifications, device–host communication latency, and even runtime settings such as SMT siblings and GPU clock frequency), the CUDA/PyTorch versions (which determine the actual implementations of the backend kernels), and many other factors all affect the measured speedup.
This is precisely why benchmarks such as Spec-Bench exist.
For example, EAGLE-2 reports 3.62× on MT-Bench in its paper, whereas the public Spec-Bench evaluation (https://github.com/hemingkx/Spec-Bench/blob/main/Leaderboard.md) obtains only 2.56×. Even within Spec-Bench, the same method obtains different speedups in different environments (GPU, CPU, and CUDA versions): 2.84× on an A100 versus 2.56× on an RTX 3090.
The consensus in speculative decoding research is therefore to guarantee a fair comparison by using exactly the same device and backend, not to compare absolute numbers across papers. Our experiments run on RTX 3090 GPUs, whereas Domino uses A100 GPUs. The two sets of numbers are not comparable.
Moreover, third-party reproductions of Domino, such as DominoTree (6.04×) and DARTree (5.79×), all report numbers lower than the official 6.47×.
On Our Evaluation Setting
Beyond hardware, several details of the experimental protocol and statistics also affect the reported results.
For consistency throughout the paper, and to stay as close as possible to how serving is measured, we compute the speedup ratio as total tokens divided by total time (i.e., overall throughput), divided by the AR throughput. This reflects the overall single-request throughput speedup.
The official Domino aggregation instead averages the per-sample per-token latency for each method and then takes the ratio of the two averages, and it only accounts for the decoding stage (prefilling is excluded).
Here, replay uses the default configuration of the paper: Mixed sampling (the top m−1 tokens plus one sampled token at each branching point), verified with UniVer.
RTX 3090:
With the code unchanged, running on an RTX PRO 6000 system instead:
Under the official Domino aggregation on the RTX PRO 6000, the Top1 chain (6.52×) already exceeds the 6.47× reported by Domino, and replay reaches 7.56×. Regardless of the protocol or the device, the gain of replay is consistently present and stable: relative to the Top1 chain, +15.7% on the RTX 3090 and +16.0% on the RTX PRO 6000 under the official Domino aggregation (+16.2% and +16.1% under our protocol).
Therefore, speedup ratios are comparable only under the same experimental conditions and the same statistics.