- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper finds that multi-path speculative decoding's path selector reshapes the proposal seen by lossless verification. It proposes Sequential Martingale Selection (SMS), which optimize retained mass under a finite-sample realizability constraint without extra target-model calls. SMS-LP1 raises decoding throughput by 20.8% over tokenwise verification, 14.7% over GBV, and 6.1% on DFlash2.
Critical strengths
- Precise and under-explored problem formulation.
- Clearly written.
Critical weaknesses
- The paper mainly validates its experiments on Qwen2.5 and Llama 3.2, both of which have long been outdated.
- The authors do not compare against other existing advanced speculative decoding models such as MTP/Eagle, so it remains questionable whether a similar approach would apply to them.
- Lack more comparable method. The authors only compare against DFlash2, and do not consider methods such as DSpark, which incorporate causality into the model itself.
- SMS-LP1 only optimizes how much of the next token can be kept at each step. It does not consider how that choice affects subsequent tokens. Maybe path A is easy at the first step but runs into an obstacle at the second, whereas path B is slightly worse at the first step but proceeds smoothly afterward. To make the first step look better, SMS-LP1 shifts the opportunity from B to A. The authors lack related ablation studies.
Questions for authors
Please see weakness. Before being accepted as a formal conference paper, this article needs to undergo significant revisions in its methodology design, and experimental results.
4 comments
Response to Reviewer 5Epy (2/3)
W2: The authors do not compare against other existing advanced speculative decoding models such as MTP/Eagle, so it remains questionable whether a similar approach would apply to them.
We thank the reviewer for the suggestion. Speculative decoding consists of two stages, drafting and verification. They are orthogonal design choices and can be combined. EAGLE and MTP are methods for the drafting stage of speculative decoding. Our contribution, SMS-LP1, is a lossless method for the verification stage. Because they address different stages, a direct comparison against EAGLE or MTP would conflate the choice of drafter with the choice of verifier.
To address the reviewer's concern about compatibility, we combined SMS-LP1 with the official EAGLE-3 drafter for Llama-3.1-8B-Instruct and used the same drafter for all four verification methods. We evaluated all five benchmarks with L=8 and T=1, using K=6 for GBV and SMS-LP1. SMS-LP1 achieves the highest BE and E2E on every benchmark. Compared with Tokenwise, SMS-LP1 improves average BE by 30.0% and average E2E by 10.9%. These results demonstrate the effectiveness of SMS-LP1 when combined with EAGLE-3. The full results are shown below.
EAGLE-3 → Llama-3.1-8B
| Method | GSM8K | HumanEval | MBPP | MATH-500 | MT-Bench | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BE | E2E | BE | E2E | BE | E2E | BE | E2E | BE | E2E | BE | E2E | |
| Tokenwise | 3.36 | 103.94 | 4.42 | 146.87 | 3.81 | 128.68 | 2.85 | 96.61 | 3.05 | 102.85 | 3.50 | 115.79 |
| BV | 3.48 | 106.09 | 4.47 | 146.82 | 3.90 | 130.05 | 2.95 | 98.39 | 3.13 | 103.88 | 3.58 (+2.3%) | 117.05 (+1.1%) |
| GBV | 4.38 | 95.69 | 5.36 | 133.02 | 4.84 | 124.00 | 3.70 | 93.90 | 4.04 | 101.61 | 4.47 (+27.7%) | 109.64 (−5.3%) |
| SMS-LP1 | 4.54 | 112.01 | 5.37 | 153.75 | 4.97 | 147.37 | 3.75 | 109.98 | 4.10 | 118.73 | 4.55 (+30.0%) | 128.37 (+10.9%) |
W3: Lack more comparable method. The authors only compare against DFlash2, and do not consider methods such as DSpark, which incorporate causality into the model itself.
DFlash2 is a state-of-the-art drafting method. It outperformed DSpark and native MTP across multiple benchmarks (Inco AI, 2026). These results motivated us to choose DFlash2 in our main experiments. Furthermore, as discussed in our response to W2, DFlash2, DSpark, and MTP are methods for the drafting stage of speculative decoding. SMS-LP1 is a lossless method for the verification stage. These two stages are orthogonal, so it would be inappropriate to compare SMS-LP1 directly against DFlash2, DSpark, or MTP.
To further address the reviewer's concern, we evaluated SMS-LP1 with the DSpark drafter and Qwen3-14B as the target across the same five benchmarks (K=6, T=1). We set L=7 to match the draft length used to train the DSpark checkpoint. Using DSpark as the shared drafter, SMS-LP1 achieves the highest BE and E2E on every benchmark, improving average BE by 16.5% and average E2E by 5.6% over Tokenwise. These results demonstrate that SMS-LP1 remains effective when paired with DSpark. The full results are shown below.
DSpark → Qwen3-14B
| Method | GSM8K | HumanEval | MBPP | MATH-500 | MT-Bench | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BE | E2E | BE | E2E | BE | E2E | BE | E2E | BE | E2E | BE | E2E | |
| Tokenwise | 5.86 | 152.44 | 4.98 | 129.47 | 5.27 | 137.50 | 5.84 | 150.92 | 3.82 | 99.78 | 5.15 | 134.02 |
| BV | 5.99 | 155.45 | 5.19 | 134.20 | 5.41 | 140.65 | 6.00 | 154.76 | 4.06 | 105.41 | 5.33 (+3.5%) | 138.09 (+3.0%) |
| GBV | 5.91 | 128.23 | 5.12 | 108.21 | 5.56 | 119.12 | 5.82 | 124.23 | 4.17 | 87.40 | 5.31 (+3.1%) | 113.44 (−15.4%) |
| SMS-LP1 | 6.69 | 159.37 | 5.89 | 136.72 | 6.06 | 144.66 | 6.73 | 158.73 | 4.65 | 108.46 | 6.00 (+16.5%) | 141.59 (+5.6%) |
We have added the EAGLE-3 and DSpark results to Table 1 of the revision.
Response to Reviewer 5Epy (3/3)
W4: SMS-LP1 only optimizes how much of the next token can be kept at each step. It does not consider how that choice affects subsequent tokens. Maybe path A is easy at the first step but runs into an obstacle at the second, whereas path B is slightly worse at the first step but proceeds smoothly afterward. To make the first step look better, SMS-LP1 shifts the opportunity from B to A. The authors lack related ablation studies.
We thank the reviewer for raising this question. In the scenario described, the selector that sacrifices path B to make the first step look better is a rank-based one such as GBV. SMS-LP1 only moves the part of B's mass that the target cannot accept at the next step, so B's next-step retained mass is unchanged. We verify this by computing the reviewer's own scenario exactly.
The reviewer's scenario. Consider a vocabulary of two tokens, A and B, with K=2 candidate paths of length L=2. The draft q and the target p are as follows (probabilities listed as (A, B)).
- First token: q = (1/2, 1/2) and p = (3/5, 2/5). The target favors A, so A is the easier first token.
- Second token after A: q = (9/10, 1/10) and p = (1/10, 9/10). Draft and target disagree, so the path through A runs into an obstacle.
- Second token after B: q = p = (9/10, 1/10). Draft and target agree, so the path through B proceeds smoothly.
How BE is computed. Each method verifies one path with BV under an effective proposal r(s), the probability that the verified path passes through prefix s. The probability that prefix s is accepted is
where s = s′x, and BE = 1 + Σₛ μ(s). The methods differ only in r. BV uses the draft itself, r(A) = q(A) = 0.5. GBV ranks A first (p/q = 1.2 vs. 0.8), so it verifies a path starting with A whenever either sampled path does: r(A) = 1 − (1/2)² = 0.75. SMS-LP1 moves to A only B's surplus above its target capacity, 0.5 − 0.4 = 0.1, so r(A) = 0.6. The resulting values are:
| BV | GBV | SMS-LP1 | Exact optimum | |
|---|---|---|---|---|
| r(A) | 0.50 | 0.75 | 0.60 | 0.60 |
| μ(A) | 0.50 | 0.60 | 0.60 | 0.60 |
| μ(B) | 0.40 | 0.25 | 0.40 | 0.40 |
| μ(AA) | 0.05 | 0.06 | 0.06 | 0.06 |
| μ(AB) | 0.05 | 0.0975 | 0.075 | 0.075 |
| μ(BA) | 0.36 | 0.2025 | 0.36 | 0.36 |
| μ(BB) | 0.04 | 0.025 | 0.04 | 0.04 |
| BE = 1 + Σₛ μ(s) | 2.400 | 2.235 | 2.535 | 2.535 |
The target accepts A as the first token with probability at most p(A) = 0.6. GBV sends 0.75 to A, so 0.15 is rejected at the first step. This mass is taken from B: μ(B) drops from 0.40 to 0.25, and B's smooth second step μ(BA) drops from 0.36 to 0.2025. This is exactly the shift the reviewer describes, and it puts GBV below BV. SMS-LP1 sends exactly 0.60 to A and keeps μ(B) and μ(BA) intact, which matches the exact optimum.
This holds in general: the proof of Theorem 6 shows that SMS-LP1 only moves mass that the target cannot accept at the next step, so it never lowers any path's next-step retained mass.
Ablation on real models. The ablation the reviewer asks for is a comparison with the full-horizon optimum, which looks ahead over the whole block and therefore accounts for all subsequent tokens. We computed the exact SMS-family optimum on 100 draft trees from real Llama-3.2-1B → Llama-3.1-8B decoding states (GSM8K and MT-Bench prompts, K=6, L=8, T=1). At each node, we take the next-token distributions of the draft and target models, keep their top-k tokens and renormalize, k ∈ {3, 4, 5, 6}. The full-horizon optimum is a linear program over the tree, which we solved exactly. SMS-LP1 is within 0.5–0.8% of the exact optimum in BE and, averaged over trees, attains 96.5–97.9% of the optimum's gain over BV. Full results (mean BE over 100 trees) are below:
| top-k | BV | GBV | SMS-LP1 | Exact optimum |
|---|---|---|---|---|
| 3 | 5.013 | 5.655 | 6.386 | 6.416 |
| 4 | 5.020 | 5.683 | 6.442 | 6.481 |
| 5 | 5.031 | 5.718 | 6.477 | 6.523 |
| 6 | 5.038 | 5.760 | 6.516 | 6.565 |
SMS-LP1 thus stays within 1% of the optimum that accounts for all subsequent tokens, while requiring no additional target evaluation and keeping exact losslessness. We have added the example and the ablation to Appendix E.5 (Tables 9 and 10) of the revision.
We hope these responses and the new results address the reviewer's concerns, and we would be grateful if the reviewer could take them into account in the final assessment. All changes are highlighted in blue in the revised paper, and we would be happy to answer any further questions.
Dear Reviewer 5Epy,
Thank you once again for your time and effort in reviewing our paper. With approximately two days remaining in the discussion, we would be grateful if you could take a look at our responses and newly added experiments. We have responded to each of your comments point by point, with new experiments on the newer Qwen3 and Qwen3.5/3.8 models (W1), the EAGLE-3 drafter (W2), the DSpark drafter (W3), and an exact comparison with the full-horizon optimum (W4). We would appreciate your feedback and would be happy to address any remaining concerns or questions.
Best regards,
The Authors
Response to Reviewer 5Epy (1/3)
We thank the reviewer for the review and for noting that our problem formulation is precise and under-explored and that the paper is clearly written. We address each point in detail below.
We thank the reviewer for the suggestion. Qwen2.5 and Llama 3 are established model families widely used in speculative decoding research. For instance, HSD (Zhou et al., 2026) uses Qwen2.5 and Llama 3.1 to evaluate its method. Our study also evaluates Muse-Glimmer-30B, which was released in August 2026. Our model choices follow prior experimental practice and include a recent open-weight model.
To further address the reviewer's concern, we evaluated SMS-LP1 with two additional model pairs, Qwen3.5-0.8B → Qwen3.8-27B and Qwen3-0.6B → Qwen3-32B. SMS-LP1 achieves the highest block efficiency (BE) and end-to-end throughput (E2E) among the compared methods on each of the five benchmarks for both pairs. On Qwen3.8, SMS-LP1 improves average BE by 26.6% and average E2E by 20.6% over Tokenwise. On Qwen3, the corresponding gains are 29.4% in BE and 15.1% in E2E. These results demonstrate the effectiveness of SMS-LP1 across both model families. The full results are shown below.
Qwen3.5-0.8B → Qwen3.8-27B
Qwen3-0.6B → Qwen3-32B
We have added these results to Table 1 of the revision.