acceptodds
Under review as a conference paper at ICLR 2027

Shaping What to Verify: Realizable Multi-Path Speculative Decoding

Abstract

Multi-path speculative decoding samples several draft continuations, but efficient lossless verification typically selects only one. This makes the selector crucial: it not only chooses a path, but also reshapes the proposal distribution seen by the verifier. We introduce *Sequential Martingale Selection* (SMS), a framework for optimizing this proposal while guaranteeing finite-sample realizability. SMS uses retained mass as the exact block-efficiency objective, represents selector-induced reweighting through locally conserved bounded densities, and realizes every proposal in the family with an explicit sequential selector. We further develop SMS-LP1, an online retained-flow optimizer that requires no additional target-model evaluation. Extensive experiments across LLM settings show that SMS-LP1 improves end-to-end decoding throughput by 20.8% over tokenwise verification and 14.7% over GBV on average. SMS-LP1 further achieves a 6.1% average gain on DFlash2, demonstrating substantial and consistent improvements on both standard model configurations and state-of-the-art drafting frameworks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Large Language Model Inference, Block Verification, Multi-Path Verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5EpyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper finds that multi-path speculative decoding's path selector reshapes the proposal seen by lossless verification. It proposes Sequential Martingale Selection (SMS), which optimize retained mass under a finite-sample realizability constraint without extra target-model calls. SMS-LP1 raises decoding throughput by 20.8% over tokenwise verification, 14.7% over GBV, and 6.1% on DFlash2.

Critical strengths
  1. Precise and under-explored problem formulation.
  2. Clearly written.
Critical weaknesses
  1. The paper mainly validates its experiments on Qwen2.5 and Llama 3.2, both of which have long been outdated.
  2. The authors do not compare against other existing advanced speculative decoding models such as MTP/Eagle, so it remains questionable whether a similar approach would apply to them.
  3. Lack more comparable method. The authors only compare against DFlash2, and do not consider methods such as DSpark, which incorporate causality into the model itself.
  4. SMS-LP1 only optimizes how much of the next token can be kept at each step. It does not consider how that choice affects subsequent tokens. Maybe path A is easy at the first step but runs into an obstacle at the second, whereas path B is slightly worse at the first step but proceeds smoothly afterward. To make the first step look better, SMS-LP1 shifts the opportunity from B to A. The authors lack related ablation studies.
Questions for authors

Please see weakness. Before being accepted as a formal conference paper, this article needs to undergo significant revisions in its methodology design, and experimental results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5Epy (1/3)

We thank the reviewer for the review and for noting that our problem formulation is precise and under-explored and that the paper is clearly written. We address each point in detail below.

W1: The paper mainly validates its experiments on Qwen2.5 and Llama 3.2, both of which have long been outdated.

We thank the reviewer for the suggestion. Qwen2.5 and Llama 3 are established model families widely used in speculative decoding research. For instance, HSD (Zhou et al., 2026) uses Qwen2.5 and Llama 3.1 to evaluate its method. Our study also evaluates Muse-Glimmer-30B, which was released in August 2026. Our model choices follow prior experimental practice and include a recent open-weight model.

To further address the reviewer's concern, we evaluated SMS-LP1 with two additional model pairs, Qwen3.5-0.8B → Qwen3.8-27B and Qwen3-0.6B → Qwen3-32B. SMS-LP1 achieves the highest block efficiency (BE) and end-to-end throughput (E2E) among the compared methods on each of the five benchmarks for both pairs. On Qwen3.8, SMS-LP1 improves average BE by 26.6% and average E2E by 20.6% over Tokenwise. On Qwen3, the corresponding gains are 29.4% in BE and 15.1% in E2E. These results demonstrate the effectiveness of SMS-LP1 across both model families. The full results are shown below.

Qwen3.5-0.8B → Qwen3.8-27B

MethodGSM8KHumanEvalMBPPMATH-500MT-BenchAverage
BEE2EBEE2EBEE2EBEE2EBEE2EBEE2E
Tokenwise4.8419.537.3928.844.3717.994.8919.922.9812.384.8919.73
BV4.9419.927.5529.584.4618.325.0120.423.1513.075.02 (+2.7%)20.26 (+2.7%)
GBV5.9322.237.6728.285.5221.236.0422.983.9515.325.82 (+19.0%)22.01 (+11.5%)
SMS-LP16.4024.368.0029.955.9523.236.4424.894.1916.566.19 (+26.6%)23.80 (+20.6%)

Qwen3-0.6B → Qwen3-32B

MethodGSM8KHumanEvalMBPPMATH-500MT-BenchAverage
BEE2EBEE2EBEE2EBEE2EBEE2EBEE2E
Tokenwise5.1823.825.5026.714.0619.835.2325.273.1015.284.6222.18
BV5.2824.185.6427.264.4121.485.4826.453.3016.204.82 (+4.4%)23.11 (+4.2%)
GBV6.1123.636.3326.995.0521.846.2526.623.9517.105.54 (+20.0%)23.24 (+4.8%)
SMS-LP16.5025.526.6929.075.5524.426.8429.704.2918.955.97 (+29.4%)25.53 (+15.1%)

We have added these results to Table 1 of the revision.

AuthorsOpenReview ↗

Response to Reviewer 5Epy (2/3)

W2: The authors do not compare against other existing advanced speculative decoding models such as MTP/Eagle, so it remains questionable whether a similar approach would apply to them.

We thank the reviewer for the suggestion. Speculative decoding consists of two stages, drafting and verification. They are orthogonal design choices and can be combined. EAGLE and MTP are methods for the drafting stage of speculative decoding. Our contribution, SMS-LP1, is a lossless method for the verification stage. Because they address different stages, a direct comparison against EAGLE or MTP would conflate the choice of drafter with the choice of verifier.

To address the reviewer's concern about compatibility, we combined SMS-LP1 with the official EAGLE-3 drafter for Llama-3.1-8B-Instruct and used the same drafter for all four verification methods. We evaluated all five benchmarks with L=8 and T=1, using K=6 for GBV and SMS-LP1. SMS-LP1 achieves the highest BE and E2E on every benchmark. Compared with Tokenwise, SMS-LP1 improves average BE by 30.0% and average E2E by 10.9%. These results demonstrate the effectiveness of SMS-LP1 when combined with EAGLE-3. The full results are shown below.

EAGLE-3 → Llama-3.1-8B

MethodGSM8KHumanEvalMBPPMATH-500MT-BenchAverage
BEE2EBEE2EBEE2EBEE2EBEE2EBEE2E
Tokenwise3.36103.944.42146.873.81128.682.8596.613.05102.853.50115.79
BV3.48106.094.47146.823.90130.052.9598.393.13103.883.58 (+2.3%)117.05 (+1.1%)
GBV4.3895.695.36133.024.84124.003.7093.904.04101.614.47 (+27.7%)109.64 (−5.3%)
SMS-LP14.54112.015.37153.754.97147.373.75109.984.10118.734.55 (+30.0%)128.37 (+10.9%)

W3: Lack more comparable method. The authors only compare against DFlash2, and do not consider methods such as DSpark, which incorporate causality into the model itself.

DFlash2 is a state-of-the-art drafting method. It outperformed DSpark and native MTP across multiple benchmarks (Inco AI, 2026). These results motivated us to choose DFlash2 in our main experiments. Furthermore, as discussed in our response to W2, DFlash2, DSpark, and MTP are methods for the drafting stage of speculative decoding. SMS-LP1 is a lossless method for the verification stage. These two stages are orthogonal, so it would be inappropriate to compare SMS-LP1 directly against DFlash2, DSpark, or MTP.

To further address the reviewer's concern, we evaluated SMS-LP1 with the DSpark drafter and Qwen3-14B as the target across the same five benchmarks (K=6, T=1). We set L=7 to match the draft length used to train the DSpark checkpoint. Using DSpark as the shared drafter, SMS-LP1 achieves the highest BE and E2E on every benchmark, improving average BE by 16.5% and average E2E by 5.6% over Tokenwise. These results demonstrate that SMS-LP1 remains effective when paired with DSpark. The full results are shown below.

DSpark → Qwen3-14B

MethodGSM8KHumanEvalMBPPMATH-500MT-BenchAverage
BEE2EBEE2EBEE2EBEE2EBEE2EBEE2E
Tokenwise5.86152.444.98129.475.27137.505.84150.923.8299.785.15134.02
BV5.99155.455.19134.205.41140.656.00154.764.06105.415.33 (+3.5%)138.09 (+3.0%)
GBV5.91128.235.12108.215.56119.125.82124.234.1787.405.31 (+3.1%)113.44 (−15.4%)
SMS-LP16.69159.375.89136.726.06144.666.73158.734.65108.466.00 (+16.5%)141.59 (+5.6%)

We have added the EAGLE-3 and DSpark results to Table 1 of the revision.

AuthorsOpenReview ↗

Response to Reviewer 5Epy (3/3)

W4: SMS-LP1 only optimizes how much of the next token can be kept at each step. It does not consider how that choice affects subsequent tokens. Maybe path A is easy at the first step but runs into an obstacle at the second, whereas path B is slightly worse at the first step but proceeds smoothly afterward. To make the first step look better, SMS-LP1 shifts the opportunity from B to A. The authors lack related ablation studies.

We thank the reviewer for raising this question. In the scenario described, the selector that sacrifices path B to make the first step look better is a rank-based one such as GBV. SMS-LP1 only moves the part of B's mass that the target cannot accept at the next step, so B's next-step retained mass is unchanged. We verify this by computing the reviewer's own scenario exactly.

The reviewer's scenario. Consider a vocabulary of two tokens, A and B, with K=2 candidate paths of length L=2. The draft q and the target p are as follows (probabilities listed as (A, B)).

  • First token: q = (1/2, 1/2) and p = (3/5, 2/5). The target favors A, so A is the easier first token.
  • Second token after A: q = (9/10, 1/10) and p = (1/10, 9/10). Draft and target disagree, so the path through A runs into an obstacle.
  • Second token after B: q = p = (9/10, 1/10). Draft and target agree, so the path through B proceeds smoothly.

How BE is computed. Each method verifies one path with BV under an effective proposal r(s), the probability that the verified path passes through prefix s. The probability that prefix s is accepted is

μ(s)=min⁡{r(s), μ(s′)⋅p(x∣s′)},μ(∅)=1,\mu(s) = \min\lbrace r(s),\ \mu(s') \cdot p(x \mid s')\rbrace, \qquad \mu(\varnothing)=1,

where s = s′x, and BE = 1 + Σₛ μ(s). The methods differ only in r. BV uses the draft itself, r(A) = q(A) = 0.5. GBV ranks A first (p/q = 1.2 vs. 0.8), so it verifies a path starting with A whenever either sampled path does: r(A) = 1 − (1/2)² = 0.75. SMS-LP1 moves to A only B's surplus above its target capacity, 0.5 − 0.4 = 0.1, so r(A) = 0.6. The resulting values are:

BVGBVSMS-LP1Exact optimum
r(A)0.500.750.600.60
μ(A)0.500.600.600.60
μ(B)0.400.250.400.40
μ(AA)0.050.060.060.06
μ(AB)0.050.09750.0750.075
μ(BA)0.360.20250.360.36
μ(BB)0.040.0250.040.04
BE = 1 + Σₛ μ(s)2.4002.2352.5352.535

The target accepts A as the first token with probability at most p(A) = 0.6. GBV sends 0.75 to A, so 0.15 is rejected at the first step. This mass is taken from B: μ(B) drops from 0.40 to 0.25, and B's smooth second step μ(BA) drops from 0.36 to 0.2025. This is exactly the shift the reviewer describes, and it puts GBV below BV. SMS-LP1 sends exactly 0.60 to A and keeps μ(B) and μ(BA) intact, which matches the exact optimum.

This holds in general: the proof of Theorem 6 shows that SMS-LP1 only moves mass that the target cannot accept at the next step, so it never lowers any path's next-step retained mass.

Ablation on real models. The ablation the reviewer asks for is a comparison with the full-horizon optimum, which looks ahead over the whole block and therefore accounts for all subsequent tokens. We computed the exact SMS-family optimum on 100 draft trees from real Llama-3.2-1B → Llama-3.1-8B decoding states (GSM8K and MT-Bench prompts, K=6, L=8, T=1). At each node, we take the next-token distributions of the draft and target models, keep their top-k tokens and renormalize, k ∈ {3, 4, 5, 6}. The full-horizon optimum is a linear program over the tree, which we solved exactly. SMS-LP1 is within 0.5–0.8% of the exact optimum in BE and, averaged over trees, attains 96.5–97.9% of the optimum's gain over BV. Full results (mean BE over 100 trees) are below:

top-kBVGBVSMS-LP1Exact optimum
35.0135.6556.3866.416
45.0205.6836.4426.481
55.0315.7186.4776.523
65.0385.7606.5166.565

SMS-LP1 thus stays within 1% of the optimum that accounts for all subsequent tokens, while requiring no additional target evaluation and keeping exact losslessness. We have added the example and the ablation to Appendix E.5 (Tables 9 and 10) of the revision.

We hope these responses and the new results address the reviewer's concerns, and we would be grateful if the reviewer could take them into account in the final assessment. All changes are highlighted in blue in the revised paper, and we would be happy to answer any further questions.

Dear Reviewer 5Epy,

Thank you once again for your time and effort in reviewing our paper. With approximately two days remaining in the discussion, we would be grateful if you could take a look at our responses and newly added experiments. We have responded to each of your comments point by point, with new experiments on the newer Qwen3 and Qwen3.5/3.8 models (W1), the EAGLE-3 drafter (W2), the DSpark drafter (W3), and an exact comparison with the full-horizon optimum (W4). We would appreciate your feedback and would be happy to address any remaining concerns or questions.

Best regards,

The Authors

Reviewer_6wUmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sequential Martingale Selection (SMS), which reshapes the proposal seen by block verification while selecting from K sampled draft paths. Bounded density martingales provide an explicit finite-K selector and a family-level coverage guarantee. SMS-LP1 then optimizes next-layer retained mass using cached target scores. The intended benefit is longer accepted blocks while preserving the target distribution and requiring no additional model training. Experiments report average throughput gains of 14.7% over greedy multi-path block verification (GBV) and 6.1% over DFlash2's token-wise verifier.

Critical strengths
  1. Explicit realizability. Theorem 4 constructs a finite-K selector together with the exact proposal conditionals required by block verification. This addresses a central difficulty: choosing a promising path changes its proposal distribution, which must remain tractable for exact verification.

  2. A coverage guarantee for the selector family. Theorem 5 bounds the expressive power lost by restricting optimization to SMS. This gives a quantitative reason to study the family, beyond demonstrating that its selectors can be implemented.

  3. Low measured overhead and varied evaluation. SMS-LP1 has a closed-form, O(|V|) update per visited prefix. Experiments include several model pairs, DFlash2, and three-seed checks for two configurations. Appendix E.2 reports selector overhead of 1.66% in the profiled Qwen setting.

Critical weaknesses
  1. The guarantee does not characterize SMS-LP1's optimization quality. Theorem 5 concerns the best proposal in the family, whereas SMS-LP1 optimizes only the next layer. The paper neither bounds nor measures the resulting full-block gap. An ablation of the minimum-movement tie-break is also missing.

  2. Comparisons after tuning the path count are limited. Main comparisons fix K=6. On Qwen2.5-0.5B/32B and GSM8K, Table 2 gives about 7.8% higher throughput than GBV at this K, but only 4.6% over GBV's best observed K. This comparison covers one setting.

  3. Exact multi-path baseline coverage is narrow. Another exact verifier under a matched candidate-scoring budget would help establish whether the gains extend beyond GBV. Appendix E.3's counterexample concerns the specific SpecTr-GBV rule and does not settle comparisons with other exact alternatives.

Questions for authors
  1. Additional experiment: How close is SMS-LP1 to the exact SMS-family optimum on small vocabulary trees? Comparing minimum movement with another deterministic, prefix-local LP1 optimum would also isolate the tie-break's contribution.

  2. Additional experiment: Can you extend the best-observed-K comparison to other model pairs and DFlash2, using the same search grid and including K=1? This would distinguish fixed-budget gains from gains after each method chooses its fastest setting.

  3. Clarification and comparison: Which GBV ranking rule and implementation revision are used? Can you compare one other exact multi-path verifier under a matched candidate-scoring budget, or explain which differences prevent a fair comparison?

  4. Related-work positioning: The discussion could briefly distinguish proposal selection from complementary approaches that adapt verification computation, such as VIA-SD (Xian et al., ICML 2026). While the two methods address different aspects of speculative decoding, this distinction may help clarify the scope of the proposed framework.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6wUm (1/3)

We sincerely thank the reviewer for the careful and insightful review, and for acknowledging the explicit realizability, the coverage guarantee and the low overhead of our method. The questions are precise and have helped us strengthen the paper considerably. We address each of them in detail below.

W1: The guarantee does not characterize SMS-LP1's optimization quality. Theorem 5 concerns the best proposal in the family, whereas SMS-LP1 optimizes only the next layer. The paper neither bounds nor measures the resulting full-block gap. An ablation of the minimum-movement tie-break is also missing.

Q1: How close is SMS-LP1 to the exact SMS-family optimum on small vocabulary trees? Comparing minimum movement with another deterministic, prefix-local LP1 optimum would also isolate the tie-break's contribution.

We computed the exact SMS-family optimum on 100 draft trees from real Llama-3.2-1B → Llama-3.1-8B decoding states (GSM8K and MT-Bench prompts, K=6K=6, L=8L=8, T=1T=1). At each node, we take the next-token distributions of the draft and target models, keep their top-kk tokens and renormalize, k ∈ {3, 4, 5, 6}.

To isolate the contribution of minimum movement (MM), we compare it with another deterministic, prefix-local LP1 optimum, greedy p/qp/q ordering. It moves the same total amount Δ=min⁡{D−,D+}\Delta=\min\lbrace D^-,D^+\rbrace as MM (Eq. 27), and is therefore also an LP1 optimum, but instead of spreading Δ\Delta proportionally as in Eq. (28), it concentrates the transfer by the likelihood ratio:

zxG=zx0−rx−+rx+,∑xrx−=∑xrx+=Δ,z^{\mathrm{G}}_x = z^0_x - r^-_x + r^+_x, \qquad \textstyle\sum_x r^-_x=\sum_x r^+_x=\Delta,

where receivers are filled in decreasing order of px/qxp_x/q_x, each up to ax+a^+_x, and donors are drained in increasing order of px/qxp_x/q_x, each up to ax−a^-_x.

SMS-LP1 is within 0.5–0.8% of the exact optimum in BE and, averaged over trees, attains 96.5–97.9% of the optimum's gain over BV. MM and greedy p/qp/q achieve nearly identical BE. Full results (mean BE over 100 trees) are below:

top-kkBVGBVSMS-LP1 (MM)Greedy p/qp/qExact optimum
35.0135.6556.3866.3876.416
45.0205.6836.4426.4436.481
55.0315.7186.4776.4756.523
65.0385.7606.5166.5146.565

The gap is small because minimum movement only removes mass that the target cannot accept at the next step. The proof of Theorem 6 shows that it never lowers any child's next-step retained mass, so the one-step objective loses only when that surplus would have been useful deeper in the tree.

We further compare the two tie-breaks end to end on GSM8K with the Qwen2.5-0.5B/32B pair. Results are below:

LP1 tie-breakBEE2E
Minimum movement (paper)6.5934.83
Greedy p/qp/q6.5832.88

Both comparisons show that the choice among LP1 optima has little effect on BE. The gain therefore comes from the SMS family and the LP1 objective rather than the tie-break. We use MM because, among LP1 optima, it perturbs the draft flow the least (Theorem 6), a conservative choice when the target's behavior deeper in the tree is unknown, and because it is computed in closed form (Eq. 28, Section 6.2), which makes it faster at nearly the same BE (34.83 vs. 32.88 tokens/s). We have added both studies to Appendix E.5 (Tables 10 and 11) of the revision.

AuthorsOpenReview ↗

Response to Reviewer 6wUm (2/3)

W2: Comparisons after tuning the path count are limited. Main comparisons fix K=6. On Qwen2.5-0.5B/32B and GSM8K, Table 2 gives about 7.8% higher throughput than GBV at this K, but only 4.6% over GBV's best observed K. This comparison covers one setting.

Q2: Can you extend the best-observed-K comparison to other model pairs and DFlash2, using the same grid and including K=1? This would distinguish fixed-budget gains from gains after each method chooses its fastest setting.

We thank the reviewer for this suggestion. As the reviewer notes, our ablation (Section 7.3, Table 2 of the paper) already varies K ∈ {1, 2, 4, 6, 8} on GSM8K with the Qwen2.5-0.5B/32B pair. At K=1, both multi-path methods reduce to single-path BV, so the grid also compares each method with its single-path baseline. In this setting, GBV is fastest at K=1, i.e., as BV itself (33.31 tokens/s), whereas SMS-LP1 reaches its best at K=6 (34.83 tokens/s).

We further ran the same grid on the Llama-3.2-1B/3.1-8B and DFlash2/Glimmer-30B pairs, both on GSM8K. At each method's own best K, SMS-LP1 remains faster than GBV in both settings. Full results are below:

Model pairKGBV BEGBV E2ESMS-LP1 BESMS-LP1 E2E
Llama15.5061.455.5061.45
26.1863.136.1365.63
46.1861.316.6969.46
66.0257.956.9869.95
85.8455.467.0670.33
DFlash216.81108.246.81108.24
27.10108.947.07107.90
47.16108.897.26109.97
67.16108.377.38111.30
87.11106.257.43110.51

On both pairs, GBV saturates early and reaches its best throughput at K=2, whereas SMS-LP1 reaches its best at K=8 on Llama and K=6 on DFlash2. We have added these results to Appendix E.4 (Table 8) of the revision.

AuthorsOpenReview ↗

Response to Reviewer 6wUm (3/3)

W3: Exact multi-path baseline coverage is narrow. Another exact verifier under a matched candidate-scoring budget would help establish whether the gains extend beyond GBV. Appendix E.3's counterexample concerns the specific SpecTr-GBV rule and does not settle comparisons with other exact alternatives.

Q3: Which GBV ranking rule and implementation revision are used? Can you compare one other exact multi-path verifier under a matched candidate-scoring budget, or explain which differences prevent a fair comparison?

GBV. We thank the reviewer for the question. We implemented GBV following Algorithm 1 of Thomas & Pal (2026), using arXiv:2602.16961v1, the version available when we submitted our paper. The anonymous repository linked in the paper has expired, and we could not find another public release of the authors' code. We therefore reproduced GBV independently from the paper. GBV ranks the K complete paths lexicographically by their per-step ratios p/q, compared depth by depth with a token-id tie-break, and verifies the top-ranked path with BV under the exact proposal induced by this selection. GBV and SMS-LP1 share the same K paths, the same batched target forward pass and the same BV implementation (Appendix D.2), so only the selector differs.

Traversal Verification. As another exact multi-path verifier, we compare with Traversal Verification (Weng et al., NeurIPS 2025). Both methods score the same K=6 paths of length L=8 in one batched target forward pass, so the candidate-scoring budget is matched. We use the paper's Qwen2.5-0.5B/32B setting on all five benchmarks with three seeds. SMS-LP1 achieves higher BE and E2E than Traversal Verification on every benchmark. Full results are below:

DatasetTraversal BETraversal E2ESMS-LP1 BESMS-LP1 E2E
GSM8K6.55 ± 0.0234.56 ± 0.576.62 ± 0.0335.22 ± 0.36
MATH-5006.65 ± 0.0537.43 ± 0.096.95 ± 0.0239.37 ± 1.54
HumanEval6.36 ± 0.1036.84 ± 0.606.48 ± 0.0537.89 ± 0.10
MBPP6.23 ± 0.0336.00 ± 0.346.33 ± 0.0737.07 ± 0.79
MT-Bench4.37 ± 0.0624.81 ± 0.344.48 ± 0.0326.06 ± 0.33

These results show that the gains of SMS-LP1 extend beyond GBV to a stronger exact multi-path verifier with a different mechanism. We have added this comparison to Appendix E.3 (Table 7) of the revision.

Q4: The discussion could briefly distinguish proposal selection from complementary approaches that adapt verification computation, such as VIA-SD (Xian et al., ICML 2026). While the two methods address different aspects of speculative decoding, this distinction may help clarify the scope of the proposed framework.

We thank the reviewer for pointing us to VIA-SD. We have read the paper carefully, and we find the suggestion on related-work positioning very valuable. VIA-SD reduces the cost of verification itself, whereas SMS-LP1 shapes which drafted path is verified and under which proposal. We have added a discussion of VIA-SD and this distinction to the related work (Section 2) of the revision.

Reviewer_HeCcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses the problem of developing a better "selector" in the context of a multi-path speculative decoding algorithm. The drafter generates multiple paths, and the selector selects one of them to pass on to the verifier. Existing techniques use simple and rigid rules, such as picking the path based on fixed ranking rules that prefer one token over another. This paper introduces SMS, which chooses among the paths in a way that makes the average number of surviving tokens per check as high as possible. It adjusts the odds of which token gets picked, favoring paths the big model is likely to accept. It provides gains of about 14% over the earlier simplistic selector model and by about 6% over DFlash2.

Critical strengths

The approach moves beyond the normal fixed-rule-based selection strategy and provides a selector that is provably valid and cheap to compute. By using local conservation rules and bounds, the paper shows that any proposal in the SMS family can be realized exactly by a simple sequential selector,

Critical weaknesses

It is unclear what the technique costs when deployed with inference batch sizes greater than one. If some of the memory-boundedness of the selection/verification work is already mitigated by a larger batch size, the cost of selecting from multiple paths could be more substantial. The paper doesn't evaluate with larger batches.

Questions for authors

What is the performance of your system against the different baselines if you were to use inference batches that have more than one query at a time?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HeCc

We sincerely thank the reviewer for the thoughtful review and for recognizing that our method moves beyond fixed-rule selection with a selector that is provably valid and cheap to compute. The question on batched inference is very helpful for understanding the practical cost of our method, and we address it in detail below.

W1: It is unclear what the technique costs when deployed with inference batch sizes greater than one. If some of the memory-boundedness of the selection/verification work is already mitigated by a larger batch size, the cost of selecting from multiple paths could be more substantial. The paper doesn't evaluate with larger batches.

Q1: What is the performance of your system against the different baselines if you were to use inference batches that have more than one query at a time?

We thank the reviewer for asking how batch size affects SMS-LP1. As batch size increases, the extra computation can become a bottleneck, posing a common challenge for speculative decoding methods. Prior work has studied this issue and proposed solutions such as the optimized batched attention in BASS (Qian et al., 2024).

In this work, our contribution is the design of a new lossless verification method, and the large-batch setting falls outside our scope. We understand that optimizing speculative decoding for large batches is an important direction for further work.

We also study the performance of our methods as suggested by the reviewer. We evaluated our method on the full GSM8K dataset with the Llama-3.2-1B → Llama-3.1-8B pair (K=6K=6, L=8L=8, T=1T=1) at batch sizes B ∈ {1, 2, 4, 8, 16}. SMS-LP1 achieves the highest throughput at every tested batch size. At B=2B=2 and B=4B=4, SMS-LP1 outperforms the Tokenwise baseline by 29.7% and 28.2%. The full results (E2E throughput, tokens/s) are below:

BTokenwiseBVGBVSMS-LP1
158.2061.4557.9569.95
2102.31108.12105.51132.71
4176.21185.67182.13225.95
8293.25311.20275.47340.94
16432.48444.31375.78449.09

We have added these results to Appendix F (Table 12) of the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all the reviewers for their valuable feedback and insightful suggestions. All reviewers recognize the contributions of our method:

  • Novel problem formulation. Our work moves beyond the normal fixed-rule-based selection strategy, with a precise and under-explored problem formulation. [ Reviewers HeCc, 5Epy ]
  • Explicit realizability. The selector is provably valid: any proposal in the SMS family can be realized exactly by a simple sequential selector, which addresses a central difficulty, since choosing a path changes its proposal distribution. [ Reviewers HeCc, 6wUm ]
  • Coverage guarantee. Theorem 5 provides a coverage guarantee for the selector family, which gives a quantitative reason to study the family. [ Reviewer 6wUm ]
  • Low overhead and varied evaluation. SMS-LP1 is cheap to compute, with a closed-form update per visited prefix and low measured overhead, and is evaluated across several model pairs and DFlash2. [ Reviewers HeCc, 6wUm ]

Based on the reviews, we have made the following revisions to our paper. All changes are highlighted in blue.

  1. Newer models. We add the Qwen3-0.6B/32B and Qwen3.5-0.8B/Qwen3.8-27B pairs to the main results. SMS-LP1 achieves the highest BE and E2E on every benchmark, improving BE/E2E over Tokenwise by 29.4%/15.1% on Qwen3 and 26.6%/20.6% on Qwen3.8. (Sections 1, 7, Table 1, and Appendix D.1) [ Reviewer 5Epy ]
  2. Advanced drafters. We combine SMS-LP1 with the EAGLE-3 drafter for Llama-3.1-8B and the DSpark drafter for Qwen3-14B. SMS-LP1 achieves the highest BE and E2E on every benchmark, improving BE/E2E over Tokenwise by 30.0%/10.9% with EAGLE-3 and 16.5%/5.6% with DSpark. The averages throughout the paper now include all new settings; the results of the original model pairs are unchanged. (Sections 1, 7, Table 1, and Appendices D.1, D.2) [ Reviewer 5Epy ]
  3. Batched inference. We evaluate all methods at batch sizes from 1 to 16. SMS-LP1 achieves the highest throughput at every batch size, outperforming Tokenwise by 29.7%/28.2% at B = 2/4. (Table 12 and Appendix F) [ Reviewer HeCc ]
  4. Gap to the exact optimum. We compute the exact SMS-family optimum on 100 real draft trees, add a tie-break ablation, and work out the two-path example exactly. SMS-LP1 is within 0.5–0.8% of the exact optimum in BE, and the tie-break ablation shows that the gain comes mainly from the SMS family and the LP1 objective. (Tables 9–11 and Appendices C.1, E.5) [ Reviewers 6wUm, 5Epy ]
  5. Path budget. We extend the path-budget ablation to the Llama and DFlash2 pairs with the same grid of K. At each method's best K, SMS-LP1 is 11.4%/2.2% faster than GBV on Llama/DFlash2. (Section 7.3, Table 8, and Appendix E.4) [ Reviewer 6wUm ]
  6. Multi-path baselines. We further compare SMS-LP1 with Traversal Verification, where SMS-LP1 achieves higher BE and E2E. We also clarify our implementation of GBV. (Table 7 and Appendices D.2, E.3) [ Reviewer 6wUm ]
  7. Related work. We add a discussion of VIA-SD. (Section 2) [ Reviewer 6wUm ]

We have also addressed each reviewer's comments with detailed point-by-point responses. We sincerely appreciate the reviewers' suggestions, which have helped us improve the clarity and empirical evaluation of our work. We welcome further feedback and look forward to answering any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.