acceptodds
Under review as a conference paper at ICLR 2027

D-VOLT: Training Speculative Drafters with Token-Conditional Acceptance Values

14 views

Abstract

Speculative decoding accelerates language model inference by verifying several draft tokens in a single target-model forward pass. Training a lightweight drafter requires allocating its limited capacity to improve the expected number of accepted draft tokens. Existing approaches improve local distribution matching or reweight losses across positions, but candidate tokens at the same prefix can enable different expected numbers of subsequent acceptances. We identify a limitation of weighting a local acceptance objective by a detached continuation weight shared across candidate tokens at the same prefix and position. Even when the weight uses the exact mean continuation value conditional on acceptance, the resulting objective can favor updates that increase one-step acceptance probability while decreasing expected accepted length. We propose D-VOLT (Drafting via Value-Optimized Lookahead Training), which assigns training credit using token-conditional continuation values. We derive a Bellman recursion over complete prefix histories and an exact gradient decomposition of expected accepted length, in which each candidate's credit combines prefix reachability with its immediate acceptance reward and continuation value. For practical training, we approximate continuation values by dynamic programming over a compact top- candidate graph that shares sampled earlier histories across branches. We combine full-vocabulary total-variation matching with a candidate-specific continuation reward, retaining the drafter architecture and standard verification procedure. Experiments with DSpark drafters on BBH, MMLU-Pro, WinoGrande, and GSM8K at decoding temperatures and show that D-VOLT improves average accepted length by 4.05%–9.74% over the DSpark baseline and by 2.04%–6.46% over D-PACE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Learning, ML System, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_WGhSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper motivates the problem statement by empirically showing that candidate tokens proposed at the same point can lead to different numbers of later accepted tokens. It derives the expected accepted length under standard lossless speculative sampling and shows that an ideal drafter would have each candidate token receive a training credit according to both the probability of reaching that point and the expected number of later acceptances after that candidate. It gives a practical implementation of this formulation via D-VOLT, which approximates this using the top four candidates and adds the resulting candidate specific reward to a distribution matching loss. The exact derivation is mathematically consistent for autoregressive speculative sampling. However, the implemented loss has a DSpark specific approximation. The reported gains support the method for one Qwen3-8B target and one DSpark drafter, but the absence of additional drafter architectures, independent training runs, uncertainty estimates, complete data splits, and broader domain evaluation leaves its generalizability unresolved.

Critical strengths

a) The problem statement has a clear motivation b) The proposed solution seems to tackle the problem statement head-on and is practical to implement.

Critical weaknesses
  1. The general exposition and presentation are abysmal and fall well below the standards required for a technical paper. The manuscript replaces a straightforward idea with unnecessary made up jargon/terminology by the authors and then fails to use that terminology consistently. For example, the same underlying notion is called a continuation value, candidate value, and token conditional continuation value, while its use in training is variously described as candidate specific credit, continuation credit, a candidate specific continuation reward, and value corrected distribution matching.

These are just two specific examples; the manuscript is full of such inconsistent unnecessary phraseology, forcing the reader to reconstruct ordinary probability and dynamic programming concepts from unnecessary jargon. The mathematical notation compounds this exposition problem with lowercase and uppercase versions of the same quantity being used inconsistently. Similarly, the paper does not clearly separate the exact full tree derivation from the substantially different training objective used in practice until a brief qualification near the end of Section 4.2. The current writing leaves a lot to be desired because it obscures which claims are proved, which quantities are approximated, and which objective is optimized. The manuscript requires a comprehensive rewrite using standardized terminologies and notation and their consistent use throughout the paper.

  1. The manuscript use a single drafter Qwen-8B model and single dataset ( full Open-PerfectBlend mixture). This is a pretty limited evaluation. Moreover, the actual objective is limited to the original DSpark objective and D-PACE, although the paper cites other training objectives that directly target speculative acceptance. In particular, AngelSpec optimizes cumulative products of target/draft overlap across draft positions, making it a close sequence-level alternative to D-VOLT’s accepted-length objective. LK Losses directly optimizes target/draft overlap with substantially lower reported training overhead. A controlled comparison under the same DSpark architecture, initialization, training data, and optimization budget is needed to establish whether candidate-specific continuation credit improves over these simpler acceptance-oriented objectives.
Questions for authors

a) The hyperparameter-selection described in the paper does not match the claimed evaluation scope. D-VOLT is trained on the broad Open-PerfectBlend mixture, but both the continuation coefficient and candidate count are selected using only a held-out GSM8K set. These hyperparameters will be susceptible to the validation dataset chosen, particularly because the top-K approximation may cover different amounts of target probability in mathematics, code, and open-ended dialogue. Although the selected setting has the largest reported point estimate on the four test benchmarks, the paper provides no uncertainty estimates and evaluates neither code nor chat. The authors should justify GSM8K as the sole validation domain or select the global configuration using a fixed multi-domain validation mixture.

b) The relationship between the theoretical objective and the implemented training loss requires clarification. The exact analysis derives the gradient of expected accepted length over the complete prefix tree, using exact reach probabilities and token-specific continuation values. The practical objective in Equation 7 instead uses sampled histories, top-K continuation estimates, shared histories across candidate branches, fixed exponential position weights, and a tuned continuation coefficient. The authors acknowledge that this is a surrogate, but provide no bound or empirical gradient-alignment analysis connecting it to the exact objective. Moreover, Equation 7 uses L1 distribution matching, whereas standard fine-tuning uses cross-entropy and the original DSpark objective combines cross-entropy, total variation, and confidence prediction. The paper should clarify the complete loss used for every baseline and compare L1, cross-entropy or KL, and an adaptive LK objective under the same candidate-specific continuation term.

c) Most of the presented results are for a single combination of Qwen3-8B target, DSpark drafter, and one block length. Is there any thing specific in your formuation that limits the evaluation to only Dspark drafter? The reported benchmarks are also concentrated on reasoning and multiple-choice tasks. No code or open-ended dialogue benchmark is included, even though code and chat constitute a substantial portion of Open-PerfectBlend. Because the candidate-specific continuation reward is estimated from sampled training histories, its benefit may depend on the input distribution. I am not sure how much of this new method's improvement would be transfer to cross domain distributions. Evaluation on code, dialogue, and a second target or drafter configuration would be needed to establish the claimed generality.

d) The main results report only point estimates, with no standard deviations for independent seeds, making the stability of the claimed 2 to 3 percent improvements impossible to assess. The paper also omits essential details of the training, validation, and test splits, so the principal empirical claims cannot be independently reproduced or verified.

e) As a reader, I would be more interested in understanding the failure patterns and tradeoffs introduced by D-VOLT, rather than seeing only improvements in mean accepted length. The paper does not report which inputs or token positions perform worse, whether the probability of accepting no draft tokens increases, or whether favoring candidates with longer predicted continuations reduces target matching elsewhere. Reporting these cases, together with performance under changes in input domain, would clarify when the method helps and when its training objective becomes harmful.

11 comments

We respectfully disagree with the reviewer's overall assessment. Several of the central criticisms rest on factual misreadings of the paper, which we correct below. Where a concern is empirical, we answer with new experiments.

1. "Single dataset." This is incorrect. Open-PerfectBlend is the training mixture. The paper evaluates on four separate benchmarks (BBH, MMLU-Pro, WinoGrande, and GSM8K) at two decoding temperatures. Treating the training corpus as the evaluation scope conflates two different things.

2. "Single target / single drafter / single block length; no code or dialogue." We have now removed this concern entirely (General Response G1). We trained matched DSpark and D-VOLT drafters for Qwen3-8B with B=16 and for a different, larger MoE target, Qwen3.5-35B-A3B with B=8. Each was evaluated at T=0 and T=1 on eight benchmarks spanning reasoning, code (MBPP, HumanEval), and open-ended chat (IFEval, Arena-Hard-Auto). D-VOLT improves accepted length in all 32 of 32 settings. Gains are +3.8% to +5.4% on code and +6.0% to +8.1% on chat. The reviewer's hypothesis that the benefit "may not transfer to cross-domain distributions" is directly contradicted. Code gains are on par with reasoning gains (mean +4.7% vs. +5.2%), and chat gains are the largest of any domain (mean +7.0%).

3. "Hyperparameters selected on GSM8K only will not generalize." The new results test this directly. With the same defaults, the gain on GSM8K itself (+4.0% to +4.9%) is at or below the median in every one of the four new settings, while chat gains exceed it in all of them. If the configuration were overfit to the selection domain, we would expect the opposite pattern. The data shows no such effect.

4. "The formulation is DSpark-specific." This is incorrect. The Bellman recursion and the exact gradient decomposition (Eqs. 4–5) are stated for any autoregressive drafter under standard lossless verification. They use no DSpark structure. Only the efficient estimator in Eq. 6 exploits DSpark's parallel base logits plus Markov head. For a sequential drafter, the same DP runs over a top-K tree built from the drafter's own proposals. The new B=8 and B=16 results confirm that the method is not tied to one block length either.

5. "No empirical gradient-alignment analysis connecting the surrogate to the exact objective." We now provide exactly this (General Response G2). Using the same checkpoint and prompts as Appendix A, we compared the practical estimate (Eq. 6) against an unbiased full-history Monte Carlo estimate of the exact value (Eq. 4). The study covers about 18.5k (block, position) records per setting at K∈{4, 8, 16}, with bootstrap CIs and an explicit MC noise ceiling.

  • Under the loss's accepted-mass weighting, the estimation error is 0.20–0.24 tokens.
  • The relative order of the two highest-mass candidates matches the exact-value MC 80–83% of the time, and 86–88% when MC resolves the gap.
  • The estimate is stable when the reference path is resampled (std 0.05–0.08 tokens; rank τ 0.91–0.94).
  • Bias and error shrink monotonically with K.

The surrogate therefore tracks the ordering that drives the exact gradient, and its residual bias is quantified rather than unknown. We also note that the paper never claims Eq. 7 is the exact gradient. Sec. 4.2 explicitly calls it a practical surrogate and lists each approximation, which contradicts the claim that the paper "does not clearly separate" the two.

6. "Made-up jargon used inconsistently." The terms the reviewer lists as synonyms denote three different mathematical objects:

  • the continuation value Vi(h,t)V_i(h,t), a quantity;
  • the candidate's credit ρi(h)⋅(1+Vi(h,t))ρ_i(h)·(1+V_i(h,t)), a gradient weight in Eq. 5;
  • the continuation reward −2βΣmin(p,q)⋅sg[V^]−2β Σ min(p,q)·sg[\hat{V}], a loss term in Eq. 7.

"Token-conditional" is not decoration. It marks the paper's core contrast with prior objectives that use a continuation weight shared across candidates. Bellman recursion, reachability, and backward DP are standard terms, used in their standard sense. That said, we will add a one-line glossary and use a single name for each object.

7. Effect size. The review refers to "the claimed 2 to 3 percent improvements." That range corresponds to the Candidate- vs. Shared-value ablation (1.76%–3.04%), which is a deliberately strict control. The headline gains over DSpark are 4.05%–9.74% (Table 1). The new experiments add 32 further settings, with gains of +1.4% to +9.1% and a median of +5.1%.

We hope that, in light of these corrections and the new evidence on generality, cross-domain transfer, and surrogate fidelity, the reviewer will reconsider the score.

Reviewer_WGhSOpenReview ↗

Rebuttal doesn't address the concerns raised

Thank you for the additional experiments. They partially address evaluation breadth, but several responses misquote my review or leave the requested evidence unanswered.

First, I did not write that “The formulation is DSpark-specific.” I wrote that the exact derivation is general, while the implemented loss contains a DSpark-specific approximation, and asked for evaluation with another drafter architecture. The response itself acknowledges that Equation 6 exploits DSpark’s parallel logits and Markov head. All new experiments still use DSpark. Changing the target model and block length does not test another drafter architecture.

The response similarly changes my concern that GSM8K-only hyperparameter selection “will be susceptible to the validation dataset chosen” into the categorical claim that it “will not generalize.” I did not make that claim. Showing gains on other benchmarks is useful, but it does not test whether another validation domain would select different values of β and K. Likewise, I asked how much the benefit transfers across input distributions. I did not predict that transfer would fail. Code and chat evaluations broaden the evidence, but Open-PerfectBlend already contains code and chat data, so these results do not establish transfer outside the training distribution.

My requested loss comparison is entirely unanswered. I asked the authors to specify the complete loss used for every baseline and compare L1, cross-entropy or KL, and an adaptive LK objective while holding the candidate-specific continuation term fixed. The Candidate Value versus Shared Value ablation changes the continuation credit while retaining the same base loss, so it answers a different question. Mentioning alternative losses in related work is not the requested controlled comparison.

The independent-seed uncertainty concern is also unanswered. Monte Carlo standard errors and bootstrap intervals for continuation-value estimation do not measure variation across independently trained models. The response provides no number of training runs, standard deviation across checkpoints, confidence interval for the main comparisons, or evaluation-seed accounting. Reviewer 9hpN independently asked how many training runs were conducted and requested variability across seeds. The authors answered that reviewer’s Questions 1, 3, and 4 while skipping Question 2 about independent training runs. Reporting improvements in 32 benchmark settings does not replace repeated training runs.

The response also says it provides “exactly” the requested gradient-alignment analysis, but it measures continuation-value error and candidate-ranking agreement. Elsewhere, the authors admit that they “have not measured end-to-end gradient cosine similarity during training.” Value-estimation accuracy is relevant, but it is not gradient alignment. The reported 80 to 83 percent ranking agreement also means disagreement in approximately 17 to 20 percent of evaluated records.

The claimed new experiments are not in the supplied manuscript. It still contains only Qwen3-8B, a DSpark drafter with block length 7, and the original four benchmarks. It contains none of the new Qwen3.5, block-length-16, code, chat, or graph-versus-Monte-Carlo results. The authors repeatedly refer to “General Response G1” for the full per-benchmark table, but that table is absent from the current OpenReview draft. Only aggregate means and ranges are visible. I also find no supporting code, checkpoints, or raw results.

Finally, the response does not address the requested failure analysis, complete data splits, or conditions under which D-VOLT performs worse. It continues to report mean accepted length without independent-seed uncertainty or the distribution of per-request outcomes.z

The new experiments provide useful preliminary evidence across another target, additional block lengths, and broader task categories. They do not resolve the controlled-loss, independent-seed, different-drafter, reproducibility, failure-analysis, or artifact concerns. Several responses instead rebut stronger claims that I did not make. My rating therefore remains unchanged.

Reviewer_WGhSOpenReview ↗

Problem with AI slop

My concern is not AI assistance itself. The concern is that the resulting prose changes genuine qualified questions into categorical claims, obscures unresolved distinctions, and makes the scientific response less precise. Instead of acknowledging this problem and striving to fix this in the revised draft, the authors are pretty defensive and doubling down

The authors dismiss the writing concerns raised by me in the review by separating “continuation value,” “candidate credit,” and “continuation reward” as distinct mathematical objects. That response addresses only a narrow subset of the problem. The submitted manuscript repeatedly renames the same continuation quantity as “continuation value,” “candidate value,” and “token-conditional continuation value,” then surrounds it with opaque labels such as “allocation headroom,” “granularity of continuation credit,” and “value-corrected distribution matching.” A first time cold reader cannot make sense of these non-standard jargon. None of these labels makes the underlying operations clearer. “Allocation headroom” means the maximum improvement in an estimated return under fixed probability constraints, while “value-corrected distribution matching” means L1 matching with an additional continuation reward. Stating those operations directly would be shorter and more precise.

This is not merely a stylistic preference. The manuscript’s compressed, AI-style phrasing obscures which quantity is estimated, which objective is optimized, and which parts of the exact derivation survive in the practical algorithm. The clearest example is the statement that the timing results compare “complete training configurations with different software versions and supervision paths.” The authors explicitly acknowledge in another response that an LLM introduced this “overly defensive statement,” yet it remains in the submitted manuscript and still does not identify what differed or how the differences affect the comparison. Their promise to add a glossary does not address the larger problem: ordinary mathematical operations are repeatedly hidden behind unnecessary labels, while consequential differences between the exact gradient and the implemented approximation receive only brief and indirect explanation.

The rebuttal itself continues the manuscript’s writing problem. Phrases such as “removed this concern entirely,” “directly contradicted,” “we now provide exactly this,” “the data shows no such effect,” “consistent, low-noise signal,” and “surrogate fidelity” are vague or inflated substitutes for stating what was actually measured. Several of these claims overstate the evidence: value-ranking agreement is presented as gradient alignment, transfer across task categories is presented as robustness to distribution shift, and a second target model is presented as resolving the absence of another drafter architecture. The authors also acknowledge in another response that they used an LLM to summarize experimental results and that it introduced “an overly defensive statement.” The same defensive, formulaic style remains throughout this rebuttal.

Reviewer_tYesOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recent work on speculative decoding aims to design better loss functions for training drafters to maximize accepted length. Due to the limited capacity of drafters and the non-convex nature of optimization, drafters cannot perfectly match the target distribution. Under these limitations, the loss function should guide the draft distribution to improve accepted length.

In this paper, the authors focus on the continuation value of each token. The authors highlight that tokens at the same prefix and position can have different continuation values. The authors derive a Bellman recursion and an exact gradient decomposition for expected accepted length, and approximate the continuation values using backward dynamic programming over a compact top-K graph. Based on these estimated continuation values, the authors propose D-VOLT, which assigns more training credit to tokens with higher continuation values.

Critical strengths

S1. The paper is well-structured, clearly presenting the motivation, methodology, and experimental results.

S2. The authors provide detailed and solid analyses.

  • For example, in Section 3.1, the authors report that the mean within-record standard deviation of the estimated continuation values across 16 candidates is 0.937 tokens. At first glance, I wondered whether this variation could simply be due to noise from Monte Carlo estimation. However, in Appendix A, the authors provide an analysis based on a concentration inequality, showing that at least some of the observed variation cannot be explained by Monte Carlo estimation noise alone. This supports the existence of genuine differences in continuation values across candidates.
Critical weaknesses

My main concerns are W2 and W3. W1 reflects my personal impression, and I do not consider it a serious weakness. I would be willing to raise my score if the authors adequately address W2, W3, and Q1.

W1. The paper presents a well-justified but relatively incremental contribution rather than a fundamental conceptual advance.

  • Previous work has already explored various training losses to improve expected accepted length. In this context, the idea of "considering future returns rather than only immediate acceptance" and "assigning different training credit to candidate tokens based on their continuation values" seems relatively natural. Therefore, I do not think it will significantly influence the future research in this area. That said, I do not mean to undervalue the contribution of this paper. The authors have carefully developed this idea and supported it with detailed analyses and experiments, which I believe is a clear contribution of the work.

W2. I think the explanations in Section 4 could be clearer. Below are some points that I found confusing.

  • In Section 4.1, the authors derive the exact gradient of J1(h0)J_1(h_0) in Equation 5. However, I am not sure what role this equation plays in the proposed method. It seems that Equations 6 and 7 could be motivated directly from Equation 4, without Equation 5.
  • It is unclear how Equation 6 is derived from Equation 4. My understanding is that Equation 4 defines the exact continuation value, but computing it is intractable, so the authors introduce Equation 6 as a practical approximation. However, the connection between these two equations is not clearly explained. In particular, Section 4.1 defines Vi(h,t)V_i(h,t) for every possible token sequence from positions 1 to i−1i-1, while Section 4.2 uses V^i(t)\hat{V}_i(t), where hh (the dependence on the history) is no longer explicit in the notation. This difference is somewhat confusing and I think the authors should explicitly explain under what assumptions and approximations Equation 4 can be reduced to the practical formulation in Equation 6, and show the steps involved.
  • Along with this explanation, I think the authors should discuss the limitations introduced by these approximations more clearly. For example, the top-K candidates at each position are selected based on a single sampled path, rather than expanding all possible branches from the previous position's top-K candidates. This could exclude important candidates in other branches and affect the accuracy of the estimated continuation values. More generally, I think the authors should clarify what information is lost when approximating the intractable VV with the tractable V^\hat{V}, and what limitations this may introduce. Although these approximations are briefly mentioned near the end of Section 4.2, their implications are not sufficiently discussed.

W3. While the structure of the paper is well-organized, I think some parts of the presentation should be improved.

  • I think there are some jargons and expressions that need to be explained more clearly or removed. For example, In line 291, the term "overlap" is not clearly defined in the main text. I understand that it refers to min{p, q}, but readers with less background in speculative decoding may find it confusing. Also, in line 445, the authors state "These measurements compare complete training configurations with different software versions". Here, I do not fully understand why “different software versions” is mentioned here or how this detail is relevant to the discussion.
  • I think the authors could better explain the purpose of each section and how it connects to the previous or following sections. Many sections focus mainly on presenting results, without clearly explaining why these results matter in the overall argument of the paper. For example, Section 3.1 msotly reports experimental results, but does not sufficiently explain their implications or how they motivate the analysis in Section 3.2. I think adding a few sentences that explain the purpose of each section and connect it to the next would make the paper much easier to follow. This suggestion applies not only to Section 3.1 but also to other subsections in Sections 3 and 4.
  • My overall impression is that, while the authors seem to have used AI quite a bit when writing this paper, they also appear to have carefully polished much of it themselves. However, they are still a few parts that may need further review, so I suggest the authors to review the paper once more to improve its overall flow.
Questions for authors

I have one major question regarding the design of the training objective.

Q1. I have a question on the training objective in equation 7. In D-PACE, the loss is designed such that ∇θLD-PACE=−∇θS~\nabla_\theta \mathcal{L}_{\mathrm{D\text{-}PACE}}=-\nabla_\theta\tilde S, where S~\tilde S is a surrogate for the expected accepted length. In contrast, D-VOLT derives the exact gradient of the expected accepted length J1(h0)J_1(h_0) in equation 5, but the final design in equation 7 combines ℓi\ell_{i} using fixed weights ωi\omega_{i} (similar to DFlash) rather than prefix-dependent reachability weights. Could the authors provide justification for this design choice?

  • Following the derivation of equation 5, I think a natural approach would be to design the D-VOLT loss such that ∇θLD-VOLT=−∇θJ1(h0)\nabla_\theta \mathcal{L}_{\mathrm{D\text{-}VOLT}} = -\nabla_\theta J_1(h_0), similar to how D-PACE constructs its loss to match the gradient of S~\tilde S. While I understand that exactly computing equation 5 would be computationally intractable, I am curious why the authors chose fixed position weights instead of approximating the reachability weights using the sampled draft prefixes.
6 comments

We sincerely appreciate the reviewer’s professional, experienced, and insightful review! It provides us with valuable guidance and profound insights into the representation aspect of our work. In the following, we would like to address W2, W3, and Q1 separately. We also welcome the reviewer to raise any questions or concerns that may arise during our discussion.

W2. I think the explanations in Section 4 could be clearer. Below are some points that I found confusing...

1. Regarding the first point, Equation (5) is indeed necessary. Equation (4) derives Ji(h)J_i(h) for an individual position under a fixed context. There are two important aspects worth noting here.

First, Vi(h,t)V_i(h,t) in Equation (4) actually has a gradient with respect to the parameters θ\theta. In contrast, sg[V^i(t)]sg[\hat{V}_i(t)] in Equation (7) is a pure scalar detached from the computation graph.

Second, what we actually optimize is only J1(h0)J_1(h_0) at the first position, rather than the sum of JJ over all positions, because J1(h0)J_1(h_0) has already taken into account the reception of subsequent positions.

Therefore, if we want to express Ji(h)J_i(h) as an accumulation of objectives across different positions, we first need to derive the corresponding gradient decomposition of ∇θJi(h)\nabla_\theta J_i(h) into the sum of gradients from individual positions. In this sense, Equation (7) is actually an optimization objective reconstructed by fitting the individual gradient terms in Equation (5). It cannot be directly derived from Equation (4) alone; additional derivations from Equation (4) are required to establish this connection.

Specifically, our actual optimization target is:

J_1(h_0) =\sum_{t\in V}m_1(h,t)[1+V_1(h,t)] =\sum_{i=1}^{B}\sum_{h\in H_{i-1}}\rho_i(h) \sum_{t\in V}sg[1+V_i(h,t)]m_i(h,t).$$ We acknowledge that there are some logical gaps in the derivation as currently presented. We plan to provide a more detailed derivation in the final version, either in the main text or in the appendix. **2. Regarding the second point**, this is actually explained in Section 4.2, where we state that $\hat{V}_{i+1}(u)$ is computed under $(h_0,s_{<i+1},u)$. This condition should indeed have been explicitly included in the corresponding equation, and its omission was an oversight on our part. To address the reviewer's concern, we provide a more complete derivation from Equation (4) to Equation (6). We first have: $$V_i(h)=J_{i+1}(h) =\sum_{t\in\mathcal{V}}m_{i+1}(h,t)[1+V_{i+1}(h,t)].$$ Let $h_i=(h_0,s_{<i})$, where $s_i$ denotes the token actually sampled at position $i$. Then, it naturally follows that: $$V_i(h_i)=J_{i+1}(h_i) =\sum_{t\in\mathcal{V}}m_{i+1}(h_i,t)[1+V_{i+1}(h_i,t)].$$ Since $(h_i,t)$ in $V_{i+1}(h_i,t)$ has $|\mathcal{V}|$ possible values, we replace the original vocabulary $\mathcal{V}$ with the Top-K vocabulary $\mathcal{V}_{i+1}'$ at position $i+1$: $$V_i(h_i)=J_{i+1}(h_i) =\sum_{t\in\mathcal{V}_{i+1}'} m_{i+1}(h_i,t)[1+V_{i+1}(h_i,t)].$$ Since computing $V_i$ requires the values of $V_{i+1}(h_i,t)$, our goal is to obtain $V_i(h_{i-1},t)$ for all possible positions $i$ and all $t\in\mathcal{V}_i'$, and then recursively compute $V_0(h_0)$. Accordingly, for $V_i(h_{i-1},t)$, we have: $$V_i(h_{i-1},t) =J_{i+1}(h_{i-1},t)\\ =\sum_{t'\in\mathcal{V}_{i+1}'} m_{i+1}(h_{i-1},t,t') [1+V_{i+1}(h_{i-1},t,t')].$$ However, recursively computing $V_{i+1}(h_{i-1},t,t')$ exactly would introduce exponential computational complexity. Therefore, in practice, we use $V_{i+1}(h_i,t')$ as an approximation of $V_{i+1}(h_{i-1},t,t')$. Our intuition is that the value of $V$ is influenced more strongly by the most recent token $t'$ than by the preceding token $t$. This approximation is motivated by the Markov property. Accordingly, we obtain: $$V_i(h_{i-1},t) =J_{i+1}(h_{i-1},t) \approx \sum_{t'\in\mathcal{V}_{i+1}'} m_{i+1}(h_{i-1},t,t') [1+V_{i+1}(h_i,t')].$$ By defining $\hat{V}_i(t)=V_i(h_{i-1},t)$ and substituting it into the above expression, while replacing $t'$ with $u$, we obtain: $$\hat{V}_i(t)= \sum_{u\in\mathcal{V}_{i+1}'} m_{i+1}(h_{i-1},t,u)[1+\hat{V}_{i+1}(u)].$$ This gives us the polynomial-complexity recursive formulation presented in Equation (6). During the writing process, we used an LLM to condense this manually derived explanation. Unfortunately, this led to some intermediate steps and logical connections being omitted from the manuscript. We will work to restore these derivation details in the final version.

W2. I think the explanations in Section 4 could be clearer. Below are some points that I found confusing...

3. Regarding the third point, we first address the concern about VV and V′V'.

As explained in the derivation above, we use Vi+1(hi,t′)V_{i+1}(h_i,t') to approximate Vi+1(hi−1,t,t′)V_{i+1}(h_{i-1},t,t'). The information lost in this approximation is precisely the dependence on the preceding token tt. Regarding this limitation, reviewers 9hpN and XqgD have raised similar concerns. We plan to conduct an additional experiment comparing our estimates with Monte Carlo (MC) sampling results to quantify the gap between the estimated values and the ground-truth values.

Regarding the Top-K approximation, we believe that Top-K already preserves the most important candidate tokens. As reported in Appendix B.1, the probability mass retained by the Top-16 distribution at arbitrary positions exceeds 99%. It is therefore highly unlikely that the target model will sample a token outside the Top-16 distribution. Even if an out-of-distribution token is sampled, its impact on the quality of multiple sampling trials remains very limited after weighting by its sampling probability.

We acknowledge that we have not sufficiently emphasized this point in the main text, which is a shortcoming in how we present our method. We appreciate the reviewer's suggestion and will revise the manuscript accordingly in the next version.

Reviewer_XqgDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes D-VOLT, a training method for speculative drafters that incorporates token-conditional continuation values to improve expected accepted length. The authors identify a limitation of existing objectives that use shared continuation weights, showing that improving one-step acceptance probability can decrease expected accepted length. They derive an exact gradient decomposition based on prefix reachability and candidate-specific continuation values, and introduce a practical approximation using dynamic programming over a compact top-K candidate graph. Experiments on four benchmarks demonstrate consistent improvements over DSpark and D-PACE.

Critical strengths
  1. Well-motivated problem. The paper identifies an important mismatch between local distribution matching and the actual objective of speculative decoding. The counterexample clearly illustrates why increasing one-step acceptance probability does not necessarily improve expected accepted length.

  2. Principled theoretical formulation. The Bellman recursion and exact gradient decomposition provide a solid theoretical justification for candidate-specific credit assignment rather than shared position-level weights.

  3. Consistent empirical improvements. D-VOLT outperforms DSpark and D-PACE across all four benchmarks at both decoding temperatures. The ablations further support the importance of candidate-specific continuation values over shared or shuffled values.

Critical weaknesses
  1. Substantial training overhead. D-VOLT incurs approximately 5.05× higher training-step time than DSpark in the reported comparison, while improving average accepted length by 4.05%–9.74%. The paper does not establish whether these improvements justify the additional training cost in practical deployments.

  2. Gap between theory and implementation. Although the theoretical gradient decomposition is exact, the practical method relies on top-K truncation, shared sampled histories, and detached approximate continuation values. The paper provides limited analysis of how these approximations affect gradient accuracy or optimization effectiveness.

  3. Limited evaluation of practical inference efficiency. The experiments focus on average accepted length rather than end-to-end decoding throughput or latency. Consequently, the actual inference speedups and deployment benefits remain unclear.

Questions for authors

See weaknesses

3 comments

We thank the reviewer for the positive assessment and for the three concrete concerns, which we address in turn.

W1: is the 5.05× training-step overhead justified?

  • Where the cost comes from. The overhead is concentrated in the forward pass: forward time rises from 2.741 s to 19.576 s, while backward time is essentially unchanged (1.385 s vs. 1.328 s). The extra forward work is target evaluation of (B−1)K = 24 candidate nodes per block. The DP itself (O(BK²) scalar operations) and the backward pass add negligible cost. As noted in Sec. 5.5, this comparison also mixes software versions and supervision paths, so 5.05× should be read as an upper estimate for the method itself rather than as an optimized implementation.
  • Training is a one-time cost; the inference gain recurs. The drafter is trained once, and D-VOLT changes neither the deployed architecture nor verification. Every gain in accepted length therefore applies to every subsequent decoded token at zero additional inference cost. In the new experiments (General Response G1), D-VOLT improves AL in all 32 settings across two targets (Qwen3-8B with B=16 and Qwen3.5-35B-A3B with B=8), two temperatures, and eight benchmarks (reasoning, code, and chat), by +1.4% to +9.1%.
  • The cost is tunable. Cost scales linearly with K through the number of target-evaluated nodes. Our new audit (G2) shows how estimate quality improves with K (MAE/H: 15.4% → 13.8% → 11.7% for K = 4 → 8 → 16), which makes the cost–accuracy trade-off explicit.

W2: gap between theory and implementation (top-K truncation, shared histories, detach). We directly measured the error that these approximations introduce into the quantity that weights the gradient, namely the per-candidate continuation value. Full details are in General Response G2. We compared the compact-graph estimate (Eq. 6) with an unbiased full-history Monte Carlo estimate of the exact value (Eq. 4) on 18.5k–19k (block, position) records at K∈{4, 8, 16}:

  • Top-K truncation plus shared histories yield a downward bias (−0.35 tokens at K=4, falling to −0.21 at K=16). The bias is concentrated on lower-probability alternatives (−16% of horizon for the second candidate vs. −4% for the dominant one).
  • The error that matters for the loss is smaller. Under the loss's min(p,q) weighting, MAE is 0.243 tokens at K=4 and 0.204 at K=16. The relative order of the two highest-mass candidates matches the MC estimate 80–83% of the time, or 86–88% when MC resolves the gap beyond 2 SE.
  • Variance is low. Resampling the reference path changes V^\hat{V} by only 0.05–0.08 tokens (std), and rankings across resamples have τ≈0.91–0.94. The approximation therefore produces a consistent, low-noise signal rather than a noisy one.
  • Detach. With the exact values and reachabilities, detaching the weights is exact (Eq. 5): differentiating the stop-gradient surrogate recovers −∇J1−∇J_1. Detaching is therefore not an additional approximation beyond the value error quantified above.

We will add this audit as an appendix. We will also state explicitly in Sec. 4.2 that Eq. 6 is a biased, low-variance surrogate that under-credits high-value alternatives, noting that increasing K reduces this bias. We have not measured end-to-end gradient cosine similarity during training, and we will list this as a limitation.

W3: practical inference efficiency. D-VOLT's inference graph is identical to DSpark's: same drafter architecture, same target, same verification, and no extra computation at inference. Per-block drafting and verification cost is therefore unchanged. Each block emits AL+1 tokens, so at matched per-block cost the decoding speedup equals the ratio of (AL+1). Under this model, the new results imply speedups over DSpark of +3.3% to +7.3% (Qwen3-8B, B=16, T=0) and +3.8% to +6.8% (Qwen3.5-35B-A3B, B=8, T=1) on the eight benchmarks.

Thank you for the detailed clarifications and additional experiments. My concerns have been addressed, and I have no further questions. I maintain my positive rating.

We sincerely thank the reviewer for the constructive feedback and for confirming that the concerns have been addressed. We appreciate the continued support of our work.

As promised, the revised manuscript will include:

  • the graph-versus-MC value audit as a new appendix;
  • an explicit statement in Sec. 4.2 that the compact-graph estimate is a biased, low-variance surrogate;
  • the additional results on Qwen3-8B with B=16 and Qwen3.5-35B-A3B with B=8 across eight benchmarks;
  • a clarified discussion of the training-cost breakdown and the inference-efficiency analysis.

Thank you again for your time and the helpful suggestions.

Reviewer_9hpNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes D-VOLT, a method for training speculative decoding draft models using the continuation value of each candidate token. Its main observation is that different tokens at the same prefix and position can lead to different expected numbers of subsequent acceptances. Giving all candidates the same continuation weight may therefore overlook their different contributions to expected accepted length.

The authors present a counterexample in which local acceptance probability increases while expected accepted length decreases. They then derive a Bellman recursion and an exact gradient decomposition over the full prefix tree. In practice, the method approximates continuation values using a graph of the top K candidates and shared sampled histories. These values provide an additional reward alongside an L1 loss that matches the target distribution over the full vocabulary.

The authors evaluate D-VOLT using Qwen3-8B as the target model and DSpark as the drafter on BBH, MMLU-Pro, WinoGrande, and GSM8K at two decoding temperatures. Results show improvements in average accepted length over both DSpark and D-PACE. Ablations comparing a shared value, separate values for individual candidates, and shuffled values provide further evidence for the proposed mechanism.

Critical strengths
  1. The motivation for assigning credit to individual candidates is clear and supported by the analysis. Different tokens at the same prefix can have different continuation values, so improving local acceptance probability does not necessarily increase expected accepted length. The counterexample illustrates this distinction, while the gradient decomposition over complete histories shows how prefix reachability and continuation value determine each candidate’s contribution to the training gradient. This provides a useful basis for designing drafter objectives that target accepted length.
  2. The ablations directly examine the proposed mechanism and show consistent gains. Under the same sampled prefix setup, the authors compare variants with no value reward, a shared value, separate values for each candidate, and shuffled values. Using separate candidate values outperforms the shared value variant on all four benchmarks, supporting the importance of matching candidates to their estimated values. The main experiments also show improvements at both decoding temperatures. The method retains the deployed drafter architecture and standard verification procedure, allowing the trained model to use the existing inference pipeline.
Critical weaknesses
  1. When estimating continuation values, the method replaces parts of a branch’s history with tokens from the reference path to reduce computation. This approximation may change the relative values of candidates and assign lower training weights to candidates with higher continuation values. The paper does not compare these estimates with estimates that retain complete branch histories for the same prefixes and candidates, or report agreement between the resulting rankings. The estimation error introduced by history sharing and its effect on candidate rankings therefore remain unclear.
  2. Reference paths are sampled during training. Repeating training with the same settings can produce different paths, leading to different candidate sets, value estimates, and training weights. The version using separate values for individual candidates improves average accepted length by 1.76% to 3.04% over the shared value variant. The paper does not specify the number of independent training runs, how the reported results were aggregated, or the variation across random seeds. It is therefore unclear how consistently these gains can be reproduced. Evaluation is also limited to Qwen3 8B, a single DSpark drafter architecture, and a fixed block length of B=7. The results do not establish whether similar gains hold for other target models, drafter configurations, or block lengths.
Questions for authors
  1. On a small representative subset, could you compare the graph estimates with Monte Carlo estimates that retain the full history of each sampled branch, using the same checkpoint, prefixes, and candidate tokens? Please report ranking agreement, the sampling budget, and the uncertainty in the estimates. On the same subset, how much do the estimated values and rankings of common candidates vary when the reference path is resampled?
  2. Are the gains in Table 2 consistent across repeated training runs? How many independent runs were conducted for each variant, and do the reported numbers come from a single run or an average? Could you report the AL differences between the Candidate value and Shared value variants across training seeds, including their mean and variability?
  3. Are draft trajectories, top K candidate sets, and continuation values recomputed at every optimizer step, or reused across multiple updates? If they are reused, please specify the refresh intervals and the number of optimizer steps performed per generated batch. Do trajectories and value estimates follow the same refresh schedule?
  4. Do these gains extend to other model configurations or block lengths? Have you evaluated settings beyond Qwen3 8B, a DSpark drafter with five layers, and B=7? Could you provide a comparison using another block length or model configuration to examine whether separate values for individual candidates still outperform a shared value?
3 comments

We thank the reviewer for the precise and constructive questions. Each one maps to a concrete experiment, and we ran the experiments for Q1 and Q4.

Q1 / W1: graph estimates vs. full-history MC. We ran exactly this comparison. We used the same checkpoint and the same 100 GSM8K prompts as Appendix A, at T=1. For every sampled prefix and draft position, we took the top-K candidates and computed two values for each candidate:

  • Graph: the training estimate from Eq. 6.
  • MC: an unbiased Monte Carlo estimate of the exact value (Eq. 4). It samples 16 draft continuations per candidate while keeping the candidate's complete history.

Budget and uncertainty. The study covers about 3,100 blocks and 18,500 (prefix, position) pairs for each K. The MC standard error is about 0.09 tokens per candidate, which is much smaller than the differences reported below.

K=4 (default)K=8K=16
(a) Value error, graph vs. MC (tokens)0.500.450.38
(b) Underestimation, most likely candidate4%4%3%
(b) Underestimation, second most likely candidate16%13%11%
(c) Ranking agreement with MC80%81%83%
(d) Value change when the reference path is resampled (tokens)0.080.060.05
(d) Top-ranked candidate unchanged after resampling92%92%91%
  • (a) Mean absolute difference between the graph and MC values of the same candidate. For scale, the mean remaining horizon is 3.5 tokens.
  • (b) Average amount by which the graph value falls below the MC value, as a percentage of the remaining horizon.
  • (c) For each prefix, we take the two candidates with the largest accepted mass min(p,q), which dominate the gradient, and report how often the graph orders their values the same way as MC does.
  • (d) We keep the prefix and candidates fixed and resample the reference path 4 times, as in training. We report the standard deviation of each candidate's graph value and how often the top-ranked candidate stays the same.

Takeaways. History sharing does bias the estimate, as the reviewer anticipated: it underestimates lower-probability alternatives (b), while the estimate for the most likely candidate is nearly unbiased. Despite this bias, the graph orders the two candidates that drive the gradient the same way as the exact MC values 80–83% of the time (c). The estimate is also stable under resampling of the reference path (d): its variation is 6–8× smaller than its error, so the error is a consistent bias rather than noise. Increasing K reduces the bias. We will add this analysis as an appendix and state the limitation explicitly in Sec. 4.2.

Q3: refresh schedule. Nothing is cached or reused across optimizer steps. For each microbatch, the draft path is sampled online from the current drafter parameters. The top-K candidate sets, the target evaluations of candidate nodes, and the backward DP for V̂ are then computed in the same forward pass, and the result is used for exactly one gradient update. Trajectories, candidate sets, and value estimates therefore all follow the same schedule: they are recomputed at every optimizer step. There is no separate value network and no value buffer. We will state this explicitly in Sec. 4.2.

Q4: other targets, drafter configurations, and block lengths. Yes. We trained matched DSpark and D-VOLT drafters for Qwen3-8B with B=16 and for a different and larger target, Qwen3.5-35B-A3B (MoE), with B=8. Both were evaluated on eight benchmarks (reasoning, code, and chat) at T=0 and T=1; the full per-benchmark table is in General Response G1.

SettingMean AL, DSpark → D-VOLTPer-benchmark gain rangeBenchmarks improved
Qwen3-8B, B=16, T=05.040 → 5.315+3.8% to +9.1%8 / 8
Qwen3-8B, B=16, T=15.078 → 5.293+1.4% to +7.6%8 / 8
Qwen3.5-35B-A3B, B=8, T=04.253 → 4.477+3.9% to +7.7%8 / 8
Qwen3.5-35B-A3B, B=8, T=14.129 → 4.386+4.9% to +8.3%8 / 8

The Monte Carlo comparison addresses my main concern about the value estimates. I’m still unsure how much the gain over shared values varies across training runs, so I’m keeping my score at 3.

Thank you for your review. The gain over shared values remains an important consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.