acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding

11 views

Abstract

In speculative decoding, a lightweight draft model proposes tokens that the target model verifies in parallel, speeding up generation without changing the target distribution. D-PACE trains such drafters with a loss that weights each position by its contribution to the expected accepted length. Both its weights and its cross-entropy (CE) loss use only the draft's probability of the recorded target token, treating the target as a point mass. Under rejection sampling, however, a sampled draft token is accepted with probability equal to the overlap of the draft and target distributions, one minus their total variation (TV). We derive D-PAL (Dynamic Position-wise Acceptance Loss) from the exact expected accepted length, using this overlap as the acceptance signal. This yields overlap-based position weights with log acceptance (LA) as the per-position loss, and the objective reduces to D-PACE for a point-mass target. Since the weights are detached, any per-position loss can replace LA in practice. We show that forward Kullback–Leibler (KL) divergence and Rényi divergence of order match or exceed LA, and that Rényi- provably bounds the true acceptance rate from above and below. Replacing CE with Rényi- improves acceptance length under both D-PACE's weights and ours. On six benchmarks with a Qwen3-4B target, D-PAL raises mean acceptance length over D-PACE from 4.50 to 4.57 at and from 4.20 to 4.28 at , and consistently achieves higher throughput at both temperatures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, large language models, acceptance-aware training, position-aware weighting, distributional losses

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KkLnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents D-PAL, a training objective for speculative decoding that extends D-PACE by replacing point-mass confidence estimates with full-distribution overlap to compute acceptance-aware position weights. The proposed method derives dynamic position weights from the expected accepted length and explores several local loss functions, including log acceptance, forward KL divergence, and Rényi-1/2 divergence. Extensive experiments across six benchmarks, multiple target models, and decoding configurations demonstrate improvements in accepted sequence length and inference throughput over D-PACE and DFlash.

Critical strengths
  • Principled theoretical analysis. The paper derives position weights from the expected accepted length and establishes theoretical bounds relating Rényi-1/2 divergence to acceptance probability, providing insights into its optimization advantages.
  • Comprehensive empirical evaluation. The experiments span multiple benchmarks, target models, decoding temperatures, draft depths, and block sizes, with ablation studies examining the contributions of position weighting and local loss functions.
Critical weaknesses
  • Insufficient evidence for the benefit of overlap-based weighting. Table 3 suggests that replacing cross-entropy with Rényi divergence accounts for most of the improvements, while overlap-based and confidence-based weighting achieve nearly identical results under matched smoothing settings.
  • Mismatch between theoretical analysis and practical verification. The overlap objective characterizes stochastic rejection sampling, whereas the evaluated DFlash implementation employs deterministic argmax proposals.
  • Modest empirical gains and limited robustness evidence. D-PAL achieves marginal improvements in mean accepted length over D-PACE (4.50 to 4.56 at T=0 and 4.20 to 4.28 at T=1), corresponding to relative gains below 2%. Furthermore, the use of a single training seed leaves the robustness of these improvements unclear.
Questions for authors
  1. Does overlap-based position weighting offer a meaningful advantage over confidence-based weighting when the local loss, smoothing parameter, and training configuration are held fixed?
  2. To what extent does distributional overlap predict actual token acceptance under the deterministic argmax proposals used during deployment, particularly across different decoding temperatures?
  3. Are D-PAL's improvements over D-PACE significant across multiple independent training seeds?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors: Response to Reviewer KkLn

We thank the reviewer for taking the time to review our paper. We will respond to their questions and concerns below. G1, G2, and G3 refer to the general response above. Weaknesses 1, 2, and 3 correspond to Questions 1, 2, and 3 below.

Question 1: Does overlap-based position weighting offer a meaningful advantage over confidence-based weighting when the local loss, smoothing parameter, and training configuration are held fixed?

We ran additional experiments across three training seeds. At a fixed floor (ρ=0.3\rho=0.3) and loss (Rényi), replacing confidence weights with overlap weights gives a small, statistically insignificant difference (+0.002 to +0.020 in τ\tau), whereas replacing CE with the Rényi loss raises τ\tau on 3/3 seeds in every condition (p=0.030, 0.085, 0.012):

Paired difference in τ\tauT=0T=0T=1T=1 argmaxT=1T=1 sampling
Overlap weights −- confidence weights (Rényi loss, ρ=0.3\rho=0.3)+0.002 (p=0.89)+0.020 (p=0.30)+0.010 (p=0.15)
Rényi loss −- CE loss (confidence weights, ρ=0.3\rho=0.3)+0.077 (p=0.030)+0.040 (p=0.085)+0.083 (p=0.012)

Testing pt(arg⁡max⁡qt)p_t(\arg\max q_t) as a third weight signal in a one-seed run also makes no measurable difference (G3). We use overlap because it is the exact acceptance rate for sampled proposals (G3); its derivation also motivates the full-distribution losses, where the measured gain comes from. With a fixed decay instead of dynamic weights, the same losses are 0.25 to 0.35 lower in τ\tau (36K training examples, one training seed, ρ=0.5\rho=0.5; reply to LtTq). We will make this split explicit in the abstract and contributions.

Question 2: To what extent does distributional overlap predict actual token acceptance under the deterministic argmax proposals used during deployment, particularly across different decoding temperatures?

Overlap is the exact per-token acceptance rate with sampled proposals and lower-bounds it with argmax proposals (Proposition 1); both proposal types are deployed. SGLang 0.5.19 proposes the argmax for original DFlash drafters; the current official DFlash implementation and DSpark in SGLang use sampled proposals at T>0T>0 (G3). We evaluated both:

  • With sampled proposals (T=1T=1): Overlap is the exact acceptance rate, and D-PAL[Rényi] improves τ\tau over D-PACE by 0.150 (3/3 seeds, p=0.001) (G1).
  • With argmax proposals (T=0,1T=0, 1): At T=1T=1, the acceptance rate is pt(arg⁡max⁡qt)p_t(\arg\max q_t), which Proposition 1 bounds via overlap. We tested directly whether the difference matters by training with pt(arg⁡max⁡qt)p_t(\arg\max q_t) itself as the weight signal (one training seed): it makes no measurable difference in τ\tau relative to overlap, including at T=1T=1 with argmax proposals (3.41 for both at ρ=0.5\rho=0.5, and 3.40 vs. 3.43 for overlap at ρ=0.3\rho=0.3; reply to 6Ra7, Question 2). With argmax proposals, the gain over D-PACE is also nearly the same at both temperatures (+0.074 at T=0T=0, +0.076 at T=1T=1, 3/3 seeds each) (G1). We will correct the wording so that "exact" refers only to sampled proposals.

Question 3: Are D-PAL's improvements over D-PACE significant across multiple independent training seeds?

Yes: D-PAL[Rényi] is above D-PACE on all three seeds (42, 43, 44), significant in all three sampler setups we evaluated (paired t-test from unrounded seed means):

Methodτ\tau, T=0T=0τ\tau, T=1T=1 argmaxτ\tau, T=1T=1 sampling
D-PACE4.475 ±\pm 0.0274.183 ±\pm 0.0224.058 ±\pm 0.017
D-PAL[Rényi]4.549 ±\pm 0.0114.259 ±\pm 0.0254.208 ±\pm 0.011
Paired difference+0.074 (3/3 seeds, p=0.019)+0.076 (3/3 seeds, p=0.003)+0.150 (3/3 seeds, p=0.001)

The values cited in the review (4.50 to 4.56, 4.20 to 4.28) are from the single training seed in the submission. Across three seeds, the relative gain in τ\tau is 1.6% at T=0T=0, 1.8% at T=1T=1 with argmax proposals, and 3.7% with sampled proposals, with inference unchanged. D-PAL[LA] and D-PAL[KL] are also above D-PACE on every seed in every condition (G1).

Reviewer_KkLnOpenReview ↗

Response for the Rebuttal

Thank you for the additional experiments and clarifications. That said, my core concern regarding the isolated contribution of overlap-based weighting remains unresolved. Under matched settings, the new results confirm that overlap-based weighting yields no statistically significant gain over confidence-based weighting; instead, the observed improvements stem largely from replacing CE with Rényi divergence. Likewise, comparing against fixed position decay does not demonstrate the value of the proposed weighting scheme, since D-PACE already uses dynamic weights.

Overall, although the expanded experiments bolster the empirical evidence, they still fall short of establishing the necessity or efficacy of the paper's central methodological contribution. I will therefore maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer KkLn's Follow-up

Thank you for the follow-up. We agree with part of your assessment and clarify our view of the contribution below.

Where we agree. Under matched settings, overlap weights give no statistically significant gain over confidence weights. The comparison with fixed decay shows the value of dynamic weights—which D-PACE introduced—rather than of our specific weight signal. The camera-ready version will state both points clearly.

What we view as the contribution. Our contribution is moving block-drafter training beyond the point-mass assumption to full distributions. D-PACE treats the target as a point mass in both its weights and its CE loss, whereas modern drafters with sampled proposals (such as the current official DFlash implementation and DSpark in SGLang, which verify via rejection sampling at T>0T>0) accept tokens based on the overlap of the two full distributions. Our derivation connects this exact acceptance rate to D-PACE's dynamic weights—showing D-PACE as the point-mass special case—and enables training smooth distributional losses (such as Rényi-1/2) under the same detached, floored weights. The gain from replacing CE with a full-distribution loss under dynamic weights is therefore the contribution we claim, not a side effect. In the camera-ready version, we will revise the abstract and contributions to frame the paper around this unified formulation rather than presenting the weight signal alone as the headline.

Why the gap over D-PACE is modest. Because D-PAL reduces to D-PACE for a point-mass target, the more concentrated the target distribution is, the smaller the room between a full-distribution objective and point-mass CE. Within that room, the gain is consistent and largest where the derivation directly applies: at the same floor (ρ=0.3\rho=0.3), D-PAL[Rényi] outperforms D-PACE on 3/3 seeds across all three conditions (+0.079 at T=0T=0, +0.060 with argmax proposals at T=1T=1, and +0.093 with sampled proposals at T=1T=1).

Experiments completed. As the discussion period nears its close, we note that the experiments your review asked for are complete and reported above: three independent training seeds with paired tests (Question 3); the matched control holding the loss, floor, and training configuration fixed while varying only the weight signal (Question 1); and evaluation under both argmax proposals and sampled proposals (Question 2), along with D-PACE at the matched floor ρ=0.3\rho=0.3. Thank you for highlighting these points—we would be grateful if you could take these results into account in your final assessment.

Reviewer_6Ra7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits D-PACE, which trains DFlash block drafters with position weights derived from a surrogate of expected accepted length using draft confidence q_t(y*) as the acceptance proxy and CE as the per-position loss. The authors argue that under rejection sampling the acceptance probability is the overlap 1 − TV(p_t, q_t), re-derive the weights with overlap in place of confidence, and obtain negative log acceptance (LA) as the matched local loss (D-PAL[LA]); the objective reduces to D-PACE for a point-mass target. Since weights are detached, they then swap LA for forward KL and Rényi-1/2 (Bhattacharyya), and prove the classical two-sided bound between Rényi-1/2 and overlap (Lemma 1) plus a simple bound on argmax acceptance (Proposition 1). Experiments train three-layer DFlash drafters in SpecForge for Qwen3-4B (100K DFlash data), Qwen3-8B and Llama-3.1-8B (36K ShareGPT), evaluated with SGLang on six benchmarks at T=0 and T=1. D-PAL[Rényi] improves mean acceptance length over D-PACE by about 0.06 to 0.08 on Qwen3-4B and about 0.12 on Llama, with ablations over block size, depth, floor ρ, batching, thinking mode, and TV-based losses (pure TV collapses, CE+TV underperforms).

Critical strengths
  1. The reformulation is clean and principled: replacing confidence with overlap in the D-PACE weight recursion is the right correction for sampled verification, the derivation (Eq. 3, App. B.3) is correct, and the objective provably reduces to D-PACE for a point-mass target. The exposition and the loss comparison (Table 11, Figure 2, the CE+TV gradient-conflict example) are clear and easy to reuse.
  2. The ablations are honest and separate the two changes (Tables 3, 4, 7). The practically useful finding is that replacing CE with a distributional loss (Rényi-1/2 or KL) helps under either weight signal, and the result direction is consistent across temperatures, block sizes, depths, three targets and thinking mode, with pure TV collapse and CE+TV underperformance documented.
Critical weaknesses
  1. The headline contribution (overlap-based weights) has no measurable effect in the paper's own ablations. At the same floor ρ=0.5, confidence weights and overlap weights with Rényi give identical mean τ (3.58/3.42 vs 3.58/3.41, Tables 3 and 7), and overlap weights with CE (D-PAL[CE]) exactly match D-PACE (3.50/3.36). All reported gains come from the CE → Rényi/KL swap, which is a modest, known direction (DistillSpec, LK Losses), so the title and abstract overstate what is new. Rényi-1/2 vs KL differ by 0.01 to 0.03 and Lemma 1 is a classical inequality, so the theoretical case for Rényi over KL does not translate into an empirical one.
  2. The improvements are small (mean τ 4.50 → 4.56, about 1.3%; SR 3.11 → 3.14, about 1%) and every configuration uses a single training seed (42). Evaluation-seed variance is reported, but at T=0 decoding is greedy so the std is trivially zero; the relevant variance (training seed) is never measured, yet conclusions such as "LA is not the best local loss" rest on differences of 0.01 to 0.03. Throughput noise in Table 2 (D-PACE beats D-PAL at K=8, T=0 despite lower τ) is already larger than the claimed SR gains, and the abstract's "consistently higher throughput" contradicts Section 4.2 (higher SR on 10 of 12 pairs). The abstract also reports the best D-PAL variant per temperature (4.57 is KL, 4.28 is Rényi) rather than the default method. Finally, the training objective targets sampled-proposal acceptance (overlap), but the evaluated system proposes argmax at both temperatures and accepts with probability p_t(argmax q_t) at T=1; that exact deployment acceptance is computable from p_t and q_t at training time and would be the natural detached weight signal, but it is never tried.
Questions for authors
  1. Please report mean τ (and ideally SR) over at least 3 training seeds for D-PACE, D-PAL[LA], D-PAL[KL], D-PAL[Rényi] in the Table 1 setting. Are the LA/KL/Rényi orderings and the D-PACE gap stable across seeds?
  2. Since the weights are detached, why not use the deployed acceptance signal directly: the top-1 match indicator 1[argmax q_t = argmax p_t] for T=0 and p_t(argmax q_t) for T=1, with the same floor ρ? How do these compare to overlap and confidence as weight signals?
  3. Given Table 3/7 show overlap and confidence weights are indistinguishable at ρ=0.5, what evidence supports overlap as the preferred weight signal beyond the theoretical argument? Is there any setting (e.g. high-entropy targets, thinking mode) where the weight signal actually matters?
  4. Table 1 uses ρ=0.3 for D-PAL and ρ=0.5 for D-PACE, with ρ selected on the same six evaluation benchmarks. What does D-PACE give at ρ=0.3 on the 100K setting, and were any hyperparameters chosen on a held-out set?
  5. What is the training-time and memory overhead of the full-vocabulary losses (KL/Rényi over V≈151K at every block position) relative to D-PACE's CE, in wall-clock per epoch on the 100K runs?
  6. Static[KL+TV] in Table 4 is below Static[CE] (3.20 vs 3.24), whereas LK Losses reports gains over CE. Your λ_t is per position (exp(−3α̂_t)) while LK aggregates acceptance over the batch. Did you verify your reimplementation reproduces LK's reported behavior?
6 comments
AuthorsOpenReview ↗

Official Comment by Authors: Response to Reviewer 6Ra7 (1/4)

We thank the reviewer for taking the time to review our paper. We will respond to their questions and concerns below. G1, G2, and G3 refer to the general response above.

The headline contribution (overlap-based weights) has no measurable effect [...]. All reported gains come from the CE →\to Rényi/KL swap [...].

The title names two parts: acceptance-derived weights and distributional losses. What is new is their combination: full-distribution losses trained under dynamic, detached, floored weights for one-pass block drafters. Relative to D-PACE, the measured gain comes from the loss (on 3/3 seeds in every condition) and, with sampled proposals, also from the lower floor (G2). Our new experiments do show that choosing overlap over confidence as the weight signal gives a small, but statistically insignificant difference (+0.002 to +0.020 at ρ=0.3\rho=0.3, G2), so we agree with the reviewer on this part. We still use overlap because it is the exact acceptance rate for sampled proposals (G3). We will reword the abstract and contributions to make this split explicit.

MethodPer-token lossWeights over draft positions
DistillSpecKL, JS, or TVuniform average (separate draft model)
DFlashCEfixed decay
D-PACECEdynamic, detached, with a floor
LK LossesLA, or KL + TVfixed decay
BebopTV, inside a product over draft stepsdynamic, not separable from the loss
D-PAL (ours)LA, KL, Rényi-1/2dynamic, detached, with a floor

For one-pass block drafters such as DFlash, dynamic weights existed only for CE, and full-distribution losses used a fixed decay; LK Losses names data-dependent weights as future work. Differentiating expected accepted length directly without detached weights and a floor does not train, as the D-PACE paper reports for its "Accept-rate" baseline. Bebop, which we cite, is the closest work: its weights are dynamic but tied to its TV loss, and it studies chains of MTP heads. D-PAL keeps D-PACE's detached, floored weights so the loss can be changed freely—which our setting requires, since training collapses without the floor (Table 7) or with pure TV (Table 1). In our measurements, D-PAL[Rényi] is 0.25 to 0.35 higher in τ\tau than the same loss with a fixed decay (36K training examples, one training seed, ρ=0.5\rho=0.5; reply to LtTq), and 0.06 to 0.09 higher than D-PACE at the same floor ρ=0.3\rho=0.3 (three seeds; Question 4 in comment 4).

Rényi against KL. With sampled proposals, where overlap is the acceptance rate, Rényi is above KL on all three seeds (4.208 vs. 4.135 in G1; paired difference +0.073, p=0.002). With argmax proposals, the two are within the spread across seeds. We will call Lemma 1 a known inequality.

AuthorsOpenReview ↗

Official Comment by Authors: Response to Reviewer 6Ra7 (Continued, 2/4)

The improvements are small [...] and every configuration uses a single training seed (42). [...] the abstract's "consistently higher throughput" contradicts Section 4.2 (higher SR on 10 of 12 pairs). [...] Finally, the training objective targets sampled-proposal acceptance (overlap), but the evaluated system proposes argmax at both temperatures [...]

We ran additional experiments and answer your questions with our new results below.

  • Size of the gain & seeds: Over three training seeds, D-PAL[Rényi] raises τ\tau over D-PACE by 1.6% at T=0T=0, 1.8% at T=1T=1 with argmax proposals, and 3.7% with sampled proposals; speedup rises by 1.2%, 1.9%, and 3.9% (G1). Every difference holds on all three seeds, inference is unchanged, and the ±\pm values in Table 1 will be over training seeds.
  • "LA is not the best local loss": Rényi is above LA on every seed in all three conditions (by about 0.04), so we will state the claim for Rényi only. KL is above LA on average with argmax proposals but below it on all three seeds with sampled proposals.
  • Throughput: For the batched runs of Table 2 (one training seed, as in the paper), we repeated the timing three times in each of 15 cases (five batch sizes ×\times three conditions). Compared with D-PACE, D-PAL[Rényi] has higher τ\tau in all 15 and higher throughput in 14; averaged over all 15, its throughput is 3.0% higher. The exception is K=8K=8 at T=0T=0. We will add the standard deviations to Table 2.
  • Abstract: Thank you for pointing this out. We will replace "consistently higher throughput" with the actual result from our new experiments (higher mean speedup on all three seeds in all three conditions), report the default method D-PAL[Rényi] at both temperatures, and use three-seed means.
  • Objective and verification rule: We now evaluate both argmax and sampled proposals, and we trained with pt(arg⁡max⁡qt)p_t(\arg\max q_t) as the weight signal (G3, and Question 2 in comment 3).
AuthorsOpenReview ↗

Official Comment by Authors: Response to Reviewer 6Ra7 (Continued, 3/4)

Question 1: Please report mean τ\tau (and ideally SR) over at least 3 training seeds for D-PACE, D-PAL[LA], D-PAL[KL], D-PAL[Rényi] in the Table 1 setting. Are the LA/KL/Rényi orderings and the D-PACE gap stable across seeds?

The gap to D-PACE is stable: all three D-PAL losses are above D-PACE on 3/3 seeds in all three conditions (G1). Among the three losses, Rényi is first at T=1T=1 in both conditions and level with KL at T=0T=0 (4.549 against 4.551). Seed by seed, Rényi is above LA on 3/3 seeds in all three conditions and above KL on 3/3 seeds with sampled proposals (4.208 vs. 4.135 in G1; paired difference +0.073, p=0.002). KL and LA swap places between argmax and sampled proposals.

Question 2: Since the weights are detached, why not use the deployed acceptance signal directly: the top-1 match indicator 1[arg⁡max⁡qt=arg⁡max⁡pt]\mathbf{1}[\arg\max q_t = \arg\max p_t] for T=0T=0 and pt(arg⁡max⁡qt)p_t(\arg\max q_t) for T=1T=1, with the same floor ρ\rho? How do these compare to overlap and confidence as weight signals?

pt(arg⁡max⁡qt)p_t(\arg\max q_t) makes no measurable difference in τ\tau relative to overlap. We trained with it as the detached weight signal, with the Rényi loss and the same floor, in the setting of Table 3 (one training seed per row):

Weight signal (Rényi loss)ρ\rhoT=0T=0T=1T=1, argmax proposalsT=1T=1, sampled proposals
overlap (D-PAL)0.53.5813.4113.290
confidence0.53.5843.4233.308
pt(arg⁡max⁡qt)p_t(\arg\max q_t)0.53.5613.4073.293
overlap (D-PAL)0.33.5863.4293.346
pt(arg⁡max⁡qt)p_t(\arg\max q_t)0.33.5553.4003.326
  • No measurable difference. At ρ=0.5\rho=0.5, pt(arg⁡max⁡qt)p_t(\arg\max q_t) is within 0.02 of overlap in all three conditions, and at ρ=0.3\rho=0.3 it is 0.02 to 0.03 below. This includes T=1T=1 with argmax proposals, the condition whose acceptance rate it is. The two new runs use a different data order, and these differences are within the spread across training seeds (G1).
  • T=0T=0. We did not train with the top-1 indicator 1[arg⁡max⁡qt=arg⁡max⁡pt]\mathbf{1}[\arg\max q_t = \arg\max p_t]. It is 0 or 1, so after the first mismatch in a block every later weight is held up only by the floor, and the pt(arg⁡max⁡qt)p_t(\arg\max q_t) runs are not above overlap at T=0T=0 either.
  • Overlap also covers drafters that sample. Since the weight signal makes no measurable difference in τ\tau, we choose overlap because it applies more broadly: current official DFlash and DSpark in SGLang use sampled proposals at T>0T>0, where the acceptance rate is the overlap (G3). We will add the table to the paper.

Question 3: Given Table 3/7 show overlap and confidence weights are indistinguishable at ρ=0.5\rho=0.5, what evidence supports overlap as the preferred weight signal beyond the theoretical argument? Is there any setting (e.g. high-entropy targets, thinking mode) where the weight signal actually matters?

Beyond the derivation, the support for overlap is breadth: it is the exact acceptance rate for sampled proposals, which confidence and pt(arg⁡max⁡qt)p_t(\arg\max q_t) are not (Question 2). We found no setting where the weight signal alone changes τ\tau measurably: with three seeds at ρ=0.3\rho=0.3, overlap and confidence differ by 0.02 or less in all three conditions, including T=1T=1, the higher-entropy condition (G2), and pt(arg⁡max⁡qt)p_t(\arg\max q_t) also makes no measurable difference (one training seed, Question 2). In thinking mode, D-PAL[Rényi] is above D-PACE with argmax and with sampled proposals (G3), but those runs use one training seed and change the loss and the weight signal together, so they do not separate the weight signal.

Reviewer_LtTqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces D-PAL, which trains DFlash speculative drafters with detached position weights built from the target–draft overlap 1−TV(p,q)1 - \mathrm{TV}(p, q), replacing D-PACE's draft confidence, and a distributional loss (Rényi-1/21/2 by default). On Qwen3-4B, mean acceptance length over D-PACE rises from 4.50 to 4.56 (T=0T = 0) and 4.20 to 4.28 (T=1T = 1).

Critical strengths

The derivation is clear, and Lemma 1 and Proposition 1 are correct. Negative results are reported openly (pure TV collapses; CE+TV underperforms). D-PAL beats D-PACE in τ\tau on all 12 benchmark: temperature pairs of Table 1, and gains hold on two 8B targets and in thinking mode.

Critical weaknesses
  1. The acceptance-derived weights add nothing measurable. At a matched floor (ρ=0.5\rho = 0.5), overlap and confidence weights give the same mean τ\tau with Rényi. The gain over D-PACE thus comes from replacing CE with Rényi-1/21/2.
  2. Speedups are within measurement noise. Headline speedups are about 1.0% (T=0T = 0) and 2.2% (T=1T = 1). Table 2 shows larger timing noise. Seed repeats and repeated timing runs are needed.
  3. The objective is not "exact" as claimed. The abstract derives D-PAL from the exact expected accepted length, but 1−TV1 - \mathrm{TV} is the acceptance rate only for sampled proposals; SGLang DFlash proposes the draft argmax at both temperatures, where Proposition 1 gives only bounds.
Questions for authors

Table 4 omits static KL and Rényi. How do they compare?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors: Response to Reviewer LtTq

We thank the reviewer for taking the time to review our paper. We will respond to their questions and concerns below. G1, G2, and G3 refer to the general response (above).

The acceptance-derived weights add nothing measurable. [...] The gain over D-PACE thus comes from replacing CE with Rényi-1/2.

We added additional training seeds to measure the effects of the weight overlap versus the loss function. At ρ=0.3\rho=0.3, replacing confidence weights with overlap weights gives a small, statistically insignificant difference (+0.002, +0.020, and +0.010 in τ\tau), whereas replacing CE with Rényi raises τ\tau on 3/3 seeds in every condition (+0.077, +0.040, and +0.083; G2), so we agree with the reviewer that the driving factor is the choice of loss function. At the same time, dynamic weights themselves raise τ\tau by 0.25 to 0.35 over a fixed decay for CE, LA, and Rényi-1/2 (36K training examples, one training seed, ρ=0.5\rho=0.5; Question below). What overlap adds is the derivation: it recovers D-PACE's weights as the point-mass case, is the exact acceptance rate for sampled proposals (G3), and points to full-distribution losses, where the measured gain comes from. We will make this split explicit in the abstract and contributions.

Speedups are within measurement noise. [...] Seed repeats and repeated timing runs are needed.

We added training seeds and repeated timing runs. Speedup over autoregressive decoding, mean ±\pm standard deviation over three training seeds (each seed is the mean of three evaluation runs; paired difference from unrounded seed means):

MethodSR, T=0T=0SR, T=1T=1 argmaxSR, T=1T=1 sampling
D-PACE3.092 ±\pm 0.0272.781 ±\pm 0.0222.715 ±\pm 0.019
D-PAL[Rényi]3.129 ±\pm 0.0152.835 ±\pm 0.0222.821 ±\pm 0.011
Paired difference+0.036 (3/3 seeds, p=0.038)+0.054 (3/3 seeds, p<0.001)+0.107 (3/3 seeds, p=0.002)

This corresponds to statistically highly significant speedups of 1.2%, 1.9%, and 3.9%, holding on 3/3 seeds in all three setups. For the batched runs of Table 2 (one training seed, as in the paper), we repeated the timing three times in each of 15 cases (five batch sizes ×\times three setups). D-PAL[Rényi] has higher τ\tau in all 15 and higher throughput in 14 of them; averaged over all 15, its throughput is 3.0% higher. The exception is K=8K=8 at T=0T=0. We will add the standard deviations to Table 2.

The objective is not "exact" as claimed. [...] SGLang DFlash proposes the draft argmax at [...] where Proposition 1 gives only bounds.

Overlap is the exact acceptance rate for sampled proposals (which current official DFlash and DSpark in SGLang use at T>0T>0), and we will state this condition in the abstract. We now evaluate both regimes (G3): D-PAL[Rényi] is above D-PACE by 0.150 with sampled proposals at T=1T=1 (3/3 seeds, p=0.001) and by 0.076 with argmax proposals at T=1T=1 (3/3 seeds, p=0.003). For argmax proposals, Proposition 1 gives a sufficient condition for acceptance at T=0T=0 and a lower bound on the acceptance probability at T=1T=1; training with pt(arg⁡max⁡qt)p_t(\arg\max q_t) as the weight signal instead of overlap makes no measurable difference in τ\tau (G3).

Question: Table 4 omits static KL and Rényi. How do they compare?

We trained both in the setting of Table 4 (36K training examples, one training seed; dynamic rows use ρ=0.5\rho=0.5):

LossWeightsτ\tau, T=0T=0τ\tau, T=1T=1 argmaxτ\tau, T=1T=1 sampling
CEstatic →\to D-PACE3.24→3.503.24 \to 3.503.11→3.363.11 \to 3.362.86→3.202.86 \to 3.20
LAstatic →\to D-PAL3.27→3.563.27 \to 3.563.15→3.403.15 \to 3.402.93→3.272.93 \to 3.27
KL (new)static3.273.152.87
Rényi-1/2 (new)static →\to D-PAL3.29→3.583.29 \to 3.583.16→3.413.16 \to 3.412.94→3.292.94 \to 3.29

With static weights, Rényi-1/2 is above CE by 0.05 to 0.08 and KL is above CE by 0.01 to 0.04; moving from static to dynamic weights raises τ\tau by a further 0.25 to 0.35 across CE, LA, and Rényi-1/2. We will add the new static rows to Table 4.

AuthorsOpenReview ↗

Follow-up to Reviewer LtTq

As the discussion period is closing soon, we would appreciate knowing whether our response addresses your concerns. All experiments requested in your review have been completed. In brief:

  • Weights vs. loss. We agree that the gain over D-PACE comes from the distributional loss rather than the weight signal, and we will state this explicitly in the abstract and contributions of the camera-ready version (G2).
  • Measurement noise. We added two training seeds and repeated the timing runs, as you asked. Over three training seeds, the speedup gain holds on 3/3 seeds in all three conditions (+0.036, +0.054, +0.107; p=0.038, p<0.001, p=0.002; G1).
  • "Exact". We will restrict "exact" to sampled proposals, and we now evaluate both sampled and argmax proposals (G3).
  • Static KL and Rényi. Both rows are reported in our reply above and will be added to Table 4 in the camera-ready version.

On the first point, moving from point-mass CE to full-distribution losses is the central reformulation needed to align drafter training with sampled-proposal verification (where acceptance equals distributional overlap, as used at T>0T>0 by the current official DFlash implementation and DSpark in SGLang). Consistently, the gain over D-PACE is largest under sampled proposals (+0.150 vs. +0.076 at default floors, and +0.093 vs. +0.060 when both use ρ=0.3\rho=0.3).

Thank you for pointing out these gaps. If any point remains open, we would be glad to clarify it before the discussion period closes, and we would be grateful if you could consider the new results in your final rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official Comment by Authors: General Response to All Reviewers (1/2)

We thank all three reviewers for their time and thoughtful reviews, and appreciate how they found the derivation "clear" (LtTq), the exposition "clear and easy to reuse" (6Ra7), the approach "principled" (6Ra7 and KkLn), and the empirical evaluation "comprehensive" (KkLn). All reviews share three questions, which we answer below:

  • Do the gains hold across training seeds? Yes. We added two more training seeds, and all three D-PAL losses are above D-PACE on each of the three seeds, in every evaluation condition (G1 below).
  • Do the gains hold when the drafter samples its tokens, as the derivation assumes, instead of using the argmax? Yes. We evaluated a drafter that samples from its distribution across three training seeds, and all three losses stay above D-PACE on every seed, with the gain of our default D-PAL[Rényi] nearly doubling (+0.150 vs. +0.076 in τ\tau; G1, and G3 in comment 2).
  • Where does the gain over D-PACE come from? We evaluate changing ρ\rho, the loss, and the weight signal separately across three training seeds (G2 in comment 2). The loss raises τ\tau on 3/3 seeds in every condition (p=0.030, 0.085, 0.012), and with sampled proposals the lower floor (ρ=0.3\rho=0.3 instead of D-PACE's 0.50.5) also contributes (+0.057 on 3/3 seeds, p=0.011). Replacing confidence with overlap as the weight signal gives a small, statistically insignificant difference (+0.002 to +0.020 at ρ=0.3\rho=0.3, G2), and pt(arg⁡max⁡qt)p_t(\arg\max q_t) makes no measurable difference in a one-seed run (G3). We use overlap because it is the exact acceptance rate for sampled proposals (G3).

The new experiments use the configuration of Table 1 unless stated otherwise. "Argmax proposals" is the sampler configuration from the paper, where the drafter proposes its most likely token. "Sampled proposals" is new: the drafter samples each token from its own distribution and the target applies standard rejection sampling. τ\tau is the mean acceptance length, and SR is the speedup over autoregressive decoding.

G1. Three training seeds

Mean ±\pm standard deviation over training seeds 42, 43, 44; each entry is the mean over six benchmarks and three evaluation runs.

Methodτ\tau, T=0T=0τ\tau, T=1T=1 argmaxτ\tau, T=1T=1 samplingSR, T=0T=0SR, T=1T=1 argmaxSR, T=1T=1 sampling
D-PACE4.475 ±\pm 0.0274.183 ±\pm 0.0224.058 ±\pm 0.0173.092 ±\pm 0.0272.781 ±\pm 0.0222.715 ±\pm 0.019
D-PAL[LA]4.512 ±\pm 0.0344.221 ±\pm 0.0294.171 ±\pm 0.0313.109 ±\pm 0.0252.804 ±\pm 0.0272.788 ±\pm 0.032
D-PAL[KL]4.551 ±\pm 0.0134.237 ±\pm 0.0084.135 ±\pm 0.0053.138 ±\pm 0.0232.812 ±\pm 0.0112.755 ±\pm 0.012
D-PAL[Rényi]4.549 ±\pm 0.0114.259 ±\pm 0.0254.208 ±\pm 0.0113.129 ±\pm 0.0152.835 ±\pm 0.0222.821 ±\pm 0.011

D-PAL[Rényi] against D-PACE, paired by training seed (from unrounded seed means; two-sided paired t-test):

MetricT=0T=0T=1T=1 argmaxT=1T=1 sampling
τ\tau+0.074 (3/3 seeds, p=0.019)+0.076 (3/3 seeds, p=0.003)+0.150 (3/3 seeds, p=0.001)
SR+0.036 (3/3 seeds, p=0.038)+0.054 (3/3 seeds, p<0.001)+0.107 (3/3 seeds, p=0.002)

D-PAL[Rényi], D-PAL[KL], and D-PAL[LA] all achieve higher τ\tau and higher speedup than D-PACE on 3/3 seeds in every evaluation condition, requiring no change at inference.

AuthorsOpenReview ↗

Official Comment by Authors: General Response to All Reviewers (Continued, 2/2)

G2. Where the gain comes from

We added two control runs with three training seeds each, changing one factor at a time. Each row is the paired difference in τ\tau over three seeds, with the p-value of a paired t-test:

ChangeT=0T=0T=1T=1 argmaxT=1T=1 sampling
Floor ρ\rho: 0.5→0.30.5 \to 0.3 (CE loss, confidence weights)−0.005-0.005 (p=0.39)+0.016 (p=0.35)+0.057 (p=0.011)
Loss: CE →\to Rényi (confidence weights, ρ=0.3\rho=0.3)+0.077 (p=0.030)+0.040 (p=0.085)+0.083 (p=0.012)
Weight signal: confidence →\to overlap (Rényi loss, ρ=0.3\rho=0.3)+0.002 (p=0.89)+0.020 (p=0.30)+0.010 (p=0.15)

The loss raises τ\tau on 3/3 seeds in all three setups (p=0.030, 0.085, 0.012), while changing the weight signal from confidence to overlap gives a small, statistically insignificant difference (+0.002 to +0.020, within the spread across seeds). This agrees with Section 4.3. At the same time, dynamic weights—which D-PACE introduced for CE—matter for every loss: under DFlash's fixed decay, CE, LA, and Rényi-1/2 are each 0.25 to 0.35 lower in τ\tau (36K training examples, one training seed, ρ=0.5\rho=0.5; table in our reply to LtTq).

We will make these findings explicit in the abstract and in contribution (i): the derivation gives the dynamic weights, recovers D-PACE as a special case, and motivates the full-distribution loss; relative to D-PACE, the measured gain comes from the loss and, with sampled proposals, also from the lower floor.

G3. Argmax proposals and sampled proposals

With standard rejection sampling, a sampled draft token is accepted with probability 1−TV(p,q)1-\mathrm{TV}(p,q), so the overlap is the acceptance rate. Recent drafters verify this way at T>0T>0, for example the current official DFlash implementation and DSpark in SGLang. SGLang 0.5.19, which the paper uses, applies rejection sampling only to the newer DFlash2 drafters; for the original DFlash drafters it proposes the most likely token at each position and accepts it with probability pt(arg⁡max⁡qt)p_t(\arg\max q_t) (argmax proposals). We have now enabled SGLang's rejection sampling for the original DFlash drafters and re-evaluated (sampled proposals). Our conclusions hold in both settings: D-PAL[Rényi] is above D-PACE on every seed with argmax proposals, and the gain is larger with sampled proposals (+0.150 against +0.076; with D-PACE at the same floor ρ=0.3\rho=0.3, +0.093 against +0.060).

D-PAL[Rényi] is also above D-PACE with sampled proposals in the other settings of the paper (mean τ\tau at T=1T=1, one training seed each):

SettingD-PACED-PAL[Rényi]
Thinking mode, argmax proposals (Table 9)3.143.17
Thinking mode, sampled proposals3.153.22
Qwen3-8B, sampled proposals3.263.33
Llama-3.1-8B, sampled proposals2.632.76

Training with pt(arg⁡max⁡qt)p_t(\arg\max q_t) as the detached weight signal, as suggested by Reviewer 6Ra7, makes no measurable difference in τ\tau relative to overlap (one training seed; table in our reply to 6Ra7, Question 2). We will state that overlap is the exact acceptance rate only for sampled proposals; for argmax proposals, Proposition 1 gives a sufficient condition for acceptance at T=0T=0 and a lower bound on the acceptance probability at T=1T=1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.