acceptodds
Under review as a conference paper at ICLR 2027

Training-free Dropout Sampling for Semantic Token Acceptance in Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference by proposing tokens with a lightweight draft model and selectively accepting them using a target model. This work introduces DropMatch, a novel approach that matches draft tokens to the predictive distribution of the target model via Monte Carlo dropout applied exclusively to the LM head, enabling sampling-based acceptance decisions. By generating multiple decoding paths, our method forms an empirical token distribution against which draft tokens are evaluated for consistency. This acceptance mechanism enables the model to adaptively control the size of decoding paths under an appropriate dropout probability, preventing substantial distortion of the target model predictive distribution. The proposed method operates in a training-free and data-free manner, requires no architectural modification to pretrained models, and can be orthogonally integrated with a wide range of existing speculative decoding and inference acceleration techniques. Experiments across multiple benchmarks demonstrate that our approach increases acceptance length while maintaining competitive task performance, yielding inference speedups ranging from 1.09× to 1.33× over the standard baseline, and up to an additional 1.09× speedup when applied on top of EAGLE3.

Primary area:
generative models
Keywords:
deep learning, speculative decoding, dropout

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rH8aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DropMatch, a training-free acceptance mechanism for lossy speculative decoding. The method applies independent dropout masks to the target model’s final hidden representation and obtains multiple predictions through the shared LM head. Draft tokens are accepted using either token matching across these predictions or a JS-divergence criterion supplemented by majority agreement. Experiments evaluate DropMatch with standard speculative decoding, Auto-Judge, and EAGLE-3, demonstrating increased acceptance lengths and additional speedups in several configurations.

Critical strengths
  1. This work explores whether predictions obtained through inference-time perturbations can support token acceptance without training an additional verifier. Reducing the dependence on learned judge components is a useful direction for making lossy speculative decoding easier to apply across models and tasks.
  2. Applying dropout only before the LM head allows the stochastic predictions to share the computation of the transformer backbone. The method requires no additional training data or model parameters, and the reported overhead analysis supports the practicality of this design in the evaluated setting.
  3. The authors conduct experiments with Llama and Qwen models on reasoning, knowledge, instruction-following, and code-generation benchmarks. The integrations with Auto-Judge and EAGLE-3, together with the batched throughput experiments, help demonstrate the potential applicability of the approach within existing decoding pipelines.
  4. The manuscript includes ablations on the dropout probability and number of paths, as well as analyses of head agreement and acceptance behavior. I also appreciate the comparisons with deterministic top-k acceptance and Gaussian perturbations in the appendix, which provide useful evidence for understanding the role of the proposed components.
Critical weaknesses
  1. Further justification of the semantic acceptance mechanism would strengthen the contribution. My main concern is the connection between variability under dropout and the semantic validity of an accepted draft token. Figure 3 provides encouraging evidence of similarity between generated paths, while Tables 9–10 characterize head agreement and substitution frequency. However, these analyses do not directly establish whether the additional substitutions admitted by DropMatch preserve meaning or downstream correctness. In particular, Eq. 6 compares entire distributions and uses the maximum divergence among dropout heads as the acceptance threshold, but its relationship to the error risk of an individual substitution remains unclear. A direct evaluation of the accepted substitutions would help substantiate the central motivation.
  2. The specific benefit of MC dropout could be established more clearly. Appendix C provides helpful comparisons, although Table 12 shows that Gaussian perturbations with a matched scale achieve essentially the same accuracy and acceptance length as DropMatch: 94.0% and 5.51 versus 93.9% and 5.47. The authors explain that dropout supplies a position-dependent perturbation scale automatically, which is a reasonable motivation. However, the current comparison with substantially smaller and larger Gaussian scales does not isolate the contribution of this position dependence. Comparisons against adaptive perturbations or entropy-based acceptance rules, preferably at matched acceptance rates or quality levels, would help clarify the methodological advantage of the proposed design.
  3. The speed–quality trade-off deserves a more qualified discussion. Although DropMatch provides useful acceleration in several settings, some results suggest that increased acceptance length does not always translate into a favorable trade-off. For Llama-3.1 on HumanEval with L=5L=5, Table 4 reports a decrease in Pass@1 from 81.70 to 79.89 with a speedup of 1.00×. At L=10L=10, Pass@1 decreases from 81.70 to 78.70, while the reported speed changes from 1.13× to 1.12× under the same normalization. I appreciate the repeated runs and paired tests in Appendix C, but the absence of a statistically significant difference in one configuration does not establish quality equivalence. Reporting speed improvements under an explicit quality tolerance would make the practical benefits easier to assess.
  4. The conclusions about robustness under distribution shift would benefit from further qualification. Avoiding additional training is an attractive property, although the multilingual results indicate that the acceptance mechanism can still be sensitive to distribution changes. In Table 13, with L=5L=5, adding DropMatch reduces accuracy from 83.2% to 78.4% on MGSM-ZH, from 84.8% to 79.2% on MGSM-DE, and from 82.0% to 78.0% on MGSM-FR, for speedups of 1.07×, 1.10×, and 1.07×, respectively. The explanation in Appendix D, involving flatter target distributions and greater sensitivity to dropout, is helpful. Bringing these limitations into the main discussion and clarifying when the method is expected to remain reliable would provide a more balanced account of its generalization behavior.
Questions for authors
  1. Could the authors directly evaluate the semantic validity and downstream correctness of the additional substitutions accepted by DropMatch, separately for the JS-divergence and token-matching criteria? Does the acceptance radius reliably distinguish harmless substitutions from harmful ones?
  2. Could the authors isolate the benefit of position-dependent dropout through comparisons with adaptive perturbation scales or entropy-based acceptance rules at matched acceptance rates or quality levels?
  3. What practical guidance would the authors recommend for selecting KK and pdropp_{\mathrm{drop}} on an unseen domain with a specified quality tolerance? Additional uncertainty estimates for the principal speed–quality comparisons would also be helpful.
4 comments

Q1. Can you directly evaluate the semantic validity and downstream correctness of the substitutions that DropMatch additionally accepts, separately for the JS criterion and the token-matching criterion? Does the acceptance radius discriminate harmless substitutions from harmful ones?

A1. The semantic validity of the substitutions that DropMatch additionally accepts is evaluated with an LLM-as-a-judge in the MT-Bench and Alpaca experiments. Similar scores do not mean that the outputs are exactly semantically equivalent, but they show that a similar level of answer quality is maintained for the given queries. In addition, by examining the difference in accuracy between SD and DropMatch on GSM8K, we confirmed that there is no significant difference in terms of correctness.

Paired comparison (α=1, 5 seeds).

Task (#questions)Lossless SDDropMatch mean±sdDiff.95% intervalCorrect only with lossless / only with DropMatchMcNemar p
GSM8K, L=5 (1,319)94.7794.04±0.24−0.73[−1.50, +0.05]30.2 / 20.60.08 – 0.40
HumanEval, L=5 (164)78.779.5±1.4+0.85[−2.80, +4.63]6.8 / 8.20.23 – 1.00
HumanEval, L=10 (164)78.780.1±2.0+1.46[−2.07, +5.12]6.0 / 8.40.02 – 1.00
  • The drop on HumanEval at L=10 (SD 81.7 → DropMatch 78.7) is at a level that lies within the error range. In our re-measurement the SD score is 78.70 (only the hardware changed; all settings are identical), and we re-measured DropMatch with five seeds.
  • HumanEval has 164 problems, and the score difference is within the standard deviation, so it is not a significant difference. With a 95% confidence interval, the performance varies within the interval. The performance difference is not at a significant level.

JS criterion vs. token-matching criterion. Each cell is: share of the accepted non-argmax tokens / median p_target / fraction within the target's top-3.

Accepted byLlama GSM8KLlama IFEvalQwen GSM8K
JS criterion only70.1% / 0.271 / 90.3%63.1% / 0.221 / 88.5%75.7% / 0.182 / 77.7%
Token-matching criterion only4.1% / 0.310 / 97.9%13.6% / 0.293 / 96.3%3.6% / 0.303 / 97.3%
Both criteria25.8% / 0.353 / 98.7%23.2% / 0.321 / 98.2%20.8% / 0.324 / 97.8%
  • The substitutions accepted by the token-matching criterion are almost always within the target's top-3, and the low-probability substitutions come from the JS criterion.

Discriminative power of the acceptance radius. Measured at T=0 on positions where the draft token is not equal to the target argmax.

Llama GSM8KLlama IFEval
Positions with draft ≠ target argmax8,56123,678
Accepted among them66.5%30.6%
Median p_target: accepted / rejected0.299 / 0.1610.256 / 0.090
Within the target's top-3: accepted / rejected92.8% / 78.7%91.8% / 71.1%
p_target < 0.05: accepted / rejected2.9% / 24.5%5.4% / 36.0%
Acceptance rate, p_target < 0.0110.4%2.5%
Acceptance rate, 0.01 – 0.0526.0%8.5%
Acceptance rate, 0.05 – 0.255.7%24.8%
Acceptance rate, 0.2 – 0.3574.9%49.2%
Acceptance rate, 0.35 – 0.582.9%65.6%
  • The higher the target probability, the higher the acceptance rate of DropMatch, with differences across tasks. On IFEval, which has strong format constraints, the acceptance rate is somewhat lower.

Does a substitution change the final answer? On GSM8K we recorded the judged positions where the draft token differs from the target argmax (8,705 positions). For each position we let the target generate to the end in two ways, once with the draft token inserted and once with the target argmax inserted, and scored the final answers.

Substitution setCountFinal answer changedHarmfulBeneficialNet change in accuracy
Substitutions accepted by DropMatch5,7464.49%1.65%1.39%−0.26%p
Substitutions rejected by DropMatch2,9598.45%3.95%2.37%−1.58%p
└ accepted: JS criterion only4,0064.49%1.55%1.42%−0.13%p
└ accepted: token-matching criterion only2309.13%3.48%3.04%−0.44%p
└ accepted: both criteria1,5103.77%1.66%1.06%−0.60%p
  • 95.5% of the accepted substitutions do not change the final answer, and the rejected substitutions have a higher harmful rate than the accepted ones. This shows that the acceptance radius discriminates between harmless and harmful substitutions.

Q2. Can you isolate the benefit of position-dependent dropout by comparing, at the same acceptance rate or the same quality, with an adaptive perturbation scale or an entropy-based acceptance rule?

A2. To isolate the benefit of position-dependent dropout, we compared against a control in which the magnitude of the perturbation is kept equal to that of dropout and only the structure is changed. The control is a Gaussian perturbation with a position- and token-wise adaptive scale: to each logit at each position we add noise with the same variance that dropout produces, but with the covariance between tokens removed. The acceptance rule and K are the same as in DropMatch. The entropy-based acceptance rule (typical acceptance) reduces to lossless SD at T=0, so we compared it at T=1.

Llama-3.1 8B→70B, GSM8K, T=0 (mean ± standard deviation over 3 runs each; τ / accuracy)

αDropMatchAdaptive-scale Gaussian
0.55.064±0.086 / 94.29±0.225.068±0.005 / 93.71±0.27
0.755.309±0.076 / 93.88±0.125.396±0.005 / 93.78±0.59
15.474±0.051 / 93.91±0.045.609±0.009 / 92.62±0.63
1.55.674±0.032 / 93.56±0.135.790±0.002 / 92.04±0.27
25.767±0.022 / 93.13±0.195.858±0.006 / 91.58±0.40
35.857±0.016 / 92.09±0.355.921±0.001 / 89.97±0.29
55.916±0.007 / 90.83±0.465.958±0.002 / 88.75±0.48
  • As τ grows, DropMatch maintains accuracy better than the adaptive-scale Gaussian. This shows that DropMatch has a separate benefit.

T=1, top_p=1.0, GSM8K

Ruleτ / accuracyDistribution shiftInter-response similarity
Lossless SD4.776 / 91.99±0.22+0.0150.481
DropMatch α=15.202±0.147 / 90.08±0.60+0.0470.489
DropMatch α=25.652±0.065 / 87.62±0.69+0.0660.463
Adaptive-scale Gaussian α=15.497 / 87.06±0.58+0.0790.449
Adaptive-scale Gaussian α=25.837 / 83.95±0.57+0.0890.437
typical ε=0.34.947 / 93.81±0.24−0.0910.608
typical ε=0.25.092 / 93.56±0.13−0.0800.583
typical ε=0.095.316 / 93.23±0.31−0.0540.549
typical ε=0.035.533 / 92.22±0.19−0.0240.517

Distribution shift is the mean negative log-probability minus the mean entropy when the output is fed back into the target (0 for exact samples of the target, negative toward greedy; the greedy output gives −0.134). Inter-response similarity is the mean word-level Ratcliff/Obershelp similarity among three responses to the same question.

Reading the T=1 table above:

  • Looking only at accuracy and τ, the typical acceptance of Cai et al. (Medusa, 2024) is higher than DropMatch.
  • However, typical acceptance fills rejected positions and the first token of every step greedily and cuts off tail tokens, so its output moves from the T=1 distribution toward greedy. As a result the output distribution becomes sharper than the T=1 sampling distribution, and its accuracy being higher than that of lossless SD includes this shift. At ε=0.3 it moves two thirds of the distance to greedy (−0.091), and the responses to the same question become more similar to each other (0.608).
  • DropMatch α=1 raises τ by 9% while keeping sample diversity at the level of lossless SD (0.489 vs. 0.481). At α=2, τ rises by 18%, while the output becomes slightly more spread out than lossless SD (0.463) and the accuracy is 4.4 points lower. If the purpose of sampling is diverse output, the two rules preserve different things.

Q3. What is the practical guidance for choosing K and p_drop in an unseen domain with a given quality tolerance? Uncertainty estimates for the main speed–quality comparisons would also help.

A3. For additional information on uncertainty, we report mean ± standard deviation in the answers above.

K and p_drop were chosen with a single ablation on GSM8K in our experiments and were then used on the other tasks and on Qwen3 without re-tuning. The experimental results show that the hyperparameters we chose work well on two model families.

During the rebuttal we added α. We confirmed that values between 1 and 2 cause a small drop in accuracy but can well be used for additional speedup. When a quality tolerance is given, keeping K and p_drop fixed and controlling α is the easier way to adjust the trade-off.

Thank for the authors' additional experiments and detailed response. The decline on GSM8K and remaining uncertainty on HumanEval, both relatively basic benchmarks, raise concerns that quality losses could be greater on harder tasks. Unpredictable quality degradation can undermine user trust in lossy SD methods. Moreover, the reported speed gains are not consistently substantial enough to justify these reliability risks. I therefore maintain my original score.

Thank you for considering our additional experiments and for your candid assessment. Your comments point to what a stronger version needs: an evaluation on harder tasks, a clearer characterization of when quality is preserved, and a larger gain in return. We will take this into account in our future work.

Thank you again for your time and for the constructive feedback throughout the review.

Reviewer_iZXMOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Context and problem. Standard speculative decoding is lossless: a draft token is accepted only if it is consistent with the target distribution at the token level. As a result, semantically equivalent tokens are still rejected, which limits acceptance length. Lossy approaches such as Judge Decoding and Auto-Judge relax this by training judge heads, but they need extra data and can degrade under distribution shift. The paper aims for a training-free relaxation of the acceptance rule.

Method. The paper proposes DropMatch, which applies Monte Carlo dropout only to the target model's LM head. At each verification position, KK independent dropout masks (K=5K=5, pdrop=0.3p_{\text{drop}}=0.3) are applied to the final hidden state, producing KK output distributions at negligible extra cost and without touching the KV cache. A draft token is accepted under either of two conditions. The first is that the Jensen–Shannon divergence between the draft distribution and the centroid of the KK distributions is no larger than the largest divergence among the KK distributions themselves (the "acceptance radius"). The second is that the draft token equals the majority top-1 token of the KK heads. Otherwise the token is rejected and replaced by the majority token. When combined with EAGLE-3, a simpler rule is used: a draft token is accepted if it matches any head's top-1 token. The method is lossy, needs no training, and leaves the model architecture unchanged.

Contributions. (i) A training-free, sampling-based acceptance rule for speculative decoding using MC dropout at the LM head. (ii) An analysis of when the rule relaxes acceptance: it relaxes at positions where the target is uncertain and stays strict where it is confident. (iii) Experiments showing that the rule can be combined with standard speculative decoding, Auto-Judge, and EAGLE-3.

Validation. The paper evaluates Llama-3.1-8B → 70B and Qwen3-4B → 32B on 4× A100 GPUs with vLLM, on GSM8K, MMLU, IFEval, and HumanEval, at batch size 1 and draft lengths 5 and 10.

  • At L=5L=5, acceptance length increases by about 10% on most tasks, with similar speedups over standard speculative decoding. The best gain is 1.33× (Qwen3, L=10L=10), and the gain on Llama GSM8K is 1.19×.
  • Accuracy is mostly close to standard speculative decoding.
  • The gain stays around 1.10× at batch sizes up to 128.
  • On top of EAGLE-3 (Llama-3.3-70B), it gives up to about 1.09× additional speedup.
  • Combined with Auto-Judge, it gives longer acceptance lengths at comparable accuracy.
  • Additional analyses cover LM-head overhead (1.64% in total), head-agreement statistics, comparisons with fixed top-kk acceptance and Gaussian logit noise, out-of-distribution results on KoMT-Bench and MGSM, and ablations on KK and pdropp_{\text{drop}}.
Critical strengths

1. Simple, training-free, and easy to integrate. DropMatch changes only the verification step. It needs no training, auxiliary data, or architectural change, and adds little overhead (1.64% of forward latency in the paper's measurement). It also composes with existing pipelines: it gives additional gains on top of standard speculative decoding, EAGLE-3, and Auto-Judge, and its gain persists in vLLM up to batch size 128 (about 1.10×, Table 3). Why it matters: a plug-in acceptance rule with this little engineering cost is easy to try in existing systems.

2. Position-adaptive relaxation instead of a fixed one. The acceptance region is set at each position by the spread of the target's own perturbed predictions. It is nearly as strict as greedy verification where the target is confident and wider only where the target is uncertain; its radius correlates strongly with target entropy (ρ = 0.924). This gives a better speed–quality balance than a fixed relaxation. In Table 11, top-2 acceptance raises acceptance length but differs significantly from lossless greedy decoding (p = 0.027), whereas DropMatch is not significantly different (p = 0.461). Why it matters: adapting the degree of relaxation to the target's confidence is a sensible principle for lossy verification, and the paper supports it with careful position-level analysis.

Critical weaknesses

1. The method is lossy, and its gains are not large enough to justify giving up exactness. Exactness is the main reason speculative decoding is attractive: many techniques (quantization, pruning, early exit) can accelerate inference if some quality loss is allowed, and speculative decoding has no clear advantage among them once losslessness is dropped. This is also why production deployments still largely rely on lossless verification, as in EAGLE-style systems. For a lossy rule to be compelling, the speedup should therefore be substantial, but the reported gains are modest: 1.09–1.33× over standard speculative decoding, and only about 1.07–1.09× on top of EAGLE-3 (Table 5, L=9L=9). Meanwhile, the quality cost is not always small:

  • accuracy drops by 2–3 points in several settings (e.g., HumanEval with Llama at L=10L=10: 81.70 → 78.70; IFEval with Qwen3 at L=5L=5: 80.72 → 78.24);
  • on MGSM, accuracy drops by 4–5.6 points in Chinese, German, and French (Table 13), which contrasts with the main text's claim of robustness under distribution shift;
  • only 91.8% of final answers match lossless greedy decoding on GSM8K (Table 11).

There is also no bound on how far the output distribution may deviate from the target. Why it matters: the practical value of a lossy acceptance rule rests on a clearly better speed–quality trade-off, and the current evidence does not show that.

2. The role of MC dropout is unclear, and the closest baselines are missing. Table 12 shows that Gaussian noise added to the logits reproduces DropMatch's acceptance length and accuracy when its scale matches the spread of the dropout paths (5.51 vs. 5.47 and 94.0 vs. 93.9). The acceptance radius also correlates strongly with target entropy (ρ = 0.924). The rule therefore behaves like an entropy-adaptive relaxed acceptance, and MC dropout serves mainly as a noise source rather than as a principled uncertainty estimate: its Bayesian interpretation assumes models trained with dropout at that point. Existing training-free rules of exactly this kind are not compared. These include Medusa's typical acceptance, which uses an entropy-adaptive threshold on the target probability; the top-kk/tolerance relaxed verification of SpecDec (Xia et al., 2023), which the paper cites; and a simple threshold on the target probability of the draft token. Only fixed top-kk acceptance is compared (Table 11). Why it matters: without these baselines, it is unclear whether DropMatch offers a better trade-off than simpler existing rules, which is central to the paper's contribution.

Questions for authors
  1. Can a moderate relaxation give a striking speedup? Within practical draft lengths (e.g., L=5L=5–1010, and EAGLE-3 with its default settings), could the authors vary how loose the acceptance criterion is, for example by scaling the acceptance radius, increasing pdropp_{\text{drop}}, or using the naive token-matching rule, and report the full speed–quality curve over standard speculative decoding and on top of EAGLE-3? In Table 8, raising pdropp_{\text{drop}} to 0.9 gives only 1.17× while losing 7.5 accuracy points, so it remains unclear whether a better operating point exists. A substantially larger speedup at small quality cost (for example, 1.5× or more on top of EAGLE-3 with accuracy within about one point) would be the most compelling result for practical use, and would significantly change my assessment.

  2. Comparison with existing relaxed acceptance rules. How does DropMatch compare, at matched acceptance length or matched accuracy, with Medusa's typical acceptance, the top-kk/tolerance relaxed verification of SpecDec, and a simple threshold on the target probability of the draft token?

  3. Overhead. The paper reports that the LM head accounts for 0.05% of the forward cost of Llama-3.1-70B. However, the LM head holds about 1.05B of roughly 70B parameters (about 1.5%), and decoding at batch size 1 is memory-bound. Could the authors explain how the 0.05% was measured, and report the overhead of the KK dropout paths and the JS computation over the full vocabulary at larger batch sizes and draft lengths?

  4. Sampling setting. Which decoding temperature and sampling scheme are used in the main experiments? How does DropMatch behave under non-greedy sampling (e.g., temperature 0.7 or 1.0), where the target distribution is itself sampled?

If the authors can address the core concerns above, especially Question 1, I would consider raising my score. I find the direction of training-free, confidence-adaptive acceptance interesting, appreciate the careful position-level analysis, and look forward to the authors' response.

5 comments

Q1. Can you report the full speed–quality curve on standard SD and on EAGLE-3 by varying the degree of relaxation (radius scaling, larger K, naive token-matching)? A larger speedup at a small quality cost would be the most convincing result.

A1. Thank you for the suggestion. We judged radius scaling to be the most natural knob for DropMatch and introduced a scale α on the radius (α=1 is the setting in the paper). Increasing K is covered in Appendix A: beyond a certain K the performance converges and only the computation cost grows.

Radius scaling on standard SD (Llama-3.1 8B→70B, GSM8K, L=5).

SettingτAccuracyτ vs. losslessHeadroom recovered
Lossless SD, L=54.96594.77——
α=0.54.970±0.20494.03±0.501.00×0%
α=0.755.220±0.19693.97±0.621.05×25%
α=1 (paper setting)5.425±0.13594.04±0.241.09×44%
α=25.737±0.06593.31±0.321.16×75%
α=3 (3 seeds)5.857±0.01692.09±0.351.18×86%
  • "Headroom recovered" is the fraction gained between the lossless τ and the ceiling L+1. One round can emit at most L+1 tokens, so for a fixed L no acceptance rule can raise τ by more than a factor of (L+1)/τ_baseline.
  • In the range α=1–2, 44–75% of the remaining headroom is recovered at a cost of 0.7–1.5 accuracy points. We adopt α as a formal hyperparameter and add its ablation to the revision (Appendix B).

On top of EAGLE-3. The same knob gives additional acceleration on EAGLE-3. In the paper we reported the gain obtained with the naive matching criterion while EAGLE-3's tree decoding was enabled; here we report the gain from adjusting α in EAGLE-3 chain mode (Llama-3.3-70B, full GSM8K).

SettingAccuracyτ (ratio)
EAGLE-3 chain81.654.532
+ naive token-matching81.78±0.785.131 (1.132×)
+ region boundary α=381.83±0.425.122 (1.130×)
+ region boundary α=580.26±0.315.763 (1.272×)
  • With a suitable value of α, additional acceleration is possible, but a boundary that is too wide can also reduce accuracy.

Q2. At the same τ or accuracy, how does DropMatch compare with Medusa's typical acceptance, SpecDec's relaxed top-k verification, and a simple threshold on the target probability of the draft token?

A2. We report comparisons at T=0 and T=1. Cells are τ / accuracy.

T=0, L=5 (Llama-3.1 8B→70B)

RuleGSM8KIFEvalHumanEvalMMLU
Lossless SD (= typical, T=0)4.965 / 94.774.128 / 86.995.415 / 78.74.015 / 85.89
DropMatch α=15.425±0.135 / 94.04±0.244.459±0.032 / 85.35±0.645.558±0.050 / 79.5±1.44.436±0.037 / 85.34±0.35
DropMatch α=25.737±0.065 / 93.31±0.325.097±0.034 / 84.21±0.565.695±0.050 / 80.0±1.25.063±0.020 / 84.61±0.14
SpecDec (top-3, gap ≤ 1)5.685 / 93.634.776 / 86.325.693 / 81.14.805 / 85.60
SpecDec (top-3, gap ≤ 2)5.813 / 93.335.201 / 84.665.835 / 78.05.222 / 84.37
SpecDec (top-3, gap ≤ 3)5.837 / 92.955.382 / 84.915.880 / 80.55.396 / 83.96
p_target > 0.35.373 / 94.624.409 / 86.675.618 / 79.94.375 / 85.77
p_target > 0.25.651 / 94.244.751 / 86.655.724 / 82.34.776 / 85.00
p_target > 0.095.849 / 93.035.248 / 85.855.832 / 77.45.283 / 84.10
  • The typical acceptance of Cai et al. (Medusa, 2024) equals lossless SD at T=0.
  • Although it varies by task, the simple threshold on the target probability of the draft token works better than expected: at 0.3 it is on par with DropMatch, and at 0.2 it outperforms all other methods.
  • For accuracy, looking directly at the target probability of the draft token turns out to be advantageous, whereas DropMatch looks only at distribution-level uncertainty.

T=1, L=5, top_p=1.0

RuleLlama GSM8KLlama HumanEvalLlama MMLUQwen3 GSM8KQwen3 IFEval
Lossless SD4.776 / 91.99±0.225.290 / 80.9±3.43.839 / 85.084.787 / 76.58±1.383.284 / 78.51±1.33
typical ε=0.34.947 / 93.81±0.245.340 / 80.1±2.53.850 / 85.225.087 / 80.64±0.933.990 / 81.52±0.26
typical ε=0.095.316 / 93.23±0.315.623 / 78.3±3.14.342 / 84.725.451 / 82.64±0.604.571 / 81.03±0.40
typical ε=0.035.533 / 92.22±0.195.725 / 78.1±1.64.770 / 83.465.678 / 82.16±0.845.069 / 80.73±0.56
p_target > 0.24.890 / 93.68±0.165.413 / 79.3±1.13.763 / 85.745.029 / 80.44±0.303.621 / 81.98±0.40
p_target > 0.095.256 / 93.48±0.405.606 / 79.7±1.24.231 / 84.565.373 / 81.80±0.824.267 / 80.86±0.64
p_target > 0.035.525 / 92.19±0.385.771 / 76.6±1.44.738 / 83.315.643 / 82.69±0.884.943 / 80.28±0.27
DropMatch α=15.202±0.147 / 90.08±0.605.423±0.068 / 80.0±1.44.050±0.072 / 84.00±0.185.535±0.068 / 80.06±1.004.240±0.137 / 79.02±0.78
DropMatch α=25.652±0.065 / 87.62±0.695.628±0.035 / 76.8±4.05.061±0.033 / 80.41±0.205.808±0.037 / 81.65±1.425.346±0.103 / 78.50±0.96
  • At T=1, typical acceptance is at the top of the Pareto front, and the fixed threshold lies on the same line.
  • Typical acceptance picks the first token of every step greedily and cuts off tail tokens, which keeps its performance toward that of T=0.

Output distribution and diversity (Llama GSM8K, T=1; three sampling seeds per question).

  • Distribution shift: mean negative log-probability minus mean entropy when the output is fed back into the target (teacher forcing). It is 0 for exact samples of the target, negative when biased toward greedy (greedy output: −0.134), and positive when the tail is heavier.
  • Similarity: mean word-level Ratcliff/Obershelp similarity over all pairs of the three responses to a question (1 = identical).
  • Self-BLEU: mean BLEU of one response against the other two as references (higher = more similar responses).
  • distinct-2: fraction of distinct 2-grams among those in the three responses (higher = more diverse).
RuleDistribution shiftSimilaritySelf-BLEUdistinct-2
Lossless SD+0.0150.48146.80.636
DropMatch α=1+0.0470.48747.70.628
typical ε=0.3 / 0.2 / 0.09 / 0.03−0.091 / −0.080 / −0.054 / −0.0240.608 / 0.583 / 0.549 / 0.51761.3 / 58.6 / 54.7 / 51.40.539 / 0.556 / 0.580 / 0.600
p_target > 0.2 / 0.09 / 0.03−0.089 / −0.061 / −0.0230.612 / 0.557 / 0.51961.7 / 55.7 / 51.60.538 / 0.574 / 0.600
SpecDec (top-3, gap ≤ 1 / 2)−0.091 / −0.0630.616 / 0.56562.1 / 56.30.536 / 0.571
  • Over three seeds on the same questions at T=1, the responses of the threshold-based rules tend to cluster, whereas DropMatch tends to follow the diversity of SD.
  • T=1 is intended for diversity rather than accuracy and is meant to include tail tokens; the threshold-based rules have the drawback of not following this.

Q3. Please explain how the 0.05% was measured, and report the overhead of the K dropout paths and the full-vocabulary JS computation at larger batch sizes and longer draft lengths.

A3. We correct the 0.05% LM head cost. The earlier measurement was wrong because of a cuda synchronization error. The re-measured value, obtained on H100, is 1.3% of the target forward pass, and DropMatch adds 2.0% of the target forward pass at batch 1, which is 0.8% of a full round.

L=5 (Llama-3.1 8B→70B, GSM8K). Times are in ms.

BatchTarget forwardStandard LM head (% of target forward)DropMatch added (K=5 paths + acceptance test)Total added% of target forward% of round
117.080.222 (1.3%)0.257 + 0.0870.342.0%0.8%
819.620.245 (1.2%)0.484 + 0.5481.035.3%2.1%
1622.750.315 (1.4%)0.780 + 0.7891.576.9%2.5%
3228.300.425 (1.5%)1.505 + 1.7123.2211.4%3.9%
6449.260.729 (1.5%)2.918 + 4.7357.6515.5%5.4%
  • Previously we ran a toy experiment; this time, for an accurate measurement, we measured on 200 GSM8K questions. The total overhead is 2.0%, slightly higher than the 1.64% in the paper.
  • As the batch grows, the cost relative to the target forward pass increases, and it also increases gradually relative to the full round. The overhead relative to the full round is similar to the rate at which the speedup decreases when we measure the speed per batch size on GSM8K.

Longer draft length (L=10, same measurement).

BatchTarget forwardStandard LM head (% of target forward)DropMatch added (K=5 paths + acceptance test)Total added% of target forward% of round
117.210.229 (1.3%)0.274 + 0.1220.402.3%0.6%
822.510.303 (1.3%)0.757 + 0.8391.607.1%2.0%
1628.110.414 (1.5%)1.438 + 1.8193.2611.6%3.2%
  • At L=10 the number of positions the target verifies at once grows from 6 to 11, so the added time becomes slightly larger (0.34 → 0.40 ms at batch 1), but the draft phase becomes longer, so the share of a round is at the same level as for L=5.

Q4. What are the decoding temperature and sampling scheme of the main experiments? How does DropMatch behave when the target distribution itself is sampled (e.g., temperature 0.7 or 1.0)?

A4. The main experiments were run with decoding temperature T=0, i.e., greedy decoding.

Re-measurement across temperatures (Llama-3.1 8B→70B, GSM8K, L=5, 3 seeds). Cells are τ / accuracy.

Ttop_pLossless SDDropMatch α=1ΔτΔacc
0.31.04.975±0.015 / 94.29±0.805.401±0.138 / 93.55±0.25+8.6%−0.74
0.51.04.962±0.031 / 94.39±0.305.369±0.133 / 93.40±0.34+8.2%−0.99
0.71.04.922±0.034 / 93.58±0.355.315±0.138 / 92.89±0.36+8.0%−0.69
1.01.04.776±0.016 / 91.99±0.225.202±0.147 / 90.08±0.60+8.9%−1.91
1.00.954.846±0.028 / 93.13±0.575.257±0.138 / 91.72±0.31+8.5%−1.41
  • For T ≤ 0.7 the results are at the same level as T=0, and the difference becomes larger at T=1.0.

T=1 on four tasks.

ModelMethodLGSM8KMMLUIFEvalHumanEval
Llama-3.1 8B/70BSD54.776 / 91.99±0.223.839 / 85.083.941 / 81.01±1.035.290 / 80.9±3.4
Llama-3.1 8B/70BSD + DM55.202±0.147 / 90.08±0.604.050±0.072 / 84.00±0.184.237±0.072 / 80.95±0.495.423±0.068 / 80.0±1.4
Llama-3.1 8B/70BSD107.073 / 92.195.075 / 84.525.090 / 82.548.552 / 76.8
Llama-3.1 8B/70BSD + DM107.496 / 90.375.531 / 82.685.949 / 80.118.802 / 77.4
Qwen3 4B/32BSD54.787 / 76.58±1.384.414 / 79.843.284 / 78.51±1.334.893 / 87.8
Qwen3 4B/32BSD + DM55.535±0.068 / 80.06±1.005.036±0.126 / 79.60±1.234.240±0.137 / 79.02±0.785.280±0.056 / 89.9±0.9
Qwen3 4B/32BSD107.089 / 76.125.878 / 79.703.831 / 76.317.384 / 89.0
Qwen3 4B/32BSD + DM109.488 / 79.238.797 / 79.575.743 / 78.668.550 / 91.5
  • On Llama, DropMatch provides a speedup at a level similar to T=0, or slightly lower.
  • On Qwen, the increase in acceptance length is larger than at T=0.
Reviewer_HGerOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

DropMatch is a training-free, lossy acceptance rule designed to accelerate speculative decoding. Instead of full-model verification, it passes the target's final hidden state through the LM head KK times with independent dropout masks (K=5,pdrop=0.3K=5, p_{\text{drop}}=0.3), generating KK perturbed next-token distributions. A draft token gets accepted if its JS divergence from the centroid of these distributions stays within the maximum divergence among the heads, or if it matches the majority head token (with the majority token emitted on rejection). Because only the LM head is re-evaluated, the overhead is kept remarkably low (e.g., around 1.64% for Llama-3.1-70B). The authors evaluate the method on Llama-3.1-8B/70B and Qwen3-4B/32B pairs using vLLM across GSM8K, MMLU, IFEval, and HumanEval. Their results show roughly 10% longer acceptance lengths and 1.00x to 1.33x speedups over standard speculative decoding with minimal accuracy drops, alongside integrations with Auto-Judge and EAGLE-3 (yielding up to 1.09x additional speedup on EAGLE-3) and OOD evaluations on KoMT-Bench and MGSM.

Critical strengths

The method is very easy to adopt: it needs no training, no extra parameters and no change to the draft model, it touches only the verification step, and it composes with existing pipelines, which the paper shows with EAGLE-3, Auto-Judge, and batch sizes up to 128 in vLLM (Table 3 stays around 1.1x). Appendix C is the most useful part of the paper. Comparing against top-k acceptance and Gaussian logit noise with paired McNemar tests over all of GSM8K is exactly the kind of control this line of work needs, and the head-agreement analysis in Tables 9 and 10 explains clearly why the gain is small on HumanEval.

Critical weaknesses

My main concern is that the paper's own controls suggest the dropout mechanism adds nothing over simpler lenient acceptance rules. Gaussian noise on the logits at the matched scale gives the same result (94.0 / τ 5.51 vs 93.9 / 5.47, Table 12), and the top-k rules in Table 11 lie on the same accuracy versus acceptance-length curve as the p_drop sweep in Table 8: top-3 gives 90.8 / 5.86 and DropMatch at p_drop = 0.6 gives 90.75 / 5.87, while top-2 (93.1 / 5.71) falls between the p_drop = 0.4 and 0.5 points. The claim that the width of the acceptance region "is not a free parameter" does not hold either, since p_drop sets it and was tuned on GSM8K just as σ or k would be. The most relevant training-free baselines are also missing: Medusa's typical acceptance (Cai et al., 2024) is an entropy-adaptive lenient rule with essentially no overhead, and divergence-threshold rules such as Fuzzy Speculative Decoding do what Eq. 6 does with an explicit knob. Without an accuracy versus speedup Pareto comparison against these, I see no evidence that DropMatch beats a one-line threshold. The second concern is that the gains are small relative to the accuracy cost, and the claims go further than the data. Speedups at L = 5 are 1.00x to 1.19x, while accuracy drops by up to 3.0 points on HumanEval (Llama, L = 10), 2.5 points on IFEval (Qwen, L = 5) and 4.0 to 5.6 points on MGSM zh/de/fr. Meanwhile the lossless SD baseline itself moves by up to 1.7 points between L = 5 and L = 10 (HumanEval with Qwen, 90.4 vs 92.1), so the single-run main table cannot resolve differences of this size. The MGSM results also contradict the out-of-distribution robustness argument, since DropMatch degrades more there than in English. Finally, the "semantic" in the title is not supported at the token level: the rule compares token distributions, and the sentence-level similarity in Figure 3 is measured between whole dropout paths, not on the accepted substitutions.

Questions for authors

(1) Can you add an accuracy versus speedup (or τ) Pareto plot on GSM8K and HumanEval that sweeps p_drop for DropMatch, k for top-k, σ for Gaussian noise, and the thresholds of Medusa's typical acceptance? Showing DropMatch clearly above those curves is what would change my assessment. (2) What decoding temperature is used in Table 4? The reproducibility statement says Section 4.2 gives it, but I could not find it there, and the lossless SD accuracy differs between L = 5 and L = 10. How does the rule behave under sampling, and what can be said about the distribution of the emitted tokens in that case? (3) How was the 0.05% LM-head share in Section 4.1 measured? For Llama-3.1-70B the head is 128256 x 8192, about 1.05B of 70.6B parameters, so in memory-bound decoding I would expect something closer to 1.5%. (4) Eq. 6 compares whole distributions and never looks at the probability of the specific draft token. How often does Eq. 6 accept a draft token that has low probability under the undropped target, and how does this change with sampled rather than greedy drafts? (5) The EAGLE-3 accuracy comparisons in Table 5 use 80 samples. Could you report full-set accuracy with variance for every draft length?

3 comments

Q1. Accuracy vs. τ Pareto comparison with top-k, Gaussian noise, and typical acceptance.

A1. We report the requested sweeps at T=0 and T=1 (Llama-3.1 8B→70B, L=5). Cells are τ / accuracy (± = std over runs); "Gaussian (VM)" = variance-matched Gaussian noise.

T=0 (lossless SD re-measured)

RuleGSM8KIFEvalHumanEval
Lossless SD4.965 / 94.774.128 / 86.995.415 / 78.7
typical ε=0.09= SD= SD= SD
DropMatch α=0.755.220±0.196 / 93.97±0.624.291±0.032 / 85.64±0.725.504±0.026 / 80.3±3.3
DropMatch α=15.425±0.135 / 94.04±0.244.459±0.032 / 85.35±0.645.558±0.050 / 79.5±1.4
DropMatch α=25.737±0.065 / 93.31±0.325.097±0.034 / 84.21±0.565.695±0.050 / 80.0±1.2
top-25.707 / 92.955.173 / 85.935.863 / 79.9
top-35.864 / 90.375.511 / 84.595.906 / 75.6
Gaussian (VM) α=0.755.396±0.005 / 93.78±0.594.441±0.000 / 85.18±0.305.557±0.006 / 81.1±1.0
Gaussian (VM) α=15.609±0.009 / 92.62±0.634.684±0.006 / 84.88±0.965.659±0.003 / 81.1±1.0
Gaussian (VM) α=25.858±0.006 / 91.58±0.405.407±0.004 / 82.86±0.885.805±0.004 / 78.0±1.3
  • As noted by Cai et al. (Medusa, 2024), typical acceptance reduces to lossless SD at T=0. For DropMatch, following Reviewer iZXM, we added a scale α that controls the region boundary (reflected in Eq. 6), so that accuracy can be compared at the same τ.
  • DropMatch is slightly ahead of top-k and variance-matched Gaussian noise on GSM8K.

T=1

RuleGSM8KHumanEval
Lossless SD4.776±0.016 / 91.99±0.225.290±0.014 / 80.9±3.4
typical ε=0.34.947±0.013 / 93.81±0.245.340±0.017 / 80.1±2.5
typical ε=0.25.092±0.002 / 93.56±0.135.446±0.020 / 79.1±2.5
typical ε=0.095.316±0.005 / 93.23±0.315.623±0.013 / 78.3±3.1
typical ε=0.035.533±0.006 / 92.22±0.195.725±0.019 / 78.1±1.6
DropMatch α=0.54.609±0.221 / 91.68±0.515.227±0.077 / 79.3±1.6
DropMatch α=0.754.932±0.208 / 90.93±0.395.332±0.055 / 80.7±0.3
DropMatch α=15.202±0.147 / 90.08±0.605.423±0.068 / 80.0±1.4
DropMatch α=25.652±0.065 / 87.62±0.695.628±0.035 / 76.8±4.0
top-25.111±0.005 / 92.19±0.475.674±0.006 / 77.0±1.5
top-35.420±0.004 / 89.94±0.125.837±0.008 / 72.0±0.6
top-55.641±0.006 / 87.04±0.465.926±0.006 / 70.7±2.7
top-105.807±0.009 / 83.37±0.125.975±0.002 / 67.9±2.1
Gaussian (VM) α=0.54.738±0.008 / 88.78±0.455.309±0.025 / 79.1±2.5
Gaussian (VM) α=15.497±0.002 / 87.06±0.585.574±0.010 / 78.5±4.6
Gaussian (VM) α=25.837±0.006 / 83.95±0.575.756±0.001 / 76.0±1.5
Gaussian (isotropic σ=1.4)5.350±0.004 / 89.74±0.765.533±0.039 / 76.0±1.5
  • Typical acceptance is the top curve on GSM8K. On HumanEval the deviations are wide and the rules cannot be separated below τ = 5.6, while top-k loses accuracy by more than the deviation.

Similarity between responses at T=1 (GSM8K). Responses to the same question from three sampling seeds are compared (Ratcliff/Obershelp similarity, distinct-n).

RuleSimilaritydistinct-2Acc.
Lossless SD0.4810.63691.99
DropMatch α=1 (mean of 3 mask seeds)0.4890.63090.31
typical ε=0.30.6080.53993.81
typical ε=0.20.5830.55693.56
typical ε=0.090.5490.58093.23
typical ε=0.030.5170.60092.22
top-20.5540.57592.19
top-50.4800.62187.04
  • DropMatch α=1 keeps the diversity of SD, whereas typical acceptance raises similarity, i.e., reduces diversity. T=1 is meant to produce diverse samples, but typical acceptance pulls the output toward greedy, trading diversity for accuracy.
  • Typical acceptance picks the first token of every step greedily and cuts off tail tokens, so its accuracy is even higher than that of SD at T=1.
  • On rejection, SD resamples from the residual distribution, so the final distribution matches the target. Typical acceptance only accepts or rejects against a threshold, so mass concentrates on high-probability tokens. Leviathan et al. (2023) also note that their lenience could hurt sample diversity.
  • DropMatch aims for speedup at SD-level diversity. top-5 also keeps SD-level diversity, but its quality drops substantially.

Relation to FSD. We were not aware of Fuzzy Speculative Decoding (Holsman et al., FSD, 2025) and confirm that it is closely related. FSD uses a fixed threshold T and lists as a limitation that one cannot know in advance which T gives which quality (it uses 0.3–0.7 depending on the model pair). In DropMatch the radius is set at each position by the dropout variance, and (K, p_drop) were chosen once on GSM8K and used unchanged for all tasks and both model families.

Q2. What decoding temperature was used in Table 4? The lossless SD accuracy differs between L=5 and L=10. How does the rule behave under sampling, and what can be said about the distribution of emitted tokens?

A2. We confirm that the temperature was missing from §4.2. The temperature used in our experiments is T=0, and the revision now states this setting.

The DropMatch entries in Table 4 are not single runs but averages over 6–10 seeds; the standard deviations were left out of the table for space.

The lossless SD accuracy differs between L=5 and L=10 for the following reason. Even under greedy decoding, L=5 and L=10 produce different tensor shapes, which changes the order of operations in the matrix multiplications, and in bfloat16 this changes the last digits of the logits. At positions where the logits are nearly tied, this small difference flips the argmax, the token changes, and the generation that follows diverges.

We add the tables below to describe the behavior under sampling.

Behavior under sampling (Llama-3.1 8B→70B, GSM8K, L=5). Cells are τ / accuracy.

Ttop_pLossless SDDropMatch α=1ΔτΔacc
0.31.04.975±0.015 / 94.29±0.805.401±0.138 / 93.55±0.25+8.6%−0.74
0.51.04.962±0.031 / 94.39±0.305.369±0.133 / 93.40±0.34+8.2%−0.99
0.71.04.922±0.034 / 93.58±0.355.315±0.138 / 92.89±0.36+8.0%−0.69
1.01.04.776±0.016 / 91.99±0.225.202±0.147 / 90.08±0.60+8.9%−1.91
1.00.954.846±0.028 / 93.13±0.575.257±0.138 / 91.72±0.31+8.5%−1.41
  • The τ gain is maintained across temperatures. The accuracy loss becomes somewhat larger at T=1. These runs use top_p=1.0, and we attribute the loss to tail probabilities being accepted: among the accepted non-argmax tokens, the fraction whose target probability is below 0.05 is 2.9% at T=0 and 28.9% at T=1. When measured with top_p=0.95, the τ gain is maintained and the accuracy difference becomes smaller.

T=1 on four tasks. Cells are τ / accuracy.

ModelMethodLGSM8KMMLUIFEvalHumanEval
Llama-3.1 8B/70BSD54.776 / 91.99±0.223.839 / 85.083.941 / 81.01±1.035.290 / 80.9±3.4
Llama-3.1 8B/70BSD + DM55.202±0.147 / 90.08±0.604.050±0.072 / 84.00±0.184.237±0.072 / 80.95±0.495.423±0.068 / 80.0±1.4
Llama-3.1 8B/70BSD107.073 / 92.195.075 / 84.525.090 / 82.548.552 / 76.8
Llama-3.1 8B/70BSD + DM107.496 / 90.375.531 / 82.685.949 / 80.118.802 / 77.4
Qwen3 4B/32BSD54.787 / 76.58±1.384.414 / 79.843.284 / 78.51±1.334.893 / 87.8
Qwen3 4B/32BSD + DM55.535±0.068 / 80.06±1.005.036±0.126 / 79.60±1.234.240±0.137 / 79.02±0.785.280±0.056 / 89.9±0.9
Qwen3 4B/32BSD107.089 / 76.125.878 / 79.703.831 / 76.317.384 / 89.0
Qwen3 4B/32BSD + DM109.488 / 79.238.797 / 79.575.743 / 78.668.550 / 91.5

Distribution of emitted tokens. We feed the output that each rule produced at T=1 back into the target (teacher forcing) and measure the mean negative log-likelihood minus the mean entropy of the target over the emitted tokens.

Rule (T=1)mean NLL − mean entropy
Lossless SD+0.015 (another sampling seed: +0.018)
Greedy output (T=0, for reference)−0.134
DropMatch α=1 / 2 / 3+0.047 / +0.066 / +0.077
typical ε=0.3 / 0.2 / 0.09 / 0.03−0.091 / −0.080 / −0.054 / −0.024
top-2 / 3 / 5 / 10−0.045 / −0.006 / +0.025 / +0.055
Gaussian noise (variance-matched α=1 / 2; isotropic σ=1.4)+0.079 / +0.089; +0.052
  • This value is 0 when the output is distributed like the target's own samples. A negative value means the output is closer to greedy, and a positive value means the tail has become heavier.
  • Typical acceptance moves in the direction opposite to the intended behavior of T=1.
  • top-5 is almost the same as SD on this metric, but its accuracy is about 5 points lower.

Q3. How was the 0.05% LM head share in §4.1 measured? For Llama-3.1-70B the head is 128256 × 8192, about 1.05B of 70.6B parameters, so a value close to 1.5% is expected for memory-bound decoding.

A3. The reviewer's expectation of a value close to 1.5% for memory-bound decoding, assuming batch-1 single-token decoding, is correct. In our experiment a cuda synchronization error caused a measurement mistake only in the LM head segment, and the re-measurement confirms about 1.3%.

However, the end-to-end overhead of the proposed method was measured from the start point to the end point of DropMatch, so it remains the same. Nothing changes in the speed results of the experiments.

Overhead on H100 (Llama-3.1 8B→70B, GSM8K, L=5). Times are in ms.

BatchTarget forwardStandard LM head (% of target forward)DropMatch added (K=5 paths + acceptance test)Total added% of target forward% of round
117.080.222 (1.3%)0.257 + 0.0870.342.0%0.8%
819.620.245 (1.2%)0.484 + 0.5481.035.3%2.1%
1622.750.315 (1.4%)0.780 + 0.7891.576.9%2.5%
3228.300.425 (1.5%)1.505 + 1.7123.2211.4%3.9%
6449.260.729 (1.5%)2.918 + 4.7357.6515.5%5.4%
  • The overhead relative to a round follows the same trend as Table 3 in the paper. At batch 1, the overhead on the target side was 1.64% on A100 in the paper; on H100 it is now 2.0% relative to the target model, which corresponds to 0.8% of a full round. We reflect the correction in Figure 5.

Q4. Eq. 6 compares whole distributions and does not look at the probability of the specific draft token. How often are draft tokens with low probability under the undropped target accepted, and how does this change with sampled drafts?

A4. For both model families, we recorded the probability that the undropped target assigns to the draft token at T=0 with α=1.

Llama GSM8KLlama IFEvalQwen GSM8K
Positions judged122,142165,352145,535
Accepted tokens that are not the target argmax4.85%4.95%4.39%
└ median p_target of those tokens0.2990.2560.220
└ p_target < 0.224.5%35.3%45.3%
└ p_target < 0.052.9%5.4%16.1%
└ within the target's top-392.8%91.8%82.5%
└ accepted by the JS criterion only / by the token-matching criterion only70% / 4%63% / 14%76% / 4%
  • 95% of the accepted tokens are equal to the target argmax, and most of the rest are tokens within the target's top-3. The acceptances of low-probability tokens come mostly from the JS criterion, not from the token-matching criterion.

Change under sampling (Llama GSM8K, α=1, top_p=1.0).

T=0T=0.7T=1.0
Accepted tokens that are not the target argmax4.85%8.88%14.18%
└ median p_target of those tokens0.2990.2350.146
└ p_target < 0.224.5%42.0%60.0%
└ p_target < 0.052.9%12.3%28.9%
└ within the target's top-392.8%85.2%69.7%
└ accepted by the JS criterion only / by the token-matching criterion only70% / 4%78% / 3%83% / 2%
  • As T increases, the draft proposes tail tokens more often and some of them are accepted. The same effect is observed in our answer A2, and it can be mitigated with the top_p setting.

Q5. The EAGLE-3 accuracy comparison in Table 5 uses 80 samples. Can you report full-set accuracy with variance for all draft lengths?

A5. The 80 questions follow the evaluation convention of Li et al. (EAGLE-3, 2025), i.e., the 80 questions of MT-Bench. MT-Bench consists of 80 questions in total, so that result is a full-set result. The paper also reports full-set accuracy on GSM8K and Alpaca.

Table 5 and the full-set results agree well in trend. Providing variance requires measuring with multiple seeds, but MT-Bench and Alpaca use a GPT-4 judge, which makes it difficult to provide variance for cost reasons. For GSM8K we provide a full-set re-measurement.

GSM8K, full test set (Llama-3.3-70B, EAGLE-3).

LEAGLE-3 (accuracy / τ)+ DropMatch (mean±sd over 3 seeds)
581.65 / 5.84281.12±0.23 / 6.170
781.65 / 6.52681.27±0.46 / 7.124
981.58 / 6.77380.92±0.23 / 7.542
1081.65 / 7.15981.17±0.31 / 7.976
  • These results agree with the full-set numbers in the paper and with the trend of Table 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.