- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DropMatch, a training-free acceptance mechanism for lossy speculative decoding. The method applies independent dropout masks to the target model’s final hidden representation and obtains multiple predictions through the shared LM head. Draft tokens are accepted using either token matching across these predictions or a JS-divergence criterion supplemented by majority agreement. Experiments evaluate DropMatch with standard speculative decoding, Auto-Judge, and EAGLE-3, demonstrating increased acceptance lengths and additional speedups in several configurations.
Critical strengths
- This work explores whether predictions obtained through inference-time perturbations can support token acceptance without training an additional verifier. Reducing the dependence on learned judge components is a useful direction for making lossy speculative decoding easier to apply across models and tasks.
- Applying dropout only before the LM head allows the stochastic predictions to share the computation of the transformer backbone. The method requires no additional training data or model parameters, and the reported overhead analysis supports the practicality of this design in the evaluated setting.
- The authors conduct experiments with Llama and Qwen models on reasoning, knowledge, instruction-following, and code-generation benchmarks. The integrations with Auto-Judge and EAGLE-3, together with the batched throughput experiments, help demonstrate the potential applicability of the approach within existing decoding pipelines.
- The manuscript includes ablations on the dropout probability and number of paths, as well as analyses of head agreement and acceptance behavior. I also appreciate the comparisons with deterministic top-k acceptance and Gaussian perturbations in the appendix, which provide useful evidence for understanding the role of the proposed components.
Critical weaknesses
- Further justification of the semantic acceptance mechanism would strengthen the contribution. My main concern is the connection between variability under dropout and the semantic validity of an accepted draft token. Figure 3 provides encouraging evidence of similarity between generated paths, while Tables 9–10 characterize head agreement and substitution frequency. However, these analyses do not directly establish whether the additional substitutions admitted by DropMatch preserve meaning or downstream correctness. In particular, Eq. 6 compares entire distributions and uses the maximum divergence among dropout heads as the acceptance threshold, but its relationship to the error risk of an individual substitution remains unclear. A direct evaluation of the accepted substitutions would help substantiate the central motivation.
- The specific benefit of MC dropout could be established more clearly. Appendix C provides helpful comparisons, although Table 12 shows that Gaussian perturbations with a matched scale achieve essentially the same accuracy and acceptance length as DropMatch: 94.0% and 5.51 versus 93.9% and 5.47. The authors explain that dropout supplies a position-dependent perturbation scale automatically, which is a reasonable motivation. However, the current comparison with substantially smaller and larger Gaussian scales does not isolate the contribution of this position dependence. Comparisons against adaptive perturbations or entropy-based acceptance rules, preferably at matched acceptance rates or quality levels, would help clarify the methodological advantage of the proposed design.
- The speed–quality trade-off deserves a more qualified discussion. Although DropMatch provides useful acceleration in several settings, some results suggest that increased acceptance length does not always translate into a favorable trade-off. For Llama-3.1 on HumanEval with , Table 4 reports a decrease in Pass@1 from 81.70 to 79.89 with a speedup of 1.00×. At , Pass@1 decreases from 81.70 to 78.70, while the reported speed changes from 1.13× to 1.12× under the same normalization. I appreciate the repeated runs and paired tests in Appendix C, but the absence of a statistically significant difference in one configuration does not establish quality equivalence. Reporting speed improvements under an explicit quality tolerance would make the practical benefits easier to assess.
- The conclusions about robustness under distribution shift would benefit from further qualification. Avoiding additional training is an attractive property, although the multilingual results indicate that the acceptance mechanism can still be sensitive to distribution changes. In Table 13, with , adding DropMatch reduces accuracy from 83.2% to 78.4% on MGSM-ZH, from 84.8% to 79.2% on MGSM-DE, and from 82.0% to 78.0% on MGSM-FR, for speedups of 1.07×, 1.10×, and 1.07×, respectively. The explanation in Appendix D, involving flatter target distributions and greater sensitivity to dropout, is helpful. Bringing these limitations into the main discussion and clarifying when the method is expected to remain reliable would provide a more balanced account of its generalization behavior.
Questions for authors
- Could the authors directly evaluate the semantic validity and downstream correctness of the additional substitutions accepted by DropMatch, separately for the JS-divergence and token-matching criteria? Does the acceptance radius reliably distinguish harmless substitutions from harmful ones?
- Could the authors isolate the benefit of position-dependent dropout through comparisons with adaptive perturbation scales or entropy-based acceptance rules at matched acceptance rates or quality levels?
- What practical guidance would the authors recommend for selecting and on an unseen domain with a specified quality tolerance? Additional uncertainty estimates for the principal speed–quality comparisons would also be helpful.
4 comments
Q2. Can you isolate the benefit of position-dependent dropout by comparing, at the same acceptance rate or the same quality, with an adaptive perturbation scale or an entropy-based acceptance rule?
A2. To isolate the benefit of position-dependent dropout, we compared against a control in which the magnitude of the perturbation is kept equal to that of dropout and only the structure is changed. The control is a Gaussian perturbation with a position- and token-wise adaptive scale: to each logit at each position we add noise with the same variance that dropout produces, but with the covariance between tokens removed. The acceptance rule and K are the same as in DropMatch. The entropy-based acceptance rule (typical acceptance) reduces to lossless SD at T=0, so we compared it at T=1.
Llama-3.1 8B→70B, GSM8K, T=0 (mean ± standard deviation over 3 runs each; τ / accuracy)
| α | DropMatch | Adaptive-scale Gaussian |
|---|---|---|
| 0.5 | 5.064±0.086 / 94.29±0.22 | 5.068±0.005 / 93.71±0.27 |
| 0.75 | 5.309±0.076 / 93.88±0.12 | 5.396±0.005 / 93.78±0.59 |
| 1 | 5.474±0.051 / 93.91±0.04 | 5.609±0.009 / 92.62±0.63 |
| 1.5 | 5.674±0.032 / 93.56±0.13 | 5.790±0.002 / 92.04±0.27 |
| 2 | 5.767±0.022 / 93.13±0.19 | 5.858±0.006 / 91.58±0.40 |
| 3 | 5.857±0.016 / 92.09±0.35 | 5.921±0.001 / 89.97±0.29 |
| 5 | 5.916±0.007 / 90.83±0.46 | 5.958±0.002 / 88.75±0.48 |
- As τ grows, DropMatch maintains accuracy better than the adaptive-scale Gaussian. This shows that DropMatch has a separate benefit.
T=1, top_p=1.0, GSM8K
| Rule | τ / accuracy | Distribution shift | Inter-response similarity |
|---|---|---|---|
| Lossless SD | 4.776 / 91.99±0.22 | +0.015 | 0.481 |
| DropMatch α=1 | 5.202±0.147 / 90.08±0.60 | +0.047 | 0.489 |
| DropMatch α=2 | 5.652±0.065 / 87.62±0.69 | +0.066 | 0.463 |
| Adaptive-scale Gaussian α=1 | 5.497 / 87.06±0.58 | +0.079 | 0.449 |
| Adaptive-scale Gaussian α=2 | 5.837 / 83.95±0.57 | +0.089 | 0.437 |
| typical ε=0.3 | 4.947 / 93.81±0.24 | −0.091 | 0.608 |
| typical ε=0.2 | 5.092 / 93.56±0.13 | −0.080 | 0.583 |
| typical ε=0.09 | 5.316 / 93.23±0.31 | −0.054 | 0.549 |
| typical ε=0.03 | 5.533 / 92.22±0.19 | −0.024 | 0.517 |
Distribution shift is the mean negative log-probability minus the mean entropy when the output is fed back into the target (0 for exact samples of the target, negative toward greedy; the greedy output gives −0.134). Inter-response similarity is the mean word-level Ratcliff/Obershelp similarity among three responses to the same question.
Reading the T=1 table above:
- Looking only at accuracy and τ, the typical acceptance of Cai et al. (Medusa, 2024) is higher than DropMatch.
- However, typical acceptance fills rejected positions and the first token of every step greedily and cuts off tail tokens, so its output moves from the T=1 distribution toward greedy. As a result the output distribution becomes sharper than the T=1 sampling distribution, and its accuracy being higher than that of lossless SD includes this shift. At ε=0.3 it moves two thirds of the distance to greedy (−0.091), and the responses to the same question become more similar to each other (0.608).
- DropMatch α=1 raises τ by 9% while keeping sample diversity at the level of lossless SD (0.489 vs. 0.481). At α=2, τ rises by 18%, while the output becomes slightly more spread out than lossless SD (0.463) and the accuracy is 4.4 points lower. If the purpose of sampling is diverse output, the two rules preserve different things.
Q3. What is the practical guidance for choosing K and p_drop in an unseen domain with a given quality tolerance? Uncertainty estimates for the main speed–quality comparisons would also help.
A3. For additional information on uncertainty, we report mean ± standard deviation in the answers above.
K and p_drop were chosen with a single ablation on GSM8K in our experiments and were then used on the other tasks and on Qwen3 without re-tuning. The experimental results show that the hyperparameters we chose work well on two model families.
During the rebuttal we added α. We confirmed that values between 1 and 2 cause a small drop in accuracy but can well be used for additional speedup. When a quality tolerance is given, keeping K and p_drop fixed and controlling α is the easier way to adjust the trade-off.
Thank for the authors' additional experiments and detailed response. The decline on GSM8K and remaining uncertainty on HumanEval, both relatively basic benchmarks, raise concerns that quality losses could be greater on harder tasks. Unpredictable quality degradation can undermine user trust in lossy SD methods. Moreover, the reported speed gains are not consistently substantial enough to justify these reliability risks. I therefore maintain my original score.
Thank you for considering our additional experiments and for your candid assessment. Your comments point to what a stronger version needs: an evaluation on harder tasks, a clearer characterization of when quality is preserved, and a larger gain in return. We will take this into account in our future work.
Thank you again for your time and for the constructive feedback throughout the review.
Q1. Can you directly evaluate the semantic validity and downstream correctness of the substitutions that DropMatch additionally accepts, separately for the JS criterion and the token-matching criterion? Does the acceptance radius discriminate harmless substitutions from harmful ones?
A1. The semantic validity of the substitutions that DropMatch additionally accepts is evaluated with an LLM-as-a-judge in the MT-Bench and Alpaca experiments. Similar scores do not mean that the outputs are exactly semantically equivalent, but they show that a similar level of answer quality is maintained for the given queries. In addition, by examining the difference in accuracy between SD and DropMatch on GSM8K, we confirmed that there is no significant difference in terms of correctness.
Paired comparison (α=1, 5 seeds).
JS criterion vs. token-matching criterion. Each cell is: share of the accepted non-argmax tokens / median p_target / fraction within the target's top-3.
Discriminative power of the acceptance radius. Measured at T=0 on positions where the draft token is not equal to the target argmax.
Does a substitution change the final answer? On GSM8K we recorded the judged positions where the draft token differs from the target argmax (8,705 positions). For each position we let the target generate to the end in two ways, once with the draft token inserted and once with the target argmax inserted, and scored the final answers.