- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PARD-2, which supports both target-dependent and target-independent drafting through stochastic gating of target model hidden features. The authors also propose CAT, which dynamically reweights the training loss for each token to improve the accepted length.
Critical strengths
Flexible support for both target-dependent and target-independent drafting. The dual-mode design enables PARD-2 to serve different target models without additional training, while also benefiting from target-specific features when available. The experimental results in Table 6 demonstrate that stochastic gating introduces only a relatively small performance degradation compared to training with a single mode.
Critical weaknesses
1 Limited evaluation of target-independent generalization. Although PARD-2 does not require hidden features from the target model in its target-independent mode, it appears to still require the draft and target models to share the same tokenizer. Furthermore, the authors only evaluate the target-independent capability across different model sizes within the Qwen3 family. This limits the demonstrated applicability of the target-independent mode. Evaluating PARD-2 across different model families would provide stronger evidence of its generalization capability.
2 Limited novelty of the CAT method. The proposed CAT method uses the target model's confidence as a proxy for token acceptance probability to optimize the accepted length. However, numerous existing studies have explored directly optimizing acceptance rates or expected accepted length in speculative decoding. The proposed approach appears closely related to these methods, but the paper lacks sufficient discussion and citations of relevant prior work [1, 2, 3]. A more detailed comparison would help clarify the novelty and contributions of CAT.
[1] Xiandong Zou, Jianshu Li, Jing Huang, and Pan Zhou. Variational speculative decoding: Rethinking draft training from token likelihood to sequence acceptance. In ICML, 2026.
[2] Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026.
[3] Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. LK losses: Direct acceptance rate optimization for speculative decoding. In ICML, 2026.
Questions for authors
Could the authors evaluate whether the same PARD-2 draft model can generalize across different target model families? In particular, it would be valuable to include additional experiments on the Llama family to further demonstrate the generalization capability of the target-independent mode.
4 comments
Thank you for the detailed response and the additional experiments.
I have one follow-up question regarding Rebuttal-Table 2: At what sampling temperature were these experiments conducted? Could the authors also compared the average acceptance length of CAT and the other training objectives under different sampling temperatures?
We thank the reviewer for the follow-up question.
The results in Rebuttal-Table 2 were obtained with greedy decoding (T = 0). We further evaluate the same four checkpoints under stochastic sampling at T = 0.5 and T = 1:
Rebuttal-Table 3: Training objectives under stochastic sampling (T = 0.5 and T = 1). Target Model: Qwen3-8B. Each cell is TPS / AL.
| Method | T | HumanEval | GSM8K | MT-Bench | MATH-500 | Average |
|---|---|---|---|---|---|---|
| PARD-2 w/ LK [3] | 0.5 | 272.09 / 5.28 | 262.66 / 5.10 | 147.42 / 2.81 | 263.46 / 5.08 | 236.41 / 4.56 |
| PARD-2 w/ VSD [1] | 0.5 | 275.00 / 5.32 | 266.68 / 5.16 | 151.42 / 2.85 | 266.05 / 5.13 | 239.79 / 4.61 |
| PARD-2 w/ D-PACE [2] | 0.5 | 292.06 / 5.67 | 279.97 / 5.43 | 155.34 / 2.93 | 276.30 / 5.36 | 250.92 / 4.85 |
| PARD-2 w/ CAT (ours) | 0.5 | 308.14 / 5.98 | 290.07 / 5.64 | 158.56 / 2.99 | 284.24 / 5.51 | 260.25 / 5.03 |
| PARD-2 w/ LK [3] | 1 | 261.38 / 5.03 | 252.94 / 4.80 | 147.32 / 2.74 | 246.52 / 4.67 | 227.04 / 4.31 |
| PARD-2 w/ VSD [1] | 1 | 261.30 / 5.03 | 251.80 / 4.84 | 145.00 / 2.73 | 247.02 / 4.72 | 226.28 / 4.33 |
| PARD-2 w/ D-PACE [2] | 1 | 269.63 / 5.20 | 268.23 / 5.15 | 150.91 / 2.85 | 252.97 / 4.85 | 235.44 / 4.51 |
| PARD-2 w/ CAT (ours) | 1 | 293.17 / 5.69 | 274.17 / 5.29 | 154.41 / 2.91 | 260.54 / 5.00 | 245.57 / 4.72 |
Rebuttal-Table 3 shows that CAT achieves the highest acceptance length and throughput on every benchmark at both temperatures. As expected, the average AL of all methods decreases as the temperature increases, but CAT's advantage is preserved: its average AL is 5.16 / 5.03 / 4.72 at T = 0 / 0.5 / 1, compared with 4.98 / 4.85 / 4.51 for D-PACE, 4.71 / 4.61 / 4.33 for VSD, and 4.70 / 4.56 / 4.31 for LK. Across the three temperatures, CAT is consistently 9–10% higher than LK and VSD and 3.6–4.7% higher than D-PACE.
Thank you for the additional comments. I will raise my score to 3.
Response to Reviewer yASV
W1 & Q1: The target-independent mode still requires the draft and target models to share the same tokenizer. Could the authors evaluate whether the same PARD-2 draft model can generalize across different target model families?
A: Thanks for the good question. It is correct that TI mode requires the drafter and target to share a tokenizer. This is a requirement of standard speculative decoding in general, since verification compares draft and target tokens directly. In comparison, target-dependent methods such as EAGLE-3 and DFlash are more restrictive, requiring a dedicated drafter for every individual target, whereas PARD-2 requires only one drafter per tokenizer, which covers all sizes and versions within a family.
To address the reviewer's concern, we extend the target-independent (TI) evaluation to a second model family. We evaluate a single TI PARD-2 drafter, trained with Llama-3.1-8B, on three models from the Llama-3 family:
Rebuttal-Table 1 shows that the average AL varies by only 2.7% across the three targets. Together with the Qwen3 results (8B–32B, Main paper-Table 2), this shows that, within each of two different model families, a single TI drafter generalizes across models. We will add these results to the revised version.
W2: The proposed approach appears closely related to existing methods that directly optimize acceptance rates or expected accepted length, but the paper lacks discussion and citations of this prior work.
A: We thank the reviewer for the comment. To address the concern, we clarify how CAT differs from [1, 2, 3] and compare with them directly under a controlled setting. LK losses [3] replace the per-token KL objective with a direct acceptance-rate objective, changing how each token is supervised rather than how much it is weighted. VSD [1] optimizes path acceptance for tree-based drafting via EM with MCMC sampling, whereas CAT targets single-chain parallel drafting, which can be applied to high-concurrency serving in vLLM, with weights computed directly from target probabilities without any sampling. D-PACE [2] uses the drafter's own confidence as the acceptance proxy, whereas CAT uses target confidence, which is more stable throughout training and needs no tuning. For a controlled comparison, we replace CAT inside PARD-2 with each of these methods while keeping the architecture, training data, and 3k-step training budget identical, with Qwen3-8B as the target model:
Rebuttal-Table 2 shows that CAT achieves the highest acceptance length and throughput on every benchmark. On average, its AL is about 10% higher than LK and VSD, and 3.6% higher than D-PACE, without requiring any additional hyperparameter. We will cite all three and discuss them in Section 3.2 and Related Work in the revised version.
[1] Xiandong Zou, Jianshu Li, Jing Huang, and Pan Zhou. Variational speculative decoding: Rethinking draft training from token likelihood to sequence acceptance. In ICML, 2026.
[2] Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026.
[3] Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. LK losses: Direct acceptance rate optimization for speculative decoding. In ICML, 2026.