acceptodds
Under review as a conference paper at ICLR 2027

PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding

Abstract

Speculative decoding accelerates Large Language Model (LLM) inference by using a draft model to propose candidate tokens that are verified in parallel by the target model. However, existing draft model training objectives are not aligned with the inference-time goal of maximizing consecutive token acceptance. In this paper, we build upon PARD to propose PARD-2, a dual-mode speculative decoding framework. First, we introduce Confidence-Adaptive Token (CAT) optimization, which reweights each token by the cumulative target-model confidence over its prefix, shifting the training focus from per-token accuracy to the acceptance length. Second, through stochastic gating of target hidden features, PARD-2 enables a single draft model to support both target-dependent and target-independent modes, serving an entire model family without retraining. Experiments on Llama3 and Qwen3 show that PARD-2 achieves up to 6.94 lossless acceleration in target-dependent mode and up to 5.71 in target-independent mode.

Primary area:
generative models
Keywords:
Speculative Decoding, LLM Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yASVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PARD-2, which supports both target-dependent and target-independent drafting through stochastic gating of target model hidden features. The authors also propose CAT, which dynamically reweights the training loss for each token to improve the accepted length.

Critical strengths

Flexible support for both target-dependent and target-independent drafting. The dual-mode design enables PARD-2 to serve different target models without additional training, while also benefiting from target-specific features when available. The experimental results in Table 6 demonstrate that stochastic gating introduces only a relatively small performance degradation compared to training with a single mode.

Critical weaknesses

1 Limited evaluation of target-independent generalization. Although PARD-2 does not require hidden features from the target model in its target-independent mode, it appears to still require the draft and target models to share the same tokenizer. Furthermore, the authors only evaluate the target-independent capability across different model sizes within the Qwen3 family. This limits the demonstrated applicability of the target-independent mode. Evaluating PARD-2 across different model families would provide stronger evidence of its generalization capability.

2 Limited novelty of the CAT method. The proposed CAT method uses the target model's confidence as a proxy for token acceptance probability to optimize the accepted length. However, numerous existing studies have explored directly optimizing acceptance rates or expected accepted length in speculative decoding. The proposed approach appears closely related to these methods, but the paper lacks sufficient discussion and citations of relevant prior work [1, 2, 3]. A more detailed comparison would help clarify the novelty and contributions of CAT.

[1] Xiandong Zou, Jianshu Li, Jing Huang, and Pan Zhou. Variational speculative decoding: Rethinking draft training from token likelihood to sequence acceptance. In ICML, 2026.

[2] Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026.

[3] Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. LK losses: Direct acceptance rate optimization for speculative decoding. In ICML, 2026.

Questions for authors

Could the authors evaluate whether the same PARD-2 draft model can generalize across different target model families? In particular, it would be valuable to include additional experiments on the Llama family to further demonstrate the generalization capability of the target-independent mode.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yASV

W1 & Q1: The target-independent mode still requires the draft and target models to share the same tokenizer. Could the authors evaluate whether the same PARD-2 draft model can generalize across different target model families?

A: Thanks for the good question. It is correct that TI mode requires the drafter and target to share a tokenizer. This is a requirement of standard speculative decoding in general, since verification compares draft and target tokens directly. In comparison, target-dependent methods such as EAGLE-3 and DFlash are more restrictive, requiring a dedicated drafter for every individual target, whereas PARD-2 requires only one drafter per tokenizer, which covers all sizes and versions within a family.

To address the reviewer's concern, we extend the target-independent (TI) evaluation to a second model family. We evaluate a single TI PARD-2 drafter, trained with Llama-3.1-8B, on three models from the Llama-3 family:

Rebuttal-Table 1: PARD-2 across the Llama-3 model family in target-independent mode. AL: average acceptance length.

Target modelHumanEval ALGSM8K ALMT-Bench ALMATH-500 ALAverage AL
Llama-3.2-3B-Instruct7.835.583.686.865.99
Llama-3.1-8B-Instruct7.765.713.676.175.83
Llama-3.3-70B-Instruct7.866.273.486.045.91

Rebuttal-Table 1 shows that the average AL varies by only 2.7% across the three targets. Together with the Qwen3 results (8B–32B, Main paper-Table 2), this shows that, within each of two different model families, a single TI drafter generalizes across models. We will add these results to the revised version.

W2: The proposed approach appears closely related to existing methods that directly optimize acceptance rates or expected accepted length, but the paper lacks discussion and citations of this prior work.

A: We thank the reviewer for the comment. To address the concern, we clarify how CAT differs from [1, 2, 3] and compare with them directly under a controlled setting. LK losses [3] replace the per-token KL objective with a direct acceptance-rate objective, changing how each token is supervised rather than how much it is weighted. VSD [1] optimizes path acceptance for tree-based drafting via EM with MCMC sampling, whereas CAT targets single-chain parallel drafting, which can be applied to high-concurrency serving in vLLM, with weights computed directly from target probabilities without any sampling. D-PACE [2] uses the drafter's own confidence as the acceptance proxy, whereas CAT uses target confidence, which is more stable throughout training and needs no tuning. For a controlled comparison, we replace CAT inside PARD-2 with each of these methods while keeping the architecture, training data, and 3k-step training budget identical, with Qwen3-8B as the target model:

Rebuttal-Table 2: Comparison of training objectives within PARD-2 on Qwen3-8B. Each cell is TPS / AL.

MethodHumanEvalGSM8KMT-BenchMATH-500Average
PARD-2 w/ LK [3]284 / 5.42274 / 5.19154 / 2.86281 / 5.34248 / 4.70
PARD-2 w/ VSD [1]286 / 5.46275 / 5.19154 / 2.86279 / 5.31248 / 4.71
PARD-2 w/ D-PACE [2]302 / 5.78295 / 5.59161 / 2.99292 / 5.55262 / 4.98
PARD-2 w/ CAT (ours)317 / 6.10300 / 5.70163 / 3.04303 / 5.78271 / 5.16

Rebuttal-Table 2 shows that CAT achieves the highest acceptance length and throughput on every benchmark. On average, its AL is about 10% higher than LK and VSD, and 3.6% higher than D-PACE, without requiring any additional hyperparameter. We will cite all three and discuss them in Section 3.2 and Related Work in the revised version.

[1] Xiandong Zou, Jianshu Li, Jing Huang, and Pan Zhou. Variational speculative decoding: Rethinking draft training from token likelihood to sequence acceptance. In ICML, 2026.

[2] Tianyu Wu, Yu Yao, Zhenting Qi, Han Zheng, Zhuohan Wang, Haoran Ma, Lawrence Liao, Himabindu Lakkaraju, Ju Li, and Yilun Du. D-PACE: Dynamic position-aware cross-entropy for parallel speculative drafting. arXiv preprint arXiv:2605.18810, 2026.

[3] Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. LK losses: Direct acceptance rate optimization for speculative decoding. In ICML, 2026.

Thank you for the detailed response and the additional experiments.

I have one follow-up question regarding Rebuttal-Table 2: At what sampling temperature were these experiments conducted? Could the authors also compared the average acceptance length of CAT and the other training objectives under different sampling temperatures?

We thank the reviewer for the follow-up question.

The results in Rebuttal-Table 2 were obtained with greedy decoding (T = 0). We further evaluate the same four checkpoints under stochastic sampling at T = 0.5 and T = 1:

Rebuttal-Table 3: Training objectives under stochastic sampling (T = 0.5 and T = 1). Target Model: Qwen3-8B. Each cell is TPS / AL.

MethodTHumanEvalGSM8KMT-BenchMATH-500Average
PARD-2 w/ LK [3]0.5272.09 / 5.28262.66 / 5.10147.42 / 2.81263.46 / 5.08236.41 / 4.56
PARD-2 w/ VSD [1]0.5275.00 / 5.32266.68 / 5.16151.42 / 2.85266.05 / 5.13239.79 / 4.61
PARD-2 w/ D-PACE [2]0.5292.06 / 5.67279.97 / 5.43155.34 / 2.93276.30 / 5.36250.92 / 4.85
PARD-2 w/ CAT (ours)0.5308.14 / 5.98290.07 / 5.64158.56 / 2.99284.24 / 5.51260.25 / 5.03
PARD-2 w/ LK [3]1261.38 / 5.03252.94 / 4.80147.32 / 2.74246.52 / 4.67227.04 / 4.31
PARD-2 w/ VSD [1]1261.30 / 5.03251.80 / 4.84145.00 / 2.73247.02 / 4.72226.28 / 4.33
PARD-2 w/ D-PACE [2]1269.63 / 5.20268.23 / 5.15150.91 / 2.85252.97 / 4.85235.44 / 4.51
PARD-2 w/ CAT (ours)1293.17 / 5.69274.17 / 5.29154.41 / 2.91260.54 / 5.00245.57 / 4.72

Rebuttal-Table 3 shows that CAT achieves the highest acceptance length and throughput on every benchmark at both temperatures. As expected, the average AL of all methods decreases as the temperature increases, but CAT's advantage is preserved: its average AL is 5.16 / 5.03 / 4.72 at T = 0 / 0.5 / 1, compared with 4.98 / 4.85 / 4.51 for D-PACE, 4.71 / 4.61 / 4.33 for VSD, and 4.70 / 4.56 / 4.31 for LK. Across the three temperatures, CAT is consistently 9–10% higher than LK and VSD and 3.6–4.7% higher than D-PACE.

Thank you for the additional comments. I will raise my score to 3.

Reviewer_4k51OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper extends PARD, a parallel draft model, with three components: Confidence-Adaptive Token (CAT) weighting, which weights each position's loss by the target model's cumulative confidence on the preceding ground-truth tokens; knowledge distillation with conditioning on hidden features from four target layers; and stochastic feature gating, which exposes the drafter to both feature-conditioned and feature-free inputs so that one checkpoint can run with or without target hidden states.

With greedy decoding (T=0) in vLLM on A100 GPUs, tree decoding disabled for all methods, and HumanEval, GSM8K, MATH-500, and MT-Bench as benchmarks, the target-dependent mode reaches average speedups of 5.81x on Qwen3-8B (vs. 4.61x for DFlash and 4.39x for PARD), 5.81x on Qwen3-14B, and 5.19x on Llama3.1-8B, with up to 6.94x on a single task. In the mode that uses no target hidden features, a single Qwen3 drafter accelerates 8B, 14B, and 32B targets without retraining, with average speedups of 4.68x-4.82x across the evaluated targets.

Critical strengths

The dual-mode design is practically useful and comes at little cost. One checkpoint supports both feature-conditioned and feature-free inference, gating at rho = 0.1 costs less than 1% in the feature-conditioned mode, and the feature-free drafter is reused across Qwen3 target sizes without hidden-feature access or retraining.

Retraining PARD on the expanded data mixture yields a 4.61x average on Qwen3-8B, compared with 5.81x for PARD-2 , and the data analysis attributes 15%-27% of the accepted-length improvement to the expanded mixture. This control establishes that the method itself, rather than the training corpus, accounts for most of the improvement.

Critical weaknesses

The interpretation of CAT is overstated. Speculative acceptance depends on both the target and draft distributions, whereas CAT's weights depend only on target probabilities along corpus tokens rather than along draft-generated prefixes. In addition, CAT's isolated benefit is modest: in Table 5 it raises average speedup from 4.23x for the best fixed decay to 4.36x, without reported variation. CAT should therefore be presented as an empirically motivated heuristic rather than as a principled approximation of expected accepted length.

All results are greedy, without seed variation. All methods are compared in single-chain mode with tree decoding disabled and with the best inference draft length chosen per method. Because parallel prediction and family-level reuse already exist in PARD, the incremental contribution rests largely on CAT and the dual-mode combination.

Questions for authors
  1. How were the acceptance labels in Figure 2 constructed, and does the confidence-acceptance relationship remain calibrated across draft checkpoints and targets? How does CAT compare with weighting based on measured acceptance or on both target and draft probabilities?
  2. At matched data, KD, training budget, and inference draft length, what are the separate gains from CAT and from gating in both inference modes, with training-seed variation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4k51 (Part 1/ 2)

W1 & Q2: CAT's weights depend only on target probabilities along corpus tokens. Under matched settings, what are the separate gains from CAT and gating in both inference modes, with training-seed variation?

A: Thanks for the question. We agree that acceptance depends on both distributions, but acceptance length also depends on whether the prefix is accepted: once a preceding token is rejected, a position contributes nothing, however accurate its prediction. Decomposing the expected acceptance length as E[L] = Σ_k s_k·a_k (Main paper Eq. 6–8) shows that each position's contribution is scaled by s_k, the probability that its entire prefix is accepted. CAT therefore targets this prefix factor. s_k can be approximated in several ways; we use target confidence as an empirical proxy, motivated by Main paper-Figure 2(b). CAT thus serves as a target-confidence-based, training-stable surrogate for s_k, the probability that a position's prefix survives verification. This makes training more efficient, concentrating capacity on tokens likely to contribute to acceptance rather than on distant positions that are rarely reached.

Main paper-Tables 5 and 6 provide initial ablations of token-weighting strategies and the gating ratio, respectively. To further clarify the separate contributions of CAT and gating, we provide an additional controlled comparison with results across two training seeds. We train three drafter variants against Qwen3-8B: neither CAT nor gating, CAT only, and gating only (ρ = 0.1). All variants use identical data, CE/KD settings, and a 10,000-step training budget, with inference draft length fixed at 15. Each variant is trained with two seeds (42 and 43).

We evaluate two inference modes: target-dependent (TD), where the draft model receives target-model hidden features, and target-independent (TI), where it generates drafts without these features. Entries are TD / TI acceptance length, reported as mean (± sample standard deviation) across training seeds. Average equally weights the three benchmarks. Values are rounded to two decimal places; an SD of 0.00 denotes an unrounded SD below 0.005.

Rebuttal-Table 1: Separate contributions of CAT and gating. Entries are TD / TI acceptance length, mean (± SD) over two training seeds.

Training variantHumanEval TD / TIGSM8K TD / TIMATH-500 TD / TIAverage TD / TI
Neither CAT nor gating6.08 (±0.04) / 4.97 (±0.04)5.64 (±0.03) / 4.17 (±0.03)6.16 (±0.04) / 4.69 (±0.02)5.96 (±0.01) / 4.61 (±0.02)
CAT only6.61 (±0.03) / 5.25 (±0.01)6.12 (±0.02) / 4.45 (±0.00)6.53 (±0.11) / 4.87 (±0.05)6.42 (±0.03) / 4.86 (±0.01)
Gating only6.07 (±0.03) / 5.59 (±0.05)5.59 (±0.05) / 4.91 (±0.01)6.13 (±0.16) / 5.33 (±0.05)5.93 (±0.03) / 5.27 (±0.00)

Rebuttal-Table 1 shows generally small variation across training seeds, and the gains from CAT and gating are consistent across all three benchmarks and both seeds.

CAT. CAT improves average acceptance length by 7.7% in TD mode (5.96 → 6.42) and by 5.4% in TI mode (4.61 → 4.86), with gains on every benchmark in both modes. The TD improvement of 0.46 is more than ten times the seed standard deviation (≤ 0.03).

Gating. Without gating, target features are present throughout training but unavailable during target-independent inference, creating a training–inference mismatch. Gating addresses this mismatch by exposing the draft model to training examples without target features. It improves average acceptance length in TI mode by 14.45%, with only a 0.43% decrease in TD mode (percentages are computed from unrounded values).

AuthorsOpenReview ↗

Response to Reviewer 4k51 (Part 2/ 2)

W2: All results are greedy without seed variation, and all methods use single-chain decoding with the best draft length chosen per method.

A: To address the reviewer's concern, we extend our evaluation from greedy decoding to stochastic sampling (T = 1) on Qwen3-8B. Each cell is TPS / AL.

Rebuttal-Table 2: Comparison under stochastic sampling (T = 1). Target: Qwen3-8B. Each cell is TPS / AL, reported as mean (± SD) over two sampling seeds (44 and 45).

MethodTHumanEvalGSM8KMT-BenchMATH-500Average
Vanilla172.94 (±0.08) / 1.0072.95 (±0.06) / 1.0073.21 (±0.04) / 1.0073.22 (±0.06) / 1.0073.08 (±0.06) / 1.00
PARD1338.60 (±4.00) / 6.20 (±0.09)322.87 (±0.12) / 5.82 (±0.02)179.84 (±0.25) / 3.16 (±0.01)309.02 (±0.10) / 5.61 (±0.01)287.58 (±1.06) / 5.20 (±0.03)
DFlash1357.62 (±4.87) / 6.92 (±0.10)283.98 (±0.46) / 5.37 (±0.00)167.11 (±2.06) / 3.09 (±0.05)309.01 (±0.76) / 5.90 (±0.01)279.43 (±1.01) / 5.32 (±0.01)
PARD-21380.32 (±3.51) / 7.55 (±0.06)347.05 (±0.41) / 6.76 (±0.00)189.40 (±1.06) / 3.61 (±0.02)340.98 (±8.60) / 6.71 (±0.17)314.44 (±1.64) / 6.16 (±0.03)

As shown in Rebuttal-Table 2, at T = 1, PARD-2 achieves an average speedup of 4.30× and AL of 6.16, versus 3.94× / 5.20 for PARD and 3.82× / 5.32 for DFlash, i.e., 9.3% and 12.5% higher throughput and 18.5% and 15.8% higher AL, respectively. It remains the best method on every benchmark in both throughput and AL, with gaps well above the variation across sampling seeds, so the conclusions drawn under greedy decoding carry over to sampling.

Evaluation. All evaluations are conducted end-to-end in vLLM to reflect real serving conditions. Our target setting is high-concurrency serving (Main paper-Figure 1, batch sizes up to 64), where tree decoding adds substantial compute and memory overhead and can increase latency, so all methods are evaluated in single-chain mode for consistency. The optimal draft length differs across methods and serving conditions (e.g., low vs. high concurrency), so fixing a single length would favor some methods over others. We therefore report each method at its own best draft length, which is the most favorable setting for every baseline.

Q1: How were the acceptance labels in Figure 2 constructed, and does the relationship hold across draft checkpoints and targets? How does CAT compare with weighting based on measured acceptance or draft probabilities?

A: Thanks for the good question. The accept rate is the conditional acceptance rate, i.e., the probability that the token at draft position k is accepted given that all preceding draft tokens were accepted.

To address the reviewer's concern about whether this relationship holds for other targets, we repeat the analysis with Qwen3-8B as the target on MATH-500. Binning output tokens by their single-token prediction score and measuring the empirical acceptance rate within each bin gives:

Rebuttal-Table 3: Calibration of the prediction score. Target: Qwen3-8B, benchmark: MATH-500.

BinMean prediction scoreEmpirical acceptance rate
10.2580.364
20.3580.390
30.4550.480
40.5520.638
50.6470.665
60.7490.757
70.8540.871
80.9930.965

Rebuttal-Table 3 shows that empirical acceptance rate increases strictly monotonically with the prediction score, with no inversion at any bin, reproducing on a second model family the behaviour reported for Llama-3.1-8B in Main paper-Figure 2(b). The proxy is therefore not an artefact of one target or one tokenizer.

To compare CAT with weighting based on measured acceptance, draft probabilities, or both target and draft probabilities, we replace CAT's weighting signal inside PARD-2 with each alternative while keeping the architecture, training data, and training budget (3k steps) identical. As shown in Rebuttal-Table 4, CAT achieves the highest acceptance length and throughput on every benchmark. This is consistent with our motivation: target confidence is stable throughout the entire training process, which keeps supervision more concentrated on the tokens that contribute to acceptance.

Rebuttal-Table 4: Comparison of weighting signals within PARD-2 on Qwen3-8B. Each cell is TPS / AL.

Weighting signalHumanEvalGSM8KMT-BenchMATH-500Average
Draft probabilities302 / 5.78295 / 5.59161 / 2.99292 / 5.55262 / 4.98
Target + draft probabilities303 / 5.82292 / 5.55160 / 2.98297 / 5.61263 / 4.99
Measured acceptance307 / 5.91295 / 5.60162 / 3.01291 / 5.55264 / 5.02
Target probabilities (CAT, ours)317 / 6.10300 / 5.70163 / 3.04303 / 5.78271 / 5.16
Reviewer_Xu9zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PARD-2, a dual-mode speculative decoding framework designed to address two key limitations in existing parallel draft models: (1) suboptimal training objectives that uniformly supervise all token positions rather than aligning with the goal of maximizing consecutive token acceptance lengths, and (2) the trade-off between peak speedup and reusability (i.e., target-dependent drafters require specific hidden states, while target-independent ones sacrifice acceleration).To tackle these, the authors propose two main techniques:Confidence-Adaptive Token (CAT) Optimization: A reweighting strategy based on the cumulative target-model confidence over a token's prefix, which shifts the training focus from per-token accuracy to expected acceptance length.Dual-Mode Inference via Stochastic Gating: By stochastically gating target hidden feature injection during training, a single draft model can operate in either a target-dependent mode (for maximum acceleration) or a target-independent mode (as a plug-and-play accelerator for an entire model family).Validation: Experiments conducted on models such as Llama3.1-8B and Qwen3-8B (including vLLM deployment tests) show that PARD-2 achieves up to 6.94×6.94\times lossless speedup in target-dependent mode, outperforms baseline parallel methods, and demonstrates robust high-concurrency throughput.

Critical strengths

Principled Alignment with Acceptance Length (CAT): Standard parallel drafters uniformly weight all predicted positions, failing to account for the fact that distant draft tokens are rarely reached if prefix tokens fail. The proposed CAT optimization is a well-motivated, elegant solution that mathematically aligns training objectives with the true inference-time metric (expected consecutive acceptance length).

Elegant Dual-Mode Flexibility: The introduction of stochastic feature gating successfully bridges the gap between target-dependent drafters (which offer high speedup but lack cross-model portability) and target-independent drafters (which are flexible but less accurate). Achieving both behaviors within a single model architecture without structural overhead is a notable engineering and algorithmic contribution.

Critical weaknesses

Analysis of Training Stability with Stochastic Gating: While stochastic gating enables dual-mode operation, the paper could benefit from a deeper discussion on whether dropping/gating target features during training causes optimization conflicts or slows down convergence compared to training dedicated single-mode models.

Generalization Across Model Families: The evaluation focuses heavily on Llama and Qwen model families. It remains slightly unclear how the target-independent mode scales or performs when applied to architectures with vastly different hidden dimensions or attention mechanisms outside of these families.

Questions for authors

In CAT optimization, how sensitive is the training stability to the choice of the confidence threshold or the scaling of the cumulative product when the prefix length grows very large (e.g., in long-context scenarios)?

During stochastic gating, what is the exact probability ratio used for training the target-dependent versus target-independent paths, and does tuning this ratio affect the peak performance of the target-dependent mode?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Xu9z

Q1: Does gating target features during training cause optimization conflicts or slow down convergence compared to dedicated single-mode models?

A: We thank the reviewer for the question. Gating is applied per sequence with a small ratio (ρ = 0.1), and since target features enter as an additive term on the input embedding, omitting them reduces the input to the original PARD setting, so the two cases are nested rather than competing objectives. We additionally trained drafters with and without gating under the same training budget, with Qwen3-8B as the target and two training seeds:

Rebuttal-Table 1: Effect of gating under the same training budget. Target: Qwen3-8B. Average acceptance length over HumanEval, GSM8K, and MATH-500, mean (± SD) over two training seeds.

Training variantTarget-dependentTarget-independent
Without gating5.96 (±0.01)4.61 (±0.02)
With gating (ρ = 0.1)5.93 (±0.03)5.27 (±0.00)

Rebuttal-Table 1 shows that the gated model matches the dedicated target-dependent model in acceptance length (within seed variation), while substantially improving target-independent acceptance length. We will add this discussion to the revised version.

Q2: Generalization Across Model Families: how the target-independent mode scales or performs when applied to architectures with vastly different hidden dimensions or attention mechanisms.

A: Thanks for the question. Speculative decoding itself requires the draft and target models to share a tokenizer, since verification compares their tokens directly; this is a general requirement of speculative decoding. Within this requirement, the target-independent (TI) mode does not depend on the target's internal architecture by construction: the drafter never consumes target hidden states, so the target's architecture, attention mechanism, and hidden dimension do not affect the drafter at all.

To address the reviewer's concern, we evaluate the Qwen3 PARD-2 drafter in TI mode on Qwen3-Next-80B-A3B-Instruct, without any retraining. The architecture of Qwen3-Next differs from Qwen3-8B: it uses a hybrid attention mechanism (Gated DeltaNet linear attention in three of every four layers, instead of full attention in every layer), a smaller hidden dimension (2048 vs. 4096), and a high-sparsity MoE (80B total parameters, about 3B active per token) instead of a dense feed-forward network.

Rebuttal-Table 2: PARD-2 in target-independent mode on Qwen3-Next-80B-A3B. AL: average acceptance length.

MethodHumanEval ALGSM8K ALMT-Bench ALMATH-500 ALAverage AL
PARD-2 (TI)5.434.692.625.144.47

Rebuttal-Table 2 shows that the drafter transfers to Qwen3-Next without any retraining, despite the different attention mechanism, hidden dimension, and MoE structure, achieving an average acceptance length of 4.47. This indicates that the target-independent mode generalizes across different attention mechanisms, hidden dimensions, and even model architectures. We will add these results to the revised version.

Q3: In CAT optimization, how sensitive is the training stability to the choice of the confidence threshold or the scaling of the cumulative product when the prefix length grows very large?

A: CAT has no threshold: ŝ_k is the plain cumulative product of target confidences (Main paper-Eq. 10), and thus requires no tuning, unlike fixed decay, which falls below uniform supervision at γ = 0.6 (Main paper-Table 5). Moreover, the product runs only over preceding positions within the draft window, so the number of multiplicative factors is bounded by the draft window size (at most K − 1 = 15) rather than the context length; increasing the context length only affects the conditioning of each ĉ_j and does not directly increase the number of factors. We will clarify in the revision.

Q4: During stochastic gating, what is the exact probability ratio used for training the target-dependent versus target-independent paths, and does tuning this ratio affect the peak performance of the target-dependent mode?

A: We thank the reviewer for the question. We use ρ = 0.1 for all models (Main paper-Table 7): each training sequence is trained with target features (target-dependent) with probability 0.9 and without them (target-independent) with probability 0.1, i.e., a 9:1 ratio. Tuning this ratio has only a minor effect on peak target-dependent performance (Main paper-Table 6). We adopt ρ = 0.1, as it preserves target-dependent performance almost entirely while still enabling the target-independent mode.

Reviewer_Xu9zOpenReview ↗

I have received the authors' rebuttal.

I have received the authors' rebuttal.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback. During the rebuttal period, we conducted additional experiments and provided detailed clarifications addressing the major concerns. We summarize our key findings below:

  • Effectiveness of CAT: Additional ablation studies demonstrate that CAT independently improves acceptance length by 7.7% and 5.4% in target-dependent and target-independent modes, respectively. We also empirically demonstrate a monotonic relationship between target confidence and token acceptance probability. In addition, we discuss LK losses, VSD, and D-PACE, and provide controlled comparisons within PARD-2 against these objectives and against alternative weighting signals (draft probabilities, target + draft probabilities, and measured acceptance) under matched architecture, training data, and training budget. Overall, CAT consistently improves acceptance length on every benchmark and in both inference modes.
  • Dual-mode training: Experiments across two random seeds show that feature gating introduces no noticeable convergence degradation, substantially improves target-independent performance (acceptance length: 4.61 → 5.27), and maintains comparable target-dependent performance (5.96 → 5.93).
  • Generalization: Without retraining, PARD-2 successfully generalizes to Qwen3-Next-80B-A3B, despite substantial architectural differences. A single target-independent drafter also achieves consistent acceptance lengths across Llama models ranging from 3B to 70B.
  • Non-greedy decoding: Under temperature T=1, PARD-2 achieves a 4.30× average speedup, compared with 3.94× for PARD and 3.82× for DFlash (averaged over two sampling seeds), demonstrating its effectiveness beyond greedy decoding.

These additional results further support our two main contributions: target-aligned parallel draft training through CAT and a unified draft model supporting both target-dependent and target-independent speculative decoding.

We believe the additional evidence addresses the major concerns raised by the reviewers and further strengthens the technical contributions and practical applicability of PARD-2. We will incorporate these results and clarifications into the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.