- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Previous work on speculative decoding (e.g., DFlash, DSpark) uses only the target model's hidden state from the accepted tokens and therefore "wastes" the hidden activations from the rejected tokens. In this work, the authors explore how the activations from rejected tokens can be used by the drafter as additional KV context and provide a draft-verify-draft method, enabling learning in parallel and preventing sequential rollouts. The authors show that on Qwen3-4B and Gemma 4 12B (using two different drafters, DFlash and Markov), acceptance length increases (6.5–14.7%) and end-to-end speedup follows (7.9–14.4%). The gain is especially visible in translation, with a speedup of up to 28.8%. As the mechanism explaining this improvement, they argue for the exploitation of re-convergence ("predictions from the rejected trajectory match the eventual continuation").
Critical strengths
-
The results show a consistent improvement using Carryover Drafting over the baseline (using only the accepted token information) across 64 dataset/model/drafter/temperature combinations. The method is simple to add to drafters, while having a low computational overhead as shown end-to-end in vLLM. The considerable speedup in translation is an interesting result that justifies further exploration in this region, i.e., why information about rejected states is so helpful.
-
The number of ablations is considerable and sufficient. The authors study what information needs to be shared about the rejected tokens, how to use it in the drafter, a matched-anchor recall analysis, as well as the attention behaviour of the transformers. The insights based on these experiments justify the claim that information from rejected tokens is useful.
Critical weaknesses
-
A critical weakness of this work is that the study of the statistical relevance of the results seems to be completely missing. The numbers from the experiments seem to be based on single runs, rather than on multiple training seeds. While the results on speedup from Table 1 seem credible based on the consistency over multiple settings, this is a problem for the explanatory re-convergence experiment. Their Pearson correlation values for re-convergence correlating with acceptance gain are computed only over 8 datasets, which, for example, would result in big 95% confidence intervals using the Fisher z-transform, making the differences likely statistically meaningless. A bigger problem there is that their analysis runs into the ecological fallacy. Their claim (that rejected states are useful when they re-converge) is about (individual) rounds, where the evidence is given over dataset (group) averages. Robinson (1950, "Ecological correlations and the behavior of individuals") has shown examples where this can even switch the sign of the Pearson correlation. Therefore, a round-based analysis is needed. A similar problem do we have for the statement "It also suggests that stronger drafters leave more useful rejected states, so the benefit of Carryover can grow as drafters improve". This is extrapolated from 2 drafters, and therefore also not statistically meaningful.
-
The implementation and justification of their method is unclear. In Table 3 (b), all 3 explored variants have a max 0.1 \tau difference from each other. So the justification for using the embedding-based Carryover Drafting is unclear. If simplification is the goal, we could use no distinction, as the information from a rejected token is already given in the position encoding of the tokens. If gain in acceptance is the goal, a separate projector is better. A third decision criterion could be the resulting speedup. Here, the embedding-based Carryover Drafting could be the best, but we have no data on the separate projector's speedup performance. Therefore, it is unclear why the authors decided on embedding-based Carryover Drafting. In general, the implementation of some components (e.g., the separate projector or the training data generation temperature) is unclear, and the code is not provided.
Questions for authors
-
What is the model architecture for separate projectors? Are they the same projectors, simply with different parameters, or a different architecture?
-
What is the speedup of separate projectors compared to the embedding version?
-
What is the uncertainty for Figure 3, Table 3, and Figure 5 (accounting for the ecological fallacy)?
-
What is the performance for Figure 4 for the OFF curve when it is not negatively biased? Currently, the other two curves (ON and baseline) are evaluated for position j if all previous tokens coincide with the tokens predicted from the target. But the OFF ablations' predictions need not coincide. Therefore, the predicted tokens 0 to j-1 can be different for the OFF method (in contrast to the ON and baseline) from the target predictions, and therefore be more prone to resulting in a wrong prediction for the j-th token?
-
What temperature was used for generating the training responses?
7 comments
Response to Reviewer H373
Response to Reviewer H373 (2/3)
W2 / Q1 / Q2. “The justification for using the embedding-based Carryover Drafting is unclear.” / “What is the model architecture for separate projectors?” / “What is the speedup of separate projectors compared to the embedding version?”
A. We agree that the three variants are close in acceptance and that the separate projector is slightly higher; our choice is based on simplicity and end-to-end speedup, which we now measured.
- Architecture of the Separate Projector (Q1): It has the same architecture as the existing projector, with its own parameters.
It is applied only to carried states, while accepted states keep the existing projector.
- Design Choices: Explicit source discrimination gives consistent speedup; the shared projector with embedding has the same speedup as a separate projector while staying simpler.
All variants we consider are already efficient architecture choices that do not add an expensive nonlinear network, and all use a bounded, temporary Carryover KV context.
Acceptance length τ, Table 3 probe (Qwen3-4B / Markov). T=1 is mean ± std over ten runs.
| Variant | T=0 τ | T=0 gain over Baseline | T=1 τ |
|---|---|---|---|
| Baseline | 3.05 [2.99, 3.12] | – | 2.481 ± 0.012 |
| No source tag | 3.33 [3.26, 3.40] | +9.1% [8.4, 9.8] | 2.640 ± 0.008 |
| Separate projector | 3.39 [3.32, 3.47] | +11.1% [10.4, 11.8] | 2.693 ± 0.009 |
| Shared projector + embedding (ours) | 3.37 [3.30, 3.45] | +10.4% [9.7, 11.1] | 2.677 ± 0.008 |
End-to-end speed relative to shared projector + embedding (ten runs)
| Variant | T=0 | T=1 |
|---|---|---|
| No source tag | −1.17% [−1.34, −0.99] | −1.22% [−1.51, −0.93] |
| Separate projector | +0.18% [−0.09, +0.45] | +0.10% [−0.37, +0.57] |
To answer Q2 directly: the separate projector is NOT better in terms of 'end to end speedup gains' than the embedding version (+0.18% and +0.10%, both within noise), while it adds a second projector (19.7M parameters against 7.7K ). The shared + embedding yields 1.2% more speedup than using no source tag. We therefore choose the embedding as the simplest design that reaches the best speedup.
Q3. “What is the uncertainty for Figure 3, Table 3, and Figure 5 (accounting for the ecological fallacy)?”
A. Table 3 is covered in the W2 response and Figure 5 in the W1 response. For Figure 3, we repeated the evaluation ten times.
- Uncertainty for Figure 3: The gain is well above the run-to-run variation on every dataset.
Acceptance-length gain per dataset (Figure 3), Qwen3-4B. T=1 is mean ± std over ten runs; T=0 runs are identical.
| Dataset | DFlash T=0 | DFlash T=1 | Markov T=0 | Markov T=1 |
|---|---|---|---|---|
| HumanEval | 1.093 | 1.067 ± 0.014 | 1.097 | 1.089 ± 0.019 |
| Math | 1.093 | 1.069 ± 0.008 | 1.077 | 1.083 ± 0.016 |
| MT-Bench | 1.085 | 1.059 ± 0.015 | 1.103 | 1.095 ± 0.021 |
| QA | 1.085 | 1.059 ± 0.018 | 1.116 | 1.092 ± 0.033 |
| RAG | 1.083 | 1.057 ± 0.025 | 1.117 | 1.107 ± 0.023 |
| Summarization | 1.101 | 1.082 ± 0.008 | 1.142 | 1.129 ± 0.009 |
| Tool Use | 1.101 | 1.070 ± 0.011 | 1.139 | 1.123 ± 0.020 |
| Translation | 1.221 | 1.170 ± 0.023 | 1.303 | 1.292 ± 0.029 |
We will add these to Figure 3, and update Table 3 with the full evaluation above.
Response to Reviewer H373
Response to Reviewer H373 (3/3)
Q4. “What is the performance for Figure 4 for the OFF curve when it is not negatively biased?”
A. We thank the reviewer for spotting this. We re-evaluated ON, OFF and Baseline on the same correct prefix at every position.
- OFF Curve without Negative Bias: OFF stays within about 2% of the Baseline, and ON remains above both at every draft position.
In the updated curves, ON is above OFF by +0.014 to +0.046 in recall at all 15 positions. The same Carryover weights with carried states withheld (OFF) also match the Baseline in overall acceptance length:
Carryover 7+3 with carried states OFF or ON, relative to Baseline 10 epochs
| Target / drafter | OFF, T=0 | ON, T=0 | OFF, T=1 | ON, T=1 |
|---|---|---|---|---|
| Qwen3-4B / Markov | 1.003 [0.998, 1.009] | 1.130 [1.124, 1.137] | 1.004 [0.994, 1.015] | 1.124 [1.111, 1.137] |
| Qwen3-4B / DFlash | 1.004 [1.000, 1.008] | 1.104 [1.099, 1.110] | 0.996 [0.984, 1.007] | 1.071 [1.060, 1.082] |
| Gemma 4 12B / Markov | 0.999 [0.994, 1.005] | 1.144 [1.138, 1.151] | 1.010 [0.998, 1.021] | 1.146 [1.132, 1.159] |
The improvement therefore comes from supplying the carried states. When the carried states are not fed, Carryover training neither degrades nor improves acceptance by more than about 2% in any setting (all 95% intervals lie between 0.984 and 1.021). We will update Figure 4 and Section 5.1.
Q5. “What temperature was used for generating the training responses?”
- Temperature for Training-Response Sampling: The temperature is 0.7 for Qwen3-4B and 1.0 for Gemma 4 12B.
| Target | Temperature | Top-p | Top-k | Thinking |
|---|---|---|---|---|
| Qwen3-4B | 0.7 | 0.8 | 20 | Off |
| Gemma 4 12B | 1.0 | 0.95 | 64 | Off |
These are the sampling settings officially recommended for each target model (non-thinking mode for Qwen3), and data generation follows the public speculator data recipe cited in the paper. Each response is sampled once and used throughout all epochs, for both Baseline and Carryover training.
We will add these settings to Appendix B.1.
Response to Reviewer H373
Response to Reviewer H373 (1/3)
We thank the reviewer for the detailed methodological questions. With new measurements, we address:
Unless stated otherwise, τ is the mean number of accepted tokens per verification, and brackets are 95% confidence intervals.
W1. “The numbers ... seem to be based on single runs ... [and] the ecological fallacy ... a round-based analysis is needed.” / “[The statement on stronger drafters] is extrapolated from 2 drafters, and therefore also not statistically meaningful.”
A. The reviewer is right on all of these points. We added a second training seed, ten evaluation runs, and a round-level analysis.
Acceptance-length gain with two training seeds, Carryover 7+3 epochs vs. Baseline 10 epochs, Qwen3-4B / Markov
Training a drafter on the full 500K-response training set is expensive, so we could afford two training runs within the rebuttal period.
Acceptance-length gain over ten evaluation runs, paper checkpoints, Qwen3-4B (mean ± std)
At T=0 decoding is deterministic, so all ten runs are identical. Per-dataset values are in our Q3 response.
Pearson correlation with Carryover's acceptance gain at T=0 (Markov / DFlash). Dataset-level intervals use the Fisher z-transform.
This confirms the reviewer's concern: the dataset-level correlation has wide intervals and overstates the relation on individual rounds. The sign does not flip, and re-convergence remains the strongest of the three statistics at both levels. In ten additional T=1 evaluations, the round-level correlation is 0.159 ± 0.009, remaining positive in all seeded comparisons, and the gain grows from +0.13 accepted tokens in rounds with no re-converged state to +1.16 in rounds with five or more. We will replace the dataset-level correlation in Figure 5 and Section 5.2 with the round-level analysis, and describe it as an association, not a causal effect.
The reviewer is right that two drafters are not enough for a statistically meaningful claim. Each additional drafter family requires a full training run, which we could not afford at this scale, so this remains an observation. We will narrow the sentence in Section 5.2 accordingly; a deeper analysis over more drafters is needed to relate drafter strength to the size of the gain.