acceptodds
Under review as a conference paper at ICLR 2027

Carryover Drafting: Recycling Rejected States for Speculative Decoding

Abstract

Speculative decoding accelerates LLM inference by verifying multiple drafted tokens in parallel, allowing a single target forward pass to generate several tokens. By construction, verification computes representations for both accepted and rejected tokens, but conventional drafters retain only the representations of accepted tokens, leaving the substantial verifier computation spent on rejected tokens effectively wasted. We show that these discarded hidden states retain useful information about future tokens that can improve subsequent drafts. However, realizing this opportunity poses two distinct challenges. At inference, reusing rejected states introduces additional projection and attention costs, which can increase drafting latency and offset the speedup gained from higher acceptance. During training, standard parallel training does not expose the drafter to rejected states encountered at inference, while generating such states through sequential rollouts would sacrifice parallelism across training positions. We introduce Carryover Drafting, which enables efficient reuse of rejected states and parallel inference-aligned training. Carryover recycles rejected target hidden states as temporary KV context for the next draft, allowing the drafter to selectively attend to the rejected states through its attention layers. A single learned embedding distinguishes rejected states from committed context, while the carried-over KV context is replaced each drafting round to bound the additional context by one proposal block. We further introduce parallel draft–verify–draft training to expose the drafter to inference-aligned rejected states without sacrificing parallelism across training positions. Across DFlash and a DSpark-derived semi-autoregressive drafter, Carryover improves average acceptance length by up to 14.7% and end-to-end serving speedup by up to 14.4% over the corresponding baselines, with gains reaching 28.8% on translation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Speculative Decoding, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H373OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous work on speculative decoding (e.g., DFlash, DSpark) uses only the target model's hidden state from the accepted tokens and therefore "wastes" the hidden activations from the rejected tokens. In this work, the authors explore how the activations from rejected tokens can be used by the drafter as additional KV context and provide a draft-verify-draft method, enabling learning in parallel and preventing sequential rollouts. The authors show that on Qwen3-4B and Gemma 4 12B (using two different drafters, DFlash and Markov), acceptance length increases (6.5–14.7%) and end-to-end speedup follows (7.9–14.4%). The gain is especially visible in translation, with a speedup of up to 28.8%. As the mechanism explaining this improvement, they argue for the exploitation of re-convergence ("predictions from the rejected trajectory match the eventual continuation").

Critical strengths
  1. The results show a consistent improvement using Carryover Drafting over the baseline (using only the accepted token information) across 64 dataset/model/drafter/temperature combinations. The method is simple to add to drafters, while having a low computational overhead as shown end-to-end in vLLM. The considerable speedup in translation is an interesting result that justifies further exploration in this region, i.e., why information about rejected states is so helpful.

  2. The number of ablations is considerable and sufficient. The authors study what information needs to be shared about the rejected tokens, how to use it in the drafter, a matched-anchor recall analysis, as well as the attention behaviour of the transformers. The insights based on these experiments justify the claim that information from rejected tokens is useful.

Critical weaknesses
  1. A critical weakness of this work is that the study of the statistical relevance of the results seems to be completely missing. The numbers from the experiments seem to be based on single runs, rather than on multiple training seeds. While the results on speedup from Table 1 seem credible based on the consistency over multiple settings, this is a problem for the explanatory re-convergence experiment. Their Pearson correlation values for re-convergence correlating with acceptance gain are computed only over 8 datasets, which, for example, would result in big 95% confidence intervals using the Fisher z-transform, making the differences likely statistically meaningless. A bigger problem there is that their analysis runs into the ecological fallacy. Their claim (that rejected states are useful when they re-converge) is about (individual) rounds, where the evidence is given over dataset (group) averages. Robinson (1950, "Ecological correlations and the behavior of individuals") has shown examples where this can even switch the sign of the Pearson correlation. Therefore, a round-based analysis is needed. A similar problem do we have for the statement "It also suggests that stronger drafters leave more useful rejected states, so the benefit of Carryover can grow as drafters improve". This is extrapolated from 2 drafters, and therefore also not statistically meaningful.

  2. The implementation and justification of their method is unclear. In Table 3 (b), all 3 explored variants have a max 0.1 \tau difference from each other. So the justification for using the embedding-based Carryover Drafting is unclear. If simplification is the goal, we could use no distinction, as the information from a rejected token is already given in the position encoding of the tokens. If gain in acceptance is the goal, a separate projector is better. A third decision criterion could be the resulting speedup. Here, the embedding-based Carryover Drafting could be the best, but we have no data on the separate projector's speedup performance. Therefore, it is unclear why the authors decided on embedding-based Carryover Drafting. In general, the implementation of some components (e.g., the separate projector or the training data generation temperature) is unclear, and the code is not provided.

Questions for authors
  1. What is the model architecture for separate projectors? Are they the same projectors, simply with different parameters, or a different architecture?

  2. What is the speedup of separate projectors compared to the embedding version?

  3. What is the uncertainty for Figure 3, Table 3, and Figure 5 (accounting for the ecological fallacy)?

  4. What is the performance for Figure 4 for the OFF curve when it is not negatively biased? Currently, the other two curves (ON and baseline) are evaluated for position j if all previous tokens coincide with the tokens predicted from the target. But the OFF ablations' predictions need not coincide. Therefore, the predicted tokens 0 to j-1 can be different for the OFF method (in contrast to the ON and baseline) from the target predictions, and therefore be more prone to resulting in a wrong prediction for the j-th token?

  5. What temperature was used for generating the training responses?

7 comments
AuthorsOpenReview ↗

Response to Reviewer H373

Response to Reviewer H373 (1/3)

We thank the reviewer for the detailed methodological questions. With new measurements, we address:

  • Different training seeds and repeated evaluation runs
  • Round-level analysis of re-convergence
  • Design choice among the Carryover variants, including the separate projector
  • Uncertainty for Figure 3 and Table 3
  • OFF curve without negative bias in Figure 4
  • Temperature for training-response sampling

Unless stated otherwise, τ is the mean number of accepted tokens per verification, and brackets are 95% confidence intervals.


W1. “The numbers ... seem to be based on single runs ... [and] the ecological fallacy ... a round-based analysis is needed.” / “[The statement on stronger drafters] is extrapolated from 2 drafters, and therefore also not statistically meaningful.”

A. The reviewer is right on all of these points. We added a second training seed, ten evaluation runs, and a round-level analysis.

  • Different Training Seeds for Robustness: A second training seed reproduces the gain.

Acceptance-length gain with two training seeds, Carryover 7+3 epochs vs. Baseline 10 epochs, Qwen3-4B / Markov

Training seedT=0T=1
First1.130 [1.124, 1.137]1.124 [1.111, 1.137]
Second1.129 [1.122, 1.136]1.125 [1.112, 1.138]

Training a drafter on the full 500K-response training set is expensive, so we could afford two training runs within the rebuttal period.

  • Repeated Evaluation Runs for Robustness: The gain is stable across ten evaluation runs.

Acceptance-length gain over ten evaluation runs, paper checkpoints, Qwen3-4B (mean ± std)

DrafterT=0T=1
DFlash1.093 ± 0.0001.065 ± 0.005
Markov1.117 ± 0.0001.106 ± 0.011

At T=0 decoding is deterministic, so all ten runs are identical. Per-dataset values are in our Q3 response.

  • Round-Level Analysis: On individual rounds, re-convergence stays positively correlated with the gain, but more weakly than the dataset-level value suggested.

Pearson correlation with Carryover's acceptance gain at T=0 (Markov / DFlash). Dataset-level intervals use the Fisher z-transform.

StatisticOver 8 dataset meansOver individual rounds
Re-convergence0.88 [0.44, 0.98] / 0.94 [0.71, 0.99]0.17 [0.16, 0.19] / 0.20 [0.18, 0.21]
Acceptance after rejection−0.57 [−0.91, 0.23] / 0.35 [−0.47, 0.85]0.11 [0.10, 0.12] / 0.10 [0.09, 0.11]
Rejected-state count−0.08 [−0.74, 0.66] / −0.52 [−0.90, 0.29]0.04 [0.03, 0.05] / 0.00 [−0.01, 0.02]

This confirms the reviewer's concern: the dataset-level correlation has wide intervals and overstates the relation on individual rounds. The sign does not flip, and re-convergence remains the strongest of the three statistics at both levels. In ten additional T=1 evaluations, the round-level correlation is 0.159 ± 0.009, remaining positive in all seeded comparisons, and the gain grows from +0.13 accepted tokens in rounds with no re-converged state to +1.16 in rounds with five or more. We will replace the dataset-level correlation in Figure 5 and Section 5.2 with the round-level analysis, and describe it as an association, not a causal effect.

  • Statement on Stronger Drafters: We will tone this statement down.

The reviewer is right that two drafters are not enough for a statistically meaningful claim. Each additional drafter family requires a full training run, which we could not afford at this scale, so this remains an observation. We will narrow the sentence in Section 5.2 accordingly; a deeper analysis over more drafters is needed to relate drafter strength to the size of the gain.

AuthorsOpenReview ↗

Response to Reviewer H373

Response to Reviewer H373 (2/3)


W2 / Q1 / Q2. “The justification for using the embedding-based Carryover Drafting is unclear.” / “What is the model architecture for separate projectors?” / “What is the speedup of separate projectors compared to the embedding version?”

A. We agree that the three variants are close in acceptance and that the separate projector is slightly higher; our choice is based on simplicity and end-to-end speedup, which we now measured.

  • Architecture of the Separate Projector (Q1): It has the same architecture as the existing projector, with its own parameters.

It is applied only to carried states, while accepted states keep the existing projector.

  • Design Choices: Explicit source discrimination gives consistent speedup; the shared projector with embedding has the same speedup as a separate projector while staying simpler.

All variants we consider are already efficient architecture choices that do not add an expensive nonlinear network, and all use a bounded, temporary Carryover KV context.

Acceptance length τ, Table 3 probe (Qwen3-4B / Markov). T=1 is mean ± std over ten runs.

VariantT=0 τT=0 gain over BaselineT=1 τ
Baseline3.05 [2.99, 3.12]–2.481 ± 0.012
No source tag3.33 [3.26, 3.40]+9.1% [8.4, 9.8]2.640 ± 0.008
Separate projector3.39 [3.32, 3.47]+11.1% [10.4, 11.8]2.693 ± 0.009
Shared projector + embedding (ours)3.37 [3.30, 3.45]+10.4% [9.7, 11.1]2.677 ± 0.008

End-to-end speed relative to shared projector + embedding (ten runs)

VariantT=0T=1
No source tag−1.17% [−1.34, −0.99]−1.22% [−1.51, −0.93]
Separate projector+0.18% [−0.09, +0.45]+0.10% [−0.37, +0.57]

To answer Q2 directly: the separate projector is NOT better in terms of 'end to end speedup gains' than the embedding version (+0.18% and +0.10%, both within noise), while it adds a second projector (19.7M parameters against 7.7K ). The shared + embedding yields 1.2% more speedup than using no source tag. We therefore choose the embedding as the simplest design that reaches the best speedup.


Q3. “What is the uncertainty for Figure 3, Table 3, and Figure 5 (accounting for the ecological fallacy)?”

A. Table 3 is covered in the W2 response and Figure 5 in the W1 response. For Figure 3, we repeated the evaluation ten times.

  • Uncertainty for Figure 3: The gain is well above the run-to-run variation on every dataset.

Acceptance-length gain per dataset (Figure 3), Qwen3-4B. T=1 is mean ± std over ten runs; T=0 runs are identical.

DatasetDFlash T=0DFlash T=1Markov T=0Markov T=1
HumanEval1.0931.067 ± 0.0141.0971.089 ± 0.019
Math1.0931.069 ± 0.0081.0771.083 ± 0.016
MT-Bench1.0851.059 ± 0.0151.1031.095 ± 0.021
QA1.0851.059 ± 0.0181.1161.092 ± 0.033
RAG1.0831.057 ± 0.0251.1171.107 ± 0.023
Summarization1.1011.082 ± 0.0081.1421.129 ± 0.009
Tool Use1.1011.070 ± 0.0111.1391.123 ± 0.020
Translation1.2211.170 ± 0.0231.3031.292 ± 0.029

We will add these to Figure 3, and update Table 3 with the full evaluation above.

AuthorsOpenReview ↗

Response to Reviewer H373

Response to Reviewer H373 (3/3)


Q4. “What is the performance for Figure 4 for the OFF curve when it is not negatively biased?”

A. We thank the reviewer for spotting this. We re-evaluated ON, OFF and Baseline on the same correct prefix at every position.

  • OFF Curve without Negative Bias: OFF stays within about 2% of the Baseline, and ON remains above both at every draft position.

In the updated curves, ON is above OFF by +0.014 to +0.046 in recall at all 15 positions. The same Carryover weights with carried states withheld (OFF) also match the Baseline in overall acceptance length:

Carryover 7+3 with carried states OFF or ON, relative to Baseline 10 epochs

Target / drafterOFF, T=0ON, T=0OFF, T=1ON, T=1
Qwen3-4B / Markov1.003 [0.998, 1.009]1.130 [1.124, 1.137]1.004 [0.994, 1.015]1.124 [1.111, 1.137]
Qwen3-4B / DFlash1.004 [1.000, 1.008]1.104 [1.099, 1.110]0.996 [0.984, 1.007]1.071 [1.060, 1.082]
Gemma 4 12B / Markov0.999 [0.994, 1.005]1.144 [1.138, 1.151]1.010 [0.998, 1.021]1.146 [1.132, 1.159]

The improvement therefore comes from supplying the carried states. When the carried states are not fed, Carryover training neither degrades nor improves acceptance by more than about 2% in any setting (all 95% intervals lie between 0.984 and 1.021). We will update Figure 4 and Section 5.1.


Q5. “What temperature was used for generating the training responses?”

  • Temperature for Training-Response Sampling: The temperature is 0.7 for Qwen3-4B and 1.0 for Gemma 4 12B.
TargetTemperatureTop-pTop-kThinking
Qwen3-4B0.70.820Off
Gemma 4 12B1.00.9564Off

These are the sampling settings officially recommended for each target model (non-thinking mode for Qwen3), and data generation follows the public speculator data recipe cited in the paper. Each response is sampled once and used throughout all epochs, for both Baseline and Carryover training.

We will add these settings to Appendix B.1.

Reviewer_qAdgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper discusses a quite interesting question: can the rejected tokens be used to predict the draft tokens in the next round? The authors claim that tokens rejected by the target model can indeed be re-used by the draft model to make predictions. Carryover drafting projects the rejected tokens into a temporary draft KV context and subsequent draft tokens can attend to the rejected tokens in the previous turn, providing additional KV context for the prediction. Experiments were conducted on the Qwen3-4B and Gemma-4-12B model versus the existing SOTA including DFlash and DSpark. The results are quite promising it shows up to 14.4% end-to-end speedup.

Critical strengths
  1. I find the research problem is quite interesting as the use of rejected tokens is rarely explored in speculative decoding. Recycling these representations is a natural and potentially valuable direction.
  2. The analysis of re-convergence provides an interesting perspective on why rejected states may be useful. The matched-anchor experiments also provide evidence that supplying carried-over states improves subsequent draft predictions.
  3. the experiment results are quite promising as it delivers improvement in terms of both the acceptance length and end-to-end speedup.
Critical weaknesses
  1. the training process of Carryover requires more compute than the baseline. The baselines are trained for 10 epochs, whereas Carryover receives an additional 3 epochs of fine-tuning starting from the baseline checkpoint. Moreover, Carryover training introduces additional proposal generation and target verification computation. Perhaps the authors should compare the baseline and Carryover in an iso-compute setup for fairness.
  2. The proposed training procedure is entirely training-inference aligned to me. The parallel draft–verify–draft procedure constructs independent one-step transitions from recorded target responses. However, actual inference involves repeated Carryover-conditioned drafting, where both the anchor distribution and rejected-state distribution depend on previous drafting rounds
Questions for authors
  1. It is not clear to me why the confidence scheduling of Dspark is dsiabled as it should help me with DSpark's efficiency.
2 comments
AuthorsOpenReview ↗

Response to Reviewer qAdg

Response to Reviewer qAdg

We thank the reviewer for the constructive evaluation. With new experiments, we address:

  • Fair training comparison against the Baseline
  • Training–inference alignment over repeated rounds
  • Confidence scheduling with Carryover

Unless stated otherwise, τ is the mean number of accepted tokens per verification, and brackets are 95% confidence intervals.


W1. “Perhaps the authors should compare the baseline and Carryover in an iso-compute setup for fairness.”

A. We measured the training cost: one Carryover epoch costs about 3.75 Baseline epochs. Starting Carryover from epoch 7 of the Baseline run, one Carryover epoch therefore totals about 10.8 Baseline-epoch equivalents, close to the 10 epochs of the Baseline.

  • Fair Training Comparison against the Baseline: With the same number of epochs, Carryover maintains its gain; in the iso-compute setting, Carryover is already 7% ahead.

Acceptance length relative to Baseline 10 epochs, Qwen3-4B / Markov (compute in approximate Baseline-epoch equivalents)

CheckpointComputeT=0T=1
Baseline, 7 epochs7.00.993 [0.989, 0.997]1.001 [0.990, 1.012]
Baseline, 8 epochs8.00.997 [0.993, 1.000]1.007 [0.995, 1.019]
Baseline, 9 epochs9.01.000 [0.998, 1.002]1.001 [0.990, 1.011]
Baseline, 10 epochs10.01.000 (reference)1.000 (reference)
Carryover 7+110.81.073 [1.066, 1.080]1.074 [1.060, 1.089]
Carryover 7+214.51.120 [1.113, 1.127]1.117 [1.106, 1.129]
Carryover 7+318.31.130 [1.124, 1.137]1.124 [1.111, 1.137]

With the same 10 training epochs (Carryover 7+3), Carryover maintains a gain of +13.0% (T=0) and +12.4% (T=1). In the iso-compute setting (Carryover 7+1, about 10.8 Baseline-epoch equivalents), a single Carryover epoch already gives +7.3% (T=0) and +7.4% (T=1). The Baseline itself is flat over its last three epochs, so more Baseline training does not close the gap.

We also see Carryover as post-training for efficient inference: the training cost is paid once, and more post-training epochs do not add any computation per inference round.


W2. “Actual inference involves repeated Carryover-conditioned drafting, where both the anchor distribution and rejected-state distribution depend on previous drafting rounds.”

A. Draft–verify–draft training reproduces the transition that every inference round performs: verification produces rejected states, and the next draft reads them. Chaining more transitions in training may follow the recursion at inference more closely, so we tried training with two chained transitions.

  • Training–Inference Alignment: Training on two chained transitions gives essentially no change over one transition.

One vs. two draft–verify–draft transitions in training (prototype DFlash-type drafter, T=0)

TrainingOne transition τTwo transitions τChange
Carryover post-training, 2 epochs3.2373.254+0.5% [0.1%, 0.9%]

We therefore keep the single transition. We also see no decrease of the acceptance-length gain in later rounds of speculative decoding, and all results in the paper come from real multi-round inference.


Q1. “It is not clear to me why the confidence scheduling of DSpark is disabled ...”

A. We used fixed-length proposals to measure Carryover's contribution at the same verification budget for both methods. As discussed in the paper, scheduling also changes the supply of rejected representations. Still, we added evaluation runs under DSpark's standard confidence scheduler.

  • Confidence Scheduling with Carryover: Carryover keeps its throughput gain under confidence scheduling.

Throughput gain with confidence scheduling, Carryover over Baseline, Qwen3-4B / Markov (mean ± std over repeated runs)

ConcurrencyT=0T=1
11.108 ± 0.0161.123 ± 0.022
21.106 ± 0.0101.087 ± 0.009
41.102 ± 0.0061.081 ± 0.015
81.084 ± 0.0061.077 ± 0.015
161.055 ± 0.0081.048 ± 0.009

Carryover outperforms the Baseline at every concurrency from 1 to 16 and at both temperatures. A scheduler that accounts for the future value of rejected states, together with scheduling-aware Carryover training, could further improve this combination. We will add these results to our Discussion (Section 6).

AuthorsOpenReview ↗

Follow-up on our response to Reviewer qAdg

Dear Reviewer qAdg,

Thank you for the review. With the discussion period closing soon, here is a brief summary of the response:

  • Fair training comparison (W1): Carryover maintains its gain under a fair training comparison. With the same number of epochs (Baseline 7 epochs + Carryover 3 epochs vs. Baseline 10 epochs) the gain remains 13.0%, and with a training compute similar to the Baseline (one Carryover epoch) Carryover is already 7.3% ahead.
  • Training–inference alignment (W2): A single training transition is enough in our tests. Training on two chained transitions gave essentially no change, and we see no decrease of the gain in later rounds of speculative decoding.
  • Confidence scheduling (Q1): Carryover keeps its throughput gain under DSpark's confidence scheduler, at every concurrency from 1 to 16.

If any question or weakness remains unresolved, please let us know; we are willing to address it further before the deadline.

Otherwise, we would be grateful if you could take these into account in your final assessment.

Reviewer_p7DNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Conventional speculative decoding often discards the rejected tokens and hidden states, while this paper claims that the rejected hidden states can be used to improve the subsequent draft. The proposed Carryover Drafting includes an inference strategy and a corresponding training pipeline. The implementation is relatively easy: it requires an additional learned embedding.

Critical strengths
  1. The idea is well-motivated. It's clever to reuse verifier computation that has already been paid for and add only one embedding vector.
  2. Consistent gains. Improvements appear on every dataset, for both targets, both drafter families, and both temperatures. They persist from concurrency 1 to 32 in real vLLM serving.
  3. Reasonable training construction. Parallel draft–verify–draft obtains roughly inference-aligned rejected states while keeping anchor-level parallelism through block-diagonal masking.
Critical weaknesses
  1. The experiment setting is not fair enough. The baseline methods are trained for 10 epochs, but the proposed Carryover is trained for 3 more epochs. A fair comparison should train the baselines for 3 more epochs, or start Carryover from the 7th epoch.
  2. The evaluation scope is limited in 2 perspectives: (1) The output cap is only 1024/2048 tokens. In real cases, responses are usually longer, and it's unclear whether Carryover maintains superiority in long-context situations. (2) The inference uses linear draft instead of tree draft. As tree draft is a powerful and easy speedup setting for speculative decoding, it's a big loss that Carryover doesn't test this setting.
  3. Questionable evaluation results. Firstly, I notice that several speedup gains exceed the corresponding acceptance gains despite the added overhead: Qwen DFlash HumanEval at T=1 (τ\tau +7.4% vs speedup +14.0%), Qwen Markov MT-Bench at T=1 (+10.8% vs +18.9%), and Qwen DFlash Avg at T=1 (+6.5% vs +8.6%). Secondly, it lacks repeated training runs to reduce the effect of randomness.
Questions for authors
  1. In tree-draft setting, how does Carryover decide which hidden state to use?
  2. Does Carryover work on EAGLE-style autoregressive draft methods? I ask this because your paper does not specify the parallel drafting case, so I would expect to see how it works with AR drafters.
3 comments
AuthorsOpenReview ↗

Response to Reviewer p7DN

Response to Reviewer p7DN (1/2)

We thank the reviewer for the constructive suggestions. With new experiments, we address:

  • Equal training epochs against the Baseline
  • Evaluation with long outputs
  • Tree decoding with Carryover
  • Speedup versus acceptance gain
  • Different training seeds and repeated evaluation runs
  • Autoregressive drafters (EAGLE-3) with Carryover

Unless stated otherwise, τ is the mean number of accepted tokens per verification, and brackets are 95% confidence intervals.


W1. “A fair comparison should train the baselines for 3 more epochs, or start Carryover from the 7th epoch.”

A. We ran the second option. Carryover now starts from epoch 7 of the same Baseline run and trains for 3 epochs, so both methods have 10 training epochs.

  • Equal Training Epochs against the Baseline: Carryover retains its acceptance-length gain over a Baseline trained for the same number of epochs.

Acceptance-length gain, Carryover 7+3 epochs vs. Baseline 10 epochs

Target / drafterT=0T=1
Qwen3-4B / Markov1.130 [1.124, 1.137]1.124 [1.111, 1.137]
Qwen3-4B / Markov, second training seed1.129 [1.122, 1.136]1.125 [1.112, 1.138]
Qwen3-4B / DFlash1.104 [1.099, 1.110]1.071 [1.060, 1.082]
Gemma 4 12B / Markov1.144 [1.138, 1.151]1.146 [1.132, 1.159]

Each value is the τ ratio averaged over the full eight datasets of Table 1 in the paper. Even with the same number of training epochs for both, Carryover keeps a gain of 10–14% at T=0 and 7–15% at T=1. We believe this resolves the concern about unequal training, and we will add these rows to Table 1.


W2 (1). “The output cap is only 1024/2048 tokens ... unclear whether Carryover maintains superiority in long-context situations.”

A. We extended generation to 5,120 tokens, and additionally ran thinking mode with up to 7,000 tokens on 140 math and AIME prompts.

  • Evaluation with Long Outputs: The gain holds throughout long outputs, including tokens 4k–7k.

Acceptance-length gain at longer outputs, Carryover 7+3 epochs vs. Baseline 10 epochs

Output segmentT=0T=1
Regular evaluation, up to 5,120 tokens1.129 [1.122, 1.137]1.123 [1.110, 1.136]
Thinking mode, up to 7,000 tokens1.110 [1.099, 1.122]1.099 [1.088, 1.110]
Thinking mode, tokens 2k–4k1.099 [1.079, 1.118]1.083
Thinking mode, tokens 4k–7k1.130 [1.090, 1.177]1.105 [1.074, 1.137]

The gain in the latest segment (tokens 4k–7k) is as large as the gain over the whole output, so Carryover does not lose its advantage as generation gets longer.


W2 (2) / Q1. “The inference uses linear draft instead of tree draft.” / “In tree-draft setting, how does Carryover decide which hidden state to use?”

A. We evaluated tree drafting with the same node budget for Baseline and Carryover, without any tree-specific retraining.

  • Tree Decoding with Carryover: Carryover increases acceptance length at every tree size, from 16 to 60 nodes.

Acceptance length τ with tree drafting, Qwen3-4B / DSpark (Markov) with PCTree [1], Baseline 10 epochs → Carryover 7+3 epochs

Tree nodesT=0T=0 gain in τT=1T=1 gain in τ
164.997 → 5.393+0.396 [0.369, 0.424]4.581 → 4.934+0.353 [0.330, 0.376]
325.429 → 5.873+0.444 [0.416, 0.471]4.991 → 5.371+0.381 [0.353, 0.408]
485.624 → 6.094+0.470 [0.440, 0.501]5.174 → 5.582+0.408 [0.380, 0.436]
605.718 → 6.195+0.476 [0.447, 0.507]5.261 → 5.675+0.413 [0.384, 0.443]

Which hidden states are carried (Q1): we keep the rejected branch closest to the accepted path, that is, the highest-scoring rejected child of the last accepted node together with its subtree. The other rejected branches are dropped.

Even this simple rule improves every tree size, and the gain grows with the tree. As noted in our Discussion, a tree produces more rejected states than a linear draft, so selecting or combining several branches is a promising next step.

[1] Li et al., From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding, arXiv:2608.02123, 2026.

AuthorsOpenReview ↗

Response to Reviewer p7DN

Response to Reviewer p7DN (2/2)


W3 (1). “Several speedup gains exceed the corresponding acceptance gains despite the added overhead.”

A. The cells the reviewer pointed out are all at T=1, where Baseline and Carryover sample different texts with different lengths. We repeated the timing ten times.

  • Speedup versus Acceptance Gain: Over ten runs, the speedup follows the acceptance gain; single T=1 runs deviate because sampled output lengths differ.

Carryover over Baseline, paper checkpoints, mean ± std over ten runs

SettingEnd-to-end speedupτ gainOutput length, Baseline / Carryover
Markov, T=0, all datasets+11.0 ± 0.2%+11.7 ± 0.0%1.000 ± 0.000
Markov, T=1, all datasets+9.9 ± 2.0%+10.6 ± 1.1%1.000 ± 0.015
DFlash, T=0, all datasets+8.9 ± 0.2%+9.3 ± 0.0%1.000 ± 0.000
DFlash, T=1, all datasets+6.3 ± 2.3%+6.5 ± 0.5%0.998 ± 0.012
Markov, MT-Bench, T=1+12.6 ± 5.5%+9.5 ± 2.1%1.031 ± 0.047
DFlash, HumanEval, T=1+7.5 ± 2.8%+6.7 ± 1.4%1.010 ± 0.021

On the same prompts, speedup is the product of the τ gain, the output-length ratio, and the per-round time ratio. At T=0 the outputs are identical, so the speedup is always slightly below the τ gain, as expected from the overhead. At T=1 the output-length ratio on a single dataset moves by 2–5% between runs, which is enough to put the speedup above the τ gain in a single run; on MT-Bench, Carryover's sampled outputs were also about 3% shorter on average. The values in the paper were single timing runs, and we will replace them with the ten-run means.


W3 (2). “[The evaluation] lacks repeated training runs.”

  • Different Training Seeds and Repeated Evaluation Runs: The gain is reproduced with a second training seed and across ten evaluation runs.

Acceptance-length gain with two training seeds, Carryover 7+3 epochs vs. Baseline 10 epochs, Qwen3-4B / Markov

Training seedT=0T=1
First1.130 [1.124, 1.137]1.124 [1.111, 1.137]
Second1.129 [1.122, 1.136]1.125 [1.112, 1.138]

Training a drafter on the full 500K-response training set is expensive, so we could afford two training runs within the rebuttal period. Over ten evaluation runs of the paper checkpoints, the T=1 τ ratio is 1.106 ± 0.011 for Markov and 1.065 ± 0.005 for DFlash.


Q2. “Does Carryover work on EAGLE-style autoregressive draft methods?”

A. Yes. We trained an EAGLE-3 drafter in the same way and compared it with its 10-epoch Baseline.

  • Autoregressive Drafter (EAGLE-3) with Carryover: Carryover improves an autoregressive drafter, beyond parallel (DFlash) and semi-autoregressive (Markov) drafters.

Acceptance-length gain, Carryover 7+3 epochs vs. Baseline 10 epochs (Qwen3-4B)

ArchitectureT=0T=1
EAGLE-3 (autoregressive)1.069 [1.064, 1.073]1.068 [1.059, 1.078]
DFlash (parallel)1.104 [1.099, 1.110]1.071 [1.060, 1.082]
Markov (semi-autoregressive)1.130 [1.124, 1.137]1.124 [1.111, 1.137]

All three architectures benefit, so reusing rejected target states is not specific to parallel block prediction. We will add these results to Section 6.

AuthorsOpenReview ↗

Follow-up on our response to Reviewer p7DN

Dear Reviewer p7DN,,

Your suggestions led to several new experiments, and we wanted to make sure the results reached you before the discussion closes. In short:

  • Fair training comparison (W1): Carryover maintains its gain under a fair training comparison. With Baseline 7 epochs + Carryover 3 epochs vs. Baseline 10 epochs, the gain remains 10–14% at T=0 and 7–15% at T=1.
  • Long outputs (W2): The Carryover gain holds throughout long outputs of up to 7,000 tokens, including tokens 4k–7k.
  • Tree decoding (W2, Q1): Carryover increases acceptance length at every tree size from 16 to 60 nodes, showing that it is applicable to tree drafting without any tree-specific training.
  • Speedup vs. acceptance gain (W3): The cases where speedup exceeded the acceptance gain come from different output lengths under temperature sampling. Over ten runs, the speedup follows the acceptance gain.
  • Repeated runs (W3): A second training seed and ten evaluation runs reproduce the gain.
  • Autoregressive drafter, EAGLE-3 (Q2): Carryover also improves an autoregressive drafter, so it is not limited to parallel or semi-autoregressive drafters.

Is there any remaining concern we can still address? If not, we would appreciate it if you could consider updating your assessment.

Comments on the paper

1 comment

General Response

Despite the strengths, some concerns remained open during the author–reviewer discussion.

Reviewer H373 confirmed the revision addresses their concerns and raised their rating.

We haven't yet heard from Reviewers p7DN and qAdg, and we remain ready to clarify anything further.

Strengths

  • Well-motivated and clever idea : reuses verifier computation already paid for and wasted; rarely explored, a natural and potentially valuable direction (p7DN, qAdg)

  • Simple and low-overhead : one added embedding (p7DN, H373)

  • Reasonable training construction : inference-aligned rejected states while keeping parallel training (p7DN)

  • Consistent gains : in acceptance length and end-to-end speedup (p7DN, qAdg, H373)


Pointed Weakness

  • Fair Training Compared to Baseline
Carryover 7+3 vs. Baseline 10 epochsAcceptance-length gain (T=0)
Qwen3-4B / Markov+13.0%
Qwen3-4B / DFlash+10.4%
Gemma 4 12B / Markov+14.4%

With equal training epochs, as Reviewer p7DN asked, the gain remains above +10%.

With equal training compute, as Reviewer qAdg asked, one Carryover epoch (7+1) already gives +7.3%.

Carryover is post-training in the spirit of inference acceleration: the cost is paid once, and the inference cost stays intact.


Requested Extensions

  • Tree decoding: Compatible. +7.9% to +8.4% at 16 to 60 nodes, with no tree-specific training (p7DN)
  • Confidence scheduling: Compatible. +5.5% to +10.8% throughput, with no scheduling-aware training (qAdg)
  • Autoregressive drafter (EAGLE-3): Compatible. +6.9% (p7DN)
  • Long outputs: Compatible. +13.0% in tokens 4k–7k (p7DN)

Tree decoding and confidence scheduling were already addressed in the Discussion (Section 6), where we expected that the best compatibility would need Carryover-aware optimization and left it as promising future work.

We tested both with no additional training (Appendix L and Appendix N), and Carryover already improves each of them without any optimization.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.