acceptodds
Under review as a conference paper at ICLR 2027

Directly Maximizing On-policy Expected Accepted Length for Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference by using a lightweight draft model to propose tokens for verification by a target model. Draft training commonly optimizes token-level distribution alignment, but this does not directly optimize the length of the accepted prefix: once a draft token is rejected, all subsequent draft tokens are discarded. We propose FReDraft, an on-policy training method that directly maximizes the expected accepted length of drafts sampled from the current draft model. This objective accounts for first-rejection truncation and induces prefix-aware credit assignment, weighting each token by its contribution to the acceptance of the current and subsequent tokens. To optimize the objective efficiently, we analytically marginalize token-level acceptance rate, using the target model's vocabulary distribution to reduce gradient variance. We further redistribute part of this distribution-level gradient into an explicit penalty for sampled tokens whose probabilities are overestimated by the draft model, while preserving the expected gradient. Experiments across multiple benchmarks and target models show that FReDraft improves accepted length and decoding speedup over strong baselines, achieving faster on-policy training convergence. Code is available at https://anonymous.4open.science/r/Code-FReDraft-A78E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Draft Model, Accepted Length

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qs6KOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FReDraft, an on-policy training method that directly maximizes the expected accepted length of drafts sampled from the current draft model. By analyzing the gradients of the expected accepted length objective, the authors introduce two techniques: Token-level Expected Acceptance Rate Marginalization and Over-sampled Token Penalty. These methods aim to reduce gradient variance and improve the efficiency of on-policy training, respectively. Experiments compare FReDraft with recent methods, including D-PACE, LK Loss, and Draft-OPD, demonstrating competitive performance in terms of accepted length.

Critical strengths

1 Insightful gradient analysis. By analyzing the on-policy sampling process, the authors identify an important issue: tokens that are frequently sampled by the draft model may provide no direct gradient signal, leading to inefficient training. To address this issue, they propose an over-sampled token penalty that improves gradient utilization without changing the expected optimization direction. I find the theoretical analysis of the gradient particularly insightful.

2 Comprehensive ablation studies. The paper provides relatively thorough ablation studies that evaluate the contributions of individual components, helping to demonstrate the effectiveness of the proposed techniques.

Critical weaknesses

1 Missing comparison with E2E TV Loss [1]. The experimental evaluation lacks a comparison with E2E TV Loss, which also aims to optimize the expected accepted length. Including this baseline would provide a more comprehensive evaluation of the proposed method.

2 Fairness of comparison with Draft-OPD. It would also be valuable to evaluate Draft-OPD combined with E2E TV Loss. Such a comparison would help clarify whether the improvements of FReDraft come from its proposed gradient optimization techniques or from the choice of training objective, making the experimental evaluation more complete and fair.

[1] Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling

Questions for authors

Please refer to the questions raised in the Weaknesses section.

Reviewer_pAXUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an on-policy method FReDraft, which maximizes the expected length of accepted prefixes by training the drafter. On the one hand, the method proposes on-policy training to address the error-correctable nature of the drafter; on the other hand, it proposes directly using the expected acceptance length as the optimization objective. Specifically, the method derives training weights that reflect the contribution of each token to both the current and subsequent accepted tokens, and reduces sampling noise by marginalizing the token-level acceptance rates. Additionally, they reallocate a portion of the TVD gradients to directly correct those sampled tokens whose probabilities were overestimated by the draft model. Experiments conducted using the DFlash draft generator on the Qwen3 and Llama target datasets demonstrate that, compared to the evaluated baseline methods, this approach achieves consistent improvements in both accepted length and decoding speed.

Critical strengths
  1. The research motivation is clear and closely tied to the actual decoding process. The paper capitalizes on the characteristic of speculative decoding where “the first rejection makes all subsequent drafts invalid,” proposing to directly optimize the expected acceptance length, and also provides a basis for training weights at different positions.
  2. The author analyzes the sampling noise that arises during direct optimization, as well as the issue that overestimated tokens lack direct sampling correction signals. To improve training, the author introduces acceptance rate marginalization in the form of TVD and redistributes a portion of the dense gradients to the sampled overestimated tokens.
  3. The method achieves improvements across mathematical, coding, and conversational tasks, as well as on multiple target models. On the main Qwen3 setup, the average decoding speed is further improved by approximately 7%–10% compared to Draft-OPD. Tests on SGLang also indicate that the increase in input length translates into throughput gains during deployment.
Critical weaknesses
  1. The sources of the performance gains relative to existing methods remain unclear. AL optimization, TVD, and on-policy training all have established foundations. The main comparison between this paper and LK Losses involves changes to the data distribution, optimization objective, and gradient estimation method. Although component ablation provides support, the lack of a direct comparison with LK Losses under the same on-policy workflow means that the additional benefits of the continuous AL objective relative to the existing objectives have not yet been fully demonstrated. In terms of novelty, while each individual contribution has precedents, the main innovations of this paper lie in the weight derivation and gradient estimator design under the continuous acceptance length objective.
  2. The training experiments for FReDraft focused on the DFlash architecture, with EAGLE-3 serving only as a baseline comparison, other advanced drafter models, such as DSpark and/or DFlash2, were not included. Although the appendix includes the Llama target model, the target sizes remain concentrated in the 3–8B range. The current results are insufficient to determine whether this method is equally effective on other draft structures or larger target models.
  3. The theoretical derivation addresses each gradient term separately. It does not directly imply a decrease in the variance of the total gradient during actual updates. The appendix describes gradient statistics by position, but the update of shared parameters involves the sum of gradients at each position and the cross-position covariance.
Questions for authors
  1. Is it possible to compare LK Losses and FReDraft under the same on-policy data collection, initialization, and training budget to more directly evaluate the additional benefits of continuously accepting length targets? Please also specify the exact loss version used for the LK baseline.
  2. Please describe the complete initialization and subsequent training processes for each method, particularly whether FReDraft starts from the original 100K SFT checkpoint or from a version that has undergone an additional 16K SFT. How were the checkpoints for each baseline selected? Could you provide performance metrics for the original DFlash checkpoint, as well as the number of training steps, token counts, and total computational budget for each method?
Reviewer_wmpLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FReDraft, a way to train a speculative drafter to produce longer accepted prefixes. It optimizes expected accepted length on drafts sampled from the current model, accounting for the fact that verification stops at the first rejection. The authors derive the gradient of this objective and make it easier to estimate through token marginalization and an expectation-preserving correction. Experiments with DFlash drafters show improvements over Draft-OPD across math, coding, and conversational tasks, with additional results under different serving and model settings.

Critical strengths
  1. I like the connection between the objective and the proposed training method. The paper follows through from accepted length on the drafter's own sampled paths to a usable gradient estimator. The marginalization step is particularly useful: the ablations show that the direct Monte Carlo version is difficult to train well, and that this step makes a substantial difference.

  2. The speed improvements in Table 2 are convincing enough to be practically interesting. The mean speedup improves by about 7–10% over Draft-OPD across the four model/temperature settings, and the gains are consistent across the evaluated tasks.

Critical weaknesses

The relationship to Acceptance-Aware Draft Model Training for Speculative Decoding could be clearer. WTV already accounts for first-rejection truncation through products of acceptance probabilities. The distinction appears to be the use of target-generated reference prefixes versus paths sampled from the current drafter, together with how the gradient is estimated. This is meaningful, but Appendix A.2 could explain it more precisely. A comparison under the same settings would be useful, though I recognize that the work is concurrent.

Questions for authors
  1. Is the distinction from WTV above accurate? Could you clarify it in the paper and, if feasible, compare the two objectives using the same drafter, data, and training budget? This would help isolate the value of optimizing on the drafter's own paths.

  2. How much do the results vary across training seeds and repeated timing runs? In particular, do the smaller serving gains and the improvement over beta=0 hold consistently in a representative setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.