FACT: Fixed-Budget Allocation and Cross-Step Training for Speculative Decoding

Speculative decoding accelerates language-model generation by verifying multi- ple draft tokens in a single target-model pass. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation. To examine this gap, we present FACT (Fixed-Budget Allocation and Cross-Step Training), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency. Using a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding, we first examine how training translates into accepted to- kens. Backpropagating through predicted-feature rollouts with greater weight on early steps improves acceptance by 15–24% over uniformly weighted detached training, yet average rollout proxy cross-entropy does not reliably rank drafters by acceptance. Beyond prediction quality, the placement of draft candidates also mat- ters: with a fixed 256-node budget and matched per-depth widths, replacing static allocation with confidence-guided allocation increases speedup over autoregres- sive decoding from 1.46× to 2.02× on HumanEval and from 1.31× to 1.75× on MT-Bench. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improve- ments do not necessarily reinforce each other. Even these speed gains depend on execution conditions: changing only host-thread settings can nearly eliminate speedup without changing acceptance. Together, these findings show why train- ing and allocation must be evaluated jointly using both acceptance and end-to-end latency under controlled execution conditions.