acceptodds
Under review as a conference paper at ICLR 2027

FACT: Fixed-Budget Allocation and Cross-Step Training for Speculative Decoding

Abstract

Speculative decoding accelerates language-model generation by verifying multi- ple draft tokens in a single target-model pass. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation. To examine this gap, we present FACT (Fixed-Budget Allocation and Cross-Step Training), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency. Using a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding, we first examine how training translates into accepted to- kens. Backpropagating through predicted-feature rollouts with greater weight on early steps improves acceptance by 15–24% over uniformly weighted detached training, yet average rollout proxy cross-entropy does not reliably rank drafters by acceptance. Beyond prediction quality, the placement of draft candidates also mat- ters: with a fixed 256-node budget and matched per-depth widths, replacing static allocation with confidence-guided allocation increases speedup over autoregres- sive decoding from 1.46× to 2.02× on HumanEval and from 1.31× to 1.75× on MT-Bench. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improve- ments do not necessarily reinforce each other. Even these speed gains depend on execution conditions: changing only host-thread settings can nearly eliminate speedup without changing acceptance. Together, these findings show why train- ing and allocation must be evaluated jointly using both acceptance and end-to-end latency under controlled execution conditions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative decoding, large language models, inference acceleration, dynamic draft trees, fixed-budget allocation, multi-step rollout training, latency evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UJ8cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies speculative decoding through a controlled empirical framework called FACT, focusing on three factors: (i) drafter training, (ii) candidate allocation under a fixed depth-wise budget, and (iii) execution conditions. The setup uses a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding.

Critical strengths

The empirical study is relatively controlled and transparent compared with many systems papers in this space. I appreciated that the paper repeatedly states what is held fixed, what is not, and what conclusions the data can support. The transfer experiment to the official EAGLE drafter is also a worthwhile check, since it shows the allocation effect is not restricted to one custom-trained checkpoint.

Critical weaknesses
  1. The paper’s novelty is limited, and the main positive results come from recombining or re-evaluating known ideas rather than introducing a clearly new algorithmic insight. The training ingredients are largely drawn from prior work (rollout exposure, decayed step weighting, cross-step gradient flow), and the allocation side is also explicitly a restricted variant of prior dynamic tree selection ideas. I do not think the resulting empirical insight is strong enough to compensate for the limited methodological novelty.
  2. Several central conclusions are not cleanly attributed because the comparisons confound multiple factors or use mismatched checkpoints/settings. The RQ1 result compares “uniform detached rollout” against “step-decayed full-gradient rollout,” which changes both weighting and gradient flow at once. The paper acknowledges this, but the conclusion is still presented prominently. Likewise, the stronger drafter used in the allocation section is trained for 20k steps and is not the same checkpoint family as in RQ1, making it hard to connect the training and allocation narratives into a single coherent claim. In the end, the paper argues for joint evaluation, but the study itself does not always isolate the interactions cleanly enough to support strong takeaways.
Questions for authors
  1. The main RQ1 result does not isolate step weighting from cross-step gradient flow. Please clarify what conclusion is supported beyond the combined comparison.
  2. Please report a default-setting cost breakdown for static vs. dynamic decoding, including tree construction/selection overhead, to support the claim that dynamic allocation improves latency under fixed budgets.
  3. The study is restricted to greedy decoding, one target model, and one hardware platform. Please clarify the scope of the claimed conclusions and which statements should be read as setting-specific.
  4. Please clarify what exactly is concluded from the proxy-loss ranking failure, given that the proxy is based on an intermediate-feature readout rather than the full target distribution.
Reviewer_ZQHfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents FACT, an empirical study of the factors that determine the end-to-end speedup of speculative decoding. Its main findings are: (1) full-gradient rollout training with step-decayed weights (γ=0.5\gamma = 0.5) achieves the highest acceptance, but this ordering is not reflected in multi-step proxy cross-entropy; (2) confidence-guided allocation under fixed depth-wise budgets improves speedup, but a stronger drafter does not consistently gain more absolute latency savings from it; (3) acceptance gains do not directly translate into speedup, since execution overhead such as tree construction and target verification can erase them. All experiments use a frozen Llama-3-8B-Instruct target with an EAGLE-style drafter under greedy decoding at batch size 1.

Critical strengths
  • The paper identifies and evaluates how training, allocation and execution settings affect the end-to-end acceleration of speculative decoding.
  • The evaluation is carefully organized, with paired prompt-level bootstrap intervals, two training seeds for RQ1, and numerical fidelity evaluated separately from speed.
  • The paper finds that a stronger drafter does not necessarily benefit more from dynamic allocation; dynamic trees are sensitive to the host-thread setting while static trees are not; and step-decayed full-gradient rollout takes about twice the training time of detached rollout (27.0 h vs. 14.1 h) for a 15–24% gain in acceptance, and about 11 times that of teacher forcing (2.4 h).
Critical weaknesses
  • The novelty is limited, and prior work that established the central message is not cited. The paper's central message, that acceptance gains do not reliably translate into speedup and that execution cost must be measured, was already established by Liu et al. [1] and earlier by Yan et al. [2]. Yan et al. show that speedup depends mainly on drafter latency rather than its language-modeling quality. Liu et al. show in vLLM that target verification takes 42–95% of runtime (Sec. 5.1), that higher acceptance can still yield lower speedup when the draft-to-target cost ratio is higher (Sec. 3.2), and give an analytic speedup model in terms of acceptance rate, proposal length and cost ratio (Sec. 4.1). FACT's RQ3 and its discussion in Section 5 restate this message in a narrower setting without citing or contrasting with either work.
  • Key comparisons are confounded. RQ1 matches optimizer steps but not training compute (2.4 h vs. 14.1 h vs. 27.0 h, Appendix D), and changes step weighting and gradient flow together. The RQ2 checkpoint (20k steps) differs from the RQ1 checkpoints (10k steps), so the joint effect of training and allocation claimed in the title and abstract is never directly measured.
  • The scope is too narrow for an empirical study. Besides being limited to batch size 1, all conclusions come from a single 8B target, one GPU, greedy decoding, a 128-token output cap, 50 prompts per benchmark, and only EAGLE-style drafters.

[1] Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, and Alvin Cheung. Speculative Decoding: Performance or Illusion? In Proceedings of Machine Learning and Systems (MLSys), 2026. arXiv:2601.11580.

[2] Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. Decoding Speculative Decoding. In Proceedings of NAACL, 2025.

Questions for authors
  • Liu et al. [1] (Sec. 3.2) report that a draft tree verifying 21 tokens (branching factor 4) gives only a narrow benefit over a 3-token chain at batch size 1 and falls below 1x speedup on all evaluated workloads at batch size 64. How do the 256-node static and dynamic trees behave at batch sizes 8, 32 and 64? Does the latency saving of dynamic allocation persist, and does either tree still outperform autoregressive decoding?
  • With only about 1% of the 256 verified candidates accepted per step, how does the 256-node budget compare against a chain or a small tree (e.g., 3–21 nodes), at batch size 1 and at larger batch sizes?
  • What is the root cause of the slowdown at high thread caps, and why does the target tree-forward time more than double when the computation runs on the GPU?
Reviewer_Ks3FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FACT studies how drafter training, candidate allocation, and execution settings affect speculative decoding. It compares several training objectives, tests static and confidence-guided allocation with the same number of candidates at each depth, and measures the effect of host-thread settings. The main message is that better prediction loss or higher acceptance does not always mean faster generation. Experiments use a frozen Llama-3-8B-Instruct target.

Critical strengths

I appreciate that the paper measures actual latency rather than only acceptance. Keeping candidate counts fixed at each depth makes the allocation comparison more informative. Testing the allocation method with an independently trained official EAGLE drafter is also useful. The paper is quite clear about its limitations. It distinguishes matched training steps from matched compute, and does not claim that the small quality probes establish quality equivalence.

Critical weaknesses

The paper is presented as a controlled empirical study. This is a reasonable direction, but the current findings seem limited. Dynamic draft trees and training with predicted features are already studied in EAGLE-2 and HASS, as the authors also acknowledge. Fixing the number of candidates at each depth makes the allocation comparison useful, but it does not by itself establish a strong new contribution. I would expect either a more informative finding about these existing methods or broader evidence that the findings hold beyond this implementation.

The training results are difficult to separate. The main comparison changes both step weighting and gradient flow. Appendix D adds a uniform full-gradient control, but it uses one seed and an earlier prompt set, and there is still no decayed-weight detached control. So we cannot tell whether cross-step backpropagation is needed for the improvement. The compute difference is also substantial: the reported training times are 2.4 hours for teacher forcing, 14.1 hours for detached rollout, and 27.0 hours for the step-weighted full-gradient configuration. Matching optimizer steps does not answer which method makes better use of the training budget. Also, the scaled checkpoint in the allocation experiments is trained for 20,000 steps, while the main training comparisons use 10,000-step checkpoints. This leaves the connection between the training improvement and the final latency improvement unclear.

The loss-ranking result also depends on a particular proxy. The readout is applied to features before the target’s last decoder layer, so it does not give the full target distribution. The ranking failure may partly reflect this mismatch. The authors acknowledge this limitation, but without a comparison using the full target distribution, it is hard to draw a broader lesson about selecting drafters by prediction loss.

Another issue is the interpretation of the training–allocation interaction. The paper measures this as a difference in absolute latency savings, in milliseconds per token. A stronger drafter may save fewer milliseconds from dynamic allocation while still receiving a larger relative improvement, because its starting latency is already lower. The paper notes that the interaction can differ on a speedup-ratio scale, but the abstract gives the broader message that stronger training does not consistently amplify allocation benefits. I think this conclusion should be stated specifically for absolute latency savings. Reporting both absolute and relative effects would make the result easier to interpret.

The evaluation is also narrow for drawing general lessons: one 8B target, one H100 platform, greedy decoding, a 128-token output cap, and 50 test prompts per benchmark. Testing the official EAGLE drafter in the shared implementation is useful, but it is not a comparison against the complete native EAGLE-2 system. The thread experiment shows a real performance problem in this implementation, although it remains unclear how much of the finding carries over to other decoding systems. Longer outputs, sampling, batching, and another target or implementation would help establish the scope of the conclusions

Questions for authors
  1. Does decayed weighting still improve acceptance when rollout features are detached? How do the training methods compare under the same compute budget?

  2. Does the loss-ranking failure remain when the proxy readout is replaced with the full target distribution?

  3. Which findings remain consistent with another target model, longer outputs, and batched decoding? How does the system compare with native EAGLE-2 under matched serving conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.