- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STEP separates the recurrent state used for drafting from the representation used for token prediction. It combines this architecture with an early-stage feature-alignment loss and confidence-guided tree expansion. Across four models, five benchmarks, and two temperatures, STEP reports average improvements of 10.1% in speed and 15.1% in acceptance length over EAGLE-3, with additional gains in SGLang.
Critical strengths
- The improvements are consistent across all 40 settings. The gains also carry over to SGLang, which makes the results practically relevant.
- Table 2 compares both drafters under both tree strategies. STEP improves performance even with the original tree construction, showing that the gains do not come solely from changing the expansion strategy.
Critical weaknesses
- The ablations show benefits from architectural separation and from adding staged alignment to STEP. However, they do not include EAGLE-3 with the same staged alignment loss or STEP with a constant alignment weight. These comparisons would help establish whether the alignment benefit depends on the new architecture and whether removing the constraint late in training is useful.
- Section 4.2 argues that narrowing the prediction branch offsets the extra MLP, but does not report concrete dimensions, parameter counts, or drafting latency. The paper specifies that the alignment weight decreases from 0.1 to zero, but not the timing or decay schedule. It also follows EAGLE-3’s training-data setup without explicitly documenting whether the compared checkpoints use matched data quantities and training steps.
Questions for authors
- How does EAGLE-3 perform with the same staged alignment loss? How does STEP perform when the alignment weight remains constant?
- Could you report the model dimensions, parameter counts, and drafting latency, specify the alignment-weight schedule, and clarify whether the compared checkpoints use the same training-data quantities and update counts?
- Since Figures 3 and 4 condition on all preceding tokens being accepted, how many eligible samples remain at later positions, and what is the uncertainty of those estimates?
5 comments
Response Q1:
We sincerely thank you for suggesting these two important ablations. To address this question, we additionally trained EAGLE-3 with the same staged feature-alignment objective and STEP with a constant alignment weight of 0.1. We evaluated both variants on GSM8K and MT-Bench using LLaMA-3.1-8B-Instruct, examining conditional token acceptance, feature similarity, decoding speedup, and average acceptance length. All four variants were evaluated using the same progressive tree expansion strategy.
The decoding efficiency results are summarized in the following table.
| Method | MT-Bench Speedup | τ | GSM8K Speedup | τ |
|---|---|---|---|---|
| STEP + staged alignment | 4.51× | 7.01 | 4.53× | 7.06 |
| STEP + constant alignment | 4.29× | 6.83 | 4.32× | 6.84 |
| EAGLE-3 + staged alignment | 4.07× | 6.60 | 4.08× | 6.42 |
| EAGLE-3 (without alignment) | 4.19× | 6.76 | 4.15× | 6.69 |
1. Applying staged alignment to EAGLE-3.
As shown in Figure 5, introducing staged alignment into EAGLE-3 improves its feature similarity to the target model. However, this improvement does not translate into better overall decoding performance. On GSM8K, compared with the original EAGLE-3, the speedup decreases from 4.15× to 4.08×, while the average acceptance length decreases from 6.69 to 6.42. Consistent results are also observed on MT-Bench.
These observations suggest that directly imposing feature alignment on EAGLE-3's shared representation may introduce competing optimization requirements for token prediction and recurrent state propagation. Although alignment can encourage more target-like recurrent states, the shared-state architecture may limit its ability to accommodate both objectives effectively. This provides further empirical support for the motivation behind our feature–logit disentangled architecture, which assigns recurrent propagation and token prediction to separate representations.
2. Maintaining a constant alignment weight in STEP.
As illustrated in Figure 5(b), maintaining the alignment weight at 0.1 throughout training yields consistently higher feature similarity than the staged schedule. However, this stronger alignment does not lead to better token prediction. In particular, STEP with staged alignment generally achieves higher conditional acceptance at the early draft positions, while constant alignment provides no consistent advantage at later positions.
This suggests that maximizing feature similarity is not necessarily equivalent to optimizing drafting performance. Feature alignment serves as an auxiliary training guide rather than the ultimate optimization objective. Maintaining the alignment constraint throughout training may unnecessarily restrict the learned representations from further adapting to the token-prediction objective.
Our staged alignment schedule addresses this trade-off by using feature alignment to guide recurrent-state learning during early training and gradually removing the auxiliary constraint thereafter. Consequently, on GSM8K, staged alignment improves the speedup from 4.32× to 4.53× (+4.9%) and the average acceptance length from 6.84 to 7.06 (+3.2%) compared with constant alignment. Consistent improvements are also observed on MT-Bench.
Overall, these additional experiments suggest that the benefit of feature alignment depends on how it is integrated into the drafter architecture and training process. Our results further support the complementary roles of architectural disentanglement and staged feature alignment: the former provides separate representations for recurrent propagation and token prediction, while the latter offers an early training reference without imposing a persistent feature-matching constraint.
We sincerely thank you again for this valuable suggestion, which has helped us further clarify these design choices.
Response Q3:
We sincerely thank you for raising this important point. We agree that conditioning on all preceding draft tokens being accepted progressively reduces the number of eligible prefixes and increases the uncertainty of estimates at later positions.
To make this attrition explicit, we have added Figure 7 to the supplementary material, which reports the number of eligible prefixes remaining at each draft position for LLaMA-3.1-8B-Instruct on MT-Bench and GSM8K. As shown in Figure 7, STEP and EAGLE-3 retain comparable numbers of eligible prefixes near the seven-step training horizon, whereas STEP retains substantially more eligible prefixes as drafting proceeds further. For example, at position 15 on MT-Bench, 799 eligible prefixes remain for STEP, compared with 230 for EAGLE-3. A consistent pattern is also observed on GSM8K.
We acknowledge that this concern is particularly relevant at the deepest positions. On GSM8K, only seven eligible prefixes remain for EAGLE-3 at position 15, making this individual endpoint estimate inherently uncertain. More broadly, however, the consistent separation between STEP and EAGLE-3 in both conditional prediction accuracy and the number of eligible prefixes across positions 9–15 provides further evidence that STEP achieves more stable long-horizon drafting. This observation is also consistent with the improvements in end-to-end decoding speed reported in our experiments.
We sincerely thank you again for this valuable suggestion, which has helped us improve the completeness and rigor of our experimental analysis.
Response Q2: Thank you for this valuable suggestion. We provide additional implementation details regarding the model dimensions, parameter counts, drafting latency, alignment-weight schedule, and training setup, using the draft models associated with LLaMA-3.1-8B-Instruct as an example.
1. Model architecture and parameter counts.
The detailed architecture of the STEP drafter is summarized below.
Specifically, the Transformer decoder takes the concatenation of the token embedding and context feature as input to its Q/K/V projections. The subsequent prediction branch does not contain self-attention and introduces no additional KV cache.
2. Parameter counts, drafting latency, and training setup.
The following table provides a direct comparison between STEP and EAGLE-3.
As shown above, STEP reduces the number of trainable parameters by approximately 3.1% compared with EAGLE-3. The parameter counts exclude the frozen token embeddings copied from the target model and the target LM head used only for generating teacher distributions during training.
Drafting latency is measured on a single NVIDIA A800 80GB GPU with a batch size of 1. STEP achieves a slightly lower average latency per autoregressive drafting step (1.60 ms vs. 1.67 ms), suggesting that the proposed architectural disentanglement does not introduce additional per-step drafting overhead in this setting. For reference, when the speculative tree contains 60 candidate tokens, a single parallel verification pass by the target model takes approximately 30 ms.
3. Alignment-weight schedule.
The feature-alignment weight follows a three-stage schedule:
This schedule allows feature alignment to guide the early training process while gradually removing the auxiliary constraint during later optimization.
4. Training-data quantities and update counts.
We follow the EAGLE-3 data setup and use the same training data sources (ShareGPT and UltraChat-200K), without introducing any additional training corpora. For the baseline, we use the officially released EAGLE-3 checkpoint, while STEP is trained until convergence. We observe no further performance improvement when extending STEP training beyond this point.
We sincerely thank you for this valuable suggestion.