acceptodds
Under review as a conference paper at ICLR 2027

STEP: Separating Trajectory Evolution and Prediction for Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference, but its efficiency depends on sustaining accurate draft predictions over multiple future positions. Autoregressive drafters such as EAGLE-3 use a shared hidden state for token prediction and recurrent state propagation. We observe that high token prediction accuracy can coexist with substantial recurrent-state deviation from target-model features, motivating separate representations for these two roles. We introduce STEP, a feature–logit disentangled drafter that separates trajectory evolution from token prediction. To stabilize recurrent propagation, we employ a staged feature-alignment curriculum that uses target-model fused features as an early training reference before gradually removing the auxiliary constraint. We further exploit the resulting long-horizon drafting capability with confidence-guided progressive tree expansion, selectively extending promising branches to obtain longer accepted continuations. Across four target models, five benchmarks, and two temperatures, STEP outperforms EAGLE-3 in all 40 settings, with average relative gains of 10.1% in decoding speed and 15.1% in acceptance length. Ablations support complementary roles for disentanglement and alignment, while progressive tree expansion further improves decoding efficiency. In SGLang, STEP also achieves 2.9–11.9% higher throughput than EAGLE-3 across all tested batch sizes under the same speculative token budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5YLsOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEP separates the recurrent state used for drafting from the representation used for token prediction. It combines this architecture with an early-stage feature-alignment loss and confidence-guided tree expansion. Across four models, five benchmarks, and two temperatures, STEP reports average improvements of 10.1% in speed and 15.1% in acceptance length over EAGLE-3, with additional gains in SGLang.

Critical strengths
  1. The improvements are consistent across all 40 settings. The gains also carry over to SGLang, which makes the results practically relevant.
  2. Table 2 compares both drafters under both tree strategies. STEP improves performance even with the original tree construction, showing that the gains do not come solely from changing the expansion strategy.
Critical weaknesses
  1. The ablations show benefits from architectural separation and from adding staged alignment to STEP. However, they do not include EAGLE-3 with the same staged alignment loss or STEP with a constant alignment weight. These comparisons would help establish whether the alignment benefit depends on the new architecture and whether removing the constraint late in training is useful.
  2. Section 4.2 argues that narrowing the prediction branch offsets the extra MLP, but does not report concrete dimensions, parameter counts, or drafting latency. The paper specifies that the alignment weight decreases from 0.1 to zero, but not the timing or decay schedule. It also follows EAGLE-3’s training-data setup without explicitly documenting whether the compared checkpoints use matched data quantities and training steps.
Questions for authors
  1. How does EAGLE-3 perform with the same staged alignment loss? How does STEP perform when the alignment weight remains constant?
  2. Could you report the model dimensions, parameter counts, and drafting latency, specify the alignment-weight schedule, and clarify whether the compared checkpoints use the same training-data quantities and update counts?
  3. Since Figures 3 and 4 condition on all preceding tokens being accepted, how many eligible samples remain at later positions, and what is the uncertainty of those estimates?
5 comments

Response Q2: Thank you for this valuable suggestion. We provide additional implementation details regarding the model dimensions, parameter counts, drafting latency, alignment-weight schedule, and training setup, using the draft models associated with LLaMA-3.1-8B-Instruct as an example.

1. Model architecture and parameter counts.

The detailed architecture of the STEP drafter is summarized below.

ComponentInput DimensionOutput DimensionConfiguration
Target feature fusion3 × 4096 = 122884096Linear projection
Transformer decoder4096 + 4096 = 81924096Hidden dim = 4096, FFN dim = 14336
Trajectory projection40962048Linear projection
Prediction branch20482048SwiGLU FFN (2048–7168–2048)
Output prediction204832000RMSNorm + trainable LM head

Specifically, the Transformer decoder takes the concatenation of the token embedding and context feature as input to its Q/K/V projections. The subsequent prediction branch does not contain self-attention and introduces no additional KV cache.

2. Parameter counts, drafting latency, and training setup.

The following table provides a direct comparison between STEP and EAGLE-3.

MetricEAGLE-3STEP
Trainable parameters424.7M411.6M
Drafting latency (ms/step)1.671.60
Training datasetsShareGPT + UltraChat-200KShareGPT + UltraChat-200K
CheckpointOfficially releasedTrained until convergence

As shown above, STEP reduces the number of trainable parameters by approximately 3.1% compared with EAGLE-3. The parameter counts exclude the frozen token embeddings copied from the target model and the target LM head used only for generating teacher distributions during training.

Drafting latency is measured on a single NVIDIA A800 80GB GPU with a batch size of 1. STEP achieves a slightly lower average latency per autoregressive drafting step (1.60 ms vs. 1.67 ms), suggesting that the proposed architectural disentanglement does not introduce additional per-step drafting overhead in this setting. For reference, when the speculative tree contains 60 candidate tokens, a single parallel verification pass by the target model takes approximately 30 ms.

3. Alignment-weight schedule.

The feature-alignment weight follows a three-stage schedule:

  • First 40% of training: The alignment weight remains constant at 0.1.
  • Next 20% of training: The weight linearly decays from 0.1 to 0.
  • Final 40% of training: The alignment weight remains 0.

This schedule allows feature alignment to guide the early training process while gradually removing the auxiliary constraint during later optimization.

4. Training-data quantities and update counts.

We follow the EAGLE-3 data setup and use the same training data sources (ShareGPT and UltraChat-200K), without introducing any additional training corpora. For the baseline, we use the officially released EAGLE-3 checkpoint, while STEP is trained until convergence. We observe no further performance improvement when extending STEP training beyond this point.

We sincerely thank you for this valuable suggestion.

Response Q1:

We sincerely thank you for suggesting these two important ablations. To address this question, we additionally trained EAGLE-3 with the same staged feature-alignment objective and STEP with a constant alignment weight of 0.1. We evaluated both variants on GSM8K and MT-Bench using LLaMA-3.1-8B-Instruct, examining conditional token acceptance, feature similarity, decoding speedup, and average acceptance length. All four variants were evaluated using the same progressive tree expansion strategy.

The decoding efficiency results are summarized in the following table.

MethodMT-Bench SpeedupτGSM8K Speedupτ
STEP + staged alignment4.51×7.014.53×7.06
STEP + constant alignment4.29×6.834.32×6.84
EAGLE-3 + staged alignment4.07×6.604.08×6.42
EAGLE-3 (without alignment)4.19×6.764.15×6.69

1. Applying staged alignment to EAGLE-3.

As shown in Figure 5, introducing staged alignment into EAGLE-3 improves its feature similarity to the target model. However, this improvement does not translate into better overall decoding performance. On GSM8K, compared with the original EAGLE-3, the speedup decreases from 4.15× to 4.08×, while the average acceptance length decreases from 6.69 to 6.42. Consistent results are also observed on MT-Bench.

These observations suggest that directly imposing feature alignment on EAGLE-3's shared representation may introduce competing optimization requirements for token prediction and recurrent state propagation. Although alignment can encourage more target-like recurrent states, the shared-state architecture may limit its ability to accommodate both objectives effectively. This provides further empirical support for the motivation behind our feature–logit disentangled architecture, which assigns recurrent propagation and token prediction to separate representations.

2. Maintaining a constant alignment weight in STEP.

As illustrated in Figure 5(b), maintaining the alignment weight at 0.1 throughout training yields consistently higher feature similarity than the staged schedule. However, this stronger alignment does not lead to better token prediction. In particular, STEP with staged alignment generally achieves higher conditional acceptance at the early draft positions, while constant alignment provides no consistent advantage at later positions.

This suggests that maximizing feature similarity is not necessarily equivalent to optimizing drafting performance. Feature alignment serves as an auxiliary training guide rather than the ultimate optimization objective. Maintaining the alignment constraint throughout training may unnecessarily restrict the learned representations from further adapting to the token-prediction objective.

Our staged alignment schedule addresses this trade-off by using feature alignment to guide recurrent-state learning during early training and gradually removing the auxiliary constraint thereafter. Consequently, on GSM8K, staged alignment improves the speedup from 4.32× to 4.53× (+4.9%) and the average acceptance length from 6.84 to 7.06 (+3.2%) compared with constant alignment. Consistent improvements are also observed on MT-Bench.

Overall, these additional experiments suggest that the benefit of feature alignment depends on how it is integrated into the drafter architecture and training process. Our results further support the complementary roles of architectural disentanglement and staged feature alignment: the former provides separate representations for recurrent propagation and token prediction, while the latter offers an early training reference without imposing a persistent feature-matching constraint.

We sincerely thank you again for this valuable suggestion, which has helped us further clarify these design choices.

Response Q3:

We sincerely thank you for raising this important point. We agree that conditioning on all preceding draft tokens being accepted progressively reduces the number of eligible prefixes and increases the uncertainty of estimates at later positions.

To make this attrition explicit, we have added Figure 7 to the supplementary material, which reports the number of eligible prefixes remaining at each draft position for LLaMA-3.1-8B-Instruct on MT-Bench and GSM8K. As shown in Figure 7, STEP and EAGLE-3 retain comparable numbers of eligible prefixes near the seven-step training horizon, whereas STEP retains substantially more eligible prefixes as drafting proceeds further. For example, at position 15 on MT-Bench, 799 eligible prefixes remain for STEP, compared with 230 for EAGLE-3. A consistent pattern is also observed on GSM8K.

We acknowledge that this concern is particularly relevant at the deepest positions. On GSM8K, only seven eligible prefixes remain for EAGLE-3 at position 15, making this individual endpoint estimate inherently uncertain. More broadly, however, the consistent separation between STEP and EAGLE-3 in both conditional prediction accuracy and the number of eligible prefixes across positions 9–15 provides further evidence that STEP achieves more stable long-horizon drafting. This observation is also consistent with the improvements in end-to-end decoding speed reported in our experiments.

We sincerely thank you again for this valuable suggestion, which has helped us improve the completeness and rigor of our experimental analysis.

Reviewer_D1q6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies that the misalignment of the rollout draft feature and target feature leads to token prediction accuracy drop in longer speculative decoding window settings. This paper proposes to separate the rollout feature (recurrent state) for future draft prediction and the feature (logit state) for predicting the current token. An MLP layer uses the recurrent state to generate the logit state. The recurrent state is trained with a feature alignment loss to better align the target feature state. A confidence threshold is used to determine whether the draft should generate deeper trees.

Critical strengths
  1. This paper identifies that the misalignment of the rollout draft feature and target feature leads to token prediction accuracy drop in longer speculative decoding window settings.
  2. This paper proposes to separate the rollout feature (recurrent state) for future draft prediction and the feature (logit state) for predicting the current token. This separation shows longer accept length and speedup.
Critical weaknesses
  1. There is a misalignment between the training and evaluation speculative decoding length. The training decoding length of both Eagle3 and STEP is 7, while they are evaluated with different decoding length. Although this paper targets the performance when decoding length is longer than trained decoding length, it's unclear whether the improvement of STEP still holds if both Eagle3 and STEP are trained with longer speculative decoding length.
Questions for authors

How do EAGLE-3 and STEP perform when both are trained with longer recurrent rollout horizons (e.g., 15 steps instead of 7)? Does STEP maintain its advantage over EAGLE-3 as the training horizon increases, or is there a point beyond which the performance gap diminishes or disappears?

3 comments

We thank you for this insightful question! To address this concern, we additionally trained both EAGLE-3 and STEP with a longer recurrent rollout horizon of H=15, compared with the original H=7 setting. We evaluated all configurations on GSM8K and MT-Bench using LLaMA-3.1-8B-Instruct and analyzed both conditional acceptance rates and recurrent-state similarity over 15 draft positions. The new results are presented in Figure 6.

First, STEP maintains its advantage over EAGLE-3 under the longer training horizon. On GSM8K, at H=15, STEP achieves a speedup of 4.13× and an average acceptance length of 6.53, compared with 4.01× and 6.30 for EAGLE-3, corresponding to relative improvements of 3.0% and 3.7%, respectively. Consistent results are also observed on MT-Bench, as shown in the table. Although the performance gap narrows compared with H=7, STEP continues to outperform EAGLE-3 in both metrics across the two benchmarks. These results suggest that STEP's advantage is not solely attributable to evaluating EAGLE-3 beyond its training rollout horizon.

MethodMT-Bench: Speedupτ\tauGSM8K:Speedupτ\tau
STEP (TTT=7)4.51×7.014.53×7.06
EAGLE-3 (TTT=7)4.19×6.764.15×6.69
STEP (TTT=15)4.08×6.594.13×6.53
EAGLE-3 (TTT=15)4.01×6.484.01×6.30

Second, increasing the training horizon does not necessarily improve end-to-end decoding efficiency. Interestingly, both methods achieve lower speedups and acceptance lengths when H increases from 7 to 15. As shown in Figure 6(a), the longer training horizon leads to reduced conditional acceptance rates at several early draft positions. One possible explanation is that extending the rollout horizon introduces additional prediction objectives, requiring the model to balance optimization across a wider range of draft positions. Since errors at earlier positions terminate the accepted continuation, maintaining strong early-position prediction quality is particularly important for overall decoding efficiency.

Third, the results provide additional evidence for the importance of recurrent-state stability in long-horizon drafting. As shown in Figure 6(b), increasing the training horizon substantially improves EAGLE-3's cosine similarity to target fused features, even without explicit feature-alignment supervision. This result is consistent with our motivation that preserving target-feature geometry is important for stable long-horizon rollout. Nevertheless, STEP maintains substantially higher feature similarity and better end-to-end performance than EAGLE-3 under the same H=15 training horizon. Together with our existing disentanglement ablations, these results further support the roles of feature–logit separation and trajectory alignment in maintaining stable recurrent prediction.

Taken together, these additional experiments demonstrate that STEP retains its advantage under a matched longer training horizon, while also providing further empirical support for our motivation to distinguish recurrent trajectory evolution from token prediction. We sincerely thank you again for suggesting this valuable experiment, which has helped us better understand the effect of training rollout horizons on speculative decoding performance.

Thank you for the additional results. I will keep the existing score.

Thank you for taking the time to review the additional results and for your follow-up. We greatly appreciate your constructive suggestion, which helped us strengthen the paper by providing a more comprehensive evaluation under matched longer rollout horizons.

Reviewer_mgCkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STEP proposes a feature-logit disentangled autoregressive drafter framework that identifies and resolves the "representation entanglement" problem in autoregressive drafters like EAGLE-3, where a single shared hidden state simultaneously serves both token prediction and recurrent state propagation, causing quality degradation in long-horizon drafting. The method comprises three key techniques: (1) Feature-logit disentangled architecture — separating trajectory state (for recurrent propagation) from logit state (derived via a lightweight MLP for token prediction); (2) Staged feature-alignment curriculum — using a weak alignment loss early in training to guide the trajectory state toward the target manifold, then fully removing it in later stages to avoid over-regularization; (3) Confidence-guided progressive tree expansion — first exploring shallow branches, then selectively deepening high-confidence branches based on cumulative confidence scores. STEP outperforms EAGLE-3 across all 40 configurations, achieving 10.1% higher decoding speed and 15.1% longer acceptance length on average, with 2.9–11.9% throughput improvements over EAGLE-3 in SGLang at batch sizes 1–16.

Critical strengths
  1. Through rigorous experiments, the paper reveals a striking divergence in EAGLE-3 between token prediction accuracy and recurrent-state similarity: at step 7, state similarity is only 0.26 while prediction accuracy remains at 69% (Figure 1). This finding provides a clear causal chain for the subsequent design — the degradation is not due to insufficient training of EAGLE-3, but rather an inevitable consequence of an architecture that forces a single state to serve two conflicting objectives. This complete narrative from phenomenon identification to architectural design offers greater methodological value than simply "adding modules."

  2. The paper carefully designs a three-stage progressive ablation (EAGLE-3 → disentanglement only without alignment → full STEP), cleanly isolating the individual contributions of disentangled architecture and staged alignment. Results show that disentanglement alone yields approximately 11 percentage points of conditional acceptance rate improvement at positions 9–12; staged alignment extends this advantage beyond position 15 (Figure 3). Progressive tree expansion further amplifies the value of long-horizon prediction capability (Table 2).

Critical weaknesses
  1. The paper tests Vicuna-13B, LLaMA-3.1-8B-Instruct, LLaMA-3.3-70B-Instruct, and DeepSeek-R1-Distill-LLaMA-8B, with the largest being 70B. In the current LLM ecosystem, MoE architectures (e.g., Mixtral-8x7B, DeepSeek-V3, Qwen3-MoE) have become mainstream inference efficiency solutions, yet the paper entirely overlooks them. The expert routing mechanism and sparse activation characteristics of MoE models may affect the stability of recurrent state propagation, but the paper provides no analysis or discussion. Furthermore, all tested models are dense architectures, leaving the method's applicability at larger scales and more complex topologies entirely unknown.

  2. Experiments are conducted only on NVIDIA A800 (80GB), lacking fine-grained modeling of hardware characteristics such as HBM bandwidth, compute intensity, kernel efficiency, and SM occupancy. No validation is performed on newer hardware like H100/H20.

Questions for authors

See Weakness

1 comment

Response (W1): We sincerely thank you for the positive assessment of our work and the constructive suggestions. We are particularly encouraged by your recognition that our work establishes a complete methodological narrative, from identifying the underlying phenomenon to developing an architectural solution, rather than merely introducing additional modules. We greatly appreciate this recognition.

Regarding the applicability of STEP to MoE architectures, we appreciate the reviewer's suggestion to further validate our method on MoE models. From an architectural perspective, although MoE models introduce input-dependent expert routing and sparse activation, their Transformer backbones still rely on hidden representations for autoregressive computation. In particular, the expert routing mechanism changes how intermediate representations are generated, but does not fundamentally alter the two distinct functional requirements of an autoregressive drafter: maintaining a reliable recurrent trajectory and producing accurate token predictions. Consequently, the representation entanglement addressed by STEP is not inherently eliminated by replacing dense FFN layers with MoE layers. Furthermore, MoE routing may introduce additional complexity into the evolution of target-model representations, potentially making accurate long-horizon feature prediction more challenging. This further motivates the separation of trajectory propagation and token prediction in STEP. We therefore believe that the proposed design has a reasonable architectural basis for extending to MoE models.

As an important direction for future work, we will further investigate the applicability and effectiveness of STEP on MoE architectures. We sincerely thank you again for highlighting this important direction.

Response (W2): We sincerely thank you for this valuable suggestion. We agree that a more fine-grained analysis of hardware characteristics, including HBM bandwidth, compute intensity, kernel efficiency, and SM occupancy, would provide additional insights into the practical performance of speculative decoding (SD).

We would like to clarify that STEP primarily focuses on algorithm-level improvements to speculative decoding, rather than hardware-specific optimizations. Its core contributions lie in improving long-horizon drafting accuracy and acceptance efficiency. These algorithmic improvements are largely independent of specific GPU architectures and can potentially complement hardware-level optimizations, as they do not rely on particular GPU architectures or hardware-specific kernel designs. While hardware characteristics inevitably influence the actual decoding latency and throughput, the underlying algorithmic mechanisms of STEP remain applicable across different hardware platforms.

Regarding the experimental platform, all our experiments were conducted on NVIDIA A800 (80GB) GPUs, based on the computational resources available in our laboratory. We acknowledge that further evaluation on newer GPU architectures, such as H100 and H20, would be valuable for providing a more comprehensive understanding of STEP's performance across different hardware platforms. We appreciate your suggestion and consider this an important direction for future investigation.

We sincerely thank you again for this constructive suggestion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewer Concerns

Dear Area Chairs and Reviewers,

We sincerely thank you for the careful evaluation of our submission and the constructive feedback. Your comments have helped us clarify the scope of STEP, strengthen the empirical evidence, and better isolate the contributions of its architectural and training designs.

The main concerns and our corresponding updates are summarized below:

  • Applicability to MoE models and different hardware platforms (Reviewer mgCk): We clarify that STEP separates two distinct functional roles of an autoregressive drafter—recurrent trajectory propagation and token prediction—and that the underlying motivation for this separation may also extend to MoE architectures. STEP does not rely on hardware-specific kernels, and all runtime comparisons are conducted under the same NVIDIA A800 hardware and software environment. We also acknowledge that evaluation on MoE models and newer GPUs remains an important direction for future work.

  • Effect of a longer matched training horizon (Reviewer D1q6): We additionally train both STEP and EAGLE-3 with a longer recurrent rollout horizon of H=15H=15, with results reported in Table 3 and Figure 6. STEP continues to outperform EAGLE-3 on both MT-Bench and GSM8K under the matched H=15H=15 setting, providing further evidence that its advantage is not solely attributable to evaluating EAGLE-3 beyond its original training horizon.

  • Roles of architectural disentanglement and staged feature alignment (Reviewer 5YLs): We add the two requested controls—EAGLE-3 with staged alignment and STEP with constant alignment—in Table 2 and Figure 5. Staged alignment improves EAGLE-3's feature similarity but reduces its decoding efficiency, while constant alignment underperforms staged STEP. These results provide additional empirical support for the complementary roles of architectural disentanglement and staged feature alignment.

  • Statistical reliability of long-horizon drafting analysis (Reviewer 5YLs): We additionally provide Figure 7 in the supplementary material, reporting the number of eligible prefixes remaining at each draft position on MT-Bench and GSM8K. This analysis makes the sample attrition at deeper positions explicit and clarifies the associated uncertainty, while providing further evidence for STEP's improved long-horizon drafting stability.

We would also like to highlight STEP's main contributions:

  • Identifying representation entanglement in autoregressive drafting: We reveal that accurate token prediction can coexist with substantial recurrent-state deviation in EAGLE-3, highlighting the potential limitations of using a shared hidden state for both token prediction and recurrent trajectory propagation over extended rollouts.

  • Separating trajectory evolution from token prediction: STEP introduces a feature–logit disentangled drafter that maintains a dedicated trajectory state for recurrent propagation and derives a separate logit state for token prediction. A staged feature-alignment curriculum provides early geometric guidance while gradually removing the auxiliary constraint to allow greater flexibility for prediction-oriented optimization.

  • Demonstrating consistent and practical acceleration: Across four target models, five benchmarks, and two temperatures, STEP outperforms EAGLE-3 in all 40 evaluated settings. Notably, on HumanEval with LLaMA-3.3-70B-Instruct, STEP achieves 20.2% higher decoding speedup (4.99× → 6.00×) and 26.4% longer average acceptance length (6.75 → 8.53) compared with EAGLE-3.

All modifications and newly added results are highlighted in red in the revised manuscript. We hope these clarifications and additional experiments help address the reviewers' concerns and facilitate the assessment of our work.

We sincerely thank you again for your time, careful consideration, and valuable guidance.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.