acceptodds
Under review as a conference paper at ICLR 2027

RELAY: Momentum Drafting from Hidden Transitions for Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference by using a lightweight drafter to propose tokens for parallel verification. Recent feature-guided drafters improve draft quality by reusing hidden features from the target forward pass, but these features are typically treated as static endpoints, exposing what the target model has computed without explicitly modeling how its representations evolve toward prediction. This leaves the lightweight drafter to infer the missing transformation, making farther-ahead predictions harder to align with the target model. We instead observe that transitions between target-model stages provide a direct drafting signal, as they capture representation changes rather than merely their resulting states. Based on this, we propose RELAY, a lightweight drafter that performs momentum drafting from hidden transitions. RELAY decomposes the target forward pass into low-, mid-, and top-stage features, converts their stagewise differences into early and late momentum states, and recurrently refines these momenta with an Attention-MLP architecture during draft-time generation. By propagating hidden-transition momentum across draft steps, RELAY uses target-side computation as evolving guidance. Experiments across multiple backbones, benchmarks, and serving settings show consistent efficiency gains, achieving up to 31.3% higher speedup and 30.0% longer average acceptance length over EAGLE-3. Code is available at https://anonymous.4open.science/r/RELAY-8421/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Momentum Drafting, Speculative Decoding, LLM Inference Acceleration, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6bVgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve the prediction quality of draft models in speculative decoding. It proposes RELAY, which initializes two momentum states from the differences between hidden representations at different layers of the target model. These momentum states are recurrently updated across drafting steps through an Attention-MLP architecture to guide subsequent token predictions. The authors evaluate RELAY on four target models and five benchmarks under different decoding settings. Its effectiveness is assessed through comparisons with EAGLE-3 and other baselines, long-horizon drafting analyses, and practical serving experiments.

Critical strengths

S1. I think RELAY's early/late momentum design is an interesting extension of EAGLE-3. Rather than relying solely on feature fusion, RELAY explicitly models the differences between hidden features and updates them during drafting. The results in Figure 1 suggest that this design is effective, particularly at later draft positions.

S2. RELAY achieves fairly consistent performance improvements over EAGLE-3 across the tested models and benchmarks. For example, under greedy decoding, the mean speedup on LLaMA-3.3-70B increases from 4.36× to 4.77× (Table 1). I also find the long-horizon drafting analysis in Figure 3 valuable, as it directly examines the core problem that motivates the method. The inference serving experiments further suggest that these improvements are also meaningful in practical settings.

Critical weaknesses

W1. I think the motivation for explicitly modeling hidden-state transitions is not sufficiently justified. These transitions can already be computed from the multi-layer features used by EAGLE-3, so it remains unclear why explicitly modeling them would be more beneficial than learning them implicitly through feature fusion. Although Figure 1 provides some empirical evidence supporting the effectiveness of this design, the overall contribution still appears relatively incremental.

W2. The experiments cover a reasonable range of models and benchmarks, but all four target models are based on the LLaMA architecture. In addition, the long-horizon drafting analysis in Figure 3 is conducted only on LLaMA-3.1-8B using MT-bench. While the results are encouraging, I am not fully convinced that the method would provide similar benefits across other model architectures.

Questions for authors

Q1. Given that hidden-state transitions can be directly computed from the multi-layer features already utilized by EAGLE-3, could the authors further clarify what unique advantages explicit transition modeling and the recurrent momentum mechanism offer over existing feature fusion methods?

Q2. Given that all evaluated target models are based on the LLaMA architecture, could the authors further explain to what extent the current experimental evidence supports the generalizability of the proposed method to other model architectures?

Q3. Equations (8) and (11) describe an additive momentum update, whereas the publicly available code implementation uses torch.lerp with an interpolation factor of 0.5. Could the authors explain this discrepancy and clarify which update rule was actually used to obtain the reported experimental results?

4 comments
  • We sincerely thank the Reviewer for the constructive feedback and for carefully examining both our paper and its public code. We appreciate your positive assessment of RELAY’s design and the consistent gains over the evaluated baselines. We have carefully considered your comments and provide our point-by-point responses below.
  1. Given that hidden-state transitions can be directly computed from the multi-layer features already utilized by EAGLE-3, could the authors further clarify what unique advantages explicit transition modeling and the recurrent momentum mechanism offer over existing feature fusion methods?

Answer: We appreciate the opportunity to clarify our contribution. EAGLE-3 gives the drafter access to hidden features from several target layers through feature fusion. The differences between successive selected features, which we call stage transitions, capture how the target representation changes across depth at each prefix position. RELAY recurrently carries forward and adapts these transitions to guide drafting, introducing an architectural inductive bias toward continuity across draft steps rather than additional target information.

The table summarizes how the two methods use target features and compute draft representations. We omit normalization and position and draft-step indices for clarity. Brackets denote concatenation, and ee is the current token embedding. The superscripts (0)(0) and −^{-} denote initialization from target features and the preceding draft state, respectively.

EAGLE-3RELAY
Use of target featuresFeature fusion h=W[zlow,zmid,ztop]h=W[z_{\mathrm{low}},z_{\mathrm{mid}},z_{\mathrm{top}}]Feature fusion zlow=Winj[e,zmid−,ztop−]z_{\mathrm{low}}=W_{\mathrm{inj}}[e,z_{\mathrm{mid}}^{-},z_{\mathrm{top}}^{-}] Early and late momentum initialization vearly(0)=zmid(0)−zlow(0)v_{\mathrm{early}}^{(0)}=z_{\mathrm{mid}}^{(0)}-z_{\mathrm{low}}^{(0)} vlate(0)=ztop(0)−zmid(0)v_{\mathrm{late}}^{(0)}=z_{\mathrm{top}}^{(0)}-z_{\mathrm{mid}}^{(0)}
Draft update ruleu=h+Attn([e,h])u=h+\mathrm{Attn}([e,h]) h+=u+MLP(u)h^{+}=u+\mathrm{MLP}(u)Early branch Δvearly=Attn([zlow,vearly−])\Delta v_{\mathrm{early}}=\mathrm{Attn}([z_{\mathrm{low}},v_{\mathrm{early}}^{-}]) vearly=12(vearly−+Δvearly)v_{\mathrm{early}}=\tfrac12(v_{\mathrm{early}}^{-}+\Delta v_{\mathrm{early}}) zmid=zlow+vearlyz_{\mathrm{mid}}=z_{\mathrm{low}}+v_{\mathrm{early}} Late branch Δvlate=MLP([zmid,vlate−])\Delta v_{\mathrm{late}}=\mathrm{MLP}([z_{\mathrm{mid}},v_{\mathrm{late}}^{-}]) vlate=12(vlate−+Δvlate)v_{\mathrm{late}}=\tfrac12(v_{\mathrm{late}}^{-}+\Delta v_{\mathrm{late}}) ztop=zmid+vlatez_{\mathrm{top}}=z_{\mathrm{mid}}+v_{\mathrm{late}}

The two-branch design draws on evidence that representations at different depths capture different aspects of language [1]. Modeling the early and late transitions separately lets each branch adapt its own direction of representation change during drafting.

As shown in Figure 1, adding stage differences to the stage-only drafter improves both Mean Acc and Late Acc under matched parameter budgets. This comparison shows that a lightweight drafter benefits from explicit access to stage transitions even when the underlying features are already available. Full RELAY provides further gains, particularly at later positions, supporting the value of recurrent transition guidance for longer draft rollouts.

Furthermore, we conduct zero-initialization and direct-prediction ablations to assess the roles of target-derived momentum initialization and explicit momentum propagation, respectively.

  • Zero momentum initialization retains the target stage features and initializes both momentum states to zero. This tests the value of a target-derived direction for the first draft update.
  • Direct momentum prediction lets each branch directly predict the current momentum, v=Fθ(z,v−)v=F_\theta(z,v^{-}), rather than a transition increment Δv\Delta v. Using the same branch inputs isolates the contribution of updating the previous momentum through a learned correction.
VariantMT-bench Speedup↑MT-bench τ\tau↑GSM8K Speedup↑GSM8K τ\tau↑
Zero momentum initialization4.23 ×\times6.794.28 ×\times6.67
Direct momentum prediction4.09 ×\times6.614.12 ×\times6.48
RELAY4.43×\times7.134.49×\times6.99

Target-derived initialization yields longer accepted sequences on both benchmarks, suggesting that changes observed across target layers provide useful guidance for drafting. Compared with direct momentum prediction, explicit propagation translates longer accepted sequences into faster decoding. This suggests that carrying the previous transition direction into each update is useful as the context evolves, whereas a branch that directly predicts the current momentum must learn how much of that direction to retain.

In the revision, we report the decoding efficiency results in Section 5.4 (Table 3). We also clarify the role of stage transitions in Sections 3 and 4.1.

[1] Jawahar G, Sagot B, Seddah D. What Does BERT Learn about the Structure of Language? ACL, 2019.

  1. Given that all evaluated target models are based on the LLaMA architecture, could the authors further explain to what extent the current experimental evidence supports the generalizability of the proposed method to other model architectures? In addition, the long-horizon drafting analysis in Figure 3 is conducted only on LLaMA-3.1-8B using MT-bench.

Answer: We sincerely thank the Reviewer for this careful observation. Our evaluation covers four widely used open-weight backbones spanning chat and reasoning models, following the target-model selection in EAGLE-3 [2]. These models enable direct comparisons with established baselines across different scales and training backgrounds.

RELAY consistently improves acceptance length and decoding speed across these targets on five benchmarks under both greedy and non-greedy decoding. These results support its effectiveness across model scales, training backgrounds, and tasks within the LLaMA family.

RELAY’s formulation does not depend on LLaMA-specific operations. It derives guidance from intermediate hidden-state differences and adapts it through recurrent updates, making it applicable in principle to other decoder architectures with compatible representations. Evaluating its effectiveness beyond the LLaMA family is an important direction for future work. As additional results become available, we plan to release them together with the corresponding drafter checkpoints.

To further strengthen the evidence, we extend the analysis in Figure 3 to Vicuna 13B and DeepSeek-R1-Distill-LLaMA-8B. Both models exhibit consistent trends, with RELAY maintaining higher conditional acceptance rates at later draft positions and higher prefix survival rates over longer horizons.

We include these results and accompanying analysis in Appendix B of the revision.

[2] Li Y, Wei F, Zhang C, Zhang H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test[C]. NeurIPS, 38, 2025.

  1. Equations (8) and (11) describe an additive momentum update, whereas the publicly available code implementation uses torch.lerp with an interpolation factor of 0.5. Could the authors explain this discrepancy and clarify which update rule was actually used to obtain the reported experimental results?

Answer: We sincerely thank the Reviewer for carefully examining our implementation! We apologize for the discrepancy between the equations and the code. All experimental results in the submitted paper were obtained using torch.lerp with an interpolation factor of 0.5 in both branches.

As detailed in our response to Reviewer sRxA’s Q1, this choice arose from our early design experiments. Our initial implementation used an Adam-like weighted formulation,

=\beta v_{i-1}^{(k-1)} +(1-\beta)\Delta v_i^{(k)}.$$ **$\beta=0.5$ achieved the highest mean draft-token accuracy** among the evaluated settings, motivating equal weighting. We include the position-wise accuracy curves and accompanying analysis in Appendix B of the revision. **To emphasize the equal contributions of the previous momentum and the learned update, we presented a simplified residual-style expression but omitted the required scaling factor**. *We have corrected Equations (8) and (11) in the revision* to match the implementation, $$v_{i,\mathrm{early}}^{(k)} =\frac{1}{2}\left( v_{i-1,\mathrm{early}}^{(k-1)} +\Delta v_{i,\mathrm{early}}^{(k)} \right),$$ $$v_{i,\mathrm{late}}^{(k)} =\frac{1}{2}\left( v_{i-1,\mathrm{late}}^{(k-1)} +\Delta v_{i,\mathrm{late}}^{(k)} \right).$$ This correction leaves the implementation and reported results unchanged. We thank the Reviewer again for helping us improve the precision of our presentation.

Thanks for your rebuttal and I will maintain my positive score.

We sincerely thank the Reviewer for the follow-up and continued positive assessment of our work! We especially appreciate your perceptive observations and careful examination of both our manuscript and implementation. Your insightful questions helped us clarify RELAY’s contribution and present it more precisely. They also guided us in strengthening the experimental evidence.

We are grateful for your constructive engagement throughout the discussion and your support for our work!

Reviewer_kPjaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RELAY, a feature-guided speculative decoding method that introduces recurrent momentum states derived from hidden-state transitions of the target model. The method employs an asymmetric Attention-MLP architecture to improve draft-token prediction and reports speedup improvements over EAGLE-3 across multiple models and benchmarks.

Critical strengths
  1. Practical significance. Improving speculative decoding efficiency is an important research problem. The paper reports consistent speedup improvements over EAGLE-3 across several target models and provides additional serving evaluations using SGLang.
  2. Promising empirical results. The reported improvements in average acceptance length and long-horizon drafting performance are encouraging, particularly for LLaMA-3.3-70B and reasoning-intensive workloads.
Critical weaknesses
  1. Insufficient justification of hidden-transition momentum. The proposed method builds heavily upon EAGLE-3's multi-layer feature extraction, with hidden transitions computed as differences between existing intermediate representations. Since these transitions are deterministic combinations of features already available to EAGLE-3, the authors should more clearly explain what fundamentally new information or inductive bias the proposed momentum mechanism introduces.
  2. Inconsistent definition of low-stage features. In Figures 1(a) and 2, the low-stage feature is presented as a hidden representation extracted from a shallow layer of the target model. However, Equation 6 defines it as a learned projection of token embeddings and previous draft states, without adequately explaining the relationship between these representations or how the learned states remain aligned with the target model's intermediate features.
  3. Insufficient motivation and ablation studies for the asymmetric Attention-MLP architecture. The authors do not adequately justify why attention is specifically used for early-stage transitions while an MLP is used for late-stage transitions. No architectural ablations comparing Attention-Attention, MLP-MLP, or MLP-Attention configurations are provided, making it difficult to determine whether this asymmetric design is necessary or optimal.
  4. Insufficient statistical analysis of the diagnostic experiments. The error bars in Figure 1(b) represent standard deviations across draft positions rather than uncertainty across independent training runs or test samples. Consequently, they do not establish whether the reported improvements are statistically reliable, and repeated-run evaluations would strengthen the conclusions.
  5. Unclear training details and computational fairness. The paper does not clearly report the drafter parameter counts, training costs, or per-backbone training details, particularly for LLaMA-3.3-70B. The authors should explicitly clarify whether all RELAY drafters were independently trained and provide sufficient information to assess the computational fairness of comparisons with EAGLE-3.
  6. Inconsistent sources of baseline results. Appendix A states that several baseline results are directly taken from the EAGLE-3 paper, while EAGLE-3 and DFlash are evaluated using released checkpoints on the authors' hardware. Combining previously reported results with newly measured results introduces potential differences in evaluation environments and configurations, which should be clearly documented and controlled wherever possible.
  7. Incomplete and questionable comparisons with DFlash and DSpark. Table 1 reports DFlash performing substantially worse than EAGLE-3, which is inconsistent with the relative performance reported in the original DFlash paper and the subsequent DSpark study. Although differences in experimental settings may explain this discrepancy, the authors provide no analysis of the reversed ranking or sufficient evidence that the comparison is fair. Furthermore, DFlash is evaluated on only one target backbone, and DSpark, a highly relevant recent baseline, is entirely absent from the experimental comparisons.
  8. Inconsistent baseline coverage across decoding temperatures. Table 1 uses different baseline sets for Temperature = 0 and Temperature = 1, particularly for Vicuna-13B. Although the authors mention restricting stochastic-decoding comparisons to methods supporting lossless verification, the exclusion criteria for individual baselines should be explained more explicitly to make the comparisons easier to interpret.
  9. Insufficient control of dynamic draft tree configurations. RELAY uses backbone-specific and temperature-specific dynamic draft tree settings, while EAGLE-3 is evaluated using its recommended configurations. Since tree construction can substantially influence acceptance length and end-to-end speedup, the authors should provide comparisons under identical tree configurations and draft-token budgets to isolate improvements attributable to the drafter architecture.
  10. Insufficient analysis of computational overhead. Although the paper reports end-to-end speedup, it lacks a detailed breakdown of drafting time, verification time, additional memory consumption, and computational overhead introduced by recurrent momentum updates. Without this analysis, it is difficult to determine whether the performance gains arise from more effective feature modeling or additional computational capacity.
  11. Presentation and equation-numbering issues. The two displayed equations defining stage-feature initialization and momentum initialization in Section 4.1 are missing equation numbers. More generally, several important architectural definitions and experimental details are introduced without sufficient explanation, reducing the clarity and reproducibility of the proposed approach.
Questions for authors

See weakness

6 comments
  • We sincerely thank the Reviewer for the detailed comments and the time invested in evaluating our work. We appreciate your recognition of the practical significance of speculative decoding and the promising empirical results. We have carefully considered each comment and provide point-by-point responses below.
  1. Insufficient justification of hidden-transition momentum. Since these transitions are deterministic combinations of features already available to EAGLE-3, the authors should more clearly explain what fundamentally new information or inductive bias the proposed momentum mechanism introduces.

Answer: We appreciate the opportunity to clarify our contribution. EAGLE-3 gives the drafter access to hidden features from several target layers through feature fusion. The differences between successive selected features, which we call stage transitions, capture how the target representation changes across depth at each prefix position. RELAY recurrently carries forward and adapts these transitions to guide drafting, introducing an architectural inductive bias toward continuity across draft steps rather than additional target information.

The table summarizes how the two methods use target features and compute draft representations. We omit normalization and position and draft-step indices for clarity. Brackets denote concatenation, and ee is the current token embedding. The superscripts (0)(0) and −^{-} denote initialization from target features and the preceding draft state, respectively.

EAGLE-3RELAY
Use of target featuresFeature fusion h=W[zlow,zmid,ztop]h=W[z_{\mathrm{low}},z_{\mathrm{mid}},z_{\mathrm{top}}]Feature fusion zlow=Winj[e,zmid−,ztop−]z_{\mathrm{low}}=W_{\mathrm{inj}}[e,z_{\mathrm{mid}}^{-},z_{\mathrm{top}}^{-}] Early and late momentum initialization vearly(0)=zmid(0)−zlow(0)v_{\mathrm{early}}^{(0)}=z_{\mathrm{mid}}^{(0)}-z_{\mathrm{low}}^{(0)} vlate(0)=ztop(0)−zmid(0)v_{\mathrm{late}}^{(0)}=z_{\mathrm{top}}^{(0)}-z_{\mathrm{mid}}^{(0)}
Draft update ruleu=h+Attn([e,h])u=h+\mathrm{Attn}([e,h]) h+=u+MLP(u)h^{+}=u+\mathrm{MLP}(u)Early branch Δvearly=Attn([zlow,vearly−])\Delta v_{\mathrm{early}}=\mathrm{Attn}([z_{\mathrm{low}},v_{\mathrm{early}}^{-}]) vearly=12(vearly−+Δvearly)v_{\mathrm{early}}=\tfrac12(v_{\mathrm{early}}^{-}+\Delta v_{\mathrm{early}}) zmid=zlow+vearlyz_{\mathrm{mid}}=z_{\mathrm{low}}+v_{\mathrm{early}} Late branch Δvlate=MLP([zmid,vlate−])\Delta v_{\mathrm{late}}=\mathrm{MLP}([z_{\mathrm{mid}},v_{\mathrm{late}}^{-}]) vlate=12(vlate−+Δvlate)v_{\mathrm{late}}=\tfrac12(v_{\mathrm{late}}^{-}+\Delta v_{\mathrm{late}}) ztop=zmid+vlatez_{\mathrm{top}}=z_{\mathrm{mid}}+v_{\mathrm{late}}

The two-branch design draws on evidence that representations at different depths capture different aspects of language [1]. Modeling the early and late transitions separately lets each branch adapt its own direction of representation change during drafting.

As shown in Figure 1, adding stage differences to the stage-only drafter improves both Mean Acc and Late Acc under matched parameter budgets. This comparison shows that a lightweight drafter benefits from explicit access to stage transitions even when the underlying features are already available. Full RELAY provides further gains, particularly at later positions, supporting the value of recurrent transition guidance for longer draft rollouts.

Furthermore, we conduct zero-initialization and direct-prediction ablations to assess the roles of target-derived momentum initialization and explicit momentum propagation, respectively.

  • Zero momentum initialization retains the target stage features and initializes both momentum states to zero. This tests the value of a target-derived direction for the first draft update.
  • Direct momentum prediction lets each branch directly predict the current momentum, v=Fθ(z,v−)v=F_\theta(z,v^{-}), rather than a transition increment Δv\Delta v. Using the same branch inputs isolates the contribution of updating the previous momentum through a learned correction.
VariantMT-bench Speedup↑MT-bench τ\tau↑GSM8K Speedup↑GSM8K τ\tau↑
Zero momentum initialization4.23 ×\times6.794.28 ×\times6.67
Direct momentum prediction4.09 ×\times6.614.12 ×\times6.48
RELAY4.43×\times7.134.49×\times6.99

Target-derived initialization yields longer accepted sequences on both benchmarks, suggesting that changes observed across target layers provide useful guidance for drafting. Compared with direct momentum prediction, explicit propagation translates longer accepted sequences into faster decoding. This suggests that carrying the previous transition direction into each update is useful as the context evolves, whereas a branch that directly predicts the current momentum must learn how much of that direction to retain.

In the revision, we report the decoding efficiency results in Section 5.4 (Table 3). We also clarify the role of stage transitions in Sections 3 and 4.1.

[1] Jawahar G, Sagot B, Seddah D. What Does BERT Learn about the Structure of Language? ACL, 2019.

  1. Inconsistent definition of low-stage features. In Figures 1(a) and 2, the low-stage feature is presented as a hidden representation extracted from a shallow layer of the target model. However, Equation 6 defines it as a learned projection of token embeddings and previous draft states, without adequately explaining the relationship between these representations or how the learned states remain aligned with the target model's intermediate features.

Answer: We appreciate the opportunity to clarify the relationship between target features and draft states. Target features initialize RELAY’s stage states, and Equation (6) defines the subsequent low-stage state updates during drafting. As described in Section 4.1 and shown in Figure 2, the target model’s low-, mid-, and top-stage features initialize the stage states over the verified prefix. Equation (6) specifies how the low-stage state evolves after the target-derived initialization. During rollout, it combines the current token embedding with the preceding mid- and top-stage states,

e_i, z_{i-1,\mathrm{mid}}^{(k-1)}, z_{i-1,\mathrm{top}}^{(k-1)} \right].$$ The resulting low-stage state serves as input to the early branch. The early and late branches then compose the current mid- and top-stage states using their updated momentum states. 3. > Insufficient motivation and ablation studies for the asymmetric Attention-MLP architecture. The authors do not adequately justify why attention is specifically used for early-stage transitions while an MLP is used for late-stage transitions. No architectural ablations comparing Attention-Attention, MLP-MLP, or MLP-Attention configurations are provided. **Answer:** We appreciate the Reviewer’s question about the roles of the two branches. RELAY *follows the familiar attention–MLP order of a Transformer block*. Early attention lets each draft position draw on the prefix, so the late MLP can refine an already contextualized representation for token prediction. *This role for the late MLP is consistent with the finding of Geva et al. [2]* that feed-forward layers associate patterns in their inputs with predictions over the output vocabulary. Using LLaMA-3.1-8B-Instruct as the target model, we **compare four branch configurations in terms of average acceptance length and end-to-end speedup on MT-bench and GSM8K**. Each pair lists the early branch first. |Branch configuration|MT-bench Speedup↑|MT-bench $\tau$↑|GSM8K Speedup↑|GSM8K $\tau$↑| |---|---|---|---|---| |MLP–MLP|2.67 $\times$|3.93|2.74 $\times$|3.82| |MLP–Attention|4.11 $\times$|6.60|4.14 $\times$|6.49| |Attention–Attention|4.02 $\times$|6.98|3.98 $\times$|**7.02**| |Attention–MLP (RELAY)|**4.43** $\times$|**7.13**|**4.49**$\times$|6.99| *Placing attention before the MLP yields longer accepted sequences than reversing their order, while replacing the late MLP with a second attention layer lowers end-to-end speedup on both benchmarks*. **These results support using early attention to incorporate prefix context and a late MLP to refine the contextualized representation efficiently**. In the revision, we report and analyze the architecture ablations in Section 5.4 (Table 3). [2] Geva M, Schuster R, Berant J, Levy O. Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP, 2021. 4. > Insufficient statistical analysis of the diagnostic experiments. The error bars in Figure 1(b) represent standard deviations across draft positions rather than uncertainty across independent training runs or test samples. Consequently, they do not establish whether the reported improvements are statistically reliable. **Answer:** We thank the Reviewer for this comment. Our diagnostic study uses the ShareGPT dataset, with approximately *50k conversations for training and 2.7k for testing*. This provides **broad coverage for comparing the variants** under the same data split, training protocol, and matched parameter budgets. **Following common practice in speculative decoding studies** [3–5], we *use Figure 1(b) as controlled diagnostic evidence for our design choices*, rather than as evidence of statistical significance. Beyond Figure 1, *RELAY consistently improves acceptance length and decoding speed* across four target models, five benchmarks, and both greedy and non-greedy decoding. The long-horizon and serving evaluations provide complementary evidence. **Together, these consistent findings support the empirical effectiveness of our design across the evaluated settings.** [3] Li Y, Wei F, Zhang C, Zhang H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test[C]. NeurIPS, 38, 2025. [4] Zhang L, Wang X, Huang Y, Xu R. Learning Harmonized Representations for Speculative Sampling[C]. ICLR, 2025. [5] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads[C]. ICML, 2024.
  1. Unclear training details and computational fairness. The paper does not clearly report the drafter parameter counts, training costs, or per-backbone training details, particularly for LLaMA-3.3-70B. The authors should explicitly clarify whether all RELAY drafters were independently trained and provide sufficient information to assess the computational fairness of comparisons with EAGLE-3.

Answer: We thank the Reviewer for this comment. Each RELAY drafter is independently trained for its corresponding target model. The per-backbone training configurations are available in our public implementation.

RELAY uses the same training datasets as EAGLE-3. As specified in Section 5, the chat-model drafters use ShareGPT and UltraChat-200K, with OpenThoughts-114k-Math additionally used for DeepSeek-R1-Distill-LLaMA-8B. Appendix A provides the training hyperparameters, including the optimizer, learning-rate schedule, batch size, sequence length, and rollout length.

We compare the drafter parameter counts below.

Target modelEAGLE-3 parameters (M)RELAY parameters (M)
Vicuna-13B638.5711.1
LLaMA-3.1-8B-Instruct424.9485.1
LLaMA-3.3-70B-Instruct1576.21861.7
DeepSeek-R1-Distill-LLaMA-8B424.9485.1

To ensure a fair inference comparison, we evaluate EAGLE-3’s official checkpoints on the same hardware. Appendix A documents the hardware and baseline settings.

  1. Inconsistent sources of baseline results. Appendix A states that several baseline results are directly taken from the EAGLE-3 paper, while EAGLE-3 and DFlash are evaluated using released checkpoints on the authors' hardware. Combining previously reported results with newly measured results introduces potential differences in evaluation environments and configurations, which should be clearly documented and controlled wherever possible.

Answer: We thank the Reviewer for this comment. To ensure a fair comparison with strong baselines, we evaluate EAGLE-3 and DFlash using their official checkpoints and implementations on the same hardware as RELAY. Following their recommended inference configurations, we obtain results comparable to those in the original reports. Appendix A documents the hardware and baseline configurations. For the remaining baselines, we use results reported in EAGLE-3 for the corresponding target models, benchmarks, and decoding settings. Combining new measurements with previously reported baseline results has precedent in the EAGLE series.

  1. Incomplete and questionable comparisons with DFlash and DSpark. Table 1 reports DFlash performing substantially worse than EAGLE-3, which is inconsistent with the relative performance reported in the original DFlash paper and the subsequent DSpark study. Although differences in experimental settings may explain this discrepancy, the authors provide no analysis of the reversed ranking or sufficient evidence that the comparison is fair. Furthermore, DFlash is evaluated on only one target backbone, and DSpark, a highly relevant recent baseline, is entirely absent from the experimental comparisons.

Answer: We thank the Reviewer for this comment. A fair comparison requires matching both the target backbone and the drafter training data. We use officially released baseline checkpoints, consistent with common practice in speculative decoding studies [6–8]. Among the available official DFlash checkpoints, only the LLaMA-3.1-8B-Instruct checkpoint matches EAGLE-3 and RELAY in both respects. The released DSpark checkpoints use different target backbones and training datasets, preventing a direct comparison under the same conditions.

We faithfully report the results measured using the official DFlash and EAGLE-3 checkpoints and implementations on the same hardware as RELAY, following their respective recommended inference configurations. Differences in inference backends and GPU models may contribute to the different relative performance. Appendix A documents our evaluation environment and baseline settings.

[6] Li Y, Wei F, Zhang C, Zhang H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test[C]. NeurIPS, 2025.

[7] Li Y, Wei F, Zhang C, Zhang H. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees[C]. EMNLP, 2024.

[8] Zhang L, Wang X, Huang Y, Xu R. Learning Harmonized Representations for Speculative Sampling[C]. ICLR, 2025.

Reviewer_sRxAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes RELAY that improves the token throughput and acceptance of draft model by using the momentum from the hidden state transitions of the target model.

Previous work such as EAGLE-3, uses hidden state representation from the target model; however it doesnt model the transition between the hidden states. But RELAY argues that these transition vectors (vector differences between low, mid and top states of the target model) can be useful to initialize the early and late momentum recurrence. Then an attention layer and MLP layer predicts the adjustments to the momentum during drafting.

Results from various datasets and model families show consistent improvements over EAGLE-3 with upto ~30% higher throughput and sequence acceptance length.

Critical strengths
  • Novel architectural approach that considers hidden-state transitions from the target model as initial momentum states and then makes updates to them through learned attention and MLP modules during drafting.

  • The results are quite encouraging, with consistent gains across various model families and task types. The improvements particularly stand out on long-horizon drafting tasks. The experiments also extend to practical serving frameworks like SGLang.

Critical weaknesses
  • The contribution of momentum is not entirely clear. The terminology of momentum seems to be borrowed from optimizer update techniques like Adam, but the update mechanism is different. Adam's momentum update weighs the previous momentum and current gradient, i.e., βmt−1+(1−β)gt\beta m_{t-1} + (1-\beta)g_t , whereas RELAY imposes no such constraints. The attention/MLP layers could potentially produce corrections that entirely overwrite the momentum from the previous step. Therefore, it is unclear whether the improvements are coming from momentum propagation or the learned attention/MLP predictions.

  • There is no clear evidence supporting the choice of an attention layer for the early stage and an MLP for the later stage. It is unclear whether this particular architecture is necessary for the reported gains.

Questions for authors
  • Did you ablations by setting the initial momentum seed to be 0 instead of the difference in state transitions? This would help in understanding the contribution of the initial momentum.

  • Did you consider an ablation where the attention/MLP layers directly predict the momentum at the current step, rather than predicting a correction that is added to the previous momentum? This would help in isolating the contribution of momentum propagation.

4 comments
  • We sincerely thank the Reviewer for the careful reading of our paper and the constructive feedback. We greatly appreciate your positive assessment of RELAY’s architectural design, empirical results, and practical relevance. Your insightful questions have helped us clarify the role of momentum and strengthen the justification of our design choices. We have carefully considered each comment and provide point-by-point responses below.
  1. The terminology of momentum seems to be borrowed from optimizer update techniques like Adam, but the update mechanism is different. Adam's momentum update weighs the previous momentum and current gradient, i.e., βmt−1+(1−β)gt\beta m_{t-1}+(1-\beta)g_t, whereas RELAY imposes no such constraints.

Answer: We sincerely thank the Reviewer for this insightful comment! By a fortunate coincidence, your suggestion closely matches the weighted formulation we explored in RELAY’s earliest pilot experiments. We especially appreciate your perceptive observation and the valuable opportunity to present the experimental reasoning behind RELAY more fully.

Following the experimental setup described in Section 3, we used LLaMA-Instruct 3.1 8B as the target model and trained on ShareGPT to examine the weighted momentum update vi(k)=βvi−1(k−1)+(1−β)Δvi(k)v_{i}^{(k)}=\beta v_{i-1}^{(k-1)}+(1-\beta)\Delta v_{i}^{(k)} with β∈0.1,0.3,0.5,0.7,0.9\beta \in \\{0.1,0.3,0.5,0.7,0.9\\}. The equal-weight setting, β=0.5\beta=0.5, achieved the highest mean draft-token accuracy. This result motivated our choice of equal weighting between the previous momentum and the learned update.

We summarize these results below. Mean Acc averages accuracy over draft positions 1–10, while Late Acc averages over positions 6–10.

Momentum coefficient β\betaMean Acc (%)↑Late Acc (%)↑
0.171.6468.70
0.372.5870.14
0.572.9870.74
0.772.9070.76
0.972.5670.32

In the revision, we present the position-wise accuracy curves and discuss the effect of the momentum coefficient in Appendix B.

  1. Did you consider an ablation where the attention/MLP layers directly predict the momentum at the current step, rather than predicting a correction that is added to the previous momentum? This would help in isolating the contribution of momentum propagation.

Answer: We greatly appreciate the Reviewer’s thoughtful suggestion. We apologize that the scope of the comparison in Figure 1 was not sufficiently clear. The stage+transitions variant provides a related comparison without explicit momentum propagation. But its inputs differ from RELAY’s, as target stage features and their differences are supplied jointly to the drafter rather than used to initialize separate representation and momentum states. We therefore additionally evaluate your proposed control, corresponding to β=0\beta=0 in the weighted update you describe in Q1. Using LLaMA-Instruct 3.1 8B as the target model, we train the variant on the full training data described in Section 5 and evaluate acceptance length and end-to-end speedup on MT-bench and GSM8K.

VariantMT-bench Speedup↑MT-bench τ\tau↑GSM8K Speedup↑GSM8K τ\tau↑
w/o explicit momentum propagation (β=0\beta=0)4.09 ×\times6.614.12 ×\times6.48
RELAY (β=0.5\beta=0.5)4.43×\times7.134.49×\times6.99

Direct momentum prediction yields shorter accepted sequences and lower end-to-end speedups on MT-bench and GSM8K. These results support the effectiveness of explicit momentum propagation in faster decoding. To better understand these results, we provide an intuitive explanation using simplified notation. Omitting branch and position indices, RELAY computes

v_k=\beta v_{k-1}+(1-\beta)\Delta v_k,\qquad z_k^{+}=z_k+v_k,$$ **Explicit propagation carries the previous transition direction forward while allowing context-dependent adaptation, encouraging continuity across draft steps** in line with the acceptance-length and efficiency gains. **If replaced with direct momentum prediction, the predictor must learn how to retain or revise historical momentum entirely through its output**. This design *draws inspiration from Momentum ResNets* [1], which maintain a velocity state across residual layers to retain information about representation changes. RELAY applies this principle across draft steps, initializing momentum from target-model hidden transitions and adapting it as the generated context evolves. In the revision, we analyze decoding efficiency in Section 5.4 (Table 3). We thank the Reviewer again for helping us assess this mechanism more directly. [1] Sander M E, Ablin P, Blondel M, Peyré G. Momentum residual neural networks[C]. ICML, 2021: 9276–9287.
  1. Did you ablations by setting the initial momentum seed to be 0 instead of the difference in state transitions? This would help in understanding the contribution of the initial momentum.

Answer: We appreciate the Reviewer’s suggestion. With LLaMA-3.1-8B-Instruct as the target model, we train a variant with zero initialization of the early and late momentum states and evaluate its end-to-end speedup and average acceptance length on MT-bench and GSM8K.

VariantMT-bench Speedup↑MT-bench τ\tau↑GSM8K Speedup↑GSM8K τ\tau↑
Zero momentum initialization4.23 ×\times6.794.28 ×\times6.67
RELAY4.43 ×\times7.134.49×\times6.99

Compared with zero initialization, initializing momentum from target hidden transitions yields longer accepted sequences and higher end-to-end speedups on both MT-bench and GSM8K. These results suggest that the drafter benefits from a target-derived direction for its first update, informed by how representations change across depth on the verified prefix.

In the revision, we clarify this rationale in Section 4.1 and analyze the initialization ablation in Section 5.4 (Table 3).

  1. There is no clear evidence supporting the choice of an attention layer for the early stage and an MLP for the later stage. It is unclear whether this particular architecture is necessary for the reported gains.

Answer: We appreciate the Reviewer’s question about the roles of the two branches. RELAY follows the familiar attention–MLP order of a Transformer block. Early attention lets each draft position draw on the prefix, so the late MLP can refine an already contextualized representation for token prediction. This role for the late MLP is consistent with the finding of Geva et al. [2] that feed-forward layers associate patterns in their inputs with predictions over the output vocabulary.

Using LLaMA-3.1-8B-Instruct as the target model, we compare four branch configurations in terms of average acceptance length and end-to-end speedup on MT-bench and GSM8K. Each pair lists the early branch first.

Branch configurationMT-bench Speedup↑MT-bench τ\tau↑GSM8K Speedup↑GSM8K τ\tau↑
MLP–MLP2.67 ×\times3.932.74 ×\times3.82
MLP–Attention4.11 ×\times6.604.14 ×\times6.49
Attention–Attention4.02 ×\times6.983.98 ×\times7.02
Attention–MLP (RELAY)4.43 ×\times7.134.49×\times6.99

Placing attention before the MLP yields longer accepted sequences than reversing their order, while replacing the late MLP with a second attention layer lowers end-to-end speedup on both benchmarks. These results support using early attention to incorporate prefix context and a late MLP to refine the contextualized representation efficiently.

In the revision, we report and analyze the architecture ablations in Section 5.4 (Table 3).

[2] Geva M, Schuster R, Berant J, Levy O. Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP, 2021.

Thank you for running the initial seed momentum set to zero, and direct momentum prediction ablations. These confirm that the presence of momentum contributes to the overall gains. However, all the results are on LLaMa class of models and the finding of optimal beta of 0.5 was from your initial set of runs. so it is unclear if they will hold more generally for other tasks and models. I have raised my soundness score to 3, but keeping the overall rating.

We sincerely thank the Reviewer for the thoughtful follow-up and improved soundness assessment! We are deeply grateful for your careful consideration and recognition of the additional ablations. Your insightful suggestions have helped us strengthen the evidence for momentum’s contribution!

We agree that the preference for β=0.5\beta=0.5 was established in our initial diagnostic experiments. We adopted equal weighting for its simple formulation and strong empirical performance. Using the same coefficient across all targets avoids model-specific tuning and reduces the cost of reproduction.

We also appreciate your emphasis on architectural generalization. LLaMA-family models are widely used in speculative-decoding studies, including EAGLE-3 [1] and Medusa [2], making improvements within this family practically relevant. Our results show consistent gains across four targets and five benchmarks under greedy and non-greedy decoding. RELAY’s formulation does not rely on LLaMA-specific operations, and extending evaluation to other families, including Qwen, is an important next step. As additional results become available, we plan to release them together with the corresponding drafter checkpoints, providing reproducible resources for the community to build upon.

Thank you again for your thoughtful guidance and continued consideration of our work!

[1] Li Y, Wei F, Zhang C, Zhang H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS, 2025.

[2] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML, 2024.

Comments on the paper

2 comments
Area_Chair_djcSOpenReview ↗

A Gentle Reminder for Discussions

Dear Reviewers,

The ICLR 2027 discussion phase will end soon on November 18 at 11:59 pm AoE. After this phase, you will not be allowed to respond. Your help is crucial during this phase. If you haven't already, please review the rebuttal and provide concrete, justified feedback to the authors on whether it addressed your concerns. Active participation is critical to the conference and the authors.

Thanks for your contribution to ICLR 2027!

-AC

AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewer Concerns

Dear Chairs and Reviewers,

We sincerely thank you for the time and effort devoted to evaluating our submission. We greatly appreciate your thoughtful feedback and the opportunity to clarify our contribution and strengthen the experimental evidence.

The main shared concerns focus on:

  • Value of explicit transition modeling and momentum propagation (Reviewers sRxA, 6bVg, kPja): In Section 5.4, we supplement the existing parameter-matched diagnostic comparisons with zero-initialization and direct-momentum-prediction ablations. Together, these experiments support the value of both extracting target-derived transition guidance and recurrently adapting it during drafting.

  • Momentum update formulation (Reviewers sRxA, 6bVg): We present a momentum-coefficient sensitivity study in Appendix B, documenting the early design exploration that informed RELAY’s update rule. The selected equal-weight setting combines a simple formulation with strong empirical performance and is used consistently across all targets, avoiding model-specific tuning and simplifying reproduction.

  • Justification of the Attention–MLP design (Reviewers sRxA, kPja): We add comparisons with MLP–MLP, MLP–Attention, and Attention–Attention configurations in Section 5.4. These ablations support using early attention to incorporate prefix context and a late MLP to refine the resulting representation efficiently.

  • Generalization across model families (Reviewers sRxA, 6bVg, kPja): We clarify that the current evidence covers different model scales, training backgrounds, and tasks within the LLaMA family. These models are widely used in speculative-decoding studies, including EAGLE-3 [1] and Medusa [2], making improvements within this family practically relevant. In Appendix B, we extend the long-horizon analysis to Vicuna 13B and DeepSeek-R1-Distill-LLaMA-8B, observing consistent trends. Extending evaluation to other families, including Qwen, is an important next step.

We would also like to emphasize RELAY’s main contributions:

  • Formulating speculative drafting around hidden-transition momentum: RELAY is the first to formulate feature-guided speculative decoding around hidden-transition momentum. This shifts the focus from target features as static context to the representation changes they encode across depth, opening a complementary direction for exploiting target-model computation.

  • Introducing an architectural inductive bias for adaptive transition guidance: RELAY initializes momentum from target hidden transitions and predicts transition increments that adapt the carried momentum across draft steps. This explicit recurrence encourages continuity while allowing context-dependent adaptation, enabling target-derived guidance to evolve throughout drafting. The diagnostic experiments and additional initialization and propagation ablations support this design.

  • Demonstrating consistent and practical acceleration: Comprehensive experiments across four target models, five benchmarks, and serving settings demonstrate RELAY’s effectiveness. Compared with EAGLE-3, RELAY achieves up to approximately 30% improvements in decoding speed and average acceptance length, with long-horizon analyses and SGLang experiments further supporting its practical value.

All changes are highlighted in red\color{red}{\text{red}} in the revision. We hope these explanations and additional results address your concerns and assist your assessment of our work. Thank you again for your careful consideration and valuable guidance.

Best regards,
The Authors

[1] Li Y, Wei F, Zhang C, Zhang H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. NeurIPS, 2025.

[2] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.