- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to improve the prediction quality of draft models in speculative decoding. It proposes RELAY, which initializes two momentum states from the differences between hidden representations at different layers of the target model. These momentum states are recurrently updated across drafting steps through an Attention-MLP architecture to guide subsequent token predictions. The authors evaluate RELAY on four target models and five benchmarks under different decoding settings. Its effectiveness is assessed through comparisons with EAGLE-3 and other baselines, long-horizon drafting analyses, and practical serving experiments.
Critical strengths
S1. I think RELAY's early/late momentum design is an interesting extension of EAGLE-3. Rather than relying solely on feature fusion, RELAY explicitly models the differences between hidden features and updates them during drafting. The results in Figure 1 suggest that this design is effective, particularly at later draft positions.
S2. RELAY achieves fairly consistent performance improvements over EAGLE-3 across the tested models and benchmarks. For example, under greedy decoding, the mean speedup on LLaMA-3.3-70B increases from 4.36× to 4.77× (Table 1). I also find the long-horizon drafting analysis in Figure 3 valuable, as it directly examines the core problem that motivates the method. The inference serving experiments further suggest that these improvements are also meaningful in practical settings.
Critical weaknesses
W1. I think the motivation for explicitly modeling hidden-state transitions is not sufficiently justified. These transitions can already be computed from the multi-layer features used by EAGLE-3, so it remains unclear why explicitly modeling them would be more beneficial than learning them implicitly through feature fusion. Although Figure 1 provides some empirical evidence supporting the effectiveness of this design, the overall contribution still appears relatively incremental.
W2. The experiments cover a reasonable range of models and benchmarks, but all four target models are based on the LLaMA architecture. In addition, the long-horizon drafting analysis in Figure 3 is conducted only on LLaMA-3.1-8B using MT-bench. While the results are encouraging, I am not fully convinced that the method would provide similar benefits across other model architectures.
Questions for authors
Q1. Given that hidden-state transitions can be directly computed from the multi-layer features already utilized by EAGLE-3, could the authors further clarify what unique advantages explicit transition modeling and the recurrent momentum mechanism offer over existing feature fusion methods?
Q2. Given that all evaluated target models are based on the LLaMA architecture, could the authors further explain to what extent the current experimental evidence supports the generalizability of the proposed method to other model architectures?
Q3. Equations (8) and (11) describe an additive momentum update, whereas the publicly available code implementation uses torch.lerp with an interpolation factor of 0.5. Could the authors explain this discrepancy and clarify which update rule was actually used to obtain the reported experimental results?
4 comments
-
Given that all evaluated target models are based on the LLaMA architecture, could the authors further explain to what extent the current experimental evidence supports the generalizability of the proposed method to other model architectures? In addition, the long-horizon drafting analysis in Figure 3 is conducted only on LLaMA-3.1-8B using MT-bench.
Answer: We sincerely thank the Reviewer for this careful observation. Our evaluation covers four widely used open-weight backbones spanning chat and reasoning models, following the target-model selection in EAGLE-3 [2]. These models enable direct comparisons with established baselines across different scales and training backgrounds.
RELAY consistently improves acceptance length and decoding speed across these targets on five benchmarks under both greedy and non-greedy decoding. These results support its effectiveness across model scales, training backgrounds, and tasks within the LLaMA family.
RELAY’s formulation does not depend on LLaMA-specific operations. It derives guidance from intermediate hidden-state differences and adapts it through recurrent updates, making it applicable in principle to other decoder architectures with compatible representations. Evaluating its effectiveness beyond the LLaMA family is an important direction for future work. As additional results become available, we plan to release them together with the corresponding drafter checkpoints.
To further strengthen the evidence, we extend the analysis in Figure 3 to Vicuna 13B and DeepSeek-R1-Distill-LLaMA-8B. Both models exhibit consistent trends, with RELAY maintaining higher conditional acceptance rates at later draft positions and higher prefix survival rates over longer horizons.
We include these results and accompanying analysis in Appendix B of the revision.
[2] Li Y, Wei F, Zhang C, Zhang H. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test[C]. NeurIPS, 38, 2025.
-
Equations (8) and (11) describe an additive momentum update, whereas the publicly available code implementation uses torch.lerp with an interpolation factor of 0.5. Could the authors explain this discrepancy and clarify which update rule was actually used to obtain the reported experimental results?
Answer: We sincerely thank the Reviewer for carefully examining our implementation! We apologize for the discrepancy between the equations and the code. All experimental results in the submitted paper were obtained using torch.lerp with an interpolation factor of 0.5 in both branches.
As detailed in our response to Reviewer sRxA’s Q1, this choice arose from our early design experiments. Our initial implementation used an Adam-like weighted formulation,
=\beta v_{i-1}^{(k-1)} +(1-\beta)\Delta v_i^{(k)}.$$ **$\beta=0.5$ achieved the highest mean draft-token accuracy** among the evaluated settings, motivating equal weighting. We include the position-wise accuracy curves and accompanying analysis in Appendix B of the revision. **To emphasize the equal contributions of the previous momentum and the learned update, we presented a simplified residual-style expression but omitted the required scaling factor**. *We have corrected Equations (8) and (11) in the revision* to match the implementation, $$v_{i,\mathrm{early}}^{(k)} =\frac{1}{2}\left( v_{i-1,\mathrm{early}}^{(k-1)} +\Delta v_{i,\mathrm{early}}^{(k)} \right),$$ $$v_{i,\mathrm{late}}^{(k)} =\frac{1}{2}\left( v_{i-1,\mathrm{late}}^{(k-1)} +\Delta v_{i,\mathrm{late}}^{(k)} \right).$$ This correction leaves the implementation and reported results unchanged. We thank the Reviewer again for helping us improve the precision of our presentation.Thanks for your rebuttal and I will maintain my positive score.
We sincerely thank the Reviewer for the follow-up and continued positive assessment of our work! We especially appreciate your perceptive observations and careful examination of both our manuscript and implementation. Your insightful questions helped us clarify RELAY’s contribution and present it more precisely. They also guided us in strengthening the experimental evidence.
We are grateful for your constructive engagement throughout the discussion and your support for our work!
Answer: We appreciate the opportunity to clarify our contribution. EAGLE-3 gives the drafter access to hidden features from several target layers through feature fusion. The differences between successive selected features, which we call stage transitions, capture how the target representation changes across depth at each prefix position. RELAY recurrently carries forward and adapts these transitions to guide drafting, introducing an architectural inductive bias toward continuity across draft steps rather than additional target information.
The table summarizes how the two methods use target features and compute draft representations. We omit normalization and position and draft-step indices for clarity. Brackets denote concatenation, and e is the current token embedding. The superscripts (0) and − denote initialization from target features and the preceding draft state, respectively.
The two-branch design draws on evidence that representations at different depths capture different aspects of language [1]. Modeling the early and late transitions separately lets each branch adapt its own direction of representation change during drafting.
As shown in Figure 1, adding stage differences to the stage-only drafter improves both Mean Acc and Late Acc under matched parameter budgets. This comparison shows that a lightweight drafter benefits from explicit access to stage transitions even when the underlying features are already available. Full RELAY provides further gains, particularly at later positions, supporting the value of recurrent transition guidance for longer draft rollouts.
Furthermore, we conduct zero-initialization and direct-prediction ablations to assess the roles of target-derived momentum initialization and explicit momentum propagation, respectively.
Target-derived initialization yields longer accepted sequences on both benchmarks, suggesting that changes observed across target layers provide useful guidance for drafting. Compared with direct momentum prediction, explicit propagation translates longer accepted sequences into faster decoding. This suggests that carrying the previous transition direction into each update is useful as the context evolves, whereas a branch that directly predicts the current momentum must learn how much of that direction to retain.
In the revision, we report the decoding efficiency results in Section 5.4 (Table 3). We also clarify the role of stage transitions in Sections 3 and 4.1.
[1] Jawahar G, Sagot B, Seddah D. What Does BERT Learn about the Structure of Language? ACL, 2019.