- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EchoSpec integrates drafting into a frozen target model through mask-position attention output biases (O-bias) and a lightweight post-target refinement block conditioned on the bonus token. It reduces adaptation requirements and avoids a separate draft KV cache, improving resource efficiency and throughput in long-context, high-concurrency serving.
Critical strengths
1.Important practical problem. The paper addresses both target-specific adaptation costs and the serving memory overhead of separate drafters. 2. Simple and well-motivated design. Systematic comparisons support the choice of output-projection adaptation, while post-target refinement addresses the missing bonus-token information. 3. Meaningful serving evidence. SGLang experiments cover three targets, multiple concurrency levels, and two sampling temperatures. 4。 4. EchoSpec retains nearly the same KV capacity as AR and reports higher average throughput than DFlash at high concurrency. Relevant controlled comparisons. Evaluations against Apple MTP and PaSS under matched training conditions strengthen the empirical contribution.
Critical weaknesses
- Moderate novelty. Target-integrated masked drafting and draft-position adaptation have precedents. The contribution lies primarily in the minimal O-bias design, refinement mechanism, and demonstrated serving efficiency.
- Limited incremental benefit from refinement. The refinement block accounts for over 99% of trainable parameters, but provides only modest throughput gains on Qwen3 and largely neutral gains on Llama. The current ablation does not separate the benefit of bonus-token information from additional model capacity.
- Incomplete cost and uncertainty reporting. Sample counts and trainable parameters establish resource efficiency, but training GPU hours and token counts would better substantiate training-cost claims. Repeated timing measurements are particularly important for the small refinement gains.
- Evaluation scope is relatively narrow. Serving experiments focus on two long-document summarization workloads with overlap scheduling disabled. The results support this setting, while broader deployment benefits remain less established.
Questions for authors
1.Does refinement still help when the same block is retained but bonus-token conditioning is removed? 2.Can the authors provide training-cost measurements and timing variability? 3.Please clarify the tree verification and rejection-sampling procedure used to preserve the target distribution at temperature 1.
4 comments
W4. Extending evaluation scope.
We thank the reviewer for this comment. To broaden the evaluation scope, we conducted additional experiments on an A100 (40GB) GPU using two code benchmarks: LongCodeCompletion and RepoBench-P. The two tables below report output throughput (tokens/s) across different concurrency levels, where 'c' denotes the concurrency level.
4K-8K
| Configuration | Method | LCC c=1 | LCC c=4 | LCC c=8 | LCC c=16 | LCC c=32 | RB c=1 | RB c=4 | RB c=8 | RB c=16 | RB c=32 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | AR | 25.8 | 35.8 | 37.9 | 37.9 | 37.8 | 25.5 | 35.3 | 37.1 | 37.1 | 37.1 |
| DFlash | 29.9 | 37.3 | 37.6 | 37.7 | 37.6 | 28.8 | 36.4 | 36.3 | 36.4 | 36.4 | |
| EAGLE-3 | 27.2 | 34.0 | 35.0 | 35.0 | 35.0 | 27.0 | 33.5 | 34.5 | 34.5 | 34.5 | |
| EchoSpec | 30.9 | 38.1 | 39.3 | 39.3 | 39.3 | 30.1 | 37.4 | 38.4 | 38.5 | 38.5 | |
| Qwen3-4B | AR | 38.1 | 51.5 | 54.8 | 55.9 | 55.9 | 37.6 | 50.8 | 54.0 | 55.1 | 55.1 |
| DFlash | 42.1 | 53.2 | 55.2 | 55.1 | 55.1 | 42.1 | 52.8 | 54.8 | 54.5 | 54.5 | |
| EAGLE-3 | 41.2 | 49.7 | 51.9 | 52.1 | 52.0 | 40.8 | 49.5 | 51.2 | 51.5 | 51.4 | |
| EchoSpec | 43.6 | 54.3 | 56.7 | 56.8 | 56.8 | 42.8 | 53.6 | 55.8 | 55.9 | 55.9 | |
| Llama-3.1-8B-Instruct | AR | 26.6 | 36.6 | 39.5 | 39.7 | 39.7 | 26.5 | 36.5 | 39.1 | 39.0 | 38.9 |
| DFlash | 32.3 | 39.5 | 40.1 | 40.1 | 40.2 | 32.0 | 39.1 | 39.7 | 39.6 | 39.7 | |
| EAGLE-3 | 22.0 | 30.6 | 32.2 | 32.2 | 32.2 | 21.9 | 30.5 | 31.8 | 31.8 | 31.8 | |
| EchoSpec | 32.4 | 39.7 | 41.1 | 41.0 | 41.0 | 32.3 | 39.1 | 40.5 | 40.8 | 40.8 |
8K+
| Configuration | Method | LCC c=1 | LCC c=4 | LCC c=8 | LCC c=16 | LCC c=32 | RB c=1 | RB c=4 | RB c=8 | RB c=16 | RB c=32 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | AR | 16.4 | 20.0 | 20.1 | 20.1 | 20.1 | 14.7 | 17.4 | 17.4 | 17.4 | 17.4 |
| DFlash | 17.3 | 19.8 | 19.8 | 19.8 | 19.8 | 15.1 | 16.9 | 16.9 | 16.9 | 17.0 | |
| EAGLE-3 | 16.7 | 18.8 | 18.8 | 18.9 | 18.8 | 14.8 | 16.1 | 15.9 | 16.1 | 16.2 | |
| EchoSpec | 18.4 | 20.9 | 21.0 | 20.9 | 20.7 | 16.2 | 18.1 | 18.1 | 18.0 | 18.1 | |
| Qwen3-4B | AR | 22.9 | 27.4 | 28.2 | 28.2 | 28.2 | 20.1 | 23.6 | 24.1 | 24.1 | 24.2 |
| DFlash | 23.7 | 27.6 | 27.9 | 27.9 | 27.9 | 20.9 | 23.8 | 23.9 | 23.9 | 23.9 | |
| EAGLE-3 | 23.7 | 26.4 | 26.7 | 26.8 | 25.9 | 20.6 | 22.7 | 22.9 | 22.9 | 22.9 | |
| EchoSpec | 25.0 | 28.5 | 29.0 | 29.0 | 29.0 | 21.9 | 24.5 | 24.7 | 24.7 | 24.7 | |
| Llama-3.1-8B-Instruct | AR | 17.3 | 21.2 | 21.6 | 21.6 | 21.6 | 15.4 | 18.5 | 18.7 | 18.6 | 18.6 |
| DFlash | 19.5 | 21.9 | 21.9 | 21.9 | 21.9 | 17.0 | 18.8 | 18.8 | 18.8 | 18.8 | |
| EAGLE-3 | 14.7 | 17.9 | 17.8 | 17.9 | 17.6 | 13.1 | 15.5 | 15.4 | 15.3 | 15.4 | |
| EchoSpec | 19.8 | 22.2 | 22.4 | 22.4 | 22.4 | 17.2 | 19.2 | 19.3 | 19.3 | 19.3 |
(Abbreviations used: LCC for LongCodeCompletion; RB for RepoBench-P)
As shown in the two tables, EchoSpec consistently achieves the highest output throughput across both code-oriented workloads and context-length ranges. Specifically, at high concurrency (c=16,32), EchoSpec outperforms DFlash, EAGLE-3, and AR by 3.7%, 15.8%, and 3.2% on average, respectively. These results demonstrate that EchoSpec's throughput advantage extends beyond long-document summarization to code-related tasks. We have also added these experimental results to Appendix C of the revised manuscript.
Furthermore, regarding the concern about disabled overlap scheduling, we would like to clarify that SGLang's overlap scheduler primarily pipelines CPU-side batch scheduling with GPU model execution. Disabling this optimization does not alter the core GPU-side operations of speculative decoding. Since GPU computation and memory usage remain key factors in speculative decoding performance, their effects can still be evaluated independently of CPU-GPU overlap. By consistently disabling overlap scheduling across all methods, we ensure a controlled comparison under identical scheduler settings. Therefore, although overlap scheduling can improve absolute serving throughput, we believe our experiments still provide a meaningful, apples-to-apples comparison of the underlying speculative decoding methods.
Recommendation After the Author Response
Thank you for your response. It addressed my questions, and I appreciate the clarifications. My original assessment remains unchanged, and I continue to lean toward acceptance.
We sincerely thank the reviewer for their thoughtful feedback and for taking the time to consider our response. We are glad that our answers addressed your questions, and we appreciate your continued positive assessment. If any further questions or concerns arise, we would be happy to provide additional clarification.
We thank the reviewer for their thoughtful comment. We would like to clarify that EchoSpec differs fundamentally from prior target-integrated methods in how it adapts the target model. In addition, we provide a systematic investigation into the practical advantages of target-integrated drafting under long-context and high-concurrency serving conditions.
Existing target-integrated methods such as PaSS and BiTA rely on additional learnable input-side components, while Apple MTP modifies the target's internal computation through gated low-rank updates. In contrast, EchoSpec directly injects a learnable bias into the attention output projection at draft positions. This enables future-token prediction with minimal target adaptation while preserving exact verification behavior and avoiding a separate drafter model, outperforming both target-integrated baselines (Apple MTP and PaSS) and standalone transformer drafters (EAGLE-3 and DFlash).
Moreover, we systematically investigate which attention projection is most effective for future-token prediction. As shown in Figures 3 and 4 of the revised manuscript, applying the bias exclusively to the attention output projection WO achieves the highest parameter efficiency, and adapting WO alone yields a higher proxy acceptance length than adapting WQ, WK, and WV jointly. These results demonstrate that the choice of adaptation site plays an important role in effective target-integrated drafting.
In addition, we identify an inherent information gap in parallel target-integrated drafting. Since draft positions are processed during the target's forward pass, they cannot access the bonus token generated after verification. To bridge this gap, we introduce a lightweight post-target refinement block that conditions on the bonus token to refine draft representations along the accepted path.
Furthermore, our work provides a systematic evaluation of speculative decoding under long-context and high-concurrency serving regimes, which, to the best of our knowledge, has not been thoroughly examined in prior work. We analyze KV-cache token capacity and examine request queueing and output throughput as concurrency scales. Our results in Section 4 demonstrate that eliminating a separate drafter and its dedicated KV cache provides substantial benefits under long-context and high-concurrency workloads.
Therefore, we believe that EchoSpec contributes not only a novel, lightweight adaptation mechanism but also valuable insights into architectural adaptation choices and practical serving efficiency.
We kindly refer the reviewer to our General Response for a detailed analysis of refinement block.
We kindly refer the reviewer to our General Response for a detailed analysis of training-cost.
We thank the reviewer for this comment. Because our draft tree is constructed without sampling from a draft distribution, we use target-only verification. Starting from the root node, at each parent node we perform a single categorical sampling trial in which each distinct candidate child token is assigned its conditional target probability, and the remaining probability mass is assigned to a residual outcome. If a child is selected, verification advances to that node. Otherwise, all candidate children are rejected, and we sample a bonus token from the renormalized residual target distribution (obtained by setting the candidate children's probabilities to zero). If verification reaches a leaf node, we sample the bonus token directly from the full conditional target distribution. This procedure preserves the target distribution regardless of how the draft tree is constructed. Similar target-only tree verification schemes have been adopted in Sequoia [1] and SpecExec [2]. In our implementation, we use SGLang's EAGLE tree-sampling function
sglang.srt.speculative.eagle_utils.eagle_sample[3, 4].We have also added this explanation to Appendix A of the revised manuscript.
Reference
[1] SEQUOIA: Scalable and Robust Speculative Decoding (NeurIPS 2024)
[2] SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024)
[3] SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024)
[4] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025)