acceptodds
Under review as a conference paper at ICLR 2027

EchoSpec: Efficient Target-Integrated Speculative Decoding via Output-Projection Adaptation and Post-Target Refinement

13 views

Abstract

Autoregressive decoding is a major bottleneck in large language model inference due to its sequential and memory-bandwidth-bound nature. Speculative decoding alleviates this bottleneck by drafting multiple future tokens for parallel verification. While recent methods improve draft quality using separate auxiliary transformers, these models introduce target-specific parameters and significant training costs. Furthermore, the auxiliary weights and KV caches increase serving memory, particularly at high concurrency. In this paper, we introduce EchoSpec, a target-integrated speculative decoding framework that reuses the frozen target model for drafting with minimal adaptation. At each layer, we apply a learned bias exclusively to the attention output projection at draft positions. This approach is highly parameter-efficient compared to broadly adapting the target model. Nonetheless, drafting in parallel with verification prevents the draft positions from accessing the bonus token. To resolve the resulting information gap, we introduce a lightweight post-target refinement block in EchoSpec rather than scaling up target-side adaptation. As a result, EchoSpec requires only 2.2-4.5K target-generated training samples, features 27.9-32.1x fewer trainable parameters than DFlash, and eliminates the need for a separate draft KV cache. Under long-context, high-concurrency workloads, it reduces the active memory footprint and delivers higher output throughput than separate transformer drafters such as DFlash and EAGLE-3.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Speculative Decoding, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KtVoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EchoSpec integrates drafting into a frozen target model through mask-position attention output biases (O-bias) and a lightweight post-target refinement block conditioned on the bonus token. It reduces adaptation requirements and avoids a separate draft KV cache, improving resource efficiency and throughput in long-context, high-concurrency serving.

Critical strengths

1.Important practical problem. The paper addresses both target-specific adaptation costs and the serving memory overhead of separate drafters. 2. Simple and well-motivated design. Systematic comparisons support the choice of output-projection adaptation, while post-target refinement addresses the missing bonus-token information. 3. Meaningful serving evidence. SGLang experiments cover three targets, multiple concurrency levels, and two sampling temperatures. 4。 4. EchoSpec retains nearly the same KV capacity as AR and reports higher average throughput than DFlash at high concurrency. Relevant controlled comparisons. Evaluations against Apple MTP and PaSS under matched training conditions strengthen the empirical contribution.

Critical weaknesses
  1. Moderate novelty. Target-integrated masked drafting and draft-position adaptation have precedents. The contribution lies primarily in the minimal O-bias design, refinement mechanism, and demonstrated serving efficiency.
  2. Limited incremental benefit from refinement. The refinement block accounts for over 99% of trainable parameters, but provides only modest throughput gains on Qwen3 and largely neutral gains on Llama. The current ablation does not separate the benefit of bonus-token information from additional model capacity.
  3. Incomplete cost and uncertainty reporting. Sample counts and trainable parameters establish resource efficiency, but training GPU hours and token counts would better substantiate training-cost claims. Repeated timing measurements are particularly important for the small refinement gains.
  4. Evaluation scope is relatively narrow. Serving experiments focus on two long-document summarization workloads with overlap scheduling disabled. The results support this setting, while broader deployment benefits remain less established.
Questions for authors

1.Does refinement still help when the same block is retained but bonus-token conditioning is removed? 2.Can the authors provide training-cost measurements and timing variability? 3.Please clarify the tree verification and rejection-sampling procedure used to preserve the target distribution at temperature 1.

4 comments

W1. Novelty of EchoSpec

We thank the reviewer for their thoughtful comment. We would like to clarify that EchoSpec differs fundamentally from prior target-integrated methods in how it adapts the target model. In addition, we provide a systematic investigation into the practical advantages of target-integrated drafting under long-context and high-concurrency serving conditions.

Existing target-integrated methods such as PaSS and BiTA rely on additional learnable input-side components, while Apple MTP modifies the target's internal computation through gated low-rank updates. In contrast, EchoSpec directly injects a learnable bias into the attention output projection at draft positions. This enables future-token prediction with minimal target adaptation while preserving exact verification behavior and avoiding a separate drafter model, outperforming both target-integrated baselines (Apple MTP and PaSS) and standalone transformer drafters (EAGLE-3 and DFlash).

Moreover, we systematically investigate which attention projection is most effective for future-token prediction. As shown in Figures 3 and 4 of the revised manuscript, applying the bias exclusively to the attention output projection WOW_O achieves the highest parameter efficiency, and adapting WOW_O alone yields a higher proxy acceptance length than adapting WQW_Q, WKW_K, and WVW_V jointly. These results demonstrate that the choice of adaptation site plays an important role in effective target-integrated drafting.

In addition, we identify an inherent information gap in parallel target-integrated drafting. Since draft positions are processed during the target's forward pass, they cannot access the bonus token generated after verification. To bridge this gap, we introduce a lightweight post-target refinement block that conditions on the bonus token to refine draft representations along the accepted path.

Furthermore, our work provides a systematic evaluation of speculative decoding under long-context and high-concurrency serving regimes, which, to the best of our knowledge, has not been thoroughly examined in prior work. We analyze KV-cache token capacity and examine request queueing and output throughput as concurrency scales. Our results in Section 4 demonstrate that eliminating a separate drafter and its dedicated KV cache provides substantial benefits under long-context and high-concurrency workloads.

Therefore, we believe that EchoSpec contributes not only a novel, lightweight adaptation mechanism but also valuable insights into architectural adaptation choices and practical serving efficiency.

W2 & Q1. Necessity of the refinement block and bonus-token conditioning.

We kindly refer the reviewer to our General Response for a detailed analysis of refinement block.

W3 & Q2. Training cost.

We kindly refer the reviewer to our General Response for a detailed analysis of training-cost.

Q3. Target-only tree verification and distribution preservation.

We thank the reviewer for this comment. Because our draft tree is constructed without sampling from a draft distribution, we use target-only verification. Starting from the root node, at each parent node we perform a single categorical sampling trial in which each distinct candidate child token is assigned its conditional target probability, and the remaining probability mass is assigned to a residual outcome. If a child is selected, verification advances to that node. Otherwise, all candidate children are rejected, and we sample a bonus token from the renormalized residual target distribution (obtained by setting the candidate children's probabilities to zero). If verification reaches a leaf node, we sample the bonus token directly from the full conditional target distribution. This procedure preserves the target distribution regardless of how the draft tree is constructed. Similar target-only tree verification schemes have been adopted in Sequoia [1] and SpecExec [2]. In our implementation, we use SGLang's EAGLE tree-sampling function sglang.srt.speculative.eagle_utils.eagle_sample [3, 4].

We have also added this explanation to Appendix A of the revised manuscript.


Reference

[1] SEQUOIA: Scalable and Robust Speculative Decoding (NeurIPS 2024)

[2] SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024)

[3] SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024)

[4] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025)

W4. Extending evaluation scope.

We thank the reviewer for this comment. To broaden the evaluation scope, we conducted additional experiments on an A100 (40GB) GPU using two code benchmarks: LongCodeCompletion and RepoBench-P. The two tables below report output throughput (tokens/s) across different concurrency levels, where 'c' denotes the concurrency level.

4K-8K

ConfigurationMethodLCC c=1LCC c=4LCC c=8LCC c=16LCC c=32RB c=1RB c=4RB c=8RB c=16RB c=32
Qwen3-8BAR25.835.837.937.937.825.535.337.137.137.1
DFlash29.937.337.637.737.628.836.436.336.436.4
EAGLE-327.234.035.035.035.027.033.534.534.534.5
EchoSpec30.938.139.339.339.330.137.438.438.538.5
Qwen3-4BAR38.151.554.855.955.937.650.854.055.155.1
DFlash42.153.255.255.155.142.152.854.854.554.5
EAGLE-341.249.751.952.152.040.849.551.251.551.4
EchoSpec43.654.356.756.856.842.853.655.855.955.9
Llama-3.1-8B-InstructAR26.636.639.539.739.726.536.539.139.038.9
DFlash32.339.540.140.140.232.039.139.739.639.7
EAGLE-322.030.632.232.232.221.930.531.831.831.8
EchoSpec32.439.741.141.041.032.339.140.540.840.8

8K+

ConfigurationMethodLCC c=1LCC c=4LCC c=8LCC c=16LCC c=32RB c=1RB c=4RB c=8RB c=16RB c=32
Qwen3-8BAR16.420.020.120.120.114.717.417.417.417.4
DFlash17.319.819.819.819.815.116.916.916.917.0
EAGLE-316.718.818.818.918.814.816.115.916.116.2
EchoSpec18.420.921.020.920.716.218.118.118.018.1
Qwen3-4BAR22.927.428.228.228.220.123.624.124.124.2
DFlash23.727.627.927.927.920.923.823.923.923.9
EAGLE-323.726.426.726.825.920.622.722.922.922.9
EchoSpec25.028.529.029.029.021.924.524.724.724.7
Llama-3.1-8B-InstructAR17.321.221.621.621.615.418.518.718.618.6
DFlash19.521.921.921.921.917.018.818.818.818.8
EAGLE-314.717.917.817.917.613.115.515.415.315.4
EchoSpec19.822.222.422.422.417.219.219.319.319.3

(Abbreviations used: LCC for LongCodeCompletion; RB for RepoBench-P)

As shown in the two tables, EchoSpec consistently achieves the highest output throughput across both code-oriented workloads and context-length ranges. Specifically, at high concurrency (c=16,32), EchoSpec outperforms DFlash, EAGLE-3, and AR by 3.7%, 15.8%, and 3.2% on average, respectively. These results demonstrate that EchoSpec's throughput advantage extends beyond long-document summarization to code-related tasks. We have also added these experimental results to Appendix C of the revised manuscript.

Furthermore, regarding the concern about disabled overlap scheduling, we would like to clarify that SGLang's overlap scheduler primarily pipelines CPU-side batch scheduling with GPU model execution. Disabling this optimization does not alter the core GPU-side operations of speculative decoding. Since GPU computation and memory usage remain key factors in speculative decoding performance, their effects can still be evaluated independently of CPU-GPU overlap. By consistently disabling overlap scheduling across all methods, we ensure a controlled comparison under identical scheduler settings. Therefore, although overlap scheduling can improve absolute serving throughput, we believe our experiments still provide a meaningful, apples-to-apples comparison of the underlying speculative decoding methods.

Reviewer_KtVoOpenReview ↗

Recommendation After the Author Response

Thank you for your response. It addressed my questions, and I appreciate the clarifications. My original assessment remains unchanged, and I continue to lean toward acceptance.

We sincerely thank the reviewer for their thoughtful feedback and for taking the time to consider our response. We are glad that our answers addressed your questions, and we appreciate your continued positive assessment. If any further questions or concerns arise, we would be happy to provide additional clarification.

Reviewer_rAfLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EchoSpec uses the target model itself to generate drafts, avoiding a separate draft model and its KV cache. It adds a small bias and a refinement block to improve draft quality, and evaluates throughput and memory use across several models and serving settings.

Critical strengths
  1. The paper is clearly written, and the overall approach is easy to understand.

  2. The design is simple and addresses the memory cost of separate draft models, which is useful for long-context serving.

Critical weaknesses
  1. The refinement block adds many parameters but brings only small gains. The ablation does not clearly separate the benefit of bonus-token information from the added model capacity.

  2. The claim of lower adaptation cost needs measurements of training time and memory use. Fewer samples and parameters alone do not establish this.

  3. The throughput advantage depends on the workload and can disappear at high concurrency. The paper should make clearer when EchoSpec is worth using.

Questions for authors
  1. How much does refinement help without bonus-token information?

  2. What are the total adaptation time and peak memory use, including data generation, compared with a separate drafter?

  3. When does EchoSpec outperform both AR and a separate trained drafter?

4 comments

W1 & Q1. Necessity of the refinement block and bonus-token conditioning.

We kindly refer the reviewer to our General Response for a detailed analysis of the refinement block.

W2 & Q2. Training cost.

We kindly refer the reviewer to our General Response for a detailed analysis of training cost.

W3 & Q3. Throughput trade-offs across concurrency and context length

We thank the reviewer for this comment. We agree that EchoSpec's throughput advantage depends on the concurrency level, context length, and available GPU memory. Below, we explain when EchoSpec outperforms AR and separate-drafter baselines such as DFlash and EAGLE-3.

When EchoSpec outperforms AR.

AR decoding processes one token per target forward pass, whereas EchoSpec processes multiple candidate positions within a single verification pass. While this verification window allows EchoSpec to commit multiple tokens per target forward pass, it also increases computational intensity and makes execution more compute-bound as concurrency grows. Thus, before computation becomes the dominant bottleneck, EchoSpec benefits from this higher token yield per target forward pass. For example, as shown in Figure 5 of the revised manuscript, EchoSpec outperforms AR on Qwen3-8B (8K+) at concurrencies 8-24 and on Qwen3-4B (4K-8K) at concurrencies 8-16, whereas AR catches up at concurrencies 28-32 and 20-32, respectively. Therefore, EchoSpec outperforms AR when the throughput gain from multi-token verification outweighs its additional computational cost.

When EchoSpec outperforms separate-drafter baselines.

A primary factor differentiating EchoSpec's serving scalability from that of separate-drafter baselines such as DFlash and EAGLE-3 is memory footprint. As shown in Table 4 of the revised manuscript, DFlash and EAGLE-3 retain only 83.283.2-93.993.9% of AR's KV-cache token capacity, whereas EchoSpec retains 99.899.8-99.999.9%. This gap directly determines how many requests can be served concurrently under a fixed memory budget. Because 8K+ requests consume substantially more KV-cache memory than 4K-8K requests, high concurrency in the 8K+ regime saturates the smaller KV-cache pool of separate drafters and restricts the effective batch size. Thus, as shown in Figure 5 of the revised manuscript, EchoSpec achieves a larger relative throughput advantage over DFlash in the 8K+ setting at high concurrency, whereas the difference is smaller at low concurrencies (e.g., 1 and 4). Overall, these results show that EchoSpec's memory efficiency translates into higher throughput when long-context workloads place heavy pressure on KV-cache capacity.

Thank you for the additional ablations and expanded serving experiments. These results clarify the contribution of bonus-token conditioning and strengthen the evidence for serving efficiency. However, the refinement block’s practical benefit remains limited relative to its added complexity, and the adaptation-cost comparison still excludes data generation and preprocessing. The serving experiments better establish where the method helps, but its advantage remains confined to particular workload and resource conditions. These additions strengthen the paper, but do not sufficiently resolve my main concerns. I therefore maintain my original score.

We thank the reviewer for carefully considering our additional experiments and for acknowledging that they strengthen the paper. We respect the reviewer's assessment and would like to offer a few additional clarifications regarding the remaining concerns.

Overall contribution beyond the refinement block.

We respectfully believe that EchoSpec's contributions extend beyond the incremental performance gains of the refinement block. In particular, we would like to emphasize that its primary contributions lie in the minimal target-integrated drafting mechanism and the resulting practical serving advantages.

Recent speculative decoding methods using separate transformer drafters (e.g., DFlash and EAGLE-3) introduce additional parameters and KV caches, reducing the memory available for target KV caches. Under long-context and high-concurrency workloads, this can increase request queueing and cause throughput to fall below AR decoding. In contrast, EchoSpec preserves nearly the same KV-cache capacity as AR, maintaining throughput advantages where separate-drafter methods lose their benefits. Therefore, EchoSpec demonstrates that target-integrated drafting is a practical alternative and, to the best of our knowledge, provides the first systematic evaluation of its serving advantages under long-context, high-concurrency workloads.

Moreover, unlike PaSS and BiTA, which rely on learnable input-side components, or Apple MTP, which uses gated low-rank updates, EchoSpec enables target-integrated drafting through a minimal bias applied to the frozen target's internal computation. Specifically, Figures 3 and 4 show that adapting WOW_O alone achieves the highest parameter efficiency, even yielding a higher proxy acceptance length than jointly adapting WQW_Q, WKW_K, and WVW_V.

We believe these findings provide valuable insights into both target-integrated drafting and practical serving efficiency. Together, they establish EchoSpec as a meaningful alternative approach for speculative decoding, supported by substantial serving experiments.

Training and adaptation cost comparison.

We agree that a fair end-to-end comparison of adaptation costs should account for data generation and preprocessing. However, we would like to emphasize the substantial difference in the amount of training data required. For example, EchoSpec and DFlash select training data from NVIDIA Nemotron Post-Training Dataset V2 and CodeAlpaca. Specifically, EchoSpec randomly selects 4.5K samples for Qwen3-8B and 2.2K for Qwen3-4B from these datasets without any specialized sampling strategy. DFlash uses approximately 800K samples from the same datasets. Therefore, DFlash uses approximately 178x and 364x as many training samples as EchoSpec for Qwen3-8B and Qwen3-4B, respectively.

As the reviewer pointed out, we agree that the number of training samples alone does not directly determine computational cost or training latency. These costs depend not only on the number of samples but also on which samples are selected, the generation hyperparameters, and the computational environment. However, DFlash does not report which specific samples were selected from the source datasets. This makes it difficult to establish a fully matched comparison of data generation and preprocessing costs. We therefore avoid making assumptions about these costs that could lead to misleading comparisons.

Nevertheless, the reported sample counts alone demonstrate that EchoSpec requires hundreds of times fewer target-generated training sequences despite using the same source datasets as DFlash. Under comparable generation conditions, this substantial reduction would also decrease the amount of data that must be generated and processed. We believe this provides meaningful evidence of EchoSpec's practical advantages in target-specific adaptation.

Serving efficiency under long-context, high-concurrency workloads.

Our work focuses on long-context and high-concurrency workloads, where the memory overhead of separate drafters can offset their speculative decoding benefits. We acknowledge that EchoSpec's throughput advantage depends on the workload and available resources. However, our evaluations identify practically relevant regimes in which preserving target KV-cache capacity enables target-integrated drafting to outperform both AR decoding and separate-drafter baselines. These findings provide a useful perspective on the role of memory efficiency in speculative decoding.

The additional evaluations included in our revised manuscript provide evidence of this advantage. Our H100 GPU results (Figure 6 of the revised manuscript) demonstrate that EchoSpec achieves 21.2% higher output throughput than DFlash and 12.3% higher than AR for Qwen3-8B on MultiNews (8K+) inputs at concurrency 32. We also evaluate LongCodeCompletion and RepoBench-P. As shown in Figure 8 of the revised manuscript, EchoSpec consistently outperforms AR, DFlash, and EAGLE-3 across all high concurrency levels (8-32) and context lengths. These results further demonstrate that EchoSpec's serving advantages extend to newer GPU architectures and workloads beyond long-document summarization.

We believe that the key strengths of EchoSpec remain substantial and that its contributions are sufficiently meaningful to warrant dissemination to the research community. Given the additional experimental evidence and the clarifications provided in our revised manuscript, we sincerely hope the reviewer will reconsider increasing their original rating.

Thank you again for your time and thoughtful evaluation.

Reviewer_ixXhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EchoSpec proposes a target-model-integrated speculative decoding framework that reuses the frozen target model for draft generation, avoiding the need for independent auxiliary Transformers. The method introduces two key techniques: (1) O-bias — applying a learnable bias vector only at mask positions on the attention output projection WO, using minimal parameters to guide the target model to generate future tokens; (2) Post-target Refinement Block — a lightweight Transformer block that leverages the bonus token to refine draft representations after verification, compensating for information gaps during parallel drafting. The method requires only 2.2–4.5K training samples, 27.9–32.1× fewer trainable parameters than DFlash, and eliminates the need for an independent draft KV cache, yielding clear throughput advantages under high concurrency and long-context settings.

Critical strengths
  1. Unlike EAGLE-3, DFlash, and other approaches that introduce independent auxiliary Transformers, EchoSpec adheres to a "target-reuse" principle, compressing all target-model modifications down to a bias vector on the WO projection. The paper systematically evaluates adaptation sites across WQ, WK, WV, and WO, revealing that modifying WO alone achieves the highest parameter efficiency—outperforming combinations involving the other three projections. This finding provides valuable guidance for the design of future target-integrated speculative decoding systems. Additionally, the Post-target Refinement Block design cleverly exploits bonus token information that only becomes available after verification, addressing a critical information gap in parallel drafting scenarios.

  2. Through four dimensions — KV cache capacity, active memory, queued request count, and output throughput — the paper clearly demonstrates EchoSpec's advantages in high-concurrency, long-context settings: KV cache capacity matches standard autoregressive decoding (99.9%), while DFlash/EAGLE-3 drop to 83–94% due to additional state overhead; at concurrency 16–32, EchoSpec achieves 6.8% higher throughput than DFlash, maintaining this advantage up to concurrency 24 (DFlash only sustains it to concurrency 8). This complete evidence chain has direct reference value for production deployment.

Critical weaknesses
  1. The paper tests only three models — Qwen3-8B, Qwen3-4B, and Llama-3.1-8B-Instruct — all at most 8B parameters. There is zero coverage of modern LLM architectures such as MoE (e.g., Mixtral, DeepSeek-V3, Qwen3-MoE) or SSM (e.g., Mamba hybrid architectures), and applicability to 70B+, 100B+, or trillion-parameter-scale models remains entirely unknown. In today's LLM ecosystem, 8B models represent a relatively limited deployment scenario, leaving the method's effectiveness at larger scales and on more architectures completely unverified.

  2. The experiments are conducted only on NVIDIA A100 (80GB/40GB), with no fine-grained modeling of hardware characteristics such as HBM bandwidth, compute intensity, or kernel efficiency. The absence of validation on newer hardware such as H100/H20 leaves unexplained why the concurrency advantage saturates in the 16–32 range.

  3. Although concurrency experiments (concurrency 1–32) are provided, how the computational overhead of EchoSpec's refinement block scales with batch size, and the latency characteristics of O-bias under batch size> 1, are both unquantified. More critically, the paper acknowledges in Appendix B that EAGLE-3 exceeds the CUDA Graph memory budget at concurrency 28/32 for Llama-3.1-8B-Instruct in 8K+ settings, and EchoSpec itself faces similar risks — this critical engineering constraint on production deployment is discussed but not resolved.

Questions for authors

See weakness.

3 comments

W1. Scalability to larger models and modern architectures.

We thank the reviewer for their thoughtful comment. To address the concerns regarding scalability and architectural diversity, we have extended our evaluation to Gemma-4-31B-it. Specifically, Gemma-4-31B-it employs a modern hybrid attention architecture that interleaves sliding-window attention with full global attention. This allows us to validate EchoSpec at a much larger model scale and under a distinct attention mechanism. The table below reports output throughput (tokens/s) across different concurrency levels, as well as KV-cache token capacity.

TaskMethodConcurrency 1Concurrency 4Concurrency 8Concurrency 16Token capacity
GovReport (4K–8K)AR18.337.436.236.234.8K
GovReport (4K–8K)DFlash31.545.745.945.722.3K
GovReport (4K–8K)EchoSpec26.245.045.145.334.8K
MultiNews (4K–8K)AR17.934.534.034.034.8K
MultiNews (4K–8K)DFlash27.840.440.440.522.3K
MultiNews (4K–8K)EchoSpec24.741.841.842.234.8K

The new results demonstrate that EchoSpec maintains strong serving efficiency on this larger-scale model. On GovReport, EchoSpec achieves nearly identical output throughput to DFlash (within a 2%2\% margin) at concurrencies 4-16. On MultiNews, EchoSpec outperforms DFlash by 3.5%-4.2% in output throughput at concurrencies 4-16. Furthermore, EchoSpec provides 1.56×1.56\times higher KV-cache token capacity than DFlash, preserving critical memory headroom for long-context serving.

We have also added these experimental results to Appendix C of the revised manuscript.

W2. Throughput generalization on H100 GPU.

We agree that absolute inference throughput can vary across hardware platforms. However, hardware characteristics such as HBM bandwidth, compute intensity, and kernel efficiency affect both target verification and drafting workloads for all speculative decoding (SD) methods. Therefore, we expect the relative performance trends among SD methods to remain largely consistent across different hardware platforms. To empirically validate this, we conducted additional experiments on an NVIDIA H100 (80GB) GPU under the same serving protocol as in the original experiments. The two tables below reports output throughput (tokens/s) across different concurrency levels.

GovReport

ModelMethodConcurrency 1Concurrency 4Concurrency 8Concurrency 16Concurrency 32
Qwen3-8BAR106263351418444
Qwen3-8BDFlash134306389420429
Qwen3-8BEAGLE-3146274324333328
Qwen3-8BEchoSpec155336430468485
Llama3.1-8B-InstructAR112281375450476
Llama3.1-8B-InstructDFlash160361451498514
Llama3.1-8B-InstructEAGLE-382172215224OOM
Llama3.1-8B-InstructEchoSpec176383476528539

MultiNews

ModelMethodConcurrency 1Concurrency 4Concurrency 8Concurrency 16Concurrency 32
Qwen3-8BAR107260342412423
Qwen3-8BDFlash122286365395392
Qwen3-8BEAGLE-3134253303314320
Qwen3-8BEchoSpec145316409450475
Llama3.1-8B-InstructAR113279381457471
Llama3.1-8B-InstructDFlash150346446492517
Llama3.1-8B-InstructEAGLE-380166209220OOM
Llama3.1-8B-InstructEchoSpec169370474522546

As shown in the two tables, EchoSpec continues to outperform the competing SD methods in output throughput, particularly under high concurrency, consistent with the results observed on an A100 GPU. These results demonstrate that EchoSpec's throughput advantage is not specific to the A100 and generalizes to newer GPU architectures.

We have also added these experimental results to Section 4.2 of the revised manuscript.

W3. Computational overhead and CUDA graph memory analysis.

We thank the reviewer for their comment. To address the concerns, we provide additional profiling results and clarify the CUDA Graph out-of-memory (OOM) issue.

Profiling of O-bias and refinement block overhead.

To quantify the computational overhead of EchoSpec, we profiled the execution time of each stage across concurrency levels. We report the execution time and percentage of total latency for the target forward pass, O-bias, the refinement block, and other operations in the two tables below. We denote the target forward pass as "Target" and the refinement block as "Refine".

Qwen3-8B

StageConcurrency 1Concurrency 4Concurrency 8Concurrency 16Concurrency 32
Target12.13ms (87.6%)15.62ms (89.3%)20.61ms (91.4%)37.81ms (94.4%)72.81ms (96.1%)
O-bias0.30ms (2.2%)0.38ms (2.2%)0.41ms (1.8%)0.51ms (1.3%)0.75ms (1.0%)
Refine1.29ms (9.3%)1.34ms (7.7%)1.37ms (6.1%)1.49ms (3.7%)1.79ms (2.4%)
Other0.12ms (0.9%)0.14ms (0.8%)0.17ms (0.7%)0.25ms (0.6%)0.38ms (0.5%)
Total13.84ms17.48ms22.55ms40.07ms75.73ms

Llama3.1-8B-Instruct

StageConcurrency 1Concurrency 4Concurrency 8Concurrency 16Concurrency 32
Target11.88ms (88.1%)17.37ms (91.0%)21.86ms (92.3%)37.06ms (94.7%)69.31ms (96.3%)
O-bias0.27ms (2.0%)0.34ms (1.8%)0.37ms (1.5%)0.45ms (1.2%)0.67ms (0.9%)
Refine1.17ms (8.7%)1.22ms (6.4%)1.26ms (5.3%)1.35ms (3.5%)1.60ms (2.2%)
Other0.16ms (1.2%)0.17ms (0.9%)0.20ms (0.8%)0.26ms (0.7%)0.36ms (0.5%)
Total13.48ms19.09ms23.69ms39.13ms71.94ms

As shown in the two tables, the target forward pass accounts for the majority of total latency, while the overhead introduced by O-bias and the refinement block remains small. Specifically, for Qwen3-8B, the combined proportion of O-bias and the refinement block decreases from 11.5% to 3.4% as concurrency increases from 1 to 32. A similar trend is observed for Llama-3.1-8B-Instruct, where the proportion decreases from 10.7% to 3.2%. These results indicate that O-bias and the refinement block introduce limited additional computation and do not become a bottleneck at high concurrency.

We have also added these experimental results to Section 4.2 of the revised manuscript.

CUDA Graph OOM and memory consumption.

To further examine CUDA Graph memory consumption, we measured the memory usage (GiB) of individual execution stages, as shown in the table below, where cc denotes the concurrency level.

MethodTarget PrefillTarget VerifyDraft PrefillDraft DecodeDraft ExtendRefinement
EAGLE-3 (c=24c=24)2.040.511.541.990.42-
EAGLE-3 (c=28c=28)2.050.571.542.290.47-
EchoSpec (c=32c=32)1.611.37---0.72

EAGLE-3 requires additional CUDA Graph memory for draft prefill, decoding, and extension, whereas EchoSpec only incurs CUDA Graph memory overhead for the refinement stage beyond target prefill and verification. This additional allocation leaves less free device memory for subsequent runtime operations. Indeed, we observed a serving-time OOM in EAGLE-3 when a Triton KV-cache copy kernel was loaded after server initialization. By avoiding the extra CUDA Graph footprint of a separate drafter, EchoSpec preserves greater memory headroom for high-concurrency serving.

We have also included this memory breakdown in Appendix B. of the revised manuscript.

We sincerely thank the reviewer again for their time and consideration. As a gentle reminder, the author–reviewer discussion period ends on November 18, 2026. We would greatly appreciate it if the reviewer could check our responses and let us know if any concerns remain.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

3. Training cost.

(1) Training-data generation cost. Constructing a target-specific training dataset is an important part of speculative decoding's adaptation cost. Existing methods such as EAGLE-3 and DFlash rely on target-generated responses to achieve high draft quality, so new training data must be prepared whenever the target model changes. As shown in Table 1 of the revised manuscript, EAGLE-3 and DFlash use approximately 532K and 800K training sequences, respectively. In contrast, EchoSpec requires only 2.2K-4.5K sequences. While the actual generation cost depends on the dataset, response length, hardware, and implementation, even a comparison based solely on the reported sequence counts shows that EchoSpec requires substantially less training data. This smaller data requirement directly translates to a lower upfront cost for adapting new target models.

(2) Time and memory consumption during training. To assess training efficiency, we measured per-training-step latency and memory consumption on a single A100 (40GB) GPU for all three target models, as detailed below. We used SpecForge [1] implementations for EAGLE-3 and DFlash, and our own implementation for EchoSpec. All experiments used a batch size of 1. For each configuration, we preloaded 110 training samples into CPU memory, performed 10 warm-up steps, and measured the average step latency over the subsequent 100 steps. We estimated the training time per epoch by multiplying the measured average step latency by the reported number of training sequences. These estimates exclude the cost of training data generation and preprocessing, and therefore do not represent end-to-end adaptation costs.

TargetMethodTrain dataTime per stepEstimated time per epochPeak memory
Q3-4BEAGLE-3532K0.210 s31.03 hours8.51 GiB
DFlash800K0.276 s61.33 hours13.32 GiB
EchoSpec2.2K0.236 s0.14 hours11.38 GiB
Q3-8BEAGLE-3532K0.294 s43.45 hours12.79 GiB
DFlash800K0.430 s95.56 hours22.67 GiB
EchoSpec4.5K0.264 s0.33 hours19.32 GiB
L3.1-8BEAGLE-3532K0.288 s42.56 hours14.02 GiB
DFlash800K0.262 s58.22 hours21.51 GiB
EchoSpec2.2K0.206 s0.13 hours18.78 GiB

(Abbreviations used: Q3 for Qwen3; L3.1-8B for Llama-3.1-8B-Instruct)

As shown in the table above, EchoSpec does not consistently achieve the lowest per-step latency. Instead, its training-time advantage comes from the substantially smaller number of required training sequences. Based on the measured step times and reported sequence counts, EchoSpec is estimated to require only 0.13-0.33 GPU-hours per epoch. In contrast, the estimates for EAGLE-3 and DFlash are tens of GPU-hours. Furthermore, under our training schedule of 2-4 epochs, EchoSpec's estimated total training time remains below one GPU-hour. These results show that EchoSpec substantially reduces the estimated model-training time, primarily due to its smaller training dataset rather than faster individual training steps.

Furthermore, we clarify that EchoSpec's memory-efficiency claim concerns inference-time serving rather than training. EAGLE-3 and DFlash use precomputed target hidden states, allowing the auxiliary drafters to be trained without loading the target model into GPU memory. However, this approach requires additional disk space to store the precomputed hidden states. In contrast, EchoSpec does not require storing precomputed target hidden states; instead, it performs forward passes through the frozen target model during training. Although EchoSpec requires the frozen target model to reside in GPU memory during training, its peak memory usage ranges from 11.38 to 19.32 GiB across the three targets, as shown in the table above. Thus, it can be easily trained on a single A100 (40GB) GPU, the smallest-memory GPU configuration used in our serving experiments.

We have also added these results to Appendix D of the revised manuscript.


Reference

[1] SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026)

AuthorsOpenReview ↗

General Response (1/2)

We sincerely thank the reviewers for their feedback. To address the reviewers' concerns, we have revised the manuscript and highlighted the changes in blue. We provide detailed responses to each reviewer in the corresponding reply threads. Because Reviewers rAfL and KtVo raised similar questions regarding the refinement block and training cost, we address these shared points below.

1. Necessity of the refinement block.

We agree that the refinement block is not strictly required for target-integrated drafting. However, the refinement block is designed to further improve draft quality by incorporating the bonus token, which is inherently unavailable during the target's parallel forward pass. Although this block accounts for most of EchoSpec's trainable parameters, EchoSpec as a whole still uses 27.927.9-32.1×32.1\times fewer trainable parameters than DFlash and 11.411.4-12.4×12.4\times fewer than EAGLE-3.

To ablate the refinement block and assess its effect on output throughput, we compared EchoSpec with and without the block under the same training configurations. As shown in Table 5 of the revised manuscript, the refinement block improves acceptance length by 1.61.6-2.62.6% and output throughput by 1.31.3-2.22.2% on the Qwen3 targets at concurrencies of 1616-3232. For Llama-3.1-8B-Instruct, the refinement block also improves acceptance length with a neutral impact on throughput. These results demonstrate that the refinement block successfully improves draft quality without incurring a throughput penalty, and can actively enhance overall serving throughput.

2. Ablation study of the benefit of bonus-token information in the refinement block.

To address the questions raised by Reviewers rAfL and KtVo regarding the contribution of bonus-token conditioning, we conducted an additional ablation study that retains the refinement block but removes the bonus-token input. The table below reports Recall@1 at each draft horizon and the proxy acceptance length (τ^\hat{\tau}). We denote EchoSpec without the refinement block as "none", EchoSpec with the refinement block but without bonus-token conditioning as "no bonus", and the original EchoSpec as "block".

HorizonQ3-8BQ3-4BL3.1-8B
noneno bonusblocknoneno bonusblocknoneno bonusblock
10.6880.7400.7750.6580.7180.7400.7750.7860.814
20.4800.5720.5970.4380.5420.5550.5960.6300.648
30.3420.4410.4540.2890.4060.4140.4560.5000.512
40.2650.3300.3330.2150.2870.2870.3480.4010.404
50.2110.2730.2790.1690.2280.2290.2740.3070.313
60.1670.2270.2290.1470.2020.2020.2220.2570.262
70.1470.1790.1830.1290.1630.1570.1840.2090.209
80.1240.1690.1620.1210.1340.1360.1700.1750.174
τ^\hat{\tau}2.1692.4332.5422.0512.3232.3832.5452.6682.767

(Abbreviations used: Q3 for Qwen3; L3.1-8B for Llama-3.1-8B-Instruct)

As shown in the table above, the gain in proxy acceptance length from conditioning on the bonus token accounts for up to 44.644.6% of the total improvement achieved by the full refinement block over O-bias alone (i.e., comparing "block" to "none"). These results demonstrate a dual benefit. While the refinement block itself improves draft quality ("no bonus"), explicitly conditioning on the bonus token yields a further performance boost.

We have also added these results to Section 3.3 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.